Add a camera#

Prerequisites: camera input requirements, Python, and NumPy. ROS is needed only for the topic-adaptation route. Outcome: determine whether configuration is sufficient, then exercise a recorded-array camera without hardware.

1. Try topic configuration first#

A camera publishing compatible ROS 2 images and point clouds often needs no new class. Set full topic names in .env:

CAMERA_BACKEND=ros2
CAMERA_COLOR_TOPIC=/camera/rgb/image_rect
CAMERA_POINT_CLOUD_TOPIC=/camera/points_in_robot_base

Replace these example names with your camera’s topics and restart GTA. Confirm the image encoding, aligned organized cloud, base frame, field layout, and units using the readiness checks.

2. Learn the interface with recorded arrays#

docs/examples/array_camera.py — recorded-array camera example#
"""A memory-only camera example; no files, ROS, or SDK connections."""

import numpy as np

from scene.camera.base import BaseCamera


class ArrayCamera(BaseCamera):
    def __init__(self, rgb, xyz=None):
        self._rgb = np.asarray(rgb).copy()
        if (self._rgb.ndim != 3 or self._rgb.shape[2] != 3
                or self._rgb.dtype != np.uint8):
            raise ValueError("RGB must be an H x W x 3 uint8 array")
        self._xyz = None if xyz is None else np.asarray(xyz, dtype=float).copy()
        if self._xyz is not None:
            if self._xyz.shape != self._rgb.shape or not np.isfinite(self._xyz).all():
                raise ValueError("XYZ must be finite H x W x 3, aligned with RGB")
        self._cached = None
        self._closed = False

    def _check_open(self):
        if self._closed:
            raise RuntimeError("Camera is closed")

    def get_rgb_image(self, use_new_frame=True):
        self._check_open()
        if use_new_frame or self._cached is None:
            self._cached = self._rgb.copy()
        return self._cached.copy()

    def get_point_cloud(self, save_points=False):
        self._check_open()
        if save_points:
            raise NotImplementedError("This memory camera does not save point clouds")
        if self._xyz is None:
            raise RuntimeError("This recording has no point cloud")
        # Already in base coordinates and metres. Preserve pixel order.
        return self._xyz.reshape(-1, 3).copy()

    def close(self):
        self._cached = None
        self._closed = True

From the repository root:

import numpy as np
from docs.examples.array_camera import ArrayCamera

rgb = np.zeros((2, 3, 3), dtype=np.uint8)
xyz = np.zeros((2, 3, 3), dtype=float)  # synthetic organized points, in metres
camera = ArrayCamera(rgb, xyz)
assert camera.get_rgb_image().shape == (2, 3, 3)
assert camera.get_point_cloud().shape == (6, 3)
camera.close()

get_rgb_image() returns RGB, not OpenCV’s usual BGR. Point ordering must retain the image-pixel correspondence. This example rejects non-finite points instead of silently deleting rows. Its cache is per instance and it raises when a point cloud was not supplied.

3. Register the camera backend#

Unlike the robot and gripper selectors, camera selection currently has a fixed set of supported names. A real recorded backend requires more than adding scene/camera/recorded.py:

  1. Implement the BaseCamera contract and resource cleanup. Keep imports hardware-free. Specify caching, freshness, timeout, and error behavior.

  2. Extend the CAMERA_BACKEND choices in config.py and the allowed set and import mapping in scene/camera/__init__.py.

  3. Update capture paths in agents/base_agent.py, agents/vision_module.py, and api/service.py. They currently distinguish file from the explicit ros1/ros2 names. A new backend must save captured images into the active workspace and must not silently fall through to file mode when capture fails.

  4. Review gta/utils/sensors.py, backend health reporting, and error messages. Generated scripts must receive the same RGB/cloud geometry as planning.

  5. Add fake-backend tests for fresh capture, cache reuse, conversion, timeout, missing dependencies, and library-workspace isolation.

See scene/camera/file.py and scene/camera/ros2.py for the existing image-file and live-camera implementations.

4. Verify end-to-end geometry#

An image-only adapter can support planning but cannot support GTA’s 3D execution preparation. For a full adapter, test pixel-to-point indexing with known recorded data, not just matching array shapes. Invalid depth must not shift every later pixel. Calibration and organized-cloud assumptions are part of the integration, not automatic properties of PointCloud2.

Verify: run the documentation example tests and existing camera tests. For a timeout, wrong annotation, or distorted 3D point see camera and grounding troubleshooting.

Download the offline example