Camera

robot.camera is a plugin namespace exposing QTrobot's RealSense 3D camera — color, depth, and IMU streams, plus intrinsics. It must be enabled before use. These examples assume you already have a connected robot — see Connection if you haven't set one up yet.

How the camera system works

Unlike the core capabilities (TTS, face, motors, ...), the camera isn't served by the main service hub on QTRP — it's served by a separate process, qtrobot-realsense-driver, running on QTPC. That's exactly why it's a plugin: robot.camera only becomes available once you explicitly point the SDK at this separate driver with enable_plugin_zmq(...).

The driver talks to the physical RealSense camera and republishes its data over the same Magpie/ZMQ pattern used everywhere else in QTrobot V3 — a few RPC calls (intrinsics, depth scale) plus several independent streams (color, depth, IMU), each individually enable-able.

The driver runs as its own systemd service on QTPC, independent of the main service hub:

sudo systemctl status qtrobot-realsense-driver
sudo systemctl restart qtrobot-realsense-driver
journalctl -u qtrobot-realsense-driver -f   # view logs

Its base ZMQ port is 50750 — RPC calls go to that port directly, while RGB, depth, and IMU each get their own stream port (50751, 50752, 50753) under the hood. You don't need to know the individual stream ports — enable_plugin_zmq("realsense-driver", endpoint="tcp://<QTPC-ip>:50750") discovers all of it from the base port, the same way every other plugin works.

What's streamed by default

Not every stream is turned on out of the box — enabling unused sensors costs CPU and bandwidth, so the driver ships with a conservative default:

Stream	Default	Resolution / rate
RGB (color)	On	848×480 @ 15 fps
Depth (depth, depth_aligned)	On	640×480 @ 15 fps
Colorized depth (depth_color)	Off	—
Gyroscope (gyro)	Off	200 Hz
Accelerometer (acceleration)	Off	63 Hz

What "aligned depth" means: the RGB and depth sensors sit at slightly different physical positions on the camera, so a raw depth pixel doesn't line up with the same pixel in the raw color image. The depth_aligned stream re-projects the depth data into the color camera's point of view, so pixel (x, y) in depth_aligned corresponds to the same (x, y) in color — this is what you want whenever you need to know "how far away is the thing at this point in the color image" (see Get the distance at a pixel below). The plain depth stream, by contrast, is in the depth sensor's own native viewpoint.

Enabling more features

To turn on something that's off by default (colorized depth, gyro, accelerometer, or to change resolution/frame rate), edit the driver's config on QTPC and restart it:

sudo nano /opt/luxai/qtrobot_realsense_driver/etc/config.yaml

use-color-depth: true   # enable colorized depth stream
use-gyro: true          # enable gyroscope
use-accel: true         # enable accelerometer

sudo systemctl restart qtrobot-realsense-driver

The same file also controls RGB/depth resolution and frame rate (rgb-width, rgb-height, rgb-fps, depth-width, depth-height, depth-fps) if you need to trade off quality for bandwidth or CPU.

Setup

mkdir ~/example
cd ~/example
python -m venv .venv
# or: uv venv .venv

source .venv/bin/activate

pip install luxai-robot
# or: uv pip install luxai-robot

Connect

from luxai.robot.core import Robot

robot = Robot.connect_zmq(robot_id="QTRD000123")
print(f"connected to {robot.robot_id} ({robot.robot_type})")

See Connection for MQTT, WebRTC, and other connection options.

Enable the plugin

robot.enable_plugin_zmq("realsense-driver", endpoint="tcp://10.231.0.1:50750")

Intrinsics and depth scale

color_intrinsics = robot.camera.get_color_intrinsics()
Logger.info(f"Color intrinsics: {color_intrinsics}")

depth_intrinsics = robot.camera.get_depth_intrinsics()
Logger.info(f"Depth intrinsics: {depth_intrinsics}")

depth_scale = robot.camera.get_depth_scale()
Logger.info(f"Depth scale: {depth_scale}")

depth_scale is the number of metres represented by one raw depth unit — multiply any raw depth pixel value by it to get a real-world distance, as used in the pixel-distance example below.

Streams

# Read a single color frame
reader = robot.camera.stream.open_color_reader()
frame = reader.read(timeout=3.0)
Logger.info(frame)

# Or subscribe to the accelerometer stream via callback
from luxai.magpie.frames import ListFrame

def on_acceleration_callback(frame: ListFrame):
    Logger.info(f"Acceleration frame received: {frame.value}")

robot.camera.stream.on_acceleration(on_acceleration_callback)

The same pattern (open_*_reader() or on_*(callback)) is available for depth, depth_aligned, depth_color, and gyro — see the Python API Reference for the full list.

Show a live color stream with OpenCV

Each color frame's .data is a raw BGR pixel buffer — exactly the layout OpenCV expects, so no color conversion is needed.

pip install opencv-python numpy

import cv2
import numpy as np

reader = robot.camera.stream.open_color_reader()

while True:
    frame = reader.read(timeout=2.0)
    if frame is None:
        continue

    image = np.frombuffer(frame.data, dtype=np.uint8).reshape(frame.height, frame.width, frame.channels)
    cv2.imshow("QTrobot Camera", image)

    if cv2.waitKey(1) & 0xFF == ord("q"):
        break

cv2.destroyAllWindows()

Get the distance at a pixel

Combining the depth_aligned stream with depth_scale tells you how far away the thing is at any given pixel of the color image — useful for "how far is the person/object I'm looking at" style logic.

import numpy as np

depth_scale = robot.camera.get_depth_scale()["scale"]
reader = robot.camera.stream.open_depth_aligned_reader()

frame = reader.read(timeout=3.0)
depth_image = np.frombuffer(frame.data, dtype=np.uint16).reshape(frame.height, frame.width)

# Distance at the center pixel, in metres
cx, cy = frame.width // 2, frame.height // 2
raw_value = depth_image[cy, cx]
distance_m = raw_value * depth_scale
Logger.info(f"Distance at center pixel: {distance_m:.2f} m")

Because depth_aligned shares the color image's coordinate system, you can reuse the exact same (x, y) from anything you detect in the color stream — e.g. a face or object bounding box center — to look up its distance here.

Next steps

Continue with the ASR tutorial, or see the full robot.camera namespace in the Python API Reference.