Human Detection

robot.perception is a plugin namespace exposing 3-D-aware human detection — bounding boxes, pose keypoints, face orientation, per-keypoint 3-D position, and an engagement score, for every person in view. These examples assume you already have a connected robot — see Connection if you haven't set one up yet.

How it works

Unlike the core capabilities, human detection isn't served by the main service hub on QTRP — raw person/pose detection comes from qtrobot-yolo-driver, a separate service that consumes the RGB (and depth) stream from qtrobot-realsense-driver and publishes a /persons stream. The human-detector plugin runs locally, in-process with your SDK client: it subscribes to /persons, enriches each frame with face yaw/pitch, a 3-D position per keypoint (using the robot's current head angles), an optional voice-activity flag, and an engagement score, then republishes the result as robot.perception's /human/presence stream.

Because it runs locally, the plugin also subscribes to robot.motor's joint-state stream on its own, so 3-D positions account for the robot's current head pose without any extra wiring on your side.

Setup

mkdir ~/example
cd ~/example
python -m venv .venv
# or: uv venv .venv

source .venv/bin/activate

pip install luxai-robot
# or: uv pip install luxai-robot

Connect

from luxai.robot.core import Robot

robot = Robot.connect_zmq(robot_id="QTRD000123")
print(f"connected to {robot.robot_id} ({robot.robot_type})")

See Connection for MQTT, WebRTC, and other connection options.

Enable and configure the plugin

The plugin runs locally, so enable it with enable_plugin_local, then point it at the running qtrobot-yolo-driver — the /persons stream endpoint is discovered automatically from its RPC endpoint:

robot.enable_plugin_local("human-detector")

ok = robot.perception.configure_human_detector(
    endpoint="tcp://10.231.0.1:50770",  # qtrobot-yolo-driver RPC endpoint
    default_depth=1.0,   # fallback depth (m) when a keypoint has no valid depth reading
    use_vad=False,       # set True to add a voice.speaking field (requires torch)
)
if not ok:
    Logger.error("Failed to configure human detector. Check that qtrobot-yolo-driver is running.")

endpoint can be left empty in favour of node_id (default "qtrobot-yolo-driver") to discover the driver via Zeroconf instead.

Subscribe to presence frames

def on_presence(frame):
    persons = frame.value.get("persons", {})
    for pid, p in persons.items():
        face = p.get("face") or {}
        nose = p.get("keypoints", {}).get("nose") or {}
        Logger.info(
            f"person {pid}: confidence={p.get('confidence', 0):.2f} "
            f"face yaw={face.get('yaw')}° pitch={face.get('pitch')}° "
            f"nose xyz={nose.get('xyz')} engagement={p.get('engagement', 0):.2f}"
        )

robot.perception.stream.on_human_presence(on_presence)

Each frame carries a persons dict keyed by a stable person ID:

Field	Description
bbox	Person bounding box in the color image.
confidence	Detection confidence (0-1).
keypoints	COCO-17 pose keypoints, each with uv (pixel), conf, depth (m), and xyz (3-D point in the robot base frame, when depth is valid).
face.yaw / face.pitch	Estimated face orientation in degrees, derived from keypoints.
engagement	Smoothed 0-1 score combining how directly the person faces the camera and how close they are.
voice	Only present when use_vad=True: {"speaking": bool, "score": float} — score is that person's share of who's most likely speaking.

Look at the most engaged person

Combine robot.perception with robot.kinematics to make the robot look at whoever is most engaged:

robot.enable_plugin_local("kinematics")

def on_presence_look(frame):
    persons = frame.value.get("persons", {})
    if not persons:
        return
    best = max(persons.values(), key=lambda p: p.get("engagement", 0))
    xyz = (best.get("keypoints", {}).get("nose") or {}).get("xyz")
    if xyz:
        robot.kinematics.look_at_point(*xyz, only_gaze=False)

robot.perception.stream.on_human_presence(on_presence_look)

See Kinematics for more on look_at_point and the robot base frame.

Annotated image stream

For debugging, the driver can also publish an annotated color image (pose skeleton, head-orientation arrow, 3-D position label) when annotated-image streaming is enabled on the driver side. Each frame's .data is a raw BGR pixel buffer — exactly the layout OpenCV expects, so no color conversion is needed:

pip install opencv-python numpy

import cv2
import numpy as np

reader = robot.perception.stream.open_human_annotated_image_reader()

while True:
    frame = reader.read(timeout=2.0)
    if frame is None:
        continue

    image = np.frombuffer(frame.data, dtype=np.uint8).reshape(frame.height, frame.width, frame.channels)
    cv2.imshow("Human Detection", image)

    if cv2.waitKey(1) & 0xFF == ord("q"):
        break

cv2.destroyAllWindows()

Next steps

Continue with the Plugins tutorial, or see the full robot.perception namespace in the Python API Reference.