Skip to main content
Version: QTrobot V3

Video

robot.media plays video on the same two lanes used for audio — foreground (FG) and background (BG) — from files or as a raw streamed RGBA signal. These examples assume you already have a connected robot — see Connection if you haven't set one up yet.

How the FG/BG video system works​

Video uses the same two-lane idea as Audio, but the lanes aren't symmetric, and there's alpha (transparency) control instead of volume:

  • BG is the opaque base layer — a plain image, no transparency.
  • FG sits on top of BG and is alpha-blended over it. robot.media.set_fg_video_alpha() controls how much of the FG layer shows through; there's no separate BG alpha, since BG is always the fully-opaque bottom layer.

Just like audio, each lane accepts a file and a stream input, and file playback takes priority — while a file is playing on a lane, frames arriving on that lane's stream are dropped, and the lane reverts to the stream once the file finishes.

An important difference from audio: there's no video equivalent of TTS using FG — it's the opposite. QTrobot's own face system uses the BG lane: emotions (robot.face.show_emotion(...)) play as a BG file, and a continuous live face rendering is pushed to the BG stream whenever no emotion file is playing. The FG lane is left free for you — anything you play or stream on FG is alpha-blended on top of whatever the face is currently showing on BG.

Creative possibilities​

Because FG always overlays whatever's on BG, you can layer your own content on top of the robot's normal face/emotions without replacing them:

  • Reactive overlays — show a small icon, sticker, or shape on FG (like the pulsing heart below) that fades in and out over an emotion or the idle face, using set_fg_video_alpha() for the fade.
  • Captions or visual cues — overlay text or a simple graphic on FG while an emotion plays on BG underneath, for accessibility or for a teaching/story context.
  • Speech-synced effects — combine FG video (a custom overlay) with FG audio (TTS) for a richer "speaking" effect, while BG continues showing the robot's current emotion or idle face.
  • Smooth transitions — fade an FG overlay in just as a BG emotion finishes, to bridge between two visual states instead of an abrupt cut.

Setup​

mkdir ~/example
cd ~/example
python -m venv .venv
# or: uv venv .venv

source .venv/bin/activate

pip install luxai-robot
# or: uv pip install luxai-robot

Connect​

from luxai.robot.core import Robot

robot = Robot.connect_zmq(robot_id="QTRD000123")
print(f"connected to {robot.robot_id} ({robot.robot_type})")

See Connection for MQTT, WebRTC, and other connection options.

Foreground alpha (transparency)​

The foreground video lane sits on top of the background lane — its alpha controls how transparent it is.

import time

# Fully opaque
robot.media.set_fg_video_alpha(1.0)

# Half-transparent
robot.media.set_fg_video_alpha(0.5)
time.sleep(2)

# Restore to fully opaque
robot.media.set_fg_video_alpha(1.0)

Play a background video file​

A file path must exist on the robot itself (not on your laptop) for these examples to work.

import time

video_file_on_robot = "/home/qtrobot/robot/data/emotions/QT/kiss.avi"

# Play and wait for it to finish
ret = robot.media.play_bg_video_file(video_file_on_robot)
Logger.info(f"Done. Result: {ret}")

# Play non-blocking, then cancel it after 2 seconds
h = robot.media.play_bg_video_file_async(video_file_on_robot)
time.sleep(2)
h.cancel()

Pause and resume​

import time

video_file_on_robot = "/home/qtrobot/robot/data/emotions/QT/kiss.avi"

play_handler = robot.media.play_bg_video_file_async(video_file_on_robot)
time.sleep(2)

robot.media.pause_bg_video_file()
Logger.info("Paused. Waiting 3 seconds...")
time.sleep(3)

robot.media.resume_bg_video_file()
play_handler.wait()  # wait for playback to finish

Stream raw video frames​

This example streams a pulsing heart shape to the foreground lane, fading its alpha in time with a heartbeat.

import time
import math
import numpy as np
from luxai.magpie.utils.common import get_uinque_id
from luxai.magpie.frames import ImageFrameRaw

def _make_heart_frame(width, height, scale=1.5):
    image = np.zeros((height, width, 4), dtype=np.uint8)  # transparent RGBA

    y, x = np.ogrid[:height, :width]
    x = (x - width / 2) / (width / 2)
    y = -((y - height / 2) / (height / 2))  # flip vertically
    x *= scale
    y *= scale

    heart_mask = (x**2 + y**2 - 1)**3 - x**2 * y**3 <= 0
    image[heart_mask] = (255, 0, 0, 255)  # red, full alpha

    return ImageFrameRaw(
        data=image.tobytes(), format="raw",
        width=width, height=height, channels=4, pixel_format="RGBA",
    )

stream_id = get_uinque_id()
fps = 30
duration_s = 5.0
total_frames = int(fps * duration_s)
frame_interval = 1.0 / fps

heart_frame = _make_heart_frame(width=400, height=280, scale=5.0)
heart_frame.gid = stream_id

bpm = 72.0
freq = bpm / 60.0  # beats per second

robot.media.set_fg_video_alpha(0.0)
writer = robot.media.stream.open_fg_video_stream_writer()

t0 = time.perf_counter()
for frame_id in range(1, total_frames + 1):
    heart_frame.id = frame_id
    writer.write(heart_frame)

    t = time.perf_counter() - t0
    alpha = 0.5 * (1 + math.sin(2 * math.pi * freq * t))
    alpha = alpha ** 2  # makes it feel more like a "beat"
    robot.media.set_fg_video_alpha(alpha)
    time.sleep(frame_interval)

robot.media.set_fg_video_alpha(1.0)
robot.media.cancel_fg_video_stream()

Next steps​

Continue with the Microphone tutorial, or see the full robot.media namespace in the Python API Reference.