Video encoding parameters#

本节介绍 LeRobot 的通用功能。请按目标机器人、数据集和运行环境配置示例；用于 AlohaMini 2 / 2 Pro 前，需要确认硬件与接口兼容性。

When video storage is enabled, LeRobot stores each camera stream as an MP4 file instead of saving one image file per timestep. Video encoding compresses across time, which usually cuts dataset size and I/O compared to a pile of PNG, while keeping MP4 — a format every player and loader understands.

Encoding frames into an MP4 is a full FFmpeg pipeline: choice of encoder, pixel format, GOP/keyframes, quality vs. speed, and optional extra encoder flags. Most of these knobs are user-tunable through rgb_encoder, a nested RGBEncoderConfig (lerobot.configs.video.RGBEncoderConfig) passed through PyAV.

You can set these parameters from the CLI with --dataset.rgb_encoder.<field> (e.g. with lerobot-record or lerobot-rollout). The same block applies to every camera video stream in that run.

[!TIP] Video storage must be on for rgb_encoder to have any effect — use_videos=True in Python APIs, or --dataset.video=true on the CLI (the recording default). With video off, inputs stay as images and rgb_encoder is ignored.

For details on when frames are written vs. encoded (streaming vs. post-episode), queues, and other top-level --dataset.* switches, see Streaming Video Encoding. For an encoding-parameter comparison and experiments, see the video-benchmark Space.

Example#

lerobot-record \
    --robot.type=so100_follower \
    --robot.port=/dev/tty.usbmodem58760431541 \
    --robot.cameras="{laptop: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}}" \
    --robot.id=black \
    --teleop.type=so100_leader \
    --teleop.port=/dev/tty.usbmodem58760431551 \
    --teleop.id=blue \
    --dataset.repo_id=<my_username>/<my_dataset_name> \
    --dataset.num_episodes=2 \
    --dataset.single_task="Grab the cube" \
    --dataset.streaming_encoding=true \
    --dataset.encoder_threads=2 \
    --dataset.rgb_encoder.vcodec=h264 \
    --dataset.rgb_encoder.preset=fast \
    --dataset.rgb_encoder.extra_options={"tune": "film", "profile:v": "high", "bf": 2} \
    --display_data=true

Tuning parameters#

[!WARNING] The defaults are tuned to balance compression ratio, visual quality, and decoding/seek speed for typical robotics datasets. Changing them can affect both recording (CPU load, frame drops) and training (decoding throughput, image quality).

Only override these parameters if you have a specific reason to, and measure the impact on your pipeline before relying on the new settings.

All flags below are prefixed with --dataset.rgb_encoder. on the CLI.

Parameter	Type	Default	Description
`vcodec`	`str`	`"libsvtav1"`	Video codec name. `"auto"` picks the first available hardware encoder from a fixed preference list, falling back to `libsvtav1`.
`pix_fmt`	`str`	`"yuv420p"`	Output pixel format. Must be supported by the chosen codec in your FFmpeg build.
`g`	`int`	`2`	GOP size — a keyframe every `g` frames. Emitted as FFmpeg option `g`.
`crf`	`int` or `float`	`30`	Abstract quality value, mapped per codec (see the mapping below). Lower → higher quality / larger output where the mapping is monotone.
`preset`	`int` or `str`	`12` *	Encoder speed preset; meaning depends on the codec. * When unset and `vcodec=libsvtav1`, LeRobot defaults to `12`.
`fast_decode`	`int`	`0`	`libsvtav1`: `0–2`, passed via `svtav1-params`. `h264` / `hevc` (software): if `>0`, sets `tune=fastdecode`. Other codecs: usually unused.
`video_backend`	`str`	`"pyav"`	Only `"pyav"` is currently implemented for video encoding.
`extra_options`	`dict`	`{}`	Extra FFmpeg or codec specific options merged after the structured fields above. Cannot override keys already set by those fields.

Depth streams#

Depth maps (Intel RealSense, Reachy 2) are stored as their own video streams alongside the RGB streams. Raw depth (uint16 millimetres or float32 metres) can't survive an 8-bit codec, so LeRobot quantizes each map to a 12-bit code ([0, 4095]) — logarithmically by default, to match the 1/depth error profile of depth sensors — then packs it into a high-bit-depth pixel format (gray12le) and encodes it with a 12-bit codec.

Raw depth uint16 mm
float32 m →

Record time Clip to [depth_min,
depth_max] → Quantize 12-bit codes 0–4095
log (default) or linear → Pack into gray12le
plane → Encode HEVC
Main 12

→ MP4 stored
stream →

Load time Dequantize to mm / m

Configure the depth pipeline through a parallel depth_encoder block (DepthEncoderConfig). It shares every RGBEncoderConfig field (vcodec, pix_fmt, crf, …) and adds four quantizer knobs, set via --dataset.depth_encoder.<field>:

lerobot-record \
    ... \
    --dataset.depth_encoder.vcodec=hevc \
    --dataset.depth_encoder.depth_min=0.05 \
    --dataset.depth_encoder.depth_max=5.0 \
    --dataset.depth_encoder.use_log=true

Parameter	Type	Default	Description
`vcodec`	`str`	`"hevc"`	HEVC Main 12 (a 12-bit-capable codec, MP4-compatible).
`extra_options`	`dict`	`{"x265-params": "lossless=1"}`	Depth defaults to lossless (exact round-trip); `crf` is ignored. Pass `extra_options={}` and set `crf` for a smaller lossy stream.
`pix_fmt`	`str`	`"gray12le"`	Single-channel 12-bit pixel format used to carry the quantized codes.
`depth_min`	`float`	`0.01`	Depth in metres mapped to quantum `0`. Values below are clipped on decode.
`depth_max`	`float`	`10.0`	Depth in metres mapped to quantum `4095`. Values above are clipped on decode.
`shift`	`float`	`3.5`	Pre-log offset (metres) used in logarithmic quantization for numerical stability near zero. Must satisfy `depth_min + shift > 0`.
`use_log`	`bool`	`True`	If `true`, quantize in log-space (recommended for typical depth sensors). Set to `false` for uniform/linear quantization.

[!TIP] depth_min, depth_max, and shift are always interpreted in metres, regardless of the input depth's unit. Inputs are auto-detected: integer arrays (e.g. uint16 millimetres straight from a RealSense) are treated as millimetres, floating arrays as metres. Pick depth_min / depth_max to bracket the actual working range of your sensor — quanta outside that range saturate, which can crush detail at the boundaries.

Depth features are flagged with "is_depth_map": true in meta/info.json, and their quantizer settings (video.depth_min, video.depth_max, video.shift, video.use_log) are persisted — which is what lets depth be dequantized back to physical units on load.

Output unit at load time#

depth_encoder is a record-time concern. The unit that depth maps are dequantized to on load (e.g. during training) is set separately by the read-time flag --dataset.depth_output_unit:

lerobot-train \
    --dataset.repo_id=<my_username>/<my_dataset_name> \
    --dataset.depth_output_unit=m \
    --policy.type=act

Parameter	Type	Default	Description
`depth_output_unit`	`str`	`"mm"`	Physical unit depth maps are dequantized to on load: `"mm"` (millimetres) or `"m"` (metres).

[!TIP] This is purely a decode-time presentation choice — it does not alter the stored video or its metadata, so the same dataset can be read as mm or m without re-encoding. It has no effect on datasets without depth cameras.

Persistence in dataset metadata#

After the first episode of a video stream is encoded, the encoder configuration is persisted into the dataset metadata (meta/info.json) under each video feature, alongside the values probed from the file itself. For a video feature observation.images.<camera>, the layout in info.json is:

{
  "features": {
    "observation.images.laptop": {
      "dtype": "video",
      "shape": [480, 640, 3],
      "info": {
        "video.height": 480,
        "video.width": 640,
        "video.codec": "h264",
        "video.pix_fmt": "yuv420p",
        "video.fps": 30,
        "video.channels": 3,
        "is_depth_map": false,
        "video.g": 2,
        "video.crf": 30,
        "video.preset": "fast",
        "video.fast_decode": 0,
        "video.video_backend": "pyav",
        "video.extra_options": { "tune": "film", "profile:v": "high", "bf": 2 }
      }
    }
  }
}

Two sources contribute to the info block:

Source	Where it comes from	Fields
Stream-derived	Read back from the encoded MP4 with PyAV.	`video.height`, `video.width`, `video.codec`, `video.pix_fmt`, `video.fps`, `video.channels`, `is_depth_map`, `audio.*`
Encoder-derived	Taken from `RGBEncoderConfig` / `DepthEncoderConfig`.	`video.g`, `video.crf`, `video.preset`, `video.fast_decode`, `video.video_backend`, `video.extra_options`

[!IMPORTANT] This block is populated once, from the first episode. It assumes every episode in the dataset was encoded with the same rgb_encoder. Changing encoder settings partway through a recording is not supported — the info.json will only reflect the parameters used for the first episode.

Merging datasets#

When aggregating datasets with merge_datasets, video files are concatenated as-is (no re-encoding), and encoder fields in info.json are merged per-key:

Merge rule	Fields	Behaviour
Must match	`video.codec`, `video.pix_fmt`, `video.height`, `video.width`, `video.fps`	Stream-derived fields must match across sources, otherwise FFmpeg's concat demuxer fails.
Merged loosely	`video.g`, `video.crf`, `video.preset`, `video.fast_decode`, `video.extra_options`	Encoder-tuning fields. If every source agrees, the value is kept; if not, it's set to `null` (or `{}` for `video.extra_options`) and a warning is logged.