Isaac Teleop → SO-101#

← 教程资料库 · 其他硬件与遥操作

本节介绍 LeRobot 的通用功能。请按目标机器人、数据集和运行环境配置示例；用于 AlohaMini 2 / 2 Pro 前，需要确认硬件与接口兼容性。

Teleoperate an SO-101/SO-100 follower arm — and record LeRobot datasets — with NVIDIA Isaac Teleop. Two input devices ship today:

  • XR (VR) controller (--teleop.type=xr_controller) — the controller's grip pose drives the end-effector through a squeeze-to-engage clutch and LeRobot's Cartesian IK pipeline; the analog trigger drives the gripper.

  • SO-101 leader arm (--teleop.type=so101_leader) — a back-drivable leader arm mirrored 1:1 onto the follower via Isaac Teleop's native so101_leader plugin (no clutch, no IK).

The full narrative guide (how the clutch works, CloudXR setup, headset pairing, tuning, and troubleshooting) is in the LeRobot docs (source: docs/source/isaac_teleop.mdx). This README is the canonical install and usage reference.

Requirements#

  • Linux workstation (see NVIDIA's system requirements for supported OS/GPU/headset combinations; isaacteleop publishes Linux wheels only).

  • An SO-101 (or SO-100) follower arm, calibrated with lerobot-calibrate.

  • For the XR device: a CloudXR-capable headset (e.g. Quest 3, Pico 4, Apple Vision Pro) on the same network.

  • For the leader device: a second, back-drivable SO-101 leader arm and the so101_leader plugin binary built from the Isaac Teleop source tree (see Build from source).

Installation#

This example lives in the LeRobot repository and is not part of the lerobot pip package, so work from a source checkout. From the repo root:

# LeRobot with the extras this example uses:
#   feetech    - SO-101 serial motor bus
#   kinematics - Placo IK solver (XR controller path)
#   dataset    - dataset recording (record.py)
# huggingface_hub >= 1.5 is needed by the automatic URDF fetch (Buckets API).
uv pip install -e ".[feetech,kinematics,dataset]" "huggingface_hub>=1.5"

# Isaac Teleop from public PyPI. `cloudxr` brings the CloudXR runtime bindings;
# `retargeters-lite` is the scipy-based retargeter path that resolves on both
# x86_64 and ARM (the full `retargeters` extra does not resolve on aarch64).
uv pip install "isaacteleop[cloudxr,retargeters-lite]~=1.3.131" "scipy>=1.14"

# Optional, x86_64 only: the full retargeter stack.
uv pip install "isaacteleop[retargeters]~=1.3.131"

One-time CloudXR EULA (the auto-launch prompts on stdin and would hang on a headless machine):

python -m isaacteleop.cloudxr --accept-eula

Usage#

Run everything from the repo root with python -m so the examples package resolves.

Teleoperate — XR controller#

python -m examples.isaac_teleop_to_so101.teleoperate \
    --robot.type=so101_follower \
    --robot.port=/dev/ttyACM0 \
    --robot.id=so101_follower_arm \
    --teleop.type=xr_controller

On startup the script launches the CloudXR runtime (~30 s), prints the workstation IP to enter in the headset's CloudXR web client, waits for the controllers to stream, slews the arm to a reset pose (--reset_to_origin=false to skip), and then: hold the squeeze/grip to engage, move the controller to drive the arm, pull the trigger to close the gripper. Releasing the squeeze freezes the arm. The SO-101 URDF is fetched automatically from the lerobot/robot-urdfs Hugging Face bucket into the LeRobot cache on first run.

To customize the reset pose: back-drive the arm to the pose you want, then

python -m examples.isaac_teleop_to_so101.override_reset_pose --port /dev/ttyACM0 --id so101_follower_arm

which writes it to HF_LEROBOT_HOME/reset_poses/<robot.name>/<robot.id>.json; runs with the same --robot.id use it automatically.

Teleoperate — SO-101 leader arm#

python -m examples.isaac_teleop_to_so101.teleoperate \
    --robot.type=so101_follower --robot.port=/dev/ttyACM0 --robot.id=so101_follower_arm \
    --teleop.type=so101_leader --teleop.port=/dev/ttyACM1 --teleop.id=so101_leader_arm \
    --launch_plugin=/path/to/IsaacTeleop/install/plugins/so101_leader/so101_leader_plugin

The follower is first slewed to the leader's pose over --align_duration seconds (--align=false to skip), then mirrors it 1:1. The plugin reuses the serial leader's calibration (HF_LEROBOT_CALIBRATION/teleoperators/so_leader/<teleop.id>.json).

Record a dataset#

record.py takes the same --robot.*/--teleop.*/loop flags plus lerobot-record-style --dataset.* flags:

python -m examples.isaac_teleop_to_so101.record \
    --robot.type=so101_follower --robot.port=/dev/ttyACM0 --robot.id=so101_follower_arm \
    --teleop.type=xr_controller \
    --robot.cameras="{ front: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}}" \
    --dataset.repo_id=<hf_user>/<dataset_name> \
    --dataset.single_task="Pick up the cube" \
    --dataset.num_episodes=3 --dataset.episode_time_s=20 --dataset.reset_time_s=5

Keyboard shortcuts (terminal-first, so they work over SSH): Right/n end episode early, Left/r re-record, Esc/q stop after the current episode.

Run either script with --help for all flags.

Layout#

isaac_teleop/            device library: session lifecycle (base.py), XRController,
                         SO101LeaderArm, Clutch, configs, and the XR→IK processor step
common.py                shared loop infra: device bundles, clutch/IK pipeline wiring,
                         reset/align slews, URDF fetch, keyboard listener
teleoperate.py           teleoperation CLI (device selected via --teleop.type)
record.py                dataset-recording CLI (same device selection + --dataset.*)
override_reset_pose.py   save the current joints as the per-arm reset pose
default.env              CloudXR device-profile overrides passed to the launcher