Train RL in Simulation#

本节介绍 LeRobot 的通用功能。请按目标机器人、数据集和运行环境配置示例；用于 AlohaMini 2 / 2 Pro 前，需要确认硬件与接口兼容性。

This guide explains how to use the gym_hil simulation environments as an alternative to real robots when working with the LeRobot framework for Human-In-the-Loop (HIL) reinforcement learning.

gym_hil is a package that provides Gymnasium-compatible simulation environments specifically designed for Human-In-the-Loop reinforcement learning. These environments allow you to:

Train policies in simulation to test the RL stack before training on real robots
Collect demonstrations in sim using external devices like gamepads or keyboards
Perform human interventions during policy learning

Currently, the main environment is a Franka Panda robot simulation based on MuJoCo, with tasks like picking up a cube.

Installation#

First, install the gym_hil package within the LeRobot environment:

pip install -e ".[hilserl]"

What do I need?#

A gamepad or keyboard to control the robot
A Nvidia GPU

Configuration#

To use gym_hil with LeRobot, you need to create a configuration file. An example is provided here. Key configuration sections include:

Environment Type and Task#

{
  "env": {
    "type": "gym_manipulator",
    "name": "gym_hil",
    "task": "PandaPickCubeGamepad-v0",
    "fps": 10
  },
  "device": "cuda"
}

Available tasks:

PandaPickCubeBase-v0: Basic environment
PandaPickCubeGamepad-v0: With gamepad control
PandaPickCubeKeyboard-v0: With keyboard control

Processor Configuration#

{
  "env": {
    "processor": {
      "control_mode": "gamepad",
      "gripper": {
        "use_gripper": true,
        "gripper_penalty": -0.02
      },
      "reset": {
        "control_time_s": 15.0,
        "fixed_reset_joint_positions": [
          0.0, 0.195, 0.0, -2.43, 0.0, 2.62, 0.785
        ]
      },
      "inverse_kinematics": {
        "end_effector_step_sizes": {
          "x": 0.025,
          "y": 0.025,
          "z": 0.025
        }
      }
    }
  }
}

Important parameters:

gripper.gripper_penalty: Penalty for excessive gripper movement
gripper.use_gripper: Whether to enable gripper control
inverse_kinematics.end_effector_step_sizes: Size of the steps in the x,y,z axes of the end-effector
control_mode: Set to "gamepad" to use a gamepad controller

Running with HIL RL of LeRobot#

Basic Usage#

To run the environment, set mode to null:

python -m lerobot.rl.gym_manipulator --config_path path/to/gym_hil_env.json

Recording a Dataset#

To collect a dataset, set the mode to record whilst defining the repo_id and number of episodes to record:

{
  "env": {
    "type": "gym_manipulator",
    "name": "gym_hil",
    "task": "PandaPickCubeGamepad-v0"
  },
  "dataset": {
    "repo_id": "username/sim_dataset",
    "root": null,
    "task": "pick_cube",
    "num_episodes_to_record": 10,
    "replay_episode": null,
    "push_to_hub": true
  },
  "mode": "record"
}

python -m lerobot.rl.gym_manipulator --config_path path/to/gym_hil_env.json

Training a Policy#

To train a policy, checkout the configuration example available here and run the actor and learner servers:

python -m lerobot.rl.actor --config_path path/to/train_gym_hil_env.json

In a different terminal, run the learner server:

python -m lerobot.rl.learner --config_path path/to/train_gym_hil_env.json

The simulation environment provides a safe and repeatable way to develop and test your Human-In-the-Loop reinforcement learning components before deploying to real robots.

Congrats 🎉, you have finished this tutorial!

[!TIP] If you have any questions or need help, please reach out on Discord.

Paper citation:

@article{luo2024precise,
  title={Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement Learning},
  author={Luo, Jianlan and Xu, Charles and Wu, Jeffrey and Levine, Sergey},
  journal={arXiv preprint arXiv:2410.21845},
  year={2024}
}