ACT (Action Chunking with Transformers)#

← Tutorial Library · policies and Models

This section introduces the general functions of LeRobot. Please configure the example according to the target robot, data set and operating environment; before using it with AlohaMini 2 / 2 Pro, you need to confirm the hardware and interface compatibility.

ACT is a lightweight and efficient policy for imitation learning, especially well-suited for fine-grained manipulation tasks. It’s the first model we recommend when you’re starting out with LeRobot due to its fast training time, low computational requirements, and strong performance.

Watch this tutorial from the LeRobot team to learn how ACT works: LeRobot ACT Tutorial

Model Overview#

Action Chunking with Transformers (ACT) was introduced in the paper Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware by Zhao et al. The policy was designed to enable precise, contact-rich manipulation tasks using affordable hardware and minimal demonstration data.

Why ACT is Great for Beginners#

ACT stands out as an excellent starting point for several reasons:

  • Fast Training: Trains in a few hours on a single GPU

  • Lightweight: Only ~80M parameters, making it efficient and easy to work with

  • Data Efficient: Often achieves high success rates with just 50 demonstrations

Architecture#

ACT uses a transformer-based architecture with three main components:

  1. Vision Backbone: ResNet-18 processes images from multiple camera viewpoints

  2. Transformer Encoder: Synthesizes information from camera features, joint positions, and a learned latent variable

  3. Transformer Decoder: Generates coherent action sequences using cross-attention

The policy takes as input:

  • Multiple RGB images (e.g., from wrist cameras, front/top cameras)

  • Current robot joint positions

  • A latent style variable z (learned during training, set to zero during inference)

And outputs a chunk of k future action sequences.

Installation Requirements#

  1. Install LeRobot by following our Installation Guide.

  2. ACT is included in the base LeRobot installation, so no additional dependencies are needed!

Training ACT#

ACT works seamlessly with the standard LeRobot training pipeline. Here’s a complete example for training ACT on your dataset:

lerobot-train \
  --dataset.repo_id=${HF_USER}/your_dataset \
  --policy.type=act \
  --output_dir=outputs/train/act_your_dataset \
  --job_name=act_your_dataset \
  --policy.device=cuda \
  --wandb.enable=true \
  --policy.repo_id=${HF_USER}/act_policy

Training Tips#

  1. Start with defaults: ACT’s default hyperparameters work well for most tasks

  2. Training duration: Expect a few hours for 100k training steps on a single GPU

  3. Batch size: Start with batch size 8 and adjust based on your GPU memory

Train using Google Colab#

If your local computer doesn’t have a powerful GPU, you can utilize Google Colab to train your model by following the ACT training notebook.

Evaluating ACT#

Once training is complete, you can evaluate your ACT policy using the lerobot-record command with your trained policy. This will run inference and record evaluation episodes:

lerobot-rollout \
  --strategy.type=base \
  --policy.path=${HF_USER}/act_policy \
  --robot.type=so101_follower \
  --robot.port=/dev/ttyACM0 \
  --robot.cameras="{ front: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}}" \
  --display_data=true \
  --task="Your task description" \ # can be skipped for ACT
  --duration=60