lerobot/examples/10_use_so100.md

This tutorial explains how to use SO-100 with LeRobot.

Source the parts

Follow this README. It contains the bill of materials, with link to source the parts, as well as the instructions to 3D print the parts, and advices if it's your first time printing or if you don't own a 3D printer already.

Important: Before assembling, you will first need to configure your motors. To this end, we provide a nice script, so let's install LeRobot. We will next provide a tutorial for assembly.

Install LeRobot

On your computer:

1. Install Miniconda:

mkdir -p ~/miniconda3
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda3/miniconda.sh
bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3
rm ~/miniconda3/miniconda.sh
~/miniconda3/bin/conda init bash

2. Restart shell or source ~/.bashrc

3. Create and activate a fresh conda environment for lerobot

conda create -y -n lerobot python=3.10 && conda activate lerobot

4. Clone LeRobot:

git clone https://github.com/huggingface/lerobot.git ~/lerobot

5. Install LeRobot with dependencies for the feetech motors:

cd ~/lerobot && pip install -e ".[feetech]"

For Linux only (not Mac), install extra dependencies for recording datasets:

conda install -y -c conda-forge ffmpeg
pip uninstall -y opencv-python
conda install -y -c conda-forge "opencv>=4.10.0"

Configure the motors

Run this script two times to find the ports (e.g. "/dev/tty.usbmodem58760432961") of your motor buses:

python lerobot/scripts/find_motors_bus_port.py

Then plug your first motor, corresponding to "shoulder_pan" and run this script to set its ID to 1 and set its present position and offset to ~2048 (useful for calibration).

python lerobot/scripts/configure_motor.py \
  --port /dev/tty.usbmodem58760432961 \
  --brand feetech \
  --model sts3215 \
  --baudrate 1000000 \
  --ID 1

Then unplug your motor and plug the second motor, corresponding to "shoulder lift", and set its ID to 2.

python lerobot/scripts/configure_motor.py \
  --port /dev/tty.usbmodem58760432961 \
  --brand feetech \
  --model sts3215 \
  --baudrate 1000000 \
  --ID 2

Redo the process for all your motors until the gripper with ID 6. Do the same for the motors of the leader arm, starting for ID 1 up to 6.

Assemble the arms

TODO

Calibrate

python lerobot/scripts/control_robot.py calibrate \
    --robot-path lerobot/configs/robot/so100.yaml \
    --robot-overrides '~cameras'

Teleoperate

Without displaying the cameras:

python lerobot/scripts/control_robot.py teleoperate \
    --robot-path lerobot/configs/robot/so100.yaml \
    --robot-overrides '~cameras' \
    --display-cameras 0

With displaying the cameras:

python lerobot/scripts/control_robot.py teleoperate \
    --robot-path lerobot/configs/robot/so100.yaml

Record a dataset

Once you're familiar with teleoperation, you can record your first dataset with so100.

If you want to use the Hugging Face hub features for uploading your dataset and you haven't previously done it, make sure you've logged in using a write-access token, which can be generated from the Hugging Face settings:

huggingface-cli login --token ${HUGGINGFACE_TOKEN} --add-to-git-credential

Store your Hugging Face repository name in a variable to run these commands:

HF_USER=$(huggingface-cli whoami | head -n 1)
echo $HF_USER

Record 2 episodes and upload your dataset to the hub:

python lerobot/scripts/control_robot.py record \
    --robot-path lerobot/configs/robot/so100.yaml \
    --fps 30 \
    --root data \
    --repo-id ${HF_USER}/so100_test \
    --tags so100 tutorial \
    --warmup-time-s 5 \
    --episode-time-s 40 \
    --reset-time-s 10 \
    --num-episodes 2 \
    --push-to-hub 1

Visualize a dataset

If you uploaded your dataset to the hub with --push-to-hub 1, you can visualize your dataset online by copy pasting your repo id given by:

echo ${HF_USER}/so100_test

If you didn't upload with --push-to-hub 0, you can also visualize it locally with:

python lerobot/scripts/visualize_dataset_html.py \
  --root data \
  --repo-id ${HF_USER}/so100_test

Replay an episode

Now try to replay the first episode on your robot:

DATA_DIR=data python lerobot/scripts/control_robot.py replay \
    --robot-path lerobot/configs/robot/so100.yaml \
    --fps 30 \
    --root data \
    --repo-id ${HF_USER}/so100_test \
    --episode 0

Train a policy

To train a policy to control your robot, use the python lerobot/scripts/train.py script. A few arguments are required. Here is an example command:

DATA_DIR=data python lerobot/scripts/train.py \
  dataset_repo_id=${HF_USER}/so100_test \
  policy=act_so100_real \
  env=so100_real \
  hydra.run.dir=outputs/train/act_so100_test \
  hydra.job.name=act_so100_test \
  device=cuda \
  wandb.enable=true

Let's explain it:

1. We provided the dataset as argument with dataset_repo_id=${HF_USER}/so100_test.
2. We provided the policy with policy=act_so100_real. This loads configurations from lerobot/configs/policy/act_so100_real.yaml. Importantly, this policy uses 2 cameras as input laptop, phone.
3. We provided an environment as argument with env=so100_real. This loads configurations from lerobot/configs/env/so100_real.yaml.
4. We provided device=cuda since we are training on a Nvidia GPU, but you can also use device=mps if you are using a Mac with Apple silicon, or device=cpu otherwise.
5. We provided wandb.enable=true to use Weights and Biases for visualizing training plots. This is optional but if you use it, make sure you are logged in by running wandb login.
6. We added DATA_DIR=data to access your dataset stored in your local data directory. If you dont provide DATA_DIR, your dataset will be downloaded from Hugging Face hub to your cache folder $HOME/.cache/hugginface. In future versions of lerobot, both directories will be in sync.

Training should take several hours. You will find checkpoints in outputs/train/act_so100_test/checkpoints.

Evaluate your policy

You can use the record function from lerobot/scripts/control_robot.py but with a policy checkpoint as input. For instance, run this command to record 10 evaluation episodes:

python lerobot/scripts/control_robot.py record \
  --robot-path lerobot/configs/robot/so100.yaml \
  --fps 30 \
  --root data \
  --repo-id ${HF_USER}/eval_act_so100_test \
  --tags so100 tutorial eval \
  --warmup-time-s 5 \
  --episode-time-s 40 \
  --reset-time-s 10 \
  --num-episodes 10 \
  -p outputs/train/act_so100_test/checkpoints/last/pretrained_model

As you can see, it's almost the same command as previously used to record your training dataset. Two things changed:

1. There is an additional -p argument which indicates the path to your policy checkpoint with (e.g. -p outputs/train/eval_so100_test/checkpoints/last/pretrained_model). You can also use the model repository if you uploaded a model checkpoint to the hub (e.g. -p ${HF_USER}/act_so100_test).
2. The name of dataset begins by eval to reflect that you are running inference (e.g. --repo-id ${HF_USER}/eval_act_so100_test).

More

Follow this previous tutorial for a more in-depth explaination.

If you have any question or need help, please reach out on Discord in the channel #so100-arm.