188 lines
7.0 KiB
Markdown
188 lines
7.0 KiB
Markdown
This tutorial explains how to use [SO-100](https://github.com/TheRobotStudio/SO-ARM100) with LeRobot.
|
|
|
|
## Source the parts
|
|
|
|
Follow this [README](https://github.com/TheRobotStudio/SO-ARM100). It contains the bill of materials, with link to source the parts, as well as the instructions to 3D print the parts, and advices if it's your first time printing or if you don't own a 3D printer already.
|
|
|
|
**Important**: Before assembling, you will first need to configure your motors. To this end, we provide a nice script, so let's install LeRobot. We will next provide a tutorial for assembly.
|
|
|
|
## Install LeRobot
|
|
|
|
On your computer:
|
|
|
|
1. [Install Miniconda](https://docs.anaconda.com/miniconda/#quick-command-line-install):
|
|
```bash
|
|
mkdir -p ~/miniconda3
|
|
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda3/miniconda.sh
|
|
bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3
|
|
rm ~/miniconda3/miniconda.sh
|
|
~/miniconda3/bin/conda init bash
|
|
```
|
|
|
|
2. Restart shell or `source ~/.bashrc`
|
|
|
|
3. Create and activate a fresh conda environment for lerobot
|
|
```bash
|
|
conda create -y -n lerobot python=3.10 && conda activate lerobot
|
|
```
|
|
|
|
4. Clone LeRobot:
|
|
```bash
|
|
git clone https://github.com/huggingface/lerobot.git ~/lerobot
|
|
```
|
|
|
|
5. Install LeRobot with dependencies for the feetech motors:
|
|
```bash
|
|
cd ~/lerobot && pip install -e ".[feetech]"
|
|
```
|
|
|
|
For Linux only (not Mac), install extra dependencies for recording datasets:
|
|
```bash
|
|
conda install -y -c conda-forge ffmpeg
|
|
pip uninstall -y opencv-python
|
|
conda install -y -c conda-forge "opencv>=4.10.0"
|
|
```
|
|
|
|
## Configure the motors
|
|
|
|
```bash
|
|
python lerobot/scripts/find_motors_bus_port.py
|
|
```
|
|
|
|
```bash
|
|
python lerobot/scripts/configure_motor.py \
|
|
--port /dev/tty.usbmodem58760432961 \
|
|
--brand feetech \
|
|
--model sts3215 \
|
|
--baudrate 1000000 \
|
|
--ID 1
|
|
```
|
|
|
|
## Assemble the arms
|
|
|
|
TODO
|
|
|
|
## Calibrate
|
|
|
|
|
|
## Teleoperate
|
|
|
|
```bash
|
|
python lerobot/scripts/control_robot.py teleoperate \
|
|
--robot-path lerobot/configs/robot/so100.yaml \
|
|
--robot-overrides '~cameras' \
|
|
--display-cameras 0
|
|
```
|
|
|
|
```bash
|
|
python lerobot/scripts/control_robot.py teleoperate \
|
|
--robot-path lerobot/configs/robot/so100.yaml
|
|
```
|
|
|
|
## Record a dataset
|
|
|
|
Once you're familiar with teleoperation, you can record your first dataset with so100.
|
|
|
|
If you want to use the Hugging Face hub features for uploading your dataset and you haven't previously done it, make sure you've logged in using a write-access token, which can be generated from the [Hugging Face settings](https://huggingface.co/settings/tokens):
|
|
```bash
|
|
huggingface-cli login --token ${HUGGINGFACE_TOKEN} --add-to-git-credential
|
|
```
|
|
|
|
Store your Hugging Face repository name in a variable to run these commands:
|
|
```bash
|
|
HF_USER=$(huggingface-cli whoami | head -n 1)
|
|
echo $HF_USER
|
|
```
|
|
|
|
Record 2 episodes and upload your dataset to the hub:
|
|
```bash
|
|
python lerobot/scripts/control_robot.py record \
|
|
--robot-path lerobot/configs/robot/so100.yaml \
|
|
--fps 30 \
|
|
--root data \
|
|
--repo-id ${HF_USER}/so100_test \
|
|
--tags so100 tutorial \
|
|
--warmup-time-s 5 \
|
|
--episode-time-s 40 \
|
|
--reset-time-s 10 \
|
|
--num-episodes 2 \
|
|
--push-to-hub 1
|
|
```
|
|
|
|
## Visualize a dataset
|
|
|
|
If you uploaded your dataset to the hub with `--push-to-hub 1`, you can [visualize your dataset online](https://huggingface.co/spaces/lerobot/visualize_dataset) by copy pasting your repo id given by:
|
|
```bash
|
|
echo ${HF_USER}/so100_test
|
|
```
|
|
|
|
If you didn't upload with `--push-to-hub 0`, you can also visualize it locally with:
|
|
```bash
|
|
python lerobot/scripts/visualize_dataset_html.py \
|
|
--root data \
|
|
--repo-id ${HF_USER}/so100_test
|
|
```
|
|
|
|
## Replay an episode
|
|
|
|
Now try to replay the first episode on your robot:
|
|
```bash
|
|
DATA_DIR=data python lerobot/scripts/control_robot.py replay \
|
|
--robot-path lerobot/configs/robot/so100.yaml \
|
|
--fps 30 \
|
|
--root data \
|
|
--repo-id ${HF_USER}/so100_test \
|
|
--episode 0
|
|
```
|
|
|
|
## Train a policy
|
|
|
|
To train a policy to control your robot, use the [`python lerobot/scripts/train.py`](../lerobot/scripts/train.py) script. A few arguments are required. Here is an example command:
|
|
```bash
|
|
DATA_DIR=data python lerobot/scripts/train.py \
|
|
dataset_repo_id=${HF_USER}/so100_test \
|
|
policy=act_so100_real \
|
|
env=so100_real \
|
|
hydra.run.dir=outputs/train/act_so100_test \
|
|
hydra.job.name=act_so100_test \
|
|
device=cuda \
|
|
wandb.enable=true
|
|
```
|
|
|
|
Let's explain it:
|
|
1. We provided the dataset as argument with `dataset_repo_id=${HF_USER}/so100_test`.
|
|
2. We provided the policy with `policy=act_so100_real`. This loads configurations from [`lerobot/configs/policy/act_so100_real.yaml`](../lerobot/configs/policy/act_so100_real.yaml). Importantly, this policy uses 2 cameras as input `laptop`, `phone`.
|
|
3. We provided an environment as argument with `env=so100_real`. This loads configurations from [`lerobot/configs/env/so100_real.yaml`](../lerobot/configs/env/so100_real.yaml).
|
|
4. We provided `device=cuda` since we are training on a Nvidia GPU, but you can also use `device=mps` if you are using a Mac with Apple silicon, or `device=cpu` otherwise.
|
|
5. We provided `wandb.enable=true` to use [Weights and Biases](https://docs.wandb.ai/quickstart) for visualizing training plots. This is optional but if you use it, make sure you are logged in by running `wandb login`.
|
|
6. We added `DATA_DIR=data` to access your dataset stored in your local `data` directory. If you dont provide `DATA_DIR`, your dataset will be downloaded from Hugging Face hub to your cache folder `$HOME/.cache/hugginface`. In future versions of `lerobot`, both directories will be in sync.
|
|
|
|
Training should take several hours. You will find checkpoints in `outputs/train/act_so100_test/checkpoints`.
|
|
|
|
## Evaluate your policy
|
|
|
|
You can use the `record` function from [`lerobot/scripts/control_robot.py`](../lerobot/scripts/control_robot.py) but with a policy checkpoint as input. For instance, run this command to record 10 evaluation episodes:
|
|
```bash
|
|
python lerobot/scripts/control_robot.py record \
|
|
--robot-path lerobot/configs/robot/so100.yaml \
|
|
--fps 30 \
|
|
--root data \
|
|
--repo-id ${HF_USER}/eval_act_so100_test \
|
|
--tags so100 tutorial eval \
|
|
--warmup-time-s 5 \
|
|
--episode-time-s 40 \
|
|
--reset-time-s 10 \
|
|
--num-episodes 10 \
|
|
-p outputs/train/act_so100_test/checkpoints/last/pretrained_model
|
|
```
|
|
|
|
As you can see, it's almost the same command as previously used to record your training dataset. Two things changed:
|
|
1. There is an additional `-p` argument which indicates the path to your policy checkpoint with (e.g. `-p outputs/train/eval_so100_test/checkpoints/last/pretrained_model`). You can also use the model repository if you uploaded a model checkpoint to the hub (e.g. `-p ${HF_USER}/act_so100_test`).
|
|
2. The name of dataset begins by `eval` to reflect that you are running inference (e.g. `--repo-id ${HF_USER}/eval_act_so100_test`).
|
|
|
|
## More
|
|
|
|
Follow this [previous tutorial](https://github.com/huggingface/lerobot/blob/main/examples/7_get_started_with_real_robot.md#4-train-a-policy-on-your-data) for a more in-depth explaination.
|
|
|
|
If you have any question or need help, please reach out on Discord in the channel `#so100-arm`.
|