Install

Create a virtual environment with Python 3.10 and activate it, e.g. with miniconda:

conda create -y -n lerobot python=3.10
conda activate lerobot

Install 🤗 LeRobot:

pip install -e . -i https://pypi.tuna.tsinghua.edu.cn/simple
pip install edge-tts
sudo apt install mpv -y

# pip uninstall numpy
# pip install numpy==1.26.0
# pip install pynput

/!\ For Linux only, ffmpeg and opencv requires conda install for now. Run this exact sequence of commands:

conda install ffmpeg=7.1.1 -c conda-forge
# pip uninstall opencv-python
# conda install "opencv>=4.10.0"

Install Realman SDK:

pip install Robotic_Arm==1.0.4.1
pip install pygame

piper集成lerobot

见lerobot_piper_tutorial/1. 🤗 LeRobot：新增机械臂的一般流程.pdf

Teleoperate

cd piper_scripts/
bash can_activate.sh can0 1000000

cd ..
python lerobot/scripts/control_robot.py \
    --robot.type=piper \
    --robot.inference_time=false \
    --control.type=teleoperate

Record

Set dataset root path

HF_USER=$PWD/data
echo $HF_USER

python lerobot/scripts/control_robot.py \
    --robot.type=realman \
    --robot.inference_time=false \
    --control.type=record \
    --control.fps=15 \
    --control.single_task="move" \
    --control.repo_id=maic/test \
    --control.num_episodes=2 \
    --control.warmup_time_s=2 \
    --control.episode_time_s=10 \
    --control.reset_time_s=10 \
    --control.play_sounds=true \
    --control.push_to_hub=false \
    --control.display_data=true

Press right arrow -> at any time during episode recording to early stop and go to resetting. Same during resetting, to early stop and to go to the next episode recording.
Press left arrow <- at any time during episode recording or resetting to early stop, cancel the current episode, and re-record it.
Press escape ESC at any time during episode recording to end the session early and go straight to video encoding and dataset uploading.

visualize

python lerobot/scripts/visualize_dataset.py \
    --repo-id ${HF_USER}/test \
    --episode-index 0

Replay

python lerobot/scripts/control_robot.py \
    --robot.type=piper \
    --robot.inference_time=false \
    --control.type=replay \
    --control.fps=30 \
    --control.repo_id=${HF_USER}/test \
    --control.episode=0

Caution

In lerobots/common/datasets/video_utils, the vcodec is set to libopenh264, please find your vcodec by ffmpeg -codecs

Train

具体的训练流程见lerobot_piper_tutorial/2. 🤗 AutoDL训练.pdf

python lerobot/scripts/train.py \
  --dataset.repo_id=${HF_USER}/jack \
  --policy.type=act \
  --output_dir=outputs/train/act_jack \
  --job_name=act_jack \
  --device=cuda \
  --wandb.enable=true

FT smolvla

python lerobot/scripts/train.py
--dataset.repo_id=maic/move_the_bottle_into_ultrasonic_device_with_realman_single
--policy.path=lerobot/smolvla_base
--output_dir=outputs/train/smolvla_move_the_bottle_into_ultrasonic_device_with_realman_single
--job_name=smolvla_move_the_bottle_into_ultrasonic_device_with_realman_single
--policy.device=cuda
--wandb.enable=false
--steps=200000
--batch_size=16

Inference

还是使用control_robot.py中的record loop，配置 --robot.inference_time=true 可以将手柄移出。

python lerobot/scripts/control_robot.py \
    --robot.type=realman \
    --robot.inference_time=true \
    --control.type=record \
    --control.fps=30 \
    --control.single_task="move the bottle into ultrasonic device with realman single" \
    --control.repo_id=maic/move_the_bottle_into_ultrasonic_device_with_realman_single \
    --control.num_episodes=1 \
    --control.warmup_time_s=2 \
    --control.episode_time_s=30 \
    --control.reset_time_s=10 \
    --control.push_to_hub=false \
    --control.policy.path=outputs/train/act_move_the_bottle_into_ultrasonic_device_with_realman_single/checkpoints/100000/pretrained_model

python lerobot/scripts/control_robot.py     \
--robot.type=realman     \
--robot.inference_time=true     \
--control.type=record     \
--control.fps=30     \
--control.single_task="move the bottle into ultrasonic device with realman single"     \
--control.repo_id=maic/eval_smolvla_move_the_bottle_into_ultrasonic_device_with_realman_single     \
--control.num_episodes=1     \
--control.warmup_time_s=2     \
--control.episode_time_s=60     \
--control.reset_time_s=10     \
--control.push_to_hub=false     \
--control.policy.path=outputs/train/smolvla_move_the_bottle_into_ultrasonic_device_with_realman_single/checkpoints/160000/pretrained_model \
--control.display_data=true

4.6 KiB Raw Permalink Blame History Unescape Escape