59 lines
2.5 KiB
Markdown
59 lines
2.5 KiB
Markdown
# Run Aloha (Real Robot)
|
|
|
|
This example demonstrates how to run with a real robot using an [ALOHA setup](https://github.com/tonyzhaozh/aloha).
|
|
|
|
## Prerequisites
|
|
|
|
This repo uses a fork of the ALOHA repo, with very minor modifications to use Realsense cameras.
|
|
|
|
1. Follow the [hardware installation instructions](https://github.com/tonyzhaozh/aloha?tab=readme-ov-file#hardware-installation) in the ALOHA repo.
|
|
1. Modify the `third_party/aloha/aloha_scripts/realsense_publisher.py` file to use serial numbers for your cameras.
|
|
|
|
## With Docker
|
|
|
|
```bash
|
|
export SERVER_ARGS="--env ALOHA --default_prompt='toast out of toaster'"
|
|
docker compose -f examples/aloha_real/compose.yml up --build
|
|
```
|
|
|
|
## Without Docker
|
|
|
|
Terminal window 1:
|
|
|
|
```bash
|
|
# Create virtual environment
|
|
uv venv --python 3.10 examples/aloha_real/.venv
|
|
source examples/aloha_real/.venv/bin/activate
|
|
uv pip sync examples/aloha_real/requirements.txt
|
|
uv pip install -e packages/openpi-client
|
|
|
|
# Run the robot
|
|
python examples/aloha_real/main.py
|
|
```
|
|
|
|
Terminal window 2:
|
|
|
|
```bash
|
|
roslaunch --wait aloha ros_nodes.launch
|
|
```
|
|
|
|
Terminal window 3:
|
|
|
|
```bash
|
|
uv run scripts/serve_policy.py --env ALOHA --default_prompt='toast out of toaster'
|
|
```
|
|
|
|
## Model Guide
|
|
The Pi0 Base Model is an out-of-the-box model for general tasks. You can find more details in the [technical report](https://www.physicalintelligence.company/download/pi0.pdf).
|
|
|
|
While we strongly recommend fine-tuning the model to your own data to adapt it to particular tasks, it may be possible to prompt the model to attempt some tasks that were in the pre-training data. For example, below is a video of the model attempting the "toast out of toaster" task.
|
|
|
|
<p align="center">
|
|
<img src="https://github.com/Physical-Intelligence/openpi/blob/main/examples/aloha_real/toast.gif" alt="toast out of toaster"/>
|
|
</p>
|
|
|
|
## Training on your own Aloha dataset
|
|
|
|
OpenPI suppports training on data collected in the default aloha hdf5 format using the `scripts/aloha_hd5.py` conversion script. Once the dataset is converted, add a new `TrainConfig` to `src/openpi/training/configs.py` (see the `aloha_static_cups_open` example config) and replace repo id with the id assigned to your dataset during conversion. Before training on a new dataset, you must first compute the norm stats using `scripts/compute_norm_stats.py`.
|
|
|
|
NOTE: When finetuning the pi0 base model on Aloha data it is recommended that you set `adapt_to_pi=True`. This maps the state and action spaces from the original aloha data to the state and action spaces of the aloha data used to train the base model. |