57 lines
1.9 KiB
Markdown
57 lines
1.9 KiB
Markdown
# LIBERO Benchmark
|
|
|
|
This example runs the LIBERO benchmark: https://github.com/Lifelong-Robot-Learning/LIBERO
|
|
|
|
Note: When updating requirements.txt in this directory, there is an additional flag `--extra-index-url https://download.pytorch.org/whl/cu113` that must be added to the `uv pip compile` command.
|
|
|
|
This example requires git submodules to be initialized. Don't forget to run:
|
|
|
|
```bash
|
|
git submodule update --init --recursive
|
|
```
|
|
|
|
## With Docker
|
|
|
|
```bash
|
|
# Grant access to the X11 server:
|
|
sudo xhost +local:docker
|
|
|
|
export SERVER_ARGS="--env LIBERO"
|
|
docker compose -f examples/libero/compose.yml up --build
|
|
```
|
|
|
|
## Without Docker
|
|
|
|
Terminal window 1:
|
|
|
|
```bash
|
|
# Create virtual environment
|
|
uv venv --python 3.8 examples/libero/.venv
|
|
source examples/libero/.venv/bin/activate
|
|
uv pip sync examples/libero/requirements.txt third_party/libero/requirements.txt --extra-index-url https://download.pytorch.org/whl/cu113 --index-strategy=unsafe-best-match
|
|
uv pip install -e packages/openpi-client
|
|
uv pip install -e third_party/libero
|
|
export PYTHONPATH=$PYTHONPATH:$PWD/third_party/libero
|
|
|
|
# Run the simulation
|
|
python examples/libero/main.py
|
|
```
|
|
|
|
Terminal window 2:
|
|
|
|
```bash
|
|
# Run the server
|
|
uv run scripts/serve_policy.py --env LIBERO
|
|
```
|
|
|
|
## Results
|
|
|
|
If you follow the training instructions and hyperparameters in the `pi0_libero` and `pi0_fast_libero` configs, you should get results similar to the following:
|
|
|
|
| Model | Libero Spatial | Libero Object | Libero Goal | Libero 10 | Average |
|
|
|-------|---------------|---------------|-------------|-----------|---------|
|
|
| π0-FAST @ 30k (finetuned) | 96.4 | 96.8 | 88.6 | 60.2 | 85.5 |
|
|
| π0 @ 30k (finetuned) | 96.8 | 98.8 | 95.8 | 85.2 | 94.15 |
|
|
|
|
Note that the hyperparameters for these runs are not tuned and $\pi_0$-FAST does not use a FAST tokenizer optimized for Libero. Likely, the results could be improved with more tuning, we mainly use these results as an example of how to use openpi to fine-tune $\pi_0$ models on a new dataset.
|