hangX/lerobot
1
0
Fork 0
forked from tangger/lerobot
Code Pull Requests Activity

Add Aloha env and ACT policy

Browse Source 
WIP Aloha env tests pass

Rendering works (fps look fast tho? TODO action bounding is too wide [-1,1])

Update README

Copy past from act repo

Remove download.py add a WIP for Simxarm

Remove download.py add a WIP for Simxarm

Add act yaml (TODO: try train.py)

Training can runs (TODO: eval)

Add tasks without end_effector that are compatible with dataset, Eval can run (TODO: training and pretrained model)

Add AbstractEnv, Refactor AlohaEnv, Add rendering_hook in env, Minor modifications, (TODO: Refactor Pusht and Simxarm)

poetry lock

fix bug in compute_stats for action normalization

fix more bugs in normalization

fix training

fix import

PushtEnv inheriates AbstractEnv, Improve factory Normalization

Add _make_env to EnvAbstract

Add call_rendering_hooks to pusht env

SimxarmEnv inherites from AbstractEnv (NOT TESTED)

Add aloha tests artifacts + update pusht stats

fix image normalization: before env was in [0,1] but dataset in [0,255], and now both in [0,255]

Small fix on simxarm

Add next to obs

Add top camera to Aloha env (TODO: make it compatible with set of cameras)

Add top camera to Aloha env (TODO: make it compatible with set of cameras)
This commit is contained in:
Remi Cadene
2024-03-08 09:47:39 +00:00
committed by Cadene
parent 060bac7672
commit 9d002032d1
116 changed files with 3658 additions and 301 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
tests/data/aloha_sim_insertion_human/action.memmap Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d789deddb081a9f4b626342391de8f48949d38fb5fdead87b5c0737b46c0877a
size 2800

3
tests/data/aloha_sim_insertion_human/episode.memmap Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:7a12e561363385e9dfeeab326368731c030ed4b374e7f5897ac819159d2884c5
size 400

3
tests/data/aloha_sim_insertion_human/frame_id.memmap Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:c202d9cfc7858fd49d522047e16948359bbbb2eda2d3825d552e45a78d5f8585
size 400

1
tests/data/aloha_sim_insertion_human/meta.json Normal file
View File

@@ -0,0 +1 @@
{"action": {"device": "cpu", "shape": [50, 14], "dtype": "torch.float32"}, "episode": {"device": "cpu", "shape": [50], "dtype": "torch.int64"}, "frame_id": {"device": "cpu", "shape": [50], "dtype": "torch.int64"}, "shape": [50], "device": "cpu", "_type": "<class 'tensordict._td.TensorDict'>"}

3
tests/data/aloha_sim_insertion_human/next/done.memmap Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:cc2786e1f9910a9d811400edcddaf7075195f7a16b216dcbefba3bc7c4f2ae51
size 50

1
tests/data/aloha_sim_insertion_human/next/meta.json Normal file
View File

@@ -0,0 +1 @@
{"done": {"device": "cpu", "shape": [50, 1], "dtype": "torch.bool"}, "shape": [50], "device": "cpu", "_type": "<class 'tensordict._td.TensorDict'>"}

1
tests/data/aloha_sim_insertion_human/next/observation/image/meta.json Normal file
View File

@@ -0,0 +1 @@
{"top": {"device": "cpu", "shape": [50, 3, 480, 640], "dtype": "torch.uint8"}, "shape": [50], "device": "cpu", "_type": "<class 'tensordict._td.TensorDict'>"}

3
tests/data/aloha_sim_insertion_human/next/observation/image/top.memmap Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:5c632e3cb06be729e5d673e3ecca1d6f6527b0f48cfe3dc03d7eea4f9eb3bbd7
size 46080000

1
tests/data/aloha_sim_insertion_human/next/observation/meta.json Normal file
View File

@@ -0,0 +1 @@
{"state": {"device": "cpu", "shape": [50, 14], "dtype": "torch.float32"}, "shape": [50], "device": "cpu", "_type": "<class 'tensordict._td.TensorDict'>"}

3
tests/data/aloha_sim_insertion_human/next/observation/state.memmap Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:e231f2e07e1cd030137ea2e938b570b112db2c694c6d21b37ceb8f8559e19088
size 2800

1
tests/data/aloha_sim_insertion_human/observation/image/meta.json Normal file
View File

@@ -0,0 +1 @@
{"top": {"device": "cpu", "shape": [50, 3, 480, 640], "dtype": "torch.uint8"}, "shape": [50], "device": "cpu", "_type": "<class 'tensordict._td.TensorDict'>"}

3
tests/data/aloha_sim_insertion_human/observation/image/top.memmap Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:a1ba64c89f4fcf9135fe34c26abf582dd5f0d573506db5c96af3ffe40a52c818
size 46080000

1
tests/data/aloha_sim_insertion_human/observation/meta.json Normal file
View File

@@ -0,0 +1 @@
{"state": {"device": "cpu", "shape": [50, 14], "dtype": "torch.float32"}, "shape": [50], "device": "cpu", "_type": "<class 'tensordict._td.TensorDict'>"}

3
tests/data/aloha_sim_insertion_human/observation/state.memmap Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:85405686bc065c6ab6c915907920a0391a57cf097b74de058a8c30be0548ade5
size 2800

BIN
tests/data/aloha_sim_insertion_human/stats.pth Normal file
View File

Binary file not shown.

3
tests/data/aloha_sim_insertion_scripted/action.memmap Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:1f5fe053b760e8471885b82c10f4a6ea40874098036337ae5cc300c4775546be
size 2800

3
tests/data/aloha_sim_insertion_scripted/episode.memmap Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:7a12e561363385e9dfeeab326368731c030ed4b374e7f5897ac819159d2884c5
size 400

3
tests/data/aloha_sim_insertion_scripted/frame_id.memmap Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:c202d9cfc7858fd49d522047e16948359bbbb2eda2d3825d552e45a78d5f8585
size 400

1
tests/data/aloha_sim_insertion_scripted/meta.json Normal file
View File

@@ -0,0 +1 @@
{"action": {"device": "cpu", "shape": [50, 14], "dtype": "torch.float32"}, "episode": {"device": "cpu", "shape": [50], "dtype": "torch.int64"}, "frame_id": {"device": "cpu", "shape": [50], "dtype": "torch.int64"}, "shape": [50], "device": "cpu", "_type": "<class 'tensordict._td.TensorDict'>"}

3
tests/data/aloha_sim_insertion_scripted/next/done.memmap Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:cc2786e1f9910a9d811400edcddaf7075195f7a16b216dcbefba3bc7c4f2ae51
size 50

1
tests/data/aloha_sim_insertion_scripted/next/meta.json Normal file
View File

@@ -0,0 +1 @@
{"done": {"device": "cpu", "shape": [50, 1], "dtype": "torch.bool"}, "shape": [50], "device": "cpu", "_type": "<class 'tensordict._td.TensorDict'>"}

1
tests/data/aloha_sim_insertion_scripted/next/observation/image/meta.json Normal file
View File

@@ -0,0 +1 @@
{"top": {"device": "cpu", "shape": [50, 3, 480, 640], "dtype": "torch.uint8"}, "shape": [50], "device": "cpu", "_type": "<class 'tensordict._td.TensorDict'>"}

3
tests/data/aloha_sim_insertion_scripted/next/observation/image/top.memmap Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:daed2bb10498ba2557983d0d7e89399882fea7585e7ceff910e23c621bfdbf88
size 46080000

1
tests/data/aloha_sim_insertion_scripted/next/observation/meta.json Normal file
View File

@@ -0,0 +1 @@
{"state": {"device": "cpu", "shape": [50, 14], "dtype": "torch.float32"}, "shape": [50], "device": "cpu", "_type": "<class 'tensordict._td.TensorDict'>"}

3
tests/data/aloha_sim_insertion_scripted/next/observation/state.memmap Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:bbad0302af70112ee312efe0eb0f44a2f1c8f6c5ef82ea4fb34625cdafbef057
size 2800

1
tests/data/aloha_sim_insertion_scripted/observation/image/meta.json Normal file
View File

@@ -0,0 +1 @@
{"top": {"device": "cpu", "shape": [50, 3, 480, 640], "dtype": "torch.uint8"}, "shape": [50], "device": "cpu", "_type": "<class 'tensordict._td.TensorDict'>"}

3
tests/data/aloha_sim_insertion_scripted/observation/image/top.memmap Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:aba55ebb9dd004bf68444b9ebf024ed7713436099c06a0b8e541100ecbc69290
size 46080000

1
tests/data/aloha_sim_insertion_scripted/observation/meta.json Normal file
View File

@@ -0,0 +1 @@
{"state": {"device": "cpu", "shape": [50, 14], "dtype": "torch.float32"}, "shape": [50], "device": "cpu", "_type": "<class 'tensordict._td.TensorDict'>"}

3
tests/data/aloha_sim_insertion_scripted/observation/state.memmap Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:dd4e7e14abf57561ca9839c910581266be90956e41bfb3bb21362ea0c321e77d
size 2800

BIN
tests/data/aloha_sim_insertion_scripted/stats.pth Normal file
View File

Binary file not shown.

3
tests/data/aloha_sim_transfer_cube_human/action.memmap Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:14fed0eed3d529a8ac0dd25a6d41585020772d02f9137fc9d604713b2f0f7076
size 2800

3
tests/data/aloha_sim_transfer_cube_human/episode.memmap Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:7a12e561363385e9dfeeab326368731c030ed4b374e7f5897ac819159d2884c5
size 400

3
tests/data/aloha_sim_transfer_cube_human/frame_id.memmap Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:c202d9cfc7858fd49d522047e16948359bbbb2eda2d3825d552e45a78d5f8585
size 400

1
tests/data/aloha_sim_transfer_cube_human/meta.json Normal file
View File

@@ -0,0 +1 @@
{"action": {"device": "cpu", "shape": [50, 14], "dtype": "torch.float32"}, "episode": {"device": "cpu", "shape": [50], "dtype": "torch.int64"}, "frame_id": {"device": "cpu", "shape": [50], "dtype": "torch.int64"}, "shape": [50], "device": "cpu", "_type": "<class 'tensordict._td.TensorDict'>"}

3
tests/data/aloha_sim_transfer_cube_human/next/done.memmap Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:cc2786e1f9910a9d811400edcddaf7075195f7a16b216dcbefba3bc7c4f2ae51
size 50

1
tests/data/aloha_sim_transfer_cube_human/next/meta.json Normal file
View File

@@ -0,0 +1 @@
{"done": {"device": "cpu", "shape": [50, 1], "dtype": "torch.bool"}, "shape": [50], "device": "cpu", "_type": "<class 'tensordict._td.TensorDict'>"}

1
tests/data/aloha_sim_transfer_cube_human/next/observation/image/meta.json Normal file
View File

@@ -0,0 +1 @@
{"top": {"device": "cpu", "shape": [50, 3, 480, 640], "dtype": "torch.uint8"}, "shape": [50], "device": "cpu", "_type": "<class 'tensordict._td.TensorDict'>"}

3
tests/data/aloha_sim_transfer_cube_human/next/observation/image/top.memmap Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:2f713ea7fc19e592ea409a5e0bdfde403e5b86f834cbabe3463b791e8437fafc
size 46080000

1
tests/data/aloha_sim_transfer_cube_human/next/observation/meta.json Normal file
View File

@@ -0,0 +1 @@
{"state": {"device": "cpu", "shape": [50, 14], "dtype": "torch.float32"}, "shape": [50], "device": "cpu", "_type": "<class 'tensordict._td.TensorDict'>"}

3
tests/data/aloha_sim_transfer_cube_human/next/observation/state.memmap Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:8c103e2c9d63c9f7cf9645bd24d9a2c4e8e08825dc75e230ebc793b8f9c213b0
size 2800

1
tests/data/aloha_sim_transfer_cube_human/observation/image/meta.json Normal file
View File

@@ -0,0 +1 @@
{"top": {"device": "cpu", "shape": [50, 3, 480, 640], "dtype": "torch.uint8"}, "shape": [50], "device": "cpu", "_type": "<class 'tensordict._td.TensorDict'>"}

3
tests/data/aloha_sim_transfer_cube_human/observation/image/top.memmap Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:7dbf4aa01b184d0eaa21ea999078d7cff86e1ca484a109614176fdc49f1ee05c
size 46080000

1
tests/data/aloha_sim_transfer_cube_human/observation/meta.json Normal file
View File

@@ -0,0 +1 @@
{"state": {"device": "cpu", "shape": [50, 14], "dtype": "torch.float32"}, "shape": [50], "device": "cpu", "_type": "<class 'tensordict._td.TensorDict'>"}

3
tests/data/aloha_sim_transfer_cube_human/observation/state.memmap Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:4fa0b9c870d4615037b6fee9e9e85e54d84352e173f2c7c1035232272fe2a3dd
size 2800

BIN
tests/data/aloha_sim_transfer_cube_human/stats.pth Normal file
View File

Binary file not shown.

3
tests/data/aloha_sim_transfer_cube_scripted/action.memmap Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:c0e199a82e2b7462e84406dbced5448a99f1dad9ce172771dfc3feb4b8597115
size 2800

3
tests/data/aloha_sim_transfer_cube_scripted/episode.memmap Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:7a12e561363385e9dfeeab326368731c030ed4b374e7f5897ac819159d2884c5
size 400

3
tests/data/aloha_sim_transfer_cube_scripted/frame_id.memmap Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:c202d9cfc7858fd49d522047e16948359bbbb2eda2d3825d552e45a78d5f8585
size 400

1
tests/data/aloha_sim_transfer_cube_scripted/meta.json Normal file
View File

@@ -0,0 +1 @@
{"action": {"device": "cpu", "shape": [50, 14], "dtype": "torch.float32"}, "episode": {"device": "cpu", "shape": [50], "dtype": "torch.int64"}, "frame_id": {"device": "cpu", "shape": [50], "dtype": "torch.int64"}, "shape": [50], "device": "cpu", "_type": "<class 'tensordict._td.TensorDict'>"}

3
tests/data/aloha_sim_transfer_cube_scripted/next/done.memmap Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:cc2786e1f9910a9d811400edcddaf7075195f7a16b216dcbefba3bc7c4f2ae51
size 50

1
tests/data/aloha_sim_transfer_cube_scripted/next/meta.json Normal file
View File

@@ -0,0 +1 @@
{"done": {"device": "cpu", "shape": [50, 1], "dtype": "torch.bool"}, "shape": [50], "device": "cpu", "_type": "<class 'tensordict._td.TensorDict'>"}

1
tests/data/aloha_sim_transfer_cube_scripted/next/observation/image/meta.json Normal file
View File

@@ -0,0 +1 @@
{"top": {"device": "cpu", "shape": [50, 3, 480, 640], "dtype": "torch.uint8"}, "shape": [50], "device": "cpu", "_type": "<class 'tensordict._td.TensorDict'>"}

3
tests/data/aloha_sim_transfer_cube_scripted/next/observation/image/top.memmap Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:30b44d38cc4d68e06c716a875d39cbdbeacbfdc1657d6366f58c279efd27c52b
size 46080000

1
tests/data/aloha_sim_transfer_cube_scripted/next/observation/meta.json Normal file
View File

@@ -0,0 +1 @@
{"state": {"device": "cpu", "shape": [50, 14], "dtype": "torch.float32"}, "shape": [50], "device": "cpu", "_type": "<class 'tensordict._td.TensorDict'>"}

3
tests/data/aloha_sim_transfer_cube_scripted/next/observation/state.memmap Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:9f484e7ea4f5f612dd53ee2c0f7891b8f7b2168a54fc81941ac2f2447260c294
size 2800

1
tests/data/aloha_sim_transfer_cube_scripted/observation/image/meta.json Normal file
View File

@@ -0,0 +1 @@
{"top": {"device": "cpu", "shape": [50, 3, 480, 640], "dtype": "torch.uint8"}, "shape": [50], "device": "cpu", "_type": "<class 'tensordict._td.TensorDict'>"}

3
tests/data/aloha_sim_transfer_cube_scripted/observation/image/top.memmap Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:09600206f56cc5b52dfb896204b0044c4e830da368da141d7bd10e52181f6835
size 46080000

1
tests/data/aloha_sim_transfer_cube_scripted/observation/meta.json Normal file
View File

@@ -0,0 +1 @@
{"state": {"device": "cpu", "shape": [50, 14], "dtype": "torch.float32"}, "shape": [50], "device": "cpu", "_type": "<class 'tensordict._td.TensorDict'>"}

3
tests/data/aloha_sim_transfer_cube_scripted/observation/state.memmap Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:60dcb547cf9a6372b78a455217a2408b6bece4371fba1df2a302b334d45c42a8
size 2800

BIN
tests/data/aloha_sim_transfer_cube_scripted/stats.pth Normal file
View File

Binary file not shown.

BIN
tests/data/pusht/stats.pth
View File

Binary file not shown.

15
tests/test_datasets.py
View File

@@ -1,4 +1,5 @@
import pytest
import torch
from lerobot.common.datasets.factory import make_offline_buffer
@@ -12,12 +13,18 @@ from .utils import init_config
# ("simxarm", "lift"),
("pusht", "pusht"),
# TODO(aliberts): add aloha when dataset is available on hub
# ("aloha", "sim_insertion_human"),
# ("aloha", "sim_insertion_scripted"),
# ("aloha", "sim_transfer_cube_human"),
# ("aloha", "sim_transfer_cube_scripted"),
("aloha", "sim_insertion_human"),
("aloha", "sim_insertion_scripted"),
("aloha", "sim_transfer_cube_human"),
("aloha", "sim_transfer_cube_scripted"),
],
)
def test_factory(env_name, dataset_id):
cfg = init_config(overrides=[f"env={env_name}", f"env.task={dataset_id}"])
offline_buffer = make_offline_buffer(cfg)
for key in offline_buffer.image_keys:
img = offline_buffer[0].get(key)
assert img.dtype == torch.float32
# TODO(rcadene): we assume for now that image normalization takes place in the model
assert img.max() <= 1.0
assert img.min() >= 0.0

17
tests/test_envs.py
View File

@@ -1,6 +1,8 @@
import pytest
from tensordict import TensorDict
import torch
from torchrl.envs.utils import check_env_specs, step_mdp
from lerobot.common.datasets.factory import make_offline_buffer
from lerobot.common.envs.factory import make_env
from lerobot.common.envs.pusht.env import PushtEnv
@@ -83,9 +85,24 @@ def test_pusht(from_pixels, pixels_only):
[
# "simxarm",
"pusht",
"aloha",
],
)
def test_factory(env_name):
cfg = init_config(overrides=[f"env={env_name}"])
offline_buffer = make_offline_buffer(cfg)
env = make_env(cfg)
for key in offline_buffer.image_keys:
assert env.reset().get(key).dtype == torch.uint8
check_env_specs(env)
env = make_env(cfg, transform=offline_buffer.transform)
for key in offline_buffer.image_keys:
img = env.reset().get(key)
assert img.dtype == torch.float32
# TODO(rcadene): we assume for now that image normalization takes place in the model
assert img.max() <= 1.0
assert img.min() >= 0.0
check_env_specs(env)