dual arm XR teleoperation: env cfg, shared client, joint locking
- Add MindRobotDualArmIKAbsEnvCfg: standalone dual-arm env inheriting ManagerBasedRLEnvCfg directly (no single-arm dependency), 20D action space (left_arm7 | wheel4 | left_gripper1 | right_arm7 | right_gripper1) - Add local mdp/ with parameterized gripper_pos(joint_names) to support independent left/right gripper observations without modifying IsaacLab - Update teleop_xr_agent.py for dual-arm mode: shared XrClient to avoid SDK double-init crash, root-frame IK command caching so arm joints are locked when grip not pressed (EEF world pos still tracks chassis) - Tune mindrobot_cfg.py initial poses with singularity-avoiding offsets - Add CLAUDE.md project instructions and debug_action_assembly.py Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
130
CLAUDE.md
Normal file
130
CLAUDE.md
Normal file
@@ -0,0 +1,130 @@
|
||||
# MindBot Project
|
||||
|
||||
基于 Isaac Lab 的移动操作机器人（MindRobot）仿真学习框架，支持强化学习（RL）和模仿学习（IL）。
|
||||
|
||||
## 环境要求
|
||||
|
||||
- **Isaac Sim**: 5.1.0
|
||||
- **Python**: 3.11
|
||||
- **依赖**: isaaclab, isaaclab_assets, isaaclab_mimic, isaaclab_rl, isaaclab_tasks
|
||||
- **Conda 环境**：
|
||||
- `env_isaaclab` — 主开发环境
|
||||
- `xr` — 另一个同样包含 isaaclab/isaacsim 的环境，两者均可运行 Isaac Lab 脚本
|
||||
|
||||
**IMPORTANT**: 所有 Isaac Lab 脚本必须通过 `~/IsaacLab/isaaclab.sh -p` 启动，直接用 `python` 会报 `KeyError: 'EXP_PATH'`（该变量只有 `isaaclab.sh` 会在启动时设置）。
|
||||
|
||||
```bash
|
||||
# 安装包
|
||||
~/IsaacLab/isaaclab.sh -p -m pip install -e source/mindbot
|
||||
```
|
||||
|
||||
## 关键环境变量
|
||||
|
||||
```bash
|
||||
export MINDBOT_ASSETS_DIR=/home/tangger/LYT/maic_usd_assets_moudle # USD 资源路径（默认值）
|
||||
```
|
||||
|
||||
## 常用命令
|
||||
|
||||
```bash
|
||||
# 列出所有注册环境
|
||||
python scripts/list_envs.py
|
||||
|
||||
# 测试环境（零动作 / 随机动作）
|
||||
python scripts/zero_agent.py --task Template-Mindbot-v0 --num_envs 4
|
||||
python scripts/random_agent.py --task Template-Mindbot-v0 --num_envs 4
|
||||
|
||||
# RL 训练（RSL-RL PPO）
|
||||
python scripts/rsl_rl/train.py --task Template-Mindbot-v0 --num_envs 4096
|
||||
|
||||
# RL 推理
|
||||
python scripts/rsl_rl/play.py --task Template-Mindbot-v0
|
||||
|
||||
# 遥操作（Spacemouse / 键盘）
|
||||
python scripts/environments/teleoperation/teleop_se3_agent.py --task Isaac-MindRobot-LeftArm-IK-Rel-v0
|
||||
|
||||
# 遥操作（XR 设备）
|
||||
python scripts/environments/teleoperation/teleop_xr_agent.py --task Isaac-MindRobot-LeftArm-IK-Rel-v0
|
||||
|
||||
# 代码格式化
|
||||
pre-commit run --all-files
|
||||
```
|
||||
|
||||
## 项目架构
|
||||
|
||||
```
|
||||
source/mindbot/mindbot/
|
||||
├── robot/mindbot.py # MindRobot 关节/执行器配置（MINDBOT_CFG）
|
||||
├── utils/assets.py # USD 资源路径（MINDBOT_ASSETS_DIR）
|
||||
└── tasks/manager_based/
|
||||
├── rl/ # 强化学习任务
|
||||
│ ├── mindbot/ # Template-Mindbot-v0（抓取）
|
||||
│ ├── centrifuge/ # Template-centrifuge-lidup-v0
|
||||
│ ├── pull/ # Template-Pull-v0
|
||||
│ └── pullUltrasoundLidUp/ # Pull-Ultrasound-Lid-Up-v0
|
||||
└── il/ # 模仿学习任务
|
||||
├── demo/ # Demo 任务（H1、Franka）
|
||||
└── open_drybox/ # Isaac-MindRobot-LeftArm-IK-{Rel,Abs}-v0
|
||||
```
|
||||
|
||||
## 注册任务列表
|
||||
|
||||
| 任务 ID | 类型 | 说明 |
|
||||
|---|---|---|
|
||||
| `Template-Mindbot-v0` | RL | MindRobot 抓取 |
|
||||
| `Template-centrifuge-lidup-v0` | RL | 离心机旋转 |
|
||||
| `Template-Pull-v0` | RL | 物体拉取 |
|
||||
| `Pull-Ultrasound-Lid-Up-v0` | RL | 超声机盖拉取 |
|
||||
| `Isaac-MindRobot-LeftArm-IK-Rel-v0` | IL | 左臂 IK 相对控制 |
|
||||
| `Isaac-MindRobot-LeftArm-IK-Abs-v0` | IL | 左臂 IK 绝对控制 |
|
||||
|
||||
## 新增任务的规范
|
||||
|
||||
### RL 任务结构
|
||||
```
|
||||
tasks/manager_based/rl/<task_name>/
|
||||
├── __init__.py # gym.register(id="<TaskId>-v0", ...)
|
||||
├── <task_name>_env_cfg.py # ManagerBasedRLEnvCfg 子类
|
||||
├── agents/
|
||||
│ └── rsl_rl_ppo_cfg.py # RslRlOnPolicyRunnerCfg 子类
|
||||
└── mdp/
|
||||
├── rewards.py
|
||||
├── terminations.py
|
||||
└── __init__.py
|
||||
```
|
||||
|
||||
### IL 任务结构
|
||||
```
|
||||
tasks/manager_based/il/<task_name>/
|
||||
├── __init__.py # gym.register with robomimic_bc_cfg_entry_point
|
||||
├── <task_name>_env_cfg.py # ManagerBasedRLEnvCfg 子类（使用 DifferentialIKController）
|
||||
└── agents/robomimic/
|
||||
└── bc_rnn_low_dim.json # RoboMimic BC 配置
|
||||
```
|
||||
|
||||
新任务的 `__init__.py` 必须在 `mindbot/tasks/manager_based/__init__.py` 中 import 才能注册到 Gym。
|
||||
|
||||
## 代码风格
|
||||
|
||||
- 文件头使用 SPDX 许可证注释：`# SPDX-FileCopyrightText: Copyright (c) 2022-2025, The Isaac Lab Project Developers.`
|
||||
- 关节角度配置用弧度，注释中可标注对应度数
|
||||
- 执行器刚度/阻尼参数调整需在注释中说明物理含义
|
||||
|
||||
## MindRobot 关节组
|
||||
|
||||
| 关节组 | 关节名 |
|
||||
|---|---|
|
||||
| 左臂（6 自由度） | l_joint1 ~ l_joint6 |
|
||||
| 右臂（6 自由度） | r_joint1 ~ r_joint6 |
|
||||
| 左夹爪 | left_hand_joint_left/right |
|
||||
| 右夹爪 | right_hand_joint_left/right |
|
||||
| 躯干升降 | PrismaticJoint |
|
||||
| 头部 | head_revoluteJoint |
|
||||
| 底盘轮子 | front/back revolute joints |
|
||||
|
||||
## 遥操作数据收集
|
||||
|
||||
- Isaac Lab 内遥操作脚本使用 `env_isaaclab` 环境
|
||||
- XR 设备接入依赖 `deps/XRoboToolkit-Teleop-Sample-Python/`，需切换到 `xr` 环境
|
||||
- 示教数据格式兼容 LeRobot / RoboMimic
|
||||
- 多相机配置：cam_head, cam_chest, cam_left_hand, cam_right_hand, cam_top, cam_side
|
||||
Reference in New Issue
Block a user