feat: add test tube pick task with custom assets and grasp annotations
- Add pick_test_tube task: USDC asset repackaging, grasp generation, task config - Add tools: usdc_to_obj.py, repackage_test_tube.py, fix_test_tube_materials.py - Add custom_task_guide.md: full Chinese documentation for creating custom tasks - Add crawled InternDataEngine online docs (23 pages) - Add grasp generation script (gen_tube_grasp.py) and pipeline config
This commit is contained in:
Tangger
561
docs_crawled/custom_robot.md
Normal file
561
docs_crawled/custom_robot.md
Normal file
@@ -0,0 +1,561 @@
|
||||
# Source: https://internrobotics.github.io/InternDataEngine-Docs/custom/robot.html
|
||||
|
||||
# New Robot [](#new-robot)
|
||||
|
||||
This guide explains how to add a new robot platform to InternDataEngine.
|
||||
|
||||
## Part 1: Obtain Robot USD [](#part-1-obtain-robot-usd)
|
||||
|
||||
We recommend converting **URDF to USD **for robot assets. Isaac Sim provides a URDF Importer that converts URDF files to USD format.
|
||||
|
||||
### Requirements for the USD File [](#requirements-for-the-usd-file)
|
||||
|
||||
-
|
||||
|
||||
**Physics Properties **: Ensure the USD has proper physics attributes - joint angles should move correctly when set, and dynamics parameters (damping, stiffness) should be configured appropriately.
|
||||
|
||||
-
|
||||
|
||||
**Collision Mesh **: Use **convex hull **collision meshes rather than convex decomposition. Convex decomposition creates many collision shapes, which significantly slows down motion planning and interaction physics.
|
||||
|
||||
-
|
||||
|
||||
**Joint Locking (Dual-arm Robots) **: For dual-arm robots, we recommend locking all joints except the two arm joints and gripper joints to simplify experiments and planning.
|
||||
|
||||
Hint
|
||||
|
||||
Isaac Sim's default URDF Importer typically produces stable USD files. However, URDF files themselves may have issues, and some damping/stiffness parameters might need adjustment after conversion.
|
||||
|
||||
Warning
|
||||
|
||||
Some grippers have underactuated joints (passive/follower joints) with a forward extension offset. These may exhibit undesirable physical interactions with objects. Test thoroughly before proceeding.
|
||||
|
||||
Good luck obtaining a physically stable robot USD!
|
||||
|
||||
## Part 2: Create Robot YAML Configuration [](#part-2-create-robot-yaml-configuration)
|
||||
|
||||
Create a robot configuration file in `workflows/simbox/core/configs/robots/ `. Here's a template `new_robot.yaml `:
|
||||
yaml
|
||||
```
|
||||
# =============================================================================
|
||||
# New Robot Configuration Template
|
||||
# =============================================================================
|
||||
# Description: [e.g., Dual-arm 6-DOF robot with lift joint]
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Robot Info
|
||||
# -----------------------------------------------------------------------------
|
||||
target_class: NewRobot # Python class name for robot wrapper
|
||||
path: "YOUR_PATH_TO_ROBOT_USD_PATH (relative to asset root)" # USD file path relative to asset root
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# CuRobo Kinematics Config (one per arm for dual-arm robots)
|
||||
# -----------------------------------------------------------------------------
|
||||
robot_file:
|
||||
- "" # Left arm CuRobo config path
|
||||
- "" # Right arm CuRobo config path (dual-arm only)
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Gripper Parameters
|
||||
# -----------------------------------------------------------------------------
|
||||
gripper_max_width: 0.0 # Maximum gripper opening (meters)
|
||||
gripper_min_width: 0.0 # Minimum gripper closing (meters)
|
||||
tcp_offset: 0.0 # Distance from EE origin to TCP (meters, MUST be positive)
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Physics Solver Parameters
|
||||
# -----------------------------------------------------------------------------
|
||||
solver_position_iteration_count: 128 # Position solver iterations
|
||||
solver_velocity_iteration_count: 4 # Velocity solver iterations
|
||||
stabilization_threshold: 0.005 # Stabilization threshold
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Joint Indices (find in Isaac Sim's articulation inspector)
|
||||
# -----------------------------------------------------------------------------
|
||||
left_joint_indices: [] # Left arm joint indices, e.g., [0, 1, 2, 3, 4, 5]
|
||||
right_joint_indices: [] # Right arm joint indices (dual-arm only)
|
||||
left_gripper_indices: [] # Left gripper joint index, e.g., [6]
|
||||
right_gripper_indices: [] # Right gripper joint index (dual-arm only)
|
||||
lift_indices: [] # Lift joint indices (if applicable)
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# End-Effector Paths (relative to robot prim path)
|
||||
# IMPORTANT: Transformation from EE to TCP must be FIXED!
|
||||
# -----------------------------------------------------------------------------
|
||||
fl_ee_path: "" # Front-left end-effector prim path
|
||||
fr_ee_path: "" # Front-right end-effector prim path (dual-arm only)
|
||||
fl_base_path: "" # Front-left base prim path
|
||||
fr_base_path: "" # Front-right base prim path (dual-arm only)
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Gripper Keypoints (in EE frame, for visualization)
|
||||
# -----------------------------------------------------------------------------
|
||||
fl_gripper_keypoints:
|
||||
tool_head: [0.0, 0.0, 0.0, 1] # Gripper fingertip (approach direction)
|
||||
tool_tail: [0.0, 0.0, 0.0, 1] # Gripper base
|
||||
tool_side: [0.0, 0.0, 0.0, 1] # Side point for width visualization
|
||||
fr_gripper_keypoints:
|
||||
tool_head: [0.0, 0.0, 0.0, 1]
|
||||
tool_tail: [0.0, 0.0, 0.0, 1]
|
||||
tool_side: [0.0, 0.0, 0.0, 1]
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Collision Filter Paths (relative to robot prim path)
|
||||
# -----------------------------------------------------------------------------
|
||||
fl_filter_paths: # Gripper fingers to filter from collision
|
||||
- ""
|
||||
fr_filter_paths:
|
||||
- ""
|
||||
fl_forbid_collision_paths: # Arm links forbidden for self-collision
|
||||
- ""
|
||||
fr_forbid_collision_paths:
|
||||
- ""
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Pose Processing Parameters
|
||||
# -----------------------------------------------------------------------------
|
||||
# Rotation matrix from GraspNet gripper frame to robot EE frame
|
||||
# See: doc/InterDataEngine-docs/concepts/robots.md#r-ee-graspnet
|
||||
R_ee_graspnet: [[0.0, 0.0, 0.0], [0.0, 0.0, 0.0], [0.0, 0.0, 0.0]]
|
||||
ee_axis: "" # Approach axis from EE to TCP: "x", "y", or "z"
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Default Joint Home Positions (radians)
|
||||
# -----------------------------------------------------------------------------
|
||||
left_joint_home: [] # Default left arm joint positions
|
||||
right_joint_home: [] # Default right arm joint positions (dual-arm only)
|
||||
left_joint_home_std: [] # Randomization std for left arm
|
||||
right_joint_home_std: [] # Randomization std for right arm (dual-arm only)
|
||||
left_gripper_home: [] # Default left gripper width (meters)
|
||||
right_gripper_home: [] # Default right gripper width (meters, dual-arm only)
|
||||
lift_home: [] # Default lift position (if applicable)
|
||||
```
|
||||
1
|
||||
2
|
||||
3
|
||||
4
|
||||
5
|
||||
6
|
||||
7
|
||||
8
|
||||
9
|
||||
10
|
||||
11
|
||||
12
|
||||
13
|
||||
14
|
||||
15
|
||||
16
|
||||
17
|
||||
18
|
||||
19
|
||||
20
|
||||
21
|
||||
22
|
||||
23
|
||||
24
|
||||
25
|
||||
26
|
||||
27
|
||||
28
|
||||
29
|
||||
30
|
||||
31
|
||||
32
|
||||
33
|
||||
34
|
||||
35
|
||||
36
|
||||
37
|
||||
38
|
||||
39
|
||||
40
|
||||
41
|
||||
42
|
||||
43
|
||||
44
|
||||
45
|
||||
46
|
||||
47
|
||||
48
|
||||
49
|
||||
50
|
||||
51
|
||||
52
|
||||
53
|
||||
54
|
||||
55
|
||||
56
|
||||
57
|
||||
58
|
||||
59
|
||||
60
|
||||
61
|
||||
62
|
||||
63
|
||||
64
|
||||
65
|
||||
66
|
||||
67
|
||||
68
|
||||
69
|
||||
70
|
||||
71
|
||||
72
|
||||
73
|
||||
74
|
||||
75
|
||||
76
|
||||
77
|
||||
78
|
||||
79
|
||||
80
|
||||
81
|
||||
82
|
||||
83
|
||||
84
|
||||
85
|
||||
86
|
||||
87
|
||||
88
|
||||
89
|
||||
90
|
||||
91
|
||||
92
|
||||
|
||||
Refer to [Robots](/InternDataEngine-Docs/concepts/robots.html)for detailed explanations of each parameter. Here we highlight several critical parameters that require special attention:
|
||||
|
||||
### Critical Parameters [](#critical-parameters)
|
||||
|
||||
fl_ee_path / fr_ee_path
|
||||
|
||||
Specify the relative path (under the robot's local prim path) to the end-effector link.
|
||||
|
||||
Why It Matters:
|
||||
|
||||
- CuRobo plans target poses for this link
|
||||
- Grasp poses are transformed based on this link's EE pose
|
||||
- The `ee_axis `(approach direction from EE origin to TCP) depends on this link
|
||||
|
||||
Selection Rule:
|
||||
|
||||
Common choices:
|
||||
|
||||
- The last link of the gripper (e.g., Robotiq, Genie-1)
|
||||
- A link on the gripper (e.g., Lift-2, Panda)
|
||||
|
||||
Warning
|
||||
|
||||
The transformation from EE to TCP must be **FIXED **(fixed rotation + fixed translation = tcp_offset).
|
||||
|
||||
Figure Examples:
|
||||
|
||||
|
||||
|
||||
*Genie-1: EE link is the last gripper link (fixed EE-to-TCP transform) *
|
||||
|
||||
|
||||
|
||||
*Lift-2: EE link is a gripper link (fixed EE-to-TCP transform) *
|
||||
|
||||
|
||||
|
||||
*Robotiq: EE link is the last gripper link (fixed EE-to-TCP transform) *
|
||||
|
||||
|
||||
|
||||
*Panda: EE link is a gripper link (fixed EE-to-TCP transform) *
|
||||
|
||||
tcp_offset
|
||||
|
||||
Once `fl_ee_path `is determined, calculate the distance from EE origin to TCP. This should be a **positive value **.
|
||||
|
||||
Config Example:
|
||||
yaml
|
||||
```
|
||||
# Example: ARX Lift-2
|
||||
tcp_offset: 0.125 # Distance in meters from EE frame origin to TCP
|
||||
```
|
||||
1
|
||||
2
|
||||
|
||||
ee_axis
|
||||
|
||||
The axis direction from EE origin pointing toward TCP, expressed in the EE frame. Valid values are `"x" `, `"y" `, or `"z" `.
|
||||
|
||||
Config Example:
|
||||
yaml
|
||||
```
|
||||
# Example: ARX Lift-2
|
||||
ee_axis: "x" # Approach direction in EE frame
|
||||
```
|
||||
1
|
||||
2
|
||||
|
||||
R_ee_graspnet
|
||||
|
||||
The rotation matrix that transforms from the GraspNet API's canonical gripper frame to the robot's end-effector frame.
|
||||
|
||||
Warning
|
||||
|
||||
This parameter is important but can be tricky! For calculation details, refer to [Robots - R_ee_graspnet](/concepts/robots.html#r-ee-graspnet).
|
||||
|
||||
Config Example:
|
||||
yaml
|
||||
```
|
||||
# Example: ARX Lift-2
|
||||
R_ee_graspnet: [[1.0, 0.0, 0.0], [0.0, -1.0, 0.0], [0.0, 0.0, -1.0]]
|
||||
```
|
||||
1
|
||||
2
|
||||
|
||||
## Part 3: Create Robot Python Class [](#part-3-create-robot-python-class)
|
||||
|
||||
Create a Python file in `workflows/simbox/core/robots/ `. Here's a template based on `lift2.py `:
|
||||
python
|
||||
```
|
||||
"""New robot implementation."""
|
||||
import numpy as np
|
||||
from core.robots.base_robot import register_robot
|
||||
from core.robots.template_robot import TemplateRobot
|
||||
|
||||
@register_robot
|
||||
class NewRobot(TemplateRobot):
|
||||
"""New robot implementation."""
|
||||
|
||||
def _setup_joint_indices(self):
|
||||
"""Configure joint indices from config."""
|
||||
self.left_joint_indices = self.cfg["left_joint_indices"]
|
||||
self.right_joint_indices = self.cfg["right_joint_indices"]
|
||||
self.left_gripper_indices = self.cfg["left_gripper_indices"]
|
||||
self.right_gripper_indices = self.cfg["right_gripper_indices"]
|
||||
# Optional: lift, body, head indices for special robots
|
||||
self.lift_indices = self.cfg.get("lift_indices", [])
|
||||
|
||||
def _setup_paths(self):
|
||||
"""Configure USD prim paths from config."""
|
||||
fl_ee_path = self.cfg["fl_ee_path"]
|
||||
fr_ee_path = self.cfg["fr_ee_path"]
|
||||
self.fl_ee_path = f"{self.robot_prim_path}/{fl_ee_path}"
|
||||
self.fr_ee_path = f"{self.robot_prim_path}/{fr_ee_path}"
|
||||
self.fl_base_path = f"{self.robot_prim_path}/{self.cfg['fl_base_path']}"
|
||||
self.fr_base_path = f"{self.robot_prim_path}/{self.cfg['fr_base_path']}"
|
||||
self.fl_hand_path = self.fl_ee_path
|
||||
self.fr_hand_path = self.fr_ee_path
|
||||
|
||||
def _setup_gripper_keypoints(self):
|
||||
"""Configure gripper keypoints from config."""
|
||||
self.fl_gripper_keypoints = self.cfg["fl_gripper_keypoints"]
|
||||
self.fr_gripper_keypoints = self.cfg["fr_gripper_keypoints"]
|
||||
|
||||
def _setup_collision_paths(self):
|
||||
"""Configure collision filter paths from config."""
|
||||
self.fl_filter_paths_expr = [
|
||||
f"{self.robot_prim_path}/{p}" for p in self.cfg["fl_filter_paths"]
|
||||
]
|
||||
self.fr_filter_paths_expr = [
|
||||
f"{self.robot_prim_path}/{p}" for p in self.cfg["fr_filter_paths"]
|
||||
]
|
||||
self.fl_forbid_collision_paths = [
|
||||
f"{self.robot_prim_path}/{p}" for p in self.cfg["fl_forbid_collision_paths"]
|
||||
]
|
||||
self.fr_forbid_collision_paths = [
|
||||
f"{self.robot_prim_path}/{p}" for p in self.cfg["fr_forbid_collision_paths"]
|
||||
]
|
||||
|
||||
def _get_gripper_state(self, gripper_home):
|
||||
"""
|
||||
Determine gripper open/close state.
|
||||
|
||||
Returns:
|
||||
1.0 if gripper is open, -1.0 if closed.
|
||||
"""
|
||||
# Adjust threshold based on your gripper's joint values
|
||||
# return 1.0 if gripper_home and gripper_home[0] >= value else -1.0
|
||||
pass
|
||||
|
||||
def _setup_joint_velocities(self):
|
||||
"""Configure joint velocity limits (optional)."""
|
||||
# Override if needed for custom velocity limits
|
||||
pass
|
||||
|
||||
def apply_action(self, joint_positions, joint_indices, *args, **kwargs):
|
||||
"""Override for custom action application (optional)."""
|
||||
super().apply_action(joint_positions, joint_indices, *args, **kwargs)
|
||||
```
|
||||
1
|
||||
2
|
||||
3
|
||||
4
|
||||
5
|
||||
6
|
||||
7
|
||||
8
|
||||
9
|
||||
10
|
||||
11
|
||||
12
|
||||
13
|
||||
14
|
||||
15
|
||||
16
|
||||
17
|
||||
18
|
||||
19
|
||||
20
|
||||
21
|
||||
22
|
||||
23
|
||||
24
|
||||
25
|
||||
26
|
||||
27
|
||||
28
|
||||
29
|
||||
30
|
||||
31
|
||||
32
|
||||
33
|
||||
34
|
||||
35
|
||||
36
|
||||
37
|
||||
38
|
||||
39
|
||||
40
|
||||
41
|
||||
42
|
||||
43
|
||||
44
|
||||
45
|
||||
46
|
||||
47
|
||||
48
|
||||
49
|
||||
50
|
||||
51
|
||||
52
|
||||
53
|
||||
54
|
||||
55
|
||||
56
|
||||
57
|
||||
58
|
||||
59
|
||||
60
|
||||
61
|
||||
62
|
||||
63
|
||||
64
|
||||
65
|
||||
66
|
||||
67
|
||||
68
|
||||
69
|
||||
|
||||
### Key Methods [](#key-methods)
|
||||
|
||||
_get_gripper_state(self, gripper_home)
|
||||
|
||||
Determine whether the gripper is open or closed based on the current joint positions.
|
||||
|
||||
Parameters:
|
||||
|
||||
- **gripper_home **( list ): Current gripper joint positions.
|
||||
|
||||
Returns:
|
||||
|
||||
- **float **: `1.0 `if gripper is open, `-1.0 `if closed.
|
||||
|
||||
Hint
|
||||
|
||||
Adjust the threshold value based on your gripper's specific joint values. Different grippers have different open/close positions.
|
||||
|
||||
Code Example:
|
||||
python
|
||||
```
|
||||
# Example 1: ARX Lift-2 (single-joint parallel gripper)
|
||||
# Gripper is open when joint value >= 0.044 (44mm)
|
||||
def _get_gripper_state(self, gripper_home):
|
||||
return 1.0 if gripper_home and gripper_home[0] >= 0.044 else -1.0
|
||||
|
||||
# Example 2: Franka with Robotiq 2F-85 (two-joint gripper)
|
||||
# Gripper is open when both joints are 0.0
|
||||
def _get_gripper_state(self, gripper_home):
|
||||
if not gripper_home or len(gripper_home) < 2:
|
||||
return 1.0
|
||||
return 1.0 if (gripper_home[0] == 0.0 and gripper_home[1] == 0.0) else -1.0
|
||||
```
|
||||
1
|
||||
2
|
||||
3
|
||||
4
|
||||
5
|
||||
6
|
||||
7
|
||||
8
|
||||
9
|
||||
10
|
||||
11
|
||||
|
||||
### Register the Robot [](#register-the-robot)
|
||||
|
||||
Add your robot to `workflows/simbox/core/robots/__init__.py `:
|
||||
python
|
||||
```
|
||||
from core.robots.new_robot import NewRobot
|
||||
|
||||
__all__ = [
|
||||
# ... existing robots
|
||||
"NewRobot",
|
||||
]
|
||||
```
|
||||
1
|
||||
2
|
||||
3
|
||||
4
|
||||
5
|
||||
6
|
||||
|
||||
## Part 4: Create Task Robot Configuration [](#part-4-create-task-robot-configuration)
|
||||
|
||||
Add the robot configuration to your task YAML file:
|
||||
yaml
|
||||
```
|
||||
robots:
|
||||
- name: "new_robot"
|
||||
robot_config_file: workflows/simbox/core/configs/robots/new_robot.yaml
|
||||
euler: [0.0, 0.0, 0.0] # Initial orientation [roll, pitch, yaw] in degrees
|
||||
ignore_substring: ["material", "table"] # Collision filter substrings
|
||||
```
|
||||
1
|
||||
2
|
||||
3
|
||||
4
|
||||
5
|
||||
|
||||
See [YAML Configuration](/InternDataEngine-Docs/config/yaml.html)for more details on task configuration.
|
||||
|
||||
## Checklist [](#checklist)
|
||||
|
||||
- [ ] Obtain or create robot USD file (URDF → USD via Isaac Sim)
|
||||
- [ ] Verify physics properties (joint movement, damping, stiffness)
|
||||
- [ ] Set up collision meshes (use convex hull, not convex decomposition)
|
||||
- [ ] Lock unnecessary joints for dual-arm robots
|
||||
- [ ] Create robot YAML config file in `workflows/simbox/core/configs/robots/ `
|
||||
- [ ] Configure `fl_ee_path `/ `fr_ee_path `(EE link with fixed EE-to-TCP transform)
|
||||
- [ ] Calculate and set `tcp_offset `(distance from EE origin to TCP)
|
||||
- [ ] Determine `ee_axis `(approach direction: "x", "y", or "z")
|
||||
- [ ] Calculate `R_ee_graspnet `(rotation from GraspNet frame to EE frame)
|
||||
- [ ] Configure joint indices ( `left_joint_indices `, `right_joint_indices `, etc.)
|
||||
- [ ] Set up gripper keypoints for visualization
|
||||
- [ ] Configure collision filter paths
|
||||
- [ ] Create robot Python class in `workflows/simbox/core/robots/ `
|
||||
- [ ] Implement `_setup_joint_indices() `
|
||||
- [ ] Implement `_setup_paths() `
|
||||
- [ ] Implement `_setup_gripper_keypoints() `
|
||||
- [ ] Implement `_setup_collision_paths() `
|
||||
- [ ] Implement `_get_gripper_state() `(1.0 = open, -1.0 = closed)
|
||||
- [ ] Register robot in `__init__.py `
|
||||
- [ ] Add robot to task YAML configuration
|
||||
- [ ] Test robot in simulation
|
||||
Reference in New Issue
Block a user