feat: add test tube pick task with custom assets and grasp annotations
- Add pick_test_tube task: USDC asset repackaging, grasp generation, task config - Add tools: usdc_to_obj.py, repackage_test_tube.py, fix_test_tube_materials.py - Add custom_task_guide.md: full Chinese documentation for creating custom tasks - Add crawled InternDataEngine online docs (23 pages) - Add grasp generation script (gen_tube_grasp.py) and pipeline config
This commit is contained in:
Tangger
373
docs_crawled/custom_assets.md
Normal file
373
docs_crawled/custom_assets.md
Normal file
@@ -0,0 +1,373 @@
|
||||
# Source: https://internrobotics.github.io/InternDataEngine-Docs/custom/assets.html
|
||||
|
||||
# New Assets [](#new-assets)
|
||||
|
||||
This guide explains how to add new assets to InternDataEngine for simulation tasks.
|
||||
|
||||
## Part 1: Rigid Objects [](#part-1-rigid-objects)
|
||||
|
||||
This section describes how to introduce new rigid objects for manipulation tasks.
|
||||
|
||||
### Step 1: Obtain Geometry and Textures [](#step-1-obtain-geometry-and-textures)
|
||||
|
||||
We recommend starting with an OBJ file that includes:
|
||||
|
||||
- An `.mtl `file storing material properties
|
||||
- With texture images
|
||||
|
||||
**Sources for OBJ Files: **
|
||||
|
||||
- **Open-source datasets **( source ): Various public 3D model repositories.
|
||||
- **ARCode **( source ): 3D reconstruction software for high-fidelity surface textures.
|
||||
- **LiDAR-based apps **( source ): Reconstruction apps with depth sensors for mesh generation.
|
||||
- **Tencent Hunyuan3D **( source ): Multi-view reconstruction for fine, non-convex, or transparent/specular objects.
|
||||
|
||||
Hint
|
||||
|
||||
For objects that are difficult to reconstruct (fine details, non-convex shapes, transparent or highly reflective surfaces), we recommend [Tencent Hunyuan3D](https://3d.hunyuan.tencent.com/)for high-quality multi-view reconstruction.
|
||||
|
||||
### Step 2: Preprocess the OBJ File [](#step-2-preprocess-the-obj-file)
|
||||
|
||||
Before conversion, preprocess the OBJ file to ensure it meets the following requirements:
|
||||
|
||||
- **Correct units **: Use real-world scale (meters recommended)
|
||||
- **Centered origin **: Place origin at the object's center
|
||||
- **Canonical pose **: Align with a reasonable axis (e.g., along a symmetry axis)
|
||||
|
||||
You can perform these adjustments in **MeshLab **:
|
||||
|
||||
|
||||
|
||||
*Setting the origin point at the object center in MeshLab *
|
||||
|
||||
|
||||
|
||||
*Setting the correct scale (units) for the object *
|
||||
|
||||
|
||||
|
||||
*Rotating the object to align with canonical pose *
|
||||
|
||||
After adjustment, export and rename to `Aligned_obj.obj `.
|
||||
|
||||
### Step 3: Convert OBJ to USD [](#step-3-convert-obj-to-usd)
|
||||
|
||||
Navigate to the tools directory and run the converter:
|
||||
bash
|
||||
```
|
||||
cd workflows/simbox/tools/rigid_obj
|
||||
python asset_usd_converter.py --folders /path/to/obj/folder
|
||||
```
|
||||
1
|
||||
2
|
||||
|
||||
This converts the OBJ to USD format and saves it as `Aligned_obj.usd `in the same directory.
|
||||
|
||||
|
||||
|
||||
*Converting OBJ to USD format *
|
||||
|
||||
### Step 4: Add Rigid Body Properties [](#step-4-add-rigid-body-properties)
|
||||
|
||||
Add rigid body physics properties (mass, etc.) to the USD file:
|
||||
bash
|
||||
```
|
||||
python make_rigid.py --usd_path /path/to/Aligned_obj.usd
|
||||
```
|
||||
1
|
||||
|
||||
|
||||
|
||||
*Adding rigid body physics properties (mass) to the USD *
|
||||
|
||||
### Step 5: Add Collision and Friction [](#step-5-add-collision-and-friction)
|
||||
|
||||
Add collider properties and physics materials with friction:
|
||||
bash
|
||||
```
|
||||
# NOTE: This requires Isaac Sim Python
|
||||
isaacsim.python.sh make_collider.py --usd_path /path/to/Aligned_obj.usd
|
||||
```
|
||||
1
|
||||
2
|
||||
|
||||
The collider uses convex decomposition to tightly wrap the object for accurate collision detection.
|
||||
|
||||
|
||||
|
||||
*Adding collision mesh and friction properties *
|
||||
|
||||
### Step 6: Verify Properties [](#step-6-verify-properties)
|
||||
|
||||
After processing, confirm the USD file has the following physics properties:
|
||||
|
||||
- Rigid body dynamics
|
||||
- Collision mesh
|
||||
- Friction coefficients
|
||||
|
||||
The USD file structure will be:
|
||||
|
||||
```
|
||||
World (defaultprim)
|
||||
├── Looks
|
||||
│ └── materials
|
||||
└── Aligned
|
||||
└── mesh
|
||||
```
|
||||
1
|
||||
2
|
||||
3
|
||||
4
|
||||
5
|
||||
|
||||
This hierarchical design ensures consistency. The `Aligned `prim (commonly set as `prim_path_child `) contains the mesh, which is used to detect force contact with the gripper.
|
||||
|
||||
### Step 7 (Optional but Common): Generate Grasp Poses [](#step-7-optional-but-common-generate-grasp-poses)
|
||||
|
||||
If the object will be used for grasping, generate grasp pose annotations:
|
||||
bash
|
||||
```
|
||||
cd workflows/simbox/tools/grasp
|
||||
# See README.md for detailed usage
|
||||
python gen_sparse_label.py --obj_path /path/to/Aligned_obj.obj --unit m
|
||||
```
|
||||
1
|
||||
2
|
||||
3
|
||||
|
||||
For detailed instructions, refer to the [README.md](https://github.com/InternRobotics/InternDataEngine/blob/master/workflows/simbox/tools/grasp/README.md)in the grasp tools directory.
|
||||
|
||||
|
||||
|
||||
*Visualization of generated grasp poses on a banana object *
|
||||
|
||||
Warning
|
||||
|
||||
The OBJ file must maintain consistent canonical frame and scale with the USD file. If you modify the OBJ file (scale, orientation) after USD conversion and then regenerate grasp poses, the grasp poses will be incorrect because they are generated from the OBJ mesh.
|
||||
|
||||
### Step 8: Final Directory Structure [](#step-8-final-directory-structure)
|
||||
|
||||
After completing all the steps above, your task's asset directory should have the following structure:
|
||||
|
||||
```
|
||||
new_task/
|
||||
└── new_objs/ # Directory containing all rigid objects
|
||||
├── new_obj0/ # Example object instance
|
||||
│ ├── Aligned_obj.obj # Preprocessed OBJ mesh (source)
|
||||
│ ├── Aligned.mtl # Material file for OBJ
|
||||
│ ├── Aligned_obj.usd # Final USD with physics properties
|
||||
│ ├── Aligned_grasp_sparse.npy # Sparse grasp poses (N × 17)
|
||||
│ ├── Aligned_grasp_dense.npz # Dense grasp poses (optional)
|
||||
│ └── textures/ # Texture files
|
||||
│ └── Scan.jpg # Object texture image
|
||||
├── new_obj1/ # Another object instance
|
||||
├── new_obj2/
|
||||
├── new_obj3/
|
||||
└── new_obj4/
|
||||
```
|
||||
1
|
||||
2
|
||||
3
|
||||
4
|
||||
5
|
||||
6
|
||||
7
|
||||
8
|
||||
9
|
||||
10
|
||||
11
|
||||
12
|
||||
13
|
||||
14
|
||||
|
||||
## Part 2: Articulated Objects [](#part-2-articulated-objects)
|
||||
|
||||
This section describes how to add new articulated objects (e.g., microwave, drawer, cabinet) for manipulation tasks.
|
||||
|
||||
### Step 1: Prepare a Stable USD Asset [](#step-1-prepare-a-stable-usd-asset)
|
||||
|
||||
First, you need to prepare a stable USD asset for the articulated object. This USD can be:
|
||||
|
||||
- Built from scratch
|
||||
- Converted from URDF format
|
||||
|
||||
The asset should have stable physical properties:
|
||||
|
||||
- **Mass **: Properly defined for each link
|
||||
- **Joint properties **: Appropriate stiffness and damping values
|
||||
- **Collision **: Properly configured colliders for realistic interaction
|
||||
|
||||
The initial hierarchy structure should look like this:
|
||||
|
||||
|
||||
|
||||
*Initial USD hierarchy structure - root positioned at articulation root *
|
||||
|
||||
### Step 2: Understand the Skills Directory Structure [](#step-2-understand-the-skills-directory-structure)
|
||||
|
||||
We provide tools for different articulated object manipulation skills in `workflows/simbox/tools/art/ `. Currently available:
|
||||
|
||||
- `close_v `: Close vertical articulated objects (e.g., microwave, oven)
|
||||
- `open_v `: Open vertical articulated objects
|
||||
|
||||
More skills will be added gradually.
|
||||
|
||||
Take `close_v `as an example - this skill handles horizontal closing motions like closing a microwave from the side. A sample asset is provided at `workflows/simbox/tools/art/close_v/7265/usd/ `.
|
||||
|
||||
### Step 3: Create keypoints_config.json (close_v example) [](#step-3-create-keypoints-config-json-close-v-example)
|
||||
|
||||
After obtaining the original asset (e.g., `microwave_0.usd `), create a `keypoints_config.json `file with the following structure:
|
||||
json
|
||||
```
|
||||
{
|
||||
"DIR": "/path/to/your/usd/directory",
|
||||
"USD_NAME": "microwave_0.usd",
|
||||
"INSTANCE_NAME": "microwave7265",
|
||||
"link0_initial_prim_path": "/root/group_18",
|
||||
"base_initial_prim_path": "/root/group_0",
|
||||
"revolute_joint_initial_prim_path": "/root/group_18/RevoluteJoint",
|
||||
"joint_index": 0,
|
||||
"LINK0_ROT_AXIS": "y",
|
||||
"BASE_FRONT_AXIS": "z",
|
||||
"LINK0_CONTACT_AXIS": "-y",
|
||||
"SCALED_VOLUME": 0.02
|
||||
}
|
||||
```
|
||||
1
|
||||
2
|
||||
3
|
||||
4
|
||||
5
|
||||
6
|
||||
7
|
||||
8
|
||||
9
|
||||
10
|
||||
11
|
||||
12
|
||||
13
|
||||
|
||||
- **DIR **( str ): Directory where USD files are stored.
|
||||
- **USD_NAME **( str ): Original USD file name.
|
||||
- **INSTANCE_NAME **( str ): Model identifier (name it yourself, preferably matching the asset).
|
||||
- **link0_initial_prim_path **( str ): Absolute path in Isaac Sim for the "door" that interacts with the gripper.
|
||||
- **base_initial_prim_path **( str ): Absolute path in Isaac Sim for the object base.
|
||||
- **revolute_joint_initial_prim_path **( str ): Absolute path for the revolute joint.
|
||||
- **joint_index **( int ): Joint number (default: 0).
|
||||
- **LINK0_ROT_AXIS **( str ): Axis pointing vertically upward in the rotating joint's local coordinate system.
|
||||
- **BASE_FRONT_AXIS **( str ): Axis facing the door in the base link's local coordinate system.
|
||||
- **LINK0_CONTACT_AXIS **( str ): Axis pointing vertically downward in the contact link's local coordinate system.
|
||||
- **SCALED_VOLUME **( float ): Default value 0.02 for microwave-like objects.
|
||||
|
||||
For detailed axis configuration with visual examples, refer to the [readme.md](https://github.com/InternRobotics/InternDataEngine/blob/master/workflows/simbox/tools/art/close_v/readme.md).
|
||||
|
||||
### Step 4: Run the Keypoint Annotation Pipeline (close_v example) [](#step-4-run-the-keypoint-annotation-pipeline-close-v-example)
|
||||
|
||||
Navigate to the tools directory and follow the pipeline in `keypoints_pipeline.sh `:
|
||||
bash
|
||||
```
|
||||
cd workflows/simbox/tools/art/close_v/tool
|
||||
```
|
||||
1
|
||||
|
||||
**Step 4.1: Rehier - Restructure Asset Hierarchy **
|
||||
bash
|
||||
```
|
||||
python rehier.py --config $CONFIG_PATH
|
||||
```
|
||||
1
|
||||
|
||||
This step reorganizes the asset hierarchy to a unified standard. After rehier, the structure should look like:
|
||||
|
||||
|
||||
|
||||
*Restructured USD hierarchy - instance prim inserted between original asset and root as articulation root *
|
||||
|
||||
An `instance.usd `file will be generated, indicating success. All joints except the specified one will be locked.
|
||||
|
||||
**Step 4.2: Select Keypoints **
|
||||
bash
|
||||
```
|
||||
python select_keypoint.py --config $CONFIG_PATH
|
||||
```
|
||||
1
|
||||
|
||||
This opens an Open3D visualization window for interactive point selection:
|
||||
|
||||
- **Ctrl + Left Click **: Add a point
|
||||
- **Ctrl + Right Click **: Remove a point
|
||||
|
||||
You need to annotate two contact points:
|
||||
|
||||
| Point | Description |
|
||||
| **First point (articulated_object_head) ** | Desired base position where the gripper contacts the door |
|
||||
| **Second point (articulated_object_tail) ** | The line from the first point should be perpendicular to the rotation axis |
|
||||
|
||||
For visual guidance, refer to the [readme.md](https://github.com/InternRobotics/InternDataEngine/blob/master/workflows/simbox/tools/art/close_v/readme.md).
|
||||
|
||||
**Step 4.3: Transfer Keypoints **
|
||||
bash
|
||||
```
|
||||
python transfer_keypoints.py --config $CONFIG_PATH
|
||||
```
|
||||
1
|
||||
|
||||
**Step 4.4: Overwrite Keypoints **
|
||||
bash
|
||||
```
|
||||
python overwrite_keypoints.py --config $CONFIG_PATH
|
||||
```
|
||||
1
|
||||
|
||||
### Step 5: Final Directory Structure (close_v example) [](#step-5-final-directory-structure-close-v-example)
|
||||
|
||||
After completing all steps, your asset directory should have the following structure:
|
||||
|
||||
```
|
||||
7265/ # Asset ID
|
||||
└── usd/
|
||||
├── microwave_0.usd # Original USD asset
|
||||
├── instance.usd # Processed USD with rehiered structure
|
||||
├── keypoints_config.json # Configuration file
|
||||
├── textures/ # Texture files
|
||||
│ ├── door_mesh_0_texture_0.jpg
|
||||
│ └── door_mesh_1_texture_0.jpg
|
||||
└── Kps/ # Keypoint annotations
|
||||
└── close_v/ # Skill directory
|
||||
├── info.json # Complete metadata
|
||||
├── keypoints.json # Raw keypoints
|
||||
└── keypoints_final.json # Final processed keypoints
|
||||
```
|
||||
1
|
||||
2
|
||||
3
|
||||
4
|
||||
5
|
||||
6
|
||||
7
|
||||
8
|
||||
9
|
||||
10
|
||||
11
|
||||
12
|
||||
13
|
||||
|
||||
You can have multiple USD files for similar objects. Each USD directory must contain:
|
||||
|
||||
- `instance.usd `: The processed USD asset
|
||||
- `Kps/ `: Keypoint annotations organized by skill
|
||||
|
||||
Each skill directory under `Kps/ `must contain:
|
||||
|
||||
- `info.json `: Complete object metadata including paths, axes, and keypoint positions
|
||||
- `keypoints.json `: Raw keypoint coordinates
|
||||
- `keypoints_final.json `: Processed keypoint data
|
||||
|
||||
### Step 6: Test with Task Configuration [](#step-6-test-with-task-configuration)
|
||||
|
||||
With the annotated asset, you can now write a task configuration file to test the manipulation task. Refer to existing task configs in `workflows/simbox/core/configs/tasks/art/ `for examples.
|
||||
|
||||
## References [](#references)
|
||||
|
||||
- [GraspNet API](https://github.com/graspnet/graspnetAPI)
|
||||
- [Tencent Hunyuan3D](https://3d.hunyuan.tencent.com/)
|
||||
Reference in New Issue
Block a user