feat: add test tube pick task with custom assets and grasp annotations
- Add pick_test_tube task: USDC asset repackaging, grasp generation, task config - Add tools: usdc_to_obj.py, repackage_test_tube.py, fix_test_tube_materials.py - Add custom_task_guide.md: full Chinese documentation for creating custom tasks - Add crawled InternDataEngine online docs (23 pages) - Add grasp generation script (gen_tube_grasp.py) and pipeline config
This commit is contained in:
Tangger
253
docs_crawled/concepts_workflows.md
Normal file
253
docs_crawled/concepts_workflows.md
Normal file
@@ -0,0 +1,253 @@
|
||||
# Source: https://internrobotics.github.io/InternDataEngine-Docs/concepts/workflows.html
|
||||
|
||||
# Workflows [](#workflows)
|
||||
|
||||
Workflows are the interfaces designed by the engine to manipulate the synthetic data pipeline. To understand workflows, you first need a basic understanding of the engine's overall architecture.
|
||||
|
||||
## Overall Architecture [](#overall-architecture)
|
||||
|
||||
The framework of the engine is shown in the figure:
|
||||
|
||||
|
||||
|
||||
Above the optimization layer, the engine provides designs related to the Stage Runner Layer and the Components Layer.
|
||||
|
||||
In the Stage Runner Layer, the standardized lifecycle of data production is defined, namely Load → Plan → Render → Store.
|
||||
|
||||
Based on iterator execution flow, a strict set of abstract base class interfaces is designed, unifying the workflow control of all tasks to ensure that different components extended based on the same interface can collaborate seamlessly.
|
||||
|
||||
For manipulating the data generation pipeline, they are integrated through Env series components combined with workflows. Its core extension principle is based on standardized component interfaces and lifecycle design:
|
||||
|
||||
| Abstract Class | Derived Class | Core Interface | Functional Duty |
|
||||
| `BaseLoader ` | `EnvLoader ` | `load_asset() ` | Resource loading and validation |
|
||||
| `BaseRandomizer ` | `EnvRandomizer ` | `randomize_scene() ` | Domain randomization of the scene |
|
||||
| `BasePlanner ` | `EnvPlanner ` | `generate_sequence() ` | Generate trajectory or action sequence |
|
||||
| `BaseRenderer ` | `EnvRenderer ` | `generate_obs() ` | Synthesize multimodal visual data |
|
||||
| `BaseWriter ` | `EnvWriter ` | `flush_to_disk() ` | Serialized storage |
|
||||
|
||||
## Workflow Base Class and Interfaces [](#workflow-base-class-and-interfaces)
|
||||
|
||||
Based on the Env component architecture, a workflow base class `NimbusWorkFlow `is defined in [workflows/base.py](https://github.com/InternRobotics/InternDataEngine/blob/master/workflows/base.py), with its core API as follows:
|
||||
|
||||
### Workflow Interface List [](#workflow-interface-list)
|
||||
|
||||
The workflow base class `NimbusWorkFlow `provides a complete set of interfaces. The table below lists all interfaces and their types:
|
||||
|
||||
| Method | Return | Calling Component | Type | Description |
|
||||
| `parse_task_cfgs(task_cfg_path) ` | `list ` | EnvLoader | ✓ Core | Parse task configuration files, return task list |
|
||||
| `get_task_name() ` | `str ` | EnvLoader | ✓ Core | Get the name of the currently executing task |
|
||||
| `reset(need_preload) ` | - | EnvLoader | ✓ Core | Reset environment to initial task state |
|
||||
| `randomization(layout_path=None) ` | `bool ` | EnvRandomizer | ✓ Core | Execute domain randomization |
|
||||
| `generate_seq() ` | `list ` | EnvPlanner | ✓ Core | Generate action sequence/trajectory |
|
||||
| `seq_replay(sequence) ` | `int ` | EnvRenderer | ✓ Core | Replay sequence and generate visual data |
|
||||
| `save(save_path) ` | `int ` | EnvWriter | ✓ Core | Save all data (trajectory + visual data) |
|
||||
| `save_seq(save_path) ` | `int ` | EnvWriter | Optional | Only save trajectory, do not save visual data |
|
||||
| `recover_seq(seq_path) ` | `list ` | EnvReader | Optional | Recover trajectory from disk |
|
||||
| `generate_seq_with_obs() ` | `int ` | EnvPlanWithRender | Optional | Generate sequence with visual data |
|
||||
| `dump_plan_info() ` | `bytes ` | EnvDumper | Optional | Serialize planning information |
|
||||
| `dedump_plan_info(ser_obj) ` | `object ` | Dedumper | Optional | Deserialize planning information |
|
||||
| `randomization_from_mem(data) ` | `bool ` | EnvRandomizer | Optional | Randomize from in-memory data |
|
||||
| `recover_seq_from_mem(data) ` | `list ` | EnvReader | Optional | Recover sequence from in-memory data |
|
||||
|
||||
**Interface Type Description: **
|
||||
|
||||
| Type | Description |
|
||||
| ✓ Core Interface | Abstract methods that must be overridden in all workflow implementations |
|
||||
| Optional | Implemented on-demand based on execution mode |
|
||||
|
||||
### Execution Modes [](#execution-modes)
|
||||
|
||||
#### Plan with Render Mode [](#plan-with-render-mode)
|
||||
|
||||
Execute planning and rendering at the same time within a single stage to generate complete trajectories and visual data. Corresponding configuration template: [de_plan_and_render_template.yaml](https://github.com/InternRobotics/InternDataEngine/tree/master/configs/simbox/de_plan_and_render_template.yaml)
|
||||
|
||||
**Lifecycle Flow: **Involved components and corresponding workflow interfaces:
|
||||
|
||||
```
|
||||
EnvLoader <-> get_task_name() → init_task() → reset()
|
||||
EnvRandomizer <-> randomization()
|
||||
EnvPlanWithRender <-> generate_seq_with_obs()
|
||||
EnvWriter <-> save()
|
||||
```
|
||||
1
|
||||
2
|
||||
3
|
||||
4
|
||||
|
||||
**Application Scenario: **Fluid simulation tasks
|
||||
|
||||
#### Plan and Render Mode [](#plan-and-render-mode)
|
||||
|
||||
Execute the planning and rendering stages sequentially. Corresponding configuration template: [de_plan_and_render_template.yaml](https://github.com/InternRobotics/InternDataEngine/tree/master/configs/simbox/de_plan_and_render_template.yaml)
|
||||
|
||||
**Lifecycle Flow: **Involved components and corresponding workflow interfaces:
|
||||
|
||||
```
|
||||
EnvLoader <-> get_task_name() → init_task() → reset()
|
||||
EnvRandomizer <-> randomization()
|
||||
EnvPlanner <-> generate_seq()
|
||||
EnvRenderer <-> seq_replay(sequence)
|
||||
EnvWriter <-> save()
|
||||
```
|
||||
1
|
||||
2
|
||||
3
|
||||
4
|
||||
5
|
||||
|
||||
**Application Scenario: **Algorithm debugging, prototype validation
|
||||
|
||||
#### Pipeline Mode (Distributed Pipeline) [](#pipeline-mode-distributed-pipeline)
|
||||
|
||||
Decouple planning and rendering stages, supporting custom dynamic pipeline scheduling. Corresponding configuration template: [de_pipe_template.yaml](https://github.com/InternRobotics/InternDataEngine/tree/master/configs/simbox/de_pipe_template.yaml)
|
||||
|
||||
**Plan Process Lifecycle: **Involved components and corresponding workflow interfaces:
|
||||
|
||||
```
|
||||
EnvLoader <-> get_task_name() → init_task() → reset()
|
||||
EnvRandomizer <-> randomization()
|
||||
EnvPlanner <-> generate_seq()
|
||||
EnvDumper <-> dump_plan_info()
|
||||
```
|
||||
1
|
||||
2
|
||||
3
|
||||
4
|
||||
|
||||
**Render Process Lifecycle: **Involved components and corresponding workflow interfaces:
|
||||
|
||||
```
|
||||
Dedumper
|
||||
EnvLoader <-> get_task_name() → init_task() → reset() → dedump_plan_info(ser_obj)
|
||||
EnvRandomizer <-> randomization_from_mem(data)
|
||||
EnvReader <-> recover_seq_from_mem(data)
|
||||
EnvRenderer <-> seq_replay(sequence)
|
||||
EnvWriter <-> save()
|
||||
```
|
||||
1
|
||||
2
|
||||
3
|
||||
4
|
||||
5
|
||||
6
|
||||
|
||||
**Application Scenario: **Large-scale distributed data generation
|
||||
|
||||
#### Plan Only Mode [](#plan-only-mode)
|
||||
|
||||
Generate only trajectory data, do not perform rendering operations; can reduce computational resource occupation.
|
||||
|
||||
**Lifecycle Flow: **Involved components and corresponding workflow interfaces:
|
||||
|
||||
```
|
||||
EnvLoader <-> get_task_name() → init_task() → reset()
|
||||
EnvRandomizer <-> randomization()
|
||||
EnvPlanner <-> generate_seq()
|
||||
EnvWriter <-> save()
|
||||
```
|
||||
1
|
||||
2
|
||||
3
|
||||
4
|
||||
|
||||
**Application Scenario: **Trajectory pre-generation, cooperate with Render Only mode to generate diversified background and material videos
|
||||
|
||||
#### Render Only Mode [](#render-only-mode)
|
||||
|
||||
Perform visual rendering on previously generated trajectory data.
|
||||
|
||||
**Lifecycle Flow: **Involved components and corresponding workflow interfaces:
|
||||
|
||||
```
|
||||
EnvLoader <-> get_task_name() → init_task() → reset()
|
||||
EnvRandomizer <-> randomization(layout_path)
|
||||
EnvReader <-> recover_seq()
|
||||
EnvRenderer <-> seq_replay(sequence)
|
||||
EnvWriter <-> save()
|
||||
```
|
||||
1
|
||||
2
|
||||
3
|
||||
4
|
||||
5
|
||||
|
||||
**Application Scenario: **Cooperate with Plan Only mode to generate diversified background and material videos, CI correctness validation
|
||||
|
||||
## SimBox Workflow Implementation [](#simbox-workflow-implementation)
|
||||
|
||||
### Workflow Registration [](#workflow-registration)
|
||||
|
||||
In ****[workflows/init.py](https://github.com/InternRobotics/InternDataEngine/blob/master/workflows/__init__.py), register workflow extensions:
|
||||
python
|
||||
```
|
||||
def import_extensions(workflow_type):
|
||||
if workflow_type == "SimBoxDualWorkFlow":
|
||||
import workflows.simbox_dual_workflow
|
||||
else:
|
||||
raise ValueError(f"Unsupported workflow type: {workflow_type}")
|
||||
```
|
||||
1
|
||||
2
|
||||
3
|
||||
4
|
||||
5
|
||||
|
||||
### Workflow Definition [](#workflow-definition)
|
||||
|
||||
In [workflows/simbox_dual_workflow.py](https://github.com/InternRobotics/InternDataEngine/blob/master/workflows/simbox_dual_workflow.py), define the specific implementation:
|
||||
python
|
||||
```
|
||||
@NimbusWorkFlow.register("SimBoxDualWorkFlow")
|
||||
class SimBoxDualWorkFlow(NimbusWorkFlow):
|
||||
def __init__(
|
||||
self,
|
||||
world,
|
||||
task_cfg_path: str,
|
||||
scene_info: str = "dining_room_scene_info",
|
||||
random_seed: int = None,
|
||||
):
|
||||
...
|
||||
```
|
||||
1
|
||||
2
|
||||
3
|
||||
4
|
||||
5
|
||||
6
|
||||
7
|
||||
8
|
||||
9
|
||||
10
|
||||
|
||||
### Workflow Architecture Design [](#workflow-architecture-design)
|
||||
|
||||
The core component structure of the SimBox workflow is as follows:
|
||||
|
||||
```
|
||||
SimBoxDualWorkFlow
|
||||
Task Module (task/banana.py)
|
||||
├── SceneObject Object Management
|
||||
├── Camera Configuration
|
||||
└── Region Definition
|
||||
Controller Module (controller/template_controller.py)
|
||||
├── CuRobo Motion Planner
|
||||
└── Gripper Controller
|
||||
Skill Module (skill/pick.py, place.py, ...)
|
||||
└── Atomic Operation Units
|
||||
```
|
||||
1
|
||||
2
|
||||
3
|
||||
4
|
||||
5
|
||||
6
|
||||
7
|
||||
8
|
||||
9
|
||||
10
|
||||
|
||||
## Related Documentation [](#related-documentation)
|
||||
|
||||
- [Controller Design](/InternDataEngine-Docs/concepts/controllers.html)- Robot motion planning and control
|
||||
- [Skill Development](/InternDataEngine-Docs/concepts/skills.html)- Implementation and extension of manipulation skills
|
||||
- [Task Definition](/InternDataEngine-Docs/concepts/tasks.html)- Construction and configuration of simulation environments
|
||||
Reference in New Issue
Block a user