yt lee
623e05f250
Replaces manual H264/TCP stereo streaming with NVIDIA CloudXR for
higher-quality stereoscopic rendering and lower latency.
Changes:
- teleop_xr_agent.py: add --cloudxr flag (enables Isaac Sim XR mode,
disables manual StreamingManager)
- deps/cloudxr/: NVIDIA CloudXR.js SDK (Early Access) with Isaac Lab
teleop React web client
- deps/cloudxr/Dockerfile.wss.proxy: HAProxy WSS proxy for PICO 4 Ultra
HTTPS mode (routes wss://48322 → ws://49100)
- deps/cloudxr/isaac/webpack.dev.js: disable file watching to avoid
EMFILE errors with large node_modules
- deps/cloudxr/INSTALL.md: full setup guide
Usage:
# Start CloudXR Runtime + Isaac Lab
cd ~/IsaacLab && ./docker/container.py start \
--files docker-compose.cloudxr-runtime.patch.yaml \
--env-file .env.cloudxr-runtime
# Run teleop with CloudXR
~/IsaacLab/isaaclab.sh -p teleop_xr_agent.py \
--task Isaac-MindRobot-2i-DualArm-IK-Abs-v0 --cloudxr
# Serve web client
cd deps/cloudxr/isaac && npm run dev-server:https
119 lines
3.6 KiB
Markdown
119 lines
3.6 KiB
Markdown
# CloudXR.js Simple Example
|
|
|
|
A minimal WebGL example demonstrating WebXR streaming from a CloudXR server to a web browser. This example shows how to integrate WebXR with CloudXR to stream immersive VR/AR content.
|
|
|
|
> **Note:** This example is for learning purposes, not production use.
|
|
|
|
## Quick Start
|
|
|
|
### Prerequisites
|
|
- Node.js (v20 or higher)
|
|
|
|
### Installation
|
|
|
|
1. **Navigate to the example folder**
|
|
```bash
|
|
cd simple
|
|
```
|
|
|
|
2. **Install Dependencies**
|
|
```bash
|
|
# For this early access release, please run the following to install SDK from the given tarball. This step will not be needed when SDK is publicly accessible.
|
|
npm install ../nvidia-cloudxr-6.0.0-beta.tgz
|
|
|
|
npm install
|
|
```
|
|
|
|
3. **Build the Application**
|
|
```bash
|
|
npm run build
|
|
```
|
|
|
|
4. **Start Development Server**
|
|
```bash
|
|
npm run dev-server
|
|
```
|
|
|
|
5. **Open in Browser**
|
|
- Navigate to `http://localhost:8080` (or the port shown in terminal)
|
|
- For desktop browsers, IWER (Immersive Web Emulator Runtime) will automatically load to emulate a Meta Quest 3 headset
|
|
|
|
### Basic Usage
|
|
|
|
1. **Configure Connection**
|
|
- Enter CloudXR server IP address (default: localhost)
|
|
- Set port (default: 49100)
|
|
- Select AR or VR mode
|
|
|
|
2. **Adjust Settings (Optional)**
|
|
- Per-eye resolution (must be multiples of 16)
|
|
- Target frame rate (72, 90, or 120 FPS)
|
|
- Streaming bitrate
|
|
- XR reference space and camera offsets
|
|
|
|
3. **Start Streaming**
|
|
- Click "CONNECT"
|
|
- Grant XR permissions when prompted
|
|
|
|
**Requirements:**
|
|
- CloudXR server running and accessible
|
|
- WebXR-compatible device (VR/AR headset) or desktop browser (IWER loads automatically for emulation)
|
|
|
|
## Architecture
|
|
|
|
### CloudXRClient Class
|
|
|
|
The main application class (`CloudXRClient` in `main.ts`) handles:
|
|
|
|
**Initialization:**
|
|
- UI element management and localStorage persistence
|
|
- Browser capability checks (WebXR, WebGL2, WebRTC)
|
|
- Event listener setup
|
|
|
|
**Connection Flow:**
|
|
1. **WebGL Setup** - Creates high-performance WebGL2 context
|
|
2. **WebXR Session** - Enters immersive VR/AR mode
|
|
3. **Reference Space** - Configures coordinate system for tracking
|
|
4. **CloudXR Session** - Establishes streaming connection to server
|
|
5. **Render Loop** - Sends tracking data, receives video, renders frames
|
|
|
|
**Key Components:**
|
|
- **WebXR Session** - Hardware access (headset, controllers)
|
|
- **WebGL Context** - Video rendering
|
|
- **CloudXR Session** - Streaming management (WebRTC-based)
|
|
- **XRWebGLLayer** - Bridge between WebXR and WebGL
|
|
|
|
## Project Structure
|
|
|
|
```
|
|
simple/
|
|
├── src/
|
|
│ └── main.ts # Main application sample
|
|
├── index.html # UI and form elements sample
|
|
├── package.json # Dependencies and scripts
|
|
├── webpack.common.js # Webpack base configuration sample
|
|
├── webpack.dev.js # Development configuration sample
|
|
├── webpack.prod.js # Production configuration sample
|
|
└── tsconfig.json # TypeScript configuration sample
|
|
```
|
|
|
|
## Code Overview
|
|
|
|
The `main.ts` file contains well-commented code explaining each step:
|
|
|
|
1. **Browser Checks** - Validates WebXR, WebGL2, and WebRTC support
|
|
2. **Connection Setup** - Reads form inputs and validates configuration
|
|
3. **WebGL Initialization** - Creates optimized rendering context
|
|
4. **WebXR Session** - Enters immersive mode with requested features
|
|
5. **CloudXR Setup** - Configures streaming session with event handlers
|
|
6. **Render Loop** - Runs 72-120 times per second:
|
|
- Sends tracking data to server
|
|
- Receives video frame
|
|
- Renders to display
|
|
|
|
Each method includes inline comments explaining the purpose and key concepts.
|
|
|
|
## License
|
|
|
|
See the [LICENSE](LICENSE) file for details.
|