Add CloudXR VR streaming support for PICO 4 Ultra (Early Access)
Replaces manual H264/TCP stereo streaming with NVIDIA CloudXR for
higher-quality stereoscopic rendering and lower latency.
Changes:
- teleop_xr_agent.py: add --cloudxr flag (enables Isaac Sim XR mode,
disables manual StreamingManager)
- deps/cloudxr/: NVIDIA CloudXR.js SDK (Early Access) with Isaac Lab
teleop React web client
- deps/cloudxr/Dockerfile.wss.proxy: HAProxy WSS proxy for PICO 4 Ultra
HTTPS mode (routes wss://48322 → ws://49100)
- deps/cloudxr/isaac/webpack.dev.js: disable file watching to avoid
EMFILE errors with large node_modules
- deps/cloudxr/INSTALL.md: full setup guide
Usage:
# Start CloudXR Runtime + Isaac Lab
cd ~/IsaacLab && ./docker/container.py start \
--files docker-compose.cloudxr-runtime.patch.yaml \
--env-file .env.cloudxr-runtime
# Run teleop with CloudXR
~/IsaacLab/isaaclab.sh -p teleop_xr_agent.py \
--task Isaac-MindRobot-2i-DualArm-IK-Abs-v0 --cloudxr
# Serve web client
cd deps/cloudxr/isaac && npm run dev-server:https
This commit is contained in:
118
deps/cloudxr/simple/README.md
vendored
Normal file
118
deps/cloudxr/simple/README.md
vendored
Normal file
@@ -0,0 +1,118 @@
|
||||
# CloudXR.js Simple Example
|
||||
|
||||
A minimal WebGL example demonstrating WebXR streaming from a CloudXR server to a web browser. This example shows how to integrate WebXR with CloudXR to stream immersive VR/AR content.
|
||||
|
||||
> **Note:** This example is for learning purposes, not production use.
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Prerequisites
|
||||
- Node.js (v20 or higher)
|
||||
|
||||
### Installation
|
||||
|
||||
1. **Navigate to the example folder**
|
||||
```bash
|
||||
cd simple
|
||||
```
|
||||
|
||||
2. **Install Dependencies**
|
||||
```bash
|
||||
# For this early access release, please run the following to install SDK from the given tarball. This step will not be needed when SDK is publicly accessible.
|
||||
npm install ../nvidia-cloudxr-6.0.0-beta.tgz
|
||||
|
||||
npm install
|
||||
```
|
||||
|
||||
3. **Build the Application**
|
||||
```bash
|
||||
npm run build
|
||||
```
|
||||
|
||||
4. **Start Development Server**
|
||||
```bash
|
||||
npm run dev-server
|
||||
```
|
||||
|
||||
5. **Open in Browser**
|
||||
- Navigate to `http://localhost:8080` (or the port shown in terminal)
|
||||
- For desktop browsers, IWER (Immersive Web Emulator Runtime) will automatically load to emulate a Meta Quest 3 headset
|
||||
|
||||
### Basic Usage
|
||||
|
||||
1. **Configure Connection**
|
||||
- Enter CloudXR server IP address (default: localhost)
|
||||
- Set port (default: 49100)
|
||||
- Select AR or VR mode
|
||||
|
||||
2. **Adjust Settings (Optional)**
|
||||
- Per-eye resolution (must be multiples of 16)
|
||||
- Target frame rate (72, 90, or 120 FPS)
|
||||
- Streaming bitrate
|
||||
- XR reference space and camera offsets
|
||||
|
||||
3. **Start Streaming**
|
||||
- Click "CONNECT"
|
||||
- Grant XR permissions when prompted
|
||||
|
||||
**Requirements:**
|
||||
- CloudXR server running and accessible
|
||||
- WebXR-compatible device (VR/AR headset) or desktop browser (IWER loads automatically for emulation)
|
||||
|
||||
## Architecture
|
||||
|
||||
### CloudXRClient Class
|
||||
|
||||
The main application class (`CloudXRClient` in `main.ts`) handles:
|
||||
|
||||
**Initialization:**
|
||||
- UI element management and localStorage persistence
|
||||
- Browser capability checks (WebXR, WebGL2, WebRTC)
|
||||
- Event listener setup
|
||||
|
||||
**Connection Flow:**
|
||||
1. **WebGL Setup** - Creates high-performance WebGL2 context
|
||||
2. **WebXR Session** - Enters immersive VR/AR mode
|
||||
3. **Reference Space** - Configures coordinate system for tracking
|
||||
4. **CloudXR Session** - Establishes streaming connection to server
|
||||
5. **Render Loop** - Sends tracking data, receives video, renders frames
|
||||
|
||||
**Key Components:**
|
||||
- **WebXR Session** - Hardware access (headset, controllers)
|
||||
- **WebGL Context** - Video rendering
|
||||
- **CloudXR Session** - Streaming management (WebRTC-based)
|
||||
- **XRWebGLLayer** - Bridge between WebXR and WebGL
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
simple/
|
||||
├── src/
|
||||
│ └── main.ts # Main application sample
|
||||
├── index.html # UI and form elements sample
|
||||
├── package.json # Dependencies and scripts
|
||||
├── webpack.common.js # Webpack base configuration sample
|
||||
├── webpack.dev.js # Development configuration sample
|
||||
├── webpack.prod.js # Production configuration sample
|
||||
└── tsconfig.json # TypeScript configuration sample
|
||||
```
|
||||
|
||||
## Code Overview
|
||||
|
||||
The `main.ts` file contains well-commented code explaining each step:
|
||||
|
||||
1. **Browser Checks** - Validates WebXR, WebGL2, and WebRTC support
|
||||
2. **Connection Setup** - Reads form inputs and validates configuration
|
||||
3. **WebGL Initialization** - Creates optimized rendering context
|
||||
4. **WebXR Session** - Enters immersive mode with requested features
|
||||
5. **CloudXR Setup** - Configures streaming session with event handlers
|
||||
6. **Render Loop** - Runs 72-120 times per second:
|
||||
- Sends tracking data to server
|
||||
- Receives video frame
|
||||
- Renders to display
|
||||
|
||||
Each method includes inline comments explaining the purpose and key concepts.
|
||||
|
||||
## License
|
||||
|
||||
See the [LICENSE](LICENSE) file for details.
|
||||
Reference in New Issue
Block a user