Add CloudXR VR streaming support for PICO 4 Ultra (Early Access)
Replaces manual H264/TCP stereo streaming with NVIDIA CloudXR for
higher-quality stereoscopic rendering and lower latency.
Changes:
- teleop_xr_agent.py: add --cloudxr flag (enables Isaac Sim XR mode,
disables manual StreamingManager)
- deps/cloudxr/: NVIDIA CloudXR.js SDK (Early Access) with Isaac Lab
teleop React web client
- deps/cloudxr/Dockerfile.wss.proxy: HAProxy WSS proxy for PICO 4 Ultra
HTTPS mode (routes wss://48322 → ws://49100)
- deps/cloudxr/isaac/webpack.dev.js: disable file watching to avoid
EMFILE errors with large node_modules
- deps/cloudxr/INSTALL.md: full setup guide
Usage:
# Start CloudXR Runtime + Isaac Lab
cd ~/IsaacLab && ./docker/container.py start \
--files docker-compose.cloudxr-runtime.patch.yaml \
--env-file .env.cloudxr-runtime
# Run teleop with CloudXR
~/IsaacLab/isaaclab.sh -p teleop_xr_agent.py \
--task Isaac-MindRobot-2i-DualArm-IK-Abs-v0 --cloudxr
# Serve web client
cd deps/cloudxr/isaac && npm run dev-server:https
This commit is contained in:
261
deps/cloudxr/INSTALL.md
vendored
Normal file
261
deps/cloudxr/INSTALL.md
vendored
Normal file
@@ -0,0 +1,261 @@
|
||||
# CloudXR VR Teleoperation Setup Guide
|
||||
|
||||
将 Isaac Lab 仿真画面通过 NVIDIA CloudXR 流式传输到 PICO 4 Ultra，实现沉浸式 VR 遥操作。
|
||||
|
||||
## 架构
|
||||
|
||||
```
|
||||
Isaac Sim (本地) ──OpenXR──► CloudXR Runtime (Docker)
|
||||
│
|
||||
WebSocket (port 49100)
|
||||
│
|
||||
HAProxy WSS Proxy (port 48322)
|
||||
│
|
||||
PICO 4 Ultra 浏览器
|
||||
https://<IP>:8080 (Web App)
|
||||
```
|
||||
|
||||
## 系统要求
|
||||
|
||||
- Ubuntu 22.04 / 24.04
|
||||
- NVIDIA GPU (RTX 系列)
|
||||
- NVIDIA Driver 最新版
|
||||
- Docker + NVIDIA Container Toolkit
|
||||
- Isaac Lab 本地安装（`~/IsaacLab`）
|
||||
- PICO 4 Ultra (OS 15.4.4U 或更高)
|
||||
- CloudXR Early Access 资格（[申请地址](https://developer.nvidia.com/cloudxr-sdk)）
|
||||
- Node.js 20 LTS
|
||||
|
||||
---
|
||||
|
||||
## 一、系统环境准备
|
||||
|
||||
### 1.1 安装 NVIDIA Container Toolkit
|
||||
|
||||
```bash
|
||||
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
|
||||
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
|
||||
sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
|
||||
sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y nvidia-container-toolkit
|
||||
sudo nvidia-ctk runtime configure --runtime=docker
|
||||
sudo systemctl restart docker
|
||||
```
|
||||
|
||||
### 1.2 配置 Docker（国内镜像 + NVIDIA runtime）
|
||||
|
||||
```bash
|
||||
sudo tee /etc/docker/daemon.json << 'EOF'
|
||||
{
|
||||
"registry-mirrors": [
|
||||
"https://docker.1ms.run",
|
||||
"https://docker.1panel.live",
|
||||
"https://hub.rat.dev"
|
||||
],
|
||||
"runtimes": {
|
||||
"nvidia": {
|
||||
"path": "nvidia-container-runtime",
|
||||
"runtimeArgs": []
|
||||
}
|
||||
}
|
||||
}
|
||||
EOF
|
||||
sudo systemctl restart docker
|
||||
```
|
||||
|
||||
### 1.3 开放防火墙端口
|
||||
|
||||
```bash
|
||||
sudo ufw allow 47998:48000/udp
|
||||
sudo ufw allow 48005/udp
|
||||
sudo ufw allow 48008/udp
|
||||
sudo ufw allow 48012/udp
|
||||
sudo ufw allow 48010/tcp
|
||||
sudo ufw allow 49100/tcp
|
||||
sudo ufw allow 48322/tcp
|
||||
sudo ufw allow 8080/tcp
|
||||
```
|
||||
|
||||
### 1.4 安装 Node.js 20 LTS（通过 nvm）
|
||||
|
||||
```bash
|
||||
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.1/install.sh | bash
|
||||
source ~/.bashrc
|
||||
nvm install 20
|
||||
nvm use 20
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 二、获取 CloudXR Early Access SDK
|
||||
|
||||
1. 前往 [ngc.nvidia.com](https://ngc.nvidia.com) 获取 API Key
|
||||
2. 登录 NGC Docker registry：
|
||||
|
||||
```bash
|
||||
sudo docker login nvcr.io
|
||||
# Username: $oauthtoken
|
||||
# Password: <你的 NGC API Key>
|
||||
```
|
||||
|
||||
3. 下载 CloudXR.js SDK（从 NGC Early Access 页面获取），解压到：
|
||||
```
|
||||
deps/cloudxr/
|
||||
├── nvidia-cloudxr-6.0.0-beta.tgz
|
||||
├── isaac/ # React Web App
|
||||
├── docs/
|
||||
└── Dockerfile.wss.proxy
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 三、配置 Isaac Lab Docker
|
||||
|
||||
修改 `~/IsaacLab/docker/.env.cloudxr-runtime`：
|
||||
|
||||
```ini
|
||||
CLOUDXR_RUNTIME_BASE_IMAGE_ARG=nvcr.io/nvidia/cloudxr-runtime-early-access
|
||||
CLOUDXR_RUNTIME_VERSION_ARG=6.0.1-webrtc
|
||||
ACCEPT_EULA=yes
|
||||
```
|
||||
|
||||
修改 `~/IsaacLab/docker/docker-compose.cloudxr-runtime.patch.yaml`，将 cloudxr-runtime service 的 `ports` 替换为 `network_mode: host`：
|
||||
|
||||
```yaml
|
||||
services:
|
||||
cloudxr-runtime:
|
||||
image: ${CLOUDXR_RUNTIME_BASE_IMAGE_ARG}:${CLOUDXR_RUNTIME_VERSION_ARG}
|
||||
network_mode: host
|
||||
# ... 其余保持不变
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 四、构建 WSS 代理（HAProxy）
|
||||
|
||||
```bash
|
||||
cd deps/cloudxr
|
||||
|
||||
# 构建镜像
|
||||
sudo docker build -t websocket-ssl-proxy -f Dockerfile.wss.proxy .
|
||||
|
||||
# 启动代理（监听 48322，转发到 CloudXR Runtime 的 49100）
|
||||
sudo docker run -d --name wss-proxy \
|
||||
--network host \
|
||||
-e BACKEND_HOST=localhost \
|
||||
-e BACKEND_PORT=49100 \
|
||||
-e PROXY_PORT=48322 \
|
||||
websocket-ssl-proxy
|
||||
|
||||
# 验证
|
||||
sudo docker logs wss-proxy
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 五、构建 Web 客户端
|
||||
|
||||
```bash
|
||||
cd deps/cloudxr/isaac
|
||||
|
||||
# 安装依赖（需要先安装本地 CloudXR SDK）
|
||||
npm install ../nvidia-cloudxr-6.0.0-beta.tgz
|
||||
npm install
|
||||
|
||||
# 启动 HTTPS 开发服务器（端口 8080）
|
||||
npm run dev-server:https
|
||||
```
|
||||
|
||||
> **注意**：首次运行会下载 WebXR controller profile 资源，需要网络连接。
|
||||
|
||||
---
|
||||
|
||||
## 六、启动流程
|
||||
|
||||
每次使用前按以下顺序启动：
|
||||
|
||||
### 6.1 启动 CloudXR Runtime + Isaac Lab
|
||||
|
||||
```bash
|
||||
cd ~/IsaacLab
|
||||
./docker/container.py start \
|
||||
--files docker-compose.cloudxr-runtime.patch.yaml \
|
||||
--env-file .env.cloudxr-runtime
|
||||
|
||||
# 进入 Isaac Lab 容器
|
||||
./docker/container.py enter base
|
||||
```
|
||||
|
||||
容器内运行 MindBot 遥操作：
|
||||
|
||||
```bash
|
||||
~/IsaacLab/isaaclab.sh -p scripts/environments/teleoperation/teleop_xr_agent.py \
|
||||
--task Isaac-MindRobot-2i-DualArm-IK-Abs-v0 \
|
||||
--cloudxr
|
||||
```
|
||||
|
||||
Isaac Sim 启动后：**AR 面板 → OpenXR → System OpenXR Runtime → 点击 Start AR**
|
||||
|
||||
### 6.2 启动 WSS 代理
|
||||
|
||||
```bash
|
||||
sudo docker start wss-proxy
|
||||
```
|
||||
|
||||
### 6.3 启动 Web 服务
|
||||
|
||||
```bash
|
||||
cd deps/cloudxr/isaac
|
||||
npm run dev-server:https
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 七、PICO 4 Ultra 配置
|
||||
|
||||
### 7.1 启用手部追踪
|
||||
|
||||
设置 → Interaction → 选择 **Auto Switch between Hands & Controllers**
|
||||
|
||||
### 7.2 接受证书
|
||||
|
||||
1. 打开 PICO 浏览器，访问 `https://<工作站IP>:48322/`
|
||||
- 点击 **Advanced** → **Proceed (unsafe)**
|
||||
- 看到 "501 Not Implemented" 是正常的
|
||||
2. 访问 `https://<工作站IP>:8080/`
|
||||
- 点击 **Advanced** → **Proceed (unsafe)**
|
||||
- Web App 加载成功
|
||||
|
||||
### 7.3 连接
|
||||
|
||||
1. 在 Web App 中输入工作站 IP
|
||||
2. 点击 **CONNECT**
|
||||
3. 点击 **Enter XR** 进入沉浸式模式
|
||||
|
||||
---
|
||||
|
||||
## 八、验证端口状态
|
||||
|
||||
```bash
|
||||
sudo ss -tlnp | grep -E "49100|48322|8080"
|
||||
```
|
||||
|
||||
预期输出：
|
||||
```
|
||||
LISTEN 0.0.0.0:49100 cloudxr-service # CloudXR Runtime
|
||||
LISTEN 0.0.0.0:48322 haproxy # WSS Proxy
|
||||
LISTEN 0.0.0.0:8080 node # Web App
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 常见问题
|
||||
|
||||
| 症状 | 原因 | 解决 |
|
||||
|---|---|---|
|
||||
| 503 Service Unavailable (48322) | CloudXR Runtime 未运行 | 重启 docker-cloudxr-runtime-1 容器 |
|
||||
| 501 Not Implemented (48322) | 正常 — HAProxy 只处理 WebSocket | 直接接受证书即可 |
|
||||
| Stream start failed 0xC0F22202 | HTTPS + ws:// 混合内容 | 必须先接受 48322 代理证书 |
|
||||
| EMFILE: too many open files | webpack 文件监听超限 | 已通过 `watchOptions.ignored` 修复 |
|
||||
| Isaac Sim AR 面板看不到 | 未传 --cloudxr 或 --xr flag | 使用 `--cloudxr` 启动脚本 |
|
||||
Reference in New Issue
Block a user