yt lee
623e05f250
Replaces manual H264/TCP stereo streaming with NVIDIA CloudXR for
higher-quality stereoscopic rendering and lower latency.
Changes:
- teleop_xr_agent.py: add --cloudxr flag (enables Isaac Sim XR mode,
disables manual StreamingManager)
- deps/cloudxr/: NVIDIA CloudXR.js SDK (Early Access) with Isaac Lab
teleop React web client
- deps/cloudxr/Dockerfile.wss.proxy: HAProxy WSS proxy for PICO 4 Ultra
HTTPS mode (routes wss://48322 → ws://49100)
- deps/cloudxr/isaac/webpack.dev.js: disable file watching to avoid
EMFILE errors with large node_modules
- deps/cloudxr/INSTALL.md: full setup guide
Usage:
# Start CloudXR Runtime + Isaac Lab
cd ~/IsaacLab && ./docker/container.py start \
--files docker-compose.cloudxr-runtime.patch.yaml \
--env-file .env.cloudxr-runtime
# Run teleop with CloudXR
~/IsaacLab/isaaclab.sh -p teleop_xr_agent.py \
--task Isaac-MindRobot-2i-DualArm-IK-Abs-v0 --cloudxr
# Serve web client
cd deps/cloudxr/isaac && npm run dev-server:https
34 lines
9.9 KiB
HTML
34 lines
9.9 KiB
HTML
<!DOCTYPE html><html class="default" lang="en" data-base=".."><head><meta charset="utf-8"/><meta http-equiv="x-ua-compatible" content="IE=edge"/><title>Isaac Lab Teleop | CloudXR.js SDK Documentation - v6.0.0-beta</title><meta name="description" content="Documentation for CloudXR.js SDK Documentation"/><meta name="viewport" content="width=device-width, initial-scale=1"/><link rel="stylesheet" href="../assets/style.css"/><link rel="stylesheet" href="../assets/highlight.css"/><script defer src="../assets/main.js"></script><script async src="../assets/icons.js" id="tsd-icons-script"></script><script async src="../assets/search.js" id="tsd-search-script"></script><script async src="../assets/navigation.js" id="tsd-nav-script"></script></head><body><script>document.documentElement.dataset.theme = localStorage.getItem("tsd-theme") || "os";document.body.style.display="none";setTimeout(() => app?app.showPage():document.body.style.removeProperty("display"),500)</script><header class="tsd-page-toolbar"><div class="tsd-toolbar-contents container"><div class="table-cell" id="tsd-search"><div class="field"><label for="tsd-search-field" class="tsd-widget tsd-toolbar-icon search no-caption"><svg width="16" height="16" viewBox="0 0 16 16" fill="none" aria-hidden="true"><use href="../assets/icons.svg#icon-search"></use></svg></label><input type="text" id="tsd-search-field" aria-label="Search"/></div><div class="field"><div id="tsd-toolbar-links"></div></div><ul class="results"><li class="state loading">Preparing search index...</li><li class="state failure">The search index is not available</li></ul><a href="../index.html" class="title">CloudXR.js SDK Documentation - v6.0.0-beta</a></div><div class="table-cell" id="tsd-widgets"><a href="#" class="tsd-widget tsd-toolbar-icon menu no-caption" data-toggle="menu" aria-label="Menu"><svg width="16" height="16" viewBox="0 0 16 16" fill="none" aria-hidden="true"><use href="../assets/icons.svg#icon-menu"></use></svg></a></div></div></header><div class="container container-main"><div class="col-content"><div class="tsd-page-title"><ul class="tsd-breadcrumb"><li><a href="../modules.html">CloudXR.js SDK Documentation</a></li><li><a href="Show_Cases.html">Show Cases</a></li><li><a href="Show_Cases.Isaac_Lab_Teleop.html">Isaac Lab Teleop</a></li></ul></div><div class="tsd-panel tsd-typography"><a id="running-isaac-lab-teleoperation-on-meta-quest-3" class="tsd-anchor"></a><h1 class="tsd-anchor-link">Running Isaac Lab Teleoperation on Meta Quest 3<a href="#running-isaac-lab-teleoperation-on-meta-quest-3" aria-label="Permalink" class="tsd-anchor-icon"><svg viewBox="0 0 24 24" aria-hidden="true"><use href="../assets/icons.svg#icon-anchor"></use></svg></a></h1><p>NVIDIA CloudXR enables seamless, high-fidelity immersive streaming to extended reality (XR) devices over the network. Developers can use CloudXR with NVIDIA Isaac Lab to build teleoperation workflows that require immersive XR rendering for increased spatial acuity and hand tracking for teleoperation of dexterous robots.</p>
|
|
<p>In these workflows, a compatible XR system captures operator head and hand motion. CloudXR sends this data to Isaac Lab, which renders and submits stereo views of the robot simulation to CloudXR. CloudXR then encodes and streams the rendered views back to the XR system for display in realtime using a low-latency, GPU-accelerated pipeline.</p>
|
|
<p>This guide provides the Isaac Lab-specific details referenced in the First Run Guide, including installation, CloudXR Runtime configuration, and teleoperation settings.</p>
|
|
<a id="system-requirements" class="tsd-anchor"></a><h2 class="tsd-anchor-link">System Requirements<a href="#system-requirements" aria-label="Permalink" class="tsd-anchor-icon"><svg viewBox="0 0 24 24" aria-hidden="true"><use href="../assets/icons.svg#icon-anchor"></use></svg></a></h2><p>Prior to using CloudXR with Isaac Lab, please review the following system requirements:</p>
|
|
<a id="🖥️-isaac-lab-workstation" class="tsd-anchor"></a><h3 class="tsd-anchor-link">🖥️ <strong>Isaac Lab Workstation</strong><a href="#🖥️-isaac-lab-workstation" aria-label="Permalink" class="tsd-anchor-icon"><svg viewBox="0 0 24 24" aria-hidden="true"><use href="../assets/icons.svg#icon-anchor"></use></svg></a></h3><ul>
|
|
<li>Ubuntu 22.04 or Ubuntu 24.04
|
|
<ul>
|
|
<li><a href="https://docs.docker.com/engine/install/ubuntu/#installation-methods">Docker</a> 26.0.0+, <a href="https://docs.docker.com/compose/install/linux/#install-using-the-repository">Docker Compose</a> 2.25.0+, and the <a href="https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html">NVIDIA Container Toolkit</a>. Refer to the <a href="https://isaac-sim.github.io/IsaacLab/main/source/deployment/docker.html#deployment-docker">Isaac Lab Docker Guide</a> for how to install.</li>
|
|
<li>For details on driver requirements, please see the <a href="https://docs.omniverse.nvidia.com/materials-and-rendering/latest/common/technical-requirements.html">Technical Requirements</a> guide</li>
|
|
<li>Follow the <a href="https://isaac-sim.github.io/IsaacLab/main/source/setup/installation/index.html#local-installation">Isaac Lab installation guide</a> to install Isaac Sim and Isaac Lab.</li>
|
|
<li>Required for best performance: 16 cores Intel Core i9, X-series or higher AMD Ryzen 9, Threadripper or higher</li>
|
|
<li>Required for best performance: 64GB RAM</li>
|
|
<li>Required for best performance: 2x RTX PRO 6000 GPUs (or equivalent e.g. 2x RTX 5090)</li>
|
|
</ul>
|
|
</li>
|
|
</ul>
|
|
<a id="🥽-meta-quest-3" class="tsd-anchor"></a><h3 class="tsd-anchor-link">🥽 <strong>Meta Quest 3</strong><a href="#🥽-meta-quest-3" aria-label="Permalink" class="tsd-anchor-icon"><svg viewBox="0 0 24 24" aria-hidden="true"><use href="../assets/icons.svg#icon-anchor"></use></svg></a></h3><p>Meta Quest 3 (OS version 79 or later).</p>
|
|
<a id="🌐-wifi-6-capable-router" class="tsd-anchor"></a><h3 class="tsd-anchor-link">🌐 <strong>WiFi 6 Capable Router</strong><a href="#🌐-wifi-6-capable-router" aria-label="Permalink" class="tsd-anchor-icon"><svg viewBox="0 0 24 24" aria-hidden="true"><use href="../assets/icons.svg#icon-anchor"></use></svg></a></h3><ul>
|
|
<li>A strong wireless connection is essential for a high-quality streaming experience. Refer to the requirements of <a href="Networking_Setup.html">Networking Setup</a> for more details.</li>
|
|
<li>We recommend using a dedicated router, as concurrent usage will degrade quality</li>
|
|
<li>The Meta Quest 3 and Isaac Lab workstation must be IP-reachable from one another
|
|
<ul>
|
|
<li>Many institutional wireless networks will prevent devices from reaching each other, resulting in the Meta Quest 3 being unable to find the Isaac Lab workstation on the network</li>
|
|
<li>See <a href="Networking_Setup.html">Networking Setup</a> guide for more details</li>
|
|
</ul>
|
|
</li>
|
|
</ul>
|
|
<a id="next" class="tsd-anchor"></a><h2 class="tsd-anchor-link">Next<a href="#next" aria-label="Permalink" class="tsd-anchor-icon"><svg viewBox="0 0 24 24" aria-hidden="true"><use href="../assets/icons.svg#icon-anchor"></use></svg></a></h2><ol>
|
|
<li><a href="Show_Cases.Isaac_Lab_Teleop.Setting_Up_Server_for_Teleoperation.html">Running Isaac Lab with the CloudXR Runtime</a></li>
|
|
<li><a href="Show_Cases.Isaac_Lab_Teleop.Teleoperating_on_Meta_Quest_3.html">Running Teleop Client on Meta Quest 3</a></li>
|
|
</ol>
|
|
<p>If you encounter issues, please see <a href="Show_Cases.Isaac_Lab_Teleop.Isaac_Lab_Teleop_Troubleshooting.html">Troubleshooting</a>.</p>
|
|
</div></div><div class="col-sidebar"><div class="page-menu"><div class="tsd-navigation settings"><details class="tsd-accordion"><summary class="tsd-accordion-summary"><h3><svg width="20" height="20" viewBox="0 0 24 24" fill="none" aria-hidden="true"><use href="../assets/icons.svg#icon-chevronDown"></use></svg>Settings</h3></summary><div class="tsd-accordion-details"><div class="tsd-filter-visibility"><span class="settings-label">Member Visibility</span><ul id="tsd-filter-options"><li class="tsd-filter-item"><label class="tsd-filter-input"><input type="checkbox" id="tsd-filter-inherited" name="inherited" checked/><svg width="32" height="32" viewBox="0 0 32 32" aria-hidden="true"><rect class="tsd-checkbox-background" width="30" height="30" x="1" y="1" rx="6" fill="none"></rect><path class="tsd-checkbox-checkmark" d="M8.35422 16.8214L13.2143 21.75L24.6458 10.25" stroke="none" stroke-width="3.5" stroke-linejoin="round" fill="none"></path></svg><span>Inherited</span></label></li></ul></div><div class="tsd-theme-toggle"><label class="settings-label" for="tsd-theme">Theme</label><select id="tsd-theme"><option value="os">OS</option><option value="light">Light</option><option value="dark">Dark</option></select></div></div></details></div><details open class="tsd-accordion tsd-page-navigation"><summary class="tsd-accordion-summary"><h3><svg width="20" height="20" viewBox="0 0 24 24" fill="none" aria-hidden="true"><use href="../assets/icons.svg#icon-chevronDown"></use></svg>On This Page</h3></summary><div class="tsd-accordion-details"><a href="#running-isaac-lab-teleoperation-on-meta-quest-3"><span>Running <wbr/>Isaac <wbr/>Lab <wbr/>Teleoperation on <wbr/>Meta <wbr/>Quest 3</span></a><ul><li><a href="#system-requirements"><span>System <wbr/>Requirements</span></a></li><li><ul><li><a href="#🖥️-isaac-lab-workstation"><span>🖥️ <wbr/>Isaac <wbr/>Lab <wbr/>Workstation</span></a></li><li><a href="#🥽-meta-quest-3"><span>🥽 <wbr/>Meta <wbr/>Quest 3</span></a></li><li><a href="#🌐-wifi-6-capable-router"><span>🌐 <wbr/>Wi<wbr/>Fi 6 <wbr/>Capable <wbr/>Router</span></a></li></ul></li><li><a href="#next"><span>Next</span></a></li></ul></div></details></div><div class="site-menu"><nav class="tsd-navigation"><a href="../modules.html">CloudXR.js SDK Documentation - v6.0.0-beta</a><ul class="tsd-small-nested-navigation" id="tsd-nav-container"><li>Loading...</li></ul></nav></div></div></div><footer><p class="tsd-generator">Generated using <a href="https://typedoc.org/" target="_blank">TypeDoc</a></p></footer><div class="overlay"></div></body></html>
|