hangX/mindbot
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add CloudXR VR streaming support for PICO 4 Ultra (Early Access)

Browse Source 
Replaces manual H264/TCP stereo streaming with NVIDIA CloudXR for
higher-quality stereoscopic rendering and lower latency.

Changes:
- teleop_xr_agent.py: add --cloudxr flag (enables Isaac Sim XR mode,
  disables manual StreamingManager)
- deps/cloudxr/: NVIDIA CloudXR.js SDK (Early Access) with Isaac Lab
  teleop React web client
- deps/cloudxr/Dockerfile.wss.proxy: HAProxy WSS proxy for PICO 4 Ultra
  HTTPS mode (routes wss://48322 → ws://49100)
- deps/cloudxr/isaac/webpack.dev.js: disable file watching to avoid
  EMFILE errors with large node_modules
- deps/cloudxr/INSTALL.md: full setup guide

Usage:
  # Start CloudXR Runtime + Isaac Lab
  cd ~/IsaacLab && ./docker/container.py start \
      --files docker-compose.cloudxr-runtime.patch.yaml \
      --env-file .env.cloudxr-runtime

  # Run teleop with CloudXR
  ~/IsaacLab/isaaclab.sh -p teleop_xr_agent.py \
      --task Isaac-MindRobot-2i-DualArm-IK-Abs-v0 --cloudxr

  # Serve web client
  cd deps/cloudxr/isaac && npm run dev-server:https
This commit is contained in:
yt lee
2026-03-26 14:29:03 +08:00
parent eef7ff838d
commit 623e05f250
133 changed files with 24869 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

307
deps/cloudxr/simple/LICENSE vendored Normal file
View File

@@ -0,0 +1,307 @@
NVIDIA SOFTWARE EVALUATION LICENSE AGREEMENT
IMPORTANT NOTICE – PLEASE READ AND AGREE BEFORE USING THE SOFTWARE
This software evaluation license agreement (“Agreement”) is a legal agreement between you, whether an
individual or entity, (“you”) and NVIDIA Corporation and its affiliates (“NVIDIA”) and governs the use of certain
NVIDIA CloudXR software and documentation that NVIDIA delivers to you under this Agreement (“Software”).
NVIDIA and you are each a “party” and collectively the “parties.”
This Agreement can be accepted only by an adult of legal age of majority in the country in which the Software is
used. If you don’t have the required age or authority to accept this Agreement, or if you don’t accept all the
terms and conditions of this Agreement, do not use the Software.
1. License Grants.
1.1 License Grant to You. The Software is licensed, not sold. Subject to the terms of this Agreement,
NVIDIA grants you a limited, non-exclusive, revocable, non-transferable, non-sublicensable (except
as expressly granted in this Agreement), license to:
(a) access, install and use copies of the Software,
(b) configure the Software using configuration files provided (if applicable),
(c) modify and create derivative works of any source code NVIDIA delivers to you as part of the
Software (“Derivatives”) (if applicable).
All the foregoing grants are only for internal test and evaluation purposes and, as applicable, for use (a) in
client systems, or (b) in server systems with NVIDIA GPUs (“Purpose”).
1.2 License Grant to NVIDIA. Subject to the terms of this Agreement, you grant NVIDIA a non-exclusive,
perpetual, irrevocable, sublicensable, worldwide, royalty-free, fully paid-up and transferable license,
under your intellectual property rights, to publicly perform, publicly display, reproduce, use, make,
have made, sell, offer for sale, distribute (through multiple tiers of distribution), import, create
derivative works of and otherwise commercialize and exploit at NVIDIA’s discretion any Derivatives
created by or for you. You may, but are not required to, deliver any Derivatives to NVIDIA.
2. License Restrictions. Your license to use the Software and Derivatives is restricted as stated in this “License
Restrictions” Section. You will cooperate with NVIDIA and, upon NVIDIA’s written request, you will confirm
in writing and provide reasonably requested information to verify your compliance with the terms of this
Agreement. You may not:
2.1 Use the Software or Derivatives for any purpose other than the Purpose, including but not limited to
in production;
2.2 Sell, rent, sublicense, transfer, distribute or otherwise make available to others (except Authorized
Users as stated in the “Authorized Users” Section) any portion of the Software or Derivatives, except
as expressly granted in Section 1.1 (“License Grant to You”);
2.3 Reverse engineer, decompile, or disassemble the Software components provided in binary form, nor
attempt in any other manner to obtain source code of such Software;
2.4 Modify or create derivative works of the Software, except as expressly granted in Section 1.1
(“License Grant to You”);
2.5 Change or remove copyright or other proprietary notices in the Software;
2.6 Bypass, disable, or circumvent any technical limitation, encryption, security, digital rights
management or authentication mechanism in the Software;
2.7 Use the Software or Derivatives in any manner that would cause them to become subject to an open
source software license; subject to the terms in Section 7 (“Components Under Other Licenses”);
2.8 Use the Software or Derivatives for the purpose of developing competing products or technologies
or assist a third party in such activities;
2.9 Replace any Software components governed by this Agreement with other software that
implements NVIDIA APIs;
2.10 Use the Software or Derivatives in violation of any applicable law or regulation in the relevant
jurisdictions; or
2.11 Use the Software in or with any system or application where the use or failure of such system or
application developed or deployed with Software could result in injury, death or catastrophic
damage (“Mission Critical Applications”). NVIDIA will not be liable to you or any third party, in whole
or in part, for any claims or damages arising from uses in Mission Critical Applications.
2.12 Disclose any evaluation or test results regarding the Software or Derivatives without NVIDIA’s prior
written consent.
3. Authorized Users. You may allow employees and contractors of your entity or of your subsidiary(ies), and
for educational institutions also enrolled students, to internally access and use the Software as authorized
by this Agreement from your secure network to perform the work authorized by this Agreement on your
behalf. You are responsible for the compliance with the terms of this Agreement by your authorized users.
Any act or omission that if committed by you would constitute a breach of this Agreement will be deemed to
constitute a breach of this Agreement if committed by your authorized users.
4. Pre-Release. Software versions identified as alpha, beta, preview, early access or otherwise as pre-release
may not be fully functional, may contain errors or design flaws, and may have reduced or different security,
privacy, availability and reliability standards relative to NVIDIA commercial offerings. You use pre-release
Software at your own risk. NVIDIA did not design or test the Software for use in production or business
critical systems. NVIDIA may choose not to make available a commercial version of pre-release Software.
NVIDIA may also choose to abandon development and terminate the availability of pre-release Software at
any time without liability.
5. Your Privacy: Collection and Use of Information.
5.1 Privacy Policy. Please review the NVIDIA Privacy Policy, located at https://www.nvidia.com/enus/about-nvidia/privacy-policy, which explains NVIDIA’s policy for collecting and using data, as well
as visit the NVIDIA Privacy Center, located at https://www.nvidia.com/en-us/privacy-center, to
manage your consent and privacy preferences.
5.2 Collection Purposes. You also acknowledge that the Software collects data for the following
purposes: (a) properly configure and optimize products for use with Software; and (b) improve
NVIDIA products and services. Information collected by the Software includes: (i) application
configuration; (ii) browser version; (iii) and session metadata (i.e. performance and usage
statistics). Additionally, NVIDIA may collect certain personal information, such as your name
and email address or those of your authorized users, and other information necessary to
authenticate and enable you or your authorized users’ access to the Software. Where appropriate
you will disclose to, and obtain any necessary consent from, your authorized users to allow NVIDIA
to collect such information.
5.3 Third Party Privacy Practices. The Software may contain links to third party websites and services.
NVIDIA encourages you to review the privacy statements on those sites and services that you choose
to visit to understand how they may collect, use and share your data. NVIDIA is not responsible for
the privacy statements or practices of third-party sites or services.
6. Updates. NVIDIA may at any time and at its option, change, discontinue, or deprecate any part, or all, of the
Software, or change or remove features or functionality, or make available patches, workarounds or other
updates to the Software. Unless the updates are provided with their separate governing terms, they are
deemed part of the Software licensed to you under this Agreement, and your continued use of the Software
is deemed acceptance of such changes.
7. Components Under Other Licenses. The Software may include or be distributed with components provided
with separate legal notices or terms that accompany the components, such as open source software licenses
and other license terms (“Other Licenses”). The components are subject to the applicable Other Licenses,
including any proprietary notices, disclaimers, requirements and extended use rights; except that this
Agreement will prevail regarding the use of third-party open source software, unless a third-party open
source software license requires its license terms to prevail. Open source software license means any
software, data or documentation subject to any license identified as an open source license by the Open
Source Initiative (http://opensource.org), Free Software Foundation (http://www.fsf.org) or other similar
open source organization or listed by the Software Package Data Exchange (SPDX) Workgroup under the
Linux Foundation (http://www.spdx.org).
8. Ownership.
8.1 NVIDIA Ownership. The Software, including all intellectual property rights, is and will remain the sole
and exclusive property of NVIDIA or its licensors. Except as expressly granted in this Agreement,
(a) NVIDIA reserves all rights, interests and remedies in connection with the Software, and (b) no
other license or right is granted to you by implication, estoppel or otherwise.
8.2 Your Ownership. Subject to the rights of NVIDIA and its suppliers in the Software, which continue to
be licensed as stated in this Agreement, even when incorporated in your products, and the extent
permitted by applicable law, as between you and NVIDIA, you hold all rights, title and interest in and
to your services, applications and Derivatives you develop as permitted in this Agreement including
their respective intellectual property rights.
9. Feedback. You may, but are not obligated to, provide suggestions, requests, fixes, modifications,
enhancements or other feedback regarding your use of the Software (“Feedback”). Feedback, even if
designated as confidential by you, will not create any confidentiality obligation for NVIDIA or its affiliates. If
you provide Feedback, you hereby grant NVIDIA, its affiliates and its designees a nonexclusive, perpetual,
irrevocable, sublicensable, worldwide, royalty-free, fully paid-up and transferable license, under your
intellectual property rights, to publicly perform, publicly display, reproduce, use, make, have made, sell,
offer for sale, distribute (through multiple tiers of distribution), import, create derivative works of and
otherwise commercialize and exploit the Feedback at NVIDIA’s discretion.
10. Confidentiality. You may use confidential information only to exercise your rights and perform your
obligations under this Agreement. You will not disclose, nor authorize others to disclose NVIDIA Confidential
Information to any third party, except as expressly authorized in this Agreement and as necessary for the
Purpose, without obtaining NVIDIA’s prior written approval. Each recipient of confidential information must
be subject to a written agreement that includes confidentiality obligations consistent with these terms and
must have a need to know for the Purpose. You will protect the NVIDIA Confidential Information with at
least the same degree of care that you use to protect your own similar confidential and proprietary
information, but no less than a reasonable degree of care. Confidential information includes, but is not
limited to, the Software, including its features and functionality, Derivatives, and any results of
benchmarking or other competitive analysis or regression or performance data relating to the Software.
No Publicity. You may not issue any public statements about this Agreement, disclose the Software or
Derivatives, or any information or results related to your use of the Software, without prior written approval
of NVIDIA.
11. Term and Termination.
11.1 Term. This Agreement has a duration of twelve (12) months starting from the date of initial
download (even if you download the same version or updates of the Software later and it is
accompanied by this Agreement or another Agreement), unless terminated earlier in accordance
with this Agreement.
11.2 Termination for Convenience. Either party may terminate this Agreement at any time with thirty (30)
days’ advance written notice to the other party.
11.3 Termination for Cause. If you commence or participate in any legal proceeding against NVIDIA with
respect to the Software, this Agreement will terminate immediately without notice. Either party may
terminate this Agreement upon notice for cause if:
(a) the other party fails to cure a material breach of this Agreement within ten (10) days of the
non-breaching party’s notice of the breach; or
(b) the other party breaches its confidentiality obligations or license rights under this
Agreement, which termination will be effective immediately upon written notice.
11.4 Effect of Termination. Upon any expiration or termination of this Agreement, you will promptly
(a) stop using and return, delete or destroy NVIDIA confidential information and all Software
received under this Agreement, and (b) delete or destroy Derivatives created under this Agreement,
unless an authorized NVIDIA representative provides prior written approval that you may keep a
copy of the Derivatives solely for archival purposes. Upon written request, you will certify in writing
that you have complied with your obligations under this “Effect of Termination” Section.
11.5 Survival. The “License Grant to NVIDIA”, “Updates”, “Components Under Other Licenses”,
“Ownership”, “Feedback”, “Confidentiality”, “No Publicity”, “Effect of Termination”, “Survival”,
“Disclaimer of Warranties”, “Limitation of Liability”, “Indemnity” and all “General” Sections of this
Agreement will survive any expiration or termination of this Agreement.
12. Disclaimer of Warranties. THE SOFTWARE IS PROVIDED BY NVIDIA AS-IS AND WITH ALL FAULTS. TO THE
MAXIMUM EXTENT PERMITTED BY APPLICABLE LAW, NVIDIA DISCLAIMS ALL WARRANTIES AND
REPRESENTATIONS OF ANY KIND, WHETHER EXPRESS, IMPLIED OR STATUTORY, RELATING TO OR ARISING
UNDER THIS AGREEMENT, INCLUDING, WITHOUT LIMITATION, THE WARRANTIES OF TITLE,
NONINFRINGEMENT, MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, USAGE OF TRADE AND
COURSE OF DEALING. NVIDIA DOES NOT WARRANT OR ASSUME RESPONSIBILITY FOR THE ACCURACY OR
COMPLETENESS OF ANY THIRD-PARTY INFORMATION, TEXT, GRAPHICS, LINKS CONTAINED IN THE
SOFTWARE. WITHOUT LIMITING THE FOREGOING, NVIDIA DOES NOT WARRANT THAT THE SOFTWARE WILL
MEET YOUR REQUIREMENTS, ANY DEFECTS OR ERRORS WILL BE CORRECTED, ANY CERTAIN CONTENT WILL
BE AVAILABLE; OR THAT THE SOFTWARE IS FREE OF VIRUSES OR OTHER HARMFUL COMPONENTS. NO
INFORMATION OR ADVICE GIVEN BY NVIDIA WILL IN ANY WAY INCREASE THE SCOPE OF ANY WARRANTY
EXPRESSLY PROVIDED IN THIS AGREEMENT. YOU ARE SOLELY RESPONSIBLE FOR DETERMINING THE
APPROPRIATENESS OF USING THE SOFTWARE OR DERIVATIVES AND ASSUME ANY RISKS ASSOCIATED WITH
YOUR USE OF THE SOFTWARE OR DERIVATIVES.
13. Limitations of Liability.
13.1 EXCLUSIONS. TO THE MAXIMUM EXTENT PERMITTED BY APPLICABLE LAW, IN NO EVENT WILL
NVIDIA BE LIABLE FOR ANY (A) INDIRECT, PUNITIVE, SPECIAL, INCIDENTAL OR CONSEQUENTIAL
DAMAGES, OR (B) DAMAGES FOR THE (I) COST OF PROCURING SUBSTITUTE GOODS OR (II) LOSS OF
PROFITS, REVENUES, USE, DATA OR GOODWILL ARISING OUT OF OR RELATED TO THIS AGREEMENT,
WHETHER BASED ON BREACH OF CONTRACT, TORT (INCLUDING NEGLIGENCE), STRICT LIABILITY, OR
OTHERWISE, AND EVEN IF NVIDIA HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES AND
EVEN IF A PARTY’S REMEDIES FAIL THEIR ESSENTIAL PURPOSE.
13.2 DAMAGES CAP. ADDITIONALLY, TO THE MAXIMUM EXTENT PERMITTED BY APPLICABLE LAW,
NVIDIA’S TOTAL CUMULATIVE AGGREGATE LIABILITY FOR ANY AND ALL LIABILITIES, OBLIGATIONS OR
CLAIMS ARISING OUT OF OR RELATED TO THIS AGREEMENT WILL NOT EXCEED ONE HUNDRED U.S.
DOLLARS (US$100).
14. Indemnity. You will defend, indemnify and hold harmless NVIDIA and its affiliates, and their respective
employees, contractors, agents, officers and directors, from and against any and all third party claims,
damages, obligations, losses, liabilities, costs or debt, fines, restitutions and expenses (including but not
limited to attorney’s fees and costs incident to establishing the right of indemnification) arising out of use of
the Software and Derivatives outside of the scope of this Agreement or in breach of the terms of this
Agreement.
15. General.
15.1 Governing Law and Jurisdiction. This Agreement will be governed in all respects by the laws of the
United States and the laws of the State of Delaware, without regard to conflict of laws principles or
the United Nations Convention on Contracts for the International Sale of Goods. The state and
federal courts residing in Santa Clara County, California will have exclusive jurisdiction over any
dispute or claim arising out of or related to this Agreement, and the parties irrevocably consent to
personal jurisdiction and venue in those courts; except that either party may apply for injunctive
remedies or an equivalent type of urgent legal relief in any jurisdiction.
15.2 Independent Contractors. The parties are independent contractors, and this Agreement does not
create a joint venture, partnership, agency or other form of business association between the
parties. Neither party will have the power to bind the other party or incur any obligation on its
behalf without the other party’s prior written consent. Nothing in this Agreement prevents either
party from participating in similar arrangements with third parties.
15.3 No Assignment. NVIDIA may assign, delegate or transfer its rights or obligations under this
Agreement by any means or operation of law. You may not, without NVIDIA’s prior written consent,
assign, delegate or transfer any of your rights or obligations under this Agreement by any means or
operation of law, and any attempt to do so is null and void.
15.4 No Waiver. No failure or delay by a party to enforce any term or obligation of this Agreement will
operate as a waiver by that party, or prevent the enforcement of such term or obligation later.
15.5 Trade Compliance. You agree to comply with all applicable export, import, trade and economic
sanctions laws and regulations, as amended, including without limitation U.S. Export Administration
Regulations and Office of Foreign Assets Control regulations. You confirm (a) your understanding
that export or reexport of certain NVIDIA products or technologies may require a license or other
approval from appropriate authorities and (b) that you will not export or reexport any products or
technology, directly or indirectly, without first obtaining any required license or other approval from
appropriate authorities, (i) to any countries that are subject to any U.S. or local export restrictions
(currently including, but not necessarily limited to, Belarus, Cuba, Iran, North Korea, Russia, Syria,
the Region of Crimea, Donetsk People’s Republic Region and Luhansk People’s Republic Region); (ii)
to any end-user who you know or have reason to know will utilize them in the design, development
or production of nuclear, chemical or biological weapons, missiles, rocket systems, unmanned air
vehicles capable of a maximum range of at least 300 kilometers, regardless of payload, or intended
for military end-use, or any weapons of mass destruction; (iii) to any end-user who has been
prohibited from participating in the U.S. or local export transactions by any governing authority; or
(iv) to any known military or military-intelligence end-user or for any known military or military-
intelligence end-use in accordance with U.S. trade compliance laws and regulations.
15.6 Government Rights. The Software, documentation and technology (“Protected Items”) are
“Commercial products” as this term is defined at 48 C.F.R. 2.101, consisting of “commercial
computer software” and “commercial computer software documentation” as such terms are used
in, respectively, 48 C.F.R. 12.212 and 48 C.F.R. 227.7202 & 252.227-7014(a)(1). Before any Protected
Items are supplied to the U.S. Government, you will (i) inform the U.S. Government in writing that
the Protected Items are and must be treated as commercial computer software and commercial
computer software documentation developed at private expense; (ii) inform the U.S. Government
that the Protected Items are provided subject to the terms of the Agreement; and (iii) mark the
Protected Items as commercial computer software and commercial computer software
documentation developed at private expense. In no event will you permit the U.S. Government to
acquire rights in Protected Items beyond those specified in 48 C.F.R. 52.227-19(b)(1)-(2) or 252.227-
7013(c) except as expressly approved by NVIDIA in writing.
15.7 Notices. Please direct your legal notices or other correspondence to legalnotices@nvidia.com with a
copy mailed to NVIDIA Corporation, 2788 San Tomas Expressway, Santa Clara, California 95051,
United States of America, Attention: Legal Department. If NVIDIA needs to contact you about the
Software, you consent to receive the notices by email and agree that such notices will satisfy any
legal communication requirements.
15.8 Severability. If a court of competent jurisdiction rules that a provision of this Agreement is
unenforceable, that provision will be deemed modified to the extent necessary to make it
enforceable and the remainder of this Agreement will continue in full force and effect.
15.9 Amendment. Any amendment to this Agreement must be in writing and signed by authorized
representatives of both parties.
15.10 Entire Agreement. Regarding the subject matter of this Agreement, the parties agree that (a) this
Agreement constitutes the entire and exclusive agreement between the parties and supersedes all
prior and contemporaneous communications and (b) any additional or different terms or conditions,
whether contained in purchase orders, order acknowledgments, invoices or otherwise, will not be
binding and are null and void.
(v. February 25, 2025)
NVIDIA Confidential

118
deps/cloudxr/simple/README.md vendored Normal file
View File

@@ -0,0 +1,118 @@
# CloudXR.js Simple Example
A minimal WebGL example demonstrating WebXR streaming from a CloudXR server to a web browser. This example shows how to integrate WebXR with CloudXR to stream immersive VR/AR content.
> **Note:** This example is for learning purposes, not production use.
## Quick Start
### Prerequisites
- Node.js (v20 or higher)
### Installation
1. **Navigate to the example folder**
```bash
cd simple
```
2. **Install Dependencies**
```bash
# For this early access release, please run the following to install SDK from the given tarball. This step will not be needed when SDK is publicly accessible.
npm install ../nvidia-cloudxr-6.0.0-beta.tgz
npm install
```
3. **Build the Application**
```bash
npm run build
```
4. **Start Development Server**
```bash
npm run dev-server
```
5. **Open in Browser**
- Navigate to `http://localhost:8080` (or the port shown in terminal)
- For desktop browsers, IWER (Immersive Web Emulator Runtime) will automatically load to emulate a Meta Quest 3 headset
### Basic Usage
1. **Configure Connection**
- Enter CloudXR server IP address (default: localhost)
- Set port (default: 49100)
- Select AR or VR mode
2. **Adjust Settings (Optional)**
- Per-eye resolution (must be multiples of 16)
- Target frame rate (72, 90, or 120 FPS)
- Streaming bitrate
- XR reference space and camera offsets
3. **Start Streaming**
- Click "CONNECT"
- Grant XR permissions when prompted
**Requirements:**
- CloudXR server running and accessible
- WebXR-compatible device (VR/AR headset) or desktop browser (IWER loads automatically for emulation)
## Architecture
### CloudXRClient Class
The main application class (`CloudXRClient` in `main.ts`) handles:
**Initialization:**
- UI element management and localStorage persistence
- Browser capability checks (WebXR, WebGL2, WebRTC)
- Event listener setup
**Connection Flow:**
1. **WebGL Setup** - Creates high-performance WebGL2 context
2. **WebXR Session** - Enters immersive VR/AR mode
3. **Reference Space** - Configures coordinate system for tracking
4. **CloudXR Session** - Establishes streaming connection to server
5. **Render Loop** - Sends tracking data, receives video, renders frames
**Key Components:**
- **WebXR Session** - Hardware access (headset, controllers)
- **WebGL Context** - Video rendering
- **CloudXR Session** - Streaming management (WebRTC-based)
- **XRWebGLLayer** - Bridge between WebXR and WebGL
## Project Structure
```
simple/
├── src/
│ └── main.ts # Main application sample
├── index.html # UI and form elements sample
├── package.json # Dependencies and scripts
├── webpack.common.js # Webpack base configuration sample
├── webpack.dev.js # Development configuration sample
├── webpack.prod.js # Production configuration sample
└── tsconfig.json # TypeScript configuration sample
```
## Code Overview
The `main.ts` file contains well-commented code explaining each step:
1. **Browser Checks** - Validates WebXR, WebGL2, and WebRTC support
2. **Connection Setup** - Reads form inputs and validates configuration
3. **WebGL Initialization** - Creates optimized rendering context
4. **WebXR Session** - Enters immersive mode with requested features
5. **CloudXR Setup** - Configures streaming session with event handlers
6. **Render Loop** - Runs 72-120 times per second:
- Sends tracking data to server
- Receives video frame
- Renders to display
Each method includes inline comments explaining the purpose and key concepts.
## License
See the [LICENSE](LICENSE) file for details.

BIN
deps/cloudxr/simple/favicon.ico vendored Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 14 KiB

593
deps/cloudxr/simple/index.html vendored Normal file
View File

@@ -0,0 +1,593 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=no, viewport-fit=cover">
<meta name="robots" content="noindex, nofollow">
<meta name="description" content="NVIDIA CloudXR.js Sample Client for VR/AR streaming">
<title>NVIDIA CloudXR.js Sample Client</title>
<link rel="icon" type="image/x-icon" href="favicon.ico">
<style>
:root {
--primary-green: #76b900;
--border-color: #e0e0e0;
--text-main: #111;
--background-main: #fff;
--input-radius: 0px;
--input-border: 2px solid var(--border-color);
}
* {
box-sizing: border-box;
-webkit-tap-highlight-color: transparent;
}
html,
body {
margin: 0;
padding: 0;
height: 100%;
overflow: hidden;
font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', 'Roboto', sans-serif;
background: var(--background-main);
color: var(--text-main);
touch-action: manipulation;
-webkit-font-smoothing: antialiased;
}
@supports (padding: max(0px)) {
body {
padding: max(0px, env(safe-area-inset-top)) max(0px, env(safe-area-inset-right)) max(0px, env(safe-area-inset-bottom)) max(0px, env(safe-area-inset-left));
}
}
/* Top Banner */
.top-banner {
width: 100vw;
height: 8px;
background: var(--primary-green);
margin: 0;
padding: 0;
}
/* Header */
header {
background: var(--background-main);
padding: 16px 0 16px 24px;
min-height: unset;
box-shadow: none;
border-bottom: 1px solid var(--border-color);
}
h1 {
font-size: 1.5rem;
font-weight: 600;
}
main {
display: flex;
height: calc(100vh - 64px);
}
aside {
width: 540px;
min-width: 400px;
padding: 32px 24px 24px;
display: flex;
flex-direction: column;
border-right: 1px solid var(--border-color);
overflow-y: auto;
}
.settings-title {
font-size: 1.5rem;
font-weight: 700;
margin-bottom: 24px;
}
/* Inputs and Selectors */
.ui-input,
select {
width: 100%;
min-width: 300px;
padding: 16px 12px;
font-size: 1rem;
border: var(--input-border);
border-radius: var(--input-radius);
background: var(--background-main);
min-height: 48px;
}
.ui-input {
margin-bottom: 16px;
}
.ui-input:focus,
select:focus {
outline: none;
border-color: var(--primary-green);
box-shadow: 0 0 0 3px rgba(118, 185, 0, 0.2);
}
.ui-input::placeholder {
color: #999;
}
.settings-table {
width: 100%;
margin-bottom: 24px;
border-collapse: collapse;
}
.settings-table td {
padding: 12px 0;
vertical-align: middle;
}
.settings-table label {
font-weight: 600;
display: inline-block;
min-width: 140px;
font-size: 1rem;
}
.settings-table select {
width: 200px;
padding: 12px 8px;
cursor: pointer;
}
button {
background: var(--primary-green);
color: #fff;
border: none;
border-radius: var(--input-radius);
padding: 16px 24px;
font-size: 1.1rem;
font-weight: 700;
cursor: pointer;
transition: all 0.3s ease;
min-height: 56px;
text-transform: uppercase;
letter-spacing: 0.5px;
}
button:hover {
background: #8ac800;
}
button:disabled {
background: #666;
cursor: not-allowed;
transform: none;
box-shadow: none;
}
.start-button {
margin-top: 16px;
width: auto;
font-size: 1rem;
padding: 12px 20px;
color: #000;
}
section {
flex: 1;
padding: 48px 32px 32px;
background: #f8f8f8;
overflow-y: auto;
}
.debug-title {
font-weight: 700;
margin-bottom: 16px;
font-size: 1.3rem;
}
.debug-list {
margin: 0;
padding: 0;
list-style: none;
color: #666;
}
.debug-list li {
padding: 8px 0;
border-bottom: 1px solid var(--border-color);
font-size: 0.95rem;
}
.debug-list li:last-child {
border-bottom: none;
}
@media (max-width: 768px) {
main {
flex-direction: column;
height: auto;
min-height: calc(100vh - 64px);
}
html,
body {
overflow: auto;
}
aside {
width: 100%;
min-width: auto;
border-right: none;
border-bottom: 1px solid #404040;
}
section {
padding: 24px 16px;
}
header {
font-size: 1.1rem;
padding: 12px 16px;
}
.settings-table select {
width: 100%;
max-width: 200px;
}
}
@media (prefers-contrast: high) {
body,
aside,
.ui-input,
select {
background: var(--background-main);
color: #000;
}
aside {
border-color: #000;
}
section {
background: #f0f0f0;
}
.ui-input,
select {
border-color: #000;
}
}
@media (prefers-reduced-motion: reduce) {
* {
animation-duration: 0.01ms !important;
animation-iteration-count: 1 !important;
transition-duration: 0.01ms !important;
}
}
.ui-input:focus-visible,
select:focus-visible,
button:focus-visible {
outline: 3px solid var(--primary-green);
outline-offset: 2px;
}
.input-label {
display: block;
font-size: 0.92rem;
color: #444;
margin-bottom: 5px;
margin-top: 10px;
font-weight: 500;
}
.input-label:first-child {
margin-top: 0;
}
/* Console Logs Section */
.console-logs-section {
margin-top: 24px;
}
.console-logs-textarea {
height: 300px;
font-family: 'Courier New', monospace;
resize: vertical;
overflow-y: auto;
}
.console-logs-buttons {
margin-top: 8px;
}
.console-logs-button {
background: #666;
font-size: 0.9rem;
padding: 8px 16px;
margin-right: 8px;
}
.console-logs-button:last-child {
margin-right: 0;
}
/* Configuration Section */
.config-section {
margin-top: 24px;
}
.config-input {
margin-bottom: 8px;
}
.config-text {
font-size: 0.85rem;
color: #666;
margin-bottom: 16px;
}
/* Exit Button */
.exit-button {
position: fixed;
top: 20px;
right: 20px;
background: #ff3300cc;
color: white;
border: none;
border-radius: 5px;
padding: 10px 15px;
font-weight: bold;
font-size: 16px;
cursor: pointer;
transition: all 0.3s ease;
display: none;
z-index: 10000;
box-shadow: 0 2px 8px rgba(0, 0, 0, 0.3);
}
.exit-button:hover {
background: #ff3300;
transform: scale(1.05);
box-shadow: 0 4px 8px rgba(0, 0, 0, 0.2);
}
/* Status Message Box */
.status-message-box {
display: none;
background: #ffebee;
border: 2px solid #f44336;
border-radius: 4px;
padding: 12px 16px;
margin: 16px 0;
color: #c62828;
font-size: 0.9rem;
font-weight: 500;
line-height: 1.4;
}
.status-message-box.show {
display: block;
}
/* Success state - green */
.status-message-box.success {
background: #e8f5e9;
border-color: var(--primary-green);
color: #2e7d32;
}
/* Info state - light blue/green */
.status-message-box.info {
background: #e3f2fd;
border-color: #2196f3;
color: #1565c0;
}
/* Error state - red (default) */
.status-message-box.error {
background: #ffebee;
border-color: #f44336;
color: #c62828;
}
.status-message-box .status-icon {
display: inline-block;
margin-right: 8px;
font-weight: bold;
}
/* Certificate Acceptance Link */
.cert-acceptance-link {
background: #e3f2fd;
border: 2px solid #2196f3;
border-radius: 4px;
padding: 12px 16px;
margin: 8px 0 16px 0;
font-size: 0.9rem;
line-height: 1.4;
}
.cert-acceptance-link .cert-icon {
display: inline-block;
margin-right: 8px;
}
.cert-acceptance-link a {
color: #1565c0;
text-decoration: none;
font-weight: 600;
border-bottom: 1px solid #1565c0;
}
.cert-acceptance-link a:hover {
color: #0d47a1;
border-bottom-color: #0d47a1;
}
</style>
</head>
<body>
<div class="top-banner"></div>
<header>
<h1>NVIDIA CloudXR.js Sample Client</h1>
</header>
<main>
<aside>
<button id="startButton" class="start-button" type="button" aria-label="Connect" disabled>CONNECT</button>
<div id="statusMessageBox" class="status-message-box" role="alert" aria-live="polite">
<span class="status-icon">⚠</span>
<span id="statusMessageText"></span>
</div>
<h2 class="settings-title">Settings</h2>
<form id="settingsForm">
<label for="serverType" class="input-label">Select Server Backend</label>
<select id="serverType" class="ui-input">
<option value="manual">Manual Input IP:Port</option>
<option value="nvcf" disabled>NVCF</option>
</select>
<div id="manualFields">
<label for="serverIpInput" class="input-label">Server IP</label>
<input id="serverIpInput" class="ui-input" type="text" placeholder="Server IP" spellcheck="false">
<label for="portInput" class="input-label">Port</label>
<input id="portInput" class="ui-input" type="number" placeholder="Port (default: 49100)"
spellcheck="false" min="1" max="65535">
<div id="certAcceptanceLink" class="cert-acceptance-link" style="display: none;">
<span class="cert-icon">🔒</span>
<a id="certLink" href="#" target="_blank" rel="noopener noreferrer"></a>
</div>
</div>
<label for="app" class="input-label">Application</label>
<select id="app" name="app" class="ui-input" aria-label="Select application">
<option value="generic">Generic Client</option>
<!-- TODO: Add other applications here -->
</select>
<label for="immersive" class="input-label">Immersive Mode</label>
<select id="immersive" name="immersive" class="ui-input" aria-label="Select immersive mode">
<option value="ar" selected>AR Immersive</option>
<option value="vr">VR Immersive</option>
</select>
</form>
</aside>
<section>
<h3 class="debug-title">Debug Settings</h3>
<div class="config-section">
<label class="input-label">Proxy Configuration</label>
<label for="proxyUrl" class="input-label">Proxy URL</label>
<input id="proxyUrl" class="ui-input config-input" type="text" placeholder="" spellcheck="false" autocapitalize="off">
<div class="config-text">
<span id="proxyDefaultText"></span>
</div>
</div>
<div class="config-section">
<label class="input-label">Frame Rate Configuration</label>
<label for="deviceFrameRate" class="input-label">Device Frame Rate</label>
<select id="deviceFrameRate" class="ui-input config-input">
<option value="72">72 FPS</option>
<option value="90" selected>90 FPS</option>
<option value="120">120 FPS</option>
</select>
<div class="config-text">
Select the target device frame rate for the XR session
</div>
</div>
<div class="config-section">
<label class="input-label">Maximum Streaming Bitrate</label>
<label for="maxStreamingBitrateMbps" class="input-label">Megabits per second</label>
<select id="maxStreamingBitrateMbps" class="ui-input config-input">
<option value="80">80 Mbps</option>
<option value="100">100 Mbps</option>
<option value="120">120 Mbps</option>
<option value="150" selected>150 Mbps</option>
<option value="180">180 Mbps</option>
<option value="200">200 Mbps</option>
</select>
<div class="config-text">
Select the maximum streaming bitrate (in Megabits per second) for the XR session
</div>
</div>
<div class="config-section">
<label class="input-label">Stream Resolution Configuration</label>
<label for="perEyeWidth" class="input-label">Per-Eye Width (default: 2048)</label>
<input id="perEyeWidth" class="ui-input config-input" type="number" placeholder="Per-eye width in pixels"
spellcheck="false" value="2048">
<label for="perEyeHeight" class="input-label">Per-Eye Height (default: 1792)</label>
<input id="perEyeHeight" class="ui-input config-input" type="number" placeholder="Per-eye height in pixels"
spellcheck="false" value="1792">
<div class="config-text">
Configure the per-eye resolution. Width and height must be multiples of 16.
</div>
</div>
<div class="config-section">
<label for="referenceSpace" class="input-label">Preferred Reference Space</label>
<select id="referenceSpace" class="ui-input config-input">
<option value="auto" selected>Auto (local-floor preferred)</option>
<option value="local-floor">local-floor</option>
<option value="local">local</option>
<option value="unbounded">unbounded</option>
<option value="viewer">viewer</option>
</select>
<div class="config-text">
Select the preferred reference space for XR tracking. "Auto" uses the original fallback logic (local-floor → local → viewer). Other options will attempt to use the specified space only.
</div>
<label class="input-label">XR Reference Space Offset</label>
<label for="xrOffsetX" class="input-label">X Offset (centimeters): Horizontal</label>
<input id="xrOffsetX" class="ui-input config-input" type="number" placeholder="X offset in centimeters"
spellcheck="false" value="0" step="1" min="-1000" max="1000">
<label for="xrOffsetY" class="input-label">Y Offset (centimeters): Vertical</label>
<input id="xrOffsetY" class="ui-input config-input" type="number" placeholder="Y offset in centimeters"
spellcheck="false" value="0" step="1" min="-1000" max="1000">
<label for="xrOffsetZ" class="input-label">Z Offset (centimeters): Depth (facing outward)</label>
<input id="xrOffsetZ" class="ui-input config-input" type="number" placeholder="Z offset in centimeters"
spellcheck="false" value="0" step="1" min="-1000" max="1000">
<div class="config-text">
Configure the XR reference space offset in centimeters. These values will be applied to the reference space transform to adjust the origin position of the XR experience.
</div>
</div>
<div class="config-section">
<label for="enablePoseSmoothing" class="input-label">Pose Smoothing</label>
<select id="enablePoseSmoothing" class="ui-input config-input">
<option value="true" selected>Enabled</option>
<option value="false">Disabled</option>
</select>
<div class="config-text">
Enable or disable secondary smoothing on predicted positions to reduce jitter. This only affects position, not orientation.
</div>
</div>
<div class="config-section">
<label for="posePredictionFactor" class="input-label">Pose Prediction Factor: <span id="posePredictionFactorValue">1.0</span></label>
<input id="posePredictionFactor" class="ui-input config-input" type="range"
min="0" max="1" step="0.1" value="1.0"
style="padding: 8px 12px;">
<div class="config-text">
Scale the pose prediction horizon (0.0 = no prediction, 1.0 = full prediction). This multiplier affects both position and orientation prediction strength.
</div>
</div>
</section>
</main>
<!-- WebGL Canvas for CloudXR rendering -->
<canvas id="webglCanvas"></canvas>
<!-- Floating Exit Button. This is a fallback option when immersive mode has no way to exit. -->
<button id="exitButton" class="exit-button" type="button" aria-label="Exit">Exit</button>
</body>
</html>

31
deps/cloudxr/simple/package.json vendored Normal file
View File

@@ -0,0 +1,31 @@
{
"name": "cloudxr-simple-example",
"version": "6.0.0-beta",
"private": true,
"description": "CloudXR.js WebGL example application",
"author": "NVIDIA Corporation",
"license": "SEE LICENSE IN LICENSE",
"keywords": [],
"scripts": {
"build": "webpack --config ./webpack.prod.js",
"dev": "webpack --config ./webpack.dev.js",
"dev-server": "webpack serve --config ./webpack.dev.js --no-open",
"dev-server:https": "HTTPS=true webpack serve --config ./webpack.dev.js --no-open",
"clean": "rimraf build"
},
"dependencies": {
"@nvidia/cloudxr": "dev"
},
"devDependencies": {
"@types/node": "^22.13.14",
"@types/webxr": "^0.5.22",
"copy-webpack-plugin": "^13.0.0",
"gl-matrix": "^3.4.3",
"html-webpack-plugin": "^5.6.3",
"rimraf": "^5.0.5",
"ts-loader": "^9.5.1",
"typescript": "^5.8.2",
"webpack-dev-server": "^5.2.1",
"webpack-cli": "^6.0.1"
}
}

515
deps/cloudxr/simple/src/main.ts vendored Normal file
View File

@@ -0,0 +1,515 @@
/**
* CloudXR.js Simple Example - WebXR Streaming Application
*
* CloudXR streams XR content from a powerful server to lightweight clients (think Netflix for VR/AR).
* Server does the heavy rendering, client displays video and sends back tracking data.
*
* Key Flow:
* 1. constructor() - Initialize UI and check browser support
* 2. connectToCloudXR() - Connect to server (called on CONNECT button click)
* 3. initializeWebGL() - Set up graphics rendering
* 4. createXRSession() - Enter VR/AR mode
* 5. createCloudXRSession() - Configure CloudXR streaming
* 6. onXRFrame() - Render loop: send tracking, receive & display video frames
*/
import { checkCapabilities } from '@helpers/BrowserCapabilities';
import { loadIWERIfNeeded } from '@helpers/LoadIWER';
import { overridePressureObserver } from '@helpers/overridePressureObserver';
import {
enableLocalStorage,
getConnectionConfig,
setupCertificateAcceptanceLink,
} from '@helpers/utils';
import { getOrCreateCanvas, logOrThrow } from '@helpers/WebGlUtils';
import * as CloudXR from '@nvidia/cloudxr';
// Override PressureObserver early to catch errors from buggy browser implementations
overridePressureObserver();
/**
* CloudXR Client - Main Application Class
*
* Architecture: WebXR (hardware access) + WebGL (rendering) + CloudXR (streaming)
*/
class CloudXRClient {
// UI Elements - Form inputs and display elements
private startButton: HTMLButtonElement;
private exitButton: HTMLButtonElement;
private serverIpInput: HTMLInputElement;
private portInput: HTMLInputElement;
private proxyUrlInput: HTMLInputElement;
private immersiveSelect: HTMLSelectElement;
private deviceFrameRateSelect: HTMLSelectElement;
private maxStreamingBitrateMbpsSelect: HTMLSelectElement;
private proxyDefaultText: HTMLElement;
private statusMessageBox: HTMLElement;
private statusMessageText: HTMLElement;
private perEyeWidthInput: HTMLInputElement;
private perEyeHeightInput: HTMLInputElement;
private referenceSpaceSelect: HTMLSelectElement;
private xrOffsetXInput: HTMLInputElement;
private xrOffsetYInput: HTMLInputElement;
private xrOffsetZInput: HTMLInputElement;
private certAcceptanceLink: HTMLElement;
private certLink: HTMLAnchorElement;
private enablePoseSmoothingSelect: HTMLSelectElement;
private posePredictionFactorInput: HTMLInputElement;
private posePredictionFactorValue: HTMLElement;
// Core Session Components
private xrSession: XRSession | null = null; // WebXR session (hardware access)
private cloudxrSession: CloudXR.Session | null = null; // CloudXR session (streaming)
private gl: WebGL2RenderingContext | null = null; // WebGL context (rendering)
private baseLayer: XRWebGLLayer | null = null; // Bridge between WebXR and WebGL
private deviceFrameRate: number = 0; // Target frame rate for XR session
private hasSetTargetFrameRate: boolean = false; // Track if we've set target frame rate
/**
* Initialize UI, enable localStorage, and check WebXR support
*/
constructor() {
// Get references to all UI elements
this.startButton = document.getElementById('startButton') as HTMLButtonElement;
this.exitButton = document.getElementById('exitButton') as HTMLButtonElement;
this.serverIpInput = document.getElementById('serverIpInput') as HTMLInputElement;
this.portInput = document.getElementById('portInput') as HTMLInputElement;
this.proxyUrlInput = document.getElementById('proxyUrl') as HTMLInputElement;
this.immersiveSelect = document.getElementById('immersive') as HTMLSelectElement;
this.deviceFrameRateSelect = document.getElementById('deviceFrameRate') as HTMLSelectElement;
this.maxStreamingBitrateMbpsSelect = document.getElementById(
'maxStreamingBitrateMbps'
) as HTMLSelectElement;
this.proxyDefaultText = document.getElementById('proxyDefaultText') as HTMLElement;
this.statusMessageBox = document.getElementById('statusMessageBox') as HTMLElement;
this.statusMessageText = document.getElementById('statusMessageText') as HTMLElement;
this.perEyeWidthInput = document.getElementById('perEyeWidth') as HTMLInputElement;
this.perEyeHeightInput = document.getElementById('perEyeHeight') as HTMLInputElement;
this.referenceSpaceSelect = document.getElementById('referenceSpace') as HTMLSelectElement;
this.xrOffsetXInput = document.getElementById('xrOffsetX') as HTMLInputElement;
this.xrOffsetYInput = document.getElementById('xrOffsetY') as HTMLInputElement;
this.xrOffsetZInput = document.getElementById('xrOffsetZ') as HTMLInputElement;
this.certAcceptanceLink = document.getElementById('certAcceptanceLink') as HTMLElement;
this.certLink = document.getElementById('certLink') as HTMLAnchorElement;
this.enablePoseSmoothingSelect = document.getElementById(
'enablePoseSmoothing'
) as HTMLSelectElement;
this.posePredictionFactorInput = document.getElementById(
'posePredictionFactor'
) as HTMLInputElement;
this.posePredictionFactorValue = document.getElementById(
'posePredictionFactorValue'
) as HTMLElement;
// Enable localStorage to persist user settings
enableLocalStorage(this.serverIpInput, 'serverIp');
enableLocalStorage(this.portInput, 'port');
enableLocalStorage(this.proxyUrlInput, 'proxyUrl');
enableLocalStorage(this.immersiveSelect, 'immersiveMode');
enableLocalStorage(this.deviceFrameRateSelect, 'deviceFrameRate');
enableLocalStorage(this.maxStreamingBitrateMbpsSelect, 'maxStreamingBitrateMbps');
enableLocalStorage(this.perEyeWidthInput, 'perEyeWidth');
enableLocalStorage(this.perEyeHeightInput, 'perEyeHeight');
enableLocalStorage(this.referenceSpaceSelect, 'referenceSpace');
enableLocalStorage(this.xrOffsetXInput, 'xrOffsetX');
enableLocalStorage(this.xrOffsetYInput, 'xrOffsetY');
enableLocalStorage(this.xrOffsetZInput, 'xrOffsetZ');
enableLocalStorage(this.enablePoseSmoothingSelect, 'enablePoseSmoothing');
enableLocalStorage(this.posePredictionFactorInput, 'posePredictionFactor');
// Update slider value display when it changes
this.posePredictionFactorInput.addEventListener('input', () => {
this.posePredictionFactorValue.textContent = this.posePredictionFactorInput.value;
});
// Set initial display value
this.posePredictionFactorValue.textContent = this.posePredictionFactorInput.value;
// Configure proxy information and port placeholder based on protocol
if (window.location.protocol === 'https:') {
this.proxyDefaultText.textContent =
'Optional: Leave empty for direct WSS connection, or provide URL for proxy routing (e.g., https://proxy.example.com/)';
this.portInput.placeholder = 'Port (default: 48322, or 443 if proxy URL set)';
} else {
this.proxyDefaultText.textContent = 'Not needed for HTTP - uses direct WS connection';
this.portInput.placeholder = 'Port (default: 49100)';
}
this.startButton.addEventListener('click', () => this.connectToCloudXR());
this.exitButton.addEventListener('click', () => this.xrSession?.end());
// Set up certificate acceptance link
setupCertificateAcceptanceLink(
this.serverIpInput,
this.portInput,
this.proxyUrlInput,
this.certAcceptanceLink,
this.certLink
);
this.checkWebXRSupport();
}
/**
* Check browser support: WebXR, WebGL2, WebRTC, and video frame callbacks
* Also loads Immersive Web Emulator if needed (for desktop development)
*/
private async checkWebXRSupport(): Promise<void> {
const { supportsImmersive, iwerLoaded } = await loadIWERIfNeeded();
if (!supportsImmersive) {
this.showStatus('Immersive mode not supported', 'error');
this.startButton.disabled = true;
return;
}
this.startButton.disabled = true;
this.startButton.innerHTML = 'CONNECT (checking capabilities)';
const result = await checkCapabilities();
if (!result.success) {
this.showStatus(
'Browser does not meet required capabilities:\n' + result.failures.join('\n'),
'error'
);
this.startButton.innerHTML = 'CONNECT (capabilities check failed)';
return;
}
if (result.warnings.length > 0) {
this.showStatus('Performance notice:\n' + result.warnings.join('\n'), 'info');
} else if (iwerLoaded) {
// Include IWER status in the final success message
this.showStatus(
'CloudXR.js SDK is supported. Ready to connect!\nUsing IWER (Immersive Web Emulator Runtime) - Emulating Meta Quest 3.',
'info'
);
} else {
this.showStatus('CloudXR.js SDK is supported. Ready to connect!', 'success');
}
this.startButton.disabled = false;
this.startButton.innerHTML = 'CONNECT';
}
private showStatus(message: string, type: 'success' | 'error' | 'info'): void {
this.statusMessageText.textContent = message;
this.statusMessageBox.className = `status-message-box show ${type}`;
console[type === 'error' ? 'error' : 'info'](message);
}
/**
* Main connection flow - orchestrates WebGL, WebXR, and CloudXR setup
* Steps: Read config → Initialize WebGL → Create XR session → Connect to CloudXR server
*/
private async connectToCloudXR(): Promise<void> {
// Read configuration from UI form
const serverIp = this.serverIpInput.value.trim() || 'localhost';
// Determine default port based on connection type and proxy usage
const useSecureConnection = window.location.protocol === 'https:';
const portValue = parseInt(this.portInput.value, 10);
const proxyUrl = this.proxyUrlInput.value;
const hasProxy = proxyUrl.trim().length > 0;
let defaultPort = 49100; // HTTP default (direct CloudXR Runtime connection)
if (useSecureConnection) {
defaultPort = hasProxy ? 443 : 48322; // HTTPS with proxy → 443, HTTPS without → 48322
}
const port = portValue || defaultPort;
const perEyeWidth = parseInt(this.perEyeWidthInput.value, 10) || 2048;
const perEyeHeight = parseInt(this.perEyeHeightInput.value, 10) || 1792;
const deviceFrameRate = parseInt(this.deviceFrameRateSelect.value, 10);
const maxStreamingBitrateKbps = parseInt(this.maxStreamingBitrateMbpsSelect.value, 10) * 1000;
const immersiveMode = this.immersiveSelect.value as 'ar' | 'vr';
const referenceSpaceType = this.referenceSpaceSelect.value as XRReferenceSpaceType;
const xrOffsetX = (parseFloat(this.xrOffsetXInput.value) || 0) / 100; // cm to meters
const xrOffsetY = (parseFloat(this.xrOffsetYInput.value) || 0) / 100;
const xrOffsetZ = (parseFloat(this.xrOffsetZInput.value) || 0) / 100;
try {
this.startButton.disabled = true;
this.startButton.innerHTML = 'CONNECT (connecting)';
this.showStatus(`Connecting to Server ${serverIp}:${port}...`, 'info');
// Initialize WebGL, WebXR session, and reference space
await this.initializeWebGL();
await this.createXRSession(immersiveMode, deviceFrameRate);
let referenceSpace = await this.getReferenceSpace(referenceSpaceType);
if (xrOffsetX !== 0 || xrOffsetY !== 0 || xrOffsetZ !== 0) {
const offsetTransform = new XRRigidTransform(
{ x: xrOffsetX, y: xrOffsetY, z: xrOffsetZ },
{ x: 0, y: 0, z: 0, w: 1 }
);
referenceSpace = referenceSpace.getOffsetReferenceSpace(offsetTransform);
}
// Create CloudXR session and connect to server
await this.createCloudXRSession(
serverIp,
port,
proxyUrl,
perEyeWidth,
perEyeHeight,
maxStreamingBitrateKbps,
referenceSpace
);
this.cloudxrSession!.connect();
this.startButton.innerHTML = 'CONNECT (waiting for streaming)';
this.showStatus(`Connected to Server ${serverIp}:${port}...`, 'info');
} catch (error) {
this.showStatus(`Connection failed: ${error}`, 'error');
this.startButton.disabled = false;
this.startButton.innerHTML = 'CONNECT';
if (this.xrSession) {
try {
await this.xrSession.end();
} catch (endError) {
console.error('Error ending XR session during cleanup:', endError);
this.clearSessionReferences();
}
} else {
this.clearSessionReferences();
}
}
}
/**
* Initialize WebGL2 context for rendering (high-performance, XR-compatible)
*/
private async initializeWebGL(): Promise<void> {
const webglCanvas = getOrCreateCanvas('webglCanvas');
const gl = webglCanvas.getContext('webgl2', {
alpha: true,
depth: true,
stencil: false,
desynchronized: false,
antialias: false, // No antialiasing (video already rendered)
failIfMajorPerformanceCaveat: true,
powerPreference: 'high-performance', // Use discrete GPU if available
premultipliedAlpha: false,
preserveDrawingBuffer: false,
}) as WebGL2RenderingContext;
if (!gl) throw new Error('Failed to create WebGL2 context');
await gl.makeXRCompatible(); // Required before using with XRWebGLLayer
this.gl = gl;
logOrThrow('Creating WebGL context', this.gl);
}
/**
* Create WebXR session, XRWebGLLayer, and start render loop
*/
private async createXRSession(
immersiveMode: 'ar' | 'vr',
deviceFrameRate: number
): Promise<void> {
const mode = immersiveMode === 'vr' ? 'immersive-vr' : 'immersive-ar';
const options = {
requiredFeatures: ['local-floor'],
optionalFeatures: ['hand-tracking', 'high-fixed-foveation-level'],
};
// Try requested mode, fallback to alternative if unsupported
try {
this.xrSession = await navigator.xr!.requestSession(mode, options);
} catch (error) {
console.warn(`${mode} session failed, trying alternative:`, error);
const altMode = immersiveMode === 'vr' ? 'immersive-ar' : 'immersive-vr';
this.xrSession = await navigator.xr!.requestSession(altMode, options);
}
// Create XRWebGLLayer - provides framebuffer for CloudXR to render into
this.baseLayer = new XRWebGLLayer(this.xrSession, this.gl!, {
alpha: true,
antialias: false,
depth: true,
framebufferScaleFactor: 1.2,
ignoreDepthValues: false,
stencil: false,
});
// Store frame rate for later use in render loop
this.deviceFrameRate = deviceFrameRate;
this.hasSetTargetFrameRate = false;
this.xrSession.updateRenderState({ baseLayer: this.baseLayer });
this.xrSession.addEventListener('end', () => this.handleXRSessionEnd());
this.xrSession.requestAnimationFrame(this.onXRFrame.bind(this));
}
/**
* Get XR reference space with fallbacks
* Reference space types: 'local-floor' (room-scale), 'local' (seated), 'viewer' (head-locked)
*/
private async getReferenceSpace(
referenceSpaceType: XRReferenceSpaceType
): Promise<XRReferenceSpace> {
try {
return await this.xrSession!.requestReferenceSpace(referenceSpaceType);
} catch (error) {
console.warn(`'${referenceSpaceType}' not supported, trying fallbacks...`);
try {
return await this.xrSession!.requestReferenceSpace('local-floor');
} catch {
try {
return await this.xrSession!.requestReferenceSpace('local');
} catch {
return await this.xrSession!.requestReferenceSpace('viewer');
}
}
}
}
/**
* Configure CloudXR session and set up event handlers
* Establishes WebRTC connection, receives video stream, sends tracking data
*/
private async createCloudXRSession(
serverIp: string,
port: number,
proxyUrl: string,
perEyeWidth: number,
perEyeHeight: number,
maxStreamingBitrateKbps: number,
referenceSpace: XRReferenceSpace
): Promise<void> {
const connectionConfig = getConnectionConfig(serverIp, port, proxyUrl);
const sessionOptions: CloudXR.SessionOptions = {
serverAddress: connectionConfig.serverIP,
serverPort: connectionConfig.port,
useSecureConnection: connectionConfig.useSecureConnection,
gl: this.gl!,
perEyeWidth, // Stream resolution: width = perEyeWidth * 2 (side-by-side)
perEyeHeight, // Stream resolution: height = perEyeHeight * 9/4 (includes metadata)
referenceSpace,
deviceFrameRate: parseInt(this.deviceFrameRateSelect.value, 10),
maxStreamingBitrateKbps,
enablePoseSmoothing: this.enablePoseSmoothingSelect.value === 'true',
posePredictionFactor: parseFloat(this.posePredictionFactorInput.value),
telemetry: {
enabled: true,
appInfo: { version: '6.0.0-beta', product: 'CloudXR.js WebGL Example' },
},
};
const delegates: CloudXR.SessionDelegates = {
onStreamStarted: () => {
console.log('CloudXR stream started');
this.startButton.innerHTML = 'CONNECT (streaming)';
this.exitButton.style.display = 'block';
this.showStatus('Streaming started!', 'success');
},
onStreamStopped: (error?: Error) => {
if (error) {
console.error('Stream stopped with error:', error);
this.showStatus(`Stream stopped: ${error}`, 'error');
} else {
console.log('Stream stopped normally');
this.showStatus('Stream stopped', 'info');
}
if (this.xrSession) {
this.xrSession
.end()
.catch(endError => console.error('Error ending XR session:', endError))
.finally(() => (this.exitButton.style.display = 'none'));
} else {
this.exitButton.style.display = 'none';
}
this.startButton.disabled = false;
this.startButton.innerHTML = 'CONNECT';
},
onWebGLStateChangeBegin: () => console.debug('WebGL state change begin'),
onWebGLStateChangeEnd: () => console.debug('WebGL state change end'),
onServerMessageReceived: (messageData: Uint8Array) => {
const messageString = new TextDecoder().decode(messageData);
console.debug('Server message:', messageString);
},
};
try {
this.cloudxrSession = CloudXR.createSession(sessionOptions, delegates);
} catch (error) {
console.error('Failed to create CloudXR session:', error);
throw error;
}
}
/**
* Main render loop - runs every frame (72-120 FPS)
* Sends tracking data to server, receives video frame, renders to display
*/
private async onXRFrame(timestamp: DOMHighResTimeStamp, frame: XRFrame): Promise<void> {
this.xrSession!.requestAnimationFrame(this.onXRFrame.bind(this));
// Set target frame rate on first frame only
if (!this.hasSetTargetFrameRate && 'updateTargetFrameRate' in this.xrSession!) {
this.hasSetTargetFrameRate = true;
try {
await this.xrSession!.updateTargetFrameRate(this.deviceFrameRate);
console.debug(
`Target frame rate set to ${this.deviceFrameRate}, current: ${this.xrSession!.frameRate}`
);
} catch (error) {
console.error('Failed to set target frame rate:', error);
}
}
if (!this.cloudxrSession) {
console.debug('Skipping frame, CloudXR session not created yet');
return;
}
if (this.cloudxrSession.state !== CloudXR.SessionState.Connected) {
console.debug('Skipping frame, session not ready');
return;
}
try {
// Send tracking (head/hand positions) → Receive video → Render
this.cloudxrSession.sendTrackingStateToServer(timestamp, frame);
this.gl!.bindFramebuffer(this.gl!.FRAMEBUFFER, this.baseLayer!.framebuffer);
this.cloudxrSession.render(timestamp, frame, this.baseLayer!);
} catch (error) {
console.error('Error in render frame:', error);
}
}
/**
* Cleanup when XR session ends (user exits, removes headset, or error occurs)
*/
private handleXRSessionEnd(): void {
try {
if (this.cloudxrSession) {
this.cloudxrSession.disconnect();
this.cloudxrSession = null;
}
this.clearSessionReferences();
this.startButton.disabled = false;
this.startButton.innerHTML = 'CONNECT';
this.exitButton.style.display = 'none';
} catch (error) {
this.showStatus(`Disconnect error: ${error}`, 'error');
}
}
private clearSessionReferences(): void {
this.baseLayer = null;
this.xrSession = null;
this.gl = null;
this.hasSetTargetFrameRate = false;
}
}
// Application entry point - wait for DOM to load, then initialize client
document.addEventListener('DOMContentLoaded', () => {
new CloudXRClient();
});

15
deps/cloudxr/simple/tsconfig.json vendored Normal file
View File

@@ -0,0 +1,15 @@
{
"compilerOptions": {
"target": "ES6",
"moduleResolution": "bundler",
"strict": true,
"lib": ["DOM", "DOM.Iterable", "ESNext", "WebWorker"],
"skipLibCheck": true,
"types": ["webxr"],
"baseUrl": ".",
"paths": {
"@helpers/*": ["../helpers/*"]
}
},
"include": ["**/*.ts", "types/**/*.d.ts"]
}

41
deps/cloudxr/simple/webpack.common.js vendored Normal file
View File

@@ -0,0 +1,41 @@
const path = require('path');
const HtmlWebpackPlugin = require('html-webpack-plugin');
module.exports = {
entry: './src/main.ts',
// Module rules define how different file types are processed
module: {
rules: [
{
test: /\.tsx?$/,
use: 'ts-loader',
exclude: /node_modules/,
},
],
},
// Resolve configuration for module resolution
resolve: {
extensions: ['.ts', '.js'],
alias: {
// @helpers can be used instead of relative paths to the helpers directory
'@helpers': path.resolve(__dirname, '../helpers')
}
},
// Output configuration for bundled files
output: {
filename: 'bundle.js',
path: path.resolve(__dirname, './build'),
},
// Webpack plugins that extend webpack's functionality
plugins: [
// Generates HTML file and automatically injects bundled JavaScript
new HtmlWebpackPlugin({
template: './index.html',
favicon: './favicon.ico'
})
]
};

42
deps/cloudxr/simple/webpack.dev.js vendored Normal file
View File

@@ -0,0 +1,42 @@
const { merge } = require('webpack-merge')
const common = require('./webpack.common.js')
const path = require('path')
// Check if HTTPS mode is enabled via environment variable
const useHttps = process.env.HTTPS === 'true'
module.exports = merge(common, {
mode: 'development',
devtool: 'eval-source-map',
devServer: {
allowedHosts: 'all',
hot: true,
open: false,
port: 8080,
// Enable HTTPS with self-signed certificate when HTTPS=true
...(useHttps && { server: 'https' }),
static: {
directory: path.join(__dirname, './build'),
watch: true,
},
watchFiles: {
paths: ['src/**/*', '../../build/**/*'],
options: {
usePolling: false,
ignored: /node_modules/,
},
},
client: {
progress: true,
overlay: {
errors: true,
warnings: false,
},
},
devMiddleware: {
writeToDisk: true,
},
},
})
console.log(module.exports);

8
deps/cloudxr/simple/webpack.prod.js vendored Normal file
View File

@@ -0,0 +1,8 @@
const { merge } = require('webpack-merge');
const common = require('./webpack.common.js');
module.exports = merge(common, {
mode: 'production'
});
console.log(module.exports);