mindbot/deps/cloudxr/react/README.md

React Three Fiber Example

This is a comprehensive CloudXR.js React Three Fiber example application that demonstrates how to integrate CloudXR streaming with modern React development patterns. This example showcases the power of combining CloudXR.js with React Three Fiber, React Three XR, and React Three UIKit to create immersive XR experiences with rich 3D user interfaces.

NOTE: This example is not meant to be used for production.

Overview

This example showcases the integration of CloudXR.js with the React Three Fiber ecosystem, providing:

• React Three Fiber Integration: Seamless integration with Three.js through React components
• React Three XR: WebXR session management with React hooks and state management
• React Three UIKit: Rich 3D user interface components for VR/AR experiences
• CloudXR Streaming: Real-time streaming of XR content from a CloudXR server
• Modern React Patterns: Hooks, context, and component-based architecture
• Dual UI System: 2D HTML interface for configuration and 3D VR interface for interaction

Quick Start

Prerequisites

• Node.js (v20 or higher)
• A CloudXR server running and accessible
• A WebXR-compatible device (VR headset, AR device)

Installation

1. Navigate to the example folder

   cd react
   

2. Install Dependencies

   # For this early access release, please run the following to install SDK from the given tarball. This step will not be needed when SDK is publicly accessible.
   npm install ../nvidia-cloudxr-6.0.0-beta.tgz
   
   npm install
   

3. Build the Application

   npm run build
   

4. Start Development Server

   npm run dev-server
   

5. Open in Browser

   • Navigate to http://localhost:8080 (or the port shown in terminal)
   • For desktop browsers, IWER (Immersive Web Emulator Runtime) will automatically load to emulate a Meta Quest 3 headset

Basic Usage

1. Configure Connection

   • Enter your CloudXR server IP address
   • Set the port (default: 49100)
   • Select AR or VR immersive mode

2. Adjust Settings (Optional)

   • Configure per-eye resolution (perEyeWidth and perEyeHeight, must be multiples of 16)
   • Set target frame rate and bitrate
   • Adjust XR reference space

3. Start Streaming

   • Click "CONNECT" to initiate the XR session
   • Grant XR permissions when prompted

NOTE: In order to connect to an actual server and start streaming, you need:

• A CloudXR server running and accessible
• A WebXR-compatible device (VR/AR headset) or desktop browser (IWER loads automatically for emulation)

Technical Architecture

Core Components

App.tsx

Main React application component managing:

• XR store configuration and session state
• CloudXR component integration
• 2D UI management and event handling
• Error handling and capability checking
• React Three Fiber Canvas setup

CloudXRComponent.tsx

Handles the core CloudXR streaming functionality:

• CloudXR session lifecycle management
• WebXR session event handling
• WebGL state management and render target preservation
• Frame-by-frame rendering loop with pose tracking
• Integration with Three.js WebXRManager

CloudXR2DUI.tsx

Manages the 2D HTML interface:

• Form field management and localStorage persistence
• Proxy configuration based on protocol
• Event listener management and cleanup
• Error handling and user feedback
• Configuration validation and updates

CloudXRUI.tsx (3D UI)

Renders the in-VR user interface:

• React Three UIKit components for 3D UI
• Interactive control buttons with hover effects
• Server information and status display
• Event handler integration

Development

Project Structure

react/
├── src/
│   ├── App.tsx              # Main React application
│   ├── CloudXRComponent.tsx # CloudXR streaming component
│   ├── CloudXR2DUI.tsx      # 2D UI management class
│   ├── CloudXRUI.tsx        # 3D VR UI component
│   ├── index.tsx            # React app entry point
│   └── index.html           # HTML template
├── public/
│   ├── play-circle.svg                   # Play button icon (Heroicons)
│   ├── stop-circle.svg                   # Stop button icon (Heroicons)
│   ├── arrow-uturn-left.svg              # Reset button icon (Heroicons)
│   └── arrow-left-start-on-rectangle.svg # Disconnect button icon (Heroicons)
├── package.json             # Dependencies and scripts
├── webpack.common.js        # Webpack configuration
├── webpack.dev.js           # Development webpack config
├── webpack.prod.js          # Production webpack config
└── tsconfig.json            # TypeScript configuration

React Three Fiber Integration

XR Store Configuration

The application uses React Three XR's store for XR session management:

const store = createXRStore({
  foveation: 0,
  emulate: { syntheticEnvironment: false },
});

Canvas Setup

React Three Fiber Canvas with WebXR integration:

<Canvas events={noEvents} gl={{ preserveDrawingBuffer: true }}>
  <XR store={store}>
    <CloudXRComponent config={config} />
    <CloudXR3DUI onAction1={handleAction1} />
  </XR>
</Canvas>

Custom Render Loop

The CloudXR component uses useFrame for custom rendering:

useFrame((state, delta) => {
  if (webXRManager.isPresenting && session) {
    // CloudXR rendering logic
    cxrSession.sendTrackingStateToServer(timestamp, xrFrame);
    cxrSession.render(timestamp, xrFrame, layer);
  }
}, -1000);

3D User Interface

React Three UIKit Components

The 3D UI uses React Three UIKit for modern VR/AR interfaces:

• Container: Layout and positioning components
• Text: 3D text rendering with custom fonts
• Button: Interactive buttons with hover effects
• Image: Texture-based image display
• Root: Main UI container with pixel-perfect rendering

UI Positioning

3D UI elements are positioned in world space:

<group position={[1.8, -0.5, -1.3]} rotation={[0, -0.3, 0]}>
  <Root pixelSize={0.001} width={1920} height={1440}>
    {/* UI components */}
  </Root>
</group>

WebGL State Tracking

This example uses WebGL state tracking to prevent rendering conflicts between React Three Fiber and CloudXR. Both libraries render to the same WebGL context, but CloudXR's rendering operations modify WebGL state (framebuffers, textures, buffers, VAOs, shaders, blend modes, etc.) which can interfere with React Three Fiber's expectations. The example wraps the WebGL context with bindGL() from @helpers/WebGLStateBinding, then uses CloudXR's onWebGLStateChangeBegin and onWebGLStateChangeEnd callbacks to automatically save and restore state around CloudXR's rendering. This ensures React Three Fiber always finds the WebGL context in the expected state after each CloudXR render operation.

See examples/helpers/WebGLStateBinding.ts, WebGLState.ts, and WebGLStateApply.ts for implementation details. Comprehensive tests are available in tests/unit/WebGLState.test.ts and tests/playwright/WebGLTests/src/WebGLStateBindingTests.ts.

License

See the LICENSE file for details.

Third-Party Assets

Icons used in the immersive UI are from Heroicons by Tailwind Labs, licensed under the MIT License. See HEROICONS_LICENSE for details.