Session API

The CloudXR.js Session API provides comprehensive management of streaming sessions with the CloudXR Runtime. This guide covers the core session lifecycle, configuration, and key operations.

Overview

CloudXR.js is a JavaScript Client SDK that enables WebXR streaming from the CloudXR Runtime. It requires compatible server applications such as Isaac Lab, or any OpenXR-compatible application running with the CloudXR Runtime.

Session Lifecycle

The CloudXR session follows a well-defined lifecycle with distinct states and transitions:

Initialized → Connecting → Connected → Disconnecting → Disconnected
Copy

Session States

• SessionState.Initialized: Session created but not yet connected
• SessionState.Connecting: Attempting to establish connection with CloudXR Runtime
• SessionState.Connected: Active streaming session with video streaming active
• SessionState.Disconnecting: Session is in the process of disconnecting from the CloudXR Runtime
• SessionState.Disconnected: Session has been disconnected and is ready to be reconnected
• SessionState.Error: Session encountered a critical error and cannot continue

Key Operations

Creating a Session

Full API reference: createSession

import { createSession } from '@nvidia/cloudxr';

const session = createSession({
  serverAddress: 'your-server',
  serverPort: 49100,
  useSecureConnection: false,
  gl: webglContext,
  perEyeWidth: 2048,
  perEyeHeight: 1792,
  referenceSpace: xrReferenceSpace,
  // Additional configuration options
});
Copy

Note: The perEyeWidth and perEyeHeight parameters specify the resolution for each eye. The actual stream resolution will be calculated automatically:

• Stream width = perEyeWidth * 2
• Stream height = perEyeHeight * 9 / 4

See SessionOptions for all available configuration options.

Session Management

Full API reference: Session

Connect

// Connect throws an error if connection cannot be initiated
try {
  session.connect();
  console.log('Session connection initiated');
  // Connection state changes will be reported via delegates
} catch (error) {
  console.error('Failed to initiate connection:', error.message);
}
Copy

Disconnect

// Disconnect from the server
session.disconnect();
Copy

The session will transition to SessionState.Disconnecting state immediately. Then to SessionState.Disconnected state when the disconnection is complete.

Note: CloudXR.js does not currently support pause/resume functionality. To temporarily stop streaming, disconnect the session and reconnect when needed.

Send Tracking State

// Send viewer pose and input tracking to server
const success = session.sendTrackingStateToServer(timestamp, xrFrame);
if (!success) {
  console.warn('Failed to send tracking state');
}
Copy

Render Frame

// Render a frame from CloudXR Runtime
session.render(timestamp, xrFrame, xrWebGLLayer);
Copy

Send Server Message

// Send custom message to server
try {
  session.sendServerMessage(messageObject);
  console.log('Message sent successfully');
} catch (error) {
  console.error('Failed to send server message:', error.message);
}
Copy

Event Handling

CloudXR uses a delegate-based event system. Pass delegates when creating the session:

const sessionDelegates = {
  onStreamStarted: () => {
    console.log('Streaming started');
  },
  
  onStreamStopped: (error) => {
    if (error) {
      console.log('Streaming stopped due to error:', error);
    } else {
      console.log('Streaming stopped normally');
    }
  },
  
  onWebGLStateChangeBegin: () => {
    // Called before session changes WebGL state
    console.log('WebGL state change beginning');
  },
  
  onWebGLStateChangeEnd: () => {
    // Called after session changes WebGL state
    console.log('WebGL state change complete');
  }
};

const session = createSession(sessionOptions, sessionDelegates);
Copy

See SessionDelegates for more details.

Monitoring Session State

// Check current session state
console.log('Current state:', session.state);
Copy

WebXR Integration

Basic WebXR Setup

// Request WebXR session
const xrSession = await navigator.xr.requestSession('immersive-vr', {
  requiredFeatures: ['local-floor']
});

// Get WebGL context and reference space
const gl = xrSession.renderState.baseLayer.context;
const referenceSpace = await xrSession.requestReferenceSpace('local-floor');

// Create CloudXR session
const sessionOptions = {
  serverAddress: 'your-server',
  serverPort: 49100,
  useSecureConnection: false,
  gl: gl,
  perEyeWidth: 2048,
  perEyeHeight: 1792,
  referenceSpace: referenceSpace,
  deviceFrameRate: 90,
  maxStreamingBitrateKbps: 150000
};

const sessionDelegates = {
  onStreamStarted: () => console.log('CloudXR streaming started'),
  onStreamStopped: (error) => console.log('CloudXR streaming stopped', error)
};

const cloudxrSession = createSession(sessionOptions, sessionDelegates);

// Connect to CloudXR Runtime
try {
  cloudxrSession.connect();
  console.log('CloudXR connection initiated');
} catch (error) {
  console.error('Failed to initiate CloudXR connection:', error.message);
}

// Begin rendering loop
function renderFrame(time, frame) {
  // Send tracking state to server
  cloudxrSession.sendTrackingStateToServer(time, frame);
  
  // Render CloudXR frame
  cloudxrSession.render(time, frame, xrSession.renderState.baseLayer);
  
  // Continue rendering
  xrSession.requestAnimationFrame(renderFrame);
}

xrSession.requestAnimationFrame(renderFrame);
Copy

Controller and Hand Tracking

CloudXR automatically handles controller input and hand tracking through the WebXR session. The tracking data is sent to the server via sendTrackingStateToServer():

// Controllers and hand tracking are automatically handled
// when you call sendTrackingStateToServer with the XR frame
function renderFrame(time, frame) {
  // This automatically includes controller and hand tracking data
  cloudxrSession.sendTrackingStateToServer(time, frame);
  
  // Render the frame
  cloudxrSession.render(time, frame, xrWebGLLayer);
}
Copy

The server will receive all input data including:

• Controller button presses and triggers
• Controller pose and orientation
• Hand tracking data (if supported by the device)
• Head tracking and viewer pose

Error Handling

Error Handling in Delegates

const sessionDelegates = {
  onStreamStarted: () => {
    console.log('Streaming started successfully');
  },
  
  onStreamStopped: (error) => {
    if (error) {
      console.error('Streaming stopped due to error:', error);
      // Handle specific error types
      if (error.message.includes('connection')) {
        console.error('Connection error - check network and server');
      }
    } else {
      console.log('Streaming stopped normally');
    }
  }
};
Copy

Error Handling

// Connect throws an error if connection cannot be initiated
try {
  session.connect();
} catch (error) {
  console.error('Failed to initiate connection:', error.message);
}

// Send tracking state returns false if not connected
if (!session.sendTrackingStateToServer(timestamp, frame)) {
  console.warn('Cannot send tracking state - session not connected');
}

// Send server message throws an error if not connected or invalid message
try {
  session.sendServerMessage(message);
} catch (error) {
  console.error('Failed to send server message:', error.message);
}
Copy

Performance Monitoring

Enable WebXR Native Performance Monitoring in Meta Quest

For comprehensive WebXR performance monitoring and debugging tools, refer to the Meta Horizon WebXR Performance Tools documentation.

Other Tools

• WebXR Performance Guide: MDN WebXR Performance Guide - Comprehensive performance optimization strategies
• Immersive Web Emulator: immersiveweb.dev - Browser extension for testing WebXR performance on various devices

Server Requirements

CloudXR.js requires a compatible server running:

• CloudXR Runtime - The streaming server component
• OpenXR-compatible application - Such as:
  • Isaac Lab simulations
  • Custom OpenXR applications
  • Unity/Unreal Engine applications with OpenXR support

The server must be accessible via WebSocket connection and properly configured for CloudXR streaming.

See our Getting Started Guide for more details.