Skip to main content
Methods for spatial queries — raycasting, screen-to-world projection, and canvas snapshots.

​
raycast(origin, direction, maxDist?)

Cast a ray and find the first intersection with a geom. 
const hit = api.raycast(
  new THREE.Vector3(0, 0, 1),    // Origin
  new THREE.Vector3(0, 0, -1),   // Direction (downward)
  10                               // Max distance
);

if (hit) {
  console.log("Hit body", hit.bodyId, "at distance", hit.distance);
  console.log("Point:", hit.point);
}
​
origin
THREE.Vector3
required
Ray origin in world coordinates.
​
direction
THREE.Vector3
required
Ray direction (will be normalized).
​
maxDist
number
default:"100"
Maximum ray distance.
Returns: RayHit | null 
interface RayHit {
  point: THREE.Vector3;
  bodyId: number;
  geomId: number;
  distance: number;
}

​
project2DTo3D(x, y, cameraPos, lookAt)

Project a 2D screen point to a 3D world point via raycasting. 
// Project center of screen
const hit = api.project2DTo3D(
  0.5, 0.5,                                    // Normalized coords (0-1)
  new THREE.Vector3(2, -1.5, 2.5),             // Camera position
  new THREE.Vector3(0, 0, 0),                  // Camera look-at
);

if (hit) {
  console.log("World point:", hit.point);
  console.log("Body:", hit.bodyId);
}
​
x
number
required
Normalized X coordinate (0 = left, 1 = right).
​
y
number
required
Normalized Y coordinate (0 = top, 1 = bottom).
​
cameraPos
THREE.Vector3
required
Camera world position.
​
lookAt
THREE.Vector3
required
Camera look-at point.
Returns: { point: THREE.Vector3, bodyId: number, geomId: number } | null

​
getCanvasSnapshot(width?, height?, mimeType?)

Capture the current canvas as a base64-encoded image. 
const base64 = api.getCanvasSnapshot();
// "data:image/png;base64,iVBORw0KGgo..."

// Custom size and format
const jpeg = api.getCanvasSnapshot(640, 480, "image/jpeg");
​
width
number
Output width. Defaults to canvas width.
​
height
number
Output height. Defaults to canvas height.
​
mimeType
string
default:"image/png"
Image format: 'image/png', 'image/jpeg', 'image/webp'.
Returns: string — base64 data URL.

​
Camera Animation

Camera state and animation are available via the standalone useCameraAnimation() hook: 
import { useCameraAnimation } from "mujoco-react";

const { getCameraState, moveCameraTo } = useCameraAnimation();

// Get current camera state
const cam = getCameraState();
console.log("Camera at:", cam.position);
console.log("Looking at:", cam.target);

// Animate camera
await moveCameraTo(
  new THREE.Vector3(3, 0, 2),
  new THREE.Vector3(0, 0, 0.5),
  1000
);
See useCameraAnimation for full documentation.

​
Example: Vision Pipeline

import { useCameraAnimation } from "mujoco-react";

async function detectObjects(api: MujocoSimAPI) {
  const { getCameraState } = useCameraAnimation();

  // 1. Capture scene
  const image = api.getCanvasSnapshot(1024, 1024);

  // 2. Send to vision model
  const detections = await callVisionAPI(image);

  // 3. Project 2D detections to 3D
  const cam = getCameraState();
  const worldPoints = detections.map(det => {
    const hit = api.project2DTo3D(
      det.x / 1000, det.y / 1000,
      cam.position, cam.target
    );
    return hit?.point;
  });

  return worldPoints.filter(Boolean);
}