Spatial Queries

Raycasting, 2D-to-3D projection, and canvas capture

Methods for spatial queries — raycasting, screen-to-world projection, and canvas snapshots.

raycast(origin, direction, maxDist?)

Cast a ray and find the first intersection with a geom.

const hit = api.raycast(
  new THREE.Vector3(0, 0, 1),    // Origin
  new THREE.Vector3(0, 0, -1),   // Direction (downward)
  10                               // Max distance
);

if (hit) {
  console.log("Hit body", hit.bodyId, "at distance", hit.distance);
  console.log("Point:", hit.point);
}

origin · THREE.Vector3 · required

Ray origin in world coordinates.

direction · THREE.Vector3 · required

Ray direction (will be normalized).

maxDist · number · default: "100"

Maximum ray distance.

Returns: RayHit | null

interface RayHit {
  point: THREE.Vector3;
  bodyId: number;
  geomId: number;
  distance: number;
}

project2DTo3D(x, y, cameraPos, lookAt)

Project a 2D screen point to a 3D world point via raycasting.

// Project center of screen
const hit = api.project2DTo3D(
  0.5, 0.5,                                    // Normalized coords (0-1)
  new THREE.Vector3(2, -1.5, 2.5),             // Camera position
  new THREE.Vector3(0, 0, 0),                  // Camera look-at
);

if (hit) {
  console.log("World point:", hit.point);
  console.log("Body:", hit.bodyId);
}

x · number · required

Normalized X coordinate (0 = left, 1 = right).

y · number · required

Normalized Y coordinate (0 = top, 1 = bottom).

cameraPos · THREE.Vector3 · required

Camera world position.

lookAt · THREE.Vector3 · required

Camera look-at point.

Returns: { point: THREE.Vector3, bodyId: number, geomId: number } | null

projectImagePointTo3D(options)

Project detector/image coordinates back into the rendered MuJoCo scene. Prefer this method for perception models because it makes the coordinate convention and camera source explicit.

const hit = api.projectImagePointTo3D({
  cameraName: "overhead",
  x: 512,
  y: 418,
  coordinateSpace: "normalized-1000",
  width: 640,
  height: 480,
  hiddenGeomGroups: [3],
});

if (hit) {
  console.log(hit.bodyId, hit.geomId, hit.point.toArray());
}

x · number · required

X coordinate in the selected coordinate space.

y · number · required

Y coordinate in the selected coordinate space.

coordinateSpace · 'normalized' | 'normalized-1000' | 'pixel' | 'ndc' · default: "normalized"

Coordinate convention for the detector result.

cameraName | siteName | bodyName | position/lookAt · CameraFrameCaptureOptions

Camera source. Uses the same fields as captureCameraFrame.

Returns: ImagePointProjectionResult | null

See Perception Projection for examples.

getCanvasSnapshot(width?, height?, mimeType?)

Capture the current canvas as a base64-encoded image.

const base64 = api.getCanvasSnapshot();
// "data:image/png;base64,iVBORw0KGgo..."

// Custom size and format
const jpeg = api.getCanvasSnapshot(640, 480, "image/jpeg");

width · number

Output width. Defaults to canvas width.

height · number

Output height. Defaults to canvas height.

mimeType · string · default: "image/png"

Image format: 'image/png', 'image/jpeg', 'image/webp'.

Returns: string — base64 data URL.

Camera Animation

Camera state and animation are available via the standalone useCameraAnimation() hook:

import { useCameraAnimation } from "mujoco-react";

const { getCameraState, moveCameraTo } = useCameraAnimation();

// Get current camera state
const cam = getCameraState();
console.log("Camera at:", cam.position);
console.log("Looking at:", cam.target);

// Animate camera
await moveCameraTo(
  new THREE.Vector3(3, 0, 2),
  new THREE.Vector3(0, 0, 0.5),
  1000
);

See useCameraAnimation for full documentation.

Example: Vision Pipeline

import { useCameraAnimation } from "mujoco-react";

async function detectObjects(api: MujocoSimAPI) {
  const { getCameraState } = useCameraAnimation();

  // 1. Capture scene
  const image = api.getCanvasSnapshot(1024, 1024);

  // 2. Send to vision model
  const detections = await callVisionAPI(image);

  // 3. Project 2D detections to 3D
  const cam = getCameraState();
  const worldPoints = detections.map(det => {
    const hit = api.project2DTo3D(
      det.x / 1000, det.y / 1000,
      cam.position, cam.target
    );
    return hit?.point;
  });

  return worldPoints.filter(Boolean);
}