> ## Documentation Index
> Fetch the complete documentation index at: https://dadd.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Spatial Queries

> Raycasting, 2D-to-3D projection, and canvas capture

Methods for spatial queries — raycasting, screen-to-world projection, and canvas snapshots.

## raycast(origin, direction, maxDist?)

Cast a ray and find the first intersection with a geom.

```tsx theme={null}
const hit = api.raycast(
  new THREE.Vector3(0, 0, 1),    // Origin
  new THREE.Vector3(0, 0, -1),   // Direction (downward)
  10                               // Max distance
);

if (hit) {
  console.log("Hit body", hit.bodyId, "at distance", hit.distance);
  console.log("Point:", hit.point);
}
```

<ParamField body="origin" type="THREE.Vector3" required>
  Ray origin in world coordinates.
</ParamField>

<ParamField body="direction" type="THREE.Vector3" required>
  Ray direction (will be normalized).
</ParamField>

<ParamField body="maxDist" type="number" default="100">
  Maximum ray distance.
</ParamField>

**Returns:** `RayHit | null`

```tsx theme={null}
interface RayHit {
  point: THREE.Vector3;
  bodyId: number;
  geomId: number;
  distance: number;
}
```

## project2DTo3D(x, y, cameraPos, lookAt)

Project a 2D screen point to a 3D world point via raycasting.

```tsx theme={null}
// Project center of screen
const hit = api.project2DTo3D(
  0.5, 0.5,                                    // Normalized coords (0-1)
  new THREE.Vector3(2, -1.5, 2.5),             // Camera position
  new THREE.Vector3(0, 0, 0),                  // Camera look-at
);

if (hit) {
  console.log("World point:", hit.point);
  console.log("Body:", hit.bodyId);
}
```

<ParamField body="x" type="number" required>
  Normalized X coordinate (0 = left, 1 = right).
</ParamField>

<ParamField body="y" type="number" required>
  Normalized Y coordinate (0 = top, 1 = bottom).
</ParamField>

<ParamField body="cameraPos" type="THREE.Vector3" required>
  Camera world position.
</ParamField>

<ParamField body="lookAt" type="THREE.Vector3" required>
  Camera look-at point.
</ParamField>

**Returns:** `{ point: THREE.Vector3, bodyId: number, geomId: number } | null`

## projectImagePointTo3D(options)

Project detector/image coordinates back into the rendered MuJoCo scene. Prefer
this method for perception models because it makes the coordinate convention and
camera source explicit.

```tsx theme={null}
const hit = api.projectImagePointTo3D({
  cameraName: "overhead",
  x: 512,
  y: 418,
  coordinateSpace: "normalized-1000",
  width: 640,
  height: 480,
  hiddenGeomGroups: [3],
});

if (hit) {
  console.log(hit.bodyId, hit.geomId, hit.point.toArray());
}
```

<ParamField body="x" type="number" required>
  X coordinate in the selected coordinate space.
</ParamField>

<ParamField body="y" type="number" required>
  Y coordinate in the selected coordinate space.
</ParamField>

<ParamField body="coordinateSpace" type="'normalized' | 'normalized-1000' | 'pixel' | 'ndc'" default="normalized">
  Coordinate convention for the detector result.
</ParamField>

<ParamField body="cameraName | siteName | bodyName | position/lookAt" type="CameraFrameCaptureOptions">
  Camera source. Uses the same fields as `captureCameraFrame`.
</ParamField>

**Returns:** `ImagePointProjectionResult | null`

See [Perception Projection](/guides/perception-projection) for examples.

## getCanvasSnapshot(width?, height?, mimeType?)

Capture the current canvas as a base64-encoded image.

```tsx theme={null}
const base64 = api.getCanvasSnapshot();
// "data:image/png;base64,iVBORw0KGgo..."

// Custom size and format
const jpeg = api.getCanvasSnapshot(640, 480, "image/jpeg");
```

<ParamField body="width" type="number">
  Output width. Defaults to canvas width.
</ParamField>

<ParamField body="height" type="number">
  Output height. Defaults to canvas height.
</ParamField>

<ParamField body="mimeType" type="string" default="image/png">
  Image format: `'image/png'`, `'image/jpeg'`, `'image/webp'`.
</ParamField>

**Returns:** `string` — base64 data URL.

## Camera Animation

Camera state and animation are available via the standalone `useCameraAnimation()` hook:

```tsx theme={null}
import { useCameraAnimation } from "mujoco-react";

const { getCameraState, moveCameraTo } = useCameraAnimation();

// Get current camera state
const cam = getCameraState();
console.log("Camera at:", cam.position);
console.log("Looking at:", cam.target);

// Animate camera
await moveCameraTo(
  new THREE.Vector3(3, 0, 2),
  new THREE.Vector3(0, 0, 0.5),
  1000
);
```

See [useCameraAnimation](/hooks/use-camera-animation) for full documentation.

## Example: Vision Pipeline

```tsx theme={null}
import { useCameraAnimation } from "mujoco-react";

async function detectObjects(api: MujocoSimAPI) {
  const { getCameraState } = useCameraAnimation();

  // 1. Capture scene
  const image = api.getCanvasSnapshot(1024, 1024);

  // 2. Send to vision model
  const detections = await callVisionAPI(image);

  // 3. Project 2D detections to 3D
  const cam = getCameraState();
  const worldPoints = detections.map(det => {
    const hit = api.project2DTo3D(
      det.x / 1000, det.y / 1000,
      cam.position, cam.target
    );
    return hit?.point;
  });

  return worldPoints.filter(Boolean);
}
```