useCameraFrameCapture

Capture offscreen frames from a chosen camera pose

Capture a still image from a specific camera pose without moving the user’s interactive viewport. This is useful for MuJoCo-mounted dataset cameras, policy rollout reports, thumbnails, and visual regression checks.

Unlike useFrameCapture, this renders the scene to an offscreen render target. It does not require preserveDrawingBuffer.

For the distinction between viewer cameras, MuJoCo cameras, explicit capture poses, and virtual debug camera overlays, see Cameras and Captures.

Signature

useCameraFrameCapture(options?: {
  camera?: THREE.Camera
  cameraName?: string
  siteName?: string
  bodyName?: string
  position?: THREE.Vector3 | readonly [number, number, number]
  lookAt?: THREE.Vector3 | readonly [number, number, number]
  quaternion?: THREE.Quaternion | readonly [number, number, number, number]
  up?: THREE.Vector3 | readonly [number, number, number]
  positionOffset?: THREE.Vector3 | readonly [number, number, number]
  quaternionOffset?: THREE.Quaternion | readonly [number, number, number, number]
  width?: number
  height?: number
  type?: string
  quality?: number
  fov?: number
  near?: number
  far?: number
  projectionMatrix?: THREE.Matrix4 | readonly number[]
  mujocoCameraCompatibility?: boolean | {
    useResolution?: boolean
    useIntrinsics?: boolean
    useClipping?: boolean
    preserveAspect?: boolean
    preferResolution?: boolean
  }
  hiddenGeomGroups?: readonly number[]
  visibleGeomGroups?: readonly number[]
  hiddenGeomNames?: readonly string[]
  background?: THREE.ColorRepresentation
  backgroundAlpha?: number
  visualOverrides?: CameraFrameVisualOverrides
  renderIsolation?: boolean | CameraFrameRenderIsolationOptions
  flipX?: boolean
}): {
  status: "idle" | "capturing" | "captured" | "error"
  error: Error | null
  isCapturing: boolean
  capture: (options?: CameraFrameCaptureOptions) => Promise<CameraFrameCaptureResult>
  captureBlob: (options?: CameraFrameCaptureOptions) => Promise<CameraFrameCaptureBlobResult>
  reset: () => void
}

CameraFrameCaptureResult and CameraFrameCaptureBlobResult include source.kind, which is one of mujoco-camera, mujoco-site, mujoco-body, custom-camera, explicit-pose, or fallback-camera. Dataset recorders should prefer the MuJoCo-mounted source kinds.

Usage

import { useCameraFrameCapture } from "mujoco-react";

function DatasetCaptureButton() {
  const capture = useCameraFrameCapture({
    width: 1024,
    height: 1024,
    siteName: "head_camera_rgb_optical_frame",
    type: "image/png",
  });

  async function saveFrame() {
    const frame = await capture.captureBlob();
    await uploadDatasetImage(frame.blob);
  }

  return (
    <button onClick={saveFrame} disabled={capture.isCapturing}>
      Capture dataset view
    </button>
  );
}

API Ref Path

MujocoSimAPI exposes the same offscreen camera capture methods:

const frame = await apiRef.current?.captureCameraFrame({
  width: 1024,
  height: 1024,
  cameraName: "head_camera",
  mujocoCameraCompatibility: true,
});
if (frame?.source.kind !== "mujoco-camera") {
  throw new Error("Expected a mounted MuJoCo camera frame.");
}

const blob = await apiRef.current?.captureCameraFrameBlob({
  width: 1024,
  height: 1024,
  siteName: "wrist_camera_rgb_optical_frame",
});

mujocoCameraCompatibility is only applied to named MuJoCo <camera> captures. It derives width/height from MJCF resolution, uses MJCF fovy, copies visual clip planes when available, and installs an asymmetric projection matrix from cam_intrinsic and cam_sensorsize when those arrays are exposed by the WASM model. Leave it off for synthetic position + lookAt probes where you are manually matching a dataset view.

Use projectionMatrix when you already have calibrated camera intrinsics. Use visualOverrides to temporarily override scene background, environment, fog, shadow maps, tone mapping, or output color space for one capture. Use renderIsolation when policy or training images should be rendered with an independent offscreen WebGLRenderer instead of inheriting viewer renderer state.

Policy Image Payloads

Use usePolicyCameraFramesFromMountedStreams or capturePolicyCameraFramesFromMountedStreams when a policy expects image keys such as observation.images.front and observation.images.wrist. The helper resolves dataset stream names to mounted MuJoCo cameras, sites, or bodies, captures each stream, and returns both raw frames and payload-ready data URLs.

const policyCameras = usePolicyCameraFramesFromMountedStreams({
  cameraKeys: ["front", "wrist"],
  aliases: {
    front: [{ cameraName: "realsense_d435i" }],
    wrist: [{ cameraName: "wrist_cam" }],
  },
  defaults: {
    width: 640,
    height: 480,
    type: "image/jpeg",
    quality: 0.82,
  },
  streamOptions: {
    front: {
      aliases: ["realsense"],
      hiddenGeomGroups: [3, 4],
      background: "#d8ddd8",
    },
    wrist: {
      quaternionOffset: [1, 0, 0, 0],
    },
  },
  requireAll: true,
});

const result = await policyCameras.capture();

await fetch("/infer", {
  method: "POST",
  body: JSON.stringify({
    state,
    images: result.images,
  }),
});

Use the non-hook helper when policy execution happens outside React:

const result = await capturePolicyCameraFramesFromMountedStreams(api, {
  cameraKeys: ["front", "wrist"],
  aliases: {
    front: [{ cameraName: "realsense_d435i" }],
    wrist: [{ cameraName: "wrist_cam" }],
  },
  requireAll: true,
});

Explicit Three camera poses are also valid policy streams. They do not move the user’s orbit camera:

const result = await capturePolicyCameraFramesFromMountedStreams(api, {
  cameraKeys: ["front", "wrist"],
  aliases: {
    wrist: [{ cameraName: "wrist_cam" }],
  },
  streamOptions: {
    front: {
      position: [1.1, -0.3, 1.3],
      lookAt: [0.45, -0.3, 0.8],
      up: [0, 0, 1],
      fov: 48.5,
    },
  },
});

Policy stream keys stay stable even when they resolve through aliases. In the example above, result.frames.wrist, result.images.wrist, and result.images["observation.images.wrist"] are populated even though the mounted MuJoCo camera is named wrist_cam.

Notes

• Use cameraName for MuJoCo <camera> elements, or siteName / bodyName for robot-mounted camera frames.
• Use positionOffset and quaternionOffset to calibrate a mounted camera pose without editing MJCF. Offsets are applied in the mounted camera/site/body local frame. Quaternion arrays use Three.js order [x, y, z, w]; convert MuJoCo [w, x, y, z] quaternions before passing them.
• Use position + lookAt only for synthetic fixed debug cameras.
• Use quaternion when replaying a recorded camera pose.
• Use camera to clone an existing THREE.Camera.
• The returned canvas is generated from an offscreen render target.
• Use hiddenGeomGroups, visibleGeomGroups, or hiddenGeomNames to match a dataset renderer without changing the interactive scene.
• Use visualOverrides and renderIsolation for canonical policy/training captures that should not inherit the interactive viewer’s visual effects.
• Use flipX to mirror captured frames horizontally when matching a policy dataset or camera convention that stores mirrored images.
• Advanced renderers can attach a function to CAMERA_FRAME_CAPTURE_RENDER_USER_DATA_KEY in scene object userData to provide custom Three pixels for capture.