mujoco-react

Composable React Three Fiber wrapper around the official @mujoco/mujoco WASM bindings. Load any MuJoCo model, step physics, render bodies, and write controllers as React components.

Demo | Docs | npm | Example Source | llms.txt

Install

npm install mujoco-react three @react-three/fiber @react-three/drei

Vite Plugin and Type-Safe Names

Use the Vite plugin to generate declaration merging for actuator, sensor, body, joint, site, geom, and keyframe names:

// vite.config.ts
import { defineConfig } from "vite";
import { mujocoReact } from "mujoco-react/vite";

export default defineConfig({
  plugins: [
    mujocoReact({
      models: {
        franka: "models/panda/scene.xml",
      },
    }),
  ],
});

The plugin writes src/mujoco-register.gen.ts during dev and build. Commit that generated file. Vite auto-loads it, so app code imports generated values from mujoco-react, not from the generated file:

// src/mujoco-register.gen.ts
// Auto-generated by mujoco-react. Do not edit.

import { registerModelResources } from "mujoco-react";

const generatedModelResources = {
  franka: {
    actuators: { joint1: "joint1", joint2: "joint2", joint3: "joint3", gripper: "gripper" },
    sensors: { force_sensor: "force_sensor", torque_sensor: "torque_sensor" },
    bodies: { link0: "link0", link1: "link1", hand: "hand" },
    joints: { joint1: "joint1", joint2: "joint2", joint3: "joint3" },
    sites: { tcp: "tcp" },
    geoms: { floor: "floor" },
    keyframes: { home: "home" },
  },
};

registerModelResources(generatedModelResources);

declare module "mujoco-react" {
  interface Register {
    models: {
      franka: {
        actuators: "joint1" | "joint2" | "joint3" | "gripper";
        sensors: "force_sensor" | "torque_sensor";
        bodies: "link0" | "link1" | "hand";
        joints: "joint1" | "joint2" | "joint3";
        sites: "tcp";
        geoms: "floor";
        keyframes: "home";
      };
    };
    actuators: "joint1" | "joint2" | "joint3" | "gripper";
    sensors: "force_sensor" | "torque_sensor";
    bodies: "link0" | "link1" | "hand";
  }
}

Once generated, hooks like useCtrl, useSensor, useBodyState, and API methods like setCtrl, applyForce, getSensorData accept the global union. For model-scoped code, use package exports such as ModelActuators.franka.gripper, ModelSites.franka.tcp, and ModelBodies.franka.hand. Generic type helpers like ModelActuators<"franka"> are still available for reusable library code. When no Register augmentation is present, names fall back to string.

Non-Vite projects can generate the same file with:

npx mujoco-react codegen franka=models/panda/scene.xml

Load a Model

import { MujocoProvider, MujocoCanvas } from "mujoco-react";
import type { SceneConfig } from "mujoco-react";

const panda: SceneConfig = {
  src: "https://raw.githubusercontent.com/google-deepmind/mujoco_menagerie/main/franka_emika_panda/",
  sceneFile: "scene.xml",
  homeJoints: [1.707, -1.754, 0.003, -2.702, 0.003, 0.951, 2.490],
};

export function App() {
  return (
    <MujocoProvider>
      <MujocoCanvas
        config={panda}
        camera={{ position: [2, -1.5, 2.5], up: [0, 0, 1], fov: 45 }}
        style={{ width: "100%", height: "100vh" }}
      />
    </MujocoProvider>
  );
}

Add Scene Tools

import { OrbitControls } from "@react-three/drei";
import { IkGizmo, ModelSites, useIkController } from "mujoco-react";

function PandaTools() {
  const ik = useIkController({ siteName: ModelSites.franka.tcp });

  return (
    <>
      <OrbitControls makeDefault />
      {ik && <IkGizmo controller={ik} />}
      <ambientLight intensity={0.7} />
      <directionalLight position={[1, 2, 5]} intensity={1.2} />
    </>
  );
}

Use it as a child of <MujocoCanvas>:

<MujocoCanvas config={panda}>
  <PandaTools />
</MujocoCanvas>

Gaussian Splat Environments

Gaussian splats are visual context; MuJoCo XML remains the source of physics, contacts, and task fixtures. Use visual-only splats when you only need rendered environment context, and add collision proxy metadata when a workflow needs simplified contact geometry.

Use VisualScenarioEffects when the same MuJoCo task should render under different camera exposure, fog/background, and deterministic material variants:

import { ScenarioLighting, VisualScenarioEffects } from "mujoco-react";

<MujocoCanvas config={sceneConfig}>
  <VisualScenarioEffects
    scenario={scenario}
    materialFilter={({ object }) => object.name.startsWith("prop_")}
  />
  <ScenarioLighting preset={scenario.lighting} />
</MujocoCanvas>;

Use the renderer-agnostic boundary from the main package. If your app stores visual scenarios as data, pass the scenario directly; the component resolves the splat asset and any paired MJCF collision proxy metadata for you. Visual-only splats are valid, and readiness tells you whether a collision proxy is required for your training/physics handoff.

import { MujocoCanvas, SplatEnvironment, useSplatSceneConfig } from "mujoco-react";

const splat = useSplatSceneConfig({ sceneConfig, scenario });

<MujocoCanvas config={splat.sceneConfig}>
  {splat.environment ? (
    <SplatEnvironment environment={splat.environment} renderer="custom">
      <MySplatRenderer src={splat.environment.splat.src} />
    </SplatEnvironment>
  ) : null}
</MujocoCanvas>;

When a splat scenario includes paired MJCF collision proxy metadata, render a generic wireframe preview from that XML with SplatCollisionProxyPreview. The component parses MJCF primitives such as planes, boxes, spheres, capsules, and mesh placeholders from any fetchable proxy XML; the tabletop example is just one possible environment.

import {
  SplatCollisionProxyPreview,
  SplatEnvironment,
  useSplatCollisionProxyGeoms,
  useSplatSceneConfig,
} from "mujoco-react";

const splat = useSplatSceneConfig({ sceneConfig, scenario });
const proxy = splat.environment?.collisionProxy;

<SplatEnvironment
  environment={splat.environment}
  collisionProxy={
    proxy ? <SplatCollisionProxyPreview collisionProxy={proxy} /> : undefined
  }
/>;

Use useSplatCollisionProxyGeoms() when your app wants to inspect or style the proxy primitives itself:

const proxyPreview = useSplatCollisionProxyGeoms({
  collisionProxy: splat.environment?.collisionProxy,
});

proxyPreview.geoms.map((geom) => geom.type);

Use splat.readiness or getSplatEnvironmentReadiness(scenario) to gate authoring and import flows. The status distinguishes disabled scenarios, missing visual assets, missing collision proxies, unsupported renderer formats, and ready environments.

Use createSplatSceneConfig() when the same scene composition needs to run outside React, such as codegen, import validation, backend handoff metadata, or tests:

import { createSplatSceneConfig } from "mujoco-react";

const splat = createSplatSceneConfig({
  sceneConfig,
  scenario,
  renderer: "spark",
});

Use createVisualScenarioExecutionContext() or useVisualScenarioExecutionContext() when recording rollouts or exporting LeRobot/HF Jobs handoff artifacts. It resolves the scenario seed, camera exposure/noise/blur/jitter, material randomization, splat source, collision proxy, and readiness into one serializable object.

import { createVisualScenarioExecutionContext } from "mujoco-react";

const visualContext = createVisualScenarioExecutionContext({
  scenario,
  renderer: "spark",
  variantId,
});

writeEpisodeManifest({
  task,
  visualExecutionContext: visualContext,
});

For MuJoCo + 3DGS composition, derive the optional collision environment from the same splat metadata and pass the resulting config to <MujocoCanvas>:

const sceneConfig = withSplatEnvironment(
  {
    src: "/models/xlerobot/",
    sceneFile: "xlerobot.xml",
  },
  kitchenScenario
);

For first-class Spark rendering, install Spark and import the optional adapter:

npm install @sparkjsdev/spark

import {
  SparkSplatEnvironment,
  useSparkSplatEnvironment,
} from "mujoco-react/spark";

function Scene() {
  const splat = useSparkSplatEnvironment({ sceneConfig, scenario });

  return (
    <MujocoCanvas config={splat.sceneConfig} gl={{ preserveDrawingBuffer: true }}>
      {splat.environment ? (
        <SparkSplatEnvironment hideGroundMeshes {...splat.props} />
      ) : null}
      <StatusBadge status={splat.lifecycle.status} error={splat.lifecycle.error} />
    </MujocoCanvas>
  );
}

SparkSplatEnvironment currently renders .spz assets. Use the renderer-agnostic SplatEnvironment for .ply/.splat metadata or when wiring a different renderer. Tune live rendering and snapshots separately with renderTuning and captureTuning:

<SparkSplatEnvironment
  {...splat.props}
  renderTuning={{ lodSplatScale: 0.75, minSortIntervalMs: 50 }}
  captureTuning={{ lodSplatScale: 1.4, lodRenderScale: 0.45, maxWarmupFrames: 6 }}
/>

Write Controllers

import { ModelActuators, useBeforePhysicsStep, useCtrl } from "mujoco-react";

function MyController() {
  const shoulder = useCtrl(ModelActuators.franka.actuator1);

  useBeforePhysicsStep(({ data }) => {
    shoulder.write(Math.sin(data.time));
  });

  return null;
}

Controllers are just React children that read sensors, write named controls, apply forces, or call the MujocoSimAPI at physics-step time.

With generated resource values, reusable controllers can be scoped to one robot without hand-typing names:

import { ModelActuators, ModelJoints, ModelSites, useCtrl, useIkController } from "mujoco-react";

function FrankaTypedControls() {
  const gripper = useCtrl(ModelActuators.franka.gripper);
  const ik = useIkController({
    siteName: ModelSites.franka.tcp,
    joints: [
      ModelJoints.franka.joint1,
      ModelJoints.franka.joint2,
      ModelJoints.franka.joint3,
      ModelJoints.franka.joint4,
      ModelJoints.franka.joint5,
      ModelJoints.franka.joint6,
      ModelJoints.franka.joint7,
    ],
    actuators: [
      ModelActuators.franka.actuator1,
      ModelActuators.franka.actuator2,
      ModelActuators.franka.actuator3,
      ModelActuators.franka.actuator4,
      ModelActuators.franka.actuator5,
      ModelActuators.franka.actuator6,
      ModelActuators.franka.actuator7,
    ],
  });

  return null;
}

Use the Sim API

import { useMujoco } from "mujoco-react";

function ResetButton() {
  const sim = useMujoco();
  if (!sim.isReady) return null;

  return <button onClick={() => sim.api.reset()}>Reset</button>;
}

Map Controls to Joints

Use named controls when a controller should write actuator commands without hard-coding data.ctrl offsets:

import {
  ModelActuators,
  defineControls,
  useBeforePhysicsStep,
  useControls,
} from "mujoco-react";

const arm = defineControls({
  shoulder: ModelActuators.so101.shoulder_pan,
  lift: ModelActuators.so101.shoulder_lift,
  elbow: ModelActuators.so101.elbow_flex,
});

function PolicyDriver({ action }: { action: number[] }) {
  const controls = useControls(arm);

  useBeforePhysicsStep(() => {
    controls.write(action);
    controls.patch({ elbow: 1.2 });
  });

  return null;
}

defineControls() preserves the alias keys for set(), patch(), get(), and read(), while write() accepts an ordered vector in the same order as the object. The Vite plugin injects the generated register module automatically, so application code imports generated names from mujoco-react, not from src/mujoco-register.gen.ts.

Use lower-level control groups when a robot's actuator order does not match a simple qpos[0..n] layout or you want actuator-name keys directly:

import { controlGroup, ModelActuators, useControlGroup } from "mujoco-react";

const gripper = controlGroup([ModelActuators.so101.gripper]);

function HoldGripper() {
  const controls = useControlGroup(gripper);
  controls.set(ModelActuators.so101.gripper, -0.17453);
  return null;
}

For selector-based introspection, resolveControlGroup() accepts { siteName }, { bodyName }, { joints }, or { actuators }. Selectors can be a name, ordered name array, regex, or predicate.

Build Observations

Build policy-ready observation vectors from common MuJoCo state without hard-coding offsets:

import { buildObservation, useBeforePhysicsStep } from "mujoco-react";

function PolicyDriver() {
  useBeforePhysicsStep(({ model, data }) => {
    const obs = buildObservation(model, data, {
      qpos: true,
      qvel: true,
      ctrl: true,
      sensors: ["imu_gyro", "imu_accel"],
      sites: ["tcp"],
      projectedGravity: "torso",
    });

    runPolicy(obs.values, obs.layout);
  });

  return null;
}

Use output: "float64" when a downstream model expects double precision. Named resources are skipped when absent, so obs.layout is the source of truth for the current model.

WebSocket Control

Stream actuator commands over a WebSocket and send simulation state back. Use a schema validator such as Zod at this boundary because socket messages are untrusted app input (npm install zod for this example):

import { useEffect, useRef } from "react";
import { z } from "zod";
import { useMujoco, useBeforePhysicsStep, useAfterPhysicsStep } from "mujoco-react";

const CtrlCommand = z.preprocess((data) => {
  try {
    return typeof data === "string" ? JSON.parse(data) : data;
  } catch {
    return undefined;
  }
}, z.object({
  type: z.literal("ctrl_command"),
  ctrl: z.array(z.number()),
}));

function useWebSocketControls(url: string) {
  const wsRef = useRef<WebSocket | null>(null);
  const latestCtrlRef = useRef<number[] | null>(null);

  useEffect(() => {
    const ws = new WebSocket(url);
    wsRef.current = ws;

    ws.onmessage = (evt) => {
      const command = CtrlCommand.safeParse(evt.data);
      if (command.success) latestCtrlRef.current = command.data.ctrl;
    };

    return () => ws.close();
  }, [url]);

  // Apply incoming actuator controls each physics step.
  useBeforePhysicsStep(({ model, data }) => {
    const ctrl = latestCtrlRef.current;
    if (!ctrl) return;
    for (let i = 0; i < Math.min(ctrl.length, model.nu); i++) {
      data.ctrl[i] = ctrl[i];
    }
  });

  // Send simulation feedback back after physics.
  useAfterPhysicsStep(({ data }) => {
    const ws = wsRef.current;
    if (!ws || ws.readyState !== WebSocket.OPEN) return;

    ws.send(JSON.stringify({
      type: "feedback",
      time: data.time,
      qpos: Array.from(data.qpos),
      qvel: Array.from(data.qvel),
      sensordata: Array.from(data.sensordata),
    }));
  });
}

For reusable controllers with typed config, default merging, and children, use the createController factory:

import {
  ModelActuators,
  createController,
  useBeforePhysicsStep,
  useCtrl,
} from "mujoco-react";

export const MyController = createController<{ gain: number }>(
  { name: "MyController", defaultConfig: { gain: 1.0 } },
  ({ config, children }) => {
    const shoulder = useCtrl(ModelActuators.franka.actuator1);

    useBeforePhysicsStep(({ data }) => {
      shoulder.write(config.gain * Math.sin(data.time));
    });

    return <>{children}</>;
  },
);

// <MyController config={{ gain: 2.0 }}>
//   <Debug showJoints />
// </MyController>

A createControllerHook factory is also available for the hook equivalent — see the Building Controllers guide.

Architecture

<MujocoCanvas> wraps R3F <Canvas> and forwards all Canvas props (camera, shadows, gl, etc.). For full control over the Canvas, use <MujocoPhysics> inside your own:

<MujocoProvider>                           <MujocoProvider>
  <MujocoCanvas config={...}>               <Canvas shadows gl={...}>
    <Scene />                                  <MujocoPhysics config={...}>
    <MyController />                             <MyController />
  </MujocoCanvas>                              </MujocoPhysics>
</MujocoProvider>                              <EffectComposer>...</EffectComposer>
                                             </Canvas>
                                           </MujocoProvider>

Custom IK Solvers

The built-in useIkController() uses Damped Least-Squares. Pass ikSolveFn to swap in your own solver (analytical, learned, etc.):

import { ModelSites } from "mujoco-react";
import type { IKSolveFn } from "mujoco-react";

const myIK: IKSolveFn = ({ position, currentQ }) => {
  return myAnalyticalSolver(position, currentQ); // return joint angles or null
};

const ik = useIkController({ siteName: ModelSites.franka.tcp, ikSolveFn: myIK });

`useIkController(config | null)`

Hook for interactive end-effector control. Pass null to disable IK (safe to call unconditionally):

import { IkGizmo, ModelSites, useIkController } from "mujoco-react";

const ik = useIkController({ siteName: ModelSites.franka.tcp });
return ik ? <IkGizmo controller={ik} /> : null;

Config	Type	Default	Description
`siteName`	`string`	required	MuJoCo site to track
`joints`	`string \| string[] \| RegExp \| (joint) => boolean`	inferred	Explicit hinge/slide joints for IK
`actuators`	`string \| string[] \| RegExp \| (actuator) => boolean`	inferred	Explicit actuators for IK output
`numJoints`	`number`	legacy only	Contiguous qpos/ctrl count from older examples
`ikSolveFn`	`IKSolveFn`	built-in DLS	Custom solver function
`damping`	`number`	`0.01`	DLS damping
`maxIterations`	`number`	`50`	Max solver iterations

Returns IkContextValue | null with methods like setIkEnabled, moveTarget, syncTargetToSite, solveIK, and getGizmoStats.

Pass the returned value to <IkGizmo controller={ik} /> or to your own controller as a prop.

By default the controller infers scalar hinge/slide joints by walking from the site body toward the model root. For robots where the MJCF control layout is not a simple chain, pass explicit names:

import { ModelActuators, ModelJoints, ModelSites } from "mujoco-react";

const leftArmIk = useIkController({
  siteName: ModelSites.franka.tcp,
  joints: [
    ModelJoints.franka.joint1,
    ModelJoints.franka.joint2,
    ModelJoints.franka.joint3,
    ModelJoints.franka.joint4,
    ModelJoints.franka.joint5,
    ModelJoints.franka.joint6,
    ModelJoints.franka.joint7,
  ],
});

const gripperIk = useIkController({
  siteName: ModelSites.franka.tcp,
  actuators: [
    ModelActuators.franka.actuator1,
    ModelActuators.franka.actuator2,
    ModelActuators.franka.actuator3,
    ModelActuators.franka.actuator4,
    ModelActuators.franka.actuator5,
    ModelActuators.franka.actuator6,
    ModelActuators.franka.actuator7,
  ],
});

Loading Models

The loader fetches src + sceneFile, parses the XML for dependencies (meshes, textures, includes), recursively fetches those too, and writes everything to MuJoCo's in-memory WASM filesystem.

// MuJoCo Menagerie
const franka: SceneConfig = {
  src: "https://raw.githubusercontent.com/google-deepmind/mujoco_menagerie/main/franka_emika_panda/",
  sceneFile: "scene.xml",
};

// Any URL
const custom: SceneConfig = {
  src: "http://localhost:3000/models/my_model/",
  sceneFile: "model.xml",
};

SceneConfig

interface SceneConfig {
  src: string;                      // Base URL for model files
  sceneFile: string;                // Entry XML/URDF file, e.g. "scene.xml"
  files?: File[];                   // Local files for browser upload workflows
  environmentFiles?: string[];      // Static MJCF environment XMLs merged before compile
  sceneObjects?: SceneObject[];     // Objects injected into scene XML at load time
  homeJoints?: number[];            // Initial joint positions
  xmlPatches?: XmlPatch[];          // Patches applied to XML files during loading
  onReset?: ({ model, data }) => void;  // Called during reset after mj_resetData
}

Use environmentFiles to compose reusable physics/collision layers with a robot model. For Gaussian splat scenes, keep the .spz as a parallel visual layer and point environmentFiles at a paired MJCF proxy scene only when contact geometry is needed:

const kitchenRobot: SceneConfig = {
  src: "/models/xlerobot/",
  sceneFile: "xlerobot.xml",
  environmentFiles: ["splats/tabletop/scene.xml"],
};

Local Files and URDF

Load browser-selected MJCF or URDF files directly. Folder uploads preserve webkitRelativePath, and flat uploads fall back to matching referenced mesh/texture assets by basename:

import { useEffect, useRef } from "react";
import { useMujoco } from "mujoco-react";

function ModelUpload() {
  const sim = useMujoco();
  const inputRef = useRef<HTMLInputElement>(null);

  useEffect(() => {
    inputRef.current?.setAttribute("webkitdirectory", "");
  }, []);

  return (
    <input
      ref={inputRef}
      type="file"
      multiple
      onChange={(event) => {
        if (sim.isReady && event.currentTarget.files) {
          sim.api.loadFromFiles(event.currentTarget.files);
        }
      }}
    />
  );
}

Adding Objects to Any Scene

const config: SceneConfig = {
  src: "https://raw.githubusercontent.com/google-deepmind/mujoco_menagerie/main/franka_emika_panda/",
  sceneFile: "scene.xml",
  sceneObjects: [
    { name: "ball", type: "sphere", size: [0.03, 0.03, 0.03],
      position: [0.5, 0, 0.1], rgba: [1, 0, 0, 1], mass: 0.1, freejoint: true },
    { name: "platform", type: "box", size: [0.2, 0.2, 0.01],
      position: [0.4, 0.3, 0], rgba: [0.5, 0.5, 0.5, 1] },
  ],
};

XML Patching

xmlPatches: [{
  target: "panda.xml",
  replace: ["name=\"actuator8\"", "name=\"gripper\""],
  inject: "<site name=\"tcp\" pos=\"0 0 0.1\" size=\"0.01\"/>",
  injectAfter: "<body name=\"hand\"",
}]

Components

`<MujocoProvider>`

Loads the MuJoCo WASM module. Wrap your entire app in this.

Prop	Type	Description
`wasmUrl`	`string?`	Custom WASM URL override
`mtWasmUrl`	`string?`	Custom multi-threaded WASM URL override
`threadedLoader`	`(options?) => Promise<unknown>`	Optional loader imported from `@mujoco/mujoco/mt`
`wasmVariant`	`"single" \| "threaded" \| "auto"`	MuJoCo WASM build. Defaults to `"single"`
`timeout`	`number`	WASM load timeout in ms
`onError`	`(error: Error) => void`	Called if WASM fails to load

The official @mujoco/mujoco package also ships a multi-threaded WASM build. Import it only in apps that opt into it:

import loadMujocoMt from "@mujoco/mujoco/mt";
import mtWasmUrl from "@mujoco/mujoco/mt/mujoco.wasm?url";

<MujocoProvider
  wasmVariant="auto"
  threadedLoader={loadMujocoMt}
  mtWasmUrl={mtWasmUrl}
>
  <App />
</MujocoProvider>

"auto" uses the threaded build only when threadedLoader and mtWasmUrl are provided and globalThis.crossOriginIsolated is true. Forced "threaded" mode requires threadedLoader, mtWasmUrl, Cross-Origin-Opener-Policy: same-origin and Cross-Origin-Embedder-Policy: require-corp.

`<MujocoCanvas>`

Thin wrapper around R3F <Canvas>. Accepts all R3F Canvas props plus:

Prop	Type	Description
`config`	`SceneConfig`	Required. Scene/robot configuration
`onReady`	`({ api }) => void`	Fires when model is loaded
`onError`	`(error: Error) => void`	Fires on scene load failure
`onStep`	`({ time, model, data }) => void`	Called each physics step
`onSelection`	`({ bodyId, name }) => void`	Called on double-click
`gravity`	`[number, number, number]`	Override model gravity
`timestep`	`number`	Override model.opt.timestep
`substeps`	`number`	mj_step calls per frame
`paused`	`boolean`	Declarative pause
`speed`	`number`	Simulation speed multiplier
`renderOptions`	`MujocoRenderOptions`	Optional render-time geometry settings such as `meshNormalSmoothing`

`<MujocoPhysics>`

Physics provider for use inside your own R3F <Canvas>. Same physics props as <MujocoCanvas> without the Canvas wrapper. Accepts a ref for the MujocoSimAPI.

<MujocoProvider>
  <Canvas shadows camera={{ position: [2, 2, 2] }}>
    <MujocoPhysics ref={apiRef} config={config} paused={paused}>
      <MyController />
    </MujocoPhysics>
    <OrbitControls />
  </Canvas>
</MujocoProvider>

Prop	Type	Description
`config`	`SceneConfig`	Required. Scene/robot configuration
`onReady`	`({ api }) => void`	Fires when model is loaded
`onError`	`(error: Error) => void`	Fires on scene load failure
`onStep`	`({ time, model, data }) => void`	Called each physics step
`onSelection`	`({ bodyId, name }) => void`	Called on double-click
`gravity`	`[number, number, number]`	Override model gravity
`timestep`	`number`	Override model.opt.timestep
`substeps`	`number`	mj_step calls per frame
`paused`	`boolean`	Declarative pause
`speed`	`number`	Simulation speed multiplier

`<Body />`

Declaratively add physics bodies to the simulation as JSX. Bodies are injected into the MJCF XML before model compilation.

<Body name="cube" type="box" size={[0.05, 0.05, 0.05]}
      position={[0.5, 0, 0.05]} rgba={[1, 0, 0, 1]}
      mass={0.1} freejoint />

// With custom Three.js visuals
<Body name="ball" type="sphere" size={[0.03, 0, 0]}
      position={[0, 0.3, 0.1]} mass={0.5} freejoint>
  <mesh>
    <sphereGeometry args={[0.03]} />
    <meshPhysicalMaterial color="gold" metalness={0.8} />
  </mesh>
</Body>

Prop	Type	Default	Description
`name`	`string`	required	Unique body name
`type`	`'box' \| 'sphere' \| 'cylinder'`	required	Geom type
`size`	`[number, number, number]`	required	Geom size
`position`	`[number, number, number]`	`[0,0,0]`	Initial position
`rgba`	`[number, number, number, number]`	`[0.5,0.5,0.5,1]`	Color (ignored with children)
`mass`	`number?`	--	Body mass in kg
`freejoint`	`boolean?`	--	Add freejoint for free movement
`friction`	`string?`	--	MuJoCo friction params
`condim`	`number?`	--	Contact dimensionality (4-6 for grasping)
`children`	`ReactNode?`	--	Custom Three.js visuals

`<IkGizmo />`

drei PivotControls gizmo that tracks a MuJoCo site and drives IK on drag. Requires a controller from useIkController().

Prop	Type	Default	Description
`controller`	`IkContextValue`	required	Controller from `useIkController()`
`siteName`	`string?`	controller's site	MuJoCo site to track
`scale`	`number?`	`0.18`	Gizmo handle scale
`onDrag`	`({ position, quaternion }) => void`	--	Custom drag handler (disables auto-IK)

`<DragInteraction />`

Click-drag to apply spring forces to bodies. Raycasts to find bodies, applies F = (mouseWorld - grabWorld) * body_mass * stiffness via mj_applyFT.

R3F Group Props

All visual components (DragInteraction, ContactMarkers, Debug, TendonRenderer, FlexRenderer) accept standard R3F group props like position, rotation, scale, visible.

<ContactMarkers visible={showContacts} />
<Debug showJoints scale={0.5} />

`<ContactMarkers />`

InstancedMesh showing MuJoCo contact points for debugging.

Prop	Type	Default	Description
`maxContacts`	`number?`	`100`	Max contacts to display
`radius`	`number?`	`0.005`	Marker sphere radius
`color`	`string?`	`"#4f46e5"`	Marker color
`visible`	`boolean?`	`true`	Toggle visibility

`<SceneLights />`

Auto-creates Three.js lights from MJCF <light> elements. Also available as useSceneLights(intensity?) hook.

`<Debug />`

Visualization overlays:

Prop	Type	Default	Description
`showGeoms`	`boolean?`	`false`	Wireframe collision geoms
`showSites`	`boolean?`	`false`	Site markers
`showJoints`	`boolean?`	`false`	Joint axes
`showContacts`	`boolean?`	`false`	Contact force vectors
`showCameras`	`boolean?`	`false`	MuJoCo camera positions, frustums, and forward rays
`virtualCameras`	`DebugVirtualCamera[]?`	`[]`	Explicit virtual policy/offscreen render camera poses to draw alongside MuJoCo cameras
`showCOM`	`boolean?`	`false`	Center of mass markers
`showInertia`	`boolean?`	`false`	Inertia ellipsoids
`showTendons`	`boolean?`	`false`	Tendon paths
`geomColor`	`string?`	`"#00ff00"`	Color for wireframe geoms
`siteColor`	`string?`	`"#ff00ff"`	Color for site markers
`contactColor`	`string?`	`"#ff4444"`	Color for contact force arrows
`comColor`	`string?`	`"#ff0000"`	Color for COM markers

Camera debug overlays use the live MuJoCo cam_xpos / cam_xmat frame, so the frustum matches mounted camera captures and follows parent body motion. Use virtualCameras for synthetic fixed policy/offscreen render viewpoints that are not declared as MJCF <camera> elements. Debug camera overlays are excluded from camera captures.

`<TendonRenderer />`

Renders tendons as tube geometry from wrap paths.

`<FlexRenderer />`

Renders deformable flex bodies from flexvert_xpos.

`<InstancedGeomRenderer />`

Opt-in renderer for repeated compatible geoms. It batches matching geom shape/material signatures into Three.js InstancedMesh objects and syncs instance transforms from data.geom_xpos / data.geom_xmat.

`<ContactListener />`

Component wrapper for contact events:

<ContactListener
  body="block_1"
  onContactEnter={(info) => console.log("contact!", info)}
  onContactExit={(info) => console.log("released", info)}
/>

`<TrajectoryPlayer />`

Plays back recorded qpos trajectories with scrubbing.

Hooks

`useMujoco()`

Access the simulation API (must be inside <MujocoCanvas> or <MujocoPhysics>). Narrow on isReady, isPending, or isError:

const sim = useMujoco();
if (sim.isReady) {
  sim.api.reset(); // fully typed
}

`useMujocoWasm()`

Access the raw WASM module lifecycle from any child of <MujocoProvider>. Most users won't need this — useMujoco() and hooks like useBeforePhysicsStep handle the model/data lifecycle for you.

import { useMujocoWasm } from "mujoco-react";

const { mujoco, status } = useMujocoWasm();

if (mujoco) {
  const model = mujoco.MjModel.from_xml_path("/path/to/scene.xml");
  const data = new mujoco.MjData(model);
  mujoco.mj_step(model, data);
  console.log(data.qpos);  // joint positions after one step
}

`useBeforePhysicsStep(callback)`

Run logic before mj_step each frame. Write to data.ctrl, apply forces, drive automation.

import { ModelActuators, useBeforePhysicsStep, useCtrl } from "mujoco-react";

const shoulder = useCtrl(ModelActuators.franka.actuator1);

useBeforePhysicsStep(({ data }) => {
  shoulder.write(Math.sin(data.time));
});

`useAfterPhysicsStep(callback)`

Run logic after mj_step each frame. Read results, compute rewards, log telemetry.

`useIkController(config | null)`

Set up IK control for a MuJoCo site. Pass null to disable. Returns IkContextValue | null.

`useCameraAnimation()`

Standalone camera animation hook:

const { getCameraState, moveCameraTo } = useCameraAnimation();

// Animate camera over 1 second
await moveCameraTo(
  new THREE.Vector3(3, 0, 2),
  new THREE.Vector3(0, 0, 0.5),
  1000
);

`useSensor(name)` / `useSensors()`

Read sensor values by name. Returns a SensorHandle with read(), dim, and name:

import { ModelSensors, useSensor } from "mujoco-react";

const imu = useSensor(ModelSensors.g1["imu-torso-angular-velocity"]);
// imu.read() -> Float64Array, imu.dim -> number

`useBodyState(name)`

Position, quaternion, linear/angular velocity of a body (ref-based):

const { position, quaternion, linearVelocity, angularVelocity } = useBodyState("block_1");

`useJointState(name)`

Joint position and velocity:

const { position, velocity } = useJointState("joint1");

`useCtrl(name)`

Read/write actuator control by name. Returns a CtrlHandle with read(), write(), name, and range:

import { ModelActuators, useCtrl } from "mujoco-react";

const gripper = useCtrl(ModelActuators.franka.gripper);
// gripper.read() -> number, gripper.write(0.04), gripper.range -> [min, max]

`useContacts(bodyName?)` / `useContactEvents(bodyName, handlers)`

Query contacts or subscribe to enter/exit events:

useContactEvents("block_1", {
  onEnter: (info) => console.log("contact!", info),
  onExit: (info) => console.log("released", info),
});

`useKeyboardTeleop(config)`

Map keyboard keys to actuators:

import { ModelActuators, useKeyboardTeleop } from "mujoco-react";

useKeyboardTeleop({
  bindings: {
    "w": { actuator: "forward", delta: 0.1 },
    "s": { actuator: "forward", delta: -0.1 },
    "v": { actuator: ModelActuators.franka.gripper, toggle: [0, 0.04] },
  },
});

`useGamepad(config)`

Map gamepad axes/buttons to actuators:

useGamepad({
  axes: { 0: "joint1", 1: "joint2" },
  buttons: { 0: "gripper" },
  deadzone: 0.1,
});

`usePolicy(config)`

Framework-agnostic decimation loop for RL policies:

const obs = useObservation({ qpos: true, qvel: true, projectedGravity: "torso" });

const policy = usePolicy({
  frequency: 50,
  onObservation: () => obs.readValues(),
  infer: ({ observation }) => policySession.run(observation),
  onAction: ({ action, data }) => applyAction(action, data),
});

`buildObservation(model, data, config)` / `useObservation(config)`

Build a flat Float32Array or Float64Array plus a layout map from qpos, qvel, ctrl, actuator activations, sensordata, named sensors, named site positions, and projected gravity.

`useTrajectoryRecorder(config)` / `useTrajectoryPlayer(trajectory, config)`

Record and play back simulation trajectories:

// Record
const recorder = useTrajectoryRecorder({ fields: ["qpos", "ctrl"] });
recorder.start();
// ... interact with simulation ...
recorder.stop();

// Play back recorded frames directly (no conversion needed)
const player = useTrajectoryPlayer(recorder.frames, {
  fps: 30,
  speed: 1.0,        // 0.5x, 1x, 2x, etc.
  loop: true,
  mode: "kinematic",  // or "physics" to replay ctrl through the sim
  onComplete: () => console.log("done"),
});
// player.play(), player.pause(), player.seek(42), player.setSpeed(2)
// player.state → "idle" | "playing" | "paused" | "completed"
// player.progress → 0-1

`useVideoRecorder(config)`

Record the canvas as video:

const video = useVideoRecorder({ fps: 30, mimeType: "video/webm" });
// video.start(), video.stop() -> returns Blob

Camera Frames, Streams, and Tensors

A camera pose — the viewer camera, a MuJoCo cameraName/siteName/bodyName, or an explicit position+lookAt — can be turned into three outputs: a snapshot (PNG/JPEG), a live on-screen stream, or a policy tensor.

Snapshots. captureFrame()/captureFrameBlob() grab the live canvas (or use useFrameCapture() / the standalone captureFrame() when you own the canvas). captureCameraFrame()/captureCameraFrameBlob() render a chosen camera offscreen at a stable resolution without moving the viewport; pass cameraName/siteName/ bodyName for true mounted frames (source.kind lets dataset pipelines reject fallback/synthetic poses). Set mujocoCameraCompatibility to inherit a MJCF camera's resolution, fovy, clipping, and calibrated intrinsics; use visualOverrides/renderIsolation for canonical training captures.

const frame = await apiRef.current?.captureCameraFrame({
  cameraName: "front_camera",
  width: 640,
  type: "image/jpeg",
  mujocoCameraCompatibility: true,
});

Live streams. useCameraStream(canvasRef, options) renders a camera into a DOM <canvas> every frame (offscreen render → blit: no PNG round-trip, no render- loop takeover, splat scenes stream safely). Call it inside <MujocoCanvas>; the canvas can live anywhere in your DOM:

function WristStream({ canvasRef }) {
  useCameraStream(canvasRef, { cameraName: "wrist_cam", width: 256, height: 192 });
  return null;
}

For a transparent picture-in-picture overlay, use <CameraView> (or useCameraViewport to track your own element). It scissors into the main canvas — cheaper, but it takes over the render loop while mounted (incompatible with postprocessing, occluded by opaque DOM), so prefer useCameraStream for panel tiles:

<MujocoCanvas config={config}>
  <CameraView cameraName="wrist_cam" style={{ right: 16, bottom: 16, width: 240, height: 180 }} />
</MujocoCanvas>

Policy tensors. For in-browser inference, capture straight into a Float32Array — no canvas, no PNG. usePolicyCameraTensors keeps one reusable session per camera and re-aims it to the live pose each step:

const cams = usePolicyCameraTensors({
  streams: [
    { key: "wrist", cameraName: "wrist_cam", width: 96, height: 96, layout: "CHW" },
    { key: "front", cameraName: "front",     width: 96, height: 96, layout: "CHW" },
  ],
});

useAfterPhysicsStep(() => {
  const { tensors } = cams.capture();
  const wrist = new ort.Tensor("float32", tensors.wrist.data, [1, ...tensors.wrist.shape]);
});

Use usePolicyCameraTensorsFromMountedStreams for dataset-name resolution, captureCameraFrameTensor() for one-offs, or createCameraFrameCaptureSession()

pixelsToPolicyImageTensor() for lower-level control. The optional mujoco-react/onnx entry point wraps ONNX Runtime Web: createOnnxPolicySession() loads a manifest plus model, and onnxTensorToPolicyActionChunk() decodes the output into action chunks.

Dataset recording. recordMountedCameraFrameSequence() steps a rollout and captures synchronized per-camera frames, resolving LeRobot-style task camera keys to MuJoCo cameras/sites/bodies. It requires every requested cameraKey by default (set requireAll: false for exploratory tooling) and returns a plan, readiness, and per-camera source provenance:

const sequence = await recordMountedCameraFrameSequence(api, {
  cameraKeys: ["head", "left_wrist", "right_wrist"],
  aliases: {
    head: [{ siteName: "head_camera_rgb_optical_frame" }],
    left_wrist: [{ siteName: "left_wrist_camera_optical_frame" }],
    right_wrist: [{ siteName: "right_wrist_camera_optical_frame" }],
  },
  defaults: { width: 640, height: 480, type: "image/png", fov: 45 },
  frames: 16,
  onFrame: ({ frameIndex, cameras }) => queueLeRobotImages(frameIndex, cameras),
});

const manifest = createMountedCameraFrameSequenceManifest(sequence); // dataset-facing manifest

Inside <MujocoCanvas>, useMountedCameraSequenceRecorder() exposes the same planning/recording with React status and a checkReadiness() preflight gate. resolveMountedCameraFrameSource() maps a single dataset feature name to a MuJoCo source (aliases first, then camera > site > body, then normalized prefix/suffix matches) when you need the selector before recording. The lower-level recordCameraSequence() / useCameraSequenceRecorder() take explicit camera configs.

`useCtrlNoise(config)`

Apply Gaussian noise to controls for robustness testing:

useCtrlNoise({ rate: 0.01, std: 0.05 });

`useGravityCompensation(enabled?)`

Applies qfrc_bias to qfrc_applied so joints hold position against gravity.

`useActuators()`

Returns actuator metadata for building control UIs.

`useSitePosition(siteName)`

Ref-based site position/quaternion tracking.

`useBodyMeshes(bodyId)`

Returns the Three.js meshes belonging to a MuJoCo body. Use for custom selection visuals, outlines, postprocessing, or any per-body mesh manipulation:

const meshes = useBodyMeshes(selectedBodyId);

// Use with drei Outline, or manipulate materials directly

`useSelectionHighlight(bodyId, options?)`

Convenience wrapper around useBodyMeshes that applies an emissive highlight:

useSelectionHighlight(selectedBodyId, { color: "#00ff00", emissiveIntensity: 0.5 });

`useSceneLights(intensity?)`

Hook form of <SceneLights>. Create Three.js lights from MJCF definitions imperatively:

useSceneLights(1.5);

MujocoSimAPI

The full API object available via ref or useMujoco() (when isReady):

Simulation Control

Method	Description
`reset()`	Reset sim, re-apply home joints
`setPaused(paused)`	Set pause state
`togglePause()`	Toggle pause, returns new state
`setSpeed(multiplier)`	Set simulation speed
`step(n?)`	Advance exactly n steps while paused
`getTime()`	Current simulation time
`getTimestep()`	Current timestep

State Management

Method	Description
`saveState()`	Snapshot qpos, qvel, ctrl, time, act
`restoreState(snapshot)`	Restore from snapshot
`setQpos(values)` / `getQpos()`	Direct qpos access
`setQvel(values)` / `getQvel()`	Direct qvel access
`setCtrl(nameOrValues, value?)`	Set control by name or batch
`getCtrl()`	Get all control values
`getControlMap()`	Map directly actuated scalar joints to qpos/dof/ctrl addresses
`getActuatedJoints()`	List scalar hinge/slide joints with matching actuators
`resolveControlGroup(selector)`	Resolve qpos/ctrl mapping from a site, body, joint selector, or actuator selector
`buildObservation(model, data, config)`	Build policy-ready observation vectors with layout metadata
`applyKeyframe(nameOrIndex)`	Apply a keyframe
`getKeyframeNames()` / `getKeyframeCount()`	Keyframe introspection

Forces

Method	Description
`applyForce(bodyName, force, point?)`	Apply force via `mj_applyFT`
`applyTorque(bodyName, torque)`	Apply torque via `mj_applyFT`
`setExternalForce(bodyName, force, torque)`	Write to `xfrc_applied`
`applyGeneralizedForce(values)`	Write to `qfrc_applied`

Model Introspection

Method	Description
`getBodies()`	All bodies with id, name, mass, parentId
`getJoints()`	All joints with id, name, type, range, bodyId
`getGeoms()`	All geoms with id, name, type, size, bodyId
`getSites()`	All sites with id, name, bodyId
`getActuators()`	All actuators with id, name, range
`getSensors()`	All sensors with id, name, type, dim
`getSensorData(name)`	Read sensor value by name
`getContacts()`	All active contacts
`getModelOption()`	Timestep, gravity, integrator

Model Mutation

Method	Description
`setGravity(g)`	Set gravity vector
`setTimestep(dt)`	Set timestep
`addBody(body)`	Add a `SceneObject` and recompile the scene
`removeBody(name)`	Remove a generated `SceneObject` and recompile
`recompile(patches?)`	Recompile current scene, optionally appending XML patches
`setBodyMass(name, mass)`	Domain randomization
`setGeomFriction(name, friction)`	Domain randomization
`setGeomSize(name, size)`	Domain randomization

Spatial Queries

Method	Description
`raycast(origin, direction, maxDist?)`	Physics raycast via `mj_ray`
`project2DTo3D(x, y, camPos, lookAt)`	Screen-to-world raycast (returns bodyId + geomId)
`getCanvas()`	Return the underlying R3F canvas element
`getCanvasSnapshot(w?, h?, mime?)`	Base64 screenshot
`captureFrame(options?)`	Capture the canvas as a data URL
`captureFrameBlob(options?)`	Capture the canvas as a Blob

Scene Management

Method	Description
`loadScene(newConfig)`	Runtime model swap
`loadFromFiles(files, options?)`	Load MJCF/URDF from a browser `FileList`

Guides

Building Controllers

See Building Controllers for full patterns including config-driven controllers, IK gizmo coexistence, multi-arm support, and the createControllerHook/createController factories.

Contact Parameters

Objects that need stable contact (grasping, stacking, etc.) require tuned MuJoCo solver parameters — friction, solref, solimp, and condim. See Contact Parameters for details.

Click-to-Select

Combine R3F raycasting with useSelectionHighlight for body selection:

function ClickSelectOverlay() {
  const selectedBodyId = useClickSelect(); // your raycasting hook
  useSelectionHighlight(selectedBodyId);
  return null;
}

See Click-to-Select for the full implementation.

useFrame Priority

Priority	Owner	Purpose
-1	MujocoSimProvider	beforeStep, mj_step, afterStep
0 (default)	SceneRenderer (internal), useIkController, your code	Body mesh sync, IK, rendering

Roadmap

Features planned but not yet implemented:

Feature	Priority	Description
Web Worker physics	P2	Run `mj_step` off main thread via SharedArrayBuffer

WASM Limitations (@mujoco/mujoco)

These MuJoCo features are not yet exposed in the WASM binding:

flex_faceadr / flex_facenum / flex_face -- FlexRenderer renders vertices without face indices
ten_rgba / ten_width -- TendonRenderer uses default color/width

License

Apache-2.0

Name		Name	Last commit message	Last commit date
Latest commit History 145 Commits
.github/workflows		.github/workflows
bin		bin
docs		docs
src		src
type-tests		type-tests
.gitattributes		.gitattributes
.gitignore		.gitignore
.npmrc		.npmrc
.releaserc.json		.releaserc.json
AGENTS.md		AGENTS.md
CHANGELOG.md		CHANGELOG.md
LICENSE		LICENSE
README.md		README.md
code_guidelines.md		code_guidelines.md
package-lock.json		package-lock.json
package.json		package.json
tsconfig.json		tsconfig.json
tsup.config.ts		tsup.config.ts

Folders and files

Latest commit

History

Repository files navigation

mujoco-react

Install

Vite Plugin and Type-Safe Names

Load a Model

Add Scene Tools

Gaussian Splat Environments

Write Controllers

Use the Sim API

Map Controls to Joints

Build Observations

WebSocket Control

Architecture

Custom IK Solvers

useIkController(config | null)

Loading Models

SceneConfig

Local Files and URDF

Adding Objects to Any Scene

XML Patching

Components

<MujocoProvider>

<MujocoCanvas>

<MujocoPhysics>

<Body />

<IkGizmo />

<DragInteraction />

R3F Group Props

<ContactMarkers />

<SceneLights />

<Debug />

<TendonRenderer />

<FlexRenderer />

<InstancedGeomRenderer />

<ContactListener />

<TrajectoryPlayer />

Hooks

useMujoco()

useMujocoWasm()

useBeforePhysicsStep(callback)

useAfterPhysicsStep(callback)

useIkController(config | null)

useCameraAnimation()

useSensor(name) / useSensors()

useBodyState(name)

useJointState(name)

useCtrl(name)

useContacts(bodyName?) / useContactEvents(bodyName, handlers)

useKeyboardTeleop(config)

useGamepad(config)

usePolicy(config)

buildObservation(model, data, config) / useObservation(config)

useTrajectoryRecorder(config) / useTrajectoryPlayer(trajectory, config)

useVideoRecorder(config)

Camera Frames, Streams, and Tensors

useCtrlNoise(config)

useGravityCompensation(enabled?)

useActuators()

useSitePosition(siteName)

useBodyMeshes(bodyId)

useSelectionHighlight(bodyId, options?)

useSceneLights(intensity?)

MujocoSimAPI

Simulation Control

State Management

Forces

Model Introspection

Model Mutation

Spatial Queries

Scene Management

Guides

Building Controllers

Contact Parameters

Click-to-Select

useFrame Priority

Roadmap

WASM Limitations (@mujoco/mujoco)

`useIkController(config | null)`

`<MujocoProvider>`

`<MujocoCanvas>`

`<MujocoPhysics>`

`<Body />`

`<IkGizmo />`

`<DragInteraction />`

`<ContactMarkers />`

`<SceneLights />`

`<Debug />`

`<TendonRenderer />`

`<FlexRenderer />`

`<InstancedGeomRenderer />`

`<ContactListener />`

`<TrajectoryPlayer />`

`useMujoco()`

`useMujocoWasm()`

`useBeforePhysicsStep(callback)`

`useAfterPhysicsStep(callback)`

`useIkController(config | null)`

`useCameraAnimation()`

`useSensor(name)` / `useSensors()`

`useBodyState(name)`

`useJointState(name)`

`useCtrl(name)`

`useContacts(bodyName?)` / `useContactEvents(bodyName, handlers)`

`useKeyboardTeleop(config)`

`useGamepad(config)`

`usePolicy(config)`

`buildObservation(model, data, config)` / `useObservation(config)`

`useTrajectoryRecorder(config)` / `useTrajectoryPlayer(trajectory, config)`

`useVideoRecorder(config)`

`useCtrlNoise(config)`

`useGravityCompensation(enabled?)`

`useActuators()`

`useSitePosition(siteName)`

`useBodyMeshes(bodyId)`

`useSelectionHighlight(bodyId, options?)`

`useSceneLights(intensity?)`

Packages