MuJoCo: building a robot learning testbed with Optimus

MuJoCo gives robotics researchers a place to study movement, contact, and control before taking an experiment to hardware. For this independent Iacon project, we gave a simulated Franka Panda a familiar task: move three cubes from one bin to another.

Our question was how to turn something a person can demonstrate into something a learner can practice. We worked with Optimus on the machinery around that question: a teleoperation interface, an inverse-kinematics controller, recorded trajectories, and an executable definition of success. MuJoCo supplied the physics; Iacon conducted the research.

A successful pick is easy to recognize in a video. Making it useful to a learning system takes more work. We need to know what the operator asked for, what the arm actually did, and whether the object ended up where it belonged. Here is how we made those questions inspectable.

The pick-and-place result

In this MuJoCo run, the Panda transfers all three cubes from the blue bin to the green bin. Two synchronized views show the robot's approach, grasp, lift, and release. Both are rendered from the same joint and object states.

Simulated robot trajectory · 4×

The motion traces below follow this same run, connecting each visible pick to the cube's measured position and its entry into the target region.

Turning a thumb movement into joint motion

We collected demonstrations through teleoperation. The native collector uses a PS5 controller; the browser implementation provides another interface for guiding the same simulated arm. The controls specify a gripper position and orientation, which inverse kinematics translates into joint motion.

The operator should be thinking about the cube, rather than coordinating seven arm joints. In the native collector, we express that intent as a target position and orientation for the gripper. At each control step, the servo compares the target with the observed pose and computes a joint velocity that reduces the error.

From a gesture to a trajectoryNative collector

The implementation uses damped least-squares inverse kinematics:

\dot{\mathbf q}=J^{\mathsf T}\!\left(JJ^{\mathsf T}+\lambda^2 I\right)^{-1}k\mathbf e

Here, J is the gripper's six-by-seven Jacobian, e is its position and orientation error with the rotational terms weighted, and k controls how quickly the servo tries to close that error. The damping term λ²I limits the amplification that an ill-conditioned Jacobian can cause. We then cap the joint velocity, integrate it over the control timestep, and clamp the command to the joint limits.

This is a feedback controller, not a learned policy. It gives the operator a usable action interface and makes the demonstrations interpretable: a pose request, a joint command, and an observed movement remain distinct. That separation is just as useful later when the request comes from a learner.

Read the trajectory, not just the last frame

The browser episode contains 5,145 saved states over 86.7 seconds of simulation. Each state includes joint positions and velocities, actuator commands, the gripper target, and the cube poses. Those records let us move from “the pick worked” to a much more useful question: what happened during it?

Three picks, one recorded trajectoryObserved motion

Cube heights during the run, with dashed lines marking entry into the target region.

Read the traces from top to bottom. The yellow cube reaches the target region at 21.7 seconds, the violet cube at 51.2 seconds, and the red cube at 80.3 seconds. The rises show each lift; the dashed markers show when the recorded cube center first enters the target region. All three remain inside through the end of this recording.

A trace makes the sequence visible in a way that a final success label cannot. We can locate the lift, inspect the transfer, and replay the release. If a future policy drops a cube or approaches it poorly, this is the level of evidence we want beside its score. These timings describe this particular demonstration; they are not an estimate of autonomous policy performance.

Make “placed” something the code can decide

A cube passing through the target region is different from a cube resting there. The browser recording uses a position-based check: all three cube centers must be inside the target box. The native collector implements a stricter check that also requires each cube to settle:

\mathrm{placed}_i=\mathbf 1\!\left[\mathbf p_i\in B\;\land\;\lVert\mathbf v_i\rVert\leq v_{\mathrm{settle}}\right]

B is the target region, pᵢ is a cube's center, and ‖vᵢ‖ is its linear speed. The native configuration sets the speed threshold to 0.05 m/s and requires all three placements to remain valid for 0.5 seconds before automatically completing the episode. The equation above describes that native check; it is not the rule used for the browser plot.

We also keep the operator's annotation separate from the computed outcome. Pressing a “success” button records a human judgment; it does not overwrite what the task evaluator observed. For training data, that is a useful boundary: an encouraging label should never erase a failed placement.

Where learning enters the picture

Once observations and actions are recorded, we can define a learning experiment precisely. One possible starting point is behavioral cloning: train a policy to predict the demonstrated action from the observation. For a continuous action representation, a basic objective is:

\mathcal L_{\mathrm{BC}}(\theta)=\frac{1}{|\mathcal D|}\sum_{(o,a)\in\mathcal D}\!\left\lVert\pi_\theta(o)-a\right\rVert^2

The dataset D supplies observation–action pairs; πθ is the learner. Before fitting it, we would choose the observation fields, action units, and sampling interval, then use the same interface during rollouts. Raw controller inputs, gripper targets, and actuator commands are different possible targets. Mixing them would give the learner an inconsistent task.

Cloning error would tell us how closely the model matches the examples. It would not tell us whether the robot reliably completes the pick. That requires separate autonomous rollouts, with a fixed placement rule and recorded failures. The materials presented here establish the collection and inspection workflow. They do not establish a trained policy's success rate or a policy-improvement curve.

Hill climbing: make the next policy earn its place

We organize Optimus's outer research loop around hill climbing. Start with an incumbent, propose a candidate, and evaluate it against the same task. Keep the candidate when it improves the score; otherwise keep the incumbent and use the failed attempt to inform the next experiment.

This search operates around training. A candidate experiment might change the demonstration mix, the reward, or the training configuration. The learning algorithm produces a policy; the outer loop decides whether that policy deserves to replace the one we already have.

Progress through policy revisionsIllustrative · not measured results

The selection rule can be written compactly:

c_{k+1}\in\underset{c\,\in\,\{c_k\}\,\cup\,\mathcal C_k}{\arg\max}\;\widehat J(c)

Here, cₖ is the incumbent policy and its configuration, Cₖ is the set of new candidates, and Ĵ is the evaluation score. We include the incumbent in the comparison so a worse candidate does not replace it; ties can leave the incumbent in place. A better score must use the same evaluation conditions to be a meaningful improvement.

The solid line illustrates candidate policies: gradual improvement, a regression, recovery, and another attempt. The faint line follows the best retained score, holding steady when a candidate gets worse. Those regressions still provide useful evidence. Their trajectories can show whether an apparent improvement in approach introduced a worse release, or whether a reward encouraged behavior that misses the task's purpose. Optimus's job in this loop is to connect that diagnosis to the next candidate.

Retaining the highest observed score does not guarantee better performance on unseen resets. A separate held-out evaluation is still needed to test whether the selected policy generalizes. The schematic explains the method; the missing training history prevents us from attaching measured gains to it.

What this taught us about working with Optimus

The useful unit of work was the whole experiment: controls that a person could use, a simulator that exposed the right state, trajectories that could be replayed, and a task check that meant something. Working with Optimus on that system gave us concrete artifacts to inspect at each step.

It also sharpened the questions we would bring to a customer's robot task. What should the learner observe? What does an action mean? Which examples are worth learning from? What must remain true before we call a trial successful? Answering those questions in code makes the next experiment much easier to evaluate.

For this project, the next test is a learned policy acting through that interface, evaluated across specified resets and compared with its own recorded failures. Physical deployment would be a further experiment. The contribution here is the inspectable path from a human demonstration to that next test.