From PS5 demonstrations to reliable robotic picking in MuJoCo

The MuJoCo team and Iacon collaborated on a manipulation study with a Franka Panda robot. The task was simple to state and difficult to make reliable: use one policy to pick the varied object categories represented in the study.

The project started with a small amount of human teleoperation data collected through a PS5 controller. Optimus turned those demonstrations into a structured training input, built the simulation and evaluation loop around them, and iterated on the policy until it completed 92% of scheduled picks in the final frozen evaluation.

The number describes autonomous simulation rollouts. It does not combine teleoperated demonstrations with evaluation attempts, and it does not claim performance on a physical robot.

Study boundary	Public description
Relationship	Joint research with the MuJoCo team
Robot	Franka Panda with a simulated gripper
Human input	Project-specific PS5 teleoperation demonstrations
Optimus built	Data, training, rollout, failure-analysis, and evaluation pipelines
Evaluation	Frozen MuJoCo environment covering every object category in the study
Result	92% successful autonomous picks

A varied picking task exposes the whole control loop

Picking one object from one known pose can hide brittle assumptions. The controller may depend on a favorable approach angle, a forgiving contact, or a reset that places the robot close to success. Introducing varied objects changes the geometry of approach and grasp acquisition. A policy must reach a usable pose, close the gripper at the right time, establish contact, and retain the object through the lift.

MuJoCo made those interactions inspectable. The teams could version the scene, robot model, contacts, solver configuration, and reset behavior. Failed attempts produced full trajectories rather than a binary label alone. That made simulation useful as an experimental instrument, provided the physics and evaluator remained fixed when policies were compared.

The study therefore began with an explicit task contract. It declared the observations available to the policy, the action interface, reset conditions, object set, and evaluator used to decide whether a pick succeeded. Fixing that contract before training prevented a common robotics mistake: adjusting the success definition after seeing where the policy failed.

Collecting demonstrations with a PS5 controller

The project used a PS5 controller as a practical teleoperation interface. Its inputs were mapped into the action space of the simulated Franka Panda, letting an operator demonstrate useful approaches and grasp sequences without a specialized robotics console.

The interface was built for this study. It exposed the controls needed by the task and recorded actions in the same coordinate and timing contract used by the training environment. That alignment mattered more than the choice of game controller. Demonstrations become difficult to use when operator actions must be reconstructed from a different control space after collection.

Optimus treated every teleoperation session as a trajectory with a clear lineage. Observations, operator actions, resets, outcomes, and simulator revision stayed connected. The resulting dataset preserved the context required to replay or inspect a demonstration instead of reducing it to an anonymous sequence of control values.

Before training, Optimus checked the episodes against the task contract. It identified incomplete or unusable trajectories, kept successful examples separate from interrupted attempts, and retained the failed demonstrations as diagnostic evidence where they helped explain the boundary of the data.

The amount of human input remained deliberately modest. Demonstrations gave the policy examples of useful behavior; they were not a substitute for autonomous evaluation across the full object set.

Building a reproducible training environment

Robotics results can move when the environment changes underneath them. A different friction value may make a grasp easier. A solver revision may alter contact stability. A reset change may place objects in more favorable poses. If those revisions are not attached to each run, a policy can appear to improve when the world has changed instead.

Optimus versioned the MuJoCo environment used by the study and attached each training and evaluation run to that revision. The run record retained the task definition, runtime configuration, policy artifact, trajectories, and evaluator version. Candidates were compared only when those boundaries matched.

The same discipline applied to data. A policy checkpoint carried the demonstration dataset revision used to train it. When the teams changed the dataset or the environment, that change created a new lineage rather than silently replacing the input to an existing result.

This record turned the simulation into a controlled experiment. The teams could reopen a failed rollout, inspect the robot and object state around the grasp, and determine whether a candidate changed behavior under the same physics.

From demonstrations to autonomous rollouts

Optimus used the teleoperation trajectories as the initial learning signal and then evaluated candidate policies through autonomous simulation rollouts. The public case study leaves the policy architecture and update rule private; the observable experiment contract remains clear.

For each iteration, Optimus trained a candidate against the versioned data, scheduled rollouts across the object set, and passed the trajectories to the fixed pick evaluator. It then grouped failures by the stage at which the task broke down.

Some failures occurred during approach, when the end effector reached an unusable pose. Others occurred during acquisition, when contact or gripper closure did not secure the object. Retention failures appeared after an initial grasp, when the object slipped before the evaluator's success condition was met. Separating these stages prevented one broad "failed pick" count from hiding several different control problems.

Optimus used those groups to choose bounded interventions. A change motivated by approach failures returned to the full object suite rather than only the cases that inspired it. Regressions stayed visible, and unsuccessful candidates remained part of the run history.

The loop continued until the policy was stable enough for a frozen evaluation. Humans did not steer those final rollouts. The policy received the declared observations, produced actions through the same interface used during training, and succeeded or failed under the evaluator fixed at the start of the run.

What the 92% result means

The final policy completed 92% of scheduled picks in the frozen MuJoCo evaluation. Every object category represented in the study appeared in that evaluation, and every scheduled failure remained in the denominator.

The metric does not mean that the robot can pick 92% of arbitrary objects. It summarizes performance on the study's object set, reset distribution, physics revision, and definition of a successful pick. The exact trial count and success criterion belong in the partner-approved technical appendix before the result is used outside this case study.

This scope keeps the result useful. A new policy can be tested under the same conditions. A revised scene can be treated as a new evaluation rather than quietly folded into the existing figure. If the policy reaches the same percentage under different physics, the record will show that as new evidence.

Where Optimus did the work

The MuJoCo and Iacon teams defined the research objective, robot, object boundary, and acceptable evidence. Human operators supplied the PS5 demonstrations.

Optimus built the machinery around that input. It organized and validated the demonstrations, maintained environment and dataset lineage, executed training, scheduled autonomous rollouts, inspected failures, proposed bounded changes, and ran the final evaluation. The 92% result measures the policy produced by that loop, not the skill of the teleoperator.

This division preserved a useful role for human demonstrations without making the research dependent on continuous human steering. People showed the robot examples of the task. Optimus converted those examples into a repeatable experiment and carried the iteration through to evaluation.

What the collaboration produced

The immediate output was a Franka Panda policy that performed reliably across the study's varied object categories. The retained capability is broader: a versioned control task, a structured demonstration dataset, a repeatable training pipeline, and a regression evaluation attached to exact MuJoCo physics.

Those artifacts support the next questions. The teams can test held-out objects, expand the reset distribution, change contact parameters, or compare another learning method without rebuilding the entire experiment. Each new claim can point to the data, environment, policy, and evaluator that produced it.

The current result remains simulation-only. The study does not establish sim-to-real transfer, performance on a physical Franka Panda, or robustness to arbitrary unseen objects. The PS5 collector was project-specific research infrastructure and is not presented as a general arkenOS teleoperation feature.

The public account excludes controller mappings, policy internals, object identities, and per-case trajectories. It still shows the technical value of the work: a small set of accessible demonstrations became a reproducible control campaign, and Optimus carried that campaign to a measured 92% success rate under a frozen simulation contract.