Discovering simpler systems through optimization

The system needed to discover a behavior can be much larger than the system needed to execute it. Search explores alternatives. An ensemble compares predictions. A planner considers actions it never takes. Once the behavior works, we have a second optimization problem: determine how much of that machinery the deployed system still needs.

This distinction matters across learned controllers, language models, and tool-using agents. A smaller network may reproduce a teacher's decisions. A short program may replace a repeated sequence of model calls. A compact state representation may preserve the distinctions that a policy uses. Each is a different hypothesis about where the capability resides.

In our work on Optimus, we treat the complete implementation as the object of investigation. The useful result is a cheaper or more legible system with a tested operating range. Removing code or parameters is only one way to reach it.

Preserve behavior, then compare cost

Let s denote a candidate system and J(s) its expected utility on a fixed target distribution. We can express simplification as minimizing a chosen operating cost C(s) subject to a tolerated loss ε:

This formulation requires a behavioral reference, a declared tolerance, and a cost boundary. It does not let a smaller average model size compensate for an unacceptable failure. Task constraints and failure rates need separate assessment, and finite evaluations leave uncertainty around the utility difference.

We measure cost at the boundary where the system will run. Parameter count measures storage more directly than latency. A short agent workflow can hide expensive retrieval or verification. Replacing a controller with a small policy may transfer computation into state estimation. These changes count as simplification only relative to the costs we actually intend to reduce.

Distinguish dependence from necessity

An ablation answers whether the current system depends on a component. Remove a memory, observation feature, or intermediate computation while holding the remaining implementation fixed. If behavior degrades, that component mattered under the intervention.

A replacement experiment asks a different question. Can the remaining system recover after adaptation, or can a cheaper component supply the same function? A failed ablation does not rule out either possibility. Conversely, successful retraining does not mean the component was unnecessary during the original learning process.

We keep the adaptation budget explicit because it changes the result. A replacement that needs substantial training may still be economical when executed frequently. If adaptation and validation have a one-time cost D, and each execution saves cᵣ − cₛ, the replacement has lower cumulative cost when:

Here all costs use the same unit, and per-execution savings are assumed stable. Maintenance, retraining, and hardware changes can move that threshold. The equation separates an expensive discovery from an expensive deployment.

Components also substitute for one another. Two recovery mechanisms may each appear redundant when removed separately because the other covers the failure. Removing both can break the system. The proposed final configuration therefore needs its own evaluation; individual ablations cannot certify the combination.

Compression changes the data distribution

Knowledge distillation uses a teacher's outputs to train another model. For an agent, matching those outputs on saved histories is only an initial test. The student makes its own decisions, which determine the states it visits next.

A small disagreement can take it outside the teacher's familiar trajectories. Errors then accumulate on inputs that were rare in the training data. This is the distribution-shift problem addressed by approaches such as DAgger. A compressed policy needs closed-loop evaluation on the trajectories it generates itself.

The same reasoning applies to replacing model calls with code. A deterministic routine may faithfully handle the common branch while discarding the checks that made uncommon branches recoverable. We need to test the conditions under which the replacement chooses a different action, including delayed feedback and states reached after an earlier mistake.

A compact implementation can improve consistency within its operating range while losing adaptability outside it. That tradeoff belongs in the result. Agreement on a benchmark does not establish equivalence under every intervention.

A simpler representation can reveal structure

Compression becomes scientifically interesting when the replacement exposes a relationship we can test. If a few variables preserve the decisions made from a much larger observation, they are candidates for a useful state representation. If a short expression predicts dynamics across informative interventions, it may explain part of the mechanism.

Sparse identification of nonlinear dynamics illustrates this approach using a library of candidate functions. The library and data determine what can be recovered. A failed fit may indicate missing functions, noisy measurements, or insufficiently varied trajectories. It cannot certify that the underlying dynamics are intrinsically complicated.

Likewise, the lottery ticket hypothesis concerns sparse subnetworks and suitable initializations in the settings studied. It motivates separating the structure needed for learning from the structure that carries a learned behavior. It does not imply that an arbitrary agent architecture contains an easily recoverable minimal implementation.

For Optimus, this makes simplification a useful form of research: construct a smaller candidate, identify where its behavior diverges, and use those divergences to revise the hypothesis. Successful reduction narrows the dependencies we need to maintain. Failed reduction can still identify a function that the proposed replacement has not captured. Both outcomes are more informative than treating implementation size as a proxy for capability.