Searching over learning systems

A learning curve describes progress inside a particular system. It tells us how a policy or model changes under the experience, interfaces, and feedback we gave it. When that curve stalls, several explanations remain open: the learner may need more experience, the update rule may be poorly suited to the task, or the system may make the desired behavior difficult to discover.

We approach this as a search problem over the learning system itself. In our work on agents, models, and simulated control, the research decisions extend beyond weight updates. They include what the agent can observe, what one action means, how experience is collected, and which outcomes produce a useful learning signal.

Optimus works at this level: implementing experiments, examining their artifacts, and revising the machinery around a learner. The central question is how to make that search effective. A system with permission to change everything has a large space of possible mistakes. It needs to identify which changes are worth testing and distinguish a useful revision from a more complicated implementation.

The object of search

We describe a candidate design, c, through six coupled choices:

• Environment: the dynamics or external systems that produce consequences.
• Interfaces: the observations available and the actions the agent can express.
• Experience: the data, reset distribution, sampling, and curriculum.
• Algorithm: the rule used to update the learner, with its configuration.
• Feedback: the rewards, targets, preferences, or verification signals guiding those updates.
• Runtime: the implementation that executes the design, including state handling and tool behavior.

These choices operate at different levels. A learning rate is a numeric parameter. Selecting an update rule is a categorical decision. Introducing a new observation or an executable action is a structural change. That structural change can alter which algorithms are applicable and whether previous experience remains usable.

For a trainable model with a compatible initialization, a candidate produces weights through a budgeted learning run:

Here, θ₀ is the initialization, b is the training budget, and ξ captures training randomness. For a frozen-model agent, θ remains fixed while the candidate changes its interfaces, context, or execution. Both cases belong in the outer search, but only the former performs a weight update.

The result we care about is the behavior of the complete candidate under a common evaluation task distribution:

For a fixed training budget b, τ denotes the evaluation trajectory produced by the resulting candidate on task z, including its learned weights when training applies. Here ξ collects training and evaluation randomness. Utility U measures the task outcome; larger values are better. Cost and operational constraints determine which candidates are acceptable alongside that outcome. This is a problem formulation, rather than a specification of Optimus's search policy.

Two levels of adaptation

The inner loop updates a learner under a chosen design. The outer loop changes that design using evidence from experiments. Their feedback is different: a policy update follows its learning signal, while the researcher asks whether that signal and the surrounding machinery produced the intended behavior.

Search over the design

Optimusrevise the design

→

Candidate c

Train, if neededθ₀ → θ_c

RunLearner ↔ Task

→

Evaluatesame task criteria

Scores + traces inform the next design

This separation lets us examine a limitation at the level where it arises. Missing information calls for an observation experiment. Sparse useful feedback calls for investigating the signal or the experience distribution. An action that is difficult to express calls for an interface experiment. These are hypotheses to distinguish, not diagnoses we can infer from a low score alone.

Changing an interface also changes the optimization problem. A robot that selects end-effector targets searches a different action space from one commanding joint torques. A software agent that executes a structured operation faces a different decision horizon from one composing the same operation through many low-level calls. Abstraction can shorten that horizon, but it can also remove flexibility or introduce dependence on a controller. The comparison must include those tradeoffs.

Our broader research direction is to give Optimus enough reach across these components to investigate such explanations. This builds on an established idea in automated learning research: AutoML-Zero searches over learning algorithms, while POET couples environment generation with agent optimization and transfer. Our focus is the research capability required to construct, run, and interpret revisions across a practical learning system.

Improvements depend on their neighbors

A component has no isolated performance value. Its effect depends on what supplies its inputs and what consumes its outputs. A richer observation can help an update rule exploit useful structure, or add noise it cannot use. Stricter verification can improve the quality of accepted experience while reducing its volume enough to impair learning.

This makes component interactions central to the search. Consider revising an interface and an update rule. Comparing the combined revision with the baseline tells us whether it helps, but cannot isolate an interaction. For that, we use a two-factor experiment with four configurations: the baseline, each individual change, and both changes together.

Isolate the interaction

	Original rule	Revised rule
Original interface	Baselinec₀₀	Rule onlyc₀₁
Revised interface	Interface onlyc₁₀	Both changesc₁₁

$$\Delta_{\mathrm{int}} = [J(c_{11}) - J(c_{10})] - [J(c_{01}) - J(c_{00})]$$

The interaction term is zero when the effects are additive on the chosen utility scale. A positive value means the joint effect exceeds the sum of the separate effects; a negative value means it falls short. Neither sign alone establishes that the combined candidate beats the baseline. Estimates require replicated runs under matched conditions and uncertainty intervals; interpretation depends on the outcome scale. A single favorable combined run is weak evidence of an interaction.

Dependencies also restrict which comparisons are valid. If a new action representation changes the meaning of recorded actions, old demonstrations need an explicit conversion or a new collection procedure. If an observation uses privileged simulator state, an evaluation intended to represent deployment cannot silently give the candidate that same advantage. Compatibility is part of the experiment design.

We treat these relationships as a reason to search in terms of explanations. “Increase capacity” proposes a change. “The available representation discards information needed to distinguish these states” proposes a mechanism that an experiment can test. The latter gives Optimus something to investigate even when the candidate fails.

The evidence changes the next decision

A run provides several kinds of evidence. The score estimates an outcome. Trajectories show how the candidate reached it. Execution records reveal whether the implementation performed the intended experiment. These support different conclusions and should remain distinguishable.

This distinction mattered in our internal harness work. A revised harness passed the same artifact checks while taking longer overall and making more calls in a small paired comparison. Inspecting execution exposed unnecessary work around terminal use and state retrieval. Those traces identified avoidable execution overhead; the comparison did not isolate its contribution to total runtime. Passing the task did not establish that the revision was better.

The experimental record should preserve the explanation under test as well as the implementation. Otherwise, the next researcher inherits a list of scores without knowing which assumptions those experiments challenged. Optimus's access to code, artifacts, and persistent records makes these earlier decisions available for subsequent investigation. Whether that access reliably improves experiment selection is itself a research question we can evaluate.

Search also consumes the budget it is trying to use well. A short probe can expose an invalid setup, but it may say little about a method whose gains emerge late. Selecting only candidates with fast initial progress introduces a bias toward their learning dynamics. We therefore need to evaluate a research strategy by the quality of the system it selects within a total budget, including unsuccessful runs and human intervention.

Improvement needs a stable reference

The training objective is part of the candidate. The intended outcome anchors the comparison. A revised reward may provide better learning signals, but its numerical value cannot be compared directly with the previous reward as evidence of progress. Both candidates need assessment under a common criterion.

Adaptive selection creates a second problem. Selection can overfit its evaluation criterion: repeatedly consulting a suite feeds information about it into the outer search, even when the learner never trains on those cases. Development evaluations can guide revisions; separate held-out assessment is needed to estimate how the selected system behaves on unfamiliar cases.

We apply the same reasoning when changing an evaluator. A revision may detect failures the old evaluator missed, but it changes the measuring instrument. Earlier and later candidates must be reassessed under the new protocol, or their results must remain separate. Otherwise, the search can appear to improve while the meaning of the score moves beneath it.

The distinction becomes especially important when Optimus changes a component it will use in later research. That creates a recursive dependency: the candidate influences how future candidates are proposed or judged. Establishing sustained improvement requires evidence across those successive revisions, with an evaluation reference the system cannot improve merely by rewriting.

The research question

Our work already includes building experimental infrastructure, diagnosing failures, and revising agent and learning-system components with Optimus. The broader hypothesis is that search across these components can reach useful designs that a restricted search would miss.

A controlled comparison would hold the total research budget and initial resources comparable, allow one approach to revise a fixed set of parameters, and allow the other to propose valid structural changes. It would measure the selected systems on held-out tasks, account for execution failures and human assistance, and test whether the research strategy transfers across task families. Expert-designed pipelines and competitive automated-search methods should serve as additional baselines.

For teams building agents, the consequence is direct. A performance limit may belong to the model, the experience available to it, or the system through which it acts. Optimus's role is to make those possibilities experimentally tractable. The capability we are pursuing is a researcher that can locate a limiting assumption, construct a valid alternative, and establish whether changing it helps.