Observability as a constraint on intelligence

Two situations can look identical to an agent and require different actions. A stationary object and a moving object can occupy the same position in one image. A completed operation and a failed operation can both produce a timeout. A summary can omit the earlier event that distinguishes a safe continuation from a repeated side effect.

These failures can resemble weak reasoning because the visible outcome is a bad decision. But improving inference and improving the available evidence are different interventions. More computation can uncover a subtle implication of an observation. It cannot guarantee a distinction that the observation process never supplied.

In our work on Optimus, we treat the agent's information boundary as part of the system design. We ask what was available at the decision point, which distinctions survived the interface, and what additional observation could change the action.

History determines what can be inferred

The relevant input is usually an interaction history, not a single observation. Earlier actions affect how current evidence should be interpreted. Timestamps, execution identifiers, and sensor history can separate states that would otherwise look alike.

A partially observable model represents uncertainty about the current state with a belief:

Here hₜ contains the observations and actions available through time t. The POMDP framework describes decisions under this uncertainty, including actions that change both the world and what the agent knows about it.

We use observability in a task-relative sense: can the available history distinguish states that require different decisions? Recovering every physical or computational detail is unnecessary if the omitted distinctions never affect the appropriate action.

Memory helps when an earlier observation contains the missing information. Better inference helps when the history contains evidence the model has not extracted. Neither resolves an ambiguity between states that induce exactly the same available evidence and prior information.

A precise limit from indistinguishable evidence

Consider two equally likely hidden states, sₐ and sᵦ, with identical available histories. Action aₐ succeeds only in the first state and aᵦ only in the second. The agent must choose one of these actions without another informative observation.

Its action distribution must be the same in both states. If it selects aₐ with probability p, expected success is:

This is an exact result for the stated example, not a benchmark measurement. Changing the prior, adding an informative query, or allowing an action that succeeds in both states changes the problem. Additional reasoning over the identical history does not.

The distinction is between discovering information already present and obtaining information that was absent. A model may initially miss a useful correlation in a long history; better reasoning can exploit it. The example removes that possibility by construction. It gives us a controlled way to test whether a proposed capability improvement depends on new evidence or better use of existing evidence.

The interface defines an observation process

Sensors, tool responses, retrieval, and summaries all transform underlying state into the input available to the policy. We can describe this transformation as an observation channel. Its design determines which distinctions are preserved and which states become aliased.

Compression is useful only relative to the decision it supports. A concise status response can preserve everything needed to choose between waiting and continuing. The same response may be inadequate for diagnosing why an artifact is incorrect. A long transcript can contain both answers while making either expensive to locate.

We therefore separate current control state from inspectable evidence. The first supports immediate coordination; the second supports diagnosis and revision. The agent needs to know which run a result belongs to and when the state was observed. Without that context, a correct value can support the wrong decision.

An evaluation must reconstruct this information boundary. Evidence written after an action cannot explain why the agent should have chosen differently beforehand. Diagnosing an apparent reasoning error requires the observations available at the time, not the full record assembled after the episode ends.

An informative action changes the problem

An agent can sometimes choose to observe: inspect an artifact, query a status field, or move to another viewpoint. The useful measurement is one that separates possibilities requiring different actions. It need not reconstruct the entire hidden state.

In the two-state example, a perfect query that identifies the state permits the correct action in either case. A noisy query permits less certainty. Its value depends on how much it changes the decision and what it costs to obtain. An observation with high information content can still be useless if it concerns details irrelevant to the task.

Observing may also perturb the system. Repeating an operation to discover whether it succeeded can create a duplicate side effect. Moving a sensor changes the time and geometry of the next measurement. A decision policy must account for those consequences along with the information gained.

For a cost-free passive signal, an ideal decision-maker can ignore irrelevant information. Implemented agents have acquisition and interpretation costs. Additional context can consume the budget needed to act or interfere with useful evidence. Observation quality is therefore a property of the complete decision process, not the volume of data returned.

Separate access from inference in the experiment

We distinguish three observation conditions: the current interface, an improved signal obtainable in deployment, and privileged state available only for diagnosis. Holding the model fixed reveals how performance responds to those changes in access.

Privileged state is not a deployment result. A gain under that condition identifies a limitation worth investigating, but the practical question remains how to supply useful evidence under real constraints. An absent gain is also ambiguous: the model may not use the signal correctly, or another bottleneck may dominate.

We then vary the model under the same observation conditions. This separates stronger inference from a more informative interface. Tests should include stale values, delayed responses, and failed queries, because an observation mechanism that works only when everything succeeds leaves its most consequential behavior unexamined.

For Optimus, the research task is to make these distinctions experimentally accessible. A failure can motivate a new representation, a memory change, or an information-gathering action. Identifying which intervention addresses the missing distinction is often more useful than asking the same model to deliberate longer over unchanged evidence.