DetectBench: evaluating investigative agents with Optimus

Resemble AI builds deepfake detection technology for audio, images, and video. Its Detect platform combines media authenticity analysis with explanations that help people understand a finding. An analyst investigating suspicious content needs both a decision and evidence they can examine.

Our collaboration focused on the agents working around that detection technology. Resemble’s experts wanted more reliable investigations, better accuracy, and a way to adapt agents to different tasks. Before we could improve those agents systematically, we needed a clear baseline for their behavior.

Together, we built DetectBench. Resemble brought domain expertise and task requirements. Iacon brought Optimus, our autonomous research agent, to implement the benchmark and turn evaluation findings into tested agent revisions.

Starting with a baseline we could trust

A detector finding is one part of an investigation. The agent still has to choose a tool, interpret what comes back, retain useful observations, and support its conclusion. Two agents can reach the same answer through very different paths. We needed to examine those paths.

We made the complete investigation the unit of evaluation. DetectBench records the final decision alongside tool activity and supporting evidence. It also checks whether the agent completed the task and returned a usable result. That gave us a way to distinguish a decision problem from an execution problem before choosing what to change.

Optimus implemented the test-case pipelines and evaluation harness. We could then compare revisions case by case, checking both corrections and regressions.

Giving every investigation a record

We treated an investigation like a lab notebook: the conclusion should lead back to the observations that produced it. Optimus implemented execution traces and structured reports so we could inspect the tools an agent used, the evidence it retained, and the point at which its reasoning became a final decision.

The benchmark needed the same discipline. Optimus built data pipelines that preserved the relationship between a test input, its transformations, and its reference label. It added output validation and checks that reconciled reports with the recorded runs. A failed run stayed in the record, even if a later recovery attempt succeeded.

Turning failures into the next experiment

Optimus worked on the code between evaluations. It revised execution and recovery paths, implemented state retention, and tested how agents carried observations through an investigation. We kept candidate state separate so a new configuration would not inherit another candidate’s accumulated experience.

Optimus also revised instructions and structured outputs so we could trace a conclusion back to specific observations. This let us examine the support for an answer alongside its correctness, rather than judge an explanation by how convincing it sounded.

Adversarial testing gave us difficult cases to work through. Optimus used variants and previous failures to test candidate changes, then broader evaluations to check behavior elsewhere. Fixing one hard case was a reason to run the next experiment; we still needed to check for regressions.

Measured progress

We measured the final classifications with a simple quantity: accuracy = correct classifications / evaluated cases. Across the evaluated revisions, the best configuration moved from 75.8% to 80.0%, then 91.7%. The total gain was 15.8 percentage points, calculated before rounding.

Best evaluated configuration+15.8 percentage points

Model A · matched track

Model B · matched track

The top curve lets the best configuration change, including its underlying model. The lower curves follow individual model tracks as we revised the agents around them. Model A improved from 45.0% to 85.8%; Model B improved from 70.8% to 79.2%. Model A went from getting fewer than half the cases right to more than four in five. Model B started higher and improved more gradually.

An agent that cannot finish an investigation cannot return a correct classification, so execution reliability matters to these gains alongside evidence use and state handling. We measured the combined revisions. The plots include complete evaluations; targeted adversarial reruns informed the changes but remain outside these comparisons.

Making the next use case easier

Resemble also needed a way to adapt agents without rebuilding the evaluation machinery each time. Optimus worked on separating task instructions and output requirements from the shared execution interfaces. That let us reuse the framework while changing what an agent was asked to investigate and how its result should be evaluated.

Resemble’s experts could set the task and success criteria while Optimus implemented and evaluated a candidate. The team gained revised agents and a framework for testing the next use case with the same infrastructure.

What we learned about Optimus

Building DetectBench gave Optimus a way to inspect agent behavior. Using it raised further questions about the evaluation itself. We learned to treat benchmark construction and agent development as connected research work: Optimus needed to be able to revise the experiment as well as the system under test.

Domain expertise kept that work grounded. Resemble’s experts defined what a useful investigation should accomplish; Optimus carried that direction into code and evaluated configurations. We want customers to retain control of the problem while Optimus takes on the implementation and experimental work.

That is the standard we hold Optimus to: implement an idea, test it, and bring back evidence for the next decision. In this collaboration, it built DetectBench and used it to improve the agents under evaluation. Resemble and Iacon gained a shared basis for deciding what to investigate next.

About these results

These are selected results for investigative agents on a development benchmark, not measurements of Resemble’s underlying detector or production accuracy. Customer information, model identities, and internal implementation details remain private. Establishing the effect of an individual change requires a separate controlled comparison.