Evaluation and benchmark partners.
Optimus uses evaluations to decide whether a change is an improvement: a new policy, a new scaffold, a different training recipe. Benchmark partners supply the test suites and scoring those decisions depend on. We integrate your benchmark so Optimus runs it the same way every time and records the results with the run, and customers can require a policy to pass it before they promote it. Your benchmark becomes part of how teams decide what ships.
What you can supply
- Benchmark suite partnersHeld-out benchmarks that show whether a change is a real improvement.
- Scoring and grading partnersGraders, verifiers, and rubrics that turn a benchmark into a decision.
- Human evaluation partnersStudies for the qualities only people can judge well.
- Safety and robustness testing partnersTests that find rare and adversarial failures before customers do.
How the evaluation and benchmark partnership works
Steps
- 01
Share access
You give us access to the product, its technical documentation, and a contact on your engineering team.
- 02
We integrate
We build the integration and write the documentation Optimus follows, so it knows when to use your product and how.
- 03
Test together
We run it in a real training setup and check the results with you before anything reaches a customer.
- 04
Go live
Optimus can select your product in customer runs, on commercial terms we agree with you.
For researchers
If you design benchmarks, evaluation methods, or safety tests, we would like to work with you. Optimus can run your evaluation across many policies and training recipes the same way every time, which is the evidence a new benchmark needs. We are open to publishing results together.
Academic labs, independent researchers, and open-source teams can apply for up to $30,000 in Iacon credits through our research grants.
Talk to us
If a training run consumes what you sell, we should talk. Book a time directly, or write to dev@iaconautonomics.com and tell us which layer you supply.
Book a call