Benchmark suite partners.
Held-out benchmarks are how a team knows whether a change is a real improvement or just a better fit to its own tests. A good benchmark is stable, versioned, and hard to game.
We would like to integrate your benchmark so Optimus runs it the same way every time, records results with the run, and keeps held-out sets held out. Customers can make your benchmark part of how they decide what ships.
How the benchmark suite partnership works
Steps
- 01
Share access
You give us access to the product, its technical documentation, and a contact on your engineering team.
- 02
We integrate
We build the integration and write the documentation Optimus follows, so it knows when to use your product and how.
- 03
Test together
We run it in a real training setup and check the results with you before anything reaches a customer.
- 04
Go live
Optimus can select your product in customer runs, on commercial terms we agree with you.
For researchers
If you design benchmarks, evaluation methods, or safety tests, we would like to work with you. Optimus can run your evaluation across many policies and training recipes the same way every time, which is the evidence a new benchmark needs. We are open to publishing results together.
Academic labs, independent researchers, and open-source teams can apply for up to $30,000 in Iacon credits through our research grants.
Talk to us
If a training run consumes what you sell, we should talk. Book a time directly, or write to dev@iaconautonomics.com and tell us which layer you supply.
Book a call