Human evaluation partners.
Some qualities only people can judge well: helpfulness, taste, safety in context, whether a robot's behaviour looks right. Human evaluation is slower than automated scoring, which makes the design of each study matter more.
If you run human evaluation programs, we would like to connect them to the runs Optimus manages, so teams can request a study at the right point and see results next to their automated metrics. We agree study formats and terms with you.
How the human evaluation partnership works
Steps
- 01
Share access
You give us access to the product, its technical documentation, and a contact on your engineering team.
- 02
We integrate
We build the integration and write the documentation Optimus follows, so it knows when to use your product and how.
- 03
Test together
We run it in a real training setup and check the results with you before anything reaches a customer.
- 04
Go live
Optimus can select your product in customer runs, on commercial terms we agree with you.
For researchers
If you design benchmarks, evaluation methods, or safety tests, we would like to work with you. Optimus can run your evaluation across many policies and training recipes the same way every time, which is the evidence a new benchmark needs. We are open to publishing results together.
Academic labs, independent researchers, and open-source teams can apply for up to $30,000 in Iacon credits through our research grants.
Talk to us
If a training run consumes what you sell, we should talk. Book a time directly, or write to dev@iaconautonomics.com and tell us which layer you supply.
Book a call