Pareto Router Challenge

The task

You have a workload of 20,000 tool-use queries, each with a known difficulty distribution. You have a fleet of six models at different price + latency points. Write a router that assigns each query to a model — within a budget — and maximises accuracy on a held-out grader.

What we score

• Pareto AUC across five (cost, latency) budgets — higher is better
• Held-out grader: 4,000 queries you never see, drawn from the same distribution
• Tie-break: lowest p95 wall-clock on the same hardware

Rules

• Submit code in a sandboxed Docker image — no internet at grading time
• One active submission per handle; switching counts as a new submission
• Subsequent submissions allowed every 30 minutes
• Winners must open-source their final submission under MIT or Apache-2 within 30 days of close
• Anyone in the world is eligible. Working scientists at frontier labs may compete but are ineligible for prize money.

Starter pack

The starter notebook ships the workload schema, a baseline random router, a baseline cost-only router, and the grading harness so you can validate locally before submitting.