Guides
How to benchmark an LLM router on your own data
Evaluate LLM routing with held-out requests, blinded grading, fixed-model baselines and cost per accepted result. Includes a reusable evaluation worksheet.
Published 3 min read
On this page
A router benchmark should answer a purchasing question: does this configuration finish our work at an acceptable level of quality, cost and speed?
A public leaderboard can help you choose what to test. It cannot answer that question for an application with different prompts, failure costs and output requirements.
Define success before choosing a winner
Write an acceptance rule for each task family. A summary might need to preserve a deadline and an unresolved question. An extraction task might require exact values and an explicit missing-value marker. Code should run in the relevant environment and pass meaningful tests.
Set any non-negotiable requirements separately. A correct answer from a prohibited provider is not an acceptable result. A valid output delivered after the product timeout may also be a failure.
Use representative, held-out requests
Sample the real workload distribution, then include important edge cases. Keep both views: a traffic-weighted aggregate and a breakdown of the difficult or uncommon cases.
Separate the data used to choose prompts and settings from the data used for final evaluation. If requests share templates, customers or conversations, split at that group level to reduce leakage. Remove or protect sensitive data before replaying it through any new service.
A small pilot finds setup errors. It does not establish a reliable savings percentage. Expand the evaluation until the uncertainty is narrow enough for the decision you need to make.
Include baselines that could actually ship
Compare the router with your current fixed model and an inexpensive fixed model that might be good enough. Add a simple rule-based route if it is a realistic option for your team.
Do not compare only with the most expensive model available. That can make any cheaper setup look compelling. Also distinguish a deployable router from an oracle that chooses the highest-scoring model after all answers have been graded. The oracle is an upper bound, not a live routing strategy.
Research benchmarks such as RouterBench provide a framework for studying these tradeoffs. Use them for context while keeping your deployment decision grounded in your own tasks.
Preserve the request and version the setup
Use the same input, relevant history, tools and output budget across configurations where the interfaces permit it. Document necessary differences instead of quietly adjusting one system's prompt.
Record the router configuration, candidate pool, model versions, dates, provider endpoints and concurrency. For NuRoute, also record X-Router-Version, capability version and fallback reason. Otherwise, a change in the underlying router can look like an unexplained result. NuRoute diagnostics
For independent text requests, replay is straightforward. For agents, the first model response changes later tool calls and history. Evaluate full task runs, and distinguish live runs from scoring a cached set of answers.
Grade without model names
Use deterministic checks where possible. For subjective work, blind the reviewer to configuration names and randomize answer order. If an LLM judge assists, state its rubric and version, check a sample with human reviewers, and investigate disagreements.
Do not treat a higher mean grade as proof of a meaningful difference. Show the size of the difference, uncertainty and the types of failure that changed.
Report quality, cost, reliability and time together
| Metric | Calculation or definition |
|---|---|
| Acceptance rate | Accepted results divided by all attempted tasks |
| Cost per accepted result | Total spend across all attempts divided by accepted results |
| Inference spend | All billable model usage, including retries |
| Total service cost | Inference plus platform charges and applicable overages |
| Reliability | Failed, timed-out, malformed and incomplete tasks |
| Completion latency | End-to-end p50 and p95 under stated concurrency |
| Human effort | Review and repair time, reported separately |
If no results are accepted, report cost per accepted result as undefined rather than zero. Include failures in the denominator. For streaming applications, measure time to first token as well as time to a complete usable result.
Estimate uncertainty by resampling the same paired tasks across configurations. If tasks belong to the same conversation or workload, resample those groups together. A confidence interval for an average score is not a guarantee for an individual request.
Read NuRoute's public results with their scope attached
NuRoute publishes enterprise, exam and RouterArena results. Its enterprise test uses 426 held-out prompts, but the documentation explicitly notes that their distribution suits Artemis. Its HLE and GPQA charts include questions used for calibration; they are not wholly untouched holdouts. Treat those as reasons to run your own test. Published methodology
The accompanying evaluation worksheet uses one row per task and configuration. Fill in outcomes and billed cost from your test; its example rows are synthetic, not product results.
Start with the NuRoute quickstart, and compare your results with our guide to cost, quality and latency.