Benchmarks: Artemis Task
Three tests answer three questions:
- Does it hold up on work it has never seen? RouterArena is an independent public benchmark for routers: 8,400 questions from 9 domains, graded by fixed rules, with a public leaderboard.
- How does it do on hard exam questions? HLE and GPQA Diamond, answered by every model it can pick.
- What does it save on business work? Our enterprise test covers 426 held-out business prompts across 15 industries. We built and graded it ourselves, and it favours Artemis Task: see why.
- RouterArena: 75.5% correct at $0.60 per 1,000 queries. GPT-5 answering every question scored 74.0% at $10.02, so Artemis Task was more accurate at 6% of the cost.
- Enterprise test: at the default Balanced preset, Artemis Task matched Claude Sonnet 5's quality (0.952 against 0.949) at 3.5% of its cost.
How Artemis Task decides
Artemis Task chooses a model per request, from the request alone:
- A small fine-tuned text encoder, not a generative model, reads the prompt and returns its mix of task types.
- The capability matrix turns that mix into each model's expected score, and an answer-length model into each model's expected cost.
- The preset picks the model with the best expected score after a penalty for extra cost; the penalty's weight is the only difference between the presets.
The decision makes no extra LLM call and uses no tokens. New models join through the capability matrix without retraining the rest: see The capability matrix.
RouterArena: an independent benchmark
RouterArena is an open benchmark and leaderboard for LLM routers (paper). Its 8,400 questions span 9 domains and 44 categories, including maths, code, science, medicine, law, finance, translation and general knowledge, at three levels of difficulty. Fixed rules grade every answer, such as the right option, the right number, or code that passes its tests. No LLM judge is involved, so no model's taste decides the score. Artemis Task was never trained or tuned on these questions.
| Correct answers | Cost per 1,000 queries | Arena score | Robustness | |
|---|---|---|---|---|
| Artemis Task (Balanced) | 75.5% | $0.60 | 72.9 | 80.0 |
| GPT-5 answering every question | 74.0% | $10.02 | 64.3 | — |
- More accurate than GPT-5, at 6% of its cost. Artemis Task sent 64% of the questions to gpt-5.6-luna and 35% to DeepSeek-V4-Flash. The remaining 1% went to Kimi-K2.6, Nemotron-3-Nano and ministral-8b. GPT-5's row is RouterArena's own measurement, from its leaderboard.
- Arena score is RouterArena's single number for accuracy and cost together, weighted towards accuracy. RouterArena's own evaluation of our submission scored 72.9. That places Artemis Task 12th of the 34 entries on the leaderboard as of 9 October 2026, where the top entry scores 76.3. The listing appears once RouterArena approves it.
- Robustness is the share of 420 reworded questions for which a router picks the same model as for the original wording. At 80.0, Artemis Task's choice stayed the same for four rewordings in five.
- Where it scores lowest: quiz-bowl trivia and questions about long stories. Both are far from the business tasks Artemis Task was built for.
Public benchmarks: HLE and GPQA Diamond
Every model Artemis Task can pick answered every question of HLE (500 text-only questions) and GPQA Diamond (190) through our API, and each preset is scored on the model it picked (capability version cap-12, 12 October 2026). The charts set the presets against a selection of those models. Half of the questions were used to tune its model ratings; on the other half the presets score within about 2 points of these charts.
- HLE
- GPQA Diamond
- Artemis Task preset
- One model answering every question
- Best score at each cost
Show the numbers
| Setup | Type | Score | Cost / 100k questions |
|---|---|---|---|
| ministral-8b | Single model | 4.2% | $50 |
| gpt-6-luna | Single model | 28.0% | $160 |
| Artemis · Cost-first | Artemis Task preset | 29.6% | $246 |
| gpt-5.6-luna | Single model | 23.8% | $282 |
| Qwen3-235B | Single model | 12.0% | $317 |
| Claude Haiku 5.5 | Single model | 31.0% | $362 |
| Artemis · Balanced | Artemis Task preset | 34.0% | $461 |
| Nemotron-3.5-Lightning | Single model | 11.2% | $686 |
| gpt-6-sol | Single model | 41.8% | $1,704 |
| Claude Sonnet 5.5 | Single model | 46.4% | $5,104 |
| Artemis · Quality-first | Artemis Task preset | 54.4% | $6,195 |
| Claude Opus 5.5 | Single model | 56.6% | $6,410 |
| gpt-5.5 | Single model | 40.8% | $13,875 |
| Kimi-K2.6 | Single model | 35.0% | $16,236 |
| gemini-3.1-pro | Single model | 45.8% | $20,203 |
Our enterprise test
This test measures what Artemis Task saves on business tasks: extraction, summarisation, drafting, classification, analysis, coding and more. The prompts are written the way people in real roles write, across 15 industries, in English and German, together with real-world requests from a public dataset. It compares Artemis Task with single models answering every request, from cheap to expensive, and with a random mix of the models it chose. The test was pre-registered: its 426 prompts were held out of training and sealed before any grade was read.
Why this test favours Artemis Task
We built this test, and it suits Artemis Task in four ways. Read its numbers as the best case, on the work Artemis Task was built for. RouterArena is the check on work it has never seen.
- The prompts are its home ground. Test and training prompts come from the same corpus, with the same industries, task types and writing styles. Whole workloads were held out, but none of the prompts is new to Artemis Task in kind.
- Cheap models handle these tasks well. On many of them, cheap models score close to frontier ones, so keeping 97% of the quality costs little. That is what makes savings this large possible. On harder tasks, cheap models fall further behind.
Against single models
At the default Balanced preset, Artemis Task scored 0.952 at $141 per 100,000 requests. That matched Claude Sonnet 5 answering every request (0.949) at 3.5% of its cost, and kept 97.5% of gpt-5.5's quality (0.977) at 2.5% of its cost.
Quality against cost per request on the held-out test, for every setup: higher is better, further left is cheaper. The line marks the best quality reached at or below each cost.
- Artemis Task preset
fallback-router:apolloat the same preset- One model answering every request
- Best score at each cost
Show the numbers
| Setup | Type | Score | Cost / 100k requests | Models used |
|---|---|---|---|---|
| ministral-8b | Single model | 0.687 | $87 | — |
| Nemotron-3-Nano | Single model | 0.775 | $91 | — |
| Artemis · Cost-first | Artemis Task preset | 0.905 | $104 | DeepSeek-V4-Flash 62%, Nemotron-3-Nano 19%, gpt-5.6-luna 18% |
| Apollo · Cost-first | fallback-router:apollo | 0.788 | $112 | — |
| DeepSeek-V4-Flash | Single model | 0.943 | $130 | — |
| Artemis · Balanced | Artemis Task preset | 0.952 | $141 | gpt-5.6-luna 54%, DeepSeek-V4-Flash 45%, Nemotron-3-Nano 1% |
| Qwen3-235B | Single model | 0.853 | $142 | — |
| gpt-5.6-luna | Single model | 0.962 | $189 | — |
| MiniMax-M3 | Single model | 0.938 | $507 | — |
| Apollo · Quality-first | fallback-router:apollo | 0.916 | $1,144 | — |
| gemini-3.8-flash | Single model | 0.909 | $1,307 | Also the judge |
| Artemis · Quality-first | Artemis Task preset | 0.967 | $1,558 | gpt-5.5 57%, gpt-5.6-luna 36%, DeepSeek-V4-Flash 7% |
| Apollo · Balanced | fallback-router:apollo | 0.837 | $2,155 | — |
| Kimi-K2.6 | Single model | 0.951 | $2,398 | — |
| Claude Sonnet 5 | Single model | 0.949 | $3,988 | — |
| gpt-5.5 | Single model | 0.977 | $5,596 | — |
The chart shows all ten models in the pool, and fallback-router:apollo, which decides the requests Artemis
Task cannot, at each preset. Each preset is its own measured operating point; nothing between them is
interpolated. Not plotted: the best model for each prompt in hindsight, 0.997 at about $302 per 100,000
requests, an upper bound rather than a router.
Per 100,000 requests, on the held-out test, cheapest first:
| Setup | Quality | Cost / 100k | vs gpt-5.5 |
|---|---|---|---|
| ministral-8b for every request | 0.687 | $87 | −98.4% |
| Nemotron-3-Nano for every request | 0.775 | $91 | −98.4% |
| Artemis Task, Cost-first | 0.905 | $104 | −98.1% |
| DeepSeek-V4-Flash for every request | 0.943 | $130 | −97.7% |
| Artemis Task, Balanced (default) | 0.952 | $141 | −97.5% |
| Qwen3-235B for every request | 0.853 | $142 | −97.5% |
| MiniMax-M3 for every request | 0.938 | $507 | −90.9% |
| gemini-3.8-flash for every request | 0.909 | $1,306 | −76.7% |
| Artemis Task, Quality-first | 0.967 | $1,558 | −72.1% |
| Kimi-K2.6 for every request | 0.951 | $2,397 | −57.2% |
| Claude Sonnet 5 for every request | 0.949 | $3,988 | −28.7% |
| gpt-5.5 for every request | 0.977 | $5,596 | — |
| The best model for each prompt, in hindsight | 0.997 | about $300 | about −95% |
Each preset matches or beats far more expensive single models:
- Balanced (the default) matches Claude Sonnet 5 and Kimi-K2.6 at 3.5% and 6% of their cost, and keeps 97.5% of gpt-5.5's quality at 2.5% of its cost.
- Quality-first scores above Claude Sonnet 5 at 39% of its cost, and keeps 99% of gpt-5.5's quality at 28% of its cost.
- Cost-first keeps 92.7% of gpt-5.5's quality at 1.9% of its cost.
Quality is the judge's mean grade from 0 to 1. The single models shown are a selection of the pool's ten; the chart above shows all ten. The last row, the best model for each prompt picked after grading, is an upper bound, not a router.
Three presets, one dial
The trade-off is a dial, and Artemis Task's dial is its preset. Each preset is fitted on the training prompts to keep a set share of the best single model's quality at the lowest cost it can. On the held-out test, each preset kept its target:
| Preset | Quality target | Quality retained [90%] | Cost saved |
|---|---|---|---|
| Quality-first | 99% | 99.1% [98.4%, 99.8%] | 72.1% |
| Balanced (default) | 97% | 97.5% [95.9%, 99.1%] | 97.5% |
| Cost-first | 92% | 92.7% [90.6%, 94.8%] | 98.1% |
Each preset is its own measured operating point; nothing between them is interpolated. Your routing
strategy selects the preset (highest_quality, balanced or cheapest): see
Presets.
Results by industry
Each row compares Artemis Task at Balanced with gpt-5.5 answering every task in that industry. Cost saved is how much less Artemis Task spent: 97.6% saved means about $2.40 where gpt-5.5 cost $100. Quality retained is Artemis Task's average grade as a share of gpt-5.5's, so above 100% means it scored higher; the brackets give the range the true value very likely falls in (90%). Tasks are the prompts tested, and workloads the business situations they come from. Fewer workloads give a wider range, and † marks industries with fewer than six, to read as indicative. The 1,475 tasks combine the 426 held-out test prompts with the 1,049 training prompts, each training prompt scored by a router fitted without that prompt's workload (cross-validation).
| Industry | Tasks | Workloads | Cost saved | Quality retained [90%] |
|---|---|---|---|---|
| Manufacturing | 114 | 11 | 97.6% | 96.4% [94.3%, 98.3%] |
| Travel | 112 | 10 | 97.6% | 98.3% [96.9%, 99.4%] |
| Insurance | 106 | 13 | 97.7% | 99.3% [96.9%, 102.0%] |
| Government | 85 | 9 | 97.4% | 96.7% [92.4%, 101.3%] |
| Logistics | 82 | 8 | 97.7% | 95.6% [93.4%, 97.8%] |
| Media | 79 | 9 | 97.8% | 94.3% [90.4%, 98.6%] |
| Banking / financial services | 77 | 6 | 97.4% | 98.7% [95.2%, 102.4%] |
| E-commerce | 77 | 6 | 97.5% | 96.4% [92.7%, 100.0%] |
| Telecommunications | 75 | 8 | 97.5% | 98.1% [96.7%, 99.3%] |
| Pharmaceutical | 73 | 8 | 97.5% | 97.5% [95.7%, 98.8%] |
| Education | 66 | 7 | 97.3% | 99.1% [96.0%, 102.8%] |
| SaaS / technology† | 64 | 5 | 97.9% | 100.2% [96.3%, 104.3%] |
| Legal | 59 | 8 | 97.2% | 95.1% [90.8%, 99.8%] |
| Consulting | 54 | 6 | 97.7% | 96.1% [93.4%, 98.6%] |
| Healthcare† | 52 | 4 | 97.8% | 98.3% [96.5%, 99.6%] |
| Real requests (no industry)* | 300 | — | 97.7% | 96.5% [94.7%, 98.3%] |
| All workloads | 1,475 | 118 | 97.6% | 97.2% [96.4%, 97.9%] |
* Real requests come from a public dataset and carry no industry.
Quality-first and Cost-first by industry
Quality-first (target: 99%):
| Industry | Tasks | Workloads | Cost saved | Quality retained [90%] |
|---|---|---|---|---|
| Manufacturing | 114 | 11 | 76.9% | 98.0% [94.9%, 100.5%] |
| Travel | 112 | 10 | 72.3% | 99.7% [99.4%, 100.0%] |
| Insurance | 106 | 13 | 81.6% | 100.8% [99.0%, 102.9%] |
| Government | 85 | 9 | 70.2% | 99.0% [97.8%, 100.0%] |
| Logistics | 82 | 8 | 77.7% | 97.1% [94.9%, 99.3%] |
| Media | 79 | 9 | 71.5% | 97.7% [96.0%, 99.7%] |
| Banking / financial services | 77 | 6 | 85.6% | 97.7% [96.8%, 98.8%] |
| E-commerce | 77 | 6 | 68.8% | 100.4% [100.0%, 101.2%] |
| Telecommunications | 75 | 8 | 78.6% | 98.7% [95.8%, 100.0%] |
| Pharmaceutical | 73 | 8 | 76.9% | 99.4% [98.8%, 100.0%] |
| Education | 66 | 7 | 68.6% | 100.8% [98.2%, 103.5%] |
| SaaS / technology† | 64 | 5 | 87.7% | 99.2% [97.4%, 101.0%] |
| Legal | 59 | 8 | 71.9% | 97.6% [95.7%, 100.3%] |
| Consulting | 54 | 6 | 84.8% | 98.1% [96.1%, 99.6%] |
| Healthcare† | 52 | 4 | 86.2% | 100.6% [100.0%, 101.8%] |
| Real requests (no industry)* | 300 | — | 48.2% | 99.5% [98.9%, 100.1%] |
| All workloads | 1,475 | 118 | 73.2% | 99.1% [98.7%, 99.5%] |
Cost-first (target: 92%):
| Industry | Tasks | Workloads | Cost saved | Quality retained [90%] |
|---|---|---|---|---|
| Manufacturing | 114 | 11 | 98.1% | 88.4% [83.4%, 93.0%] |
| Travel | 112 | 10 | 98.2% | 91.9% [89.2%, 94.4%] |
| Insurance | 106 | 13 | 98.3% | 95.9% [92.5%, 99.2%] |
| Government | 85 | 9 | 98.1% | 91.5% [86.9%, 96.4%] |
| Logistics | 82 | 8 | 98.2% | 90.3% [85.2%, 95.2%] |
| Media | 79 | 9 | 98.4% | 89.3% [85.4%, 94.2%] |
| Banking / financial services | 77 | 6 | 98.1% | 92.1% [88.3%, 94.3%] |
| E-commerce | 77 | 6 | 98.3% | 90.5% [87.8%, 93.0%] |
| Telecommunications | 75 | 8 | 98.3% | 94.2% [91.8%, 96.8%] |
| Pharmaceutical | 73 | 8 | 98.3% | 93.2% [91.9%, 94.0%] |
| Education | 66 | 7 | 98.1% | 95.3% [90.2%, 100.7%] |
| SaaS / technology† | 64 | 5 | 98.5% | 93.5% [90.5%, 95.8%] |
| Legal | 59 | 8 | 97.8% | 88.1% [83.6%, 94.2%] |
| Consulting | 54 | 6 | 98.6% | 87.1% [82.6%, 91.7%] |
| Healthcare† | 52 | 4 | 98.4% | 88.8% [82.5%, 95.0%] |
| Real requests (no industry)* | 300 | — | 98.3% | 95.3% [93.2%, 97.2%] |
| All workloads | 1,475 | 118 | 98.3% | 92.2% [91.3%, 93.2%] |
Speed
Every figure above is at Speed Off, which weighs quality and cost only. At Off, Balanced's median answer takes longer than the frontier model's. The Speed setting trades cost for time and keeps each preset's quality target.
These figures are at Balanced, cross-validated on the training prompts. Answer times are as timed when the corpus was answered in batches; in production, Artemis Task uses the latencies it observes.
| Configuration | Quality retained | Cost / 100k | Median answer time | 90th percentile |
|---|---|---|---|---|
| Artemis Task, Balanced, Speed Off | 97.1% | $135 | 11.4 s | 97.2 s |
| Artemis Task, Balanced, Speed Faster | 97.1% | $145 | 7.5 s | 63.0 s |
| Artemis Task, Balanced, Speed Fastest | 97.1% | $174 | 5.8 s | 25.2 s |
| Frontier model (gpt-5.5), everything | 100% | $5,744 | 7.4 s | 35.2 s |
At Fastest, Balanced answers sooner than the frontier model at the median and at the 90th percentile, for 3% of its cost.
See also
- Routing with Artemis Task: how it decides and the capability matrix
- Cost, latency and quality controls: the presets and the Speed setting