How to reduce LLM API costs while meeting your quality requirements

A lower token price is helpful only if the application still finishes the job. If a cheaper model creates more retries, longer answers or extra review work, the saving can disappear.

Start with cost per accepted result. Then identify which parts of the workload can cost less while still meeting your quality requirements.

1. Count the full request chain

A customer action might trigger classification, retrieval, drafting and an agent loop. Count all those calls, including failed attempts, rather than dividing the monthly bill by the number of visible responses.

For each feature, record input tokens, eligible cached-input tokens, output tokens, model, attempts, latency and whether the final result passed review. Separate the inference bill from platform fees and operational costs.

Here is a constructed example, not a NuRoute measurement:

The second route cuts inference spend by 30% but costs more per accepted result. It also leaves more unfinished work. If those failures go to people, account for that review separately.

2. Define acceptable quality for each feature

For extraction, require the right fields, correct values and no invented information. For summaries, identify facts that must survive. For code, use execution and tests. For customer replies, check policy adherence and whether the suggested next step is correct.

Do not use one vague "helpfulness" score for all of them. A fluent summary with a missing deadline can be a failed result even when a reviewer likes the writing.

Keep a held-out set of representative requests for final evaluation. Repeatedly tuning on the same examples makes the result less useful as evidence about new traffic.

3. Remove waste before adding routing

Send relevant context rather than every document available. Remove repeated boilerplate where the application does not need it. Set output limits appropriate to the task and check for truncated answers.

Use deterministic code for deterministic work. Parsing a known date format or filtering by an exact identifier may not need an LLM. Keep the model for the part that needs interpretation.

Investigate repeated attempts. Authentication errors, invalid parameters and policy rejections need fixes; retrying them adds latency without improving the request.

4. Test an inexpensive fixed model

If one less expensive model passes your task rubric, start there. It may be simpler and more predictable than introducing per-request selection.

Routing becomes more interesting when the workload contains both easy requests and a significant number of difficult cases. Research on learned routing and cascades explores that opportunity, but the published savings apply to their experiments. They are not forecasts for your traffic. RouteLLM, FrugalGPT

5. Evaluate model routing against that baseline

Test the same held-out requests with your current model, the inexpensive candidate and the router. Compare acceptance rate, total spend, errors and completion latency. For agents, replay complete tasks because changing one turn can change the later conversation.

Artemis estimates answer quality and cost before selecting a model. You can inspect the served model and decision trace. Its predictions still need checking against your acceptance rules. Artemis routing

6. Budget the deployed product, not just token usage

For direct NuRoute Platform API usage, current production documentation lists Artemis at $0.75 per million input tokens and $3.20 per million output tokens, with eligible cached input at $0.03. Gateway plans and applicable overages are separate. Artemis uses NuRoute's keys. Enterprise customers can use their own keys for named-model requests; standard access is Artemis-only. Billing details

A request with 2,200 uncached input tokens and 500 output tokens therefore costs $0.00325 in Artemis token charges. At 100,000 identical requests, that is $325 before gateway charges, overages and applicable taxes. Actual output lengths, reasoning usage and cache discounts change the bill.

Ship one controlled change

Move a small share of traffic, watch failures and expensive outliers, and retain a way to return to the previous model. Compare the same workload mix over time so a quieter week does not masquerade as an optimization.

Use the LLM router evaluation guide for the test design, then apply project spending controls to contain unexpected volume.