If you run a large language model behind a product, you have probably internalized an uncomfortable trade-off. The big frontier model handles the hard queries, the long documents, the gnarly reasoning chains. The small model costs five to ten times less per token and answers twice as fast, but it falls apart on exactly the requests that stress your users the most. Most teams resolve this tension by picking one model for everything, and the invoice (or the latency SLO) reflects that choice every single day.
There is a better option: route each request to the cheapest model that can handle it well. This is LLM routing, and it has quietly become a first-class architecture decision, alongside caching and batch processing. The research community has produced a full taxonomy of techniques — query classifiers, confidence-based cascades, LLM-as-judge escalation — and production frameworks now ship these strategies out of the box. This post walks through the design space with a bias toward what you can actually build: how routers decide, where cascades beat classification, and the failure modes that turn a cost optimization into a quality regression.
Why static model selection leaves money on the table
Traffic is not uniformly difficult. In a typical product, a large fraction of requests are routine: short-form chat, simple lookups, format conversions, rewrites. A small model handles these at a fraction of the cost, and often indistinguishably from the flagship. But the tail — multi-step reasoning, niche domains, ambiguous instructions — genuinely needs the larger model, and a single-model deployment either overpays on every request or underperforms on the hard ones.
Recent research on dynamic model routing frames the problem cleanly: static deployment ignores the complexity and domain of each incoming query, while routing systems adaptively select models per request. The same work makes a point that practitioners keep rediscovering: a well-designed routing system can beat the single best model in its pool, because it leverages specialized capabilities per query while maximizing efficiency. Routing is not just a cost play — done right, it improves quality too.
Measure first: instrument your traffic with a difficulty proxy (outcome quality, judge scores, escalation rates) and you may find that most of your subtasks are small-model-shaped. Only then start building.
Three dimensions that define every router
A useful way to cut through the zoo of published routers is a conceptual framework from the survey literature: every routing system can be characterized by when the decision is made, what information it uses, and how the decision is computed. Get these three axes right and most implementation choices fall out naturally.
When: pre-generation vs. post-generation
A pre-generation router inspects only the query and metadata (user tier, expected domain, time of day) before spending a token. This is the cheapest point to decide, but it is also the least informed — you are guessing how hard the request will be from its text alone.
A post-generation router runs the cheap model first, then evaluates the response — with token probabilities, confidence scores, or a verifier — before deciding whether to escalate. You pay for the small model’s attempt on every request, but you buy real evidence about quality. Production systems frequently combine both: a low-cost pre-router sets the entry point, and a post-generation verifier gates escalation.
What: the signals worth trusting
- Query features — embeddings, length, task type, detected language or domain. Cheap, available before generation, but only correlates with difficulty.
- Model metadata — capability ratings (for example Elo scores) per domain, cost per token, context limits. This is how domain-aware routers avoid sending medical or legal queries to a generalist model.
- Response signals — token-level logprobs, self-consistency across samples, self-verification verdicts, or a separate judge model’s score. These are the highest-value signals but they arrive last and cost the most to compute.
How: classifiers, matrices, bandits, and judges
The computation spans a spectrum from heuristics (“queries matching X go to model Y”) through supervised models (a BERT classifier or matrix-factorization router trained on preference data) to reinforcement-learned policies and POMDP formulations. In practice, supervised routers trained on preference pairs remain the most common production choice: training data is easy to collect and the inference cost is negligible compared to a single LLM call.
RouteLLM: the reference implementation of preference-based routing
The open-source RouteLLM framework is the clearest worked example of the pre-generation approach. The setup: given a strong model and a weak model, train a router that predicts which one would produce a better response for a given query, then send the query accordingly. The framework ships four router architectures — a similarity-weighted ranking, a matrix-factorization model, a BERT classifier, and a causal LLM router — and an evaluation harness built on benchmarks like MT Bench, MMLU, and GSM8K.
Two findings from that work carry over to almost any deployment. First, preference data beats task labels: the router is trained on observed pairwise comparisons (which response users actually preferred), aligning its objective with response quality rather than a proxy like topic classification. Second, data augmentation with an LLM judge works: synthetic preference labels generated by a judge model substantially improved router quality across all four architectures when human feedback was sparse. If you are bootstrapping a router without much production feedback data, judge-augmented training is the cheapest lever available.
The reported trade-off curve is the key deliverable: by sweeping the routing threshold, you choose a point on the quality-versus-cost frontier — for example, retaining a large majority of the strong model’s quality while invoking it for only a fraction of queries. The exact numbers depend on your model pair and traffic mix, which is precisely why the framework ships the evaluation harness. Do not copy someone else’s split; measure your own.
One operational caveat: a router adds a lightweight model call (or embedding lookup) in front of every request. That is usually negligible, but it is a new dependency on the hot path, and it must meet the same latency and availability bar as your inference endpoint.
Cascades: escalate only when the cheap answer fails verification
Routing decides up front; a cascade answers first and escalates only when needed. The canonical pattern, popularized by the FrugalGPT work, runs a chain of models from cheapest to most expensive with a scoring function (a DistilBERT-style quality estimator) deciding after each attempt whether to stop or continue. The AutoMix approach refined this with three steps: generate with a small model, self-verify the answer with few-shot prompting, then route to a larger model only if verification fails. AutoMix showed that while self-verification is unreliable for repairing wrong answers, it is a surprisingly useful signal for deciding when to escalate — a subtle but important distinction.
Cascades shine when verification is cheap and reliable: structured extraction with a schema validator, code that must pass a test suite, arithmetic with a calculator check. They are weaker on open-ended generation, where “is this response good?” is itself a hard judgment call. The research also flags a trap: models verbalizing their own confidence (“I’m 80% sure”) align poorly with actual correctness. If you want confidence-like signals, derive them from verifier behavior or aggregate statistics, not from the model’s self-report.
A pragmatic middle ground that works well in production systems combines three coordinated stages:
- a cheap pre-router using query and model metadata, subject to hard cost constraints,
- a post-generation verifier that scores the efficient model’s response, and
- an escalation policy that accepts, refines, or defers to a stronger model.
This is compositional by design — most successful systems integrate multiple paradigms rather than betting on one.
A minimal Go implementation of a threshold cascade
The following example sketches the core escalation loop: answer with the small model, score the result with a verifier, retry with the strong model if the score falls below threshold. The scoring function is stubbed — in a real system it would be a judge call, a logprob aggregate, or a schema check — but the control flow is complete.
package router
import (
"context"
"errors"
"time"
)
type Completion struct {
Text string
Confidence float64 // verifier score in [0,1]; not the model's self-report
}
type Model interface {
Name() string
Complete(ctx context.Context, prompt string) (Completion, error)
}
type Verifier interface {
Score(ctx context.Context, prompt string, c Completion) (float64, error)
}
type Cascade struct {
Primary Model // small, cheap
Fallback Model // large, expensive
Verifier Verifier
Threshold float64
Timeout time.Duration
}
var ErrAllModelsFailed = errors.New("router: all models failed")
func (c *Cascade) Complete(ctx context.Context, prompt string) (Completion, error) {
attempts := []Model{c.Primary, c.Fallback}
for i, m := range attempts {
actx, cancel := context.WithTimeout(ctx, c.Timeout)
resp, err := m.Complete(actx, prompt)
cancel()
if err != nil {
continue
}
// Last resort: return whatever we have rather than nothing.
if i == len(attempts)-1 {
return resp, nil
}
score, err := c.Verifier.Score(ctx, prompt, resp)
if err == nil && score >= c.Threshold {
return resp, nil
}
// Low score or verifier failure: escalate to the next model.
}
return Completion{}, ErrAllModelsFailed
}Two details matter more than they look. The verifier failure path escalates rather than accepts — if your scorer errors out, an aggressive threshold would silently degrade quality, so treat verifier failure as “no evidence, escalate.” And the per-attempt timeout keeps a hung small-model call from consuming the entire request budget before the strong model ever runs.
The failure modes that bite in production
- Distribution drift. A router trained on last quarter’s traffic will misroute as your product shifts. Retrain or recalibrate on fresh preference data, and monitor the routed-model distribution — a sudden shift toward the strong model is usually a quality problem in disguise, while a shift toward the weak model may signal an overfit router.
- Thresholds tuned on benchmarks. MT Bench numbers do not transfer to your support ticket stream. Sweep the threshold on your own judge-scored traffic and pick the operating point from your data.
- Verifier bottleneck. If the verifier is another LLM call, cascades can cost more than always using the strong model. Budget verifier latency and cost explicitly, and prefer structured checks where a schema or test suite can do the job deterministically.
- Observable regressions, not average quality. Routing makes per-request behavior heterogeneous: two users asking the “same” question may get visibly different quality. Track the tail of the quality distribution per routed model, not just the mean, and give hard queries a deterministic path to the strong model where the stakes are high.
Getting started without overbuilding
The migration path that works for most teams: start with static rules (task-type heuristics and cost caps), add a preference-trained router once you have accumulated judge-scored traffic, and introduce cascading only where verification is cheap and objective. Instrument every hop — which model answered, what the router scored, whether escalation happened — so the system stays diagnosable.
Routing will not replace the models in your stack, but it changes what you owe them: instead of one model carrying every request, each request gets the model it deserves — and your budget and latency stop being hostages to the hardest query in the tail.