An LLM router only pays if your traffic has the right shape. Here’s how to measure it.

We built a local-first LLM router that works. It answers the easy, checkable majority of queries on hardware we own — a chunk of them in zero milliseconds, correct by construction, with no model involved — and escalates the rest to a frontier model. It runs grammar-constrained classification in about 200 milliseconds. It sheds load gracefully instead of queueing. It is deployed.

It is also switched off, and has been since July 22.

Not because it broke. Because we finally measured the traffic we had built it for, and the traffic was the wrong shape. That measurement took an afternoon and would have saved two weeks if we had run it first. Here it is, so you can run it before you write any code.

The bet a router makes

Every LLM router is a wager: enough of my traffic is cheap enough to answer locally, and checkable enough to trust, that displacing it beats the cost of the hop.

Three things have to be true at once. Most write-ups only argue the first.

1. The work has to be small. A router saves you the tokens it does not send. If your frontier calls generate long output, a local model that answers short questions displaces almost nothing.
2. The answer has to be checkable. Serving a small model’s output is only safe if something cheaper than the model can verify it. Arithmetic has a free verifier. A constrained label set has a free verifier. Prose does not.
3. It has to repeat. Local inference pays off through caching, prefix reuse and eventually fine-tuning — all of which need the same shapes to recur.

Miss any one and the router is a hop with extra steps. We missed the first, and the third turned out to be an illusion.

Question one: how long are your outputs?

This is a single query against your own usage log, and it is the highest-leverage thing in this post. Group your frontier calls by model tier and take the mean output tokens.

Ours, over one week:

  tier          calls    mean output tokens
  cheap           327                 1,097
  mid             243                10,700
  frontier         94                18,500
                  ---
  total           664

A classification call outputs one token.

That table ended the project. We had built a router whose entire guarantee is that it verifies short, deterministic answers, and pointed it at a platform whose spend is overwhelmingly generative. Even the cheap tier — the one you would assume is full of little classify-and-extract calls — averages over a thousand output tokens, because it is writing summaries and drafting text, not picking from a list.

There was no short-label traffic to displace. The one classifier we did have fires about once a day.

If your version of this table looks like ours, stop reading and go do something else. If your cheap tier averages tens of tokens across thousands of calls, you have a real router opportunity and the rest of this is worth your time.

Question two: can something cheaper than the model check the answer?

This is the axis that separates a router you can put in production from a demo.

Our ladder is ordered by how cheaply an answer can be produced and proven: exact arithmetic and unit conversions first, then word-problem shapes solved in closed form, then the model with a verifier re-deriving its work, then escalation. The bottom of that ladder is production-grade in the strong sense — zero milliseconds, correct by construction, and effectively unlimited in concurrency, because it never touches the model or the memory bus.

For labels, the equivalent is a grammar. Declare the label set, constrain the sampler so the model can only emit one of them, sample a few times, and serve on-box only if the samples agree and the answer is provably in-set. The model cannot invent a category.

For prose there is nothing. No free verifier exists for “is this summary good.” Which means the largest category of real LLM spend is exactly the category a verifying router cannot safely serve.

Worth being precise about what the guarantee buys, because it is narrower than it sounds. A grammar guarantees the answer is in the set. It does not guarantee the answer is right. We learned that expensively, below.

Question three: does your traffic actually repeat?

We assumed yes. We had an execution log with 4,056 rows, a findings table with 963, a ticket queue with 1,100 and a signals table with 3,116. That looked like a fine-tuning corpus.

Every family failed, and the failures generalize better than the successes would have.

The execution log was mechanical. It looked strongest — 4,056 rows, 29 classes, 98% distinct openings. That 98% was timestamps and identifiers. Normalize them out and one agent’s 636 executions collapse to 13 distinct prompts. Another’s 236 collapse to five. A third’s 194 collapse to four. Trigger fires, fixed template renders, fixed agent runs. That is a lookup table wearing a corpus costume.

If you take one thing from this post: a uniqueness metric computed over text containing timestamps measures the timestamps. Normalize before you trust any diversity or dedup statistic. This is a general-purpose way to fool yourself and it is very easy to do.

The signals were not text. 3,116 rows averaging 18 characters — telemetry scalars with a value column.

The severity labels were a dozen raters in one column. 963 rows of genuinely distinct text, but one agent marks 0.7% of findings critical and another marks 33.3%. Train on the mixture and you learn to predict the average rater, who does not exist. Slice per rater and no slice is big enough.

The ticket categories were 61 free-text values with 277 nulls and 52 classes under twenty rows — docs and documentation, ops and operations and studio_ops.

The lesson is not “our data was bad.” It is that volume is not a corpus. Four tables with thousands of rows each contained, between them, almost no learnable judgment — because the judgment had already been encoded upstream, in templates and triggers.

The failure that was worth the whole exercise

Before we shut the lane down, we tuned a small classifier on six labels and then added two more. The wrong-serve rate went from 22% to 44%, and the entire delta was the two new labels.

Across twenty live requests, neither new label was served once. All eight of their requests came back on-box, wrong, at 0.90–0.98 confidence. Including this one:

“the discord bot stopped responding to slash commands” → classified monitor, confidence 0.91.

The category name is in the sentence.

Probing the real logit-read path found the mechanism. Mean posterior mass on the two unseen labels was 0.0007 on the tuned model against 0.0461 untuned; peak 0.003 against 0.431. The fine-tune had driven those labels out of the distribution. No confidence threshold recovers a label the weights do not support. A gate catches an uncertain answer. It cannot catch a confident wrong one.

And the reason generalizes further than the number. Those two labels were not semantic categories, they were ownership categories — which team owns the surface. “Add a DNS record” genuinely is build-shaped. No general small model can infer a fact about your org chart, and no grammar fixes it, because the grammar constrains the vocabulary and not the judgment.

My first instinct — just offer the caller fewer labels — was wrong and measurably so. Offering six labels does not stop misrouting; a Discord request still comes back monitor, and monitor passes validation cleanly.

What we kept

The router is still deployed and the code is still public, because the parts that hold are worth having.

• The deterministic tiers are correct by construction on every build. That is the claim I will defend without qualification, and it is deliberately weaker-sounding than “zero wrong answers served” — which was true for us on one binary on one afternoon and not the week after. Classic reasoning traps flip between solved, caught and served-wrong across builds and transports at any temperature. Any headline number on that class is a fact about a runtime, not a property of a router.
• Slot pinning. Modern inference servers cache prompts in host memory but cannot infer which requests are one family. Pinning a family to a single slot took a vote of three identical prompts from three simultaneous prefills down to one — 2.6× cold, 2.0× warm on a quiet rig.
• Small is smaller than you think. A 0.5B model matched a 3B’s classification accuracy on an identical battery at five times the decode speed. On our own hardware, classification went from about 1,400 ms to about 300 ms at no accuracy cost.
• Load shedding. On a bandwidth-bound CPU, concurrent model requests queue rather than parallelize, so under load the right move is to escalate immediately instead of making a caller wait. One second against fifty-nine, measured in production.

All of that is real engineering and none of it mattered, because the wager underneath was wrong.

Run this first

Before you build a router: four queries and an honest afternoon.

1. Mean output tokens per call, grouped by model tier. If your cheap tier is in the hundreds or thousands, a verifying router has nothing to displace.
2. What fraction of calls have a checkable answer? Arithmetic, a closed label set, structured extraction against a schema. Not “could a model check it” — could something free check it.
3. Normalize timestamps and identifiers out of your prompts, then count distinct. If it collapses by an order of magnitude, your traffic is templated and you want a cache or a lookup table, not a model.
4. For any label set, check which labels your data actually contains — and ask whether each one is inferable from the text alone, or whether it encodes a fact about your organization that no general model can know.

If those four come back the way ours did, the correct move is to not build the thing, and that is a good outcome to buy for an afternoon.

The part I would say again

“Own your inference” still means what it meant. The easy, private, verifiable majority of your traffic never has to leave your machine or meter a token. What I would add is the sentence I skipped: not every workload has that majority, and you can find out which you are before you write a line of code.

We built a very good answer to a question our own platform was not asking. The two weeks proving that are worth more than the two days building it — but only because they ended in a measurement anyone can run, instead of a router nobody needed.