How far is a CPU from serving production LLMs? We found the exact line.

“Own your inference” is a good slogan until someone asks the operational question: can a machine you own actually serve this, to real traffic, at production quality? Last month I measured why a CPU is slow: decode is memory-bandwidth-bound, and one law predicts every number. This month the question got sharper: given that wall, how close is “own your inference” to something you could put behind a real endpoint? So I measured that too, on the same hostile box: a 2013 Intel i7-4770, sixteen gigabytes of DDR3, no GPU.

Two questions wearing the same words

“Serve production LLMs” splits cleanly in two the moment you press on it, and the two halves have opposite answers.

One: can a CPU stand in for a frontier model (general chat, open-ended generation, interactive latency, for arbitrary users)? No, and it isn’t close. Two: can a machine you own run a production routing gateway (answer the free, checkable majority of queries on-box and escalate only the genuinely hard minority to a frontier model)? Yes, and the distance is a short engineering step. Almost every argument about “local LLMs in production” is really two people answering these two different questions and mistaking it for a disagreement.

The wall you can’t engineer around

The first answer is a hard no because it’s physics, and I want to be precise instead of hand-wavy about it. Generating a token means streaming every weight through the processor once, so decode speed is a reading speed, capped by memory bandwidth. On this box a 7-billion-parameter model decodes at 2.6 tokens per second; a modern GPU does the same model at around 140. That 34× gap is purely the ratio of memory bandwidths, not kernels, not “nobody optimized the CPU.” No quantization trick closes it, because quantization saves bytes and the gap is already all about bytes.

Prefill (reading the prompt before the first token) is worse, because it’s compute-bound rather than bandwidth-bound, and this CPU is compute-weak: a long prompt takes the better part of a minute just to read. And the quality half is its own wall: a 7B’s open-ended prose is 7B prose, and there is no free verifier for prose the way there is for arithmetic. Stack those up and “a CPU replaces your frontier model for chat” isn’t a roadmap item. It’s a category error. Anyone selling you the other story is selling.

The distance that’s just engineering

The second answer is where the interesting work is, and it turns on a reframe: you don’t serve every query on the local model. Most of what people ask an LLM is easy (a fact, a rewrite, a bit of arithmetic, a shaped word problem), and a good chunk of it is deterministically answerable with no model at all. hybrid’s whole design is a ladder ordered by how cheaply an answer can be produced and proven: exact arithmetic and unit conversions first, then word-problem shapes it can transcribe and solve in closed form, then, only for what’s left, the model, with a Python verifier re-deriving its work, and finally a frontier escalation for the genuinely hard.

The part that is already production-grade in the strong sense is the bottom of that ladder. The deterministic tiers answer in zero milliseconds, correct by construction, and (this is the operational point) infinitely concurrent, because they never touch the model or the memory bus. On a real held-out set, close to half the queries were answered there. That half is free, private, instant, and scales to any load you like. The distance to “production” is entirely about the other half, the model path, and today it got two levers and one missing tier.

Two levers that moved a 2013 box 2×

The friendly way to run a local model is Ollama; the fast way is llama.cpp’s own server, and pointing hybrid at it unlocked two wins that come straight from the physics rather than from a faster kernel. First, a prefix cache: each of hybrid’s tiers sends the same fixed instructions ahead of the query, so caching that prefill means a transcription call reads ~24 tokens instead of ~128 after the first. On a CPU, where prefill is the compute-bound wall, that’s about five seconds back per call. Second, a grammar: the model’s structured output is constrained at the sampler to exactly the shape the verifier parses. That kills the failure mode where the same 7B that answers a rate problem in 23 tokens instead writes 210 tokens of LaTeX nothing can read: measured, on the worst case, 54 seconds down to six and a half.

Same box, same weights, same benchmark, measured back to back:

                     22-query bench     26-query stress
  Ollama (standalone)     3m 23s             9m 59s
  llama.cpp transport     2m 03s             4m 23s
                         ------             ------
                          1.65×              2.3×

A 2013 desktop, no GPU, running a real router roughly twice as fast, at identical safety on the stress set. That’s not a benchmark stunt; it’s the difference between a demo and a thing you’d let a person wait on.

The tier that makes “production” an honest word

Speed isn’t the thing that separates a demo from a service, though. Concurrency is, and here the physics bites in a way most local-LLM demos quietly ignore. Because decode saturates the memory bus, two model requests at once don’t run in parallel: they queue on the same bus and each takes longer. Firing a second user’s query into a busy box doesn’t make it slow; it makes it unboundedly slow under load. A demo serves one query at a time and never notices. A service falls over.

So the missing tier isn’t more parallelism: you can’t parallelize a saturated bus. It’s the opposite: under load, shed. When the box is already busy with model work, a new model-path query escalates to the frontier now rather than joining a queue it would wait forty seconds in. That one rule turns “the box is slow under load” into “the box is a cost optimizer that gracefully overflows to the frontier.” It’s the difference between a router that’s cheap-until-it-isn’t and one you can actually point concurrent traffic at.

I built it as a shared, thread-safe count of in-flight model calls with a cap you set. Below the cap, serve locally; at it, escalate. Then I proved it on the box with the cap set to one and two real requests fired three seconds apart:

[A]  LOCAL     re-derived on the 7B          24.1s
[B]  ESCALATE  "load shed: 1 in flight,       →  frontier
               cap 1 → frontier"

The first query took the single slot and answered locally; the second, arriving while that slot was held, went straight to the frontier instead of waiting behind it. A latency budget does the same thing from the caller’s side: if the projected local time would blow your SLA, don’t start the call, escalate it. Both default off; the deterministic tiers never shed, because they cost nothing. That tier is the real boundary between “we have a router” and “you can put production traffic through it.”

What we shipped turned off (and why that’s the point)

Here’s the part I care about most, because it’s the part most write-ups leave out. Three ideas from this push were built, measured, and shipped turned off, because the measurement disqualified them. Documenting exactly why is not an admission; it’s the product. A router that hides where it breaks is worth nothing; the map of where it breaks is the whole value.

• Fusing two model calls into one. The quantitative path can make two calls (transcribe, then verify); folding them into one is an obvious ~2× win, and it worked, until it didn’t. Asking one call to transcribe and self-check degraded the transcription itself: a unit conversion the two-call path nails (five feet four inches → 162.56 cm) came back mangled as 5.33, and the verifier then cheerfully “checked” the mangled answer’s internally-consistent arithmetic. One call is only cheaper if its answers stay worth serving. It ships as an opt-in flag with that sentence attached.
• A smaller, faster model for the easy votes. A 3B is ~2.8× faster than the 7B and, we’d measured in June, safe on simple factual votes. On this month’s runtime it voted (three times, unanimously, in a second and a half) that Brazil won the 2014 World Cup. A plainly wrong fact the 7B doesn’t miss. So the split-model speedup ships opt-in, with that receipt printed next to it.
• Voting across temperatures to catch reasoning traps. The tempting idea: sample the same problem at different randomness and escalate when they disagree. Measured, the model gives the same confident-wrong transcription at every temperature: a resample of one blind spot is still one blind spot. It buys nothing, so it isn’t there.

And the single most useful finding of the day was itself a negative one. The classic reasoning traps a router is graded on (the chicken-and-a-half, the sibling riddles) turn out to be runtime-fragile: the same model and prompts flip between solved, caught, and served-wrong across llama.cpp builds and transports, at any temperature, under every prompt shape I tried. Which means a “zero wrong answers served” headline on that class of problem is a fact about one binary on one afternoon, not a property of the router. The tiers that are runtime-stable are exactly the deterministic ones: the arithmetic and the shapes, which are correct by construction on every build. That’s not a caveat buried in a footnote; it’s the reason the whole architecture leans on proof over vibes.

Two answers, kept separate

So, how far? If you mean a CPU that replaces a frontier model for general use: you can’t get there from here, and no amount of engineering changes it. That’s the 34× bandwidth wall and the un-verifiable prose, both physics. If you mean a production routing gateway that answers the free, verifiable majority on your own hardware and overflows the rest to a frontier: a day’s work got a 2013 desktop a 2× transport speedup and the one tier (load shedding) that was genuinely missing, and what remains is ordinary service hardening, not a missing capability. The expensive part, the routing decision, is done and measured.

The reason to keep the two answers separate is that the accurate one is the useful one. “Own your inference” doesn’t mean your potato out-runs a datacenter. It means the easy, private, verifiable majority of your traffic never has to leave your machine or meter a token, and the hard minority goes exactly where it should, on a budget you control, with the box shedding to it gracefully when the queue fills. That’s not a slogan. On a thirteen-year-old desktop, it’s a few hundred lines of Python you can read, and every number here is one you can rerun.