Two releases before midnight. Live-test the deployed agent, not the repo.

deepdive, my local research agent, has a benchmark I trust. It caught the dead row. It gates changes to anything grounding-adjacent. It is also, structurally, a lab instrument: it runs repo code with bench config on bench questions. The thing users actually hit is a deployment: a pinned container with an env file, a real search stack, real documents on the other end of real fetches. That gap is a bug surface, and no amount of bench coverage reaches it.

So the test plan for the evening was one sentence: run real questions through the deployed runner, and grade what comes back (the source-trust mix, the citation-verification numbers, the confidence level, the cost) against what the bench says these numbers should look like.

Receipt one: the answer was right, the badge was wrong

First live question: "What changed in the most recent stable OpenSSH release, and are there any security fixes admins should prioritize?" The answer came back excellent: 18/18 citations lexically supported, the release correctly identified, reporter credits intact. Sixteen of those eighteen citations grounded to a single source: the official OpenSSH release notes.

And the trust badge said mixed. Here's the kept-source table:

primary   1.0  primary/official source (learn.microsoft.com)
unknown   0.4  unrecognized domain (openssh.org)   ← releasenotes.html
unknown   0.4  unrecognized domain (openssh.org)
unknown   0.4  unrecognized domain (en.wikiversity.org)
unknown   0.4  unrecognized domain (app.opencve.io)

The scorer awarded top authority to a marginally relevant Windows install guide, and had never heard of the OpenSSH project's own site. The answer was anchored to the primary source; the badge just couldn't see it. Worse than cosmetics: under slot competition, the ranking that decides which sources win fetch slots would rank the actual release notes below any recognized domain.

This is exactly the coverage class the dead row exposed for nginx.org and serverfault.com, canonical project sites with no docs. prefix, invisible to both the curated list and the prefix rule. The fix is the same shape as before: three receipted entries in the curated primary list (#142 → #143), a pinning test, released as v0.27.1 the same hour. The receipt is the part I care about: the identical invocation, re-run on the rebuilt runner, went from 1 primary / 4 unknown (mixed) to 5 primary / 0 unknown (high), and the two filler sources that had been kept before got outranked out of the slots entirely, because the keep-stage finally had real primaries to prefer. cdn.openbsd.org scored primary through subdomain matching on an entry added twenty minutes earlier.

Receipt two: dead code in production

Second live question was an ops one: nginx proxy buffers, the dead row's home turf. The bench says this class of question now keeps a majority of primary/reputable sources. The live runner kept 9 unknowns out of 11, SEO content farms filling slot after slot, and zero Stack Exchange sources. Not zero kept: zero candidates. The Stack Exchange keyword ladder I shipped after the dead row, live-verified at 0 → 12 results, benchmarked, tested, had never run in production at all.

The cause was one line of deployment config. The runner's search adapter was set to searxng alone, a perfectly sensible first configuration: in-network, keyless, one moving part. But the Stack Exchange adapter only participates through the multi-adapter fan-out, so in the deployed product it was dead code. The bench tested the code. Nothing tested the config.

# before
DEEPDIVE_SEARCH: searxng
# after
DEEPDIVE_SEARCH: multi:searxng,stackexchange,wikipedia

Same question, re-run: 9 unknown-tier farms → 0, stackoverflow answers in the kept set, trust mixed → high, citation support up. One env line, but no repo change would have found it, because the repo was never the thing that was wrong.

Receipt three: the extractor was innocent

Third probe: force the PDF path. The runner had just regained PDF support (its old image ran a Node version too old for the extractor, also a deployment fact, also invisible from the repo), so I asked a question whose best source only exists as a PDF: NIST SP 800-63B on password composition and rotation, with sources restricted to NIST's publication host. The runner fetched the real 129-page standard, parsed it, kept it, cited it. And answered like this:

However, the extracted text supplied from source [1] consists only
of publication metadata, author information, the abstract, and the
table of contents. ... I cannot responsibly answer the specific
question with inline citations from the material given.

That refusal is the system working: the synthesizer is forbidden from citing what it can't see, and it held the line instead of reciting NIST guidance from model memory. The starvation upstream of it was the bug. I ran the extractor directly against the same PDF bytes, deterministically, no LLM: 129 pages, 18,265 words extracted, flawless, with the famous normative language ("Other composition requirements for passwords SHALL NOT be imposed") sitting at word ~7,300. The per-source cap keeps 2,000 words. And it kept them head-first: words.slice(0, maxWords).

Head-first is a completely standard way to cap content, and for the web at large it's roughly right: articles lead with their point. Long formal documents are the worst case for it: they lead with a title page, authors, an abstract, and a table of contents, and 2,000 words is not enough to get past them. So exactly the class of source the authority ranking works hardest to keep, standards, official publications, papers, arrived at the synthesizer as front matter and nothing else.

The fix (#145 → #146, released as v0.28.0) makes the cap spend its budget where the answer lives: keep a head anchor so the model knows what document it's reading, score fixed word-windows across the whole document by content-token overlap with the question and the round's queries (the same deterministic token machinery the keyword ladder already uses), keep the best windows, fill any remaining budget in document order, and reassemble chronologically with elision markers. No new dependencies, no LLM in the loop, and with no query terms it degrades to exactly the old behavior. Ten new tests pin it, including a fixture in the NIST failure shape.

The receipt, same invocation, same PDF, same 2,000-word budget:

v0.27.1  "the extracted text ... consists only of publication
          metadata ... I cannot responsibly answer"
v0.28.0  "Minimum length: at least 15 characters [1]"
         "Verifiers and CSPs SHALL NOT impose other
          composition rules [1]"
         "SHALL NOT require subscribers to change
          passwords periodically [1]"    — 8/8 citations supported

Three cents of inference. The budget just goes where the answer lives now.

Making the receipts regenerate themselves

Everything above was me, one evening, driving queries by hand. The obvious failure mode of a good evening is that it stays an evening. So the last build of the night was a canary: five fixed questions, one per axis these bugs lived on (ops, recency, academic, general, and a long-formal-PDF question), run through the deployed runner every night, with the trust mix, citation support, confidence, cost, and latency of each appended to a ledger that survives container rebuilds. Regression rules compare each night against that query's own trailing history, medians and modes, not absolutes, because run-to-run variance is real, and anything that trips writes an alert. The PDF question is marked advisory: it exists to watch the #145 fix stay fixed, every night, automatically.

The dead-row post ended on: compute the number and let it disagree with you out loud. This is that stance pointed at the deployment instead of the lab, and put on a timer.

What I didn't fix that night

Live testing also surfaced things I did not fix that night, and they're filed with receipts rather than quietly forgotten. Restricting sources to one host turns out to be a coin flip: the planner rewrites your question into semantic sub-queries, and whether any of them happens to surface the allowed domain decides between a great answer and an empty run: two of three identical invocations died while a raw search probe returned the target at rank 1 (#147). And authority ranking stops discriminating exactly when every candidate is trustworthy: an all-arxiv answer set happily kept a December 2010 paper in a question about 2024-era ternary LLMs, and wikipedia took four of seven slots on an ops question, because nothing in the ranking speaks to topical relevance once the trust tier is uniform (#148). Both need bench evidence before anything ships; the dead row taught me what guessing from a symptom costs.

And to be clear about what this does not say: none of it retires the bench. The bench catches regressions cheaply, repeatably, before merge; it gates both of these releases. What it cannot do is see the deployment: the env line, the Node version, the real backend's real results, the real document's real shape. Each of the three bugs lived in a place the bench structurally cannot reach. You want both instruments, and a canary bridging them.

The rule this leaves

The repo is not the product. The bench was green and the product was still wrong in three places: a curated list missing the one project that answers the question, a config line that quietly benched a shipped feature, and a cap whose worst case is precisely the documents worth capping. Test the deployed thing with real questions, grade it with the same numbers the lab uses, fix from evidence, and re-run the identical invocation as the receipt. Then put the receipts on a timer, because the next regression won't schedule itself for an evening you're watching.