The receipt that failed. 1/3 → 3/6 → 3/3 across three releases.

Quick recap of the two open items. #147: restricting deepdive's sources to one host (--allow-domain=nvlpubs.nist.gov, say) was a coin flip, the planner rewrites your question into semantic sub-queries, correctly drops incidental details like a host name, and whether any query happens to surface the allowed domain decides between a cited answer and an empty run. Two of three identical invocations died while a raw search probe found the target at rank 1. #148: authority ranking stops discriminating exactly when every candidate is trustworthy, an all-arxiv answer set kept a December 2010 paper in a question about 2024-era ternary LLMs, and wikipedia took four of seven slots on an ops question, because nothing in the ranking speaks to topical fit once the trust tier is uniform.

Measure first, then fix

#148 went first, and the instrument went in before the fix did. The benchmark's authority boards measure trust composition, how many primary and reputable sources got kept, but a kept set can be all-primary and still miss the one source that actually answers the question. So each bench question now declares its canonical source (nginx.org for the nginx question, an RFC for HTTP/3, arxiv for the retrieval-augmented-generation question), and every board reports a ✓/✗ column: did the kept set contain it? Reported, never gated, same stance as the authority column, because a run can answer well from mirrors.

Then two additive ranking changes: inside equal authority tiers, candidates now order by term overlap between their title-plus-snippet and the question, the same deterministic token machinery the keyword ladder already uses, no model in the loop, and the keep stage caps any one registrable domain at two slots unless slots would otherwise go empty. Authority stays the primary key; relevance never promotes a lower tier; turning ranking off remains a strict no-op, which is what makes the before/after measurable at all. The boards: the academic question went 8 primary / 3 unknown → 11 primary / 0 unknown, the ops question 2P 2R 4U → 6P 5R 1U, and the canonical column read ✓ on all four runs. Shipped as v0.29.0 alongside the #147 work.

The fix that fired and missed

The #147 fix had two halves. The recovery half: when a round yields zero candidates under an allow list, retry the round's queries once with the allowed host appended, a relevance hint, with the filter still enforcing. The honesty half: stop printing returned 0 usable results when search actually returned plenty and the filter dropped them all, and stop running the fallback pass when the fallback's entire corpus (wikipedia's, say) structurally cannot satisfy the allow list, skip it and say so.

Release out, runner rebuilt, receipt protocol run: the NIST question, three times, on the deployed runner. 1/3, identical to baseline. And the two failing runs printed this:

deepdive: no sources gathered: 10 queries via
  multi(searxng,stackexchange,wikipedia) found 16 result(s),
  but the domain filter dropped every one
  fallback (wikipedia) skipped: its results cannot
  satisfy --allow-domain

Read it closely, because the release is diagnosing itself. Ten queries, five planned, five hinted: the retry fired on every failing run. Sixteen results found, every one dropped: search was healthy and the allowed host never entered the candidate pool. The hint was working as built and not working as intended. A bare host token appended to a query is a suggestion, and the search aggregator's ranking was free to ignore it, engines tokenize nvlpubs.nist.gov as URL-ish noise, where the probe that motivated the issue had used the bare label nvlpubs, which matches page text. Worse, two of the fan-out's three backends couldn't respond to any hint at all: Stack Exchange's API is literal-match over one site's corpus, and wikipedia only ever serves wikipedia. The one steerable backend was getting the weakest possible steering.

An hour-old error message explaining the insufficiency of the fix it shipped with is not a bad morning. It's the entire point of shipping diagnostics and receipts together: the failure was visible in one read, with the root cause sitting in the message.

A hint is not a directive

The replacement (#157, shipped as v0.30.0) stops asking nicely. Search adapters that pass engine query syntax through (searxng, DuckDuckGo, Brave) now expose a hinted-search capability: they receive the raw query plus the structured host list and send site:nvlpubs.nist.gov themselves. site: is an operator the engine enforces, not a token it may weigh. The multi-adapter fan-out dispatches per backend: engine-syntax backends run their own form; fixed-domain backends are skipped outright, the plain pass already asked them, and re-walking a keyword ladder over a hinted string a literal-match API can never satisfy was pure waste; open-web backends without query syntax get an improved token form that includes the bare label. And a primary adapter that structurally can't serve the allowed host is never hinted at all, you can't steer a corpus into containing something.

Same protocol, rebuilt runner:

protocol: same question x3, --allow-domain=nvlpubs.nist.gov

v0.27.1   1/3   the original coin flip
v0.29.0   3/6   two runs: 1/3, 2/3 - hint fired, couldn't steer
v0.30.0   3/3   exit 0 * trust high * ~$0.05/run
                citations supported: 19/19, 14/14, 14/15

Two of the three v0.30.0 runs also kept two allowed-host sources where every earlier success had scraped by on one, the directive doesn't just save the failing runs, it deepens the succeeding ones.

The accidental control run

One more row in that table deserves its footnote: v0.29.0 shows six runs because the rebuild script raced the release once, the runner rebuild ran before the new version existed on npm, the container build failed loudly, and the old container stayed up. The receipt script ran anyway and did the one thing that mattered: it printed which version it was measuring before measuring it. Result, a free second baseline run (2/3 this time, a coin flip landing the other way, which is what coin flips do) and zero confusion about what the numbers meant. Instruments that report their own configuration are cheap to build and they pay for themselves the first time reality gets out of order.

What the failed receipt actually bought

It's worth being precise about what happened across these three releases, because the sequence is the method. The honesty layer shipped with the first fix, so when the fix missed, the system said so in its own exit message instead of leaving a mystery. The receipt protocol was defined before the fix, same question, three times, on the deployed thing, so "did it work" was never a matter of opinion. And the bench instrument for #148 went in before the ranking change it was built to judge. None of this is extra process; each piece is a sentence long. The discipline is only this: every fix ships alongside the instrument that's allowed to call it insufficient.

The nightly canary from the last post now runs the deployed v0.30.0 across all five question shapes, so the 3/3 doesn't have to stay an afternoon's claim, it gets re-earned while I sleep.

The rule this leaves

A receipt that can't fail is a press release. The protocol here was allowed to come back 1/3, did, and pointed at its own root cause, that's the difference between shipping a fix and shipping a claim. And when a filter has to hold the line, prefer directives to suggestions: the enforcement layer was never the problem, but recovery built on hoping a ranking notices your token loses to recovery built on an operator the engine must obey.