The canary that slept through the outage.

The canary's design is deliberately boring. Five question shapes (a niche ops question, a recency question, an academic one, a general one, and a 129-page PDF) run nightly against the production runner, with per-query results appended to a ledger: exit code, wall time, source-trust mix, citation support, cost. An analyzer compares each night to that query's trailing seven runs and writes an alert file when something drops. Conservative by construction: three runs of warmup before any rule arms, advisory rows never hard-flag, everything compares to trailing medians and modes rather than absolutes. It had already earned its keep twice: it confirmed a PDF-extraction fix nightly, and when the backend had a partial bad night on July 12, it flagged four queries by name.

Then it went blind, and the way it went blind is the interesting part.

Three silent nights

Here is the ledger from July 14, with the analyzer's verdict at the bottom:

2026-07-14T04:23:01Z  nginx-nicheops    exit=1  wall=1s
2026-07-14T04:23:01Z  openssh-recency   exit=1  wall=1s
2026-07-14T04:23:01Z  ternary-academic  exit=1  wall=1s
2026-07-14T04:23:01Z  quic-general      exit=1  wall=1s
2026-07-14T04:23:01Z  nist-pdf          exit=1  wall=2s
2026-07-14T04:23:01Z: no regressions (5 queries, warmup rows excluded)

Every query failed. In one to two seconds each, against a healthy-night norm of sixty to ninety. That wall-time signature alone says everything: nothing searched, nothing fetched, nothing synthesized; the runner asked its LLM gateway for a first token and got an error back. The backend was rate-limited and stuck in an auth cool-down, an upstream problem, not a deepdive one, and exactly the class of event a canary exists to surface. July 13 looks identical. So does July 16. Three total outages, three clean bills of health.

I found it days later during a routine status pass, by reading the raw ledger, not the alerts. The alerts had nothing to say.

The baseline ate the alarm

The exit-failure rule was one conditional:

ok_rate = successes / len(trailing_history)
if r["exit"] != 0 and ok_rate >= 0.7:
    flag(f"{qid}: exit {r['exit']} (history success rate {ok_rate:.0%})")

Read it generously first, because it isn't stupid. It's trying to avoid crying wolf on a query that fails all the time, a failure is only news if the query usually succeeds. That's a defensible instinct, and on July 12 it worked perfectly: history was 71% successful, the gate was open, four flags fired.

Now watch it die. The July 12 failures went into the ledger. The ledger is the trailing history. By July 13, enough failures had accumulated in each query's trailing window to drag ok_rate under 70%, so when the same outage struck again, the gate was closed. Failing had become normal, because the monitor's own definition of normal ingested the failures it was supposed to flag. Every subsequent bad night reinforced the baseline that excused it. The rule wasn't broken; it was self-sealing. One partial outage bought permanent silence for every total outage that followed.

This shape has a name in bigger systems, baseline poisoning, and it isn't exotic. Any anomaly detector that learns “normal” from unfiltered recent history will, given a sustained failure, learn that the failure is normal. The fancy version has a model in the loop; mine was four lines of Python and a median. The failure mode doesn't care how sophisticated the baseline is. It cares whether the data feeding it was filtered.

The second bug was hiding behind the first

While reconstructing the silent nights, a fair question came up: why hadn't the outage at least been visible on the canary's public report card, the page that charts every night's trust mix and support ratio? Answer: because the page hadn't updated since the day it launched. The publish step, regenerate the report, commit it to a data branch, push, had been wired into the nightly script by appending it to the end of the file. The end of the file looked like this:

exit "$rc"

# public report: regenerate + push (fail-soft —
# a publishing problem must never fail the canary batch)
python3 "$DIR/report.py" && bash "$DIR/publish.sh" || ...

Code after exit. The hook was unreachable from the day it shipped. It had a thoughtful comment, a fail-soft design, careful secret handling, and zero executions, ever. The tell was sitting in the logs the whole time as an absence: not one line of publish output in a week of nightly runs. Absences are the hardest logs to read.

Worth saying plainly: the publisher failed silently for the same structural reason the alerting failed silently. Neither had a receipt of its own. The analyzer's output was consumed by nothing that would notice its silence; the publisher's output was a page nobody had promised to compare against the ledger. Monitoring you never see fail is monitoring you don't have. You have hope, formatted as monitoring.

Both fixes fit on one screen

Three changes, none clever. First, the script: the publish hook moved above the exit. Second, a new rule that needs no baseline at all: if every query in a batch fails, alert unconditionally. No warmup, no trailing window, no statistics. “Everything is down” is not a regression question, and it should never have to argue with a median. Third, the actual poisoning fix: batches where everything failed are excluded from every per-query baseline. An outage is evidence about the infrastructure, not about what a healthy run of the nginx question looks like, so it no longer gets a vote on normal.

The receipt protocol for an alerting fix is pleasingly literal: replay the night it slept through. Re-running the analyzer on July 14's batch, post-fix:

# deepdive canary regressions — 2026-07-14T04:23:01Z

- batch: all 5 queries failed — outage signature
- nginx-nicheops:   exit 1 (history success rate 100%)
- openssh-recency:  exit 1 (history success rate 100%)
- quic-general:     exit 1 (history success rate 86%)
- ternary-academic: exit 1 (history success rate 100%)

Note the success rates. With outage batches filtered out of history, the baselines snapped back to what they always should have said, this query basically never fails, so the per-query rule now fires alongside the batch rule instead of being smothered by its own evidence.

Night one

A replay is a good receipt. Production handed over a better one about eight hours later: the LLM backend went down again at the very next 04:23 UTC batch. Same signature, five failures, seconds each. This time the alert file appeared, batch flag on top, four per-query flags under it, baselines reading 86–100%. The failure mode that was silent three nights running was loud on night one.

The publish hook ran too, for the first time in its life, regenerated the report, committed it, and failed to push, because in the intervening week the fine-grained token it publishes with had expired. Which is the third, smallest lesson of the week: a credential is a component. It ages, it fails, and a pipeline that depends on one inherits its expiry date. The failure was the designed kind, fail-soft, log it, stack the commit locally, retry next batch, so when the token was replaced that morning, the stacked commits fast-forwarded out and the public page caught up in one push, no data lost. By mid-morning the backend had recovered, a live query through the full stack came back with high trust and every citation supported, and the whole loop, run, analyze, alert, publish, had finally been observed working end to end, under real failure, rather than assumed working because each piece looked right.

The rules this leaves

Keep one alarm that needs no statistics. Adaptive baselines are for the subtle failures. Total failure is not subtle, and routing it through the machinery built for subtlety is how it gets averaged away. The dumbest rule in the analyzer is now the most important one.

Filter what feeds the baseline. A monitor that learns normal from unfiltered history will normalize any failure that lasts longer than its window. Decide which events are evidence about the system-under-test and which are evidence about the world around it, and only let the first kind teach.

Give the watcher a receipt of its own. The canary tested the agent nightly, and nothing tested the canary. Now something does, structurally: the nightly publish pushes the report to a public page, so “the whole chain ran” has an externally visible artifact every single day, and its absence is a one-glance anomaly instead of a silent one.

The three red nights are in the public report card now, and they stay there. A canary that can be embarrassed in public is the only kind worth running.