When your security badge cries wolf

The setup: exact bytes or nothing

truecopy’s watch scans ~1,800 skills across 257 plugins every run. A handful of skills legitimately trip the scanner, security products that ship attack strings as test fixtures, docs that quote “ignore previous instructions” in order to teach resisting it. Those get hand-read and recorded in an accept file: these findings, reviewed benign. The acceptance was keyed the strictest way possible, a content hash over the entire skill. Change one byte anywhere and the acceptance lapses, the skill re-flags, the badge goes red until a human re-reads it.

Exact-bytes is the right default. Drift means re-review; that’s the whole product. But it has a failure mode, and an actively-developed vendor found it for us.

The failure mode: a vendor who ships

Salesforce’s agentforce-adlc plugin ships security-testing skills whose fixtures are, by design, the kind of strings a poison scanner exists to catch. Reviewed, accepted, fine. Then Salesforce did something entirely healthy: they shipped. Two releases inside a day, a docs-only verification workflow, then a v0.9 with four new reference documents. Neither release touched a fixture. Both lapsed the whole-skill acceptance, re-flagged the skill, and turned a public badge red over unrelated documentation.

Each re-review reached the same verdict: fixture bytes identical, everything else benign docs churn. When your alarm fires on the vendor’s changelog instead of the vendor’s payload, you are training everyone watching, including yourself, that red means “probably nothing.” That is how gates get turned off. We’ve written before that a gate you don’t trust is a gate you disable; this was that lesson arriving at our own front door.

The fix: key the acceptance to what the human actually read

The insight is that a reviewed-benign acceptance is a claim about specific bytes, the files that carry the findings, not about a repository’s whole working tree. So acceptance entries can now opt into per-file granularity: the entry records a hash per finding-bearing file, and the acceptance holds only while everything else in the skill scans clean on the same detection pipeline.

The security analysis is the part worth stealing. The acceptance path only runs when the full skill has already flagged, detection always runs on everything. So at decision time there are exactly three cases:

• A reviewed fixture file changed. Its hash no longer matches, so it rejoins the scan, its content re-flags, the acceptance lapses. The guarantee that matters, changed fixture means re-review, survives intact.
• A new finding appeared anywhere else. The remainder scan catches it and the skill flags. Acceptance never silences anything nobody reviewed.
• Unrelated files churned. Docs, changelogs, version bumps, the remainder scans clean, the acceptance holds, the badge stays green. The false-alarm class is gone.

And the degenerate case fails closed: an entry that opts into per-file granularity but lists no usable files excludes nothing, so the remainder is the whole flagged skill and it stays flagged. The authoring helper enforces honesty from the other side, it attributes findings to files by scanning each file alone, then verifies its own attribution with exactly the predicate the watch will use, and refuses per-file mode if the findings can’t be pinned to individual files.

Same-day receipts

The fix merged, and the very next watch run was its trial by fire: the directory took a wave of vendor pin bumps that afternoon. The three converted acceptances held green through the churn, and the run still went red, correctly this time, on four genuinely new flags from the bump wave. Each was hand-read at the pinned bytes: AWS’s own OIDC hardening module whose refusal strings (“Refusing to send credentials over a non-HTTPS connection”) pattern-match exfiltration, two PostHog skills quoting injection strings inside their own injection defenses, and a new Anthropic plugin that mines local transcripts and never touches the network, we verified its 1,353-line miner has no network primitives at all. All four benign, all four accepted per-file, badge back to brightgreen.

One more receipt from the drill rig: while verifying, we tampered a reviewed fixture file in a local corpus copy and the acceptance lapsed exactly as designed, then forgot the tamper was still on disk and got flagged by our own watch minutes later. An alarm that catches its own operator is an alarm that works.

The watch became a dependency this week too

The same release turns the watch from a badge into something you can consume. Every run now publishes directory-manifest.json, a content hash for every skill scanned, plus the currently-flagged names, and truecopy check-manifest compares the plugin skills installed on your machine against exactly the bytes the watch vetted:

curl -fsSLo directory-manifest.json \
  https://raw.githubusercontent.com/askalf/truecopy/watch/directory-manifest.json
npx @askalf/truecopy check-manifest directory-manifest.json

Drifted from what was scanned fails. Watch-flagged fails even byte-identical, a hash match is not an endorsement. Skills from your own plugins or other marketplaces are reported, never fatal. It’s offline like everything else truecopy does: you fetch the manifest, it reads bytes. On the machine that wrote this post, 29 installed directory skills check out as exact matches.

What to take from this

Precision maintenance is part of running a detector, not a distraction from it. The whole-skill model wasn’t wrong, it was the correct strictest-possible default, and it’s still the default today. What changed is that a week of real vendor churn told us precisely where strictness stopped buying safety and started spending trust, and we re-keyed exactly that boundary.

Scope every acceptance to what a human actually reviewed. Whole-artifact hashes overclaim: they assert a review of bytes nobody read. Per-file hashes plus a clean-remainder proof state exactly what was reviewed and re-verify everything else, every run.

A public alarm is a promise about its own false-positive rate. Red has to mean look now. The day it means probably the changelog again, the watch is decorative, and the fix is never to stop looking, it’s to make the looking mean something.

The mechanism, the authoring helper, and the manifest are all shipped and public in truecopy 0.9.0 (as of 2026-07-17). The watch runs weekly; the badge is on the README.