The agent-firewall leaderboard I refused to build.

Every agent firewall has the same demo. Someone types rm -rf /, the tool flashes red, the room nods. I've shipped that demo myself. It proves almost nothing, a single blocked command is a screenshot, not a measurement, and the interesting question was never “does it catch the one attack you chose to show me.”

The question is: across a corpus of real attacks and real benign work, what fraction does it catch, what fraction of your legitimate actions does it break, and does it answer the same way twice? Recall, precision, and determinism, together, on labeled data. Almost nobody in “agent security” publishes that, because the demo is easier and the honest number is scarier. So I built the thing that produces it. It's called arena, it lives in the warden repo, and the first thing it taught me was that the benchmark I wanted to build was a lie.

Two numbers, and each one lies alone

Start with why one number is never enough. I put two deliberately stupid firewalls in the benchmark to make the point unarguable.

The first blocks everything. Perfect recall, it catches 100% of attacks, because it catches 100% of anything. It also blocks every file you write and every command you run, so it's useless, and its recall score is a perfect 100%. The second blocks nothing. Perfect precision, zero false positives, because it never says no. Also useless; also a perfect score on its chosen metric.

That's the whole trap. Any vendor can hand you a 100% on recall or a 100% on precision by being maximally paranoid or maximally lazy, and either number, alone, is marketing. So arena always shows both, side by side, with those two anchors sitting right there in the table as a reminder:

firewall              recall   precision   deterministic
warden                96.4%    100%        yes
regex deny-list       15.8%     98.4%      yes
allow-all (null)       0%      100%        yes
block-all (paranoid) 100%        0%        yes

block-all is the ceiling that proves recall alone is gameable. allow-all is the floor that proves precision alone is gameable. A useful firewall is the one that keeps both high, and the naive regex deny-list in the third row, the “I'll just write some patterns” approach, is there to show how far that falls short the moment the corpus stops being canonical.

One pipe, so a Python firewall and a JavaScript one get the identical test

The mechanics matter, because a benchmark you can't reproduce is just my word. Every firewall plugs in as an adapter: any executable that reads samples as JSON lines on stdin and writes verdicts as JSON lines on stdout. A Node firewall and a Python firewall are scored by the exact same harness, over the exact same 234-sample corpus, with no shared library between them. The runner strips the label off every sample before handing it over, so an adapter physically cannot read the answer key it's being graded against.

And there's a column most benchmarks don't have: determinism. The runner scores every tool twice and checks the verdicts are byte-identical. A deterministic firewall is reproducible and auditable, you can put it in front of an auditor and it will always explain itself the same way. A guard with an LLM in the decision path can't make that claim; it answers differently on Tuesday. This is the same discipline I wrote about grading deepdive with a lexical verifier instead of a second model: a benchmark that uses an LLM to judge is a benchmark that will happily agree with whatever you just told the model to do. If the grader can be sweet-talked, the score is fiction.

The category error I almost shipped

Here's where I nearly embarrassed myself. The obvious next move, once the harness worked, was to install the famous tools, run them through it, and publish the bar chart. I started to. Then I actually read what those tools are.

Meta's LlamaFirewall is an LLM-I/O guardrail. Its PromptGuard scanner is a classifier for prompt-injection and jailbreak text. Point it at rm -rf / and it correctly shrugs, that string isn't a jailbreak prompt, it's a shell command, and screening shell commands is simply not the job it was built for. Deno's Claw Patrol is something else again: a gateway that sits on the network wire, parses protocol traffic, SQL verbs, Kubernetes calls, HTTP paths, gates it against rules, and swaps real credentials in mid-flight so the agent never holds them. It never sees a shell command as text at all. It operates a layer below the one my firewall lives on.

Neither of those tools is worse than mine. They're not on the same axis as mine. warden classifies an agent's tool call before it runs; LlamaFirewall screens the model's input and output; Claw Patrol gates protocol traffic on the wire. Three different jobs. A bar chart that stacks all three on one axis and crowns my tool the winner isn't a benchmark, it's a category error in a lab coat, and any engineer who actually knows these tools would see through it in about four seconds. It would cost me the exact credibility the benchmark exists to earn.

So I didn't build the leaderboard. I built a map:

tool             what it actually guards                fair on this corpus?
warden           each agent tool call, before it runs   yes — its axis
regex deny-list  tool calls (naively)                    yes — the floor
LlamaFirewall    model input/output (injection text)    partial — injection only
Claw Patrol      protocol traffic on the wire           no — a different layer

warden and the deny-list are the same-axis rows. LlamaFirewall competes only on the injection slice of the corpus, so its adapter is in the repo scoring exactly that slice, honestly, expected to pass shell commands because that isn't what it screens. Claw Patrol isn't a row at all; it's a different layer, and I've written separately about running the two of them together rather than against each other. The deliverable was never a ranking. It was a map of who guards what, which is the thing a bar chart actively hides.

The number on my own firewall, with the misses left in

So what does warden score, run through the same cold pipe every other tool uses, not grading itself, but answered over stdin by a harness that only knows how to strip labels and count? On the 234-sample corpus (19 attack families, from destructive filesystem and reverse shells to cloud-metadata SSRF and poisoned skills): 96% recall, 100% precision, deterministic. Zero false positives across 62 benign samples, and identical verdicts on both passes.

Now the asterisk, because leaving it out is how benchmarks start lying. I wrote the corpus. warden topping a corpus written by warden's author is the expected result, not proof of superiority, and I'd distrust anyone who presented it as the latter. What keeps it honest is structural: the labels are assigned by what a correct firewall should do, not by what warden happens to do; and the misses are right there in the output, not hidden. Five attacks slip the deterministic gate, pure shell metaprogramming like X=rm; $X -rf /, ${IFS}-padding, a hex string piped into a shell, and one risky action gets under-gated. Those are the same five I've written about catching with an LLM judge tier, deliberately kept out of the deterministic hot path.

A benchmark like this earns neutrality the only way one can: more tools on it, and corpus samples contributed by people who don't own the tool that's winning. Both are open pull requests. Until then it's a scoreboard with one serious entrant and its own misses on display, which is still worth more than a bar chart I'd have to walk back.

Why this is the whole point

A while ago I wrote about the four times my own benchmark lied to me about my own model, a throttled backend, noise inverting on a stable run, a timeout that wasn't watching the thing that hung. That was the discipline pointed inward: audit the substrate before you trust the number. This is the same discipline pointed outward, at the benchmarks you read comparing other people's tools.

So the next time you see an agent-security bar chart where one product wins clean, ask the three questions this whole thing is built around: which axis is it measuring, were the losers even playing the same game, and would the author show you their own misses? If the answer to any of those is uncomfortable, you're looking at marketing, not measurement.

This is what Own Your Stack means one layer under the slogan: own your evaluation, not just your infrastructure. If you can't see which axis a comparison is measuring, you don't own the conclusion, you're renting someone's framing and calling it a fact. I'd rather hand you a map I can defend than a leaderboard I'd have to take back. The corpus, the harness, and my firewall's five misses are all in the open. Bring your own firewall and run it.