I put two rival firewalls in my benchmark. Here's the honest read.

A few days ago I wrote about the agent-firewall leaderboard I refused to build: why a bar chart where my own tool wins is a category error, and why I shipped a benchmark that draws a map of who guards what instead. That post ended with a challenge I'd left mostly unanswered: bring your own firewall and run it. It's a cheap thing to say when the only serious entrant on your own scoreboard is you.

So I ran two. Not the LLM-guardrail tools from last time that guard a different layer, two firewalls that actually sit where mine sits, or close enough to make the comparison honest: Pipelock, an open-source agent egress firewall, and AEGIS, a pre-execution tool-call firewall. Both real, both open source, both with a per-call decision surface I could drive fairly. And this time the scores don't come off my laptop.

The part that makes this reproducible instead of “trust me”

A benchmark is only as neutral as the host it runs on. It's trivially easy, even unintentionally, to tune your own machine until your own tool looks good. So the arena's competitor runs happen in continuous integration, on a GitHub-hosted runner, which is about as close to a disinterested third party as I can get without hiring one.

Each competitor is pinned so the run is repeatable to the byte: Pipelock is downloaded as a specific released binary and checked against a hardcoded SHA-256 before it's allowed to run; AEGIS, which ships no prebuilt image, is built from a pinned git commit into its own gateway container. The workflow starts each tool, waits for it to answer a health check, feeds it the corpus through the same stdin/stdout pipe warden uses, and then asserts that each competitor actually produced verdicts, because a tool that silently failed to start would otherwise score a fake zero, and a fake zero flatters me exactly as dishonestly as a rigged win. Anyone can re-run the whole thing by re-running the workflow. The numbers below regenerated on that neutral host, not on mine.

Pipelock: strong where its axis is, quiet where it isn't

Pipelock's primary job is the egress wire: it proxies an agent's outbound traffic (HTTP, MCP, WebSocket) and scans it for credential exfiltration, SSRF, and injection. It also exposes an on-demand scan API for per-call verdicts, which its own docs recommend as the “forward each tool call through before execution” integration. That's the surface I drove, and I drove it generously: every sample gets checked as a tool call and as raw command text and as skill text, and a block from any of those counts as a block. Composition can only ever raise its score, never lower it.

On the 245-sample tool-call corpus, out of the box:

firewall              recall (block)   recall (+gate)   precision
warden                96.5%            96.5%            100%
Pipelock v3.0.0        7.0%             7.0%             95.5%

Read that through the map, not as a knockout. Pipelock's catches land exactly where its design says they should: it flags the credential-exfiltration and poisoned-skill-injection samples, the ones where a secret or an injection string is physically present in the call, and it passes the shell-semantics families, because rm -rf / carries no credential and no URL for a wire-scanner to catch. Its enforcement for that class of attack lives at the network layer, where the destructive command either does or doesn't try to phone home. On a corpus of tool-call strings, that layer simply isn't in frame. An operator who wrote a few of Pipelock's tool-policy rules would move that number; I ran it with no custom rules, and I labeled the row that way. It's not a low score. It's a differently-shaped tool photographed from the one angle this corpus can see.

AEGIS: the first competitor that answers “ask a human”

AEGIS is the interesting one, because it does something warden and Pipelock don't: it has three answers instead of two. Allow, block, and pending, escalate to a human, hold the call, wait for a click. Run in its blocking mode, a risky action doesn't get hard-blocked; it gets parked for review. Which means the column almost every other benchmark ignores, the difference between “blocked” and “stopped and asked,” is the entire story for a tool like this.

The arena has always scored those separately: recall (block) is what a tool hard-stops, recall (+gate) is what it either stops or escalates, both of which prevent silent execution. Force AEGIS into a block-or-allow world and you'd score it as missing nearly everything, which would be a lie about a tool whose whole thesis is “a human decides the hard ones.” So I mapped its pending verdict to the gate column and read it there:

firewall              recall (block)   recall (+gate)   precision
warden                96.5%            96.5%            100%
AEGIS v0.2.0           4.9%            59.4%            100%

Read the block column alone and you'd write AEGIS off: 4.9%, catches almost nothing. That reading is exactly the mistake. Look one column right. When you count the calls it escalated, parked for a human instead of letting them run silently, it prevents silent execution on 59.4% of the attacks, at 100% precision: not one benign command hard-blocked. It's a fundamentally different posture from warden's. warden decides; AEGIS, in this mode, defers the hard calls to a person. Both stop the attack from running unattended, which is what the +gate column measures, and neither number is visible if your benchmark only knows how to say block-or-allow.

The posture has a price, and the benchmark shows that too, in a column vendors never print: friction. Twelve benign actions, legitimate work, also got sent for human review. That's the tax of “when unsure, ask”: a human has to clear those twelve before the agent proceeds. Whether that trade is worth it is a judgment about your team and your blast radius, not a number I can rank for you, which is the whole reason I'd rather hand you the two columns and the friction count than a single score with the tradeoff hidden inside it.

Two caveats worth stating plainly, both noted in the adapter itself so nobody has to take my word for it. AEGIS has a behavioral-anomaly layer that learns each agent's normal pattern over time, and a one-shot corpus, where every sample is a stranger, never lets that layer warm up. So what I'm scoring is its static policy-and-classifier layer, not its learned one; the tool has a dimension this benchmark structurally can't test, and pretending otherwise would be its own kind of dishonesty. And AEGIS screens the action, not poisoned-skill text, so the injection-via-skill samples that carry no dangerous argument pass it, again, its axis, honestly drawn.

The asterisk I'm still wearing

The same one as last time, and it doesn't go away because I added competitors: I wrote the corpus. warden topping a corpus by warden's author is the expected result, not a proof of superiority, and the fix isn't for me to insist I was fair, it's structural. The labels are set by what a correct firewall should do, not by what warden does. The competitors are driven through their own documented interfaces, in their out-of-box posture, with the generous reading, not a strawman one. Their misses are the corpus's blind spots as much as the tools', which is exactly why the write-up of a rival's 7% is three paragraphs about what the corpus can't see, not a touchdown dance.

What would make this genuinely neutral is the thing I keep asking for: a corpus contributed by someone who doesn't own any tool on the board. The pipe is built for it: any labeled JSON-lines file with an action and an expected verdict scores every tool unchanged, no warden code in the loop. Until then this is an open, reproducible scoreboard with three serious entrants, each read on its own axis, misses and all. That's still worth more than a bar chart I'd have to walk back.

Why I'd rather publish this than a win

I could have posted the first table and stopped. “Rival agent firewall scores 7% against warden's 96%” is a tweet that would do numbers. It would also be the exact category error I spent a whole post swearing off: a real measurement bent into a fake ranking, and any engineer who's actually used Pipelock would see the bend instantly and trust nothing else I say.

This is what Own Your Stack means below the slogan: own your evaluation, not just your infrastructure. A comparison you can't reproduce, on a host you don't trust, reading one axis as if it were the only axis, that's a conclusion you're renting from whoever drew the chart. I'd rather hand you a map I can defend, with two competitors' real numbers on it and the caveats printed next to them, than a leaderboard I'd have to take back. The corpus, the pipe, the two competitor adapters, and the CI that runs them on a neutral host are all in the open. Bring a third firewall. Better yet, bring a corpus I didn't write.