Writing

Research & build notes

Field reports from an independent studio that runs on its own agent fleet. AI infrastructure, the wire-level reality of building on someone else's models, and notes from shipping real software end-to-end. No think-pieces — just what we learned doing the work.

• August 15, 2026 Release notes 4 min read

  What shipped — Aug 1–8, 2026

  172 merged PRs across 35 repos — the busiest week we’ve logged. dario’s health check finally proves it can serve, not just that it’s configured — and building that probe surfaced two real bugs: an unkeyed proxy leaking OAuth internals on health responses, and a request-queue slot leak. Plus security patches across five repos (fast-uri, hono, a poison-detector bypass in strongroom’s MCP gate), truecopy 0.10.3, redstamp 0.7.5, and five tools onboarded to the Glama registry.

• August 10, 2026 AI infrastructure 10 min read

  An LLM router only pays if your traffic has the right shape. Here’s how to measure it.

  A router only pays if your traffic is short, checkable and repetitive. Ours was long, generative and rare: even the cheap tier averaged 1,097 output tokens against a classifier’s one, so there was nothing to displace. The execution log that looked like a fine-tuning corpus was 98% timestamps — 636 runs collapsed to 13 distinct prompts. Four queries that tell you whether to build the thing at all.

• August 1, 2026 Agent security 7 min read

  npm called my firewall malware. So does my firewall.

  npm’s content scanner rejects redstamp because its detection corpus is, byte for byte, a file full of attack strings — and it isn’t exactly wrong. A benign canary published fine through the identical pipeline seconds later, which is how you prove a block is keyed on content rather than credentials. Then I went to write up all the times my own firewall did the same thing to me, measured them first, and watched most of them fail to reproduce — including the one I was loudest about, which turned out to be the rule working correctly. On the false-positive class that is real, why the obvious workaround is detection evasion lifted from the corpus we ship, and why a signed release tarball you verify before installing is the better answer anyway.

• August 1, 2026 Release notes 6 min read

  What shipped — Jul 5–Aug 1, 2026

  Four weeks in one catch-up post, every figure pulled from a changelog, release, or public ledger. warden became redstamp and keeper became strongroom. dario shipped v5.0 pool-as-primitive and dropped shim mode, then added an upstream wire-format drift watch. redstamp closed seventeen classifier bypasses across three releases in a single day — a white-box sweep, a boot-persistence find, and a live red-team of the LLM judge tier — with precision never dropping below 100%. And plumbline joined the stack as the thirteenth tool.

• July 26, 2026 AI engineering 9 min read

  Two rebake loops and the bug that was my own laptop.

  Four findings in one day on dario’s template pipeline. Three of them weren’t what the first report said: a foreign capture that was inherited context, upstream drift that was my own bake host, an unstable prompt that was git noise — and the marker that would have failed every build if I hadn’t counted it first. What measuring before writing code changed each time.

• July 18, 2026 AI engineering 7 min read

  The canary that slept through the outage.

  The nightly canary that tests our research agent alerted correctly on a partial outage — then said “no regressions” through three total outages in a row. The alerting rule was statistically reasonable and structurally wrong: its baseline ingested the failures it was supposed to flag, and one bad night bought silence for every night after. Behind that bug hid a second — the job publishing the public report card sat below an exit line and had never run. Both fixes fit on a screen; production receipted them on night one by failing again, this time loudly. On baseline poisoning, alarms that need no statistics, and giving the watcher a receipt of its own.

• July 17, 2026 Agent security 7 min read

  When your security badge cries wolf.

  Our standing watch over the official Claude Code plugin directory went red three times in one week — every time a false alarm caused by our own acceptance model, not an attack. A vendor who ships docs shouldn’t lapse a security review that never covered the docs. So we re-keyed reviewed-benign acceptances from whole-skill hashes to the exact finding-bearing files a human read, with everything else re-scanned every run — and watched the fix hold green through a real vendor bump wave the same day, while four genuinely new flags still fired correctly. Plus: the watch is now consumable — truecopy check-manifest verifies the plugin skills installed on your machine against the exact bytes the watch vetted.

• July 16, 2026 Agent security 8 min read

  After zero raw credentials: hardening the vault the fleet stands on.

  The migration post ended with agents holding zero raw credentials — which is the moment the vault stops being something you evaluate and becomes something you stand on. So we attacked it. A coverage-guided fuzzer found two real bugs in the broker’s path-allowlist parser: a climbing .. that survived canonicalization, and a ? in an endpoint glob that turned a valid lease into one that 502s on every request. Grant ceilings became vault policy instead of caller discipline — over-cap asks are rejected and audited, never silently clamped, and a NaN TTL no longer mints a lease that never expires. keeper became strongroom and ships to npm via trusted publishing with no long-lived token anywhere in the release path. And the fleet swapped its last git pin for the attested npm artifact — because dogfooding doesn’t count until you run what you ship.

• July 11, 2026 AI engineering 7 min read

  The receipt that failed. 1/3 → 3/6 → 3/3 across three releases.

  The allow-domain fix shipped with a production receipt protocol: the same question, three times, on the deployed runner. It came back 1/3 — unchanged from baseline — and the diagnostics that shipped in the same release explained exactly why: the retry fired, and a bare host token can’t steer an aggregator’s ranking. The next release replaced the hint with a site: directive and taught the fan-out to skip backends that structurally can’t respond. Same protocol: 3/3. A receipt that can’t fail is a press release.

• July 11, 2026 AI infrastructure 8 min read

  The model is a witness: live-testing a fail-closed PII gateway.

  cordon sits between an application and an LLM and promises raw PII never reaches the model: placeholders go up, real values come back restored, and a detection failure blocks the request instead of forwarding it. Its hardening release had 150 green tests — and we still refused to deploy until we'd booted real processes and watched their exit codes, attacked the running gateway over live HTTP with adversarial placeholder-shaped text, killed it mid-suite to prove tenant policy survives a restart, and used strip mode to make the model itself testify: it echoed [EMAIL] back because the placeholder is all it ever received. Then the deploy pipeline earned the same treatment — and paid for itself by catching an expired registry credential.

• July 11, 2026 AI infrastructure 7 min read

  dario 5.0: a major version earned by subtraction.

  Most software earns a major version by adding. dario earned its 5.0 by removing — two breaking deletions that leave the codebase smaller and the one job it does easier to audit. The multi-account pool becomes the one credential model, so a plain dario login is now just a pool of one and the request handler stops forking two ways on every call. And shim mode — a half-working transport deprecated since 4.2 — is deleted outright, about a thousand lines of code and test gone. The discipline that makes deleting a working path safe: prove the path you're keeping first. Zero-effort migration for solo users.

• July 10, 2026 AI infrastructure 9 min read

  How far is a CPU from serving production LLMs? We found the exact line.

  The question splits in two, with opposite answers. A GPU-less box replacing a frontier model for general chat is a physics wall — a 34× memory-bandwidth gap and un-verifiable prose, unfixable in software. But a production routing gateway that answers the free, verifiable majority on-box and sheds the rest to a frontier is one honest engineering step away. A day of measured receipts on a 2013 desktop: a 1.65–2.3× transport win from a prefix cache and grammar-locked output, a live load-shedding tier that escalates under load instead of queueing, and — the part worth reading — three ideas we built, measured, and shipped turned off because the measurement disqualified them.

• July 10, 2026 Agent security 7 min read

  One-time audits rot. We turned ours into a standing watch.

  Both marketplace audits ended with the same caveat: a scan is a snapshot, and a skill can be updated tomorrow. We retired the caveat. truecopy now re-audits the entire official Claude Code plugin directory daily — every catalog plugin, including the external vendor plugins re-fetched at their catalog-pinned commits — and publishes the verdict to a public badge and report. The August 7, 2026 run: 280 plugins · 1,966 skills, zero poisoned, reviewed-benign findings accepted byte-exactly so any drift re-flags. And the honest part: on its first full run the watch caught two bugs in its own harness — which is exactly the property a tripwire is for.

• July 9, 2026 Agent security 7 min read

  I put two rival firewalls in my benchmark. Here’s the honest read.

  Last time I refused to build the leaderboard and drew a map instead. A map is worth nothing until someone else’s tool is on it — so I put two real competitors through the same cold pipe, live in CI, on a host neither of us controls. Pipelock, an egress-DLP gateway, scores strong on its own axis and quiet off it. AEGIS is the first three-tier firewall on the board: near-zero hard-block by design, but it escalates 59% of attacks to a human at 100% precision. Each read on its own axis, with the friction cost printed next to it — not force-ranked into a bar chart.

• July 5, 2026 Release notes 4 min read

  What shipped — Jun 28–Jul 5, 2026

  153 PRs merged across the 13 public repos that moved this week, pulled straight from GitHub — no estimates. dario shipped 33 releases through its self-healing pipeline (Sonnet 5 day-one, Fable 5 back). warden 0.3.0 added four framework adapters and a MITRE-mapped benchmark. canon’s marketplace audits reached 66,541 ClawHub skills (68,560 across all marketplaces) and became a CI gate. keeper migrated our own fleet off 119 raw keys. hands learned to turn repeated tasks into $0 macros.

• July 4, 2026 Agent security 10 min read

  We scanned the marketplace that started the poisoned-skills panic. It’s clean.

  The obvious rejoinder to auditing the well-run marketplace: scan the wild one. So we did — ClawHub, OpenClaw’s open registry, the marketplace whose poisoned-skills incident started this category. All 66,541 skills, poison-scanned with truecopy and cross-checked against ClawHub’s own scanner. Zero confirmed malicious. The output is the map of why 813 deterministic alarms are all benign — and why two scanners agreeing isn’t proof — plus the single most-installed skill in the registry, which runs clean but ships obfuscated code you can’t read.

• July 3, 2026 Agent security 5 min read

  The category every agent roundup skipped

  The 2026 open-source agent roundups rank what agents can do; none rank what agents are allowed to do. An open-source agent means you can read its code — not that you know which shell command it ran at 3 a.m., which marketplace skill it just installed, or where your keys went. The governance trilogy we built because our own fleet forced the question: redstamp the firewall, truecopy the supply-chain scanner, strongroom the lease broker — plus the receipts, including a 2,019-skill marketplace audit.

• July 3, 2026 Agent security 5 min read

  MCP tools should return leases, not keys

  MCP made tool results the lingua franca of agent context — which is exactly why a credential must never appear in one. A key in the server's env is one compromise from gone; a key in a tool result is already in the model's history, logs, and traces. The pattern we run instead: MCP servers spawned keyless, holding scoped leases redeemed at egress, and credential-granting tools that return capabilities. The receipt is checked in — every JSON-RPC frame on the wire captured, the secret in zero of them.

• July 3, 2026 Agent security 9 min read

  We scanned 2,019 Claude Code skills. The hard part wasn't finding poison.

  A skill is prose that steers an agent holding your credentials, and marketplaces distribute them by the thousand. So we audited the supply chain: every skill in the official Claude Code marketplace plus nine community ones — 2,019 skills across 177 vendor repos, fetched at pinned commits and poison-scanned. None were malicious. The engineering was in earning that answer: getting a deterministic scanner from a one-in-ten false-alarm rate down to 0.6%, every fix measured against the corpus and pinned with a test.

• July 3, 2026 Agent security 6 min read

  The agent-firewall leaderboard I refused to build

  The easy benchmark is a bar chart where my firewall wins; the honest one is a map, because half the tools I'd be "beating" aren't on the same axis. A neutral agent-firewall benchmark: one labeled corpus, one language-agnostic pipe, recall and precision and determinism together, with the two gameable-alone anchors (block-all, allow-all) sitting in the table to prove why. LlamaFirewall screens injection text, Claw Patrol gates the network wire — different layers, not a leaderboard — so I built the map instead. My own firewall scored through the same cold pipe, sanity anchors included — and when the corpus caught five misses, they were documented, not hidden. The current 298-sample arena run: 100% recall, 100% precision, deterministic.

• July 3, 2026 AI engineering 8 min read

  Two releases before midnight: live-test the deployed agent, not the repo

  The bench was green and the product was wrong in three places the bench can't reach: a scorer that had never heard of openssh.org, a config line that quietly benched a shipped feature, and a word cap that reduced a 129-page NIST standard to its title page — the synth honestly refused to answer from it. Live-tested, root-caused, two npm releases same night, each receipted with the identical invocation in production. Then a nightly canary to keep the receipts fresh.

• July 2, 2026 Agent security 8 min read

  keeper: zero raw credentials in a production agent fleet

  Spawn a worker with the parent's environment and it inherits every secret the platform holds — ours were getting 132 keys they didn't need. keeper hands agents leases instead: TTL-bound, use-limited, host-scoped, revocable, audited, with the real key injected only at egress. The migration case study — 13 env keys kept, 119 withheld, git tokens and the model key leased — plus the bugs the rollout surfaced on both sides, fixed in public, down to the reflected-secret sanitizer proven against a live echo endpoint.

• July 2, 2026 AI engineering 6 min read

  The dead row: when a benchmark refuses to move, it's a pointer, not a verdict

  One benchmark row sat at zero through a feature built to move it. The board couldn't say why — the evidence could: two independent bugs wearing one symptom, a scorer blind to the ops canon and a search adapter returning zero for natural-language queries. Fixed, composed, receipted: 42% → 100% trusted sources, trust flipping mixed → high. Plus the release pipeline that shipped empty output twenty-eight times without a symptom at all.

• July 1, 2026 Agent security 7 min read

  picket: an injection firewall for the agentic browser

  An agentic browser has the lethal trifecta by construction — the page is untrusted, the logged-in session is private, navigation is the exfil channel. picket quarantines indirect prompt injection before the model sees it, gates dangerous actions, and keeps credentials out of agent context. 8/8 planted payloads withheld, validated against real Chrome 149 — with the real false positives and parity bugs found and fixed in public.

• July 1, 2026 Agent security 5 min read

  Deno built an agent firewall too — Claw Patrol, warden, and two layers of the same idea

  Deno’s Claw Patrol and warden answer the same thesis — an agent can’t police itself — at two different layers. Claw Patrol gates the network wire (SQL, Kubernetes); warden gates the MCP tool surface, stripping a poisoned tool before the agent can load it. An honest comparison — no primacy claim, theirs is the bigger product.

• July 1, 2026 Agent security 6 min read

  One gate, four frameworks: governing agents below the framework

  The capstone: CrewAI, LangGraph, the OpenAI Agents SDK, and Microsoft AutoGen — four unrelated runtimes governed by one MCP-layer gate. Not four integrations — each merged PR touches only examples/, with zero lines changed in the gate’s own source. The structural argument, with a proof grid to four merged PRs and four live write-ups.

• June 30, 2026 Agent security 7 min read

  Running a Microsoft AutoGen agent under askalf's control plane

  The fourth third-party framework under the same gate: Microsoft's AutoGen, run by its own AssistantAgent loop, governed at the MCP layer by askalf's warden — the poisoned tool stripped before the agent can load it, a destructive call blocked at the gate, every verdict in a tamper-evident audit — without patching AutoGen.

• June 30, 2026 Agent security 6 min read

  Running an OpenAI Agents SDK agent under askalf's control plane

  OpenAI's first-party Agents SDK, run by its own Runner, with every tool call governed by askalf's warden at the MCP layer — poisoned tool stripped, destructive call blocked, every verdict in a tamper-evident audit — without patching the SDK. Third framework in the series, same gate.

• June 30, 2026 Agent security 5 min read

  Running a LangGraph StateGraph under askalf's control plane

  The same gate that governed a CrewAI Flow, proven on a second, unrelated framework: a LangGraph.js StateGraph governed at the MCP layer — the poisoned tool stripped before the graph can load it, a destructive call blocked at the gate, every verdict in a tamper-evident audit — without patching LangGraph.

• June 29, 2026 Agent security 5 min read

  Running CrewAI FlowDefinition under askalf's control plane

  CrewAI orchestrates agents; it doesn't own the boundary between an agent and its tools. Here's a real CrewAI v1.15 Flow governed end-to-end by askalf's warden at the MCP layer — the poisoned tool stripped before the model sees it, a destructive call blocked at the gate, every verdict in a tamper-evident audit — without patching CrewAI.

• June 28, 2026 Release notes 3 min read

  What shipped — Jun 21–28, 2026

  A weekly, verifiable changelog from a studio running on its own agent fleet. 242 PRs merged across 32 repos — signing keys into the OS keychain, a tightened secret-store socket, deepdive v0.26.1, dario v4.8.101, and the internal tooling the fleet runs on. Every public claim links to a real PR or release.

• June 26, 2026 Source trust 6 min read

  deepdive: a source-trust engine for a local research agent

  A research agent is only as good as the sources it trusts — and search ranks an anyone-can-publish Google Doc next to a peer-reviewed paper. A measured, two-axis source-authority engine, built as a decomposed P1–P4 epic with a calibration benchmark — down to the real bug that scored a fabricable doc as authoritative, found, measured, and fixed in one auditable PR.

• June 26, 2026 Release engineering 7 min read

  dario: a self-healing autonomous release pipeline

  A published package that has to track a dependency Anthropic moves every few days — with no manual bandwidth to do it. The closed-loop pipeline that watches, validates, ships to npm, and heals itself: an idempotency gate, three trigger paths engineered around GitHub's own constraints, and self-monitoring guards. Six verifiable auto-releases in 72 hours, zero humans in the loop.

• June 24, 2026 AI infrastructure 4 min read

  The Gateway Layer Isn't the Stack

  “Own your AI stack” isn't a product category — it's a practice. Six receipts from the actual work: a routing proxy that corrupted code in transit and made the model look guilty, a command injection closed at the class level, an 18-minute hang where every timeout had passed, and the four-day silent outage that kept returning 200. Every claim links to a real PR.

• June 24, 2026 AI infrastructure 4 min read

  Why askalf isn't a framework — and what that means when you build with LangGraph, CrewAI, or AutoGen

  Every 2026 roundup of open-source AI agent tooling covers LangGraph, CrewAI, and AutoGen — and askalf, the platform this studio runs on, is in none of them. Partly a visibility gap we're fixing; partly correct, because askalf isn't a framework. It ships as a running system — dashboard, ticket queue, a fleet of agents, an LLM proxy with cost caps, a queryable execution log, human-in-the-loop — the operational layer those frameworks could run inside.

• June 23, 2026 Operations 8 min read

  Your embedding model vanished and nothing paged you

  The outages that cost you aren't the ones that fall over loudly — they're the ones that keep returning 200 and are quietly wrong for days. A four-day silent outage of the fleet's memory: a model gone after a rebuild, a zero-vector fallback that hid the blast radius, a watchdog guarding the wrong thing, and the guard that turns the next one into a ten-minute alert.

• June 19, 2026 Agent security 7 min read

  The whole agent-security stack, behind one MCP server

  Five separate tools — a tool-action firewall, a skill supply-chain gate, a secrets vault, a PII redactor, a browser injection firewall — composed into one MCP server an agent can call mid-task. The bug only stacking reveals, the live proof all five fire, and the dull question that turned out to have a real answer: was the stack actually up to date? It wasn't.

• June 18, 2026 AI engineering 6 min read

  Citation verification isn't enough. My research agent scored a perfect 1.00 on content farms.

  Every AI research tool now verifies its citations. Mine gave a flawless score to an answer where every source was an AI content farm — because a verified citation proves the source said it, not that it's true. The orthogonal fix has no model in it.

• June 16, 2026 AI infrastructure 11 min read

  Your CPU isn't bad at LLMs — it's bandwidth-starved

  One law predicts every CPU inference number: decode tok/s × model-bytes ≈ a constant 14–18 GB/s memory wall. So I ran an 8B model at 7.6 tok/s and a 30B model larger than the RAM it ran in, on a GPU-less 2013 desktop — and kept the limits no lever fixes: prefill is ~240× off a GPU, speculative decoding is GPU-shaped, and a cheap router can't catch its own confident wrongness.

• June 15, 2026 AI engineering 9 min read

  You can't use an LLM to grade an LLM. I tried, for money.

  I sell a $1,500 code audit. The obvious way to scale it is one model finding the bugs and a second checking the first's work. I built that — it doesn't work, and the way it fails is worth more than if it had. A negative result, with the receipts.

• June 15, 2026 Agent security 8 min read

  The agent guardrails that were theater

  Controls that existed but never ran: a "jail" that was a shell, an approval gate never wired into the exec path, a denylist that forgot the Windows shell — several found by attacking my own agent. The fix is almost always to remove a capability, not add a sanitizer.

• June 15, 2026 Security 8 min read

  Every untrusted byte is an attacker

  AI systems ingest hostile input by design — search results, robots.txt, synced files, a URL a model picked. So classic appsec applies: a ReDoS that blew 0.04ms to 218ms, a cloud-metadata SSRF, an unauthenticated browser takeover — found by reading the live deployment against the source, including one gap I didn't close.

• June 15, 2026 AI infrastructure 7 min read

  The bugs your CI will never catch (because it runs clean Linux)

  The failures that only surface on real hardware and the real install path: one AUTH_FAILED symptom hiding two unrelated bugs, a regex that returned zero sessions on Windows, a global CLI that was a silent no-op for every npm -g user. Test the published artifact, not the source.

• June 15, 2026 AI engineering 9 min read

  Half my model regressions were the measurement lying to me

  A search backend rate-limiting my test IP and silently falling back to Wikipedia, a date tweak that tanked scholarly citations 0.68→0.44, an 18-minute hang where every timeout passed. Half my "model regressions" were the harness lying. Fix the harness before you trust the number.

• June 15, 2026 Agents 8 min read

  How I let an agent run commands on production and still sleep

  A four-tier risk engine gated at one chokepoint with automatic rollback, a once-consumed approval queue, autonomy earned by repeated approvals — and the deliberate choice to route a fleet with a deterministic keyword bus instead of an LLM. The architecture for letting an agent touch production without burning it down.

• June 15, 2026 Agent security 6 min read

  I built a firewall for my AI agents — what it actually took

  OpenClaw became 2026's first big AI-security disaster: one-click RCE, a poisoned skill marketplace, exfil. So I built warden — a firewall that decides what an agent's tools may do before they run. A deterministic gate at 95% recall / 98% precision, an LLM judge that deobfuscates evasion, and a tamper-evident audit. The honest version, misses included.

• June 13, 2026 AI infrastructure 5 min read

  When Anthropic pulled two models overnight — what it takes to keep a proxy honest

  On 2026-06-12 a government directive disabled Claude Fable 5 and Mythos 5 for every Anthropic customer. If your tooling keeps advertising a model that no longer answers, it's not neutral — it's broken. Here's how we shipped a fix to dario in a day, and the design rule behind it.