keeper: zero raw credentials in a production agent fleet

The engineering problem

A key in an agent's context is a key in every log, every trace, every crash dump, and every place a poisoned tool can read. OpenClaw made the failure mode concrete at scale: on the order of 135,000 exposed instances with long-lived credentials sitting in agent environments and config. But the interesting part isn't the headline; it's why this keeps happening.

It happens because the default is wrong. When a platform spawns an agent process, the path of least resistance is {...process.env}: hand the child everything the parent has. Our own platform did exactly this. When we measured it, every spawned agent was inheriting 132 environment keys, most of them secrets or connection strings: cloud credentials, database URLs, service tokens for every integration the platform speaks to. The agent used almost none of them. It needed model access and, for repo tasks, a git credential, and it was being handed the keys to the entire studio.

The fix isn't a lecture about not putting keys in prompts. It's changing what the agent holds: an agent should hold a lease — an opaque handle bound to a TTL, a use count, and a destination — and the raw secret should exist only at the network boundary, injected by something that isn't the agent.

That is what keeper does.

What keeper is

keeper is an encrypted secrets vault built for agents, with two egress mechanisms that keep the key out of the agent's hands:

• Vault + leases: secrets encrypted at rest (AES-256-GCM with the secret's name bound in as AAD; master key from scrypt, the OS keychain, or a locked-down key file). grant mints a lease bound to TTL, use count, and host; only the lease's hash is stored, so reading keeper's files can't redeem anything. Redeem is an atomic check-and-consume: a single-use lease can't be double-spent, and a denial never burns a use.
• The broker: for HTTP APIs. The agent's client points at http://127.0.0.1:<port>/<lease> with a placeholder key; the broker checks the lease (endpoint allowlist, rate cap, concurrency cap), redeems it, and makes the real upstream request itself with the secret injected at the network boundary. The lease is bound to one upstream, so the secret physically cannot be sent anywhere else.
• The redeem-daemon: for credentials a tool consumes directly. A long-lived local process holds the master key and answers lease→secret over a token-gated local socket, so the redeeming process is keyless: compromise it and you get its leases (scoped, expiring, revocable), not the vault.
• Tamper-evident audit: every grant / redeem / deny / revoke is hash-chained, with an HMAC-authenticated tip so truncating or splicing the log is detectable, not just editing it. Leases are logged by fingerprint, never raw.

The design decision that matters most: every failure fails closed, and a failure never costs the caller anything. A corrupt vault denies rather than throws; a decrypt failure denies without consuming the use; an unavailable lock denies rather than running the atomic section unlocked. These are stated properties with regression tests, because a secrets tool's behavior under failure is the product.

• Repository: github.com/askalf/keeper, public, MIT, zero runtime dependencies beyond the stack's shared warden audit primitive
• 61-test suite: functional, robustness, adversarial-security, fail-safe, and end-to-end batteries, CI on Linux and Windows
• Ships a CLI, a library, the broker, and the daemon; npm run demo walks the whole story offline

The migration: three seams, all flag-gated

Our platform dispatches a fleet of specialist agents: each execution spawns a worker that runs a coding agent against real repos, real infrastructure, real APIs. The migration touched the three places a credential crossed into agent territory, in production, behind flags that default off, each with a one-line rollback.

1. Git credentials: from a token on disk to a keyless redeem

Before: the worker wrote the raw GitHub token into a credential-helper script under /tmp and pointed GIT_ASKPASS at it. A token, on disk, in the agent's sandbox, for the lifetime of the run.

After: the worker stashes the token in keeper (encrypted), grants a short-TTL, few-uses lease, and hands the agent a credential helper that runs keeper redeem, routed through the in-process redeem-daemon. The agent's environment carries the lease id and a socket path. Zero token bytes on disk, and the agent-side process never holds the master key. The lease is revoked and the stashed secret deleted in the execution's cleanup, and the whole lifecycle (add → grant → redeem → revoke → remove) shows up in the audit chain per execution. Live in production since June 16.

2. The spawn environment: 13 keys kept, 119 withheld

The core win was making the worker stop spawning agents with the platform's environment. We built a default-deny allowlist with three modes: off (unchanged), observe (full env, but log the names that would be withheld), and enforce. We ran observe first against live traffic to earn the list rather than guess it. Two findings shaped it: the agent reaches internal tools through a config file whose bearer token never needed to be in env at all, and git needed no token in env because of the lease above.

Under enforce, measured in the running container: the agent environment is exactly 13 keys (HOME, PATH, model routing, the keeper handles, the git credential-helper config), and 119 keys are withheld, including every cloud, database, and integration secret the platform holds. The keeper passphrase itself is explicitly blanked: agents can hold leases, not the vault.

3. The model key: a placeholder in env, the real key at egress

The last raw credential in the agent's environment was the model API key. It now goes through the keeper broker: at spawn, the worker mints a per-execution lease bound to our model gateway, sets the agent's base URL to http://127.0.0.1:8771/<lease>, and sets the API key to an obviously fake placeholder. The broker injects the real key per request, at the boundary. We validated streaming end-to-end before flipping — server-sent events flow through the broker chunk-for-chunk — and the flip completed on June 27.

Net state, verified live in the deployed container: an agent in our fleet holds zero raw fleet or model credentials. Its model key is fake, its git access is a revocable lease, and the hundred-plus platform secrets are simply absent.

Integration is a flashlight

Wiring a security layer through a real system exercises paths nothing else does, and it found two pre-existing platform bugs that had been latent because the credential path had never actually run.

First: the worker's credential-helper script could never have executed. The container mounts /tmp with noexec, so git's askpass exec was doomed from the start, for the raw-token path as much as for keeper. The fix dropped the script file entirely: the credential helper is now injected inline via git's GIT_CONFIG_* environment variables, nothing on disk to execute, and it supplies the username the clone URL lacked. Second: repo tasks tried to build a git worktree on a read-only mount, impossible by construction; they now get a fresh writable working directory per execution, and a repo task whose credentials can't resolve fails fast with an actionable error instead of a fake infrastructure one. (Our platform is a private monorepo, so those fixes aren't linkable; the keeper-side artifacts below all are.)

The honest takeaway: a credential path that has never been forced to run is a credential path you should assume is broken. The security migration was the first thing that forced ours.

What real bugs look like in a secrets tool

keeper's own history is public, and the most instructive parts are the defects its adversarial batteries and live rollouts surfaced, each one a merged, linkable fix with a regression test.

• A decrypt failure burned a use. The end-to-end daemon test caught redeem consuming the use before decrypting, so a worker's expected local decrypt-failure was silently exhausting its own single-use lease, contradicting a documented guarantee. The secret is now materialized inside the atomic consume; failure denies without spending.
• The lock could fail open. An orphaned lockfile (a holder killed mid-section) made the spin loop give up and run the critical section unlocked, quietly reopening the single-use double-spend window the lock exists to close. PR #5: stale locks are reclaimed, and if the lock still can't be acquired, redeem denies. Fail closed, never unlocked.
• The daemon's socket obeyed the umask. The one keeper artifact not locked to its owner was the socket that serves decrypted secrets: the docker.sock class of bug. PR #10 chmods it 0600 the instant it binds, pinned by a test that forces umask 000.
• The secret could come back. The broker injects the key upstream, but an upstream that reflects it (an echo or debug endpoint, a verbose error, a misconfigured proxy) would hand the raw key straight back into the agent's context. PR #14 sanitizes responses on the way out: any occurrence of the secret in relayed headers or body is redacted, streaming-safe (a secret split across chunk boundaries is still caught), with a sanitize audit event, because a reflected secret is an incident worth seeing, not just silently fixing. The same PR added a per-lease concurrency cap alongside the endpoint allowlist and rate cap.

The receipt

The sanitizer shipped with unit coverage, but the claim is easier than that to check. Point a lease at a real echo service — one that reflects request headers back at you — and make a call through the broker with no key:

$ curl http://127.0.0.1:8899/$LEASE/anything
{
  "headers": {
    "Accept-Encoding": "identity",
    "Authorization": "Bearer [keeper:redacted]",
    ...
  }
}

Read it closely, because the whole design is in there: the upstream did receive the real key (the surviving Bearer prefix is where it sat), the reflection came back redacted so the caller never sees it, and the broker pinned the response uncompressed so the scan sees real bytes. The audit log for the session tells the rest of the story: two parallel requests through a concurrency-1 lease, one refused without consuming a use:

redeem   LIVEKEY · httpbin.org
sanitize 0904cd29dfd9
redeem   LIVEKEY · httpbin.org
deny     508d76d00570 (concurrency)
sanitize 508d76d00570

✓ audit chain intact (12 entries)

Composable, like the rest of the stack

keeper is the secrets leg of Own Your Stack: warden gates actions, canon vets skills, picket governs the browser, keeper holds the keys. The composition runs in production on our own fleet and ships as a public demo in agent-security-stack. And the pattern isn't platform-specific: the repo carries a worked example of an OpenAI Agents SDK agent whose API key never enters its process; any framework that accepts a base URL inherits it.

What this demonstrates

Least privilege as a migration, not a rewrite. Three seams, each behind a default-off flag, observe-before-enforce, canaried on live traffic, one-line rollback, on a production fleet, without an outage.

Failure properties as product. Deny-don't-throw, deny-don't-spend, locked-or-refused, redact-and-audit: the behaviors that matter are the ones under failure, and each is a stated design decision pinned by tests.

Receipts over claims. The repository, the PR history, the demo, and the example are public; the numbers in this piece (13 kept, 119 withheld, the redacted reflection) were measured against the running system, and the ones you can reproduce offline take one npm run demo.

Scope and limits

Scope matters. When the redeem-daemon runs inside the same trust domain as the platform that owns the vault, keeper's win against a platform-level compromise is bounded: the platform holds the master key, and something has to. What keeper changes is the agent's exposure: the blast radius of a compromised or manipulated agent drops from “every credential the studio has, indefinitely” to “a lease: scoped, expiring, counted, revocable, audited.” The bootstrap secret (the vault passphrase) is irreducible and lives with the platform, stripped from every agent. And keeper is one studio's production system, not a standard; the code is public precisely so the claims don't have to be taken on faith.

All claims are verifiable at github.com/askalf/keeper (as of 2026-07-02).