Shadow report · Agent Gateway demo
26 calls blocked in shadow mode, including 9 refunds worth $60,153.
Blocked: $60,153 of refund money in 9 refunds. The other 17 blocked calls moved data, not money: 7 customer exports (36,745 rows), 6 emails to unlisted domains and 4 malformed calls.Held for a person: 25 more actions, 23 of them refunds worth $36,746.
The week, call by call
Each mark is one tool call. Nothing was enforced: every call ran. The mark shows what enforce mode would have done. Select an amber mark to open its diff.
The actions it would have stopped
What the agent tried, against what the policy allows, field by field. Each line cites the key in policies/v2.yaml that decided it.
The policy that judged it
One file, validated on load and hashed into every verdict. The right column counts how often each key fired this week. It is production v1 with limits added for the agent’s two new actions, customer exports and outbound email; every other line is the same as v1.
Every verdict is in a hash chain
Each entry hashes the one before it, with the same function the live audit log uses. This page re-hashes all 208 in your browser, so an edited verdict cannot pass unnoticed. Try it: flip one blocked refund to allowed.
How we measured
The verdicts are real engine output, on a demo company’s week.
- All 208 calls are demo traffic for Northwind Supply, a demo company: 170 from the gateway demo’s generated week, plus 38 export and outbound-email calls written to exercise the two new limits.
- Each verdict comes from the gateway’s own engine: identity and scope, the strict tool contract, then the policy rules, on the context recorded at call time (rate, spend, customer history, invoice balance, export size). The same engine passes 46 engine tests and 304 test cases against the in-browser copy.
- “Worth” means refund dollars only: the amount the agent asked to refund. Exports, emails, deletes and malformed calls carry no dollar figure; we count rows and recipients instead of guessing a breach cost.
- Shadow mode means observe-only: every call ran, and the gateway wrote down what enforce would have done. Held actions would have waited for approval, and some would have been approved, so they are reported apart from blocked ones.
- On every refund, delete and lookup, v2 decides exactly as production v1 (a test checks this); it changes only the export and email limits, so it can only block more, never less. The chain was built 12 Nov 2026; all 208 entries are verified at build time and again in this browser.
For engineers
- Traffic:
traffic/northwind_week.jsonl(tools/make_seed_traffic.py) andtraffic/northwind_week_ext.jsonl(tools/make_shadow_traffic.py). - Verdicts:
gateway/shadow.pydecide(), which callspolicy.check_scope_and_schema()andpolicy.evaluate()on each record’s context. - Chain:
engine.entry_hash, SHA-256 over the previous hash plus canonical JSON.tools/build_shadow_report.pywrites every number on this page;tests/test_shadow_report.pyfails if the page data is stale.
Want an agent that acts, with this gate in front of it?
We build the AI feature and put every action it can take behind a gate like this one. Shadow mode first, so you see what it would stop before anything is enforced.
Build an AI Feature
$3,500two-week sprint
One AI feature shipped in two weeks, hitting one agreed number
A support bot, copilot, document AI or workflow automation. Production from $7,500.
- You pay when the agreed metric passes.
- Every action the AI takes goes behind a safety gate: identity, schema, policy and risk, a person for the risky ones. This demo is that gate.
- Evals on your data, a cost budget and an attack test, included.
Start here, free
The 48-hour AI check. Send your agent’s tool list, your bot’s link or your AI bill. Within 48 hours you get a one-page report: the three biggest problems and what each one costs you.
or email hello@khlabs.devWe reply within 24 hours.