Protocol Agent-Readiness Audit

Version protocol-readiness-v1 · code: packages/protocol-audit · tables: migration 027_protocol_audit.sql · API: /v1/protocol-audits · pages: /protocols · operations: docs/runbooks/protocol-audit.md

The question: before AI agents use a DeFi protocol in production, how do they fare on it? Where do they succeed, fail, get confused or lose funds, and what would the protocol change to make it safer and clearer for them?

A protocol gives us its contracts (an address on Base or Ethereum, or creation bytecode to deploy into a fork). We generate the flows an agent would attempt from the ABI, run a battery of scripted agents through them on an Anvil fork at a recent block, and report an Agent Readiness Score, findings with concrete fixes, and reproducible traces. The report hash is attested on Ethereum Sepolia.

This is a measurement of how scripted agents behave against the agent-facing interface of a protocol at one block, on a fork. It is not a security audit, not a risk rating of the protocol, and the agents are scripts, not language models. Nothing is ever broadcast to a real network.

#1. Intake

A config (ProtocolConfigSchema in packages/protocol-audit/src/types.ts, validated with zod at the API and again in the worker):

ABIs. From the config if given, else from Sourcify (v2 API, with proxy resolution) and then the chain's Blockscout, both at fixed hosts. For a proxy the implementation's ABI is fetched and merged (deduplicated by signature); if the registry does not report a proxy, the EIP-1967 implementation slot is read from the fork. A contract with no ABI anywhere is audited only through flows with explicit signatures, and says so (unverified-source).

ABIs are read at the time of the run, not at the fork block; the ABI hash is part of the report, so a registry change shows up as a different hash on re-run.

Fork. The Anvil fork is re-pinned with anvil_reset to the config's block, or to the chain head minus 16 blocks. Contracts must have code at that block. Block timestamps and base fees are set explicitly before every transaction and agents use deterministic keys, so the same config at the same block produces the same transactions. Agents' ETH and token balances are written directly into the fork's storage (the balance-mapping slot of each token is discovered by probing and verified with balanceOf).

#2. Task generation (deterministic, no language model)

Each flow becomes a task spec: plain-language instruction, the exact call (canonical signature plus argument template), the approvals needed, and success checks read from chain state (balance of an input token down, balance of an output token or a receipt token up, a protocol position view up or down). Argument templates use placeholders ($agent, $recipient, $onBehalf, $amountIn, $tokenIn, $tokenOut, $minOut, $deadline) that the agent profile binds at run time; an unbound placeholder is an error, never sent.

The interface is recognised from the ABI:

A flow whose shape is not recognised and has no explicit call is reported as not generated with the reason; it is excluded from the score, shown under coverage, and a low coverage caps the grade. Flows that need a position (withdraw, borrow, repay) list the flows that create it in requires; the harness performs those first, with a careful agent, before every case.

#3. The agents

Scripted policies (packages/protocol-audit/src/exec.ts); they run through the same harness, record the same trace and are judged by the same checks.

All agents are scripts. "Adversarial" describes how careless or exposed their behaviour is, not that they attack. The attacker in the price-move condition is a separate harness actor.

#4. The battery

For every flow, every selected agent runs from the same pristine snapshot, so cases are independent. The price-move and probe cases run only for flows the careful agent completed on a clean market (a flow it cannot complete is reported as such and has no meaningful hostile-input behaviour to compare against):

1. Baseline: each agent, clean market. Outcome: success, reverted, refused (its own simulation reverted, so it did not send), check_failed (the transaction succeeded but chain state does not show the intended result), or unsupported (the harness could not run it: reported as coverage, never as a protocol failure).
2. Price move (flows with a reverse path, i.e. swaps): after the agent has quoted, approved and simulated, and before its transaction lands, an attacker repeats the agent's own swap in the same direction with growing size (×1.5 steps; a step that overshoots the target by more than 25% is undone and retried at half size) until a probe quote for the agent's amount has fallen by about 100 bps. The agent's transaction then lands. The move actually reached is recorded per case; if the pool cannot be moved to at least half the target the case is unsupported and measures nothing (it is not scored and not praised). The careful agent's call reverts on its output bound (it never reaches its own refusal, because it simulated before the move). Loss is the output shortfall against the careful baseline, in bps; funds at risk are that fraction of the flow amount in the input token (USD when the token is a stablecoin or priced). For sandwich-prone the attacker then unwinds with the reverse swap and the report states the attacker's net in input-token base units (negative = unprofitable after fees in this model).
3. Probes (a bare agent, no safeguards, one hostile or careless input each, on flows where the input exists): amount 0; amount 1 base unit; recipient 0x0; no approval; amount above the wallet balance; deadline in the past; minimum output ten times the quote; an input token address with no code. A probe's verdict is rejected (reverted), accepted (harmless) or accepted_unsafe (funds left the agent and the result did not arrive, or a guard that must hold did not).
4. Re-entrancy probe: the reentrancy-probe agent's transaction is traced (debug_traceTransaction, call tracer). We count calls from the configured protocol contracts into the agent's wallet or recipient (and those carrying value), distinct contracts touched, delegate calls, and call loops X → Y → X among the configured contracts. It does not deploy attacker contracts, and contracts that are not in the config (a pool behind a router) are not counted as protocol; it tells you whether the configured contracts hand control to an agent-controlled address during the flow.

Every revert is decoded against the protocol's published ABI plus standard token errors, and attributed from the call tree to the contract that raised it: a revert raised by a token (the protocol only bubbled it up) is counted apart and not held against the protocol; its legibility is 1 for a named custom error with parameters, 0.8 for a sentence in Error(string), 0.65 for other strings, 0.4 for short or numeric codes ("STF", "26"), 0.3 for a panic, 0.2 for a selector missing from the published ABI, 0 for an empty revert.

#5. What is measured

Per agent and flow: success rate, revert reasons (raw and decoded), gas and transaction count, funds lost (value that left without the result), allowance left behind (none, bounded, unlimited), loss under a price move, call-tree facts. Per protocol: hostile-input probes, error legibility, approval surface (who the spender is and whether it is an upgradeable proxy; whether exact approvals suffice), and the static interface findings below.

#Static interface findings (functions an agent would use)

Computed from the ABI alone, so reproducible without a fork. "User-facing" means a state-changing function whose name matches a user intent (swap, exact-in/out, deposit, withdraw, redeem, mint, burn, supply, borrow, repay, claim, liquidate, flash, …) and is not an admin/setup/callback function, plus every function used by a flow.

#Dynamic findings

careful-agent-failed (high) and max-approval-required (high); one finding per unsafe probe class and contract (probe-zero-recipient, probe-impossible-min-out high; probe-expired-deadline, probe-non-contract-token, probe-overspend medium; probe-dust-amount, probe-zero-amount low); price-move-exposure (≥300 bps high, ≥100 medium, ≥25 low); error-opacity (at least 3 reverts observed and mean legibility below 0.6: medium, below 0.2: high); callback-to-agent (medium with value, else low); events-not-in-abi (medium); no-protocol-event (info). Every finding carries a concrete fix and the ids of the cases that demonstrate it.

#6. The Agent Readiness Score

A fixed weighting of six components, summing to 100; each component is computed only from the cases and the ABI in the report.

Grade: A ≥ 85, B ≥ 70, C ≥ 55, D ≥ 40, F below. If fewer than half of the requested flows could be executed the grade is capped at C and the report says why; if none could be executed the score is at most 39 and the grade F. The score orders protocols by how their agent-facing interface fares under this battery. A lending protocol with no price-sensitive flow earns the safety points by construction; the report states that no price move was measured rather than implying it was tested.

#7. The report, its hash and the attestation

The report (karatum.protocol-readiness.v1) contains: the protocol and contracts (ABI source, ABI hash, code hash, proxy facts), fork block and timestamp, the generated task specs, the static analysis, the agents, every metric, the findings, what held, the score with its components, an index of all cases with per-case hashes, the caveats, and a reproduction command. It contains no wall-clock times.

reportHash = keccak256(canonical JSON of the report) (sorted keys, bigints as decimal strings); each case has its own caseHash, indexed in the report, so the hash commits to every trace. The cases (with full traces: calldata, selectors, gas, revert data and decoded reason, events, tx hashes of the fork) are stored separately and served by GET /v1/protocol-audits/:id/cases/:caseId.

The hash is attested on Ethereum Sepolia through EAS (0xC2679fBD37d54388Ce493F1DB75320D236e1815e) by a dedicated testnet key, schema string protocol,uint256 chainId,uint64 forkBlock,uint16 score,string grade,uint16 findings,uint16 high,bytes32 reportHash,string methodology. The attestation proves what Karatum published at that time, not that the measurement is true; to check the measurement, re-run it.

Reproduction. Start an Anvil fork of the chain at the report's block from an archive-capable RPC, then karatum-protocol-audit run --config audit.json --rpc http://127.0.0.1:8545 --block N. Because actors, nonces, timestamps and fees are deterministic, the same config at the same block yields the same report hash; each case study below was re-run from scratch at its block and reproduced its hash exactly.

#8. Access

• POST /v1/protocol-audits {config, visibility?}: self-serve, free during the testnet phase, no sign-up. Returns 202 with the id and, for unlisted audits (the default), a view token shown once (only its sha256 is stored). Limits: 3 requests per minute, 6 per day and one waiting-or-running audit anonymously (10, 40 and 3 with an API key), at most 30 audits waiting; the daily, active and queue caps are enforced atomically in the database. Audits run one at a time per chain fork (also enforced in the database). The whole body, inline ABIs and creation bytecode included, must fit 64 KB. Prefer the X-Audit-Token header to ?token=: query strings end up in access logs.
• GET /v1/protocol-audits lists the published case studies; GET /v1/protocol-audits/:id (id or case-study slug) returns status and, when complete, the report; ?token= or X-Audit-Token opens an unlisted audit; a missing or wrong token looks exactly like a missing audit (404).
• /protocols (index and case studies) and /protocols/:id (report; ?t=<token> for unlisted). Public reports are readable by id but only curated audits are listed.
• CLI: karatum-protocol-audit run|list|worker|case-studies|attest.

#9. Case studies

Three well-known Base protocols, audited read-only on forks of Base mainnet, using the configs in packages/protocol-audit/case-studies/. Block, scores and hashes below are those of the published runs; the live numbers and every trace are on /protocols.

Every run completed all generated flows with the careful agent (2/2, 5/5, 5/5). Score components: Uniswap: task completion 30/30, safety for a naive agent 15/25, input hardening 11.9/15, error legibility 5.3/10, approval surface 10/10, interface clarity 7.5/10. Aave: task completion 30/30, safety for a naive agent 25/25, input hardening 15/15, error legibility 1.4/10, approval surface 6/10, interface clarity 6/10. Morpho: task completion 30/30, safety for a naive agent 25/25, input hardening 15/15, error legibility 7.4/10, approval surface 10/10, interface clarity 6/10.

Uniswap v3 SwapRouter02 (Base, USDC to WETH 1,000 USDC and WETH to USDC 1 WETH, single-hop; fee tier 500 chosen by simulated output).

• Task completion held for every agent: exact approvals suffice, the spender is the router itself, no callback from the router into the agent's wallet.
• Price move: with the pool pushed 125 and 124 bps against the trade, the careful agent's amountOutMinimum turned the move into a revert with nothing lost, while a naive, slippage-insensitive or sandwich-prone agent executed 125 and 124 bps below the careful baseline. In this model the attacker's unwind netted +7.2655 USDC on the USDC trade and -0.1292 WETH on the WETH trade, so the loss is the price move, not necessarily extractable value.
• Hostile inputs: with recipient 0x0 the USDC to WETH swap succeeds and the WETH is burned (the reverse direction reverts because USDC refuses transfers to the zero address); an input of one base unit is accepted and returns nothing. Missing approval, balance overspend, an impossible minimum output and a code-less token all revert.
• Interface: exactInputSingle has amountOutMinimum but no deadline (a deadline exists only through multicall(deadline, …)), no events and no custom errors in the published ABI, and sqrtPriceLimitX96 must stay 0. Reverts are short strings ("STF", "AS", "Too little received") or empty.

Aave v3 Pool (Base, proxy at 0xA238…d1C5, implementation 0xA4Ab…dFdB; supply and withdraw USDC, supply WETH as collateral, borrow USDC at the variable rate, repay).

• Task completion 5/5; all 25 weighted hostile-input probe points were rejected or harmless (a repay larger than the wallet balance was accepted and delivered everything it promised, since Aave repays at most the outstanding debt). No price-sensitive flow was configured, so the naive-safety component is full by construction and says nothing about Aave's pricing.
• Error legibility is the weak point: the pool reverts with custom errors (for example 0x2c5211c6 on a zero amount) that are not in the published ABI, so an agent decoding with the ABI sees an unknown selector; mean legibility 0.14 over 10 protocol reverts.
• Interface: six functions (supply, withdraw, borrow, repay, repayWithATokens, liquidationCall) also exist as opaque packed bytes32 overloads (the L2 encoding) next to the readable ones, deposit and supply are duplicate entry points, and the spender is an upgradeable proxy (the tested behaviour is that of one implementation).

Morpho Blue (Base, the WETH/USDC market with 86% LLTV, id 0x8793…1bda; supply and withdraw USDC, supply WETH collateral, borrow USDC, repay).

• Task completion 5/5, all 22 hostile-input probe points rejected or harmless, reverts legible (sentences such as "inconsistent input", "zero address"), exact approvals suffice, no callbacks. As with Aave, no price-sensitive flow was configured.
• Interface: four functions take both assets and shares (exactly one must be zero, which the ABI cannot express), the contract emits events that its verified ABI does not describe (events defined in a library), there are no custom errors in the ABI, and data must stay empty (a non-empty value triggers a callback into the caller).

The three runs show how the score moves: the AMM loses points for what a careless agent can lose, the lending protocols for error design and interface shape. Read the scores as "how forgiving and legible is this interface to an agent", not as a ranking of the protocols' quality. All three configs are in packages/protocol-audit/case-studies/; the previous iterations of these runs (earlier harness versions, same contracts) were discarded and re-run after an independent code review of the harness; their EAS attestations on Sepolia still exist but refer to superseded reports.

#10. What this does not tell you

• It does not find contract bugs. A protocol can score A and still be exploitable, and a C does not mean it is unsafe.
• The agents are scripts. A language-model agent can make mistakes the scripts do not (mis-scaling decimals, picking the wrong overload) and can avoid ones they make. An optional LLM follow-up can be requested with llmAgents: true; it is queued behind the deterministic battery, needs the LLM inference host (currently unavailable), and is not part of the score. When it exists it will execute the same task specs through the same harness and checks.
• The price-move test moves one pool with same-direction swaps. It does not model mempool ordering, private order flow, builders or cross-pool arbitrage, and the number of bps lost depends on pool depth at the block.
• The re-entrancy probe reads call trees; absence of a finding is not proof of safety.
• Flows are generated for the interface shapes in section 2. Everything else needs an explicit call, and untested flows are listed as coverage gaps.
• The fork is a snapshot. Liquidity, rates, caps, oracles and implementations change; a report describes the protocol at its block. Reproduction needs an archive-capable RPC for that block, which free public RPCs may not provide for old blocks.
• Funding is synthetic (token storage is written), so a token with unusual storage (rebasing, fee-on-transfer, a layout the probe does not recognise) may be unsupported and shows as a coverage gap.
• Score weights are a judgement, published here so they can be argued with; changing them is a version bump of METHODOLOGY_VERSION.