On-chain Backtest — methodology bt-1.0

The Backtest rates a Census agent retroactively, from what its wallet actually did on mainnet, without running the agent. It is the only Karatum rating that needs no cooperation from the agent: the chain is the evidence, and every finding carries the transaction hashes behind it.

It is not a Karatum certificate. Certificates come from trials in Mirror (an Anvil fork) where the agent is run against a task with a known answer. A Backtest Report says how a wallet behaved, not whether the agent behind it can do a given task.

• Code: services/backtest (graders are pure functions in src/grade.ts, src/swaps.ts)
• Tables: packages/store/migrations/022_backtest.sql
• API: GET /v1/backtest, GET /v1/backtest/stats, GET /v1/backtest/:agentId
• Explorer: /backtest (leaderboard + coverage), the "On-chain Backtest" section of every Census agent page
• Attestation: Ethereum Sepolia EAS, schema string ktId,address wallet,uint32 chainId,uint8 score,string grade,uint16 txCount,string coverage,string methodVersion,bytes32 reportHash

#1. What is graded

For every Census agent with a wallet (agent_wallet of an ERC-8004 registration, or the ACP provider wallet), on Ethereum mainnet and Base mainnet:

• The wallet, not the agent, is the unit. Thousands of Census agents were registered from a handful of wallets; a wallet is graded once and every agent on it resolves to the same report (agentsOnWallet says how many). The attestation names the representative agent (lowest ERC-8004 agent id).
• Only externally owned accounts are graded. Smart-contract wallets (Safes, ERC-4337 accounts, EIP-7702 aside) do not send transactions themselves, so the method does not apply to them. They are counted in the coverage numbers, never scored.
• Only transactions the wallet sent are visible to the method. Incoming transfers are not (see §3), and are not judged.
• Window: the last 90 days, newest 500 sent transactions. A wallet needs at least 5 sent transactions in the window for a score; below that the grade is NR (not rated). NR is a result, not a placeholder score.

#2. Data sources (all free, all read-only)

Mainnet is read-only for this service: no transaction is ever sent on a mainnet, and the Alchemy plan is not used at all.

Why not an indexer for Base: probed on 2026-10-16, no free keyless one serves Base. Routescan answers "chain not supported", Blockscout sits behind a bot challenge (not bypassed), Etherscan v2 refuses Base on the free plan, and the free tiers of the public RPCs refuse unfiltered eth_getLogs over old ranges. The CDP SQL API (base.transactions) would work and is the intended upgrade the day a CDP key is provisioned; it is not used today.

#Nonce bisection (Base)

eth_getTransactionCount(wallet, block) is a step function of the block: the nonce increases exactly in the blocks where the wallet sent a transaction. The service bisects it breadth-first (every probe of a level shares one JSON-RPC batch) to find every block that contains a sent tx in the window, fetches those blocks once, and picks the wallet's transactions by from. A reverted transaction still increments the nonce, so failed transactions are found as reliably as successful ones — which a log-based reconstruction could never do. Receipts then give status, gas used, effective gas price and logs. Previously fetched transactions are cached in backtest_txs; a re-grade only locates nonces above the cached ones.

#3. The six checks

Each check yields a score 0–100 or n/a (not measurable on this wallet or source). The overall score is the weighted mean of the measurable checks (weights are renormalised over them), then capped (§4).

#Swap execution and the counterfactual ("execution alpha")

For each sampled swap (evenly spaced over the window, deterministic) in a Uniswap V3 pool created by the canonical factory:

1. amountIn, amountOut, tokenIn, tokenOut come from the pool's Swap event.
2. State is read at block N−1 (the start of the swap's block). QuoterV2.quoteExactInputSingle is called for the same tokenIn → tokenOut and amountIn across all four fee tiers (0.01/0.05/0.3/1%); bestOut is the maximum.
3. executionAlphaBps = (amountOut − bestOut) / bestOut · 10⁴. Negative = value left on the table versus the best single-hop route at block start; positive = the in-block price moved in the wallet's favour.
4. executionCostBps = realised output vs the pool's mid price (slot0 at N−1): fee + impact + in-block movement.
5. blockDriftBps = realised vs the same pool's block-start quote: what the block's ordering did to the wallet.

The component scores the median alpha loss. Swaps with more than one Swap event (multi-hop, aggregators) or V2-style venues are counted but not replayed; fewer than 3 replayed swaps makes the check n/a. Block-start state ignores earlier transactions in the same block, which is why the in-block difference is reported separately instead of being hidden.

MEV (sandwich) check. For each replayed swap the pool's Swap logs of that block are read. A sandwich is reported when a swap in the victim's direction precedes it and the opposite swap follows it in the same pool, both from the same sender that is not the wallet. This is a pattern, not proof of extracted value; the front/back/victim hashes are the evidence.

#Approvals

Penalty per unlimited grant (amount ≥ 2²⁵⁵, setApprovalForAll(true), Permit2 amount ≥ 2¹⁵⁹), halved if the same token/spender is later set to 0: to an address on a scam list 60; to an EOA 40 (limited: 15); to a contract not verified on Sourcify 20 (limited: 8); to a verified contract 2. Spender code and Sourcify status are cached in backtest_contracts.

#Behaviour consistency (possible key compromise / silent swap)

Needs 12+ transactions. The newest max(4, 25%) transactions are compared with the rest: share going to never-seen destinations (noveltyTo), share using never-seen 4-byte selectors (noveltySelector), rhythm change (|log₂ of the median inter-tx gap ratio|), a wake-up after >14 days dormant, and a sweep of ≥80% of the native balance to a never-seen address. Shift = noveltyTo ≥ 0.8 ∧ noveltySelector ≥ 0.6 ∧ at least two of {that novelty, rhythm ≥ 2, dormant wake, sweep}. It is a signal ("possible key change or a silent swap of the agent behind the wallet"), never a verdict.

#4. Overall score, grade, caps

score = round(Σ wᵢ·sᵢ / Σ wᵢ) over measurable checks. Caps: funds sent to a listed address ⇒ ≤ 40; any listed counterparty ⇒ ≤ 60; behaviour shift ⇒ ≤ 60. Grade: A ≥ 90, B ≥ 80, C ≥ 65, D ≥ 50, else F; NR when fewer than 5 transactions or nothing measurable. Flags (funds-sent-to-flagged-address, flagged-counterparty, sudden-pattern-shift, sandwiched, approval-to-flagged-address) are listed beside the score.

#5. Determinism, hashing, attestation

Everything is a deterministic function of (wallet, chain history, label lists, Sourcify, method version). The report hash is sha256 of the canonical JSON of the report without generatedAt and the sliding window bounds, plus the digest of the analysed transaction hashes: re-grading an unchanged wallet yields the same hash and writes no new row. A new report content ⇒ a new row; the newest row per wallet is current. Reports with a score are attested in batches on Ethereum Sepolia EAS (summary + reportHash only; evidence stays in the report the hash commits to). Changing any threshold or weight bumps METHOD_VERSION; reports of different versions are not comparable.

#6. What it does not do (stated on every report)

• Sent-transaction history only. A wallet that mostly receives, or acts through a contract wallet, is not graded.
• The 90-day / 500-tx window, and the ≤12-swap sample, bound what a report knows. coverage.notes says when a cap bit.
• Swap replay: single-hop Uniswap V3 on Base only. Ethereum reports (explorer index without logs) have no swap, MEV or tip checks.
• Open scam lists are evidence that an address was reported; they are chain-agnostic address lists and can lag or err.
• A good score means the wallet behaved well, not that the agent is competent. It does not replace a trial-based certificate.
• Public RPCs can rate-limit or prune. Failures are retried across endpoints; a wallet that cannot be read is skipped and retried on a later run, never scored from partial data.

#8. What would raise coverage

• A CDP SQL API key (base.transactions, free tier 1,000 queries/month, batched by wallet list) would give Base incoming history and remove the per-wallet bisection cost.
• A dedicated Anvil fork for the Mirror backend (BACKTEST_MIRROR_URL) makes the re-simulation run on a real EVM fork; the archive eth_call backend gives the same quotes for the same block.
• Multi-hop and Aerodrome/Slipstream venues for the swap replay.