Underwriting: pricing the risk that an agent fails a task

Status: testnet pilot (Ethereum Sepolia). Model family uw-hbb-1. Code: packages/intel/src/underwriting/, API services/api/src/routes/underwriting.ts, contracts contracts/src/PolicyManager.sol + GuaranteePool.sol, page /underwriting on the Explorer.

This document says what is estimated, from what data, how a premium is built, what the chain enforces, how the model was tested, and where the numbers must not be trusted. Read the last two sections before reading any price as more than a pilot.

#1. What is insured

A policy covers one KaratumEscrow job. The insured event is the one on which the oracle path already acts: the evaluator rules FAILURE or PARTIAL and the escrow rejects the job (JobStatus.Rejected). If that happens the policy pays its coverage to the job's client. If the job completes (SUCCESS) or lapses unresolved, nothing is paid and the premium stays with the pool's underwriters.

What a policy is not: it does not cover a verdict that was later revoked, a dispute about the verdict, an agent that is slow, or a loss that is not a failed verdict. The escrow refund of the job amount is separate from the policy and always happens on a reject.

#2. Data

Source: the verdict store (attempts joined to tasks), one row per (trial, task, build) with the verdict status, the agent's own claim, and the finish time. A row is an observation when the verdict is SUCCESS, FAILURE or PARTIAL. UNKNOWN, DISPUTED and REVOKED verdicts do not decide whether the escrow would reject, so they are excluded and their count is published with the model.

Development sample behind the figures in section 8 (a snapshot, see GET /v1/underwriting/model for the live numbers): 4,792 decided verdicts (124 excluded: 72 revoked, 52 unknown), 17 builds, 16 task types, 80 non-empty build x task-type cells, 3 to 12 October 2026. The trials run as automated campaigns, so verdicts arrive in bursts, not as a steady stream.

Difficulty tier. Each task type belongs to one tier (routine, adversarial, frontier, code; unclassified for types we have never seen), defined in difficulty.ts by how the suites were designed. The tier is a fixed function of the task type, so build x task type x difficulty is not a three-dimensional grid: difficulty is an upper pooling level (tier, then task type, then the build's own cell), not an independent axis. We say so rather than present 3-D cells.

#3. The risk model

Target: p(build, task type) = probability that a verdict on that build and task type is FAILURE or PARTIAL.

Hierarchical Beta-Binomial with empirical-Bayes pooling. Each level is pulled towards the one above it with a strength measured in pseudo-observations:

Consequences that matter for pricing: a build with a few clean outcomes is not priced at zero; an empty cell uses the task type's rate shifted by how the build does elsewhere; an unseen task type starts from its tier (and a weaker prior, so it can move fast); an unclassified type starts from the global rate. The failure probability used for pricing is the posterior mean, floored at 0.25%. The 90% credible interval is the central interval of the cell's Beta posterior.

Recency. Observations are weighted 0.5^(age / halfLife), with age measured against the newest verdict in the data, so a pause in the pipeline does not erase everything at once. The half-life, the pooling strengths and sBuild are the hyper-parameters. They are chosen by a grid search on training data only (inner rolling-origin log-loss), refit per fold in the backtest, and the stored run for production uses the same search on all data. Half-lives below 3 days are excluded from the grid on purpose: in the 9-day development sample they score better (pooled log-loss 0.235 against 0.259), but they would erase any record older than a few days, and trials run in campaigns separated by days, so a build could not be priced between campaigns. That is a design constraint, not a fit.

Limits of the model itself. Attempts are treated as independent although several builds run the same tasks and builds are re-run on the same suites, so effective sample sizes are smaller than the counts. There is no task-level difficulty. The credible intervals condition on the fitted hyper-parameters and on the task-type and tier rates, so they are somewhat too narrow (the backtest measures by how much).

#4. Adjustments that are assumptions, not fits

Three adjustments exist because a pricing model for agents should react to them, and because the history to estimate them does not. They are stated as assumptions, sized conservatively, and tested against held-out data in section 8 (which is where they may be contradicted).

• Fingerprint drift. The newest behavioural-fingerprint drift d (0 to 1) discounts the build's own evidence by 1 / (1 + (d / 0.25)^2), pulling the cell back towards the pooled prior. Fingerprints exist for few builds, with a handful of readings each, and none as of an earlier date, so this cannot be backtested.
• Certificate state. A STALE certificate keeps 70% of the evidence, EXPIRED 50%, REVOKED 25%. Certificates are recomputed daily and only the latest state is stored, so there is no history to test it on.
• Staleness. Days g since the build's newest verdict keep 1 / (1 + (g / 14)^2) of its evidence (half after 14 days of silence). Recency inside the data is fitted; a silent data set is not something any window can teach.
• Over-claim. The share of times the build claimed success and the verdict was not (smoothed Beta(1, 4); the Honesty Index value from migration 018 is used when present) raises the margin on expected loss by up to 2x that rate. It prices the cases the buyer's own monitoring would not catch. Section 8 reports whether held-out failures are ordered by it; so far they are not cleanly ordered, so it is a precaution.

#5. From probability to premium

premium  = expected loss + risk load + capital cost            (never below the minimum, never above coverage)
expected loss = p x coverage
risk load     = coverage x [ rho x p x (1 + 2 x overClaim)  +  zeta x (pHi - p) ]
capital cost  = coverage x costOfCapital x days / 365

Defaults (all in the model card): rho 15% margin on expected loss; zeta 0.5 of the gap between the upper 90% bound pHi and the mean, so thin evidence is priced part-way to its upper bound; cost of capital 12% a year, because the pool reserves the full coverage while a policy is open; 7-day default horizon (the quote request can override it). Rates are applied in parts per million, rounded up, in integer token units.

Maximum coverage is the per-policy cap (1,000 tUSDC on the testnet pool), scaled down by effective evidence nEff (after recency and the adjustments above): full at 100 or more, 50% at 30, 20% at 10, 5% below that. It is zero for a build with no verified outcomes, and zero when pHi is above 50% ("declined"). It is also bounded by what the pool can still reserve for policies. Required bond is coverage x pHi: the bond an agent should hold for its first-loss stake to match the upper-bound expected loss. In v0 the policy payout does not touch the AgentBond (section 7), so this is a recommendation attached to the quote, not an enforced condition.

#6. The quote

POST /v1/underwriting/quote with {buildId, taskType, coverage} returns premium, rate, required bond, maximum coverage, the failure probability with its 90% interval, the evidence (cell counts, effective evidence, pooling chain, adjustments, premium breakdown, notes in plain words), the model version uw-hbb-1+<digest> and the dataset digest. coverage and all amounts are token base units (tUSDC, 6 decimals). Also: GET /v1/underwriting/model (model card, assumptions, backtest), GET /v1/underwriting/risk-table, GET /v1/underwriting/policies.

Add jobId, buyer and provider (a funded KaratumEscrow job whose client is buyer) and the answer carries an EIP-712 signed quote the buyer submits to PolicyManager.buy:

domain  { name: "KaratumUnderwriting", version: "1", chainId, verifyingContract: PolicyManager }
Quote(address buyer,address provider,uint256 jobId,uint256 coverage,uint256 premium,uint64 validUntil,bytes32 quoteId,bytes32 riskHash)

quoteId is random and single-use on-chain; riskHash commits to build, task type, model version and the priced probabilities (an audit trail, never interpreted on-chain); the quote is valid for 10 minutes. Every signed quote is logged in underwriting_quotes.

Controls in the signer, before it signs: coverage above the evidence-scaled maximum, or above the pool's room, or a declined risk, gets no signature (the response says why); if the priced build has a registered wallet it must equal the job's provider (PROVIDER_BUILD_MISMATCH otherwise); with UNDERWRITING_REQUIRE_BINDING=1 a build without a wallet is refused (PROVIDER_NOT_BOUND); and the coverage signed for one provider in 24 hours is capped (default 3 x the per-policy cap).

Known gap, stated plainly. None of the benchmarked builds has a registered escrow wallet today, so on the pilot a signed quote prices the named build as a stand-in for the job's provider and says so in the quote's notes. The e2e below uses exactly that. Production would run with UNDERWRITING_REQUIRE_BINDING=1, which needs agents to register the wallet that signs their jobs.

#7. On-chain

Ethereum Sepolia, deployed 16 October 2026 (contracts/deployments/sepolia.json, key underwriting):

• GuaranteePool v2 0x1aA4c5cE96dD2b31b92Bd1E4388957061D1F1842: the existing pool contract plus an optional policy module, a separate instance with the old pool's parameters, the same AgentBond and escrow, its own (initially empty) capital. The older pool at the top level of sepolia.json is untouched and keeps serving the claim path.
• PolicyManager 0x75cA4B98111631228A5346C364F62feE80D926D9.

Flow. buy(quote, signature) (buyer = the job's client, job still Funded, provider and token match, quote fresh and unused, one policy per job, coverage within maxCoverage, reserved coverage within maxUtilizationBps of the pool's assets, at least minLead before the job's deadline) pulls the exact premium, adds it to the pool as underwriter income through fund(), and reserves coverage in the pool. settle(policyId) is a permissionless crank: if the escrow says Rejected the pool pays the job's client directly (credited instead if the transfer fails, so a payout cannot be blocked), limited per epoch by the pool's payout cap with the remainder paid by later calls; if the job Completed, was Reclaimed, or can no longer be resolved, the reserve is released and the premium stays. No claim, no challenge window, no arbiter: the payout follows the same oracle verdict the escrow already acts on.

Solvency: coverage is reserved one-to-one inside the pool's existing totalReserved, so balance >= owed and totalReserved <= totalAssets keep holding; redemptions are priced as if every open policy will pay. That is capital-inefficient and provably solvent. A job with a policy cannot also be claimed through fileClaim (no double recovery). The pool re-checks the escrow state itself and only ever pays the job's client; the manager is trusted for limits and the signature only.

Tests: 268 Foundry tests pass (unit, fuzz, handler-based invariants with fail_on_revert), forge fmt clean; PolicyManager.sol 100% line and branch coverage. The security review (threat, mitigation, residual risk) is in contracts/README.md. Residual risks that matter for reading any price:

• One trusted signer sets the price. The chain enforces the signature and the limits (per-policy cap, utilisation, free capital), not the price. A compromised signer key can sell cheap policies up to those caps until the owner rotates it or pauses.
• Self-dealing. A client that also controls the provider could fail its own job and collect coverage - premium. Bounded by the caps and the signer's provider-exposure limit; not eliminated. A per-provider on-chain cap and subrogation against the agent's bond are the obvious next steps.
• Adverse selection through the mempool. A client who sees the evaluator's reject pending could buy a policy first. minLead and the 15-minute quote limit narrow the window; they cannot close it while the evaluator may resolve a Funded job at any time.
• No subrogation. A policy payout does not draw on the failing agent's AgentBond.
• Owner powers (rotate the signer, swap the manager, lower the payout cap) must sit behind a multisig and timelock before mainnet.

#8. Backtest

Protocol. Time-ordered, expanding window: four folds that each train on all verdicts before a cut and predict the next 12.5% of the data (about 600 verdicts, 1 to 2 days) with the model frozen at the cut, so staleness is paid for. Hyper-parameters are selected per fold on that fold's training data alone. Nothing from a test window reaches the model that predicts it (a unit test checks that pure noise yields no skill). Baselines get the same folds: global rate, per task type, per build, and per build x task type without pooling (all Laplace-smoothed).

Results on the development sample (2,396 held-out verdicts, 401 failures, observed failure rate 16.7%; model uw-hbb-1+04c2319d):

The model has the best log-loss, Brier and discrimination. It is not best on every fold (in fold 2 the plain per-task-type rate scored 0.296 against 0.314) and the unpooled cell estimate is better calibrated on average. The 90% predictive interval of a held-out cell's failure count covered 87.9% of 107 held-out cells (88.7% without the one task type that fails for everyone), slightly under the nominal 90%.

What it got wrong, in the observed-over-expected failure ratio (1.0 = calibrated):

• Task types never seen in training: 7.1 (23 failures where 3.2 were expected; 83 verdicts). The model cannot know a new, harder suite is coming.
• The code tier: 5.7 in the development run (87 failures where 15 were expected): the first small code trials were easy and the generated ones that followed were not.
• Evidence 1 to 3 days old: 4.6 against 1.2 for fresh evidence, which supports a staleness discount, though it is confounded with the two points above.
• Over-claim terciles: 1.4, 1.5, 1.0 for low, middle, high: no ordering in the direction the adjustment assumes. The over-claim margin is therefore not supported by this sample.
• The routine and adversarial tiers were slightly over-predicted (0 failures where 0.4 expected; 0.67); the frontier tier was calibrated (1.0).

Pricing realism: selling one unit of coverage on every held-out verdict at the model's own price, the average premium was 18.1% against a realised loss rate of 16.7%, a loss ratio of 0.92 on the full premium and 1.19 on the pure expected-loss part. By predicted-risk bin the loss ratio ranges from 0.09 (0 to 1%) to 3.1 (5 to 10%). A portfolio total near 1 hides mispriced bins.

Limits of this evidence. Nine days, one lab, mostly one family of agents. Folds are 1 to 2 days long. The data is non-stationary: new suites changed failure rates mid-sample, which is what the model failed on and what pooling only partly absorbs. The backtest cannot say how the model behaves over months, on agents or tasks outside the suites, or against anyone trying to game it. Do not read the loss ratio as profitability.

Reproduce: pnpm --filter @karatum/intel exec tsx src/underwriting/backtest-cli.ts [--csv attempts.csv] [--save] --out report.json (DATABASE_URL for the store; --save writes underwriting_model_runs, which the API and the Explorer pick up). Unit tests: pnpm vitest run packages/intel/src/underwriting services/api/src/routes/underwriting.test.ts.

#9. Sample size, in plain terms

• 80 build x task-type cells; the median cell has about 30 verdicts and a quarter have fewer than 15. A cell with 0 failures in 14 is not priced at 0%: its mean sits near the pooled rate and its 90% interval spans several percentage points.
• Coverage limits shrink with effective evidence precisely because of this: below 10 effective outcomes at most 5% of the per-policy cap is written.
• Premium rates on this testnet are not market prices and not forecasts for production traffic. Test tokens only.
• When new verdicts arrive the model refits (the API every 5 minutes) and the version string changes; a signed quote pins the version it was priced with.

#10. Versioning and changes

uw-hbb-1 is the model family. The version string is uw-hbb-1+<first 8 hex of the dataset digest>: the digest covers the cell counts, the newest verdict time and the hyper-parameters, so two parties can check they priced from the same data. Changing the model structure, the adjustments or the pricing constants changes the family (uw-hbb-2) and is recorded in this document with the backtest of the change.

#11. Sepolia end-to-end (16 October 2026)

scripts/e2e-underwriting.ts, run through the public API and the production evaluator. Both jobs are 100 tUSDC escrow jobs asking the provider for a 25 tUSDC transfer; each policy covers 40 tUSDC for a premium of about 1.2 tUSDC, priced with 1 day horizon from the stand-in build ext-claude-code-radikws-haiku-v1 / web3.transfer (0 failures in 14 outcomes, 2 effective, so coverage was capped at 5% of the limit and the quote says so; the provider binding was not verified, as in section 6). Underwriter capital in the pool: 2,000 tUSDC. Explorer: /underwriting (table "Policies on Sepolia").

The first run of the failure scenario completed on-chain but my balance check read a lagging RPC node and reported a false failure; the script now pins reads to the receipt's block and checks the Transfer logs of the settle transaction. Two policies on test tokens show that the mechanism works end to end; they say nothing about pricing quality.