Methodology
Challenge Market
On this page
- Two kinds of challenge
- Verdict challenge: "this verdict does not follow from its trace"
- Certificate challenge ("short the agent"): "this agent no longer performs at its certificate's lower bound"
- Money
- Escalation to a validator vote
- Anti-griefing
- Trust assumptions (v0)
- Security self-review of ChallengeManager
All documents
Methodology
- Agent Honesty Index (AHI) — methodology
- Verifiable judge: who checks the checker?
- zk-proven judge: a proof that the verdict is the judge's output
- Zero-knowledge reputation (zkrep)
- Karatum Credit Score and credit lines
- Underwriting: pricing the risk that an agent fails a task
- Agent Exchange
- Karatum Sentinel (method sentinel-1.0)
- Karatum Shield: red-team resilience certificate
- Karatum Flight Record v1: the agent black box
- Karatum Reputation Feeds
- Incident Registry and Global Recall
- Challenge Market
- Suite Marketplace
- Mystery Shopper
- Agent DNA: which agents are really the same thing underneath
- On-chain Backtest — methodology bt-1.0
- Protocol Agent-Readiness Audit
- Compliance Evidence Pack
Specifications
Integrations
Source files
An optimistic oracle for Karatum's own outputs. Anyone can put a bond against a Karatum verdict or certificate and force a re-check. A challenger who is right gets its bond back plus most of Karatum's matching stake; a challenger who is wrong loses its bond. The point is that the truth of a verdict or a certificate stops depending on trusting Karatum alone: the money is in a contract, every re-check is reproducible from public data, and the verdict path can be settled by a zero-knowledge proof without Karatum at all.
Everything runs on testnets: ChallengeManager on Ethereum Sepolia, bonds in test USDC (open mint), re-trials on a private Anvil fork of Base. No real money is involved.
Two kinds of challenge
Verdict challenge: "this verdict does not follow from its trace"
Since October 2026 every verdict is a pure function of a stored trace: verdict = judge(trace) (see verifiable-judge.md). A
verdict is registered on-chain with its attested verdictHash and its traceHash. The resolver re-runs the published evaluator
bundle on the stored trace and compares:
- the recomputed trace hash with the registered trace hash, and
- the recomputed verdict hash with the registered (attested) verdict hash.
Both equal: the challenger loses. Either differs, or the trace cannot be judged: the challenger wins, the verdict is revoked in the store and its EAS attestation is revoked.
Trustless settlement. When the verdict is in a batch that ZkJudgeVerifier has proven (an SP1 Groth16 proof that
(traceHash, verdictHash) = judge(trace) for every trace in the batch), anyone can call
resolveVerdictByProof(challengeId, root, merkleProof). The contract asks the verifier itself and, on level 4 (ZK_PROVEN),
settles the challenge against the challenger at once, with no dispute window and no vote. The resolver prefers this path
whenever a proof exists. It answers exactly the challenge's claim (the verdict follows from its trace). It does not prove the
trace is what happened on-chain: evidence collection is outside the judge (see zk-judge.md).
Eligible: public verdicts with a stored trace, attested on Sepolia, not revoked, within 30 days of the decision.
Certificate challenge ("short the agent"): "this agent no longer performs at its certificate's lower bound"
A certificate claims a build's success rate is at least ciLow, the lower end of the 95% Wilson interval over its verified
outcomes. The resolver runs a fresh re-trial:
- When it picks the challenge up it draws a random 32-bit seed and a 32-byte salt and publishes only
seedCommitment = keccak256(abi.encode("karatum.retrial-seed/1", challengeId, seed, salt)). - The seed generates a new frontier web3 task set (
generateFrontierTrial(seed)); 24 tasks are taken in the order ofsha256(seed:taskId). Nobody, Karatum included, knows the tasks before the commitment is published, and the challenger cannot pick them. - Each task runs on a private Anvil fork of Base (pinned block 52136000, the block of the frontier suite; nothing is broadcast)
from the same snapshot, and every verdict is decided by the published judge from its trace. Re-trial verdicts are stored in
challenge_retrial_attemptswith their full traces and are never written toattempts: they cannot feed ratings, certificates or attestations. - With
kverified successes (SUCCESS only) out ofntasks, the challenger wins iff the upper end of the re-trial's own 95% Wilson interval is below the certifiedciLow. Even the most generous reading of the new evidence cannot reach the certified floor. The rule leans towards the certificate on purpose: a borderline drop (say 20/24 against a 0.90 floor) is not enough. At least 12 decided tasks are required; harness errors are reported and not counted. - The evidence bundle (claim, seed, salt, commitment, fork block, scenario, every verdict hash and trace hash, the decision) is
published at
/v1/challenges/:id/evidence, andkeccak256(canonical JSON)is the evidence hash posted with the resolution. Every trace is at/v1/challenges/:id/retrialfrom the moment the resolution is posted (not before, so the seed stays hidden).
A won certificate challenge:
- opens an Incident Registry entry (class
cert_revoked, the same dedupe key the registry's own revoked-certificate detector uses, the drill flag carried over) and confirms it; - when the build has wallets with Guard leaves, runs the incident recall (revokeCert on every Guard registry, IncidentLog);
- sets the certificate
REVOKEDin the store. A trigger in migration 030 keeps it REVOKED (and keeps its dates) when the daily certify job rewrites certificates, until a maintainer lifts the revocation (karatum-challenges lift <id> <note>). The next Guard root no longer carries it; - marks the subject
Overturnedon-chain, permanently.
Which builds can be re-tried today. Karatum's deterministic scripted solvers (scripted-reference-frontier-v1,
scripted-naive-frontier-v1) are re-tried at once: they need no model. Every other build needs an LLM. Karatum's LLM gateway
(the LLM inference host) is down, so those re-trials are queued and the challenge shows pending_llm. If a queued re-trial cannot
run before the resolve deadline, anyone can expire the challenge and every bond goes back where it came from.
Eligible: VALID certificates, until they expire. A drill may register a certificate record of a Karatum house build that is not VALID; it is labelled as a drill everywhere.
Money
- Bond =
max(10 tUSDC, 1% of the subject's value). A verdict is valued at 0 (bond 10 tUSDC); a certificate at 100 tUSDC per karat (a 23-karat certificate takes a 23 tUSDC bond). The bond is exactly this amount, never more, so a large bond cannot lock the defender pool. - Defender match. Opening a challenge reserves the same amount from Karatum's defender pool (2,000 tUSDC at launch). No pool, no challenge: Karatum must have stake behind its claims.
- Payout. Winner: own stake back plus 80% of the loser's stake. Treasury: the other 20%. Challenger loses: the defender pool
gets its match back plus 80% of the bond. Credits are withdrawn with
withdraw, which keeps working while the contract is paused. - Expiry. Resolver silent past the deadline (2 days): anyone calls
expire, the bond is credited back and the match returns to the pool.
Escalation to a validator vote
A challenger can dispute a resolution against it (not a zk-settled one) within the dispute window (1 hour on Sepolia) by
escalate, posting an escalation bond equal to its bond. For 1 day, validators allowed by the owner and staked in
ValidatorStaking vote; a vote weighs the validator's stake minus any pending unstake (minimum 100 KARAT). The challenger and
the resolver cannot vote. After the deadline anyone calls settleVote:
- quorum (600 KARAT voted) and a strict overturn majority: the challenger wins everything as above, escalation bond included;
- quorum and no overturn majority (ties uphold): the resolution stands, the challenger loses bond and escalation bond;
- no quorum: the resolution stands and the escalation bond is refunded (validators not showing up is not the challenger's fault).
Validators re-derive the outcome from public data only: karatum-challenges vote <id> fetches the public verdict record or the
evidence bundle and every re-trial trace, checks the bundle hash against the on-chain resolution hash, re-runs the judge on every
trace, re-applies the rule and votes. A bundle that does not hash to the posted value, or a trace that does not recompute, is a
vote for the challenger.
Anti-griefing
One active challenge per subject; a 1-day cooldown on a subject after a challenge against it fails; a 3-day cooldown on a challenger who lost; bonds that scale with the value at stake; the resolver key cannot open challenges; subjects can only be registered by the resolver, and only with facts re-checked from the store (registration requests through the API are rate-limited like every API call).
Trust assumptions (v0)
- The resolver (dedicated key
0xD20C…e60b) registers subjects and posts resolutions. It cannot touch bonds or credits, cannot resolve after the deadline, cannot stop a zk settlement and cannot stop withdrawals. A wrong resolution can be escalated (except a zk one, which is not a Karatum statement at all). - The owner sets parameters, the treasury and the voter allow-list. Voter weights come from ValidatorStaking, but who may vote is curated: an open voter set would let anyone stake just before a vote. Today the three validators are Karatum's own nodes, so the vote is not yet independent of Karatum; that changes as outside validators stake.
- Re-trials of scripted builds are fully reproducible; a re-trial of an LLM build would be a fresh, non-deterministic sample (its traces are still judged by the same pure judge).
Security self-review of ChallengeManager
- Accounting:
token.balanceOf(this) == defenderPool + defenderReserved + bondsHeld + totalCreditson every path; checked by a 1,000-run fuzz over all four resolution paths (win, loss + finalize, expiry, escalation with either vote result) and by_assertSolventin every unit test.recoverExcesscan only take the bond token abovetotalOwed. - Reentrancy: every state-changing function that moves tokens is
nonReentrant(transient guard); bonds are pulled withExactTransfer.pullExactafter state is written, so fee-on-transfer tokens revert (tested) and a reverting pull rolls back everything. - Payouts are pull-based credits; no external call to a user-controlled address happens during resolution.
- No funds can be frozen: expiry needs no one but time; finalize, settleVote and withdraw stay available while paused; the owner cannot renounce (that would strand the pool).
- Time: deadlines are uint64 seconds via SafeCast; windows bounded (resolve 1 h to 30 d, dispute and vote 10 min to 14 d, cooldowns up to 30 d).
- Votes read
ValidatorStaking.getValidator(a view) at vote time; double votes, votes after the deadline, and votes by the challenger or resolver revert. The arithmetic (votesOverturn > votesUpholdwith quorum) cannot overflow (uint128 weights). - zk path: the contract trusts ZkJudgeVerifier's
check(no owner, program keys allow-listed in the EvaluatorRegistry). A verifier answer of anything other than level 4 reverts with its reason code. - Known limits: the resolver and owner are single keys (testnet); voter allow-listing is owner-curated; the subject value (bond scaling) is set by the resolver from store facts.
- Tooling:
forge test --match-path 'test/challenges/*'(31 tests including the fuzz),forge lint src/challenges(notes only: the two small view interfaces live in the same file).