Verifiable judge: who checks the checker?

Karatum's verdicts decide payouts, bonds and reputation, so the code that produces them has to be checkable by anyone, not trusted. This document explains how a verdict proves which code produced it, what that proof does and does not cover, and how to re-check a verdict yourself.

Status (2026-10-16): live on Ethereum Sepolia and karatum.com for every new verdict. Verdicts decided before 2026-10-16 have no trace (see Re-verification results).

#The idea in one line

verdict = judge(trace)

judge is a pure function: no clock, no network, no randomness, no environment. Its input, the trace, contains the task spec and the evidence the runner collected. Its code is one published JavaScript file, the evaluator bundle. The bundle's SHA-256 is the evaluator digest. Anyone who has the trace and the bundle gets the same verdict hash that Karatum attested on-chain.

#Components

Both contracts are Sourcify exact_match. Deployment details are in contracts/deployments/judge-sepolia.json and the judge object in contracts/deployments/sepolia.json.

#Hashes

• Trace hash = sha256(canonicalJson(trace)). Canonical JSON sorts keys, omits undefined members and encodes bigints as {"$n":"<decimal>"}. JSONB key reordering therefore never changes the hash.
• Verdict hash = sha256(JSON.stringify([trialId, taskId, buildId, status, evidenceClass, checks[name,pass,expected,actual], sorted txHashes])). This is the bytes32 already attested in every Karatum verdict attestation. A test pins attemptVerdictHash to packages/store verdictHash.
• Evaluator digest = sha256(karatum-judge.mjs).
• TEE report data = sha256(evaluatorDigest ‖ traceHash ‖ verdictHash), zero-padded to 64 bytes. A quote over it says: this measured machine ran this evaluator on this trace and got this verdict.

#Reproducible build

The evaluator is reproducible without Nix or Docker. The lockfile pins every dependency. build.mjs uses a fixed esbuild version (0.28.2) and repo-relative paths, writes no timestamps and does not minify. The manifest records inputsDigest, a digest over all 113 input files, so two builds can be compared input by input.

Release 1.0.0:

Measured reproducibility:

• The bundle built on macOS arm64 (Node 25) and on the Linux x64 server (Node 22) is byte-identical (same digest above).
• The test packages/judge/src/judge.test.ts builds the bundle twice per run, compares the bytes and checks that the bundle judges every fixture exactly like the TypeScript source.
• The image built twice with --no-cache and rewrite-timestamp=true, SOURCE_DATE_EPOCH=1792100419 (the commit time) gave the same OCI manifest digest and the same tarball sha256.

git checkout 804cee374f3e47016e2e2126c77ace20816de77c && pnpm install --frozen-lockfile
node packages/judge/scripts/build.mjs --check 0x6f1b3cdd7035afc9797e9895db335e1bfb1f3b18b9efffa95eea7532e12adcbe
docker buildx build --build-arg EVALUATOR_DIGEST=0x6f1b3cdd…cbe --build-arg SOURCE_DATE_EPOCH=1792100419 \
  --output type=oci,dest=karatum-judge.oci.tar,rewrite-timestamp=true packages/judge

To publish a new release, run these on the server:

1. node packages/judge/scripts/build.mjs (with SOURCE_COMMIT set).
2. Build the image.
3. tsx packages/judge/src/ops-cli.ts release --image-digest sha256:…. The owner key calls EvaluatorRegistry.allow, and the release is recorded in evaluator_releases.

#Trust boundary

Be precise about what each layer proves.

Level 3 (verifyWithQuote) means Automata's on-chain DCAP verifier accepted the quote whose keccak256 is teeQuoteHash. JudgeVerifier does not parse the DCAP output, so the measurement and report_data must still be checked off-chain from the quote bytes (verifyTdxQuote). Full on-chain DCAP verification costs roughly 4–5M gas per TDX quote, which is why it is optional and done per verdict on demand.

What the level 3 label (LEVEL_TEE_VERIFIED, reason TEE_VERIFIED) does and does not mean:

• It means: some genuine Intel TDX platform produced a quote with exactly these bytes (Automata's DCAP verifier checked the certificate chain and signature on-chain), and the provenance attestation names that quote by hash.
• It does NOT mean, on-chain: that the quote's measurement is on the EvaluatorRegistry measurement allow-list, or that its report_data equals sha256(evaluatorDigest ‖ traceHash ‖ verdictHash). JudgeVerifier checks neither, because it does not parse the DCAP output. A quote from an unrelated TDX workload passes the on-chain step.
• So a consumer that relies on level 3 must also check the measurement and report_data off-chain from the quote bytes (verifyTdxQuote in packages/judge/src/tee/tdx.ts, or the re-verify page), or treat level 3 as "a TEE quote exists" and nothing more. Level 2 (TEE_CLAIMED) is weaker still: only Karatum's attester says a quote exists.
• Level 4 (ZK_PROVEN, ZkJudgeVerifier) does not have this gap: the proof binds the judge program, the trace hash and the verdict hash on-chain.

#Attested execution: pluggable backends

KARATUM_TEE_BACKEND selects the backend: reproducible-only (default), tdx-dstack, tdx-configfs or nitro. The judge and the verdict are identical across backends. Only the evidence attached to the trace differs: verdict_traces.tee_*, and teeQuoteHash / teeMeasurement in the provenance.

Tests (packages/judge/src/tee/tee.test.ts) build structurally exact TDX v4 quotes and Nitro documents through TEST-ONLY certificate chains (fixtures/gen.sh). They cover acceptance, tampered bodies, wrong report data, an unbound attestation key, bad COSE signatures and foreign roots. They also show that the production roots reject the synthetic chains. No quote from real hardware has been verified yet: none was available without an account (see below).

#What it would cost, and what is still open

Researched 2026-10-16. Nothing was signed up for or spent. No TEE was found that issues a genuine hardware quote with neither an account nor a card.

Public free verification without an account: Phala's POST https://cloud-api.phala.com/api/v1/attestations/verify (multipart file= or hex=), with shareable reports at proof.t16z.com.

#How to re-verify a verdict

1. Browser. Open https://karatum.com/verify/<verdict-id> (also linked as "Re-verify this verdict" on each verdict page) and click Re-verify this verdict. The page downloads the record and the bundle. It executes the bundle only if sha256(bundle) equals the evaluator digest, then compares the trace hash and the verdict hashes. Nothing leaves the browser.
2. Zero-dependency kit (Node ≥ 20): curl -sO https://karatum.com/verify/kit.mjs && node kit.mjs '<verdict-id>'. Exit code 0 = MATCH, 1 = MISMATCH, 2 = no trace or error.
3. From source: pnpm --filter @karatum/judge build, then pnpm verify-verdict '<verdict-id>'. This also reads the Sepolia verdict attestation, the provenance attestation, JudgeVerifier.checkVerdict and EvaluatorRegistry.isAllowed over a free public RPC. Use --offline to skip the chain checks and --bundle <file> to run a specific bundle.
4. Container: curl -s https://karatum.com/verify/<id>/record | docker run --rm -i --network none karatum-judge:1.0.0

Public endpoints:

• GET /verify/<id>/record: the stored verdict, the full trace, attestation and provenance UIDs and the release.
• GET /verify/evaluator/<digest>.mjs: bundle bytes, served only if they hash to the digest.
• GET /verify/kit.mjs

#Re-verification results

Run reverify-2026-10-16 (rows in judge_reverify; the latest run is shown on karatum.com/verify).

A. 100 traced verdicts, recomputed: 100 / 100 match.

• Where they come from. Production LLM trials were paused on the LLM inference host quota, so no new LLM verdicts were produced on 2026-10-16. The traced verdicts therefore come from trial judge-sample-2049: the 67-task frontier web3 suite (seed 2049), solved by two scripted, LLM-free builds (scripted-reference-frontier-v1, scripted-naive-frontier-v1) on a private Anvil fork of Base at block 52136000.
• The trial. 134 verdicts: reference 67 SUCCESS; naive 59 FAILURE, 4 PARTIAL, 4 SUCCESS. The runner decided each one with bundle 0x6f1b3cdd…cbe and stored its trace. All 134 are attested on Sepolia in one EAS tx, . Each also has a provenance attestation: 7 txs, the first . The schema was registered in .
• What was checked. A random sample of 100 was recomputed with the published bundle. 100 / 100 passed every one of these checks:
  • trace hash
  • task-spec hash, equal to the trial's stored tasks.spec
  • verdict hash equal to the judge's hash at decision time
  • verdict hash equal to the stored hash
  • same evaluator build as the runner
  • Sepolia EAS verdict attestation equal to both the stored and the recomputed hash
  • provenance trace hash and evaluator digest
  • JudgeVerifier.checkVerdict = level 1 REPRODUCIBLE
  • EvaluatorRegistry.isAllowed
• Second, independent path. The same 100 records went through the karatum-judge:1.0.0 container with --network none: 100 / 100 verdict hashes match.
• Example. judge-sample-2049:injected-04:scripted-naive-frontier-v1 (FAILURE: the naive solver paid the injected address):
  • verdict attestation
  • provenance attestation
  • re-verify page

B. 100 historical verdicts (no trace): 100 / 100 integrity match. They were not recomputed.

• Why only an integrity check. Verdicts decided before 2026-10-16 (4,916 rows, all attested) never had their raw evidence stored, and their Mirror forks are gone. They cannot be recomputed by any evaluator.
• The sample. 100 random verdicts from 20 trials: 70 SUCCESS + 6 FAILURE web3, 21 SUCCESS + 3 PARTIAL code. The 72 REVOKED rows were excluded because their status was changed after the fact.
• What was checked:
  • the stored fields re-hash to the stored verdict hash (packages/store verdictHash)
  • the status is consistent with its checks
  • the Sepolia EAS attestation carries the same hash, status and ids, and is not revoked
• Result. 100 / 100. This shows the stored and attested records are intact and internally consistent. It does not show that the evidence behind them was judged correctly.

Mismatches found: none. Bugs found along the way: none in the verdict path. In the existing code the judge's attemptVerdictHash and packages/store verdictHash agree bit for bit, and a test pins this.

#Limitations and open items

• Source availability. The bundle is public and unminified at /verify/evaluator/<digest>.mjs, and the kit runs it without the repository. The GitHub repository is currently private, though, so "rebuild from the source commit" is only possible for people with access. The owner should publish the evaluator sources (packages/judge, packages/validator, packages/kos and the lockfile), for example as a public mirror, so the source → digest step is independently checkable.
• Historical verdicts (before 2026-10-16) have no trace. Their raw evidence was never stored and their Mirror forks are gone. They can only be integrity-checked (see above), never recomputed.
• Evidence collection is outside the judge until the runner runs inside a TEE (see Trust boundary).
• Revoked verdicts. REVOKED is set after the fact. The trace still recomputes the original decision, and re-verification reports that instead of a mismatch.
• Registry owner. The registry owner is a single EOA on testnet. On mainnet it should be a multisig behind a timelock, like the other Karatum contracts.