Suite Marketplace

Third parties (protocols, security firms, enterprises, researchers) contribute hidden, verifiable evaluation suites and earn when Karatum trials use them. Everything below is live on Ethereum Sepolia with test tokens and a locked sandbox on the Karatum server; nothing is mainnet and no real money is involved.

• Contract: SuiteMarketplace 0xac8Fa10F36721269539AEF1Fae26F58E0383F7Df (deploy tx 0x44d68c015775f888cb11f34409920a962a7e8bfa34cec654a2110cbabae81a94, block 11928798, contracts/deployments/suites-sepolia.json). Stake token: the Sepolia test KARAT mock 0x54BF…a73f (18 decimals). Fee token: the Sepolia test USDC mock 0x9ed7…f13D (6 decimals). Both mocks mint freely; they stand in for $KARAT and USDC.
• Code: packages/suites (spec, DSL, runner, validation, quality, economics, seed suites), services/suites (worker, chain access, indexer, CLI karatum-suites), contracts/src/suites/SuiteMarketplace.sol (53 forge tests), API /v1/suites, Explorer /suites, store migration 039_suites.sql.
• Why it matters: every suite that is listed, trialled and paid for adds hidden test material no competitor can copy from a public repository, and the people who write that material are paid for the use of it, not for the hype.

#1. Suite Spec v1 (karatum-suite/1)

A suite is data. It contains task templates, a restricted expression language for the success checks, scripted agents (also in that language) and, optionally, EVM fixture bytecode that runs only in the sandbox. No field can name a URL, a shell command or a module; static review rejects URLs and 32-byte key-like strings in any task text.

Values and checks are a small DSL over bigints (addresses, booleans and bytes32 included): literals, params, fixture addresses, read (an eth_call with a typed signature), math (add, sub, mul, div, mod, min, max), the agent's footprint (txCount, revertedTxCount, refused, completed), pre-agent captures, and document fields. Expressions are comparisons, within (basis points), all, any, not. Agent steps are call, send (repeat at most 16), readDoc, warp, complete, refuse, each with an optional when. There are no loops, no strings beyond document text and no host access. Limits (spec size 56 KiB, 24 templates, 48 steps per script, 64 transactions per attempt, 1,200 DSL nodes per template, 8 fixtures of at most 24 KiB, 12 bits of variant space for a hidden template) are part of the spec and are enforced before anything runs. The authoritative reference is GET /v1/suites/guide.

Determinism by construction. Transaction order, timestamps (2027-01-15 08:00:00 UTC + 12 s per transaction), actor addresses and fixture addresses are fixed; checks read the chain at the end of the attempt. A suite whose results depend on anything else is caught by the double run below.

Contributor identity. The submission carries an EIP-191 personal_sign signature by contributor.address over a fixed message that names the slug, the version, the license, the salt commitment and the content hash (keccak256 of the canonical JSON of the parsed spec). The same wallet lists, stakes and claims on the contract, where the suite id is keccak256(abi.encode(contributor, keccak256(slug))).

#2. Submission pipeline

1. POST /v1/suites/submissions {spec, salt, signature}. Immediate checks: schema (strict, unknown fields are errors), the salt opens the commitment, the signature is the contributor's, static review (reference integrity, what each context may read, entropy, URL and key-like strings, size). Failures answer 422 with the findings. At most three open submissions per contributor; a version must exceed the latest.
2. The worker (karatum-suites.timer, every 2 minutes) takes the oldest queued version and validates it in the sandbox: an Anvil container started with --network none, --read-only, all capabilities dropped, 1 CPU, 768 MB, 64 processes, no fork and no accounts, reachable only through a unix socket (karatum-suites-sandbox.service). Fixture bytecode executes only there; the worker talks to it over the socket. A fork would need an upstream RPC, which is why validation runs on an empty chain and fixtures are deployed from bytecode.
3. Validation runs every template on six variants with every scripted agent plus two platform baselines (idle: does nothing and reports success; refuse-all), twice each, and a replay probe (the reference solution of variant 0 replayed verbatim on ten other variants).

Hard gates (any failure rejects the version with the reasons attached to the report):

Quality score (0 to 100) = 100 × (0.30 discrimination + 0.20 (1 − worse blind pass) + 0.20 variant sensitivity + 0.15 novelty + 0.15 calibration). Variant sensitivity is 1 − replay pass (0 for templates that are not hidden). Calibration compares the declared difficulty with the measured one (1 − mean pass of the careless, partial, idle and refuse-all agents). The report also carries per-template numbers, the capability flags of fixture bytecode (self-destruct, delegatecall, create; informational because the sandbox contains all of it) and resource use.

4. A passing version gets an EIP-712 listing attestation signed by the dedicated attester key (Listing(contributor, slugHash, version, contentHash, saltCommitment, license, embargoUntil, qualityBps, reportHash, deadline); reportHash is the keccak256 of the stored report). It is served at GET /v1/suites/submissions/:hash.

Anti-leak. Variants are a pure function of the contributor's salt, Karatum's own per-trial salt (committed before a trial, revealed after) and the template id and index, so neither side alone can predict a trial's values. Parameter values and traces of a version are published only when its salt is revealed on-chain (superseded versions: anyone can call revealSupersededSalt; retirement: finishRetire). Templates that are not sensitive to their variants fail the replay gate; the deep re-validation (100 replays) is the resolver's evidence in a leak challenge. Honest limit: Karatum stores the contributor's salt in its database, so contributors trust Karatum with it until reveal; a contributor who leaks their own salt loses the value of the suite and can be challenged.

#3. Economics (testnet)

Flow: the contributor calls submitVersion(listing, attesterSignature, stake); a trial calls recordUsage(suiteId, usageRef, amount, buyer) from the distributor key (the fee is pulled in test USDC, usageRef is single-use, 25% is credited to the contributor and the rest to the protocol); the contributor calls claim(to). Payments are pull-only. Accounting invariants are checked by an invariant test (stake, bond and fee totals equal the sum of the per-account credits and the token balances; zero dust after a full lifecycle fuzz).

Slashing. Two paths: openChallenge by anyone with a bond, ruled by the resolver with an evidence hash; and slashByResolver for automated findings. ChallengeManager (the Challenge Market contract) has no suite subject kind, so suite challenges live in SuiteMarketplace itself with the same optimistic pattern (bond, resolver evidence hash, timeout refund). Folding suites into ChallengeManager and letting ValidatorStaking validators vote on escalations is the intended next step, not done today. The resolver is a single key (a deliberate v0 trust assumption, as in the other contracts).

Mapping to the tokenomics paper: staking by those who do the work and are slashed for bad work, a share of protocol fees to the party that produced the data, no payment for holding. The numbers above are testnet calibration, not a proposal for mainnet parameters.

#4. What is live

Three suites were submitted through the public API by wallet-signed requests, validated in the sandbox, listed on SuiteMarketplace with a 1,000 test KARAT stake each, trialled with scripted house builds, and the contributors claimed their rewards.

karatum-frontier-payments v1 was rejected by the pipeline (a salt produced a variant that needed 20 transfers against the limit of 16, so the reference agent failed); v2 narrows the cap range. It is visible as a rejected version on the suite page.

Trials (3 variants per template; builds: house reference, house careless, idle and refuse-all baselines, plus the partial build where a suite has one) and the usage fees recorded on-chain:

The house reference build passed 100% of the attempts on every suite, careless builds 11% to 25%, the idle baseline 0%. Claims: house wallet 0.435 tUSDC (0x0bf099144469306bc4678dfea018bcdcbcab3985ae8e05cf8acdf88389938399), external wallet 0.24 tUSDC (0xa09977aac51444dcadd6e8521f2cffd69d7ad8406d79605c5e12d2c679794789).

Slashing drill. drill-memorizable-payments is a deliberately weak suite (19 of 20 variants share one answer) whose contributor ground the salt (two candidates) until the listing gate's ten replays happened to include the one different variant. It was listed (0x84de5055307606358753a40e96a3010de00b9e59de8e85a5656e9625f159e44e), challenged with a 100 test KARAT bond (0xe293cb1f5721f1189b84967631e9c0b3b0691d5030fcc72038b412d8b32d11bd), and the resolver's deep re-validation (100 replays: 94% pass, gate 90%) upheld the challenge (0x6bdf1d66523ea3fd4ad828bdf1e8a1f32eedc7a8d4588e81e9ea704f79e874db): half the stake slashed, the suite quarantined. The evidence JSON and its hash are on the suite page.

#Seeds are portable subsets

The two internal suites are re-expressions, not the originals: the Frontier trial and Shield run on a Base mainnet fork with real Uniswap pools; the portable versions use synthetic fixtures (a 6-decimal token with transfer bookkeeping, a payment router, a desk, an ERC-4626-style vault) on an empty chain, so a third party can run them. The Base-fork Frontier trial and the 13-class Shield remain the sources of the certificates.

#Trials use scripted builds

Trial builds today are the suite's own scripted agents and two blind baselines, which is enough to make the accounting real and measurable. External builds attach through the session tool surface that the scripts already use (SuiteSession in packages/suites/src/runner.ts: read document, call, send, warp, complete, refuse); wiring an LLM harness to it, and charging real certification fees for it, is not done.

#5. Contributor guide

1. Read GET /v1/suites/guide (limits, gates, DSL, signing message). Look at the three seed suites in packages/suites/src/seed/ (data builders) for working templates.
2. Write the spec. Every hidden template needs at least 12 bits of variant space that the solution depends on: amounts and modes, not just addresses. Provide a reference agent (the careful behaviour; it must pass every variant) and a careless agent (the default behaviour the task must catch); add a partial agent to show gradations.
3. Test locally: SuiteSpec.parse, lintSpec and, with an Anvil, validateSuite (all exported by @karatum/suites; @karatum/suites/node has the IPC and HTTP transports and @karatum/suites/testing an in-memory chain for unit tests).
4. Pick a secret 32-byte salt, set saltCommitment = saltCommitmentOf(salt), parse the spec, sign submissionMessage(spec) with your wallet, POST /v1/suites/submissions {spec, salt, signature}.
5. Poll GET /v1/suites/submissions/:contentHash. A rejected version lists the reasons; fix and submit the next version number.
6. When validated: approve the minimum stake to the contract, then submitVersion(listing, signature, stake) from the contributor wallet using the attestation from the same endpoint (before its deadline, 7 days).
7. Earn: each trial that runs your suite credits claimable(you); claim(to).
8. Rotate: publish a version N+1; the salt of N becomes revealable by anyone. Leave: requestRetire, wait the delay, finishRetire(salt), withdrawStake.

#7. Limits and non-claims

• Single sandbox, validation is serial; a submission can wait behind others.
• The salt is held by Karatum until reveal. Variant privacy rests on that, on the second (Karatum) salt and on the replay probes, not on cryptography.
• Similarity is a token-overlap heuristic; it stops copies, not a determined paraphrase.
• The usage fee is a testnet constant; no real certification fee is charged through this path and no mainnet contract exists.
• The listing gate samples ten replays, which a contributor can grind; that is what the deep re-validation and the stake are for, and the drill above shows the cost of being caught, not that every weak suite is caught.
• Suite challenges are a separate contract from the Challenge Market today (see section 3).