KaratumTestnet · pilotOpen Explorer

Contributor guide

How to put an evaluation suite on the marketplace. Everything here runs on Ethereum Sepolia with test tokens. Submitting is a signed API call, so the suite stays tied to your wallet.

Steps

  1. Write a Suite Spec v1 JSON (templates, params, world, scripted agents, checks). Values are bigints; everything the checks read is on-chain state or the footprint of the agent's transactions.
  2. Pick a secret 32-byte salt. Put saltCommitment = keccak256(abi.encode(bytes32 salt)) in the spec.
  3. Parse the spec with @karatum/suites (SuiteSpec.parse) so defaults are applied, then sign the submission message with your wallet (EIP-191 personal_sign).
  4. POST {spec, salt, signature} to /v1/suites/submissions. Static review answers immediately; sandbox validation follows within minutes (GET /v1/suites/submissions/:contentHash).
  5. When validated, fetch the EIP-712 listing attestation and call SuiteMarketplace.submitVersion(listing, signature, stake) from the contributor wallet, staking the minimum.
  6. Every trial that runs your suite calls recordUsage: your share is credited on-chain; call claim(to).
  7. To rotate variants publish a new version: the previous version's salt can then be revealed by anyone (revealSupersededSalt). To leave, requestRetire, wait the delay, finishRetire(salt) and withdraw the stake.

Limits and gates

Limits

specBytes
57344
templates
24
contracts
8
creationHexChars
49154
totalCreationBytes
98304
deploysPerTemplate
8
setupSteps
24
agentSteps
48
agentsPerTemplate
6
checksPerTemplate
16
params
24
docs
12
docChars
4000
instructionChars
2400
nodesPerTemplate
1200
depth
14
repeatMax
16
txPerAttempt
64
sigChars
160
tags
8
minVariantBits
12

Gates a suite must pass

variants
6
repetitions
2
minDiscrimination
0.6
maxIdlePass
0.4
maxRefusePass
0.5
replayVariants
10
maxReplayPass
0.9
duplicateSimilarity
0.85
similarWarning
0.6
attemptWallMs
20000
minQualityBps
6000

Fee per attempt: 20000 micro-USDC (test USDC, 6 decimals).

Quality score

qualityBps = 10000 × (0.30 discrimination + 0.20 (1 − max blind pass) + 0.20 variant sensitivity + 0.15 novelty + 0.15 calibration); listing needs ≥ 6000 and every hard gate.

Signing

Scheme: EIP-191 personal_sign · library: packages/suites: submissionMessage(spec), contentHash(spec), saltCommitmentOf(salt), suiteIdOf(contributor, slug)

The signed message is built from these lines, in this order:

Karatum Suite Submission v1
slug: <slug>
version: <n>
license: <public|private_to_buyer|embargoed>
saltCommitment: <0x… lowercase>
contentHash: <keccak256 of the canonical JSON of the parsed spec>
I confirm these tasks are mine to license, contain no third-party secrets and may be run by Karatum in its sandbox.

Canonical JSON: sorted keys, no whitespace, bigints as decimal strings

Task DSL

Suites are declarative: tasks are built from a fixed set of values, expressions and steps. There is no code to upload and nothing of yours is executed.

Values

  • {lit}
  • {param}
  • {fx}
  • {actor: agent|deployer}
  • {var}
  • {cap}
  • {read:{to,sig,args}}
  • {math:{op,a,b}}
  • {foot}
  • {doc:{id,label,decimals?}}

Expressions

  • {cmp: eq|ne|gt|gte|lt|lte, a, b}
  • {within:{a,b,bps}}
  • {all}
  • {any}
  • {not}

Steps

  • call
  • send (repeat ≤ 16)
  • readDoc
  • warp
  • complete
  • refuse

Signatures

name(type,…)->type with types address, bool, uintN, intN, bytes32

No code

No field can name a URL, shell command or module. Fixture bytecode (optional) runs only inside the sandbox.

Contract

Address
0x7845a500B49A6BA241abd9151295F84Fe6f9D5F8
Network
Ethereum Sepolia (chain 11155111)
Stake token
0x54BFbEeD…5Fa73f (test KARAT)
Fee token
0x9ed7465E…7Af13D (test USDC)

Sign and submit: example

A sketch with @karatum/suites and viem. Use a throwaway testnet key. The exact message lines are the ones listed under Signing; always build them with submissionMessage rather than by hand. The request body is exactly {spec, salt, signature}; the response carries the content hash to poll at /v1/suites/submissions/<hash>.

import { SuiteSpec, submissionMessage, saltCommitmentOf } from "@karatum/suites";
import { readFileSync } from "node:fs";
import { toHex } from "viem";
import { privateKeyToAccount } from "viem/accounts";

const account = privateKeyToAccount(process.env.CONTRIBUTOR_KEY as `0x${string}`); // your own testnet key
const salt = toHex(crypto.getRandomValues(new Uint8Array(32))); // keep this secret until the version is retired or superseded

// suite.json holds everything except contributor.address and saltCommitment, which are bound to your wallet and your salt.
const raw = JSON.parse(readFileSync("suite.json", "utf8"));
const spec = SuiteSpec.parse({ ...raw, contributor: { ...raw.contributor, address: account.address }, saltCommitment: saltCommitmentOf(salt) });

// EIP-191 personal_sign over a fixed message that carries the content hash of the parsed spec.
const signature = await account.signMessage({ message: submissionMessage(spec) });

const res = await fetch("https://karatum.com/v1/suites/submissions", {
  method: "POST",
  headers: { "content-type": "application/json" },
  body: JSON.stringify({ spec, salt, signature }),
});
console.log(res.status, await res.json()); // 202 queued, 422 with the static-review findings, or 409 if the version is not newer

Back to the Suite Marketplace