Compliance Evidence Pack

Status: live on testnets, 2026-10-16. Code: packages/compliance; API: services/api/src/routes/compliance.ts; table compliance_packs (migration 033); control mapping: docs/compliance/.

#What it is, and what it is not

Operators who deploy AI agents are asked by auditors, customers and regulators what they can show. A Compliance Evidence Pack gathers what Karatum has measured about one build, as of one moment, into one signed set of files: a PDF for a reader, a canonical JSON for a machine, and a manifest of hashes with references to the upstream attestations. It then maps that evidence to the controls of three frameworks and lists, for every control, what the evidence does not show and what the operator has to supply.

It is evidence, not an opinion:

• It is not legal advice, a conformity assessment, a certification or an audit opinion, and it never says that a requirement is met. The strongest status a control can have is "evidence attached" (a primary Karatum artefact for the control exists for this build). The status words are fixed in packages/compliance/src/types.ts and the tests reject wording that claims compliance.
• Karatum is not a notified body, a market surveillance authority, an accredited certification body or an ISO/IEC 42001 auditor.
• Karatum measures fixed task suites, probe sets and attack suites on testnets and sandboxes. It does not measure the operator's production system unless the operator records it (Flight Recorder).

#Inputs (all read-only)

All inputs are Karatum's own records. Each collector filters by the pack's asOf time where the table keeps a timestamp, so a pack requested with an explicit asOf can be reproduced.

Statistics: 95% Wilson score intervals (same formula as K-ratings and the Honesty Index); medians take the lower middle value so they are always an observed value; durations are measured between recorded timestamps.

#Control mapping

docs/compliance/ holds the mapping: 47 controls in map version 1.0.0 (EU AI Act: Arts. 6, 9, 11, 12, 13, 14, 15, 26, 72, 73; NIST AI RMF 1.0 subcategories in GOVERN, MAP, MEASURE, MANAGE; ISO/IEC 42001 Annex A controls A.3.3, A.5.2, A.6.2.4 to A.6.2.8, A.8.2, A.8.4, A.9.4, A.10.3). It is generated from packages/compliance/src/controls; docs/compliance/CHANGELOG.md records versions, and every pack records the map version and hash it used.

Every entry carries:

• applicability: for the EU AI Act, whether the article binds providers of high-risk systems, deployers of high-risk systems or both. Whether a system is high-risk is decided by Art. 6 and Annex III from its intended purpose; Karatum does not classify (EU-AIA-6 has no Karatum evidence by design). NIST AI RMF is voluntary and uncertifiable; ISO/IEC 42001 certifies an organisation's management system, never a tool. General-purpose AI model obligations are out of scope.
• evidence: which artefact bears on the control, marked primary or supporting, with one sentence on how.
• limits: what that evidence does not show.
• operator gaps: what the operator must supply.

Framework wording is paraphrased: titles and requirement lines are our own, and article, subcategory and control numbers are checked against the official texts at each map version. Application dates of the EU AI Act have been subject to amendment proposals; the pack states no dates.

Status of a control in a pack, per build:

An empty incident list is none recorded, not evidence of safety: it counts as "partial" for a control that needs incidents as primary evidence.

#Gaps

Two lists, both printed in the PDF and returned in the JSON:

1. Karatum evidence gaps, computed from the data: no Shield certificate, no Flight Recorder coverage, no judge traces, an Honesty Index entry without a published score, a certificate that is not VALID at asOf, fewer than 30 decided attempts, verdicts on weak evidence (E2 or lower), unattested verdict hashes, batch gaps or forks, unresolved incidents, no Guard account declared.
2. Declaration gaps: what the operator left undeclared (purpose, classification, oversight, stop procedure, log retention, incident process). The operator declaration is printed as supplied and is not verified.

Per-control operator obligations are listed in the control matrix.

#Determinism and hashes

A pack is a pure function of (evidence snapshot, declaration, ownership flag). Same inputs give byte-identical output:

• pack.json is canonical JSON (keys sorted, no whitespace). contentHash = keccak256(pack.json).
• pack.pdf is written by an in-repo PDF writer (A4, standard fonts, uncompressed streams, no dates, a trailer id derived from the content hash). Rendering works from the canonical form, so key order cannot change it. pdfHash = keccak256(pack.pdf). Output was confirmed identical on Node 22 and Node 25.
• control-map.json is the control map as used; the manifest records its hash.
• manifest.json lists every file with its size and keccak256, the content, PDF and references hashes, and how to reproduce. manifestHash = keccak256(canonical manifest).

The pack id is cp_ plus the first 16 hex characters of SHA-256 of contentHash|ownerKeyId, so asking again for the same evidence returns the same pack. Pass asOf to make a request reproducible; the default is the current second.

#Attestation

A worker (karatum-compliance worker, systemd unit karatum-compliance) attests each new pack on Ethereum Sepolia (a test network) with a dedicated key, in the same EAS deployment as verdicts and Shield. Schema:

string buildId, string packId, uint64 asOf, bytes32 contentHash, bytes32 pdfHash,
bytes32 controlMapHash, bytes32 manifestHash, uint16 evidencedSections, uint16 gaps, string methodology

The attestation shows that exactly this content, PDF and control map were committed by Karatum's attester key at that time. It says nothing about whether the content is correct, complete or reviewed.

#Verifying a pack

Offline, with only the downloaded files and this repository:

pnpm --filter @karatum/compliance cli verify <dir with pack.json, pack.pdf, manifest.json, control-map.json>

It checks that pack.json is the canonical form, that its hash and the PDF's hash equal the manifest's, that the PDF re-renders from pack.json byte for byte, that the references hash and control-map hash match, and the method version. To re-derive the pack from the database, run karatum-compliance build --build <id> --as-of <iso> with the same declaration and compare the three hashes.

#API and privacy

• POST /v1/compliance/packs (API key): {buildId, asOf?, declaration?}. Burst 6 per minute and 20 new packs per day per key. 201 for a new pack, 200 when it already exists.
• GET /v1/compliance/packs/:id: JSON envelope; ?format=pdf|pack|manifest|control-map returns that file. Only the key that generated a pack, or anyone for a published sample, can read it; everything else answers 404.
• GET /v1/compliance/packs: your packs and the published samples. GET /v1/compliance/controls: the control map.
• Any key may generate a pack for any build, because the evidence is Karatum's public record. The pack states ownership: "owner_key" when the requesting key owns the build, else "not_verified". Non-public (embargoed or private) incidents are included only for the build's owner.
• The declaration is stored with the pack and visible to the key that generated it only.

#Legal wording review

The disclaimer text (packages/compliance/src/disclaimer.ts) and every control entry were reviewed for overclaiming with these rules, enforced by tests where they can be:

1. No sentence says or implies that a requirement is "met", "satisfied", "compliant" or "guaranteed"; a test scans the control map and the disclaimer for such words.
2. Every EU AI Act control states that it binds only high-risk systems (and which role), and that classification is the operator's.
3. Karatum is described as none of: notified body, authority, accredited certifier, ISO auditor.
4. Karatum's incident classes are never equated with the legal "serious incident" of Art. 3(49); the registry is not a report to an authority.
5. Evidence is always tied to its scope: fixed suites, testnets, one as-of time. An empty list is "none recorded".
6. House builds are labelled as such: Karatum is then both operator and evaluator, so the sample shows format, not independence.
7. Framework text is paraphrased and the reader is sent to the official texts; no dates of application are asserted.
8. The attestation is described as a commitment to bytes, never as validation.

This review is an engineering review, not legal advice. Have counsel read the pack wording before it is used in a regulated sale.

#Limits worth repeating

• Karatum has no Flight Recorder data for most builds, no Shield certificate for the LLM builds while the the LLM inference host is down, and judge traces only for newer verdicts. Packs say so instead of hiding it.
• Evidence is on testnets and sandboxes; no real funds are involved in any measurement.
• The mapping covers a subset of each framework. Everything organisational (policy, roles, training, impact assessment, data governance, supplier management, management-system clauses) is the operator's.