@coderifts/conformance

Offline adapter-acceptance suite for every consumer of a CodeRifts decision.

Run the same case file the CodeRifts SDKs are tested against — no network, no API key, no CodeRifts account.

Cases live in cases.v1.json (language-agnostic). Adapters supply a subject that maps each case input to an outcome. Scoring rules and the subject interface are defined in CONTRACT.md.

# one-shot (after publish)
npx @coderifts/conformance --subject reference

# from a local checkout of this package
node bin/coderifts-conformance.js --subject reference

Exit code 0 iff every selected case passes.

node bin/coderifts-conformance.js --subject reference
node bin/coderifts-conformance.js --subject sdk
node bin/coderifts-conformance.js --subject agent-guard
node bin/coderifts-conformance.js --subject agent-guard --profile enforcement_consistent
node bin/coderifts-conformance.js --subject model-acceptance
node bin/coderifts-conformance.js --subject data-plane

• normative (default) — full consumer branching suite
• enforcement_consistent — subset for guards that fail-closed on decision↔action inconsistency
• all — every case

test/ and scripts/ ship with the package. From an install you can run the suite (node --test test/*.test.js) and re-mint the tamper-negative (node scripts/mint-tampered-attestation-negative.js --out /tmp/mine), then measure your own copy. Rows that need something absent — a sibling checkout, a database, the published Python reader — are SKIPPED with the reason printed, never passed quietly. See VERIFY.md §4.

See CONTRACT.md. A subject is:

subject(case) → result

No network. No API keys. Branch on execution_action, not on decision or safe_for_agent.

Point your CI at this package so a decision-only brancher fails the same way CodeRifts’ own SDKs would.

Honesty. model-acceptance proves that the host wired executeOpenAIToolCall / executeAnthropicToolCall / executeGeminiToolCall / executeLangGraphToolCall (which call bind*GuardOutcome) over the same Change-IR (artifacts[] → change_fp) and got the same verdict on all four provider shapes. It does not prove that OpenAI/Claude/Gemini/LangGraph spontaneously call preflight_change_set, and it does not prove raw tools outside the returned table are inescapable.

Measured bind signatures ([email protected]; dispatcher later):

Canned tool-use fixtures: fixtures/model-acceptance.v1.json. Offline; no provider API key. --json prints the machine scorecard.

node bin/coderifts-conformance.js --subject model-acceptance
# or: npm run model-acceptance

Exit 0 iff all 4 providers × allow/block rows pass and change_fp + verdict are identical across providers. Two-run determinism is asserted in test/model-acceptance.test.js.

Local guard newer than the published npm copy: set CODERIFTS_AGENT_GUARD_DIR (or keep a sibling ../coderifts-agent-guard checkout with dist/).

This suite used to report a single conformance number. One number collapses seven claims of very different strength: "an adapter branches on execution_action" and "a merge was refused at the provider" are not the same evidence, and a total cannot tell you which of them is held. It is also silent about what was never attempted.

So a run is reported as seven profiles, in chain order, on two axes that are never conflated:

• coverage — COVERED / PARTIAL / NOT_COVERED
• evidence_tier — LIVE / RECORDED / MODELLED / NOT_RUN
• plus an execution result PASS / FAIL (COVERED + LIVE + FAIL means the property is covered AND the run found a regression)

Every profile is listed, including the empty ones — a reader who sees only the profiles that pass has learned nothing new. Default --evidence recorded verifies vendored pinned external artifacts. --evidence live produces new proof on available infra and is NOT_RUN without it — it does not fall back to recorded bytes.

PROFILE COVERAGE 7/7 · EVIDENCE 2 LIVE + 5 RECORDED + 0 MODELLED · OVERALL RECORDED · FULL LIVE false. That is a count of COVERED rows plus an evidence breakdown, not a pass-rate over unequal claims. MODELLED cannot become COVERED. Every RECORDED operational profile carries a non-empty does_not_prove. Conformance never mints evidence (self_minted:false).

COVERED requires both polarities — at least one positive vector (the capability works) and one negative vector (a real mismatch is refused). A profile with only one half is not covered. PARTIAL is earned when recorded evidence exists but a required negative or read-back is missing — the gap is named, never filled in.

• RECEIPT_CRYPTO — RETIRED from cases.v1.json in 0.4.0; RECORDED from receipt-verifier in this release. The 23 EG-* / EG2-* / EG-A-* / MON-A-* cases were vendored from the CodeRifts app into cases.v1.json, and every shipped subject implements only the decide and tool_selection kinds. A runner here was the wrong fix: the inputs carried a scenario name, not a token, so running them would have meant minting the signed token from that name — a generator and a verifier in one repository agreeing with itself. The removal could not start here (cases.v1.json is vendored from the app and gated byte-identical).

  What changed: this suite now verifies receipt-verifier's committed signed token bytes (fixtures/recorded/receipt-crypto/vectors.json, pin sha256:b2ac4482763ad3c4d743e0073f418740f08083f37b397d6074fea3a4ccf93532, generator test/gen-vectors.js, producer commit 4d3cc48d36a2ee7ff256eec8d76f819843bfd429). Positive VALID tokens pass; the byte-level FORGED negative (tampered_fp) reuses the VALID signature over broken protected content and must fail; field-level tampers (sig / key / audience / operation / expired / version) are scored. Conformance does not mint these tokens (self_minted:false). A digest-pin mismatch is an error. does_not_prove: the live kernel mints this today; the production signing key is current; the key-discovery endpoint is fresh; the grant is currently executable. --evidence live without a kernel is NOT_RUN and does not fall back.

• CREDENTIAL_BOUNDARY — COVERED / RECORDED. The signed prove-transcript DENY panel carries a real target-side denial: cr_host INSERT → Postgres SQLSTATE 42501 (not Node 403, not exit-78), with articles count BEFORE and AFTER the attempt (unchanged). POINT 3 is that denial, not the catalog posture receipt. does_not_prove is non-empty (another credential, another target, raw shell, current config). db.js comments are MODELLED source and are not counted.

• ATOMIC_COMMIT — COVERED / RECORDED. The same correlated transcript shows single-use (replay 201 then 409 GRANT_CONSUMED), concurrency (ok=1 grew=1), CAS-stale (STATE_DRIFT, row unchanged, jti not consumed), no-consume-only (crash-before-seal rolled back article AND ledger), no-mutation-only (executor raw INSERT 42501), and before/after read-backs on the positive commit and every negative. EG-A-STATE-NONCE-MISMATCH was never evidence here. POINT 8 merge stays MODELLED and is not this profile. The artifact records working_tree_dirty:false (generated from a clean checkout of 3a34079).

• PROVIDER_ENFORCED — COVERED / RECORDED. Raw GitHub API dumps of coderifts/demo, not a CodeRifts summary. Ruleset 22074842 (coderifts-enforcement, required CodeRifts / contract-gate, integration_id 2860592, refs/heads/main, enforcement active). Negative pole: PR#4 head 4b2062b9, REST mergeable_state blocked, required context CodeRifts / contract-gate COMPLETED FAILURE (app 2860592). Positive pole: PR#5 head df76f7a7, required context CodeRifts / contract-gate SUCCESS (same app). Both poles on that required context — not the differently-named CodeRifts — API Contract Check. Capture provenance: local gh api by zsobpeter-code on 2026-09-05, oidc_attested:false. Negative dump is REST (PR + check-runs); positive dump is GraphQL — the shapes as captured. ADV-1 still does not count here. The envelope also carries the merge-API 405 bodies (PR#4 contract-gate is failing; PR#5 is expected as control) — the gate refused the admin merge, not only the check. does_not_prove names HISTORICAL freshness, bypass actors, local gh token, that PR#5 remains BEHIND / 405-expected, and that no merge landed. The 1105 canary design remains the cost model for a live 405.

• observed_bypass_failure — recorded 405 pin, not an eighth profile. The bytes live in fixtures/recorded/bypass-attempt/ and are scored on PROVIDER_ENFORCED (1395). Admin PUT …/pulls/{4,5}/merge returned HTTP 405 naming CodeRifts / contract-gate. PR#4 reason is failing; PR#5 is expected (BEHIND, not a gate-refusal). HISTORICAL, local gh, oidc_attested:false. The 405 bodies do not identify the actor or the PR; the pin binds path → payload.

• END_TO_END — PARTIAL / RECORDED. The two earlier gaps stay closed: authorization is continuous (one challenge-first cr.exec.v2 ATOMIC grant through authorize, consume, attestation and correlation) and every token authenticates against its issuer's key. The gap that reopened the grade is a different class — CROSS-RUN COLLAGE. The measure checks each token individually, so it cannot see that a set of individually-authentic tokens came from two runs; swapping in the negative pole's own signed tokens was accepted in all three positions. This is stated rather than graded around: cross_run_collage is a conjunct of the verdict, not a footnote. COVERED returns when an artifact carries a signed evidence root, and the measure verifies that rather than being told. prove-transcript is a Postgres executor whose POINT 8 merge is MODELLED. The provider bundle is GitHub PRs with different commits and a different run_id. A collage of separate artifacts is layer-coverage, not end-to-end. Not COVERED.

0/0 is rejected as a rendering, in every format. It is not merely unclear — it is the same shape as a pass: 14/14 and 0/0 both read as "everything selected succeeded", and a dashboard averaging ratios scores 0/0 as 100%. NOT COVERED cannot be misread that way because it is not a number. A count is printed only where a count means something.

node bin/coderifts-conformance.js --profiles          # the table above (default --evidence recorded)
node bin/coderifts-conformance.js --profiles --json   # machine shape, explicit `green` per profile
node bin/coderifts-conformance.js --evidence live --profiles   # NOT_RUN for recorded profiles; no fallback
node bin/coderifts-conformance.js --assurance RECEIPT_CRYPTO   # exit 0 in recorded mode (COVERED / RECORDED)
node bin/coderifts-conformance.js --assurance ATOMIC_COMMIT    # exit 0 in recorded mode (COVERED / RECORDED)
node bin/coderifts-conformance.js --assurance PROVIDER_ENFORCED # exit 0 in recorded mode (COVERED / RECORDED)
node bin/coderifts-conformance.js --assurance END_TO_END        # exit 3 — PARTIAL (cross_run_collage)

--assurance <ID> exits 0 only when that profile is COVERED (LIVE or RECORDED), 3 when it is PARTIAL, NOT RUN or NOT COVERED, and 2 for an unknown id. Exit 3 is distinct from exit 1 on purpose: "nothing proved this" is not "this was disproved". A CI job pointed at a PARTIAL or empty profile fails instead of going green forever.

One vector fits none of the seven and is recorded rather than forced: ADV-6 (SHA pin vs the moving @v0 tag) asserts supply-chain integrity of the enforcing component, which is a precondition for provider enforcement rather than enforcement, and is not a verdict, a token or a tool. Forcing it into the nearest profile would make that profile's name overclaim. See UNPLACED in lib/assurance-profiles.js.

The ten COVERAGE BOUNDARY / RE-VERIFICATION RECORD tests count toward no profile: they assert properties of the fixture, and letting the suite score itself for describing itself is exactly the inflation this split prevents.

Every other subject here is a pure function of case input. This one executes real code — the four hops of examples/atomic-v2/run.js in the pinned capability-demo checkout — because CREDENTIAL_BOUNDARY and ATOMIC_COMMIT are properties of a running system and no pure function can reach them.

Measured result, and it is not the flattering one: the keyless run fills zero profiles. All four hops pass and none of them is admissible as evidence here:

With CODERIFTS_DATAPLANE_PG pointing at a migrated capability-demo database, two rows become real — and they are the only two the chain can honestly fill:

# keyless — reports skips with reasons, exits 0, and says in words that nothing was proved
node bin/coderifts-conformance.js --subject data-plane

# with a live database
CODERIFTS_DATAPLANE_PG="postgres://demo:demo@localhost:5432/demo" \
  node bin/coderifts-conformance.js --subject data-plane

No API key on any path. pg is not a dependency of this package; it is borrowed from the capability-demo checkout, and its absence is a named skip. An absent or off-pin checkout is capability_demo_absent / capability_demo_commit_mismatch — never silently COVERED.

Every database row runs inside a transaction that is rolled back, so a caller's database is left exactly as it was found.

These rows do not feed --profiles or --assurance, on purpose. Those drive CI gates, and a gate whose colour depends on whether a database happened to be reachable is worse than one that is honestly red. The seven-profile report takes CREDENTIAL_BOUNDARY / ATOMIC_COMMIT from the recorded prove-transcript (COVERED / RECORDED), not from this live subject. Data-plane skips remain true of the live path; this subject reports separately what a run with a database observed.

DP-PG-SEAL-REQUIRED was not designed — it was found. The first version of DP-PG-SINGLE-USE INSERTed and committed, and the deferred trigger refused it. That refusal turned out to be the closest thing this suite can observe to ATOMIC_COMMIT's actual sentence, so it became a row.

cases.v1.json is vendored from the private CodeRifts app (test/adapter-acceptance/cases.v1.json). The app CI asserts byte-identity so this public copy cannot silently drift from what the product tests against.

App-private anchor checks (that cases still match live app source text) stay out of this package — they require the private app tree.

MIT

A passing adversarial vector proves that ONE NAMED ATTACK SHAPE fails against the behaviour this suite can reach. It never proves the class is closed, and it is not a security guarantee.

WHAT A PASS ASSERTS: the specific shape named in the vector — that payload, that ordering, that ref form — does not produce the outcome the attacker wanted, at the version recorded in the fixture.

WHAT A PASS DOES NOT ASSERT: that a variant of the same shape fails; that the class the shape belongs to is covered; that the product still behaves this way at a version other than the one recorded; or, where a vector mirrors published logic rather than importing it, that the mirror and the product still agree. That last one is not hypothetical — re-verifying against [email protected] found two divergences in the issuer mirror while every test stayed green, and both would have reported a WRONG verdict to an auditor.

WHY SOME VECTORS ARE ABSENT: four of the auditor's seven are not expressible in an offline, credential-free suite. They are listed in fixtures/adversarial.v1.json under excluded, each with its reason and where it IS covered, if anywhere. An absent vector is recorded, never silently dropped — a suite that hides what it cannot test is worse than a smaller honest one.

Run them with the rest of the suite (npm test), or alone:

node --test test/adversarial.test.js

No CodeRifts credential, no network, no account. Three families are carried — issuer collision, MCP negative schema, action tag mutation — and the coverage boundary is a test of its own, so the suite fails if it stops stating what it does not cover.

lib/attack-matrix-runner.js executes a vector only when the row names an execute id that maps to a shipped adapter. COVERED means all five points (target, nonce, executor, attestation, gate) were checked by that run — not that the JSON declared an expected outcome. Rows with no execute id stay NOT_RUN, named. The runner does not invent a verifier and does not call issueExecutionGrant. ATOMIC_COMMIT stays NOT COVERED.

Executed regressions (fail-closed): git missing expected_old_sha (4742476), HTTP missing/weak ETag (0988a20), reconciler forged attestation (eeae7e7). Stated-contract consumeAndCommit rows (replay, expired-nonce, payload-swap, missing-attestation) remain recorded, not executed. Pending-contract: raw-tool, concurrent, stale-state. This file does not enlarge adversarial.v1.json excluded[] (still exactly four).

Those three regressions load the shipped adapters from capability-demo (git-atomic.js, http-atomic.js, reconcile.js). capability-demo is a demo repo (private: true in its package.json) — it is not on npm. The pin lives in this package's package.json, under coderifts.capability_demo. It is not an npm dependency: installing this package does not fetch it.

Pinned commit (the checkout COVERED expectations were measured against):

75a09d8b74f1161d40415fb98c2537f116d9ffdc

Sibling checkout (path the runner looks for by default):

<parent>/
  capability-demo/          # git clone + checkout the pin
  coderifts-conformance/    # this repo

cd <parent>
git clone https://github.com/coderifts/capability-demo.git
cd capability-demo
git checkout 75a09d8b74f1161d40415fb98c2537f116d9ffdc

Adapters are loaded from ../capability-demo/demo/src relative to this repo. Override with CODERIFTS_CAPABILITY_DEMO (repo root or demo/src).

When the checkout is absent (or at a different commit): the three regressions are NOT_RUN / capability_demo_absent (or capability_demo_commit_mismatch), named, with the expected path and commit. They are never silently COVERED. The rest of this suite still runs. An external user who follows the clone+checkout above can reproduce the COVERED run.