Skip to main content

Benchmark Execution Trace — Compliance Exemption

The benchmark e2e_full pipeline emits a full per-case execution trace — input query, retrieve I/O, sufficiency decision, external evidence chain, synthesizer I/O, hallucination check, score breakdown, verdict — surfaced through three audience projections. This page documents the compliance exemption that permits raw content in step traces (the default AiMOneHealth posture forbids it) and the gates that keep the surface safe.

Source of approval: docs/compliance/benchmark-trace-debug-exemption.md (effective 2026-04-27, project owner).

Why an exemption is needed​

The default reasoning-flow contract (Architecture → Reasoning Flow) says:

Never embed raw prompt text, retrieval hit bodies, or LLM output in step inputs/outputs. Those stay in their backing collections; the step carries an opaque ref.

That rule prevents PHI leaks through the reasoning surface. The benchmark needs to prove the system's outputs are complete and grounded — for that, the operator must see the actual retrieve-response body, the actual evidence-chain bodies, and the actual synthesized answer, not just counts and IDs.

The exemption permits raw content narrowly and only for benchmark e2e_full traces against synthetic patients.

Two hard server-side gates​

Both gates are fail-closed on a non-synthetic patient id. Either gate alone refuses the request — defense in depth.

Gate 1 — Capture​

apps/research/benchmark/services/e2e/case_trace.py::is_capture_eligible(patient_id) returns True only when patient_id.startswith("PAT_TEST_"). The benchmark pipeline records nothing in the trace when this returns False, so there is no captured content to leak.

Gate 2 — Projection​

apps/api/src/modules/benchmark-runs/services/trace-projector.ts::assertSynthetic() raises NonSyntheticPatientError on a non-PAT_TEST_* trace. The service maps it to HTTP 400. Even if a trace somehow lands in the artefact (it can't — gate 1 blocks it), the projector refuses to serve it.

Three audience projections​

Computed server-side from a single canonical capture. The wire payload to the browser is the projection, never the canonical capture, so an admin viewing the patient projection cannot exfiltrate admin-only fields via DOM inspection.

AudienceSeesDoes not see
patientPlain-English narrative of the question, evidence sources used, outcome, caveatsInternal IDs, raw retrieve text, timing details
providerClinical detail: query, expected vs predicted, keyword coverage, evidence counts, verdict, hallucination/latency notesInternal latency timings, raw token sets, internal scoring formulas
adminFull canonical DAG — every phase's raw input/output, fact pool, leaked tokens, score formula(admin sees the full DAG)

Role → max-audience​

apps/api/src/modules/benchmark-runs/services/trace-projector.ts::isAudiencePermitted(role, audience):

Caller rolePermitted audiences
adminpatient, provider, admin
provider, nurse, coordinatorpatient, provider
patientpatient

A patient-role caller cannot request the admin projection — server-side 403 before the projector runs.

Audit trail​

Every read emits a benchmark.trace.read event regardless of audience. Payload carries case_id, audience, verdict. The events ride the same SHA-256 hash chain as the rest of audit_events (Compliance → Audit Chain).

Operational readback:

db.audit_events.find({
  action: "benchmark.trace.read",
  ts: { $gte: ISODate("2026-04-27") },
}).sort({ seq: 1 })

Surface​

EndpointNotes
GET /api/benchmark-runs/:run_id/cases/:case_id/trace?audience=<patient|provider|admin>Returns the audience-specific projection. 400 on missing trace, 403 on audience > role-max, 404 on missing run/case.
/dashboard/admin/benchmark/runs/:run_id/cases/:case_id/traceAudience-tabbed page with Download document (markdown export) for evidence packets.

Revocation​

The exemption is scoped to the benchmark e2e_full surface only. To extend it to any other surface (real-patient flows, other strategies, other code paths) requires a new approval recorded under docs/compliance/.