Benchmark Console (SPEC-068)

Admin-only web surface that turns the Spec 067 case-study benchmark CLI into an interactive workflow: initiate runs, watch progress live, compare runs side-by-side, drill into a single case, and see regression badges on PR runs.

PHI-free by construction (FR-068-026): every payload is a count, an id, or a synthetic narrative from the Spec 067 fixture (FR-067-003). The repository deliberately omits @phi_repository envelope encryption — see apps/api/src/modules/benchmark-runs/benchmark-runs.repository.ts for the documented justification.

Audience

• Admins — only role with access (@Roles('admin')); enforced server-side at every endpoint.
• PMs / clinicians reviewing reasoning quality — read-only via the same surface; admins are gatekeepers.

The 5 user stories

US	Priority	Surface	What it does
US1	P1	/dashboard/admin/benchmark/initiate	POST /api/benchmark-runs with strategy + smoke + (optional) fixture pin; single-flight per fixture (FR-068-004); audit-logged.
US2	P1	/dashboard/admin/benchmark/live/[runId]	Polls GET /api/benchmark-runs/:id/events?since=<ts> every 1.5s; server clock is the cursor for SC-068-003 reconnect-replay; Cancel-with-confirm dialog.
US3	P2	/dashboard/admin/benchmark/history + /history/compare	Past runs list with short git_sha (FR-068-030) + Reproduce button (FR-068-028); compare two runs → case-level flips + axis-rate deltas + fixture-version-drift banner (FR-068-013/014).
US4	P2	Drill-down dialog from any case row on a terminal run	Native <dialog> with axis + observed verdict + Spec 062 query as narrative + Spec 066 reasoning-flow fallback.
US5	P3	Red <RegressionBadge> on PR runs in History	At terminal status, PR runs auto-compute regressions vs the most-recent main-branch baseline (FR-068-018); deep-link to the comparison filtered to PASS→FAIL only.

Architecture

Why the sidecar

The API container stays Python-free. Spec 067's CLI has heavy ML dependencies (apps/research/) that we don't want in the API image. The sidecar wraps the CLI behind a stable HTTP/SSE boundary; the API consumes the SSE stream as if it were the original stdout NDJSON.

Auth: bearer token (BENCHMARK_RUNNER_TOKEN), constant-time compare, fail-closed in production. In dev the binding is loopback-only (127.0.0.1:6206) so the token is optional.

Why HTTP polling instead of Socket.IO

The platform MessagesGateway.setServer() is defined but never invoked in production code, so a benchmark-specific Socket.IO gateway would inherit the broken bootstrap. FR-068-012 explicitly authorizes HTTP polling as a fallback transport; we promoted it to primary. The polling client tracks a server_time cursor returned on every poll, which gives SC-068-003 reconnect-replay for free — a transient blip resumes from the cursor on the next tick.

Why an artefact file

At terminal status the API writes a RunArtefact JSON file at output_paths.json. Comparison reads it for case-level flips (US3); drill-down reads it for the per-case observed view (US4); regression detection reads both runs' artefacts (US5). Single source of truth, no shared volume needed between API and sidecar.

Per-fixture narrative resolver

services/fixture-resolver.ts reads the Spec 062 fixture file and projects query_text into the drill-down's case.narrative. Per-fixture cache because fixtures are immutable inside a version. Fail-soft: missing file or unknown case_id renders the dialog's fallback copy.

Reproduction-recipe loop

Every History row carries a Reproduce button that copies a paste-and-run CLI command:

git checkout <git_sha>
&& python3 -m apps.research.benchmark.cli run \
  --strategy=<strategy> [--smoke] [--fixture-version=<sha>] \
  --out=./reports/<run_id>

This closes the clinical-quality / debug loop: see a regression in the UI → paste the command → reproduce locally with the exact same code + fixture (FR-068-028, FR-068-029, FR-068-030).

Role + audit + PHI invariants

• Role: every endpoint carries @Roles('admin'). No other role sees the surface.
• Audit: benchmark_run.created, benchmark_run.conflict, benchmark_run.cancelled all routed through AuditService.append() BEFORE state changes are user-visible. Hash-chained per Spec 008.
• PHI: zero by construction. The benchmark_runs collection holds counts/IDs only. The Spec 067 fixture is synthetic (PAT_TEST_*, no real MRN/SSN). The Playwright DOM-grep test (SC-068-007, owed) verifies at runtime.

Reasoning-flow integration (Spec 066)

Every benchmark e2e_full case opens a ReasoningFlow on memory-store and emits one step per pipeline phase:

Phase	StepKind	Recorded
1. Retrieve	retrieval	entities returned, retrieval latency, force_route
2. Sufficiency gate	evidence_assembly	sufficient: bool, reason
3. External chain (conditional)	evidence_assembly	pubmed_count, manual_count, requires_manual_review, gap_logged
4. Synthesizer	summary	predicted char count, evidence_refs count
5. Hallucination check	provenance_check	clean / leaked-token count
6. Verdict	verdict	PASS|FAIL|SKIP, keyword_coverage, answer_f1, latency vs budget

The case-detail dialog deep-links to /dashboard/admin/reasoning/{flow_id} for the chain-of-thought DAG. Best-effort per FR-RFT-020 — recorder failures never block the verdict.

Execution-trace surface (compliance-approved)

For full forensic visibility — input → DB query + response → external evidence → synthesizer I/O → score breakdown — the dashboard surfaces a per-case execution trace with three audience projections (patient narrative / provider clinical detail / admin canonical DAG) plus a downloadable markdown document. Synthetic-only (PAT_TEST_*); raw-content embedding is a narrowly scoped exemption to the default reasoning-flow contract.

Surface: /dashboard/admin/benchmark/runs/[run_id]/cases/[case_id]/trace

Audit: every read emits benchmark.trace.read with the audience tag.

See Compliance → Benchmark Trace Exemption for the approval record, the two server-side gates (capture + projection), and the role → max-audience mapping.

Synthetic-patient consent bypass (FR-033)

The benchmark CLI calls POST /api/memory/retrieve against PAT_TEST_* patients. Memory-store's consent guard short-circuits to a synthesized ResolvedScope(auth_source='synthetic_test') for those ids — the /api/consent/check service has no records for synthetic patients and would reject every benchmark request. Real PHI patient ids do not match the prefix; the bypass fires only on the explicit FR-033 fixture convention. See Compliance → Consent Model.

Cross-references

• Spec 067 — the case-study benchmark CLI itself: Benchmark + Monitoring
• Spec 066 — Reasoning Flow Tracker (now integrated)
• Trace exemption — Compliance → Benchmark Trace
• Plan + tasks: specs/068-benchmark-console-ux/{plan,tasks,spec}.md

What's deferred

	Why
Playwright e2e suite	Repo doesn't yet have a Playwright framework; setup is its own slice.
Inline reasoning-step timeline in the drill-down dialog	Dialog currently deep-links to the admin reasoning page; inline projection is a follow-up if operators ask.
CI workflow for triggered_by: "ci" PR auto-runs	Needs CreateBenchmarkRunDto extension + service-account auth + a new GitHub workflow.
Socket.IO gateway (T037)	Platform IoAdapter blocker.