Reasoning Flow Tracker

Spec 066 — captures every reasoning step an agent takes, chains them into a queryable DAG, and surfaces the result to four audiences:

1. Patient / HCP transparency — "why did the system recommend this?"
2. Audit / compliance — FDA CDS exemption evidence trail + HIPAA incident response
3. Pathway library (Phase 3) — recognize and reuse known decision paths
4. Eval + fine-tuning (Phase 4) — DPO training signal + decision-tree evals

The three-level model

A flow opens on a trigger, accumulates steps, and closes with an outcome. Every step carries opaque back-refs to its contributing collections so operators can drill down to raw artifacts. Every write also lands in audit_events for hash-chain integrity.

Collections

Collection	Purpose	PHI posture
reasoning_flows	One doc per rooted reasoning tree (trigger → outcome). Metadata-only (ids, state, trace_id, patient_id).	No PHI on the flow doc; patient_id is an opaque id
reasoning_steps	One doc per step. inputs/outputs carry declared pii_fields	@phi_repository envelope; PHI fields encrypted at Motor boundary

Indexes (created at lifespan startup):

• reasoning_flows: (patient_id, created_at) sparse, (trace_id, created_at), (flow_state, created_at)
• reasoning_steps: (flow_id, created_at), (parent_step_id, created_at) sparse

HTTP surface

Route	Audience	Emits
POST /api/reasoning/flows	reasoning.write scope	reasoning.flow.started
POST /api/reasoning/flows/{id}/steps	reasoning.write	reasoning.step.created
PATCH /api/reasoning/flows/{id}	reasoning.write	reasoning.flow.completed or .aborted
GET /api/reasoning/flows/{id}	reasoning.read scope	patient.reasoning.flow.read
GET /admin/v1/reasoning/flows/{id}	admin	admin.reasoning.flow.read
GET /admin/v1/reasoning/flows?trace_id=...	admin	(read-only, no audit on list)
GET /admin/v1/reasoning/pathways/stats	admin	admin.reasoning.pathway.stats_read (counts only — total/active/embedded/by_model/by_trigger_type/maturity_by_trigger_type/demoted/top_hits)
GET /api/admin/reasoning/pathways/match	admin (proxy)	admin.reasoning.pathway.match.proxy (count + top reason)
POST /api/admin/reasoning/pathways/:id/hit	admin (proxy)	admin.reasoning.pathway.hit.proxy (pinned to actor)
POST /api/admin/reasoning/pathways/research-deeper	admin (proxy)	admin.reasoning.research.deeper_requested + memory-store reasoning.research.deeper_request_created (writes to reasoning_research_backlog collection; falls back to audit-only when memory-store unreachable)
GET /api/admin/reasoning/pathways/research-backlog	admin (proxy)	admin.reasoning.research.backlog_read.proxy (count summary only)
POST /admin/v1/reasoning/research/backlog/next	admin	reasoning.research.backlog.dispatched (atomic FIFO pull + status flip; optional trigger_type filter; returns {request: null} on empty queue)
POST /admin/v1/reasoning/research/backlog/{id}/complete	admin	reasoning.research.backlog.completed (marks completed_at + records completed_flow_id; 409 if not in dispatched state — prevents double-complete + skip-pull bugs)

Dispatcher contract: any external dispatcher (research-engine, scheduled poller, lambda, etc.) consumes these two routes — pull-next gives an exclusive lock on a backlog row via atomic status flip, complete records the resulting flow_id back-reference. The contract is the integration point that unblocks real deeper-research dispatch without forcing memory-store to depend on research-engine internals.

Pathway maturity buckets (per trigger_type, classified server-side from active count + high-hit count): mature (≥ 5 active AND ≥ 1 with hit_count ≥ 3), growing (≥ 1 active), sparse (0 active). Surfaced on the dashboard tile + the reasoning:pathway-inspect CLI so operators see at a glance which trigger families have proven neural pathways vs which need more research investment.

Decision Dimensions radar — every pathway match candidate AND every patient narrative response carries a 6-axis radar profile derived server-side in services/reasoning_dimensions.py. Five positive axes (Confidence, Coverage, Maturity, Recency, Consensus) + one negative axis (Risk — high score = caution flag). Each axis is a 0–7 integer with a low/medium/high bucket. Operators read the radar on the Research Console (compact inline + click-to-expand); patients read it on /dashboard/reasoning/[decisionId] with audience-appropriate labels (Coverage → "Evidence breadth", Consensus → "Care team agreement", Risk → "Caution flags"). Pure-SVG component — no chart library, accessible via <title> tooltips + summary aria-label. The composite match score is no longer the operator's only signal: they see the trade-offs visually and can pick a moderate-but-fresh pathway over a strong-but-stale one.

Snapshot at decision time — the close handler computes the radar at flow close and persists it as dimensions_snapshot on the reasoning_flows document. The patient narrative route prefers the snapshot when present; falls back to fresh compute for backwards compatibility. This is the audit-trail integrity move: patients viewing the narrative weeks later see what the system believed when it decided, not what would be recomputed after pathway demotions, hit-count climbs, or Atlas data freshness shifts.

Comparison radar — operators on the Research Console check 2 candidates → overlay panel renders both polygons (indigo + pink) on one chart with a differential legend ranking axes by absolute delta. The dominant pick is declared with explicit reasoning; tie-breaker prefers lower Risk. Operators see trade-offs as ranked deltas instead of mentally diffing two score lists.

Research Console at /dashboard/admin/reasoning/research-console (ScreenProfile admin-reasoning-research-console) is the operator workbench for building the knowledge base: submit a (trigger_type, trigger_ref) → see ranked candidates → "Reuse pathway" on strong matches (bumps hit_count + last_hit_at, strengthening the pathway in the library) → "Trigger deeper research" when nothing clears the threshold (lodges an audit-tracked dispatch request; real research-engine dispatch is the deferred multi-day cross-package integration).

Operator CLI: npm run reasoning:pathway-inspect [-- --json] wraps the stats route — set MEMORY_STORE_ADMIN_TOKEN (and optionally MEMORY_STORE_BASE_URL, default http://localhost:6401). Use --json to pipe through jq (e.g. jq '.by_model' to watch the embedding-model breakdown converge after flipping PATHWAY_EMBEDDING_MODEL).

Browser surface: /dashboard/admin/reasoning/pathway-library renders the same shape (totals + by-model + demoted + top-5) for operators who'd rather watch from the dashboard. ScreenProfile admin-reasoning-pathway-library (admin-only). The apps/api proxy at /api/admin/reasoning/pathways/stats bridges the cookie session to the memory-store service token and emits an admin.reasoning.pathway.stats_read.proxy audit pinned to the calling admin (memory-store emits its own audit too). Page falls back to a demo fixture if the proxy errors.

Patient narrative route is rate-limited at 30/5min to prevent log-mining.

Self-improvement loop

The system feeds itself. Two producer signals enqueue follow-up work; a consumer drains the queue with trigger-specific dispatch strategies; recovery + observability + a human-triage queue close the loop.

Critic dispatch strategy is operator-chosen via env at lifespan-construction time:

CRITIC_DISPATCH_STRATEGY	Behavior
unset / mark_for_review (default)	Rejected claims go to the human-triage queue (claims_pending_review). Curator clicks Resolve/Dismiss.
re_research	Rejected claims trigger a Researcher re-run with dispatched_from_backlog=True and the rejection reason in the question framing. No human in the loop.

The two strategies are mutually exclusive at runtime; existing pending entries from the prior strategy stay in their queue (not auto-converted). Activation runbook + rollback at research/loop-activation.

Producer flags (default OFF)

Flag	Producer	Trigger	Reason label
RESEARCH_AUTO_ENQUEUE_ENABLED	Researcher	composite_k < 0.6 OR zero hits	no_evidence_hits / weak_composite_k
CRITIC_AUTO_ENQUEUE_ENABLED	Critic	verdict == 'reject'	claim_rejected

Independent flags so producer sides roll out separately. Both pass dispatched_from_backlog=False from organic invocations; the consumer dispatcher passes True to suppress re-enqueue and break the loop.

Consumer flag (default OFF)

BACKLOG_CONSUMER_ENABLED — research-engine BacklogConsumerJob auto-starts in lifespan. With no real dispatcher wired (worker-absent path), every pulled row is marked failed_voluntary with failure_reason='no_dispatcher_configured' — honest signal but actively destructive to a queue with no real consumer, hence default-OFF.

Routes

Verb + path	Service	Purpose
POST /admin/v1/reasoning/research/deeper-request	memory-store	Producer enqueue
POST /admin/v1/reasoning/research/backlog/next	memory-store	Atomic FIFO pull (status flip pending → dispatched)
POST /admin/v1/reasoning/research/backlog/{id}/complete	memory-store	Close out; body {flow_id?, error?} — error flips to failed_voluntary
GET /admin/v1/reasoning/research/backlog	memory-store	Operator observability — status_distribution, oldest pending/dispatched age, retry distribution, recent failures, loop_health verdict
POST /admin/v1/reasoning/claims-pending-review	memory-store	Mark a claim for human triage (idempotent on claim_id)
GET /admin/v1/reasoning/claims-pending-review	memory-store	List pending entries + last-24h recent_resolutions
PATCH /admin/v1/reasoning/claims-pending-review/{id}	memory-store	Resolve or dismiss; 409 on already-terminal (no double-resolve)
GET /api/admin/reasoning/pathways/research-backlog	apps/api proxy	Operator UI fetch
GET /api/admin/reasoning/pathways/claims-pending-review	apps/api proxy	Operator UI fetch
PATCH /api/admin/reasoning/pathways/claims-pending-review/:id	apps/api proxy	Operator UI Resolve/Dismiss

Audit shapes

Action	When
reasoning.research.deeper_request_created	Producer enqueue
reasoning.research.backlog.dispatched	Consumer pull
reasoning.research.backlog.completed	Consumer success
reasoning.research.backlog.failed_voluntary	Consumer voluntary-fail (e.g., dispatcher gave up)
reasoning.research.backlog.recovered	Stale-sweep flips back to pending
reasoning.research.backlog.failed_after_retries	Stale-sweep at retry-cap
memory.job.backlog_stale_sweep.completed	Stale-sweep lifecycle
reasoning.claims_pending_review.marked	Consumer marks for review
reasoning.claims_pending_review.resolved	Curator resolves/dismisses
admin.reasoning.research.backlog_read	Observability route call
admin.reasoning.claims_pending_review.listed	Triage queue listing

All events hash-chain through audit_events; backlog rows carry their own 30-day TTL on completed_at so terminal-state queue rows expire while the audit chain remains the long-term record.

Operational hygiene

• Stale-sweep recovery — 30-min threshold flips stuck dispatched rows back to pending (or to failed after 3 retries). Worst-case lockup = sweep cadence (5 min) + threshold = ~35 min.
• Loop-break marker — inputs.dispatched_from_backlog=True suppresses producer auto-enqueue, preventing weak-result→re-enqueue→re-dispatch infinite loops.
• Backlog TTL — completed/failed rows expire 30 days after completed_at; pending/dispatched rows are unaffected (TTL ignores null indexed fields).
• Loop health verdict — server-computed healthy / warn / critical on the observability route. Thresholds: critical at oldest-dispatched ≥25min OR oldest-pending ≥1h; warn at oldest-dispatched ≥15min OR ≥5 recent failures. Single source of truth across UI + monitoring.

Operator surfaces

• Research Console (/dashboard/admin/reasoning/research-console) — backlog status chips · age callouts · recent failures · Loop health pill · Pending review panel · Recently resolved strip · Resolve/Dismiss action buttons.
• Recently resolved — last 10 terminal entries from past 24h, mirrored as audit-trail confirmation; audit_events is the long-term record.

SDK methods (apps/research/research-engine/services/clients/memory_client.py)

Method	Wraps
enqueue_deeper_research	POST deeper-request
pull_next_research_request	POST backlog/next
complete_research_request	POST backlog/{id}/complete
mark_claim_for_review	POST claims-pending-review
list_claims_pending_review	GET claims-pending-review
resolve_claim_review	PATCH claims-pending-review/{id}

All async, all fail-soft (return None on transport error / non-2xx) so caller agents never block on the queue.

The summary_text contract

Agents emit a ≤280-char plain-English summary_text on every step close. The server runs each summary through reasoning_summary_guard at flow close; a flow whose summaries all pass is marked patient_safe_summary=True and the patient narrative route renders it. A flow with any unsafe summary stays admin-only (the patient route returns 403).

Banned patterns (conservative — more false positives than false negatives):

• MRN labeled / bare 9-10 digit ids
• SSN (\d{3}-\d{2}-\d{4})
• ISO-date DOB shapes + DOB: prefix
• US phone numbers
• Email addresses
• Medication-with-dose (e.g. metformin 500mg)
• ICD-10 codes
• Titled proper names (Dr. <First> <Last>)

Agents' rule of thumb: summary describes the step, not the content. "Reviewed recent medications" is safe; "reviewed metformin 500mg" is not.

Lifecycle + decay

ReasoningFlowDecayJob runs hourly (same cadence as ShortTermConsolidationJob + DecayJob). Any flow stuck active for more than 24h transitions to aborted with reason=timeout. This is the cleanup for agent crashes / container cycles that leave zombie flows.

OTel

Metric	Kind	Attributes
reasoning.flow.started	counter	trigger_type
reasoning.flow.completed	counter	outcome_type, trigger_type, patient_safe_summary
reasoning.flow.aborted	counter	outcome_type, trigger_type, patient_safe_summary
reasoning.step.writes	counter	step_kind, agent
reasoning.flow.latency_ms	histogram	outcome_type, trigger_type, step_count_bucket

Phased delivery

• Phase 1a (shipped) — schemas, memory-store routes, decay job, OTel counters, summary guard.
• Phase 1b — SDK ReasoningFlowRecorder context manager (sync + async).
• Phase 1c (shipped) — Researcher / Critic / Correlator each emit a reasoning flow on every real-mode run via emit_agent_flow(...) (fire-and-forget helper in services/agents/_reasoning_flow.py). Flag-gated by REASONING_FLOW_CAPTURE_ENABLED (default OFF) — NoOp recorder means zero behavioral change on merge; flipping the flag in staging activates capture across all 3 agents simultaneously. Each agent emits 3 canonical steps (Researcher: retrieval → llm_prompt → verdict; Critic: graph_walk → llm_prompt → verdict; Correlator: aggregation → llm_prompt → verdict) with non-PHI summaries and back-refs to the LLM call + claim.
• Phase 2a (shipped) — Admin /dashboard/admin/reasoning/[flowId] page renders the full DAG: flow header (state, outcome, trace_id, patient_id) + metadata grid (trigger / trace / patient / outcome / timing) + ordered step timeline with each step's kind, agent, summary, latency, confidence, back-refs (llm_call_log_ref / experiment_trace_ref / decision_graph_node_ref / memory_claim_ref). apps/api proxies to memory-store's admin routes and emits its own admin.reasoning.flow.read audit pinning the calling admin.
• Phase 2b (shipped) — Patient /dashboard/reasoning/[decisionId] narrative page. Renders the sanitizer-passed shape: trigger summary → per-step cards with agent label + safe summary + timestamp → outcome summary + disclaimer. 403 from the server when summaries didn't pass the guard (page shows a gentle "ask your care team" message). 202 in-progress renders a "still working on this" banner with partial step count. Route lives under /api/reasoning/flows/:id on apps/api (distinct from admin). Per-request JWT consent forwarding shipped as a follow-up — the controller extracts the patientrx_access cookie and forwards it as the bearer to memory-store, so memory-store's require_scope("reasoning.read") runs against the patient's identity (defense-in-depth). Falls back to service token when the cookie is absent (rare given the Roles gate). The reasoning.read consent toggle UI is now live at /dashboard/profile (component apps/web/src/components/profile/ReasoningReadConsentToggle.tsx); patients can opt in or out from their profile page.
• Phase 3a (shipped) — PathwayExtractor captures high-confidence completed flows as reasoning_pathways templates. Qualifier: outcome_confidence ≥ 0.9 (min across steps) + outcome_type ∈ {claim_created, recommendation_delivered} + patient_safe_summary=True. Idempotent on seed_flow_id unique index. Extraction fires at the end of the flow-close route; failures don't block the close response. Emits reasoning.pathway.extracted or reasoning.pathway.skipped with the disqualifier reason. embedding + embedding_model fields are stubbed null for Phase 3c.
• Phase 3b (shipped) — PathwayMatcher scores candidate pathways against a new flow's (trigger_type, trigger_ref). Structural score today: exact_ref (1.0) / prefix_ref (0.7) / trigger_type_only (0.3) / no-match (0.0). Admin routes: GET /admin/v1/reasoning/pathways/match returns ranked candidates (read-only; no hit_count bump); POST /admin/v1/reasoning/pathways/{id}/hit atomically bumps hit_count + stamps last_hit_at when the pathway is actually reused. Sorted by score desc then hit_count desc (cold pathways lose to established ones on ties). Emits reasoning.pathway.match_requested and reasoning.pathway.matched audits.
• Phase 3c (shipped) — Pathway embedding layer. Three pieces: (1) pluggable EmbeddingClient Protocol with a HashBucketEmbedding 128-dim deterministic stub + two real adapters: OpenAIEmbedding (text-embedding-3-small, dimensions=128 native to match the Atlas index) and VoyageEmbedding (voyage-3.5-lite at output_dimension=256, truncated to 128 client-side — Voyage's smallest native is 256, so the head-128 slice is a degraded but valid signal until a 256-dim Atlas index is provisioned). Voyage entry is registered in the BAA registry with pending_compliance per the dev-phase deferral policy; (2) PathwayEmbeddingBackfillJob on hourly cadence — populates embedding + embedding_model on un-embedded active pathways, idempotent on re-run, re-embeds when the model name changes; (3) PathwayMatcher composite score = 0.3 × structural + 0.7 × semantic when both query + candidate carry same-model embeddings. Cross-model / un-embedded candidates fall back to pure structural (reason stays structural; +semantic suffix marks blended scores). Negative cosine clips to 0 so anti-similar vectors never push a candidate below baseline. Active adapter resolved at startup via build_embedding_client() reading PATHWAY_EMBEDDING_MODEL (default hash_bucket_v1, opt-in openai_text_3_small — fails fast on unknown value or missing OPENAI_API_KEY). Atlas Search vector-index config at infrastructure/atlas-search/reasoning_pathways_vector.json (128-dim, cosine, filters on trigger_type / demoted_at / embedding_model) — apply via the Admin API runbook when promoting the env. Until the index is applied, matcher uses in-app cosine (scales to ~2k pathways).
• Phase 3d (shipped) — PathwayDecayJob runs daily. Demotes pathways under two rules: cold (hit_count == 0 AND created_at < now - 30d — reason cold_no_hits_30d) or stale (last_hit_at < now - 90d — reason stale_no_recent_hits_90d). Non-destructive: demoted_at + demotion_reason fields stamp the row; PathwayMatcher.match filters demoted pathways out of candidates. Idempotent on re-run. Thresholds are constructor-tunable. Emits memory.job.pathway_decay.{started,completed,error} lifecycle audits + one reasoning.pathway.demoted audit per demoted row (count-only {pathway_id, reason, hit_count, age_days, trigger_type}).
• Phase 4a (shipped) — scripts/export-reasoning-dataset.mjs (a.k.a. npm run reasoning:export). Mongo-direct JSONL export of completed reasoning flows + their projected steps. Flags: --format raw|dpo / --since / --until / --outcome / --min-confidence / --limit / --out. DPO filter keeps only flows whose pathway got reused (hit_count ≥ 1) — downstream scripts pair those against aborted / demoted flows to build preference pairs. Output is PHI-safe: drops inputs/outputs payloads, surfaces back-refs as booleans only, passes summary_text verbatim (sanitizer guard already enforced it at emission time).
• Phase 4b.1 (shipped) — scripts/lint-reasoning-flows.mjs (npm run reasoning:lint). Scans JSONL output from 4a for quality anomalies. 3 critical rules (empty step sequence / verdict-before-retrieval / no LLM ref) + 4 warnings (empty agent_set / aggregate_confidence=1.0 / duplicate consecutive kinds / summary-hidden > 50%). Exit 1 on any critical, 0 on PASS / PASS-WITH-NOTES. --json emits machine-readable findings. Foundation for the full decision-tree eval harness — the linter catches stub-mode regressions + obvious agent-shape drift in seconds without needing a full replay.
• Phase 4b.2-a (shipped) — golden signature corpus + drift comparator. Corpus at specs/066-reasoning-flow-tracker/fixtures/expected-signatures.json declares the canonical (step_kind_sequence, allowed_outcomes) per trigger_type (Researcher / Critic / Correlator). scripts/reasoning-signature-check.mjs (npm run reasoning:signature-check -- --input flows.jsonl) reads a JSONL from reasoning:export and emits per-flow drift findings (sequence_drift / outcome_drift / skipped). Default threshold is 0 (any drift is a regression); --threshold 0.05 loosens for historical sweeps. CI gate reasoning-signature-drift fires on any PR touching the corpus / comparator / fixture and runs the comparator against sample-flows-clean.jsonl (5 matched + 1 intentionally-skipped flow) — corpus drift, comparator regression, and fixture drift all surface here. Pairs cleanly with reasoning:lint (4b.1: universal shape rules) by adding agent-specific signature checks.
• Phase 4b.2-a CI gate (shipped) — reasoning-signature-drift PR-time gate runs the comparator against sample-flows-clean.jsonl on any PR touching the corpus, comparator, or fixture (catches corpus/comparator/fixture triangle drift). reasoning-signature-drift-sweep is a workflow_dispatch job that exports a 24h window of real staging flows and runs the comparator against them; flip on the daily cron after the first green dispatch confirms creds + network. Two complementary signals: synthetic self-test catches code regressions; real-flow sweep catches agent-runtime regressions.
• Phase 4b.2-b foundation (shipped) — replay harness scaffold + 5-case fixture corpus. ReplayHarness + AgentInvoker Protocol + StubAgentInvoker (canonical signatures per trigger_type) + ReplayCase/ReplayResult/ReplayOutcome dataclasses. Fixture corpus at specs/066-reasoning-flow-tracker/fixtures/replay-cases/ (Researcher happy/abort, Critic promote/quarantine, Correlator recommendation). Pytest self-test (14 probes) covers the contract pieces + a corpus self-test that asserts harness ↔ corpus consistency.
• Phase 4b.2-b CLI + CI gate (shipped) — npm run reasoning:replay [-- --threshold 0.05] [--json] [--cases dir] [--corpus path] wraps the harness in a Node CLI mirroring the reasoning:signature-check shape. PR-time gate reasoning-replay-drift fires on any PR touching the replay-cases / expected-signatures / harness / CLI; runs the stub invoker end-to-end (always green today, but catches CLI/harness/fixture regressions). The --invoker flag is reserved for real Researcher/Critic/Correlator adapters; until those ship the CLI accepts only stub.
• Phase 4b.2-b real invokers (deferred) — real Researcher/Critic/Correlator adapters that implement the AgentInvoker Protocol against actual agent code paths. Once shipped, dispatch the reasoning-replay-drift workflow manually with the matching --invoker flag, confirm green, then uncomment its schedule: block to enable the daily cadence.

Benchmark integration

The Spec 067 / Spec 068 benchmark e2e_full pipeline opens a ReasoningFlow per case (trigger_type=user_question, trigger_ref=benchmark/{run_id}/{case_id}) and emits one step per pipeline phase: retrieval → evidence_assembly (sufficiency + optional external chain) → summary (synthesizer) → provenance_check (hallucination guard) → verdict. The Benchmark Console's case-detail dialog deep-links to /dashboard/admin/reasoning/{flow_id} for the chain-of-thought DAG. See Research → Benchmark Console.

The benchmark's recorder is a slim in-package httpx wrapper (apps/research/benchmark/services/e2e/reasoning_recorder.py) rather than the aimonehealth-memory-sdk AsyncReasoningFlowRecorder. The benchmark-runner Docker image deliberately does not ship the SDK; the wrapper covers the three reasoning endpoints in ~100 lines and keeps the sidecar boundary honest. Best-effort per FR-RFT-020 — recorder failures never block the verdict.

For full forensic visibility (raw retrieve bodies, raw synthesizer output) the benchmark also persists a separate per-case execution trace alongside the run artefact. That surface uses a narrowly scoped compliance exemption — see Compliance → Benchmark Trace.

Related specs

• Memory Hierarchy — the three-tier storage this feature indexes against
• Audit Chain — every reasoning.* action hash-chains here
• Data Model — where the two new collections fit in the research tier
• Compliance → Benchmark Trace — narrowly scoped exemption that complements the standard reasoning surface for synthetic benchmark runs
• Spec 035 — Decision Graph (the DAG structure this layer enriches with narrative + multi-agent context)
• Spec 023 — llm_call_logs (one of the four back-ref targets on each step)