Memory Hierarchy

One Health stores knowledge in three tiers with distinct durability, audit posture, and retention contracts. Two of them ship today (episodic audit chain + long-term knowledge graph); the third (short-term working memory) lands with Spec 065 Phase 2.

This page is the canonical map for operators, new engineers, and compliance reviewers.

The three tiers

Tier summary

Tier	Surface	Durability	PHI handling	Governs
Episodic	audit_events	Immutable · 6y HIPAA floor · indefinite in practice	Payloads contain only identifiers + metadata; no PHI content	Principle IV — audit by default
Short-Term	memory_short_term (Spec 065)	TTL-bounded per scope_kind (1h default) · auto-deletes on expiry · anonymizes-in-place if PHI	@phi_repository envelope encryption; pii_fields per-row; patient_id scope required for non-admin	FR-KIA-003/004/005/009
Long-Term	memory_claims + memory_entities + memory_relationships	Indefinite while Active; decay_at → anonymize-in-place; freshness decays on-read per source tier	Same envelope + consent posture as short-term	Specs 01, 05, 063, 064

Per-scope TTL defaults (Short-Term)

scope_kind	Default TTL	Meaning
research_task	3600s (1h)	Matches worker PRIORITY SLA
agent_run	600s (10m)	Short enough that agent restarts lose it cleanly
user_session	900s (15m)	Matches session idle-timeout
admin_session	1800s (30m)	Extended window for admin debug workflows

Per-scope caps live in apps/research/memory-store/services/config/memory_short_term_config.yml. See Spec 065 FR-MEM-011.

Freshness + consolidation (Long-Term)

Tier-1 sources (NEJM · JAMA · Cochrane · NCCN) have a 365-day freshness half-life; tier-2 (PubMed · Elsevier · Wiley · LWW · ClinicalTrials.gov · ClinVar · DrugBank) have 180 days; tier-3 (patient-note · open-web · clinical-narrative) have 30 days. Computed on-read via compute_freshness(claim, now, config) in aimonehealth_contracts.freshness.

Three consolidation paths govern STM → LTM graduation (Spec 065 Phase 3, shipped):

1. Explicit — agent calls MemoryClient.consolidate(stm_id, claim_input). Routes through POST /api/memory/short-term/{stm_id}/consolidate, creates the LTM claim via the shared ClaimRepository, stamps consolidated_to on the STM row, emits paired memory.claim.create + memory.stm.consolidated audit events with trigger="explicit". 409 on already-consolidated rows.
2. Age-triggered — hourly ShortTermConsolidationJob scans memory_short_term for rows with ttl_expires in the next 15 minutes. Per-scope_kind policy is hardcoded drop for v1 (all 4 scope_kinds drop silently — agents that want durability call the explicit path). Emits memory.stm.expired with trigger="age_triggered" and action_detail="age_triggered_drop" (or _unknown_kind for future scope_kinds). Idempotent via warn_emitted_at stamp.
3. Signal-triggered — STMSignalWatcher tails the memory_claims change stream for transitions to state="active". For each promoted claim it scans STM rows where related_claim_ids contains the claim id AND consolidate_on_critic_promote=true AND consolidated_to=None. Stamps each match and emits memory.stm.signal_triggered with trigger="signal_triggered" and signal="claim_promoted_to_active". Opt-in per STM row (default false). Falls back silently on standalone Mongo (no replica set) — age-triggered path still covers those rows.

A fourth re-research trigger — FreshnessConfidenceJob (Spec 063 FR-063-003) — is conceptually Tier-3 re-consolidation: claims below confidence threshold or past max age are re-enqueued for Critic re-evaluation. It is not an STM → LTM path; it operates entirely within LTM and triggers the per-claim refresh pipeline.

Transition matrix (19 flows)

Every move between tiers (or within one) has a named transition, a tool, and an audit event. See Requirements /Group6/SPEC-08-Memory-Hierarchy.md § 2.8 for the complete table with retention per row.

Classes:

• Write-side (T1–T7) — Source→Episodic · Source→STM · STM refresh · Source→LTM · STM→LTM consolidate · LTM→LTM supersede · LTM→STM demote
• Read-side (R1–R4) — per-tier reads + blended get_context_bundle() (Phase 4)
• Recovery (RC1–RC3) — audit replay → STM reconstruction · claim history · LTM → STM workbench seed
• Deletion (D1–D5) — STM TTL non-PHI · STM DecayJob anon PHI · LTM archive · LTM anon · Episodic never

Illegal transitions (fail-closed):

• ❌ Source → Episodic bypassing originating tier
• ❌ STM → STM cross-scope (admin-only override)
• ❌ LTM → LTM direct content rewrite (corrections only via supersede)
• ❌ Episodic → any writable tier (replay produces derived data, never mutates)

Audit event catalog

Event	Tier	Emitted by
memory.claim.create / .state_transition / .supersede	LTM	claim routes
memory.entity.create / .state_transition	LTM	entity routes
memory.retrieval.hybrid	LTM read	HybridRetriever
memory.stm.write / .read / .expired / .anonymized	STM	STM repo + route + DecayJob
memory.govern.admin_stm_read	STM admin	admin bypass path
memory.stm.consolidated	STM → LTM (explicit)	POST /short-term/{id}/consolidate
memory.stm.signal_triggered	STM → LTM (signal)	STMSignalWatcher on Active-promotion
memory.job.stm_consolidation.{started,completed,error}	STM → LTM (age)	ShortTermConsolidationJob lifecycle
memory.bundle.resolved	all tiers	POST /api/memory/bundle (every get_context_bundle server resolve)
research.freshness_confidence.tick / .enqueued	LTM re-consolidation	FreshnessConfidenceJob (Spec 063)
memory.govern.retire_attempt / .archived_entity_audit	LTM governance	entity retirement

All payloads carry only identifiers + numeric scalars + action metadata. No claim text, entity names, MRN/SSN/DOB, or narrative content ever reaches the audit payload.

OTel metrics (T065-109)

Every surface in the catalog above also emits an OTel counter, shipped via Azure Monitor (APPLICATIONINSIGHTS_CONNECTION_STRING). Counters are bucketed low-cardinality to stay within the Azure Monitor metric dimension limit:

Metric	Kind	Attributes
memory.stm.writes	counter	scope_kind
memory.stm.reads	counter	row_count_bucket (0 | 1+)
memory.stm.deletes	counter	—
memory.stm.expired	counter	scope_kind, reason (age_triggered_drop | unknown_kind)
memory.stm.consolidated	counter	trigger (explicit | age_triggered | signal_triggered), scope_kind
memory.bundle.resolved	counter	query_provided
memory.bundle.latency_ms	histogram	query_provided, stm_count_bucket, ltm_count_bucket, episodic_count_bucket
memory.ltm.consolidation	counter	trigger, scope_kind

If the OTel SDK isn't configured (dev box without APPLICATIONINSIGHTS_CONNECTION_STRING), every counter is a safe no-op — code paths never raise because telemetry is absent. KQL alert JSON is tracked separately (follow-up).

Retention policy

Tier	PHI rows	Non-PHI rows
Episodic	Append-only; 6y HIPAA floor; indefinite in practice	Same
STM	TTL → anonymize-in-place → 6y metadata retention (audit parity)	TTL → hard-delete
LTM	decay_at → anonymize-in-place → 6y metadata; freshness decays on-read but row stays Active until state transition	Decay-at → hard-delete

The 6-year floor for anonymized-but-metadata-retained rows matches the audit retention. Patient-deletion requests trigger soft-delete via state transition; hard-delete waits for the audit floor.

How new engineers use this

1. Agent wants working state between ticks → STM (MemoryClient.create_stm once Phase 2 ships).

2. Agent wants to compose an LLM prompt → MemoryClient.get_context_bundle(scope_id, patient_id, query=..., include_episodic=true) (FR-MEM-030, shipped). Returns ContextBundle with STM rows + optional LTM RetrievalResult + live episodic events (audit replay filtered by related_claim_ids, T065-108). 30s in-process cache keyed on (scope_id, patient_id, query) (FR-MEM-031). Emits memory.bundle.resolved count-only audit on every server-side resolve (FR-MEM-032). The research-engine's async MemoryStoreClient.get_context_bundle() mirrors the sync SDK method and is safe to drop into any agent's prepare.py / experiment.py on top of the existing retrieve() fallback path.

3. Agent produces a durable fact → POST /api/memory/claims (spec 01) OR MemoryClient.consolidate(stm_id, claim_input) (Phase 3, shipped). Set consolidate_on_critic_promote=True on the STM row at create time to opt in to signal-triggered auto-promotion when the referenced claim flips to Active.

4. Agent corrects a fact → supersede via POST /api/memory/claims with supersedes: [...].

5. Compliance reviewer → query audit_events filtered by the tier-specific action names above.

6. Operator observability → /dashboard/admin/memory-hierarchy renders STM counts (grouped by scope_kind) + LTM counts (grouped by lifecycle_state) + a 24h rolling episodic count via GET /api/admin/memory-hierarchy/counts (apps/api proxy) → GET /admin/v1/memory-hierarchy/counts (memory-store) (FR-MEM-050, T065-106 + T065-108). Episodic wiring via AuditReplayService.count_recent — on replay error the field returns null and the card renders "—". The proxy hop emits an admin.memory_hierarchy.counts.read audit event pinning the calling admin; memory-store emits its own admin-layer audit too. Page gracefully degrades to demo-mode fixture if the proxy 5xx's. Screen Profile admin-memory-hierarchy.

7. Operator debugging (CLI) → npm run memory:inspect counts (or --json for piping). Calls GET /admin/v1/memory-hierarchy/counts, renders a tier table, and emits admin.memory_tier.cli_invocation audit per FR-MEM-082. Three additional read subcommands ship today:

   • replay --scope <id> [--since <ts>] [--limit <n>] — calls GET /admin/v1/memory-hierarchy/replay/{scope_id} (RC1) and renders a memory.stm.* timeline for a single STM scope. Use when an agent crashed mid-tick and you need to reconstruct the STM state the agent saw.

   • stm --scope <id> --justification "<text>" [--patient <id>] — calls GET /admin/v1/memory-hierarchy/stm/{scope_id} and renders the current non-expired STM rows for a scope. PHI-redacted — admins get metadata + identifier fields but not plaintext content (the @phi_repository interceptor redacts because the route passes scope=None). --justification is required per FR-MEM-082.

   • ltm --entity <id> [--state <s>] [--patient <id>] [--limit <n>] — calls GET /admin/v1/memory-hierarchy/ltm/{entity_id} and lists claims on that entity across ALL lifecycle states (active / pending / superseded / retracted / archived / critic_quarantine). object bodies surface as <encrypted> when envelope-encrypted so admins never see plaintext PHI here; consent-scoped GET /api/memory/claims/{id} is the path for full decryption.

   • consolidate --stm <id> --from-file <path> --justification "<text>" [--apply] — dry-run by default (prints what would post); --apply actually performs the STM → LTM consolidation via POST /api/memory/short-term/{stm_id}/consolidate. Audit payload records {stm_id, claim_id, dry_run}.

   demote stays scaffolded — the LTM→STM workbench-demote route (§2.8.3 RC3) isn't built yet; the CLI exits 99 with a clear pointer when invoked.

Agent integration: Spec 065 T065-105 is complete. Researcher / Critic / Correlator all call memory_client.get_context_bundle(scope_id, patient_id, query, ...) on their LTM path and extract bundle["ltm"]. MemoryRetrievalError behavior is preserved for fallback paths. Replicator + Librarian don't do retrieval (both are write-side agents).

Related specs

• Knowledge graph — the LTM schemas in depth
• Knowledge lifecycle — add / change / delete operations on LTM
• Hybrid retrieval — the read surface that returns LTM + freshness
• Audit chain — episodic hash-chain mechanics
• Data model — collection-level schemas + retention
• Reasoning flow — Spec 066; captures the "why" behind every claim by threading llm_call_logs + decision_graph_nodes + memory_claims + audit_events via a shared trace_id
• Spec 01: Knowledge Memory Store
• Spec 063: Confidence & Freshness Wiring
• Spec 064: Numeric Freshness Scorer
• Spec 065 (this spec): Memory Hierarchy
• Requirements /Group6/SPEC-08-Memory-Hierarchy.md — the requirements-level source of truth