Feat/p3 tenant aware correlation

CHANGELOG.md

@@ -8,6 +8,27 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Changed
 
+- **Bounded, tenant/principal-aware proxy conversation correlation store** (P3):
+  the proxy `ConversationCorrelationStore` lineage map is now keyed by
+  `(principal, canonical_history_hash)` instead of the bare history hash, so
+  byte-identical conversation histories from different tenants no longer collide
+  onto one `conversation_id`. The two hash functions
+  (`canonical_history_hash` / `canonical_parent_history_hash`) are **byte-for-byte
+  unchanged** — isolation lives entirely in the map key, and an empty principal
+  reproduces the previous behavior exactly (`("", hash)` keyspace), so existing
+  single-turn and history-based multi-turn correlation is preserved. The store is
+  now bounded (TTL lazy-expiry on read + a max-entries FIFO cap, mirroring
+  `InMemorySessionStore`; defaults 3600 s / 20 000 entries, overridable via
+  `MORALSTACK_CORRELATION_TTL_SECONDS` / `MORALSTACK_CORRELATION_MAX_ENTRIES`),
+  removing the previous unbounded-growth/OOM risk. Principal is derived per
+  request as `X-Moralstack-Tenant-Id` header → HMAC-SHA256 of an
+  `Authorization: Bearer` token (secret from `MORALSTACK_PRINCIPAL_HMAC_SECRET`,
+  read per request, raw token/digest never logged) → empty-string sentinel.
+  `create_app` gains an optional `correlation_store=` parameter. `resolve()` is
+  best-effort on the TTL/eviction path (PROJECT_SPEC §5 invariant #6): a helper
+  failure never propagates into the request handler and a valid `msconv-*` id is
+  always minted. `ConversationLockManager._locks` bounding is deferred
+  (`TODO(P3-followup)`). No P0 decision/governance invariant is affected.
 - **Constitution retrieval unified to a single pass**: `get_relevant_principles`
   now runs exactly once per request, owned inside the risk thread
   (`models/risk/estimator.py:_get_principles_context`) at the unified
@@ -54,6 +75,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Development
 
+- **memory-guard trace-doc path casing fixed**: `scripts/check_memory_updated.py`
+  referenced `docs/TRACES/*` while the repository directory is `docs/traces/*`, making
+  the guard structurally unsatisfiable for `moralstack/server/` and
+  `moralstack/compliance/` changes (their only mapped docs were the mis-cased trace
+  files). Aligned the four `BEHAVIOR_DOC_MAP` entries to the actual `docs/traces/`
+  paths so the gate enforces the intended doc updates again.
 - **AI harness**: added a self-maintaining memory contract and evolved the Claude Code
   hook mechanics (Stop verify dedup, PreCompact snapshot, SessionEnd diary, docs-gate)
   with offline harness tests under `tests/harness/`. Test-suite isolation hardened

README.md

@@ -408,6 +408,9 @@ Environment is loaded via `moralstack/utils/env_loader.py`.
 | `MORALSTACK_OBSERVABILITY_JSONL_DIR` | `logs/observability` | JSONL output directory |
 | `MORALSTACK_DB_PATH` / `MORALSTACK_PERSIST_MODE` | — | Deprecated aliases — still work |
 | `MORALSTACK_ORCHESTRATOR_BORDERLINE_REFUSE_UPPER` | `0.95` | Upper boundary for the borderline-refuse zone |
+| `MORALSTACK_CORRELATION_TTL_SECONDS` | `3600` | TTL (seconds) for the server proxy's lineage correlation store; must exceed the max expected inter-turn delay, and should stay aligned with the session-store TTL so a lineage entry doesn't outlive (or expire well before) its governance session |
+| `MORALSTACK_CORRELATION_MAX_ENTRIES` | `20000` | Max entries in the server proxy's lineage correlation store (FIFO eviction beyond this cap) |
+| `MORALSTACK_PRINCIPAL_HMAC_SECRET` | *(unset)* | Secret used to HMAC the bearer token into a tenant/principal identifier for conversation correlation; when unset, the HMAC principal path is disabled (falls back to the empty-string sentinel) |
 
 ### Default models by component
 

docs/MORALSTACK_CODEBASE_INDEX.md

@@ -270,10 +270,11 @@ Python `>=3.11` (`pyproject.toml:11`). Runtime deps: `openai>=2.24`, `pydantic>=
 - `emit_helpers.py` — `persist_*` / `async_persist_*` telemetry wrappers.
 
 ### Server proxy — `moralstack/server/`
-- `proxy.py` — `create_app(openai_client, orchestrator, config, session_store)`
-  returns a FastAPI app exposing `POST /v1/chat/completions`, `/chat/completions`,
-  `GET /healthz`. `ConversationLockManager` (per-conversation locks),
-  `_handle_chat_completion_sync` (runs in a threadpool). Reads client
+- `proxy.py` — `create_app(openai_client, orchestrator, config, session_store,
+  correlation_store)` returns a FastAPI app exposing `POST /v1/chat/completions`,
+  `/chat/completions`, `GET /healthz`. `ConversationLockManager` (per-conversation
+  locks; `_locks` growth-bounding is deferred — see FACTS), `_handle_chat_completion_sync`
+  (runs in a threadpool). Reads client
   `max_tokens`/`max_completion_tokens`/`temperature`/`top_p` into
   `ProcessedRequest.generation_overrides` via
   `GenerationOverrides.from_mapping(body, passthrough_unset=True)`; the SDK
@@ -283,11 +284,22 @@ Python `>=3.11` (`pyproject.toml:11`). Runtime deps: `openai>=2.24`, `pydantic>=
   defaults do not apply there; on the **SDK/CLI** an unset field falls back to the env
   default (precedence override > `GenerationConfig` > env defaults
   `OPENAI_MAX_TOKENS`/`OPENAI_TEMPERATURE`/`OPENAI_TOP_P`). REFUSE wording is excluded
-  on every path. See `docs/modules/policy.md`.
-- `conversation_correlation.py` — `ConversationCorrelationStore` (lineage hashing
-  → conversation_id) + `canonical_history_hash`, `canonical_parent_history_hash`.
+  on every path. See `docs/modules/policy.md`. Per request, `_extract_principal(request)`
+  derives a tenant/principal string (A: `X-Moralstack-Tenant-Id` header → B: HMAC-SHA256
+  of an `Authorization: Bearer` token via `MORALSTACK_PRINCIPAL_HMAC_SECRET`, read
+  per-request → C: empty-string sentinel) that keys the correlation store's lineage map
+  (P3 / P0-3 / A3; see `docs/traces/openai_compatible_multiturn.md`).
+- `conversation_correlation.py` — `ConversationCorrelationStore` (bounded, TTL +
+  max-entries FIFO eviction; lineage hashing → conversation_id, internal map keyed by
+  `(principal, history_hash)`) + `canonical_history_hash`, `canonical_parent_history_hash`
+  (unchanged by the principal-keying design — isolation lives entirely in the map key).
+  Constructor: `ttl_seconds` (default `DEFAULT_CORRELATION_TTL_SECONDS=3600`),
+  `max_entries` (default `DEFAULT_MAX_CORRELATION_ENTRIES=20_000`), `time_fn` (clock seam
+  for tests). `create_app` wires `MORALSTACK_CORRELATION_TTL_SECONDS` /
+  `MORALSTACK_CORRELATION_MAX_ENTRIES` (best-effort parse + range-validate, never raises).
 - `headers.py` — `build_governance_headers` (X-Moralstack-* response headers).
-- `fingerprint.py` — request fingerprinting.
+- `fingerprint.py` — request fingerprinting (calls the unchanged `canonical_history_hash`;
+  verified unaffected by the principal-keying change).
 
 ### UI — `moralstack/ui/`
 - `app.py` — FastAPI dashboard (`moralstack-ui`). Reads exclusively from the
@@ -718,10 +730,19 @@ See `docs/traces/complai_llm_rules_flow.md`.
 
 ## 17. Known fragile areas
 
-- **Lineage-based conversation correlation can collide.** Two samples with
-  byte-identical histories (and identical assistant outputs) map to the same
-  `conversation_id` (`conversation_correlation.py` docstring + `resolve`). This
-  is the central COMPL-AI risk — see `docs/traces/complai_llm_rules_flow.md`.
+- **Lineage-based conversation correlation can still collide within a principal.**
+  Two samples with byte-identical histories (and identical assistant outputs) **for
+  the same principal** map to the same `conversation_id` (`conversation_correlation.py`
+  docstring + `resolve`) — this is the central COMPL-AI risk (see
+  `docs/traces/complai_llm_rules_flow.md`). As of P3 / P0-3 / A3, this no longer
+  crosses tenant/principal boundaries: the internal lineage map is keyed by
+  `(principal, history_hash)`, so identical histories from *different* principals
+  never collide (see `docs/traces/openai_compatible_multiturn.md`).
+- **`ConversationLockManager._locks` growth is still unbounded (deferred).** One
+  `threading.Lock` is created per distinct `conversation_id` and never removed
+  (`proxy.py:83` area, `# TODO(P3-followup)`); the correlation store's TTL/max-entries
+  bound already removes the dominant growth source (~2 entries/turn vs 1 lock/conversation).
+  A safe idle-prune has a real race (see the plan for P3 / P0-3 / A3) and is a follow-up.
 - **UI requires SQLite.** `file_only` runs never appear in the dashboard; the UI
   reads only the DB.
 - **Cache governance.** Ledger fast-path can reuse a prior decision; the

docs/traces/openai_compatible_multiturn.md

@@ -65,54 +65,127 @@ Empty governed content fails closed to a governed refusal
 guard fields are retained as audit metadata, but they no longer route delivery to
 an upstream call. The SDK uses the same governed delivery invariant.
 
-### conversation_id generation / propagation (`proxy.py:218-219`, `121-136`)
-Resolution precedence:
+### conversation_id generation / propagation (`proxy.py:134-152`, resolver;
+`proxy.py:654-709`, route handler)
+Resolution precedence (principal keying only affects step 3):
 1. HTTP header `X-Moralstack-Conversation-Id` (highest).
 2. `extra_body.moralstack_conversation_id`.
-3. **Lineage correlation** (`ConversationCorrelationStore.resolve`).
+3. **Lineage correlation** (`ConversationCorrelationStore.resolve`), keyed
+   additionally by `principal` (see below).
+
+**Principal derivation (P3 / P0-3 / A3, `proxy.py:178-215`, `_extract_principal`)**
+— layered, first match wins:
+- **(A)** Trusted internal header `X-Moralstack-Tenant-Id` — used verbatim
+  (whitespace-stripped) when present and non-empty. Set by a trusted fronting
+  layer; this is a **documented trust boundary, not enforced by MoralStack**
+  (a hostile client could forge the header unless the fronting layer strips
+  client-supplied copies). Note this header starts with `x-` and therefore
+  passes `_collect_safe_headers` (`proxy.py:164-175`) and appears in the
+  correlation debug line (`proxy.py:683-691`) — acceptable for an identifier,
+  not for a secret.
+- **(B)** HMAC-SHA256 of an `Authorization: Bearer <token>` value, keyed by
+  `MORALSTACK_PRINCIPAL_HMAC_SECRET` (read **per-request** via `os.environ.get`,
+  not captured once at `create_app`, so secret rotation takes effect
+  immediately). Only a `Bearer` scheme triggers this path; a non-Bearer scheme
+  (e.g. `Basic`) or a malformed/empty value skips to (C). If the secret is
+  unset, (B) is silently skipped (no hardcoded fallback). Neither the raw
+  token nor the derived digest is ever logged (the token stays filtered by
+  `_SENSITIVE_HEADER_MARKERS`, `proxy.py:155-161`). **Token rotation mid-conversation
+  changes the principal and silently splits lineage into a new conversation_id**
+  (expected, not a bug).
+- **(C)** Empty-string sentinel `""` — the pre-change, no-principal behavior.
+  All existing single-tenant deployments and the byte-equality-locked tests
+  take this path.
+
+Composite keying is **anti-collision only, not an authentication/authorization
+boundary**: it prevents two tenants' identical histories from sharing one
+`conversation_id`, but it does not verify tenant identity beyond the (A)/(B)
+trust assumptions above.
 
 Lineage correlation (`server/conversation_correlation.py`):
-- `canonical_history_hash(messages)` = SHA-256 over canonicalized role+content.
+- `canonical_history_hash(messages)` = SHA-256 over canonicalized role+content
+  (`:67-75`). **Unchanged by the principal-keying design** — no `salt` param, no
+  envelope; the isolation lives entirely in the store's internal map key.
 - `canonical_parent_history_hash` = hash of `messages[:-1]` when the last message
-  is `user`.
-- `resolve`: if the request hash is known → return its conversation_id; else if
-  the parent hash is known → inherit that conversation_id and record the request
-  hash; else mint a new `msconv-<uuid16>` id (`conversation_correlation.py:99-114`).
-- After a turn completes, `observe_completed_turn` records the history *including*
-  the assistant reply so the next request's parent hash links back
-  (`conversation_correlation.py:116-129`).
-
-### turn_index handling (`proxy.py:526-541`)
+  is `user` (`:78-91`).
+- The store's internal map is `OrderedDict[tuple[str, str], _Entry]` keyed by
+  `(principal, hash)` (`:112-138`) — bounded via TTL (default
+  `DEFAULT_CORRELATION_TTL_SECONDS=3600`) and a max-entries FIFO cap (default
+  `DEFAULT_MAX_CORRELATION_ENTRIES=20_000`, `:41-42`), overridable via
+  `create_app(correlation_store=...)` or the env vars below. `resolve`: if the
+  `(principal, request_hash)` key is known and not expired → return its
+  conversation_id (hit path does **not** refresh the entry's TTL); else if
+  `(principal, parent_hash)` is known and not expired → inherit that
+  conversation_id and record the request hash under the same principal; else
+  mint a new `msconv-<uuid16>` id (`:140-159`).
+- After a turn completes, `observe_completed_turn` records the completed history
+  (including the assistant reply) under `(principal, completed_hash)` so the next
+  request's parent hash links back within the same principal
+  (`:161-176`).
+- Env wiring (`create_app`, `proxy.py:544-574`, `608-611`):
+  `MORALSTACK_CORRELATION_TTL_SECONDS` / `MORALSTACK_CORRELATION_MAX_ENTRIES`.
+  Best-effort parsed **and** range-validated (`ttl_seconds > 0`, `max_entries >= 1`);
+  a missing var, a parse error, or an out-of-range value falls back to the
+  constructor defaults — `create_app` never raises on a malformed value.
+  **Caveat**: the correlation TTL is not automatically aligned with the
+  `SessionStore` TTL; if an operator overrides only one, a lineage entry can
+  outlive (or expire before) its governance session state.
+
+### turn_index handling (`proxy.py:717-732`)
 Stateless: `turn_index = max(0, user_message_count - 1)`. Turn 0 = first request
 with one user message; turn 1 = two user messages, etc. Chosen so a server
 restart or multiple clients sharing a conversation_id don't desync from the
 client's view.
 
 ### Collision risks
-- **Identical histories collide.** The module docstring states it explicitly:
-  two distinct samples whose histories (and assistant outputs) are byte-identical
-  cannot be distinguished without an external id
-  (`conversation_correlation.py:10-12`). For benchmarks that reuse the same
-  opening user message across many samples, all of them hash-collide to **one**
-  conversation_id.
+- **Identical histories from the same principal still collide.** The module
+  docstring states it explicitly: two distinct samples whose histories (and
+  assistant outputs) are byte-identical, **for the same principal**, cannot be
+  distinguished without an external id (`conversation_correlation.py:17-18`).
+  For benchmarks that reuse the same opening user message across many samples
+  under one principal (the common no-tenant-header COMPL-AI case, principal
+  `""`), all of them hash-collide to **one** `conversation_id`. As of P3 / P0-3 /
+  A3, this collision **no longer crosses principal/tenant boundaries**: the
+  internal lineage map is keyed by `(principal, hash)`, so two tenants with
+  byte-identical histories resolve to different `conversation_id`s.
 - **Consequence (verified mechanism)**: the resolved conversation_id is the key
-  for the per-conversation lock (`ConversationLockManager`, `proxy.py:87-110`),
-  the `SessionStore` entry (`proxy.py:256,303-304`), and the ledger key
-  (`ledger.py:254`). So colliding requests are serialized under one lock and
-  share one governance-state/ledger entry — i.e. a decision or posture from one
-  sample can be read by another. (Whether a given run actually collides depends
-  on the benchmark data containing identical-history samples.)
+  for the per-conversation lock (`ConversationLockManager`, `proxy.py:78-131`),
+  the `SessionStore` entry (`proxy.py:381`, `:438`). **The ledger is NOT keyed by
+  `conversation_id`** — its key is `(contract_hash, posture, domain)`
+  (`orchestration/ledger.py:262`, `LedgerKey`); an earlier version of this trace
+  incorrectly cited `ledger.py:254` (a `LedgerResult` return statement, not a
+  key) as a collision-shared surface — corrected here (code wins, PROJECT_SPEC
+  §9). So same-principal colliding requests are serialized under one lock and
+  share one governance-state entry — i.e. a decision or posture from one sample
+  can be read by another sample under the *same* principal. (Whether a given
+  run actually collides depends on the benchmark data containing
+  identical-history samples under the same principal.)
 - **Mitigation available**: send a unique `X-Moralstack-Conversation-Id` header
   (or `extra_body.moralstack_conversation_id`) per logical conversation to bypass
-  lineage hashing entirely.
-
-### Worker / concurrency implications (`proxy.py:72-119`, `234-373`)
+  lineage hashing entirely, or a per-tenant `X-Moralstack-Tenant-Id` /
+  `Authorization` to isolate by principal.
+- **Cross-tenant availability effect (new, P3 / P0-3 / A3)**: all principals
+  share one `max_entries` eviction pool, so a noisy tenant can evict another
+  tenant's lineage entries under load, causing a mid-conversation
+  `conversation_id` split for the affected tenant. This is an availability
+  effect, not a confidentiality leak (composite keying still prevents any
+  content/state sharing across principals). A per-principal cap is a possible
+  follow-up.
+
+### Worker / concurrency implications (`proxy.py:78-131`)
 - `ConversationLockManager` hands out one `threading.Lock` per conversation_id;
   empty conversation_id → no lock (independent request). Acquire timeout 30s →
-  HTTP 503 with `Retry-After: 10` (`ConversationLockTimeout`).
+  HTTP 503 with `Retry-After: 10` (`ConversationLockTimeout`). **`_locks` growth
+  is unbounded and its bounding is deferred** (`proxy.py:90-96`,
+  `# TODO(P3-followup)`): a naive idle-prune races `acquire()` releasing
+  `_meta_lock` before the blocking `lock.acquire()` call. The correlation
+  store's TTL/max-entries bound (above) already removes the dominant growth
+  source (~2 entries/turn vs 1 lock/conversation); a refcounted-waiter design is
+  the follow-up.
 - Must run a **single uvicorn worker** for multi-turn: each `--workers N` process
-  has its own pipeline, session store, and lock namespace; routing turns of one
-  conversation to different workers breaks continuity
+  has its own pipeline, session store, and lock/correlation-store namespace;
+  routing turns of one conversation to different workers breaks continuity, and
+  principal isolation and bounding are per-process, not cluster-wide
   (`examples/server_quickstart.py:16-21`).
 - Different conversation_ids run concurrently (threadpool); same conversation_id
   is serialized.