Superseded by #9

nvflow/provenanceguard/README.md

@@ -0,0 +1,109 @@
+# ProvenanceGuard Open v1
+
+Uncalibrated, not paper-faithful open approximation of the research
+ProvenanceGuard system.  Provides claim decomposition, embedding-centroid
+source routing, DeBERTa NLI scoring, and protected-value checking to produce
+per-rollout-row `allow` / `block` / `unavailable` verdicts.
+
+## Algorithm
+
+**routing-nli-v1** — pipeline: decompose, route, NLI, protected-value, decision.
+
+## Decision Semantics
+
+- **allow**: every claim is entailed AND every protected value found in evidence.
+- **block**: any claim is contradiction, neutral, no_source, or protected_value_mismatch.
+- **unavailable**: no evidence, no claims extracted, or any NLI/model error on any claim.  If any claim has an error, the row is unavailable even alongside block-worthy verdicts.
+
+## Evidence Basis
+
+`retrieval_model_excerpt` — evidence comes from `retrieve_information` tool-call
+outputs in the rollout trace, NOT from primary SEC filing verification.
+
+## Generic vs Finance Layers
+
+| Layer | Location | Contents |
+|---|---|---|
+| Generic | `nvflow/provenanceguard/` | Types, protocols, decomposer, embedder, router, NLI, evaluator, protected_values |
+| Finance sidecar + CLI | `nvflow/recipes/finance/utils/rl/provenanceguard.py` | Trace extraction, deterministic/atomic sidecar evaluator, CLI |
+| Finance stage | `nvflow/recipes/finance/stages/rl/evaluate_provenance.py` | Registered as `finance/grpo/evaluate_provenance` |
+
+The generic package contains no finance-specific code.  Trace extraction,
+sidecar evaluation, and CLI live in the finance recipe layer.
+
+## Enablement
+
+Uncomment `# - evaluate_provenance` in `pipeline_stages` in
+`nvflow/recipes/finance/workflows/grpo/base.yaml`.  The stage config
+(`stages.evaluate_provenance`) is active by default; only the pipeline_stages
+entry is commented out (opt-in).
+
+## Paths
+
+- **Input**: `${directories.step-5-collect-rollouts}/{env}/rollout/output-rs<seed>.jsonl`
+- **Input marker** (required): `output-rs<seed>.jsonl.done`
+- **Output**: `${directories.provenanceguard-eval}/{env}/provenanceguard-rs<seed>.jsonl`
+- **Output marker** (created atomically after `os.replace`): `provenanceguard-rs<seed>.jsonl.done`
+- **Directory**: `${model_output_dir}/provenanceguard-eval`
+
+Depends only on `collect_rollouts`; no downstream dependencies.
+
+## Sidecar Protocol
+
+- Exactly one sidecar row for every physical input line (including blank/malformed/non-object).
+- No random UUID; `evaluation_uuid` is a deterministic SHA-256 of raw-line fingerprint + seed + line number + algorithm + config digest.  Duplicate identical rows get distinguishable IDs by physical line position.
+- Input gate first: both the input file and its sibling `.done` marker must exist before any output state is touched.  A missing input gate preserves prior output and output `.done` unchanged.
+- Stale output `.done` marker is cleared only after the input gate passes, before mkdir/temp/evaluation.  Atomic temp write + `os.replace`; a fresh empty `.done` marker is created only after successful replace.  If evaluation fails after the valid input gate, prior output remains intact but the stale `.done` marker stays absent.
+- Input file is never modified.
+
+## Source ID Stability
+
+Stable canonical IDs (`sec:cik=<10-digit>:accession=<...>:doc=<...>`) are only
+produced when a 10-digit zero-padded CIK + accession + document are all
+present.  Never falls back to storage keys or URLs for the canonical ID.
+Multiple keys, any unknown key, or no canonical ID yields `attribution_state:
+unavailable`; `source_ids` contains only known canonical candidates.
+
+## Protected Values
+
+Full normalized dates/numbers/currency/percent are required for a match.
+A bare 4-digit year in the evidence does NOT satisfy a date protected value.
+
+## NLI Label Validation
+
+The NLI scorer validates actual model config label names
+(entailment/neutral/contradiction, case-insensitive).  It never assumes
+arbitrary `LABEL_0` ordering — if a label does not match one of the three
+canonical classes, it raises `ValueError`.
+
+## Model IDs and Licenses
+
+| Model | ID | License (from public model card) |
+|---|---|---|
+| Routing embedder | `sentence-transformers/all-MiniLM-L6-v2` | Apache-2.0 |
+| NLI scorer | `MoritzLaurer/DeBERTa-v3-base-mnli-fever-anli` | MIT |
+
+Both are lazy-loaded (torch/transformers imports happen inside `_ensure_loaded`,
+not at module import time).  Production deployments should pin revisions from
+Hugging Face Hub metadata; revisions are optional and not hardcoded.
+
+## Differences from Private/Research Harness
+
+This open implementation differs from the private/research ProvenanceGuard
+harness in several ways.  No parity is claimed.
+
+- **Rule-based decomposition** rather than LLM/Gemma-based decomposition.
+- **No token alignment/conflation head** — claims are not aligned to specific
+  evidence tokens.
+- **No RF calibration** — NLI scores are used directly without random forest
+  calibration.
+- Evidence is model-retrieved excerpts, not primary SEC verification.
+
+## Tests
+
+```bash
+uv run --frozen --offline pytest tests/test_provenanceguard.py -q -o addopts=""
+```
+
+Tests use deterministic fakes (no model downloads, no network).  No real model
+execution or download is performed.

nvflow/provenanceguard/__init__.py

@@ -0,0 +1,52 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""ProvenanceGuard Open v1 — routing-nli-v1.
+
+An uncalibrated, not paper-faithful open approximation of the research
+ProvenanceGuard system.  Provides claim decomposition, embedding-centroid
+source routing, DeBERTa NLI scoring, and protected-value checking to
+produce per-rollout-row allow / block / unavailable verdicts.
+
+All Hugging Face model loading is strictly lazy — no model download or
+real model execution is performed at import time.
+"""
+
+from nvflow.provenanceguard.protocols import (
+    ClaimDecomposer,
+    Embedder,
+    NLIScorer,
+    RoutedClaim,
+    SourceRouter,
+)
+from nvflow.provenanceguard.types import (
+    AtomicClaim,
+    ClaimVerdict,
+    Decision,
+    EvidenceChunk,
+    NLIResult,
+)
+
+__all__ = [
+    "AtomicClaim",
+    "ClaimDecomposer",
+    "ClaimVerdict",
+    "Decision",
+    "Embedder",
+    "EvidenceChunk",
+    "NLIScorer",
+    "NLIResult",
+    "RoutedClaim",
+    "SourceRouter",
+]

nvflow/provenanceguard/decomposer.py

@@ -0,0 +1,116 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""Deterministic rule-based claim decomposer (v1 default).
+
+**Limitation**: This is a conservative sentence/claim splitter that does
+not call any LLM.  It splits on sentence boundaries and then further
+splits compound claims on conjunctions (``and``, ``;``, ``--``).  It may
+over-merge compound claims or split mid-clause.  The research
+ProvenanceGuard uses Gemma-based decomposition, which is not included
+in this open implementation.
+
+Claim IDs are deterministic SHA-256 hashes of the claim text and the
+source sentence, so re-evaluation of the same answer yields stable IDs.
+"""
+
+from __future__ import annotations
+
+import hashlib
+import re
+from collections.abc import Sequence
+
+from nvflow.provenanceguard.types import AtomicClaim
+
+# SEC filing URL pattern — used to extract stated SEC IDs/URLs from claim text.
+_SEC_URL_RE = re.compile(
+    r"https?://(?:www\.)?sec\.gov/[^\s\"'<>]+",
+    re.IGNORECASE,
+)
+
+# CIK pattern — 10-digit zero-padded numbers.
+_CIK_RE = re.compile(r"\b(\d{10})\b")
+
+# Accession number pattern — e.g. 0001811414-25-000010
+_ACCESSION_RE = re.compile(r"\b(\d{10}-\d{2}-\d{6})\b")
+
+# Sentence boundary: period, exclamation, question mark followed by space
+# or end of string.  Also splits on newlines.
+_SENTENCE_END_RE = re.compile(r"(?<=[.!?])\s+|\n+")
+
+# Compound claim splitters within a sentence.
+_COMPOUND_SPLIT_RE = re.compile(r"\s*;\s*|\s+--\s+|\s+but also\s+", re.IGNORECASE)
+
+
+def _stable_claim_id(text: str, source_sentence: str) -> str:
+    """Deterministic 16-char hex hash for a claim."""
+    raw = f"{text}|{source_sentence}"
+    return hashlib.sha256(raw.encode("utf-8")).hexdigest()[:16]
+
+
+def _extract_stated_sec_ids(text: str) -> tuple[tuple[str, ...], tuple[str, ...]]:
+    """Extract SEC CIKs and accession numbers mentioned in claim text."""
+    ids: list[str] = []
+    for m in _CIK_RE.finditer(text):
+        ids.append(f"cik:{m.group(1)}")
+    for m in _ACCESSION_RE.finditer(text):
+        ids.append(f"accession:{m.group(1)}")
+    urls = tuple(m.group(0) for m in _SEC_URL_RE.finditer(text))
+    return tuple(ids), urls
+
+
+class RuleBasedDecomposer:
+    """Deterministic sentence/claim splitter.
+
+    Splits the answer into sentences, then further splits compound
+    claims on semicolons, em-dashes, and ``"but also"`` connectors.
+    Filters out empty fragments and very short noise (< 3 words).
+    """
+
+    def decompose(self, answer: str) -> Sequence[AtomicClaim]:
+        if not answer or not answer.strip():
+            return []
+
+        # Normalize whitespace but preserve sentence structure.
+        text = answer.strip()
+
+        # Split into sentences.
+        sentences = [s.strip() for s in _SENTENCE_END_RE.split(text) if s.strip()]
+        if not sentences:
+            sentences = [text]
+
+        claims: list[AtomicClaim] = []
+        for sentence in sentences:
+            # Further split compound claims.
+            fragments = _COMPOUND_SPLIT_RE.split(sentence)
+            for frag in fragments:
+                frag = frag.strip()
+                if not frag:
+                    continue
+                # Skip very short fragments (noise).
+                if len(frag.split()) < 3:
+                    continue
+                sec_ids, sec_urls = _extract_stated_sec_ids(frag)
+                claim_id = _stable_claim_id(frag, sentence)
+                claims.append(
+                    AtomicClaim(
+                        claim_id=claim_id,
+                        text=frag,
+                        source_sentence=sentence,
+                        stated_sec_ids=sec_ids,
+                        stated_sec_urls=sec_urls,
+                    )
+                )
+
+        return claims

nvflow/provenanceguard/embedder.py

@@ -0,0 +1,104 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""Lazy Hugging Face embedder for claim/evidence routing.
+
+Default public model: ``sentence-transformers/all-MiniLM-L6-v2``
+(Apache 2.0).  Model construction is strictly lazy — ``torch`` and
+``transformers`` imports happen inside ``_ensure_loaded``, not at
+module import time.
+
+Uses the raw Transformers ``AutoTokenizer`` / ``AutoModel`` recipe with
+mean pooling over ``last_hidden_state`` with attention mask, L2-normalized,
+returning plain float lists — so no ``sentence-transformers`` runtime
+dependency is needed.
+"""
+
+from __future__ import annotations
+
+from collections.abc import Sequence
+
+DEFAULT_EMBEDDING_MODEL = "sentence-transformers/all-MiniLM-L6-v2"
+
+
+class HFEmbedder:
+    """Transformers embedder with lazy model loading and mean pooling.
+
+    Args:
+        model_id: Hugging Face model ID.  Defaults to
+            ``sentence-transformers/all-MiniLM-L6-v2``.
+        revision: Git revision to pin.  ``None`` uses the latest
+            available revision.  Production deployments should pin a
+            specific revision fetched from Hugging Face Hub metadata
+            (do not hardcode unverified hashes).
+    """
+
+    def __init__(
+        self,
+        model_id: str = DEFAULT_EMBEDDING_MODEL,
+        revision: str | None = None,
+    ) -> None:
+        self._model_id = model_id
+        self._revision = revision
+        self._model = None
+        self._tokenizer = None
+        self._dim: int | None = None
+
+    def _ensure_loaded(self) -> None:
+        if self._model is not None:
+            return
+        from transformers import AutoModel, AutoTokenizer
+
+        kwargs: dict = {}
+        if self._revision:
+            kwargs["revision"] = self._revision
+
+        self._tokenizer = AutoTokenizer.from_pretrained(self._model_id, **kwargs)
+        self._model = AutoModel.from_pretrained(self._model_id, **kwargs)
+        self._model.eval()
+        self._dim = self._model.config.hidden_size
+
+    @property
+    def dimension(self) -> int:
+        self._ensure_loaded()
+        assert self._dim is not None
+        return self._dim
+
+    def embed(self, texts: Sequence[str]) -> Sequence[Sequence[float]]:
+        import torch
+
+        self._ensure_loaded()
+        assert self._tokenizer is not None
+        assert self._model is not None
+
+        encoded = self._tokenizer(
+            list(texts),
+            padding=True,
+            truncation=True,
+            max_length=512,
+            return_tensors="pt",
+        )
+        with torch.no_grad():
+            outputs = self._model(**encoded)
+            token_embeddings = outputs.last_hidden_state
+            attention_mask = encoded["attention_mask"]
+            mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size())
+            sum_embeddings = torch.sum(token_embeddings * mask_expanded, dim=1)
+            sum_mask = torch.clamp(mask_expanded.sum(dim=1), min=1e-9)
+            sentence_embeddings = sum_embeddings / sum_mask
+            sentence_embeddings = torch.nn.functional.normalize(sentence_embeddings, p=2, dim=1)
+        return [[float(x) for x in row] for row in sentence_embeddings.detach().cpu().tolist()]
+
+
+__all__ = ["DEFAULT_EMBEDDING_MODEL", "HFEmbedder"]