diff --git a/INSTALL.md b/INSTALL.md index 2e041a2..349560e 100644 --- a/INSTALL.md +++ b/INSTALL.md @@ -167,6 +167,25 @@ uv run hf download google/gemma-3-4b-it \ --local-dir /path/to/models/hf_models/google/gemma-3-4b-it ``` +If you enable the finance **GroundingVerifier** stage, pre-stage its two public +models at the pinned revisions used by the workflow: + +```bash +# Claim-to-evidence routing model +uv run hf download sentence-transformers/all-MiniLM-L6-v2 \ + config.json model.safetensors special_tokens_map.json \ + tokenizer.json tokenizer_config.json vocab.txt \ + --revision 1110a243fdf4706b3f48f1d95db1a4f5529b4d41 \ + --local-dir /path/to/models/hf_models/sentence-transformers/all-MiniLM-L6-v2 + +# Natural-language-inference model +uv run hf download MoritzLaurer/DeBERTa-v3-base-mnli-fever-anli \ + added_tokens.json config.json model.safetensors special_tokens_map.json \ + spm.model tokenizer.json tokenizer_config.json \ + --revision 6f5cf0a2b59cabb106aca4c287eed12e357e90eb \ + --local-dir /path/to/models/hf_models/MoritzLaurer/DeBERTa-v3-base-mnli-fever-anli +``` + **Storage location:** Models should go in your mounted HuggingFace models directory (see cluster config `mounts` section). ### Mount Path in Cluster Config @@ -191,13 +210,15 @@ stage_kwargs:
Which models does each workflow need? -| Model | Demo SDG | Demo SFT | Demo GRPO | Demo Eval | Production GRPO | -|-------|:--------:|:--------:|:---------:|:---------:|:---------------:| -| `Qwen/Qwen3-4B` | | ✓ | ✓ | ✓ | | -| `openai/gpt-oss-20b` | ✓ | | | ✓ | | -| `google/gemma-3-4b-it` | | | | ✓ | | -| `openai/gpt-oss-120b` | | | ✓ | | ✓ | -| `Qwen/Qwen3-30B-A3B` | | | | | ✓ | +| Model | Demo SDG | Demo SFT | Demo GRPO | Demo Eval | Production GRPO | GroundingVerifier | +|-------|:--------:|:--------:|:---------:|:---------:|:---------------:|:-----------------:| +| `Qwen/Qwen3-4B` | | ✓ | ✓ | ✓ | | | +| `openai/gpt-oss-20b` | ✓ | | | ✓ | | | +| `google/gemma-3-4b-it` | | | | ✓ | | | +| `openai/gpt-oss-120b` | | | ✓ | | ✓ | | +| `Qwen/Qwen3-30B-A3B` | | | | | ✓ | | +| `sentence-transformers/all-MiniLM-L6-v2` | | | | | | ✓ | +| `MoritzLaurer/DeBERTa-v3-base-mnli-fever-anli` | | | | | | ✓ | **Tip:** Download commonly used models once and reuse across all workflows.
diff --git a/docs/recipes/finance/README.md b/docs/recipes/finance/README.md index 2350a4a..c509611 100644 --- a/docs/recipes/finance/README.md +++ b/docs/recipes/finance/README.md @@ -164,6 +164,8 @@ Detailed technical specifications for each stage: - **[SFT Stages](stages/sft.md)** - 6 stages - **[Eval Stages](stages/eval.md)** - 7 stages - **[GRPO Stages](stages/grpo.md)** - 10 stages +- **[GroundingVerifier](grounding-verifier/README.md)** - Finance integration, + evaluation protocol, and held-out results ## Quick Command Reference diff --git a/docs/recipes/finance/grounding-verifier/README.md b/docs/recipes/finance/grounding-verifier/README.md new file mode 100644 index 0000000..494d1f3 --- /dev/null +++ b/docs/recipes/finance/grounding-verifier/README.md @@ -0,0 +1,122 @@ +# GroundingVerifier for finance + +GroundingVerifier evaluates completed `finance_agent` rollouts against the SEC +evidence preserved in their native tool-call history. It writes a separate +sidecar and does not modify the original rollout, reward, or training data. + +The evaluator: + +1. deterministically splits the submitted answer into claims; +2. routes each claim to retrieved evidence with + `sentence-transformers/all-MiniLM-L6-v2`; +3. checks entailment with + `MoritzLaurer/DeBERTa-v3-base-mnli-fever-anli`; and +4. verifies SEC attribution, entities, metrics, dates, protected values, + calculations, comparisons, and evidence-scoped refusals. + +Both public model revisions can be pinned. Incomplete attribution or model +failure produces `unavailable`; unsupported or conflicting claims are blocked. +Evidence is limited to `retrieve_information` excerpts retained by the rollout. + +## Air-gapped setup + +NVFlow workers run with Hugging Face and Transformers offline flags enabled, so +the two public models must be staged before submitting `evaluate_grounding`. +From a connected host, download the pinned revisions into the cluster directory +mounted as `/hf_models`: + +```bash +uv run hf download sentence-transformers/all-MiniLM-L6-v2 \ + config.json model.safetensors special_tokens_map.json \ + tokenizer.json tokenizer_config.json vocab.txt \ + --revision 1110a243fdf4706b3f48f1d95db1a4f5529b4d41 \ + --local-dir /path/to/models/hf_models/sentence-transformers/all-MiniLM-L6-v2 + +uv run hf download MoritzLaurer/DeBERTa-v3-base-mnli-fever-anli \ + added_tokens.json config.json model.safetensors special_tokens_map.json \ + spm.model tokenizer.json tokenizer_config.json \ + --revision 6f5cf0a2b59cabb106aca4c287eed12e357e90eb \ + --local-dir /path/to/models/hf_models/MoritzLaurer/DeBERTa-v3-base-mnli-fever-anli +``` + +The workflow loads `/hf_models/sentence-transformers/all-MiniLM-L6-v2` and +`/hf_models/MoritzLaurer/DeBERTa-v3-base-mnli-fever-anli`. See [INSTALL.md → +Download Models](../../../../INSTALL.md#download-models) for the mount +configuration and general model-staging procedure. + +## NVFlow integration + +Add `evaluate_grounding` after `collect_rollouts` in a finance GRPO workflow. +The stage reads completed `output-rs*.jsonl` files and writes matching +`grounding-verifier-rs*.jsonl` sidecars. + +```yaml +pipeline_stages: + - collect_rollouts + - evaluate_grounding +``` + +## Repeated feature acceptance + +Every GroundingVerifier feature can define a native acceptance profile with a +fixed number of finance-agent seeds. The first profile covers the air-gapped +model setup and runs one prepared `finance_sec_search` task three times: + +```bash +uv run nflow run-all \ + --config nvflow/recipes/finance/workflows/grpo/feature_gates/grounding_airgap.yaml \ + -e finance_sec_search +``` + +Change `feature_runs` in the profile to increase the repetition count. The +profile always regenerates its dedicated rollouts, evaluates every row, and +then fails unless all expected files and completion markers exist, every +sidecar matches its rollout fingerprint, the offline model contract holds, and +the unavailable rate stays within the configured limit. Its compact summary is +written beside the sidecars as `feature-gate-grounding_airgap.json`. + +This native profile requires the demo `prepare_data` output and SEC cache from +the quick-start workflow. Ordinary CI tests validate the gate logic without a +GPU; executing the finance agent itself requires the configured NVFlow Slurm +environment. `allow` and `block` are both valid execution outcomes because an +unsupported agent answer should be blocked. Detection efficacy remains covered +by the pinned public-model benchmark below. + +## Public-model benchmark + +The controlled benchmark uses eight FY2024 facts from Amazon, Alphabet, Meta, +and Tesla, which were held out from the earlier AAPL/MSFT/NVDA development set. +Each fixed evidence trace is evaluated once with its grounded answer and against +six mutations: numeric fabrication, entity conflation, metric fabrication, +temporal conflation, accession conflation, and an unsupported claim. This gives +eight grounded and 48 adversarial cases. + +```bash +uv sync +uv run python -m nvflow.recipes.finance.utils.rl.grounding_benchmark \ + --output-dir artifacts/grounding-verifier-benchmark \ + --routing-model-revision 1110a243fdf4706b3f48f1d95db1a4f5529b4d41 \ + --nli-model-revision 6f5cf0a2b59cabb106aca4c287eed12e357e90eb +``` + +The command writes `cases.jsonl`, `results.jsonl`, and `summary.json`. It also +records latency, throughput, memory use, model revisions, and host details. +It succeeds only when grounded acceptance and attack rejection are each at +least 90% and the unavailable rate is at most 5%. + +The same public-model gate and native rollout-to-sidecar check can be run with: + +```bash +GROUNDING_VERIFIER_RUN_MODELS=1 uv run pytest -q \ + tests/test_grounding_benchmark.py -k pinned_public_models --no-cov +``` + +For the native held-out study, see the [frozen evaluation +protocol](evaluation-protocol.md) and [results](evaluation-results.md). + +## Scope + +This integration verifies whether a submitted answer is supported by the +retrieval excerpts and SEC metadata preserved in an NVFlow rollout. It does not +independently re-verify the complete SEC filing or establish unrestricted +production hallucination detection. diff --git a/docs/recipes/finance/grounding-verifier/evaluation-protocol.md b/docs/recipes/finance/grounding-verifier/evaluation-protocol.md new file mode 100644 index 0000000..03a2d3e --- /dev/null +++ b/docs/recipes/finance/grounding-verifier/evaluation-protocol.md @@ -0,0 +1,57 @@ +# Native finance evaluation protocol + +The confirmation protocol was frozen before task generation, rollout +collection, or GroundingVerifier evaluation. + +## Separation + +- All 60 issuers were excluded from development and three earlier studies. +- GroundingVerifier was not used for task generation, rollout collection, + eligibility, labeling, or adversarial-answer creation. +- Evaluation inputs and labels were frozen before the one-shot guard run. + +## Trace generation + +An independent generator used official SEC ticker, submissions, filing, and +XBRL Company Facts data to select a 10-K and establish its company, metric, +year, value, and unit. Each task supplied the target filing date, canonical SEC +URL, and bounded retrieval range. + +A pinned, locally hosted `Qwen3-4B-Instruct-2507` policy then executed the native +NVFlow `finance_agent` path: + +```text +sec_filing_search -> parse_html_page -> retrieve_information -> submit_final_result +``` + +The tasks covered year-over-year calculations, cross-issuer comparisons, and +closed-world evidence-based refusals. This design tests evidence use and answer +verification, not autonomous discovery of an unknown filing. + +## Eligibility and freezing + +A calculation or comparison was eligible only when every retrieved +company/metric/year/value/unit tuple agreed with the independent SEC/XBRL +reference. A refusal was eligible only when its bounded evidence contained a +valid coverage fact but omitted the requested metric. + +We collected 180 native rollouts and selected 120 eligible traces by task-ID +hash, balanced as 40 calculations, 40 comparisons, and 40 refusals. Selection +did not use GroundingVerifier outcomes. + +Each trace contributed one grounded answer and one independently assigned +adversarial answer. Attacks covered numeric fabrication; entity, metric, +temporal, and source conflation; wrong or contradictory comparison winners; +and unsupported refusal claims. + +## Success criteria + +The preregistered GO gate required: + +- at least 95% grounded acceptance and attack rejection; +- Wilson 95% lower bounds of at least 90%; +- at least 90% accuracy in every task category; and +- zero adversarial `allow` decisions. + +Intervals used two-sided Wilson estimates and 20,000 issuer-cluster bootstrap +resamples with seed `26081432`. diff --git a/docs/recipes/finance/grounding-verifier/evaluation-results.md b/docs/recipes/finance/grounding-verifier/evaluation-results.md new file mode 100644 index 0000000..6e12e7e --- /dev/null +++ b/docs/recipes/finance/grounding-verifier/evaluation-results.md @@ -0,0 +1,58 @@ +# Native finance evaluation results + +## Verdict + +The frozen confirmation passed its preregistered GO criteria. + +| Endpoint | Correct | Rate | Wilson 95% interval | +|---|---:|---:|---:| +| Grounded acceptance | 119/120 | 99.17% | 95.43-99.85% | +| Adversarial rejection | 120/120 | 100% | 96.90-100% | + +- Calculations: 80/80 paired decisions correct. +- Comparisons: 80/80 paired decisions correct. +- Refusals: 79/80 paired decisions correct. +- False allows: 0. +- Exact two-sided McNemar comparison with an accept-all baseline: + `p = 9.18e-35`. + +All attacks were blocked. They included numeric fabrication; entity, metric, +temporal, and source conflation; wrong or contradictory comparison winners; +and unsupported refusal detail or assertion. + +## Collection and validation + +- Native collection: 180 rollouts in 1:46:40. +- Frozen evaluation: 120 independent traces from 60 unseen issuers. +- Guard evaluation: 240 paired rows in 33.60 seconds. +- Native exclusions: 40 missing or incorrect source-bound tuples, nine + unexpected retrieval counts, one unpaired submission, and ten eligible + reserve traces. Exclusions did not use guard outcomes. +- Repository tests: 385 passed and one skipped. +- Pinned-public-model integration test, Ruff, `uv lock --check`, and + `git diff --check`: passed. + +## Residual error + +The only false block was a grounded refusal for a company ending in `N.V.`. +The deterministic decomposer split the legal suffix into an incomplete clause, +which the NLI model classified as contradiction. The candidate was not changed +after observing the confirmation result. + +## Public models + +```text +sentence-transformers/all-MiniLM-L6-v2 +revision 1110a243fdf4706b3f48f1d95db1a4f5529b4d41 + +MoritzLaurer/DeBERTa-v3-base-mnli-fever-anli +revision 6f5cf0a2b59cabb106aca4c287eed12e357e90eb +``` + +## Claim boundary + +These results demonstrate source-bound detection on controlled, issuer-held-out +retrieval excerpts preserved in native NVFlow finance traces. They do not +independently validate complete SEC documents, estimate production failure +prevalence, prove unrestricted hallucination detection, or demonstrate an +NVIDIA Slurm deployment. diff --git a/nvflow/grounding_verifier/__init__.py b/nvflow/grounding_verifier/__init__.py new file mode 100644 index 0000000..6c5a95b --- /dev/null +++ b/nvflow/grounding_verifier/__init__.py @@ -0,0 +1,15 @@ +# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +"""Evidence-based provenance decisions for NVFlow agent outputs.""" diff --git a/nvflow/grounding_verifier/conflation.py b/nvflow/grounding_verifier/conflation.py new file mode 100644 index 0000000..49ea3db --- /dev/null +++ b/nvflow/grounding_verifier/conflation.py @@ -0,0 +1,117 @@ +# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +"""Deterministic explicit SEC source-attribution checks.""" + +from __future__ import annotations + +import re +from urllib.parse import urlsplit, urlunsplit + +from nvflow.grounding_verifier.types import AtomicClaim, EvidenceChunk + +_SEC_URL_RE = re.compile(r"https?://(?:www\.)?sec\.gov/\S+", re.IGNORECASE) +_ATTRIBUTION_RE = re.compile( + r"(?:according to\s+)?(?:SEC\s+)?(?:accession|CIK)(?:\s+number)?\s+" + r"[\d-]+(?:\s+(?:reports?|reported|states?|stated)\s+that)?", + re.IGNORECASE, +) +_GENERIC_SEC_ATTRIBUTION_RE = re.compile( + r"\s+as\s+reported\s+in\s+(?:the\s+)?SEC\s+filings?", + re.IGNORECASE, +) +_TICKER_RE = re.compile(r"\(([A-Z][A-Z0-9.]{0,7})\)") +_CORPORATE_SUFFIX_RE = re.compile(r"\b(?:inc|corp|corporation|ltd|llc)\b", re.IGNORECASE) +_COMPANY_STOPWORDS = {"co", "company", "corp", "corporation", "inc", "incorporated", "ltd", "llc"} + + +def _digits(value: str) -> str: + return re.sub(r"\D", "", value) + + +def _source_fields(source_id: str | None) -> dict[str, str]: + if not source_id: + return {} + return { + key: value + for part in source_id.split(":") + if "=" in part + for key, value in [part.split("=", 1)] + } + + +def _canonical_url(url: str) -> str: + parts = urlsplit(url.rstrip(".,;")) + return urlunsplit((parts.scheme.lower(), parts.netloc.lower(), parts.path, "", "")) + + +def content_for_routing(text: str) -> str: + """Remove explicit SEC attribution tokens before semantic routing.""" + content = _SEC_URL_RE.sub(" ", text) + content = _ATTRIBUTION_RE.sub(" ", content) + content = _GENERIC_SEC_ATTRIBUTION_RE.sub(" ", content) + return " ".join(content.split()) or text + + +def explicit_source_match(claim: AtomicClaim, chunk: EvidenceChunk) -> bool | None: + """Compare an explicit SEC citation with a source, or return ``None``. + + This intentionally covers only identifiers stated in the claim. It does not + infer attribution from company names or replace token-level alignment. + """ + fields = _source_fields(chunk.source_id) + compared = False + stated: dict[str, set[str]] = {"cik": set(), "accession": set()} + for identifier in claim.stated_sec_ids: + kind, _, value = identifier.partition(":") + if kind in stated: + stated[kind].add(_digits(value)) + + for kind, expected in stated.items(): + actual = _digits(fields.get(kind, "")) + if expected and actual: + compared = True + if actual not in expected: + return False + + if claim.stated_sec_urls and chunk.sec_url: + compared = True + routed_url = _canonical_url(chunk.sec_url) + if routed_url not in {_canonical_url(url) for url in claim.stated_sec_urls}: + return False + return True if compared else None + + +def has_explicit_source_conflation(claim: AtomicClaim, chunk: EvidenceChunk) -> bool: + """Return true when an explicit SEC citation disagrees with routed evidence.""" + return explicit_source_match(claim, chunk) is False + + +def has_explicit_entity_conflation(claim: AtomicClaim, chunk: EvidenceChunk) -> bool: + """Compare explicit company/ticker mentions with SEC source metadata.""" + claim_tickers = set(_TICKER_RE.findall(claim.text)) + if claim_tickers and chunk.sec_ticker and chunk.sec_ticker.upper() not in claim_tickers: + return True + + if not chunk.sec_company_name or not _CORPORATE_SUFFIX_RE.search(claim.text): + return False + company_tokens = { + token + for token in re.findall(r"[a-z0-9]+", chunk.sec_company_name.lower()) + if token not in _COMPANY_STOPWORDS + } + claim_lower = claim.text.lower() + return bool(company_tokens) and not any( + re.search(rf"\b{re.escape(token)}\b", claim_lower) for token in company_tokens + ) diff --git a/nvflow/grounding_verifier/decomposer.py b/nvflow/grounding_verifier/decomposer.py new file mode 100644 index 0000000..808746f --- /dev/null +++ b/nvflow/grounding_verifier/decomposer.py @@ -0,0 +1,136 @@ +# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +"""Deterministic rule-based claim decomposer (v1 default). + +This conservative splitter uses sentence boundaries and conjunctions. It may +over-merge compound claims or split mid-clause. + +Claim IDs are deterministic SHA-256 hashes of the claim text and the +source sentence, so re-evaluation of the same answer yields stable IDs. +""" + +from __future__ import annotations + +import hashlib +import re +from collections.abc import Sequence + +from nvflow.grounding_verifier.types import AtomicClaim + +# SEC filing URL pattern — used to extract stated SEC IDs/URLs from claim text. +_SEC_URL_RE = re.compile( + r"https?://(?:www\.)?sec\.gov/[^\s\"'<>]+", + re.IGNORECASE, +) + +# CIK pattern — 10-digit zero-padded numbers. +_CIK_RE = re.compile(r"\b(\d{10})\b") + +# Accession number pattern — e.g. 0001811414-25-000010 +_ACCESSION_RE = re.compile(r"\b(\d{10}-\d{2}-\d{6})\b") + +# Keep common corporate suffixes intact ("Apple Inc. reported ..."). +_SENTENCE_END_RE = re.compile( + r"(? bool: + """Ignore provenance narration that contains no submitted factual value.""" + has_factual_value = bool(re.search(r"[$€£¥%]|\b\d{1,3}(?:,\d{3})+\b", text)) + return bool(_CITED_EVIDENCE_RE.search(text)) or ( + bool(_META_EVIDENCE_RE.search(text)) and not has_factual_value + ) + + +def _stable_claim_id(text: str, source_sentence: str) -> str: + """Deterministic 16-char hex hash for a claim.""" + raw = f"{text}|{source_sentence}" + return hashlib.sha256(raw.encode("utf-8")).hexdigest()[:16] + + +def _extract_stated_sec_ids(text: str) -> tuple[tuple[str, ...], tuple[str, ...]]: + """Extract SEC CIKs and accession numbers mentioned in claim text.""" + accessions = tuple(m.group(1) for m in _ACCESSION_RE.finditer(text)) + # An accession begins with ten digits, but that prefix is not necessarily + # the issuer CIK. Remove accession spans before looking for standalone CIKs. + without_accessions = _ACCESSION_RE.sub(" ", text) + ids = [f"cik:{m.group(1)}" for m in _CIK_RE.finditer(without_accessions)] + ids.extend(f"accession:{accession}" for accession in accessions) + urls = tuple(m.group(0) for m in _SEC_URL_RE.finditer(text)) + return tuple(ids), urls + + +class RuleBasedDecomposer: + """Deterministic sentence/claim splitter. + + Splits the answer into sentences, then further splits compound + claims on semicolons, em-dashes, and ``"but also"`` connectors. + Filters out empty fragments and very short noise (< 3 words). + """ + + def decompose(self, answer: str) -> Sequence[AtomicClaim]: + if not answer or not answer.strip(): + return [] + + # Normalize whitespace but preserve sentence structure. + text = answer.strip() + + # Split into sentences. + sentences = [s.strip() for s in _SENTENCE_END_RE.split(text) if s.strip()] + if not sentences: + sentences = [text] + + claims: list[AtomicClaim] = [] + for sentence in sentences: + # Further split compound claims. + fragments = _COMPOUND_SPLIT_RE.split(sentence) + for frag in fragments: + frag = _REFUTED_SUFFIX_RE.sub("", frag).strip() + if not frag: + continue + if _is_meta_evidence_fragment(frag): + continue + # Keep short clauses created by an explicit compound split + # (for example, "Acme won"); otherwise skip short noise. + if len(frag.split()) < 3 and len(fragments) == 1: + continue + sec_ids, sec_urls = _extract_stated_sec_ids(frag) + claim_id = _stable_claim_id(frag, sentence) + claims.append( + AtomicClaim( + claim_id=claim_id, + text=frag, + source_sentence=sentence, + stated_sec_ids=sec_ids, + stated_sec_urls=sec_urls, + ) + ) + + return claims diff --git a/nvflow/grounding_verifier/embedder.py b/nvflow/grounding_verifier/embedder.py new file mode 100644 index 0000000..5082917 --- /dev/null +++ b/nvflow/grounding_verifier/embedder.py @@ -0,0 +1,101 @@ +# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +"""Lazy Hugging Face embedder for claim/evidence routing. + +Default public model: ``sentence-transformers/all-MiniLM-L6-v2`` +(Apache 2.0). Model construction is strictly lazy — ``torch`` and +``transformers`` imports happen inside ``_ensure_loaded``, not at +module import time. + +Uses the raw Transformers ``AutoTokenizer`` / ``AutoModel`` recipe with +mean pooling over ``last_hidden_state`` with attention mask, L2-normalized, +returning plain float lists — so no ``sentence-transformers`` runtime +dependency is needed. +""" + +from __future__ import annotations + +from collections.abc import Sequence + +DEFAULT_EMBEDDING_MODEL = "sentence-transformers/all-MiniLM-L6-v2" + + +class HFEmbedder: + """Transformers embedder with lazy model loading and mean pooling. + + Args: + model_id: Hugging Face model ID. Defaults to + ``sentence-transformers/all-MiniLM-L6-v2``. + revision: Git revision to pin. ``None`` uses the latest + available revision. Production deployments should pin a + specific revision fetched from Hugging Face Hub metadata + (do not hardcode unverified hashes). + """ + + def __init__( + self, + model_id: str = DEFAULT_EMBEDDING_MODEL, + revision: str | None = None, + ) -> None: + self._model_id = model_id + self._revision = revision + self._model = None + self._tokenizer = None + self._dim: int | None = None + + def _ensure_loaded(self) -> None: + if self._model is not None: + return + from transformers import AutoModel, AutoTokenizer + + kwargs: dict = {} + if self._revision: + kwargs["revision"] = self._revision + + self._tokenizer = AutoTokenizer.from_pretrained(self._model_id, **kwargs) + self._model = AutoModel.from_pretrained(self._model_id, **kwargs) + self._model.eval() + self._dim = self._model.config.hidden_size + + @property + def dimension(self) -> int: + self._ensure_loaded() + assert self._dim is not None + return self._dim + + def embed(self, texts: Sequence[str]) -> Sequence[Sequence[float]]: + import torch + + self._ensure_loaded() + assert self._tokenizer is not None + assert self._model is not None + + encoded = self._tokenizer( + list(texts), + padding=True, + truncation=True, + max_length=512, + return_tensors="pt", + ) + with torch.no_grad(): + outputs = self._model(**encoded) + token_embeddings = outputs.last_hidden_state + attention_mask = encoded["attention_mask"] + mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()) + sum_embeddings = torch.sum(token_embeddings * mask_expanded, dim=1) + sum_mask = torch.clamp(mask_expanded.sum(dim=1), min=1e-9) + sentence_embeddings = sum_embeddings / sum_mask + sentence_embeddings = torch.nn.functional.normalize(sentence_embeddings, p=2, dim=1) + return [[float(x) for x in row] for row in sentence_embeddings.detach().cpu().tolist()] diff --git a/nvflow/grounding_verifier/evaluator.py b/nvflow/grounding_verifier/evaluator.py new file mode 100644 index 0000000..d3b988a --- /dev/null +++ b/nvflow/grounding_verifier/evaluator.py @@ -0,0 +1,462 @@ +# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +"""Core evaluator for GroundingVerifier. + +Pipeline: decompose → route → NLI → protected-value check → decision. + +Decision strictness (conservative): + +- **allow**: every claim is entailed AND every protected value is + found in routed evidence. +- **block**: any claim is contradiction, neutral, no_source, or + protected_value_mismatch. +- **unavailable**: no evidence chunks were provided, no claims were + extracted from the answer, or any NLI/model error prevented at + least one claim from being fully evaluated. If any claim has an + error, the row is unavailable even if another claim is + contradiction or neutral. + +Evidence is retrieval_model_excerpt only, never primary SEC verification. +""" + +from __future__ import annotations + +import re +from collections.abc import Sequence +from dataclasses import dataclass +from typing import Any + +from nvflow.grounding_verifier.conflation import ( + content_for_routing, + has_explicit_entity_conflation, + has_explicit_source_conflation, +) +from nvflow.grounding_verifier.embedder import DEFAULT_EMBEDDING_MODEL +from nvflow.grounding_verifier.nli import DEFAULT_NLI_MODEL +from nvflow.grounding_verifier.protected_values import ( + check_protected_values, + extract_protected_values, + has_financial_metric_mismatch, +) +from nvflow.grounding_verifier.protocols import ClaimDecomposer, NLIScorer, SourceRouter +from nvflow.grounding_verifier.types import ( + AtomicClaim, + ClaimVerdict, + Decision, + EvidenceChunk, +) + +ALGORITHM_VERSION = "routing-nli-sec-attribution-v5" + +_TOKEN_RE = re.compile(r"[A-Za-z0-9]+") +_DIRECT_SUPPORT_STOPWORDS = { + "a", + "about", + "an", + "and", + "approximately", + "are", + "as", + "at", + "by", + "for", + "from", + "had", + "has", + "in", + "is", + "of", + "on", + "or", + "the", + "to", + "was", + "were", +} + + +def _fact_tokens(text: str) -> set[str]: + """Return conservative lexical tokens for direct fact alignment.""" + return { + token.lower() + for token in _TOKEN_RE.findall(text) + if token.lower() not in _DIRECT_SUPPORT_STOPWORDS + } + + +def _has_direct_value_support( + claim_text: str, + evidence: EvidenceChunk, + pv_outcome: str, +) -> bool: + """Recognize near-verbatim numeric facts when an NLI model is under-confident. + + This is deliberately narrower than a generic lexical fallback: every protected + value must already match, at least one non-date numeric value must be present, + and the evidence must cover both the claim vocabulary and two descriptive + (non-numeric) terms. A changed number therefore remains fail-closed. + """ + if pv_outcome != "pass": + return False + protected = extract_protected_values(claim_text) + if not any(value.kind in {"currency", "number", "percentage"} for value in protected): + return False + + if not evidence.sec_company_name and not evidence.sec_ticker: + return False + claim_lower = claim_text.lower() + company_tokens = { + token + for token in _fact_tokens(evidence.sec_company_name or "") + if token not in {"co", "company", "corp", "corporation", "inc", "incorporated", "ltd"} + } + ticker = (evidence.sec_ticker or "").lower() + has_entity_match = any(token in claim_lower for token in company_tokens) or ( + bool(ticker) and re.search(rf"\b{re.escape(ticker)}\b", claim_lower) is not None + ) + if not has_entity_match and not re.search(r"\b(?:the company|registrant)\b", claim_lower): + return False + + claim_tokens = _fact_tokens(claim_text) + evidence_tokens = _fact_tokens(evidence.text) + if not claim_tokens: + return False + + overlap = claim_tokens & evidence_tokens + descriptive_overlap = {token for token in overlap if not token.isdigit()} + return len(descriptive_overlap) >= 2 and len(overlap) / len(claim_tokens) >= 0.8 + + +@dataclass(frozen=True) +class GroundingVerifierConfig: + """Configuration for the GroundingVerifier evaluator. + + The fail-closed policy is **fixed and non-configurable**: + contradiction, no_source, and protected_value_mismatch always block; + neutral blocks unless deterministic entity/metric/value checks establish + direct support. Any model/trace error yields unavailable. + There are no ``block_on_*`` toggles — the policy cannot be + disabled. + + Attributes: + evidence_excerpt_length: Maximum characters of evidence text + to include in each claim verdict for auditability. + routing_model: Model ID used for embedding/routing (metadata). + nli_model: Model ID used for NLI scoring (metadata). + """ + + evidence_excerpt_length: int = 500 + routing_model: str = DEFAULT_EMBEDDING_MODEL + nli_model: str = DEFAULT_NLI_MODEL + + def to_dict(self) -> dict[str, Any]: + return { + "algorithm": ALGORITHM_VERSION, + "policy": "fixed_fail_closed", + "evidence_excerpt_length": self.evidence_excerpt_length, + "routing_model": self.routing_model, + "nli_model": self.nli_model, + } + + +class GroundingVerifierEvaluator: + """Orchestrates claim decomposition, routing, NLI, and protected-value checking. + + All model-bearing collaborators (``Embedder``, ``NLIScorer``) are + injected so that tests can use deterministic fakes. No Hugging Face + imports happen during construction or evaluation of this class — + that is the responsibility of ``HFEmbedder`` / ``HFNLI``, which load + lazily. + """ + + def __init__( + self, + *, + decomposer: ClaimDecomposer, + router: SourceRouter, + nli_scorer: NLIScorer, + config: GroundingVerifierConfig | None = None, + ) -> None: + self._decomposer = decomposer + self._router = router + self._nli = nli_scorer + self._config = config or GroundingVerifierConfig() + + @property + def config(self) -> GroundingVerifierConfig: + return self._config + + def evaluate( + self, + answer: str, + evidence: Sequence[EvidenceChunk], + ) -> Decision: + """Evaluate an answer against evidence and return a Decision. + + Args: + answer: The final answer text submitted by the agent. + evidence: Evidence chunks extracted from the tool-call trace. + + Returns: + A :class:`~nvflow.grounding_verifier.types.Decision` with status + ``allow``, ``block``, or ``unavailable``. + """ + errors: list[str] = [] + + if not evidence: + return Decision( + status="unavailable", + reason="no_evidence", + errors=("No evidence chunks were provided.",), + ) + + claims = self._decomposer.decompose(answer) + if not claims: + return Decision( + status="unavailable", + reason="no_claims_extracted", + errors=(), + ) + + try: + routed = self._router.route(claims, evidence) + except Exception as exc: + return Decision( + status="unavailable", + reason="routing_error", + errors=(f"Routing failed: {exc!s}",), + ) + + routed_by_claim_id = {item.claim.claim_id: item for item in routed} + + verdicts: list[ClaimVerdict] = [] + for claim in claims: + verdicts.append(self._evaluate_claim(claim, routed_by_claim_id, errors)) + + nli_errors = sum(bool(verdict.errors) for verdict in verdicts) + if nli_errors > 0: + reason = "all_claims_errored" if nli_errors == len(claims) else "partial_nli_errors" + return Decision( + status="unavailable", + reason=reason, + verdicts=tuple(verdicts), + errors=tuple(errors), + ) + + status, reason = self._aggregate(verdicts) + return Decision( + status=status, + reason=reason, + verdicts=tuple(verdicts), + errors=tuple(errors), + ) + + def verify_against_premise(self, answer: str, premise: str) -> Decision: + """Require every submitted claim to be entailed by a canonical premise. + + This is used for derived facts, such as arithmetic or comparisons, for + which callers can construct a deterministic premise from attributed + source facts. No source routing or lexical entity vocabulary is used. + """ + claims = self._decomposer.decompose(answer) + if not claims: + return Decision(status="unavailable", reason="no_claims_extracted") + + verdicts = [] + errors = [] + for claim in claims: + claim_text = " ".join(claim.text.split()) + premise_text = " ".join(premise.split()) + if claim_text in premise_text: + verdicts.append( + ClaimVerdict( + claim_id=claim.claim_id, + claim_text=claim.text, + raw_nli_label="entailment", + raw_nli_probabilities=(("entailment", 1.0),), + final_label="entailment", + evidence_excerpt=premise[: self._config.evidence_excerpt_length], + ) + ) + continue + try: + result = self._nli.score(premise=premise, hypothesis=claim.text) + except Exception as exc: + error = f"NLI error for claim {claim.claim_id}: {exc!s}" + errors.append(error) + verdicts.append( + ClaimVerdict( + claim_id=claim.claim_id, + claim_text=claim.text, + final_label="neutral", + evidence_excerpt=premise[: self._config.evidence_excerpt_length], + errors=(error,), + ) + ) + continue + verdicts.append( + ClaimVerdict( + claim_id=claim.claim_id, + claim_text=claim.text, + raw_nli_label=result.label, + raw_nli_probabilities=result.probabilities, + final_label=result.label, + evidence_excerpt=premise[: self._config.evidence_excerpt_length], + ) + ) + + if errors: + return Decision( + status="unavailable", + reason="semantic_verification_error", + verdicts=tuple(verdicts), + errors=tuple(errors), + ) + for label in ("contradiction", "neutral"): + if any(verdict.final_label == label for verdict in verdicts): + return Decision( + status="block", + reason=f"semantic_{label}", + verdicts=tuple(verdicts), + ) + return Decision(status="allow", reason="semantic_entailment", verdicts=tuple(verdicts)) + + def _evaluate_claim( + self, + claim: AtomicClaim, + routed_by_claim_id: dict[str, Any], + errors: list[str], + ) -> ClaimVerdict: + """Evaluate a single claim and return its verdict.""" + routed = routed_by_claim_id.get(claim.claim_id) + + if routed is None: + return ClaimVerdict( + claim_id=claim.claim_id, + claim_text=claim.text, + final_label="no_source", + protected_value_outcome="not_applicable", + errors=(), + ) + + chunk = routed.chunk + excerpt = chunk.text[: self._config.evidence_excerpt_length] + routed_fields = { + "claim_id": claim.claim_id, + "claim_text": claim.text, + "routed_source_id": chunk.source_id, + "routed_source_ids": chunk.source_ids, + "routed_attribution_state": chunk.attribution_state, + "routed_chunk_id": chunk.chunk_id, + "routing_score": routed.score, + "routing_margin": routed.margin, + "evidence_excerpt": excerpt, + } + + if chunk.attribution_state in ("unavailable", "composite", "unknown"): + return ClaimVerdict( + final_label="no_source", + protected_value_outcome="not_applicable", + errors=(), + **routed_fields, + ) + + if has_explicit_source_conflation(claim, chunk): + return ClaimVerdict( + final_label="conflation", + protected_value_outcome="not_applicable", + errors=(), + **routed_fields, + ) + + if has_explicit_entity_conflation(claim, chunk): + return ClaimVerdict( + final_label="entity_conflation", + protected_value_outcome="not_applicable", + errors=(), + **routed_fields, + ) + + factual_text = content_for_routing(claim.text) + try: + nli_result = self._nli.score(premise=chunk.text, hypothesis=factual_text) + except Exception as exc: + err = f"NLI error for claim {claim.claim_id}: {exc!s}" + errors.append(err) + return ClaimVerdict( + raw_nli_label="neutral", + raw_nli_probabilities=(), + final_label="neutral", + protected_value_outcome="not_applicable", + errors=(err,), + **routed_fields, + ) + + raw_label = nli_result.label + pv_outcome, _missing = check_protected_values(factual_text, chunk.text) + + if has_financial_metric_mismatch(factual_text, chunk.text): + return ClaimVerdict( + raw_nli_label=raw_label, + raw_nli_probabilities=nli_result.probabilities, + final_label="financial_metric_mismatch", + protected_value_outcome=pv_outcome, + errors=(), + **routed_fields, + ) + + final_label = self._apply_policy(raw_label, pv_outcome) + if raw_label == "neutral" and _has_direct_value_support(factual_text, chunk, pv_outcome): + final_label = "entailment" + + return ClaimVerdict( + raw_nli_label=raw_label, + raw_nli_probabilities=nli_result.probabilities, + final_label=final_label, + protected_value_outcome=pv_outcome, + errors=(), + **routed_fields, + ) + + def _apply_policy(self, raw_nli_label: str, pv_outcome: str) -> str: + """Apply fixed fail-closed policy to the raw NLI label.""" + if raw_nli_label == "entailment": + if pv_outcome == "fail": + return "protected_value_mismatch" + return "entailment" + if raw_nli_label == "contradiction": + return "contradiction" + return "neutral" + + def _aggregate(self, verdicts: list[ClaimVerdict]) -> tuple[str, str]: + """Aggregate per-claim verdicts into a top-level decision (fixed fail-closed).""" + labels = [v.final_label for v in verdicts if not v.errors] + if not labels: + return "unavailable", "all_claims_errored" + + for blocking_label in ( + "conflation", + "entity_conflation", + "financial_metric_mismatch", + "contradiction", + "protected_value_mismatch", + "neutral", + "no_source", + ): + if blocking_label in labels: + return "block", blocking_label + if all(lbl == "entailment" for lbl in labels): + return "allow", "all_entailed" + return "block", "unverifiable" diff --git a/nvflow/grounding_verifier/nli.py b/nvflow/grounding_verifier/nli.py new file mode 100644 index 0000000..a0c0ba1 --- /dev/null +++ b/nvflow/grounding_verifier/nli.py @@ -0,0 +1,167 @@ +# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +"""Lazy Hugging Face NLI scorer for claim-evidence entailment checking. + +Default public model: ``MoritzLaurer/DeBERTa-v3-base-mnli-fever-anli`` (MIT). + +Validates the exact unique label set (entailment, neutral, contradiction) +from the model config — fails on duplicates, extras, or missing labels. +Model construction is strictly lazy — the ``transformers`` import happens +inside ``__init__``, not at module import time. + +No lexical/hash runtime fallback. If model loading fails, the caller +must catch the exception and produce an ``unavailable`` decision. +""" + +from __future__ import annotations + +from nvflow.grounding_verifier.types import NLIResult + +DEFAULT_NLI_MODEL = "MoritzLaurer/DeBERTa-v3-base-mnli-fever-anli" + +_VALID_NLI_LABELS = frozenset({"entailment", "neutral", "contradiction"}) + + +def _normalize_nli_label(raw_label: str) -> str: + """Normalize a model config label to one of the three canonical NLI classes. + + Validates the *actual* label name from the model config + (``config.id2label``). Never assumes arbitrary ``LABEL_0`` ordering. + + Raises ``ValueError`` if the label does not match entailment, + neutral, or contradiction (case-insensitive). + """ + normalized = raw_label.strip().lower() + if normalized in _VALID_NLI_LABELS: + return normalized + raise ValueError( + f"Unknown NLI label '{raw_label}': expected one of " + f"entailment, neutral, contradiction (case-insensitive). " + "Refusing to assume arbitrary label ordering." + ) + + +class HFNLI: + """DeBERTa sequence-classification NLI scorer with lazy model loading. + + Args: + model_id: Hugging Face model ID. + revision: Git revision to pin. ``None`` uses the latest + available revision. Production deployments should pin a + specific revision fetched from Hugging Face Hub metadata + (do not hardcode unverified hashes). + """ + + def __init__( + self, + model_id: str = DEFAULT_NLI_MODEL, + revision: str | None = None, + ) -> None: + self._model_id = model_id + self._revision = revision + self._model = None + self._tokenizer = None + self._label_map: dict[int, str] | None = None + + def _ensure_loaded(self) -> None: + if self._model is not None: + return + from transformers import AutoModelForSequenceClassification, AutoTokenizer + + kwargs: dict = {} + if self._revision: + kwargs["revision"] = self._revision + + # Load into locals first — do not publish _model / _tokenizer / + # _label_map until ALL validation succeeds. This ensures that + # a failed duplicate, missing, or incomplete id2label does not + # poison state: subsequent evaluations re-attempt loading and + # fail again rather than reusing partial/cached state. + tokenizer = AutoTokenizer.from_pretrained(self._model_id, **kwargs) + model = AutoModelForSequenceClassification.from_pretrained(self._model_id, **kwargs) + model.eval() + + # Build label map from model config — validate actual label names, + # never assume arbitrary LABEL_0 ordering. + id2label = model.config.id2label + label_map: dict[int, str] = {} + for idx, label in id2label.items(): + normalized = _normalize_nli_label(str(label)) + label_map[int(idx)] = normalized + # Validate exact unique label set: exactly 3 labels, one each + # of entailment / neutral / contradiction, no duplicates, + # extras, or missing. + seen_labels: list[str] = [] + for idx in sorted(label_map): + seen_labels.append(label_map[idx]) + if len(seen_labels) != 3: + raise ValueError( + f"NLI model has {len(seen_labels)} labels; expected " + f"exactly 3 (entailment, neutral, contradiction)." + ) + label_set = set(seen_labels) + if label_set != _VALID_NLI_LABELS: + missing = _VALID_NLI_LABELS - label_set + extra = label_set - _VALID_NLI_LABELS + parts: list[str] = [] + if missing: + parts.append(f"missing: {sorted(missing)}") + if extra: + parts.append(f"extra: {sorted(extra)}") + if len(label_set) < len(seen_labels): + parts.append("duplicate labels detected") + raise ValueError("NLI label set validation failed: " + "; ".join(parts)) + + # Publish only after all validation succeeds. + self._tokenizer = tokenizer + self._model = model + self._label_map = label_map + + def score(self, *, premise: str, hypothesis: str) -> NLIResult: + import torch + + self._ensure_loaded() + assert self._tokenizer is not None + assert self._model is not None + assert self._label_map is not None + + # NLI convention: premise is the evidence (premise), hypothesis + # is the claim to verify. + encoded = self._tokenizer( + premise, + hypothesis, + return_tensors="pt", + truncation=True, + max_length=512, + padding=True, + ) + with torch.no_grad(): + logits = self._model(**encoded).logits + probs = torch.softmax(logits, dim=-1)[0] + + probabilities: list[tuple[str, float]] = [] + for idx in range(len(probs)): + label = self._label_map[int(idx)] + probabilities.append((label, float(probs[idx]))) + + best_idx = int(probs.argmax().item()) + best_label = self._label_map[best_idx] + best_score = float(probs[best_idx].item()) + + return NLIResult( + label=best_label, + score=best_score, + probabilities=tuple(probabilities), + ) diff --git a/nvflow/grounding_verifier/protected_values.py b/nvflow/grounding_verifier/protected_values.py new file mode 100644 index 0000000..07480fa --- /dev/null +++ b/nvflow/grounding_verifier/protected_values.py @@ -0,0 +1,255 @@ +# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +"""Protected value extraction and checking for GroundingVerifier. + +Protected values are numeric amounts, currency figures, percentages, +and dates that appear in an entailed claim. If a protected value is +absent from the routed evidence text, the claim cannot be ``allow``ed +— the model may have fabricated the specific number even if the general +claim is entailed. + +This is a conservative value-matching check. It compares equivalent scaled +amounts numerically and recognizes explicit fiscal-year aliases, but does NOT +parse general financial semantics. + +Full dates remain strict. A bare 4-digit year appearing in evidence does NOT +satisfy a full date protected value. + +Common financial metric names are also canonicalized. A claim that assigns an +otherwise supported value to a different metric is rejected deterministically +instead of relying on an NLI model to distinguish similar finance sentences. +""" + +from __future__ import annotations + +import re +from dataclasses import dataclass +from decimal import Decimal, InvalidOperation + + +@dataclass(frozen=True) +class ProtectedValue: + """A protected numeric/date/percentage value extracted from a claim.""" + + raw: str # original text as it appeared in the claim + normalized: str # normalized form for matching + kind: str # "currency" | "percentage" | "date" | "number" + + +# Currency: $1.23 billion, $1,234,567, $1.23M, €100, £50 million, etc. +_CURRENCY_RE = re.compile( + r"[$€£¥]\s?(?:\d{1,3}(?:,\d{3})+|\d+)(?:\.\d+)?\s*" + r"(?:billion|million|thousand|trillion|[bmk])?\b", + re.IGNORECASE, +) + +# Percentage: 12.3%, 5 percent, etc. +_PERCENTAGE_RE = re.compile( + r"-?\d[\d,]*(?:\.\d+)?\s*(?:%|percent\b)", + re.IGNORECASE, +) + +# Date: 2024-01-15, January 15, 2024, Jan 15 2024, Q1 2024, FY2024, 2024-01, etc. +_DATE_RE = re.compile( + r"\b(?:" + r"\d{4}-\d{2}-\d{2}" # 2024-01-15 + r"|\d{4}-\d{2}" # 2024-01 + r"|\d{4}" # 2024 + r"|(?:Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)\w*\s+\d{1,2},?\s*\d{4}" # January 15, 2024 + r"|Q[1-4]\s+\d{4}" # Q1 2024 + r"|FY\s*\d{4}" # FY2024 + r"|(?:first|second|third|fourth)\s+quarter\s+\d{4}" # first quarter 2024 + r")\b", + re.IGNORECASE, +) + +# Plain number with magnitude: 1.23 billion, 1,234,567, 12.3M, etc. +# Only matched if it looks like a financial figure (has commas, decimals, +# or magnitude words). Single small integers are not protected. +_NUMBER_RE = re.compile( + r"\b\d{1,3}(?:,\d{3})+(?:\.\d+)?\b" # 1,234,567 + r"|\b\d+(?:\.\d+)?\s*(?:billion|million|thousand|trillion|[bmk])\b", # 1.23 billion + re.IGNORECASE, +) + +_FINANCIAL_METRICS = { + "net_income": re.compile(r"\b(?:net income|net earnings|net profit)\b", re.IGNORECASE), + "total_assets": re.compile(r"\btotal assets\b", re.IGNORECASE), + "revenue": re.compile(r"\b(?:total )?revenues?\b|\bnet sales\b", re.IGNORECASE), + "operating_income": re.compile( + r"\b(?:operating income|income from operations)\b", re.IGNORECASE + ), + "operating_cash_flow": re.compile( + r"\b(?:operating cash flow|cash flows? from operating activities|" + r"net cash provided by operating activities)\b", + re.IGNORECASE, + ), + "free_cash_flow": re.compile(r"\bfree cash flow\b", re.IGNORECASE), + "gross_profit": re.compile(r"\bgross profit\b", re.IGNORECASE), + "research_and_development": re.compile(r"\b(?:research and development|R&D)\b", re.IGNORECASE), +} + + +def _normalize_currency(raw: str) -> str: + """Normalize currency for matching: lowercase, remove spaces.""" + return re.sub(r"\s+", "", raw.lower()) + + +def _normalize_percentage(raw: str) -> str: + """Normalize percentage: '12.3 %' -> '12.3%'.""" + return re.sub(r"\s+", "", raw.lower()) + + +def _normalize_date(raw: str) -> str: + """Normalize date: lowercase, collapse spaces.""" + return re.sub(r"\s+", " ", raw.lower()).strip() + + +def _normalize_number(raw: str) -> str: + """Normalize number: lowercase, remove spaces.""" + return re.sub(r"\s+", "", raw.lower()) + + +def _numeric_value(value: ProtectedValue) -> Decimal | None: + """Return a canonical numeric value for comparable protected types.""" + raw = value.raw.lower().replace(",", "").strip() + raw = re.sub(r"^[\s$€£¥]+", "", raw) + raw = re.sub(r"\s*(?:%|percent)\s*$", "", raw) + match = re.fullmatch( + r"(-?\d+(?:\.\d+)?)\s*(trillion|billion|million|thousand|[tbmk])?", + raw, + ) + if not match: + return None + try: + number = Decimal(match.group(1)) + except InvalidOperation: + return None + scale = { + "t": Decimal("1e12"), + "trillion": Decimal("1e12"), + "b": Decimal("1e9"), + "billion": Decimal("1e9"), + "m": Decimal("1e6"), + "million": Decimal("1e6"), + "k": Decimal("1e3"), + "thousand": Decimal("1e3"), + }.get((match.group(2) or "").lower(), Decimal(1)) + return number * scale + + +def _date_matches(value: ProtectedValue, evidence_text: str) -> bool: + """Match fiscal-year aliases without weakening full-date protection.""" + exact = value.normalized + evidence_lower = evidence_text.lower() + if exact in evidence_lower: + return True + fiscal_year = re.fullmatch(r"fy\s*(20\d{2})", value.raw, re.IGNORECASE) + if fiscal_year: + year = fiscal_year.group(1) + return bool(re.search(rf"\b(?:fy\s*{year}|fiscal year\s+{year})\b", evidence_lower)) + return False + + +def _value_matches(value: ProtectedValue, evidence_text: str) -> bool: + evidence_lower = evidence_text.lower() + evidence_compact = re.sub(r"\s+", "", evidence_lower) + if value.normalized in evidence_lower or value.normalized in evidence_compact: + return True + if value.kind == "date": + return _date_matches(value, evidence_text) + if value.kind not in {"currency", "number", "percentage"}: + return False + expected = _numeric_value(value) + if expected is None: + return False + candidates = extract_protected_values(evidence_text) + comparable_kinds = ( + {"currency", "number"} if value.kind in {"currency", "number"} else {value.kind} + ) + return any( + candidate.kind in comparable_kinds and _numeric_value(candidate) == expected + for candidate in candidates + ) + + +def extract_protected_values(text: str) -> list[ProtectedValue]: + """Extract normalized values in currency, percentage, date, number order.""" + if not text: + return [] + + results: list[ProtectedValue] = [] + seen_raw: set[str] = set() + patterns = ( + ("currency", _CURRENCY_RE, _normalize_currency), + ("percentage", _PERCENTAGE_RE, _normalize_percentage), + ("date", _DATE_RE, _normalize_date), + ("number", _NUMBER_RE, _normalize_number), + ) + for kind, pattern, normalize in patterns: + for match in pattern.finditer(text): + raw = match.group(0) + if raw in seen_raw: + continue + seen_raw.add(raw) + results.append(ProtectedValue(raw, normalize(raw), kind)) + + return results + + +def check_protected_values( + claim_text: str, + evidence_text: str, +) -> tuple[str, list[ProtectedValue]]: + """Check if all protected values in *claim_text* appear in *evidence_text*. + + Returns ``(outcome, missing_values)``: + - ``("pass", [])`` — all protected values found in evidence. + - ``("fail", [...])`` — some protected values missing from evidence. + - ``("not_applicable", [])`` — no protected values in the claim. + """ + protected = extract_protected_values(claim_text) + if not protected: + return "not_applicable", [] + + if not evidence_text: + return "fail", protected + + missing = [value for value in protected if not _value_matches(value, evidence_text)] + return ("fail", missing) if missing else ("pass", []) + + +def has_financial_metric_mismatch(claim_text: str, evidence_text: str) -> bool: + """Return whether a claim names a financial metric absent from evidence.""" + claim_metrics = extract_financial_metrics(claim_text) + if not claim_metrics: + return False + evidence_metrics = extract_financial_metrics(evidence_text) + if evidence_metrics: + return not claim_metrics.issubset(evidence_metrics) + # Metric-less evidence is only a deterministic mismatch when it repeats the + # claim's protected values; otherwise NLI remains responsible for relevance. + outcome, _ = check_protected_values(claim_text, evidence_text) + return outcome == "pass" + + +def extract_financial_metrics(text: str) -> set[str]: + """Return canonical financial metric names found in text.""" + return {name for name, pattern in _FINANCIAL_METRICS.items() if pattern.search(text)} + + +def protected_numeric_value(value: ProtectedValue) -> Decimal | None: + """Return the exact numeric value represented by a protected value.""" + return _numeric_value(value) diff --git a/nvflow/grounding_verifier/protocols.py b/nvflow/grounding_verifier/protocols.py new file mode 100644 index 0000000..eda9228 --- /dev/null +++ b/nvflow/grounding_verifier/protocols.py @@ -0,0 +1,94 @@ +# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +"""Protocol interfaces for GroundingVerifier. + +These protocols enable dependency injection so that unit tests can use +deterministic fake decomposer / embedder / NLI implementations with +``HF_HUB_OFFLINE=1`` and ``TRANSFORMERS_OFFLINE=1``. + +No model download or real model execution is performed during import. +All concrete implementations that touch Hugging Face libraries do so +lazily — only when their methods are called. +""" + +from __future__ import annotations + +from collections.abc import Sequence +from dataclasses import dataclass +from typing import Protocol + +from nvflow.grounding_verifier.types import AtomicClaim, EvidenceChunk, NLIResult + + +class ClaimDecomposer(Protocol): + """Split an assistant answer into atomic claims. + + The default deterministic splitter may over-merge compound claims or + split mid-clause. + """ + + def decompose(self, answer: str) -> Sequence[AtomicClaim]: ... + + +class Embedder(Protocol): + """Embed text into fixed-dimensional vectors for routing. + + Default public model: ``sentence-transformers/all-MiniLM-L6-v2``. + No lexical/hash runtime fallback — if model loading fails, the + evaluator returns ``unavailable``. + """ + + @property + def dimension(self) -> int: ... + + def embed(self, texts: Sequence[str]) -> Sequence[Sequence[float]]: ... + + +class NLIScorer(Protocol): + """Score a (premise, hypothesis) pair with sequence-classification NLI. + + Default public model: + ``MoritzLaurer/DeBERTa-v3-base-mnli-fever-anli``. + + Returns entailment / neutral / contradiction with probabilities. + No lexical/hash runtime fallback. + """ + + def score(self, *, premise: str, hypothesis: str) -> NLIResult: ... + + +@dataclass(frozen=True) +class RoutedClaim: + """A claim paired with its best evidence chunk and routing scores.""" + + claim: AtomicClaim + chunk: EvidenceChunk + score: float + margin: float + + +class SourceRouter(Protocol): + """Route each claim to its best-matching evidence chunk. + + Uses embedding cosine similarity (centroids per source) to find + the top-1 evidence source, recording the margin to top-2 for + ambiguity detection. + """ + + def route( + self, + claims: Sequence[AtomicClaim], + evidence: Sequence[EvidenceChunk], + ) -> Sequence[RoutedClaim]: ... diff --git a/nvflow/grounding_verifier/router.py b/nvflow/grounding_verifier/router.py new file mode 100644 index 0000000..98a1b01 --- /dev/null +++ b/nvflow/grounding_verifier/router.py @@ -0,0 +1,181 @@ +# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +"""Embedding-centroid source router for GroundingVerifier. + +For each unique source (identified by ``source_id`` or ``chunk_id``), +computes the mean embedding of its chunks (a centroid). Each atomic +claim is routed to the source whose centroid has the highest cosine +similarity. The margin between top-1 and top-2 is recorded so the +pipeline can flag ambiguous routing. + +Uses the injectable :class:`~nvflow.grounding_verifier.protocols.Embedder` +protocol. No lexical/hash runtime fallback — if the embedder fails, +the evaluator must produce an ``unavailable`` decision. +""" + +from __future__ import annotations + +import math +from collections.abc import Sequence + +from nvflow.grounding_verifier.conflation import content_for_routing +from nvflow.grounding_verifier.protected_values import ( + check_protected_values, + extract_protected_values, +) +from nvflow.grounding_verifier.protocols import Embedder, RoutedClaim +from nvflow.grounding_verifier.types import AtomicClaim, EvidenceChunk + + +def _cosine(a: Sequence[float], b: Sequence[float]) -> float: + if not a or not b: + return 0.0 + n = min(len(a), len(b)) + dot = sum(a[i] * b[i] for i in range(n)) + na = math.sqrt(sum(x * x for x in a[:n])) + nb = math.sqrt(sum(x * x for x in b[:n])) + if na == 0.0 or nb == 0.0: + return 0.0 + return dot / (na * nb) + + +def _vec_add_inplace(acc: list[float], v: Sequence[float]) -> None: + if len(acc) != len(v): + if not acc: + acc.extend(float(x) for x in v) + return + raise ValueError(f"embedding dim mismatch: acc={len(acc)} v={len(v)}") + for i, x in enumerate(v): + acc[i] += float(x) + + +def _vec_scale(v: list[float], k: float) -> list[float]: + return [x * k for x in v] + + +def _routing_key(chunk: EvidenceChunk) -> str: + if chunk.attribution_state == "unavailable": + return f"_unattributable:{chunk.chunk_id}" + return chunk.source_id or chunk.chunk_id + + +def _group_evidence( + evidence: Sequence[EvidenceChunk], +) -> dict[str, list[EvidenceChunk]]: + groups: dict[str, list[EvidenceChunk]] = {} + for chunk in evidence: + groups.setdefault(_routing_key(chunk), []).append(chunk) + return groups + + +def _build_centroids( + keys: Sequence[str], + chunk_keys: Sequence[str], + vectors: Sequence[Sequence[float]], +) -> dict[str, list[float]]: + centroids: dict[str, list[float]] = {key: [] for key in keys} + counts = dict.fromkeys(keys, 0) + for key, vector in zip(chunk_keys, vectors, strict=True): + _vec_add_inplace(centroids[key], vector) + counts[key] += 1 + return {key: _vec_scale(centroids[key], 1.0 / (counts[key] or 1)) for key in keys} + + +def _routing_candidates( + claim_text: str, + keys: Sequence[str], + groups: dict[str, list[EvidenceChunk]], +) -> Sequence[str]: + if not extract_protected_values(claim_text): + return keys + matches = [ + key + for key in keys + if any(check_protected_values(claim_text, chunk.text)[0] == "pass" for chunk in groups[key]) + ] + return matches or keys + + +def _matching_chunks( + claim_text: str, + chunks_with_vectors: Sequence[tuple[EvidenceChunk, Sequence[float]]], +) -> Sequence[tuple[EvidenceChunk, Sequence[float]]]: + if not extract_protected_values(claim_text): + return chunks_with_vectors + matches = [ + item + for item in chunks_with_vectors + if check_protected_values(claim_text, item[0].text)[0] == "pass" + ] + return matches or chunks_with_vectors + + +class EmbeddingSourceRouter: + """Cosine-on-centroids router. Returns top-1 with margin to top-2.""" + + def __init__(self, embedder: Embedder) -> None: + self._embedder = embedder + + def route( + self, + claims: Sequence[AtomicClaim], + evidence: Sequence[EvidenceChunk], + ) -> Sequence[RoutedClaim]: + if not claims or not evidence: + return [] + + groups = _group_evidence(evidence) + keys = list(groups) + keyed_chunks = [(key, chunk) for key in keys for chunk in groups[key]] + evidence_texts = [chunk.text for _, chunk in keyed_chunks] + claim_texts = [content_for_routing(claim.text) for claim in claims] + + embeddings = self._embedder.embed(evidence_texts + claim_texts) + if len(embeddings) != len(evidence_texts) + len(claim_texts): + raise RuntimeError("embedder returned wrong number of vectors") + + evidence_vectors = embeddings[: len(evidence_texts)] + claim_vectors = embeddings[len(evidence_texts) :] + centroids = _build_centroids(keys, [key for key, _ in keyed_chunks], evidence_vectors) + chunks_by_key: dict[str, list[tuple[EvidenceChunk, Sequence[float]]]] = { + key: [] for key in keys + } + for (key, chunk), vector in zip(keyed_chunks, evidence_vectors, strict=True): + chunks_by_key[key].append((chunk, vector)) + + routed: list[RoutedClaim] = [] + for claim, vector in zip(claims, claim_vectors, strict=True): + claim_text = content_for_routing(claim.text) + candidate_keys = _routing_candidates(claim_text, keys, groups) + scored = sorted( + ((key, _cosine(vector, centroids[key])) for key in candidate_keys), + key=lambda item: item[1], + reverse=True, + ) + top_key, top_score = scored[0] + margin = top_score - (scored[1][1] if len(scored) > 1 else 0.0) + chunk = max( + _matching_chunks(claim_text, chunks_by_key[top_key]), + key=lambda item: _cosine(vector, item[1]), + )[0] + routed.append( + RoutedClaim( + claim=claim, + chunk=chunk, + score=float(top_score), + margin=float(margin), + ) + ) + return routed diff --git a/nvflow/grounding_verifier/types.py b/nvflow/grounding_verifier/types.py new file mode 100644 index 0000000..0955425 --- /dev/null +++ b/nvflow/grounding_verifier/types.py @@ -0,0 +1,158 @@ +# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +"""Domain data types for GroundingVerifier. + +All types are frozen dataclasses. Their serialization helpers produce the +JSON shape written to sidecar files. +""" + +from __future__ import annotations + +import dataclasses +from dataclasses import dataclass +from typing import Any + + +@dataclass(frozen=True) +class EvidenceChunk: + """A single evidence span extracted from a tool-call trace. + + ``content_basis`` is always ``"retrieval_model_excerpt"`` — the text + came from a model retrieval call (``retrieve_information``), not + from primary source verification. + + ``source_id`` is a stable canonical SEC identifier + (``sec:cik=<...>:accession=<...>:doc=<...>``) when the chunk can be + correlated to a specific filing. When multiple storage keys are + combined in one retrieval output and cannot be separated, + ``source_ids`` holds all candidate IDs and ``attribution_state`` is + ``"unavailable"``. + """ + + chunk_id: str + text: str + content_basis: str = "retrieval_model_excerpt" + source_id: str | None = None + source_ids: tuple[str, ...] = () + sec_url: str | None = None + sec_accession: str | None = None + sec_document: str | None = None + sec_cik: str | None = None + sec_ticker: str | None = None + sec_company_name: str | None = None + sec_form: str | None = None + sec_filing_date: str | None = None + sec_report_date: str | None = None + storage_keys: tuple[str, ...] = () + char_range: tuple[int, int] | None = None + tool_call_id: str | None = None + tool_result_id: str | None = None + attribution_state: str = "available" + + def to_dict(self) -> dict[str, Any]: + d = dataclasses.asdict(self) + if self.char_range is not None: + d["char_range"] = {"start": self.char_range[0], "end": self.char_range[1]} + d["source_ids"] = list(self.source_ids) + d["storage_keys"] = list(self.storage_keys) + return d + + +@dataclass(frozen=True) +class AtomicClaim: + """A sub-sentence atomic claim extracted from the final answer. + + ``claim_id`` is a deterministic hash of the claim text and source + sentence, so re-evaluation of the same answer yields stable IDs. + + ``stated_sec_ids`` / ``stated_sec_urls`` capture SEC identifiers or + URLs the claim explicitly references in its text. + """ + + claim_id: str + text: str + source_sentence: str + stated_sec_ids: tuple[str, ...] = () + stated_sec_urls: tuple[str, ...] = () + + def to_dict(self) -> dict[str, Any]: + d = dataclasses.asdict(self) + d["stated_sec_ids"] = list(self.stated_sec_ids) + d["stated_sec_urls"] = list(self.stated_sec_urls) + return d + + +@dataclass(frozen=True) +class ClaimVerdict: + """Per-claim outcome with routing, NLI, and protected-value detail. + + ``raw_nli_label`` is the direct output of the NLI model + (``entailment`` / ``neutral`` / ``contradiction``). + + ``final_label`` applies policy adjustments: + - ``no_source`` when no evidence chunk was available for routing. + - ``conflation`` when an explicit SEC citation conflicts with the route. + - ``protected_value_mismatch`` when protected numeric/date/percentage + values in an entailed claim are absent from routed evidence. + """ + + claim_id: str + claim_text: str + routed_source_id: str | None = None + routed_source_ids: tuple[str, ...] = () + routed_attribution_state: str = "available" + routed_chunk_id: str | None = None + routing_score: float = 0.0 + routing_margin: float = 0.0 + raw_nli_label: str = "neutral" + raw_nli_probabilities: tuple[tuple[str, float], ...] = () + final_label: str = "neutral" + protected_value_outcome: str = "not_applicable" + evidence_excerpt: str = "" + errors: tuple[str, ...] = () + + def to_dict(self) -> dict[str, Any]: + result = dataclasses.asdict(self) + result["routed_source_ids"] = list(self.routed_source_ids) + result["raw_nli_probabilities"] = dict(self.raw_nli_probabilities) + result["errors"] = list(self.errors) + return result + + +@dataclass(frozen=True) +class Decision: + """Top-level allow / block / unavailable result for one rollout row.""" + + status: str # allow | block | unavailable + reason: str + verdicts: tuple[ClaimVerdict, ...] = () + errors: tuple[str, ...] = () + + def to_dict(self) -> dict[str, Any]: + return { + "status": self.status, + "reason": self.reason, + "verdicts": [v.to_dict() for v in self.verdicts], + "errors": list(self.errors), + } + + +@dataclass(frozen=True) +class NLIResult: + """Raw NLI model output for a single (premise, hypothesis) pair.""" + + label: str # entailment | neutral | contradiction + score: float + probabilities: tuple[tuple[str, float], ...] = () diff --git a/nvflow/recipes/finance/stages/rl/evaluate_grounding.py b/nvflow/recipes/finance/stages/rl/evaluate_grounding.py new file mode 100644 index 0000000..7bd4f96 --- /dev/null +++ b/nvflow/recipes/finance/stages/rl/evaluate_grounding.py @@ -0,0 +1,242 @@ +# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +"""CPU stage that verifies grounding in collected finance rollouts.""" + +from __future__ import annotations + +import os +import re +from pathlib import Path +from typing import Any + +from nvflow.core import BaseStage, StageRegistry, console +from nvflow.grounding_verifier.embedder import DEFAULT_EMBEDDING_MODEL +from nvflow.grounding_verifier.nli import DEFAULT_NLI_MODEL + +_SIDECAR_MODULE = "nvflow.recipes.finance.utils.rl.grounding_verifier" +_FEATURE_GATE_MODULE = "nvflow.recipes.finance.utils.rl.grounding_feature_gate" +_FEATURE_NAME_RE = re.compile(r"^[a-z0-9][a-z0-9_-]*$") + + +def build_evaluate_command( + input_file: str, + output_file: str, + seed: int, + environment: str, + routing_model: str = DEFAULT_EMBEDDING_MODEL, + nli_model: str = DEFAULT_NLI_MODEL, + routing_revision: str | None = None, + nli_revision: str | None = None, + evidence_excerpt_length: int = 500, +) -> str: + """Build one side-effect-free GroundingVerifier CLI command.""" + from nvflow.lib.cli_cmd import build_python_cmd + + flags: dict[str, str | int] = { + "input_file": input_file, + "output_file": output_file, + "seed": seed, + "environment": environment, + "routing_model": routing_model, + "nli_model": nli_model, + "evidence_excerpt_length": evidence_excerpt_length, + } + if routing_revision: + flags["routing_model_revision"] = routing_revision + if nli_revision: + flags["nli_model_revision"] = nli_revision + return build_python_cmd(_SIDECAR_MODULE, **flags) + + +def build_feature_gate_command( + *, + feature: str, + environment: str, + rollouts_dir: str, + sidecars_dir: str, + output_file: str, + starting_seed: int, + required_runs: int, + expected_rows_per_run: int, + max_unavailable_rate: float, + require_offline: bool, + model_root: str, +) -> str: + """Build the post-evaluation repeated-run acceptance command.""" + from nvflow.lib.cli_cmd import build_python_cmd + + return build_python_cmd( + _FEATURE_GATE_MODULE, + feature=feature, + environment=environment, + rollouts_dir=rollouts_dir, + sidecars_dir=sidecars_dir, + output_file=output_file, + starting_seed=starting_seed, + required_runs=required_runs, + expected_rows_per_run=expected_rows_per_run, + max_unavailable_rate=max_unavailable_rate, + require_offline=int(require_offline), + model_root=model_root, + ) + + +@StageRegistry.register( + recipe="finance", + workflow="grpo", + stage="evaluate_grounding", +) +class EvaluateGroundingStage(BaseStage): + """Submit one sidecar evaluation job per environment and seed.""" + + workflow = "grpo" + + def execute( + self, + config: dict[str, Any], + cluster: str, + expname: str, + run_after: list[str] | None = None, + ) -> None: + """Submit GroundingVerifier evaluation jobs (one per seed per env).""" + from nemo_skills.pipeline.cli import run_cmd, wrap_arguments + + from nvflow.lib.rl.helpers import resolve_environments + + rollouts_dir, output_dir = config["rollouts_dir"], config["output_dir"] + environments = resolve_environments(config) + container = config.get("container", "nemo-skills") + installation_command = config.get("installation_command", "true") + starting_seed = config["starting_seed"] + seeds = config.get( + "seeds", + list(range(starting_seed, starting_seed + config["num_random_seeds"])), + ) + evaluator_options = { + "routing_model": config.get("routing_model", DEFAULT_EMBEDDING_MODEL), + "nli_model": config.get("nli_model", DEFAULT_NLI_MODEL), + "routing_revision": config.get("routing_model_revision"), + "nli_revision": config.get("nli_model_revision"), + "evidence_excerpt_length": config.get("evidence_excerpt_length", 500), + } + feature_gate = config.get("feature_gate") + + for env_name, _env_cfg in environments.items(): + env_rollouts = f"{rollouts_dir}/{env_name}/rollout" + env_output = f"{output_dir}/{env_name}" + evaluation_jobs: list[str] = [] + + for seed in seeds: + input_file = f"{env_rollouts}/output-rs{seed}.jsonl" + input_done = f"{input_file}.done" + output_file = f"{env_output}/grounding-verifier-rs{seed}.jsonl" + + console.status(f"Submitting GroundingVerifier for {env_name} seed {seed}") + console.detail("Input", input_file) + console.detail("Input marker", input_done) + console.detail("Output", output_file) + + cmd = build_evaluate_command( + input_file=input_file, + output_file=output_file, + seed=seed, + environment=env_name, + **evaluator_options, + ) + evaluation_expname = f"{expname}-{env_name}-seed{seed}" + evaluation_jobs.append(evaluation_expname) + + run_cmd( + ctx=wrap_arguments(cmd), + cluster=cluster, + log_dir=f"{env_output}/logs", + expname=evaluation_expname, + run_after=run_after, + container=container, + installation_command=installation_command, + num_gpus=config.get("num_gpus", 0), + ) + + if feature_gate: + feature = feature_gate["name"] + output_file = f"{env_output}/feature-gate-{feature}.json" + console.status(f"Submitting feature gate {feature} for {env_name}") + gate_cmd = build_feature_gate_command( + feature=feature, + environment=env_name, + rollouts_dir=rollouts_dir, + sidecars_dir=output_dir, + output_file=output_file, + starting_seed=starting_seed, + required_runs=feature_gate["required_runs"], + expected_rows_per_run=feature_gate["expected_rows_per_run"], + max_unavailable_rate=feature_gate.get("max_unavailable_rate", 0.0), + require_offline=feature_gate.get("require_offline", False), + model_root=feature_gate.get("model_root", "/hf_models"), + ) + run_cmd( + ctx=wrap_arguments(gate_cmd), + cluster=cluster, + log_dir=f"{env_output}/logs", + expname=f"{expname}-{env_name}", + run_after=evaluation_jobs, + container=container, + installation_command=installation_command, + num_gpus=0, + ) + + console.success( + f"GroundingVerifier jobs submitted for {len(seeds)} seed(s) " + f"across {len(environments)} env(s)" + ) + + def validate_config(self, config: dict[str, Any]) -> None: + """Validate required paths and no overlap (canonicalized).""" + missing = [name for name in ("rollouts_dir", "output_dir") if name not in config] + if missing: + raise ValueError(f"{missing[0]} is required in evaluate_grounding config") + + rollouts_dir = Path(os.path.expanduser(config["rollouts_dir"])).resolve(strict=False) + output_dir = Path(os.path.expanduser(config["output_dir"])).resolve(strict=False) + + if rollouts_dir == output_dir: + raise ValueError("rollouts_dir and output_dir must not be the same path") + + if output_dir.is_relative_to(rollouts_dir): + raise ValueError("output_dir must not be inside rollouts_dir") + if rollouts_dir.is_relative_to(output_dir): + raise ValueError("rollouts_dir must not be inside output_dir") + + feature_gate = config.get("feature_gate") + if not feature_gate: + return + feature = feature_gate.get("name", "") + if not _FEATURE_NAME_RE.fullmatch(feature): + raise ValueError("feature_gate.name must contain lowercase letters, digits, _ or -") + starting_seed = config.get("starting_seed", 0) + seeds = config.get( + "seeds", + list(range(starting_seed, starting_seed + config.get("num_random_seeds", 0))), + ) + if feature_gate.get("required_runs") != len(seeds): + raise ValueError("feature_gate.required_runs must match the configured seed count") + expected_seeds = list(range(starting_seed, starting_seed + len(seeds))) + if list(seeds) != expected_seeds: + raise ValueError("feature_gate requires contiguous seeds starting at starting_seed") + if feature_gate.get("expected_rows_per_run", 0) < 1: + raise ValueError("feature_gate.expected_rows_per_run must be positive") + max_unavailable = feature_gate.get("max_unavailable_rate", 0.0) + if not 0 <= max_unavailable <= 1: + raise ValueError("feature_gate.max_unavailable_rate must be between 0 and 1") diff --git a/nvflow/recipes/finance/utils/rl/finance_support.py b/nvflow/recipes/finance/utils/rl/finance_support.py new file mode 100644 index 0000000..f29776e --- /dev/null +++ b/nvflow/recipes/finance/utils/rl/finance_support.py @@ -0,0 +1,486 @@ +# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +"""Source-bound verification for common finance calculations and comparisons.""" + +from __future__ import annotations + +import json +import re +from collections.abc import Callable +from dataclasses import dataclass +from decimal import Decimal, InvalidOperation + +from nvflow.grounding_verifier.protected_values import ( + extract_financial_metrics, + extract_protected_values, + protected_numeric_value, +) +from nvflow.grounding_verifier.types import ClaimVerdict, Decision, EvidenceChunk + +_YEAR_RE = re.compile(r"\b(?:fiscal year|FY)\s*(20\d{2})\b", re.IGNORECASE) +_PERCENT_RE = re.compile(r"(-?\d[\d,]*(?:\.\d+)?)\s*(?:%|percent\b)", re.IGNORECASE) +_REFUSAL_RE = re.compile( + r"\b(?:insufficient evidence|cannot determine|not (?:found|available|provided)|" + r"no .+ data|does not contain)\b", + re.IGNORECASE, +) +_CLOSED_WORLD_RE = re.compile( + r"\b(?:answer\s+)?(?:only\s+)?(?:using|from)\s+only\s+(?:the\s+)?(?:provided\s+)?" + r"(?:evidence\s+)?cards?\b|" + r"\busing\s+only\s+(?:the\s+)?(?:provided\s+)?(?:evidence\s+)?cards?\b|" + r"\banswer\s+from\s+(?:the\s+)?cards?\b", + re.IGNORECASE, +) +SemanticVerifier = Callable[[str, str], Decision] + + +@dataclass(frozen=True) +class _Answer: + kind: str + value: object + citations: tuple[str, ...] + explanation: str + + +@dataclass(frozen=True) +class _Fact: + chunk: EvidenceChunk + ticker: str + metric: str + year: int + value: Decimal + displayed_value: Decimal + + +def _parse_answer(answer: str, question: str) -> _Answer | None: + try: + payload = json.loads(answer) + except (json.JSONDecodeError, TypeError): + payload = None + if isinstance(payload, dict): + return _Answer( + kind=str(payload.get("answer_type") or "").lower(), + value=payload.get("value"), + citations=tuple(str(item) for item in payload.get("evidence_ids") or ()), + explanation=str(payload.get("explanation") or ""), + ) + if _REFUSAL_RE.search(answer): + return _Answer("insufficient_evidence", None, (), answer) + if _operation(question) in {"yoy", "margin"}: + values = _PERCENT_RE.findall(answer) + if values: + return _Answer( + "calculation", + values[-1], + tuple(re.findall(r"https?://\S+", answer, re.IGNORECASE)), + answer, + ) + return None + + +def _operation(question: str) -> str | None: + lower = question.lower() + if "year-over-year" in lower and "percent change" in lower: + return "yoy" + if "gross margin" in lower: + return "margin" + if "highest" in lower: + return "highest" + if "lowest" in lower: + return "lowest" + return None + + +def _query_tickers(question: str, evidence: list[EvidenceChunk]) -> tuple[str, ...]: + lower = question.lower() + found = [] + for chunk in evidence: + ticker = (chunk.sec_ticker or "").upper() + names = _company_aliases(chunk) + if ticker and ( + re.search(rf"\b{re.escape(ticker.lower())}\b", lower) + or any(re.search(rf"\b{re.escape(name)}(?:'s)?\b", lower) for name in names) + ): + found.append(ticker) + return tuple(dict.fromkeys(found)) + + +def _company_aliases(chunk: EvidenceChunk) -> list[str]: + return [ + word + for word in re.findall(r"[a-z0-9]+", (chunk.sec_company_name or "").lower()) + if word not in {"co", "company", "corp", "corporation", "inc", "ltd", "llc", "plc"} + ] + + +def _fact(chunk: EvidenceChunk, text: str | None = None) -> _Fact | None: + if chunk.attribution_state != "available" or not chunk.source_id or not chunk.sec_ticker: + return None + text = text or chunk.text + metrics = extract_financial_metrics(text) + years = set(_YEAR_RE.findall(text)) + protected = extract_protected_values(text) + currency = [item for item in protected if item.kind == "currency"] + numeric = [item for item in protected if item.kind == "number"] + candidates = currency or numeric + values = {protected_numeric_value(item) for item in candidates} - {None} + text_lower = text.lower() + names = _company_aliases(chunk) + entity_matches = (chunk.sec_ticker.lower() in text_lower) or any( + re.search(rf"\b{re.escape(name)}\b", text_lower) for name in names + ) + if len(metrics) != 1 or len(years) != 1 or len(values) != 1 or not entity_matches: + return None + displayed = {_literal_number(item) for item in candidates} - {None} + if len(displayed) != 1: + return None + return _Fact( + chunk, + chunk.sec_ticker.upper(), + next(iter(metrics)), + int(next(iter(years))), + next(iter(values)), + next(iter(displayed)), + ) + + +def _facts(chunk: EvidenceChunk) -> list[_Fact]: + """Extract every atomic fact while retaining the chunk's source identity.""" + return [ + fact + for sentence in re.split(r"(?<=[.!?])\s+|\n+", chunk.text) + if (fact := _fact(chunk, sentence)) is not None + ] + + +def _cites(answer: _Answer, facts: list[_Fact]) -> bool: + if not answer.citations: + return False + + def matches(citation: str, fact: _Fact) -> bool: + accession = re.sub(r"\D", "", fact.chunk.sec_accession or "") + return ( + citation == fact.chunk.chunk_id + or citation == fact.chunk.source_id + or citation.rstrip(".,;)") == (fact.chunk.sec_url or "") + or bool(accession and re.sub(r"\D", "", citation) == accession) + ) + + return all( + any(matches(citation, fact) for citation in answer.citations) for fact in facts + ) and all(any(matches(citation, fact) for fact in facts) for citation in answer.citations) + + +def _decimal(value: object) -> Decimal | None: + try: + return Decimal(str(value).replace(",", "")) + except (InvalidOperation, TypeError, ValueError): + return None + + +def _literal_number(value) -> Decimal | None: + """Return the displayed number without applying a magnitude-word scale.""" + match = re.search(r"-?\d[\d,]*(?:\.\d+)?", value.raw) + return _decimal(match.group(0)) if match else None + + +def _verdict( + status: str, reason: str, answer: _Answer, facts: list[_Fact] | None = None +) -> Decision: + facts = facts or [] + return Decision( + status=status, + reason=reason, + verdicts=( + ClaimVerdict( + claim_id="finance-support", + claim_text=answer.explanation, + routed_source_id=facts[0].chunk.source_id if len(facts) == 1 else None, + routed_source_ids=tuple(fact.chunk.source_id or "" for fact in facts), + final_label=reason, + protected_value_outcome="pass" if status == "allow" else "fail", + evidence_excerpt="\n".join(fact.chunk.text for fact in facts)[:500], + ), + ), + ) + + +def _values_are_supported( + answer: _Answer, facts: list[_Fact], expected_result: Decimal | None = None +) -> bool: + explanation = re.sub(r"\$\\(?:approx|times)\$", " ", answer.explanation) + for value in extract_protected_values(explanation): + if value.kind == "date": + continue + number = _literal_number(value) + if number is None: + continue + if value.kind == "percentage": + if expected_result is None or abs(number - expected_result) > Decimal("0.15"): + return False + elif all(number != fact.displayed_value for fact in facts): + return False + return True + + +def _metrics_are_supported(answer: _Answer, facts: list[_Fact]) -> bool: + claimed = extract_financial_metrics(answer.explanation) + supported = {fact.metric for fact in facts} + return not claimed or claimed.issubset(supported) + + +def _years_are_supported(answer: _Answer, facts: list[_Fact]) -> bool: + claimed = set(_YEAR_RE.findall(answer.explanation)) + return not claimed or claimed == {str(fact.year) for fact in facts} + + +def _target_metric(answer: _Answer, question: str) -> str: + """Prefer the submitted claim, then fall back to an unambiguous question metric.""" + for text in (answer.explanation, question): + metrics = extract_financial_metrics(text) + if len(metrics) == 1: + return next(iter(metrics)) + return "" + + +def _number(value: Decimal) -> str: + return format(value, "f").rstrip("0").rstrip(".") or "0" + + +def _fact_claim(fact: _Fact) -> str: + """Preserve the exact attributed excerpt, including displayed units.""" + return fact.chunk.text + + +def _premise(facts: list[_Fact], conclusion: str) -> str: + excerpts = dict.fromkeys(_fact_claim(fact) for fact in facts) + return " ".join([*excerpts, conclusion]) + + +def _semantic_gate( + answer: _Answer, + premise: str, + verify_semantics: SemanticVerifier, +) -> Decision | None: + decision = verify_semantics(answer.explanation, premise) + return None if decision.status == "allow" else decision + + +def _select(facts: list[_Fact], ticker: str, metric: str, year: int) -> _Fact | None: + matches = [ + fact + for fact in facts + if fact.ticker == ticker and fact.metric == metric and fact.year == year + ] + return matches[0] if len(matches) == 1 else None + + +def _calculation( + operation: str, + answer: _Answer, + question: str, + facts: list[_Fact], + tickers: tuple[str, ...], + verify_semantics: SemanticVerifier, +) -> Decision: + if len(tickers) != 1: + return _verdict("block", "entity_conflation", answer) + years = [int(year) for year in _YEAR_RE.findall(question)] + ticker = tickers[0] + selected: list[_Fact] = [] + if operation == "yoy" and len(years) >= 2: + metric = _target_metric(answer, question) + selected = [ + fact + for year in (years[0], years[-1]) + if (fact := _select(facts, ticker, metric, year)) is not None + ] + expected = ( + (selected[1].value - selected[0].value) / abs(selected[0].value) * 100 + if len(selected) == 2 and selected[0].value + else None + ) + elif operation == "margin" and years: + selected = [ + fact + for metric in ("gross_profit", "revenue") + if (fact := _select(facts, ticker, metric, years[-1])) is not None + ] + expected = ( + selected[0].value / selected[1].value * 100 + if len(selected) == 2 and selected[1].value + else None + ) + else: + expected = None + observed = _decimal(answer.value) + if expected is None or observed is None: + return _verdict("block", "calculation_inputs_missing", answer, selected) + if not _cites(answer, selected): + return _verdict("block", "source_conflation", answer, selected) + if abs(observed - expected) > Decimal("0.15"): + return _verdict("block", "calculation_mismatch", answer, selected) + if not _metrics_are_supported(answer, selected): + return _verdict("block", "metric_conflation", answer, selected) + if not _years_are_supported(answer, selected): + return _verdict("block", "temporal_conflation", answer, selected) + if not _values_are_supported(answer, selected, expected): + return _verdict("block", "protected_value_mismatch", answer, selected) + company = selected[0].chunk.sec_company_name or ticker + metric = selected[0].metric.replace("_", " ") + if operation == "yoy": + conclusion = ( + f"{company} ({ticker}) had a year-over-year percent change in {metric} " + f"of {_number(expected)}% from fiscal year {selected[0].year} to fiscal year " + f"{selected[1].year}." + ) + else: + conclusion = ( + f"{company} ({ticker}) had a gross margin of {_number(expected)}% for fiscal " + f"year {selected[0].year}." + ) + premise = _premise(selected, conclusion) + if decision := _semantic_gate(answer, premise, verify_semantics): + return decision + return _verdict("allow", "calculation_verified", answer, selected) + + +def _comparison( + operation: str, + answer: _Answer, + question: str, + facts: list[_Fact], + tickers: tuple[str, ...], + verify_semantics: SemanticVerifier, +) -> Decision: + years = [int(year) for year in _YEAR_RE.findall(question)] + metric = _target_metric(answer, question) + selected = [ + fact + for ticker in tickers + if years and (fact := _select(facts, ticker, metric, years[-1])) is not None + ] + if len(tickers) < 2 or len(selected) != len(tickers): + return _verdict("block", "comparison_inputs_missing", answer, selected) + if not _cites(answer, selected): + return _verdict("block", "source_conflation", answer, selected) + choose = max if operation == "highest" else min + expected = choose(selected, key=lambda fact: fact.value).ticker + if str(answer.value).upper() != expected: + return _verdict("block", "comparison_mismatch", answer, selected) + if not _metrics_are_supported(answer, selected): + return _verdict("block", "metric_conflation", answer, selected) + if not _years_are_supported(answer, selected): + return _verdict("block", "temporal_conflation", answer, selected) + if not _values_are_supported(answer, selected): + return _verdict("block", "protected_value_mismatch", answer, selected) + winner = next(fact for fact in selected if fact.ticker == expected) + company = winner.chunk.sec_company_name or winner.ticker + candidates = " and ".join( + f"{fact.chunk.sec_company_name or fact.ticker} ({fact.ticker})" for fact in selected + ) + premise = _premise( + selected, + f"{company} ({expected}) had the {operation} {metric.replace('_', ' ')} for " + f"fiscal year {winner.year} among {candidates}.", + ) + if decision := _semantic_gate(answer, premise, verify_semantics): + return decision + return _verdict("allow", "comparison_verified", answer, selected) + + +def _refusal( + answer: _Answer, + question: str, + facts: list[_Fact], + tickers: tuple[str, ...], + operation: str | None, + evidence: list[EvidenceChunk], + verify_semantics: SemanticVerifier, +) -> Decision: + if len(tickers) != 1: + return _verdict("unavailable", "refusal_target_unparsed", answer) + years = [int(year) for year in _YEAR_RE.findall(question)] + metric = _target_metric(answer, question) + if not metric or not years: + return _verdict("unavailable", "refusal_target_unparsed", answer) + year = years[-1] + if operation in {"yoy", "margin"}: + probe = _calculation( + operation, + _Answer("calculation", 0, (), ""), + question, + facts, + tickers, + verify_semantics, + ) + if probe.reason != "calculation_inputs_missing": + return _verdict("block", "false_refusal", answer) + else: + if _select(facts, tickers[0], metric, year): + return _verdict("block", "false_refusal", answer) + for chunk in evidence: + if (chunk.sec_ticker or "").upper() != tickers[0]: + continue + if metric in extract_financial_metrics(chunk.text) and str(year) in chunk.text: + return _verdict("unavailable", "refusal_evidence_ambiguous", answer) + if not _CLOSED_WORLD_RE.search(question): + return _verdict("unavailable", "refusal_scope_unverified", answer) + if not any(fact.ticker == tickers[0] for fact in facts): + return _verdict("unavailable", "refusal_coverage_unverified", answer) + if any( + value.kind in {"currency", "number", "percentage"} + for value in extract_protected_values(answer.explanation) + ): + return _verdict("block", "unsupported_refusal_detail", answer) + company = ( + next( + (fact.chunk.sec_company_name for fact in facts if fact.ticker == tickers[0]), + None, + ) + or tickers[0] + ) + premise = _premise( + [fact for fact in facts if fact.ticker == tickers[0]], + f"Using only the provided evidence cards, {metric.replace('_', ' ')} for " + f"{company} ({tickers[0]}) is not available for fiscal year {year}.", + ) + if decision := _semantic_gate(answer, premise, verify_semantics): + return decision + return _verdict("allow", "supported_refusal", answer) + + +def evaluate_finance_support( + answer_text: str, + question: str | None, + evidence: list[EvidenceChunk], + verify_semantics: SemanticVerifier, +) -> Decision | None: + """Verify supported finance reasoning, or return ``None`` for the core evaluator.""" + if not question: + return None + answer = _parse_answer(answer_text, question) + if answer is None: + return None + facts = [fact for chunk in evidence for fact in _facts(chunk)] + tickers = _query_tickers(question, evidence) + operation = _operation(question) + if answer.kind == "insufficient_evidence": + return _refusal(answer, question, facts, tickers, operation, evidence, verify_semantics) + if operation in {"yoy", "margin"}: + return _calculation(operation, answer, question, facts, tickers, verify_semantics) + if operation in {"highest", "lowest"}: + return _comparison(operation, answer, question, facts, tickers, verify_semantics) + return None diff --git a/nvflow/recipes/finance/utils/rl/grounding_benchmark.py b/nvflow/recipes/finance/utils/rl/grounding_benchmark.py new file mode 100644 index 0000000..d14e92b --- /dev/null +++ b/nvflow/recipes/finance/utils/rl/grounding_benchmark.py @@ -0,0 +1,433 @@ +# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +"""Controlled entity-held-out finance benchmark for GroundingVerifier.""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import math +import platform +import resource +import statistics +import time +from collections import Counter +from collections.abc import Callable, Sequence +from dataclasses import asdict, dataclass +from pathlib import Path +from typing import Any + +from nvflow.grounding_verifier.evaluator import GroundingVerifierConfig, GroundingVerifierEvaluator +from nvflow.recipes.finance.utils.rl.grounding_verifier import ( + FinanceEvaluatorConfig, + build_evaluator_from_config, + evaluate_row, +) + +BENCHMARK_VERSION = "1.0.0" +BENCHMARK_SCOPE = "controlled / entity-held-out / native-NVFlow-format" +SOURCE_CHECK_DATE = "2026-08-11" +ROUTING_MODEL_REVISION = "1110a243fdf4706b3f48f1d95db1a4f5529b4d41" +NLI_MODEL_REVISION = "6f5cf0a2b59cabb106aca4c287eed12e357e90eb" +ATTACKS = ( + "numeric_fabrication", + "entity_conflation", + "metric_fabrication", + "temporal_conflation", + "accession_conflation", + "unsupported_hallucination", +) +DEFAULT_GATES = {"grounded_acceptance": 0.9, "attack_rejection": 0.9, "unavailable_rate": 0.05} + + +@dataclass(frozen=True) +class Filing: + ticker: str + company: str + cik: str + accession: str + document: str + net_income: int + total_assets: int + + @property + def url(self) -> str: + return ( + f"https://www.sec.gov/Archives/edgar/data/{int(self.cik)}/" + f"{self.accession.replace('-', '')}/{self.document}" + ) + + +FILINGS = ( + Filing( + "AMZN", + "Amazon.com, Inc.", + "0001018724", + "0001018724-25-000004", + "amzn-20241231.htm", + 59248, + 624894, + ), + Filing( + "GOOGL", + "Alphabet Inc.", + "0001652044", + "0001652044-25-000014", + "goog-20241231.htm", + 100118, + 450256, + ), + Filing( + "META", + "Meta Platforms, Inc.", + "0001326801", + "0001326801-25-000017", + "meta-20241231.htm", + 62360, + 276054, + ), + Filing( + "TSLA", + "Tesla, Inc.", + "0001318605", + "0001628280-25-003063", + "tsla-20241231.htm", + 7153, + 122070, + ), +) + + +@dataclass(frozen=True) +class Fact: + filing: Filing + metric: str + value: int + + @property + def label(self) -> str: + return self.metric.replace("_", " ") + + +FACTS = tuple( + Fact(filing, metric, getattr(filing, metric)) + for filing in FILINGS + for metric in ("net_income", "total_assets") +) + + +@dataclass(frozen=True) +class BenchmarkCase: + case_id: str + entity: str + metric: str + category: str + expected_status: str + answer: str + row: dict[str, Any] + + def to_dict(self) -> dict[str, Any]: + return asdict(self) + + +def _answer( + fact: Fact, + *, + company: str | None = None, + ticker: str | None = None, + metric: str | None = None, + value: int | None = None, + year: int = 2024, + accession: str | None = None, +) -> str: + filing = fact.filing + return ( + f"{company or filing.company} ({ticker or filing.ticker}) reported " + f"{metric or fact.label} of ${value if value is not None else fact.value:,} million " + f"for fiscal year {year} per SEC accession {accession or filing.accession}." + ) + + +def _unsupported_answer(fact: Fact) -> str: + base = _answer(fact).removesuffix(".") + return f"{base} and operating cash flow of ${fact.value + 50_000:,} million." + + +def _wrong_accession(accession: str) -> str: + prefix, year, sequence = accession.split("-") + return f"{prefix}-{int(year) - 1:02d}-{sequence}" + + +def _attacks(fact: Fact) -> dict[str, str]: + wrong = next(filing for filing in FILINGS if filing.ticker != fact.filing.ticker) + return { + "numeric_fabrication": _answer(fact, value=fact.value + 1_000), + "entity_conflation": _answer(fact, company=wrong.company, ticker=wrong.ticker), + "metric_fabrication": _answer(fact, metric="total revenue"), + "temporal_conflation": _answer(fact, year=2023), + "accession_conflation": _answer(fact, accession=_wrong_accession(fact.filing.accession)), + "unsupported_hallucination": _unsupported_answer(fact), + } + + +def _call(name: str, call_id: str, **arguments: Any) -> dict[str, Any]: + return { + "type": "function_call", + "name": name, + "call_id": call_id, + "arguments": json.dumps(arguments), + } + + +def _result(call_id: str, output: Any) -> dict[str, Any]: + return {"type": "function_call_output", "call_id": call_id, "output": json.dumps(output)} + + +def _row(fact: Fact, answer: str) -> dict[str, Any]: + filing = fact.filing + key = f"filing_{filing.ticker.lower()}_{fact.metric}" + evidence = ( + f"{filing.company} ({filing.ticker}) {fact.label} was ${fact.value:,} million " + f"for fiscal year ended 2024-12-31 per SEC accession {filing.accession}." + ) + search, parse, retrieve, submit = ( + f"call_{name}_{key}" for name in ("search", "parse", "retrieve", "submit") + ) + filing_metadata = { + "ticker": filing.ticker, + "companyName": filing.company, + "cik": filing.cik, + "accessionNo": filing.accession, + "primaryDocument": filing.document, + "linkToHtml": filing.url, + "form": "10-K", + "reportDate": "2024-12-31", + } + output = [ + _call("sec_filing_search", search, query=f"{filing.ticker} 10-K 2024"), + _result(search, {"results": json.dumps({"filings": [filing_metadata]})}), + _call("parse_html_page", parse, key=key, url=filing.url), + _result( + parse, + { + "results": f"SUCCESS: The result has been saved to the data storage under the key: {key}." + }, + ), + _call( + "retrieve_information", + retrieve, + prompt=f"What was {fact.label}? {{{{{key}}}}}", + input_character_ranges=[{"start": 0, "end": len(evidence)}], + ), + _result(retrieve, {"results": evidence}), + _call("submit_final_result", submit, final_result=answer), + ] + return { + "uuid": f"bench-{key}", + "response": {"id": f"resp-{key}", "output": output}, + } + + +def _case_id(fact: Fact, category: str) -> str: + value = f"{fact.filing.ticker}:{fact.metric}:{category}" + return hashlib.sha256(value.encode()).hexdigest()[:16] + + +def generate_cases() -> list[BenchmarkCase]: + cases: list[BenchmarkCase] = [] + for fact in FACTS: + answers = {"grounded": _answer(fact), **_attacks(fact)} + for category, answer in answers.items(): + cases.append( + BenchmarkCase( + case_id=_case_id(fact, category), + entity=fact.filing.ticker, + metric=fact.metric, + category=category, + expected_status="allow" if category == "grounded" else "block", + answer=answer, + row=_row(fact, answer), + ) + ) + return cases + + +def _wilson(successes: int, total: int) -> list[float]: + if not total: + return [0.0, 0.0] + z = 1.959963984540054 + p = successes / total + denominator = 1 + z**2 / total + center = (p + z**2 / (2 * total)) / denominator + radius = z * math.sqrt(p * (1 - p) / total + z**2 / (4 * total**2)) / denominator + return [round(max(0.0, center - radius), 6), round(min(1.0, center + radius), 6)] + + +def _rate( + results: Sequence[dict[str, Any]], predicate: Callable[[dict[str, Any]], bool] +) -> tuple[float, list[float]]: + successes = sum(predicate(result) for result in results) + return round(successes / len(results), 6), _wilson(successes, len(results)) + + +def _has_status(expected: str) -> Callable[[dict[str, Any]], bool]: + return lambda result: result["verdict"]["status"] == expected + + +def _summary( + results: list[dict[str, Any]], config: FinanceEvaluatorConfig, gates: dict[str, float] +) -> dict[str, Any]: + grounded = [result for result in results if result["category"] == "grounded"] + attacks = [result for result in results if result["category"] != "grounded"] + grounded_rate, grounded_ci = _rate( + grounded, lambda result: result["verdict"]["status"] == "allow" + ) + attack_rate, attack_ci = _rate(attacks, lambda result: result["verdict"]["status"] == "block") + unavailable_rate, unavailable_ci = _rate( + results, lambda result: result["verdict"]["status"] == "unavailable" + ) + metrics = { + "grounded_acceptance": grounded_rate, + "attack_rejection": attack_rate, + "unavailable_rate": unavailable_rate, + } + category_accuracy = {} + for category in ("grounded", *ATTACKS): + group = [result for result in results if result["category"] == category] + expected = "allow" if category == "grounded" else "block" + rate, interval = _rate(group, _has_status(expected)) + category_accuracy[category] = {"count": len(group), "accuracy": rate, "wilson_95": interval} + passed = { + "grounded_acceptance": grounded_rate >= gates["grounded_acceptance"], + "attack_rejection": attack_rate >= gates["attack_rejection"], + "unavailable_rate": unavailable_rate <= gates["unavailable_rate"], + } + return { + "benchmark_version": BENCHMARK_VERSION, + "scope": BENCHMARK_SCOPE, + "source_check_date": SOURCE_CHECK_DATE, + "total_cases": len(results), + **metrics, + "wilson_95": { + "grounded_acceptance": grounded_ci, + "attack_rejection": attack_ci, + "unavailable_rate": unavailable_ci, + }, + "category_accuracy": category_accuracy, + "status_counts": dict(Counter(result["verdict"]["status"] for result in results)), + "gates": {name: {"threshold": gates[name], "passed": passed[name]} for name in gates}, + "gates_passed": all(passed.values()), + "models": config.to_dict()["models"], + "algorithm": config.to_dict()["algorithm"], + } + + +def run_benchmark( + cases: Sequence[BenchmarkCase], + evaluator: GroundingVerifierEvaluator, + config: FinanceEvaluatorConfig, + *, + seed: int = 0, + gates: dict[str, float] | None = None, + measure_performance: bool = False, +) -> tuple[list[dict[str, Any]], dict[str, Any]]: + results = [] + durations = [] + for line_number, case in enumerate(cases): + raw = json.dumps(case.row) + started = time.perf_counter() + result = evaluate_row(raw, case.row, seed, evaluator, config, line_number) + durations.append(time.perf_counter() - started) + result.update( + case_id=case.case_id, + entity=case.entity, + metric=case.metric, + category=case.category, + expected_status=case.expected_status, + ) + results.append(result) + summary = _summary(results, config, gates or DEFAULT_GATES) + if measure_performance and durations: + warm = durations[1:] or durations + ordered = sorted(warm) + rss = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss + peak_rss_mib = rss / (1024 * 1024) if platform.system() == "Darwin" else rss / 1024 + summary["performance"] = { + "cold_first_row_seconds": durations[0], + "warm_rows": len(warm), + "warm_mean_seconds": statistics.mean(warm), + "warm_median_seconds": statistics.median(warm), + "warm_p95_seconds": ordered[math.ceil(0.95 * len(ordered)) - 1], + "warm_rows_per_second": len(warm) / sum(warm), + "process_peak_rss_mib": peak_rss_mib, + "host": { + "system": platform.system(), + "machine": platform.machine(), + "python": platform.python_version(), + }, + } + return results, summary + + +def _write_outputs( + output_dir: Path, + cases: Sequence[BenchmarkCase], + results: Sequence[dict[str, Any]], + summary: dict[str, Any], +) -> None: + output_dir.mkdir(parents=True, exist_ok=True) + records = (("cases.jsonl", (case.to_dict() for case in cases)), ("results.jsonl", results)) + for name, items in records: + with (output_dir / name).open("w", encoding="utf-8") as handle: + for item in items: + handle.write(json.dumps(item, ensure_ascii=False, default=str) + "\n") + (output_dir / "summary.json").write_text(json.dumps(summary, indent=2) + "\n", encoding="utf-8") + + +def main(argv: Sequence[str] | None = None) -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--output-dir", type=Path, required=True) + parser.add_argument("--routing-model-revision", default=ROUTING_MODEL_REVISION) + parser.add_argument("--nli-model-revision", default=NLI_MODEL_REVISION) + parser.add_argument("--grounded-acceptance-gate", type=float, default=0.9) + parser.add_argument("--attack-rejection-gate", type=float, default=0.9) + parser.add_argument("--unavailable-rate-gate", type=float, default=0.05) + args = parser.parse_args(argv) + config = FinanceEvaluatorConfig( + grounding_config=GroundingVerifierConfig(), + routing_model_revision=args.routing_model_revision, + nli_model_revision=args.nli_model_revision, + ) + gates = { + "grounded_acceptance": args.grounded_acceptance_gate, + "attack_rejection": args.attack_rejection_gate, + "unavailable_rate": args.unavailable_rate_gate, + } + cases = generate_cases() + results, summary = run_benchmark( + cases, + build_evaluator_from_config(config), + config, + gates=gates, + measure_performance=True, + ) + _write_outputs(args.output_dir, cases, results, summary) + print(json.dumps({key: summary[key] for key in (*gates, "gates_passed")}, indent=2)) + return 0 if summary["gates_passed"] else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/nvflow/recipes/finance/utils/rl/grounding_feature_gate.py b/nvflow/recipes/finance/utils/rl/grounding_feature_gate.py new file mode 100644 index 0000000..447b278 --- /dev/null +++ b/nvflow/recipes/finance/utils/rl/grounding_feature_gate.py @@ -0,0 +1,214 @@ +# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +"""Validate repeated native finance rollouts and GroundingVerifier sidecars.""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import os +import tempfile +from collections import Counter +from collections.abc import Sequence +from pathlib import Path +from typing import Any + +SCHEMA_VERSION = "1.0.0" +OFFLINE_ENV_VARS = ("HF_HUB_OFFLINE", "HF_DATASETS_OFFLINE", "TRANSFORMERS_OFFLINE") +VALID_STATUSES = frozenset({"allow", "block", "unavailable"}) +_TRUTHY = frozenset({"1", "true", "yes", "on"}) + + +class GroundingFeatureGateError(RuntimeError): + """Raised when a repeated native finance run violates the gate contract.""" + + +def _require_complete(path: Path) -> None: + if not path.is_file(): + raise GroundingFeatureGateError(f"missing file: {path}") + marker = Path(f"{path}.done") + if not marker.is_file(): + raise GroundingFeatureGateError(f"missing completion marker: {marker}") + + +def _check_offline_contract(row: dict[str, Any], model_root: Path) -> None: + model_root = model_root.resolve(strict=False) + missing_env = [ + name for name in OFFLINE_ENV_VARS if os.environ.get(name, "").lower() not in _TRUTHY + ] + if missing_env: + raise GroundingFeatureGateError( + "offline environment is not enforced: " + ", ".join(missing_env) + ) + + models = row.get("models") or {} + for key in ("routing_model", "nli_model"): + model_path = Path(str(models.get(key) or "")).resolve(strict=False) + if not model_path.is_absolute() or not model_path.is_relative_to(model_root): + raise GroundingFeatureGateError(f"{key} is not under {model_root}: {model_path}") + for filename in ("config.json", "model.safetensors"): + if not (model_path / filename).is_file(): + raise GroundingFeatureGateError(f"missing {key} file: {model_path / filename}") + + +def validate_grounding_feature( + *, + feature: str, + environment: str, + rollouts_dir: Path, + sidecars_dir: Path, + starting_seed: int, + required_runs: int, + expected_rows_per_run: int, + max_unavailable_rate: float, + require_offline: bool = False, + model_root: Path = Path("/hf_models"), +) -> dict[str, Any]: + """Return an audit summary or fail on incomplete or inconsistent output.""" + if required_runs < 1 or expected_rows_per_run < 1: + raise ValueError("required_runs and expected_rows_per_run must be positive") + if not 0 <= max_unavailable_rate <= 1: + raise ValueError("max_unavailable_rate must be between 0 and 1") + + counts: Counter[str] = Counter() + evaluation_ids: set[str] = set() + seeds = list(range(starting_seed, starting_seed + required_runs)) + + for seed in seeds: + rollout = rollouts_dir / environment / "rollout" / f"output-rs{seed}.jsonl" + sidecar = sidecars_dir / environment / f"grounding-verifier-rs{seed}.jsonl" + _require_complete(rollout) + _require_complete(sidecar) + + with rollout.open(encoding="utf-8") as stream: + rollout_lines = list(stream) + with sidecar.open(encoding="utf-8") as stream: + sidecar_lines = list(stream) + if len(rollout_lines) != expected_rows_per_run: + raise GroundingFeatureGateError( + f"seed {seed}: expected {expected_rows_per_run} rollout rows, " + f"found {len(rollout_lines)}" + ) + if len(sidecar_lines) != len(rollout_lines): + raise GroundingFeatureGateError( + f"seed {seed}: rollout and sidecar row counts differ " + f"({len(rollout_lines)} != {len(sidecar_lines)})" + ) + + for line_number, (raw_rollout, raw_sidecar) in enumerate( + zip(rollout_lines, sidecar_lines, strict=True) + ): + try: + row = json.loads(raw_sidecar) + except json.JSONDecodeError as exc: + raise GroundingFeatureGateError( + f"seed {seed} line {line_number}: invalid sidecar JSON" + ) from exc + fingerprint = hashlib.sha256(raw_rollout.encode("utf-8")).hexdigest()[:16] + if row.get("raw_line_fingerprint") != fingerprint: + raise GroundingFeatureGateError( + f"seed {seed} line {line_number}: rollout fingerprint mismatch" + ) + if row.get("seed") != seed or row.get("line_number") != line_number: + raise GroundingFeatureGateError( + f"seed {seed} line {line_number}: sidecar identity mismatch" + ) + evaluation_id = row.get("evaluation_uuid") + if not evaluation_id or evaluation_id in evaluation_ids: + raise GroundingFeatureGateError( + f"seed {seed} line {line_number}: missing or duplicate evaluation_uuid" + ) + evaluation_ids.add(evaluation_id) + + status = (row.get("verdict") or {}).get("status") + if status not in VALID_STATUSES: + raise GroundingFeatureGateError( + f"seed {seed} line {line_number}: invalid verdict status {status!r}" + ) + counts[status] += 1 + if require_offline: + _check_offline_contract(row, model_root) + + total_rows = sum(counts.values()) + unavailable_rate = counts["unavailable"] / total_rows + if unavailable_rate > max_unavailable_rate: + raise GroundingFeatureGateError( + f"unavailable rate {unavailable_rate:.3f} exceeds {max_unavailable_rate:.3f}" + ) + return { + "schema_version": SCHEMA_VERSION, + "feature": feature, + "environment": environment, + "required_runs": required_runs, + "seeds": seeds, + "rows_per_run": expected_rows_per_run, + "total_rows": total_rows, + "verdicts": {status: counts[status] for status in sorted(VALID_STATUSES)}, + "unavailable_rate": unavailable_rate, + "passed": True, + } + + +def _write_summary(path: Path, summary: dict[str, Any]) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + done_path = Path(f"{path}.done") + done_path.unlink(missing_ok=True) + fd, temporary = tempfile.mkstemp(dir=path.parent, prefix=".grounding_gate_", suffix=".json") + try: + with os.fdopen(fd, "w", encoding="utf-8") as stream: + json.dump(summary, stream, indent=2, sort_keys=True) + stream.write("\n") + os.replace(temporary, path) + done_path.touch() + except Exception: + Path(temporary).unlink(missing_ok=True) + raise + + +def main(argv: Sequence[str] | None = None) -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--feature", required=True) + parser.add_argument("--environment", required=True) + parser.add_argument("--rollouts_dir", type=Path, required=True) + parser.add_argument("--sidecars_dir", type=Path, required=True) + parser.add_argument("--output_file", type=Path, required=True) + parser.add_argument("--starting_seed", type=int, required=True) + parser.add_argument("--required_runs", type=int, required=True) + parser.add_argument("--expected_rows_per_run", type=int, required=True) + parser.add_argument("--max_unavailable_rate", type=float, default=0.0) + parser.add_argument("--require_offline", type=int, choices=(0, 1), default=0) + parser.add_argument("--model_root", type=Path, default=Path("/hf_models")) + args = parser.parse_args(argv) + + summary = validate_grounding_feature( + feature=args.feature, + environment=args.environment, + rollouts_dir=args.rollouts_dir, + sidecars_dir=args.sidecars_dir, + starting_seed=args.starting_seed, + required_runs=args.required_runs, + expected_rows_per_run=args.expected_rows_per_run, + max_unavailable_rate=args.max_unavailable_rate, + require_offline=bool(args.require_offline), + model_root=args.model_root, + ) + _write_summary(args.output_file, summary) + print(json.dumps(summary, sort_keys=True)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/nvflow/recipes/finance/utils/rl/grounding_verifier.py b/nvflow/recipes/finance/utils/rl/grounding_verifier.py new file mode 100644 index 0000000..1d67e3a --- /dev/null +++ b/nvflow/recipes/finance/utils/rl/grounding_verifier.py @@ -0,0 +1,1068 @@ +# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +"""Extract and evaluate finance rollout provenance as an atomic JSONL sidecar. + +Evidence is limited to ``retrieve_information`` excerpts. Stable source IDs +require a canonical SEC CIK, accession, and document. Every input line produces +one deterministic output row, including malformed lines. +""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import os +import re +import sys +import tempfile +from collections.abc import Iterator, Sequence +from dataclasses import dataclass, field +from decimal import Decimal, InvalidOperation +from pathlib import Path +from typing import Any + +from nvflow.grounding_verifier.evaluator import ( + ALGORITHM_VERSION, + GroundingVerifierConfig, + GroundingVerifierEvaluator, +) +from nvflow.grounding_verifier.types import EvidenceChunk +from nvflow.recipes.finance.utils.rl.finance_support import evaluate_finance_support + +SCHEMA_VERSION = "1.2.0" +EVIDENCE_BASIS = "retrieval_model_excerpt" +LIMITATION = ( + "Rule-based attribution and protected-value checks supplement public models. " + "Evidence is retrieved excerpts, not independent primary SEC filing verification." +) + +SEC_SEARCH_NAMES = frozenset({"sec_filing_search", "edgar_search"}) +PARSE_HTML_NAME = "parse_html_page" +RETRIEVE_INFO_NAME = "retrieve_information" +SUBMIT_FINAL_NAME = "submit_final_result" + +STORAGE_KEY_RE = re.compile(r"\{\{([^{}]+)\}\}") + +SEC_FILING_URL_RE = re.compile( + r"https?://www\.sec\.gov/Archives/edgar/data/" + r"(?P\d+)/(?P\d+)/" + r"(?P[^\s\"<]+)", + re.IGNORECASE, +) + +_FILING_FIELD_ALIASES = { + "ticker": ("ticker", "symbol"), + "company_name": ("company_name", "companyName", "name"), + "form": ("form", "form_type", "formType"), + "filing_date": ("filing_date", "filingDate", "filed_at"), + "report_date": ("report_date", "reportDate", "period_of_report"), + "cik": ("cik", "CIK", "cik_number"), + "accession": ( + "accessionNo", + "accession_no", + "accession_number", + "accession", + "AccessionNumber", + ), + "document": ("primaryDocument", "document", "file_name", "filename"), + "url": ("linkToHtml", "url", "filing_url", "link"), +} + + +def _render_value(value: object, unit: str) -> str: + raw = str(value).replace(",", "") + try: + number = Decimal(raw) + except InvalidOperation: + return str(value) + if unit.lower() == "usd": + return f"${int(number):,}" if number == number.to_integral() else f"${number:,}" + if unit.lower() == "percent": + return f"{number}%" + return str(value) + + +def canonicalize_finance_answer(answer: str | dict) -> str: + """Preserve structured result values in the text evaluated by the guard.""" + parsed = answer + if isinstance(answer, str): + try: + candidate = json.loads(answer) + except (json.JSONDecodeError, TypeError): + return answer + if not isinstance(candidate, dict): + return answer + parsed = candidate + + explanation = str(parsed.get("explanation") or "").strip() + value = parsed.get("value") + parts = [explanation.rstrip(".")] if explanation else [] + if value is not None: + parts.append( + f"The submitted answer is {_render_value(value, str(parsed.get('unit') or ''))}" + ) + evidence_ids = parsed.get("evidence_ids") or [] + if isinstance(evidence_ids, list) and evidence_ids: + parts.append("Cited evidence: " + ", ".join(str(value) for value in evidence_ids)) + if not parts: + return json.dumps(parsed, sort_keys=True) + return ". ".join(parts) + "." + + +@dataclass(frozen=True) +class FilingMetadata: + """Canonical SEC filing metadata.""" + + cik: str | None = None + accession: str | None = None + document: str | None = None + url: str | None = None + ticker: str | None = None + company_name: str | None = None + form: str | None = None + filing_date: str | None = None + report_date: str | None = None + identity_conflict: bool = False + + def source_id(self) -> str | None: + """Return an ID only for a complete canonical filing.""" + if not self.identity_conflict and self.cik and self.accession and self.document: + cik = self.cik.zfill(10) + acc = str(self.accession) + if len(cik) == 10 and cik.isdigit() and len(acc) == 18 and acc.isdigit(): + return f"sec:cik={cik}:accession={self.accession}:doc={self.document}" + return None + + def to_dict(self) -> dict[str, Any]: + return { + "cik": self.cik, + "accession": self.accession, + "document": self.document, + "url": self.url, + "ticker": self.ticker, + "company_name": self.company_name, + "form": self.form, + "filing_date": self.filing_date, + "report_date": self.report_date, + "identity_conflict": self.identity_conflict, + "source_id": self.source_id(), + } + + +@dataclass +class TraceExtraction: + """Evidence and final answer extracted from one rollout.""" + + answer: str | None = None + evidence: list[EvidenceChunk] = field(default_factory=list) + extraction_errors: list[str] = field(default_factory=list) + submit_call_id: str | None = None + has_submit: bool = False + + +@dataclass +class SidecarRow: + """Parsed representation of one rollout line.""" + + raw_line_fingerprint: str + parse_error: str | None = None + trace: TraceExtraction | None = None + + def to_dict(self) -> dict[str, Any]: + d: dict[str, Any] = { + "raw_line_fingerprint": self.raw_line_fingerprint, + } + if self.parse_error: + d["parse_error"] = self.parse_error + if self.trace: + d["answer"] = self.trace.answer + d["has_submit"] = self.trace.has_submit + d["submit_call_id"] = self.trace.submit_call_id + d["evidence"] = [c.to_dict() for c in self.trace.evidence] + d["extraction_errors"] = list(self.trace.extraction_errors) + return d + + +@dataclass(frozen=True) +class FinanceEvaluatorConfig: + """Finance-specific evaluator metadata.""" + + grounding_config: GroundingVerifierConfig = field(default_factory=GroundingVerifierConfig) + environment: str = "finance_sec_search" + routing_model_revision: str | None = None + nli_model_revision: str | None = None + + def to_dict(self) -> dict[str, Any]: + return { + "schema_version": SCHEMA_VERSION, + "algorithm": ALGORITHM_VERSION, + "evidence_basis": EVIDENCE_BASIS, + "limitation": LIMITATION, + "environment": self.environment, + "thresholds": { + "policy": "fixed_fail_closed", + "evidence_excerpt_length": self.grounding_config.evidence_excerpt_length, + }, + "models": { + "routing_model": self.grounding_config.routing_model, + "routing_model_revision": self.routing_model_revision, + "nli_model": self.grounding_config.nli_model, + "nli_model_revision": self.nli_model_revision, + }, + } + + +def _raw_line_fingerprint(raw_line: str) -> str: + """SHA-256 fingerprint (first 16 hex chars) of the raw JSONL line.""" + return hashlib.sha256(raw_line.encode("utf-8")).hexdigest()[:16] + + +def _iter_conversation_items(row: dict[str, Any]) -> Iterator[dict[str, Any]]: + """Yield conversation items from a rollout row. + + Tries common locations in order: + 1. row["output"] (Responses API output items list) + 2. row["response"]["output"] (nested under response) + 3. row["messages"] (chat completions format) + 4. row["conversation"] (alternative trace field) + """ + for source_key in ("output", "response"): + source = row.get(source_key) + if isinstance(source, dict): + items = source.get("output") + if isinstance(items, list): + yield from (i for i in items if isinstance(i, dict)) + return + if isinstance(source, list): + yield from (i for i in source if isinstance(i, dict)) + return + + for key in ("messages", "conversation"): + items = row.get(key) + if isinstance(items, list): + yield from (i for i in items if isinstance(i, dict)) + return + + +def _canonicalize_document(document: str | None) -> str | None: + """Strip a document URL to its basename.""" + if not document: + return None + doc = str(document).strip() + doc = doc.split("?")[0].split("#")[0] + doc = os.path.basename(doc) + return doc if doc else None + + +def _merge_filing(primary: FilingMetadata, fallback: FilingMetadata) -> FilingMetadata: + return FilingMetadata( + cik=primary.cik or fallback.cik, + accession=primary.accession or fallback.accession, + document=primary.document or fallback.document, + url=primary.url or fallback.url, + ticker=primary.ticker or fallback.ticker, + company_name=primary.company_name or fallback.company_name, + form=primary.form or fallback.form, + filing_date=primary.filing_date or fallback.filing_date, + report_date=primary.report_date or fallback.report_date, + identity_conflict=primary.identity_conflict or fallback.identity_conflict, + ) + + +def _extract_filing_records(result: Any) -> list[FilingMetadata]: + """Parse SEC records independently so fields never cross filing boundaries.""" + if isinstance(result, list): + records = [value for value in result if isinstance(value, dict)] + elif isinstance(result, dict): + nested = result.get("filings") + records = ( + [value for value in nested if isinstance(value, dict)] + if isinstance(nested, list) + else [result] + ) + else: + return [] + + filings = [] + for record in records: + values = { + field: next((record[alias] for alias in aliases if record.get(alias)), None) + for field, aliases in _FILING_FIELD_ALIASES.items() + } + url = values["url"] + cik = str(values["cik"]).strip().zfill(10) if values["cik"] else None + accession = _format_accession(str(values["accession"] or "").strip()) + document = _canonicalize_document(values["document"]) + url_metadata = _extract_filing_from_url(url) if url else FilingMetadata() + identity_conflict = any( + left and right and left != right + for left, right in ( + (cik, url_metadata.cik), + (accession, url_metadata.accession), + (document, url_metadata.document), + ) + ) + metadata = FilingMetadata( + cik=cik, + accession=accession, + document=document, + url=url, + ticker=str(values["ticker"]).strip() if values["ticker"] else None, + company_name=(str(values["company_name"]).strip() if values["company_name"] else None), + form=str(values["form"]).strip() if values["form"] else None, + filing_date=(str(values["filing_date"]).strip() if values["filing_date"] else None), + report_date=(str(values["report_date"]).strip() if values["report_date"] else None), + identity_conflict=identity_conflict, + ) + filings.append(_merge_filing(metadata, url_metadata) if url else metadata) + return filings + + +def _extract_filing_metadata(result: Any) -> FilingMetadata: + filings = _extract_filing_records(result) + return filings[0] if filings else FilingMetadata() + + +def _extract_filing_from_url(url: str) -> FilingMetadata: + """Parse canonical filing fields from an SEC EDGAR URL.""" + match = SEC_FILING_URL_RE.search(url) + if not match: + return FilingMetadata(url=url) + + cik = match.group("cik").zfill(10) + accession_raw = match.group("accession") + document = _canonicalize_document(match.group("document")) + + accession = _format_accession(accession_raw) + return FilingMetadata(cik=cik, accession=accession, document=document, url=url) + + +def _format_accession(accession_raw: str) -> str | None: + """Return exactly 18 accession digits, otherwise ``None``.""" + digits = re.sub(r"\D", "", accession_raw) + if len(digits) == 18: + return digits + return None + + +def _parse_tool_arguments(raw: Any) -> dict[str, Any]: + """Parse tool call arguments which may be a JSON string or dict.""" + if isinstance(raw, dict): + return raw + if isinstance(raw, str): + try: + parsed = json.loads(raw) + if isinstance(parsed, dict): + return parsed + except (json.JSONDecodeError, TypeError): + pass + return {} + + +def _parse_tool_result(raw: Any) -> Any: + """Parse a tool result output which may be a JSON string or already parsed.""" + if isinstance(raw, dict | list): + return raw + if isinstance(raw, str): + try: + return json.loads(raw) + except (json.JSONDecodeError, TypeError): + return raw + return raw + + +def _try_parse_json(text: str) -> Any: + """Attempt to parse a string as JSON; return None on failure.""" + try: + return json.loads(text) + except (json.JSONDecodeError, TypeError): + return None + + +def _decode_tool_envelope(raw: Any) -> tuple[str | None, str | None]: + """Strictly decode a function_call_output envelope for pinned Gym. + + Returns ``(payload, error_reason)``. On success, ``payload`` is the + unwrapped content string and ``error_reason`` is ``None``. On failure, + ``payload`` is ``None`` and ``error_reason`` explains why. + + Accepts: + - Pinned Gym envelope: ``{"results": }`` with no top-level + ``error`` key, a string payload that does not begin ``ERROR:`` and + is not a nested JSON ``{"error": ...}`` payload. + - Legacy producer shape: ``{"success": true, "result": }`` + — accepted only when ``success is True`` plus a string ``result``. + + Rejects (returning ``(None, reason)``): + - Top-level agent ``{"error": ...}`` (timeout / exception envelope). + - Missing or unknown envelope keys. + - Non-string ``results`` / ``result`` payloads. + - ``results`` payload beginning ``ERROR:``. + - Nested JSON ``{"error": ...}`` inside ``results`` (time-budget / + no-company error payloads from the resource server). + - Legacy ``success is False`` or missing ``success``. + - All raw strings and other non-dict outputs (unstructured). + """ + parsed = _parse_tool_result(raw) + + if not isinstance(parsed, dict): + return None, "unstructured output" + + if "error" in parsed: + return None, "agent error envelope" + + if "results" in parsed: + results = parsed["results"] + if not isinstance(results, str): + return None, "non-string results payload" + + stripped = results.strip() + if stripped.upper().startswith("ERROR:"): + return None, "ERROR: payload" + + nested = _try_parse_json(results) + if isinstance(nested, dict) and "error" in nested: + return None, "nested JSON error payload" + + return results, None + + if "success" in parsed: + if parsed["success"] is not True: + return None, "legacy shape with success=False" + result = parsed.get("result") + if not isinstance(result, str): + return None, "legacy shape with non-string result" + return result, None + + return None, "unknown envelope" + + +def _is_parse_success(payload: str, key: str) -> bool: + """Check whether a parse_html_page results payload indicates success. + + In pinned Gym, a successful ``parse_html_page`` returns a results + string whose lines include the exact marker (the + ``_save_tool_output`` line):: + + SUCCESS: The result has been saved to the data storage under the key: {key}. + + Failed parses return arbitrary ``str(e)`` text without the marker. + This function checks for the exact marker on any line, bound to the + expected ``key``, so that the ``WARNING:`` overwrite case (which + still contains the exact line) is accepted while abbreviated, + unrelated, or wrong-key ``SUCCESS:`` strings are rejected. + """ + stripped = payload.strip() + if not stripped: + return False + expected = f"SUCCESS: The result has been saved to the data storage under the key: {key}." + for line in stripped.splitlines(): + if line.strip() == expected: + return True + return False + + +def _call_id(item: dict[str, Any]) -> str: + return item.get("call_id") or item.get("id") or item.get("tool_call_id") or "" + + +def _tool_output(item: dict[str, Any]) -> Any: + return item.get("output", item.get("result", "")) + + +def _has_filing_data(filing: FilingMetadata) -> bool: + return bool(filing.cik or filing.accession or filing.url) + + +def _record_search_output( + call_id: str, + raw_output: Any, + filings_by_call: dict[str, FilingMetadata], + filings_by_url: dict[str, FilingMetadata], +) -> None: + existing = filings_by_call.get(call_id, FilingMetadata()) + payload, _ = _decode_tool_envelope(raw_output) + decoded = _try_parse_json(payload) if payload is not None else None + filings = _extract_filing_records(decoded) + if filings and _has_filing_data(filings[0]): + filings_by_call[call_id] = _merge_filing(filings[0], existing) + for filing in filings: + if filing.url: + filings_by_url[filing.url] = filing + + +def _record_parse_output( + call_id: str, + raw_output: Any, + parse_calls: dict[str, tuple[str, str]], + urls_by_key: dict[str, str], +) -> None: + key, url = parse_calls[call_id] + payload, _ = _decode_tool_envelope(raw_output) + if payload is not None and _is_parse_success(payload, key): + urls_by_key[key] = url + + +def _build_key_to_filing_map( + items: list[dict[str, Any]], +) -> dict[str, FilingMetadata]: + """Correlate successful parse storage keys with SEC filing metadata.""" + urls_by_key: dict[str, str] = {} + filings_by_call: dict[str, FilingMetadata] = {} + filings_by_url: dict[str, FilingMetadata] = {} + search_calls: set[str] = set() + parse_call_ids: dict[str, tuple[str, str]] = {} + + for item in items: + item_type = item.get("type", "") + name = item.get("name", "") + call_id = _call_id(item) + + if item_type == "function_call" and name in SEC_SEARCH_NAMES: + search_calls.add(call_id) + filing = _extract_filing_metadata(_parse_tool_arguments(item.get("arguments"))) + if _has_filing_data(filing): + filings_by_call[call_id] = filing + continue + + if item_type == "function_call" and name == PARSE_HTML_NAME: + args = _parse_tool_arguments(item.get("arguments")) + key, url = args.get("key", ""), args.get("url", "") + if key and url and call_id: + parse_call_ids[call_id] = (key, url) + continue + + if item_type not in ("function_call_output", "tool_result") or not call_id: + continue + if call_id in search_calls: + _record_search_output(call_id, _tool_output(item), filings_by_call, filings_by_url) + elif call_id in parse_call_ids: + _record_parse_output(call_id, _tool_output(item), parse_call_ids, urls_by_key) + + key_to_filing: dict[str, FilingMetadata] = {} + for key, url in urls_by_key.items(): + filing = filings_by_url.get(url) + filing = ( + _merge_filing(filing, _extract_filing_from_url(filing.url)) + if filing and filing.url + else _extract_filing_from_url(url) + ) + key_to_filing[key] = filing + + return key_to_filing + + +def _extract_storage_keys_from_text(text: str) -> list[str]: + """Extract ``{{key}}`` storage key references from text.""" + return [m.strip() for m in STORAGE_KEY_RE.findall(text)] + + +def _build_evidence_chunk( + chunk_id: str, + text: str, + keys: list[str], + key_to_filing: dict[str, FilingMetadata], + tool_call_id: str | None, + tool_result_id: str | None, + char_range: tuple[int, int] | None = None, +) -> EvidenceChunk: + """Build a chunk, allowing attribution only for one canonical source.""" + filings = [key_to_filing[key] for key in keys if key in key_to_filing] + common = { + "chunk_id": chunk_id, + "text": text, + "tool_call_id": tool_call_id, + "tool_result_id": tool_result_id, + "char_range": char_range, + } + + if not filings: + return EvidenceChunk( + source_id=None, + source_ids=(), + attribution_state="unavailable", + **common, + ) + + if len(keys) == 1 and len(filings) == 1: + filing = filings[0] + sid = filing.source_id() + if sid: + return EvidenceChunk( + source_id=sid, + source_ids=(), + sec_url=filing.url, + sec_cik=filing.cik, + sec_accession=filing.accession, + sec_document=filing.document, + sec_ticker=filing.ticker, + sec_company_name=filing.company_name, + sec_form=filing.form, + sec_filing_date=filing.filing_date, + sec_report_date=filing.report_date, + storage_keys=tuple(keys), + attribution_state="available", + **common, + ) + return EvidenceChunk( + source_id=None, + source_ids=(), + sec_url=filing.url, + storage_keys=tuple(keys), + attribution_state="unavailable", + **common, + ) + + source_ids = tuple(f.source_id() for f in filings if f.source_id() is not None) + return EvidenceChunk( + source_id=None, + source_ids=source_ids, + sec_url=None, + storage_keys=tuple(keys), + attribution_state="unavailable", + **common, + ) + + +def _collect_trace_items( + items: Sequence[dict[str, Any]], +) -> tuple[dict[str, Any], list[dict[str, Any]], str | None, str | None]: + outputs: dict[str, Any] = {} + retrievals: list[dict[str, Any]] = [] + submit_call_id = answer = None + + for item in items: + item_type, name = item.get("type", ""), item.get("name", "") + call_id = _call_id(item) + if item_type in ("function_call_output", "tool_result"): + if call_id and _tool_output(item) is not None: + outputs[call_id] = _tool_output(item) + elif item_type == "function_call" and name == RETRIEVE_INFO_NAME: + retrievals.append(item) + elif item_type == "function_call" and name == SUBMIT_FINAL_NAME: + final_result = _parse_tool_arguments(item.get("arguments")).get("final_result", "") + if isinstance(final_result, str) and final_result: + submit_call_id, answer = call_id, final_result + return outputs, retrievals, submit_call_id, answer + + +def extract_trace(row: dict[str, Any]) -> TraceExtraction: + """Extract answer and evidence from a single rollout row.""" + items = list(_iter_conversation_items(row)) + extraction = TraceExtraction() + key_to_filing = _build_key_to_filing_map(items) + results, retrieve_calls, submit_call_id, answer = _collect_trace_items(items) + + if answer is not None: + extraction.answer = answer + extraction.submit_call_id = submit_call_id + extraction.has_submit = True + else: + extraction.extraction_errors.append("No valid submit_final_result call found in trace") + + for idx, call in enumerate(retrieve_calls): + call_id = _call_id(call) or f"retrieve_{idx}" + result_raw = results.get(call_id, "") + if not result_raw: + extraction.extraction_errors.append( + f"retrieve_information call {call_id}: no result output" + ) + continue + + text, error_reason = _decode_tool_envelope(result_raw) + if text is None: + extraction.extraction_errors.append( + f"retrieve_information call {call_id}: {error_reason}" + ) + continue + + if not text.strip(): + extraction.extraction_errors.append( + f"retrieve_information call {call_id}: empty result" + ) + continue + + prompt = _parse_tool_arguments(call.get("arguments")).get("prompt", "") + ranges = _parse_tool_arguments(call.get("arguments")).get("input_character_ranges") + char_range = None + if isinstance(ranges, list) and len(ranges) == 1 and isinstance(ranges[0], dict): + start, end = ranges[0].get("start"), ranges[0].get("end") + if isinstance(start, int) and isinstance(end, int): + char_range = (start, end) + chunk = _build_evidence_chunk( + chunk_id=f"ev:{call_id}", + text=text, + keys=_extract_storage_keys_from_text(prompt), + key_to_filing=key_to_filing, + tool_call_id=call_id, + tool_result_id=call_id, + char_range=char_range, + ) + extraction.evidence.append(chunk) + + return extraction + + +def parse_rollout_line(raw_line: str) -> SidecarRow: + """Parse a single JSONL line and return a SidecarRow. + + Every valid or malformed JSONL line yields exactly one SidecarRow. + """ + fingerprint = _raw_line_fingerprint(raw_line) + stripped = raw_line.strip() + if not stripped: + return SidecarRow( + raw_line_fingerprint=fingerprint, + parse_error="empty line", + ) + + try: + row = json.loads(stripped) + except json.JSONDecodeError as exc: + return SidecarRow( + raw_line_fingerprint=fingerprint, + parse_error=f"JSONDecodeError: {exc!s}", + ) + + if not isinstance(row, dict): + return SidecarRow( + raw_line_fingerprint=fingerprint, + parse_error=f"expected JSON object, got {type(row).__name__}", + ) + + trace = extract_trace(row) + return SidecarRow( + raw_line_fingerprint=fingerprint, + trace=trace, + ) + + +def _response_id(row: dict[str, Any]) -> str | None: + """Extract response ID from a rollout row.""" + for key in ("response_id", "id", "response_id_str"): + val = row.get(key) + if isinstance(val, str) and val: + return val + resp = row.get("response") + if isinstance(resp, dict): + rid = resp.get("id") + if isinstance(rid, str) and rid: + return rid + return None + + +def _row_uuid(row: dict[str, Any]) -> str | None: + """Extract UUID from a rollout row.""" + for key in ("uuid", "task_uuid", "sample_id"): + val = row.get(key) + if isinstance(val, str) and val: + return val + return None + + +def _row_indices(row: dict[str, Any]) -> tuple[int | None, int | None]: + """Extract (_ng_task_index, _ng_rollout_index) from a rollout row.""" + task_idx = row.get("_ng_task_index") + rollout_idx = row.get("_ng_rollout_index") + return ( + int(task_idx) if isinstance(task_idx, int | float) else None, + int(rollout_idx) if isinstance(rollout_idx, int | float) else None, + ) + + +def _row_question(row: dict[str, Any]) -> str | None: + """Extract the user question from native Responses API inputs.""" + params = row.get("responses_create_params") + if not isinstance(params, dict): + return None + inputs = params.get("input") + if not isinstance(inputs, list): + return None + for item in reversed(inputs): + if not isinstance(item, dict) or item.get("role") != "user": + continue + content = item.get("content") + if isinstance(content, str): + marker = re.search(r"(?:^|\n)Question:\s*", content, re.IGNORECASE) + return content[marker.end() :].strip() if marker else content.strip() + return None + + +def _row_fingerprint(row: dict[str, Any]) -> str: + """Deterministic fingerprint of the rollout row content.""" + raw = json.dumps(row, sort_keys=True, default=str) + return hashlib.sha256(raw.encode("utf-8")).hexdigest()[:16] + + +def _evaluation_uuid( + fingerprint: str, + seed: int, + line_number: int, + config: FinanceEvaluatorConfig, +) -> str: + """Deterministic evaluation UUID from raw-line fingerprint + seed + line number + algorithm/config. + + Uses the raw-line fingerprint (SHA-256 of the raw JSONL line) rather + than the parsed dict fingerprint, avoiding collisions on malformed + lines that all parse to ``{}``. The physical line number is included + so duplicate identical rows still get distinguishable IDs by their + position in the input file. The config digest captures the + algorithm version and policy settings so the same row evaluated under + different configurations yields a different UUID. + """ + config_digest = hashlib.sha256( + json.dumps(config.to_dict(), sort_keys=True, default=str).encode("utf-8") + ).hexdigest()[:16] + raw = f"eval:{fingerprint}:{seed}:{line_number}:{ALGORITHM_VERSION}:{config_digest}" + return hashlib.sha256(raw.encode("utf-8")).hexdigest() + + +def _unavailable(reason: str, errors: Sequence[str]) -> dict[str, Any]: + return { + "status": "unavailable", + "reason": reason, + "verdicts": [], + "errors": list(errors), + } + + +def _parse_failure(output: dict[str, Any], message: str, reason: str) -> dict[str, Any]: + output.update( + parse_error=message, + verdict=_unavailable(reason, [message]), + evidence=[], + extraction_errors=[message], + ) + return output + + +def evaluate_row( + raw_line: str, + row: dict[str, Any], + seed: int, + evaluator: GroundingVerifierEvaluator, + config: FinanceEvaluatorConfig, + line_number: int = 0, +) -> dict[str, Any]: + """Evaluate a single rollout row and produce a complete sidecar dict.""" + sidecar = parse_rollout_line(raw_line) + trace = sidecar.trace + + task_idx, rollout_idx = _row_indices(row) + output: dict[str, Any] = { + **config.to_dict(), + "seed": seed, + "line_number": line_number, + "uuid": _row_uuid(row), + "_ng_task_index": task_idx, + "_ng_rollout_index": rollout_idx, + "response_id": _response_id(row), + "fingerprint": _row_fingerprint(row), + "raw_line_fingerprint": sidecar.raw_line_fingerprint, + "evaluation_uuid": _evaluation_uuid( + sidecar.raw_line_fingerprint, seed, line_number, config + ), + } + + if sidecar.parse_error: + return _parse_failure(output, sidecar.parse_error, "parse_error") + + if trace is None: + return _parse_failure(output, "trace extraction returned None", "trace_extraction_failed") + + evidence = trace.evidence + extraction_errors = list(trace.extraction_errors) + question = _row_question(row) + + evaluated_answer = ( + canonicalize_finance_answer(trace.answer) if trace.answer is not None else None + ) + output["answer"] = trace.answer + output["evaluated_answer"] = evaluated_answer + output["question"] = question + output["has_submit"] = trace.has_submit + output["submit_call_id"] = trace.submit_call_id + output["evidence"] = [c.to_dict() for c in evidence] + output["extraction_errors"] = extraction_errors + + if not trace.has_submit or trace.answer is None: + decision = _unavailable("no_submit_final_result", extraction_errors) + else: + assert evaluated_answer is not None + decision_obj = ( + evaluate_finance_support( + trace.answer, question, evidence, evaluator.verify_against_premise + ) + if question + else None + ) + if decision_obj is None: + decision_obj = evaluator.evaluate(evaluated_answer, evidence) + decision = decision_obj.to_dict() + if extraction_errors: + decision["status"] = "unavailable" + decision["reason"] = "trace_extraction_errors" + decision["errors"] = extraction_errors + list(decision.get("errors", [])) + + output["verdict"] = decision + return output + + +def evaluate_seed( + input_path: str, + output_path: str, + seed: int, + evaluator: GroundingVerifierEvaluator, + config: FinanceEvaluatorConfig, +) -> int: + """Evaluate one rollout file and atomically replace its sidecar.""" + done_path = output_path + ".done" + + input_done = input_path + ".done" + if not os.path.isfile(input_path): + raise FileNotFoundError(f"Input file does not exist: {input_path}") + if not os.path.isfile(input_done): + raise RuntimeError(f"Input completion marker not found: {input_done}") + + if os.path.exists(done_path): + os.remove(done_path) + + out_dir = os.path.dirname(output_path) + os.makedirs(out_dir or ".", exist_ok=True) + + count = 0 + fd, tmp_path = tempfile.mkstemp( + dir=out_dir or ".", + prefix=".pg_tmp_", + suffix=".jsonl", + ) + try: + with ( + os.fdopen(fd, "w", encoding="utf-8") as out_f, + open(input_path, encoding="utf-8") as in_f, + ): + for raw_line in in_f: + parsed = _try_parse_json(raw_line.strip()) + row = parsed if isinstance(parsed, dict) else {} + result = evaluate_row(raw_line, row, seed, evaluator, config, count) + out_f.write(json.dumps(result, ensure_ascii=False, default=str) + "\n") + count += 1 + + os.replace(tmp_path, output_path) + except Exception: + if os.path.exists(tmp_path): + os.remove(tmp_path) + raise + + Path(done_path).touch() + return count + + +def sidecar_filename(seed: int) -> str: + """Return the sidecar output filename for a given seed.""" + return f"grounding-verifier-rs{seed}.jsonl" + + +def merged_filename(seed: int) -> str: + """Return the merged rollout filename for a given seed.""" + return f"output-rs{seed}.jsonl" + + +def done_marker(seed: int) -> str: + """Return the completion marker filename for a given seed.""" + return f"{merged_filename(seed)}.done" + + +def build_evaluator_from_config( + config: FinanceEvaluatorConfig, +) -> GroundingVerifierEvaluator: + """Build a GroundingVerifierEvaluator with lazy HF model collaborators. + + Model loading is deferred; this function constructs the objects but + does not download or load any model weights. + """ + from nvflow.grounding_verifier.decomposer import RuleBasedDecomposer + from nvflow.grounding_verifier.embedder import HFEmbedder + from nvflow.grounding_verifier.nli import HFNLI + from nvflow.grounding_verifier.router import EmbeddingSourceRouter + + verifier_config = config.grounding_config + embedder = HFEmbedder( + model_id=verifier_config.routing_model, + revision=config.routing_model_revision, + ) + nli_scorer = HFNLI( + model_id=verifier_config.nli_model, + revision=config.nli_model_revision, + ) + return GroundingVerifierEvaluator( + decomposer=RuleBasedDecomposer(), + router=EmbeddingSourceRouter(embedder), + nli_scorer=nli_scorer, + config=verifier_config, + ) + + +def main(argv: Sequence[str] | None = None) -> int: + """CLI entry point for GroundingVerifier finance evaluation.""" + parser = argparse.ArgumentParser( + description="Run GroundingVerifier evaluation on a finance rollout file." + ) + parser.add_argument("--input_file", required=True) + parser.add_argument("--output_file", required=True) + parser.add_argument("--seed", type=int, required=True) + parser.add_argument("--environment", default="finance_sec_search") + parser.add_argument( + "--routing_model", + default="sentence-transformers/all-MiniLM-L6-v2", + ) + parser.add_argument( + "--nli_model", + default="MoritzLaurer/DeBERTa-v3-base-mnli-fever-anli", + ) + parser.add_argument("--routing_model_revision", default=None) + parser.add_argument("--nli_model_revision", default=None) + parser.add_argument("--evidence_excerpt_length", type=int, default=500) + + args = parser.parse_args(argv) + + verifier_config = GroundingVerifierConfig( + evidence_excerpt_length=args.evidence_excerpt_length, + routing_model=args.routing_model, + nli_model=args.nli_model, + ) + finance_config = FinanceEvaluatorConfig( + grounding_config=verifier_config, + environment=args.environment, + routing_model_revision=args.routing_model_revision, + nli_model_revision=args.nli_model_revision, + ) + + evaluator = build_evaluator_from_config(finance_config) + + count = evaluate_seed( + input_path=args.input_file, + output_path=args.output_file, + seed=args.seed, + evaluator=evaluator, + config=finance_config, + ) + print(f"GroundingVerifier: {count} rows written to {args.output_file}") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/nvflow/recipes/finance/workflows/grpo/base.yaml b/nvflow/recipes/finance/workflows/grpo/base.yaml index cdf0e87..2ecd589 100644 --- a/nvflow/recipes/finance/workflows/grpo/base.yaml +++ b/nvflow/recipes/finance/workflows/grpo/base.yaml @@ -98,6 +98,7 @@ pipeline_stages: - prepare_data # Step 4: Run ng_prepare_data (CPU) - prefetch_cache # Pre-warm SEC metadata cache (CPU, optional, no step-N) - collect_rollouts # Step 5: Rollout collection + reward profiling + filter + # - evaluate_grounding # GroundingVerifier evaluation (opt-in; depends on collect_rollouts) # - compute_rewards # Step 6: Re-judge rollouts with different judge (optional) - train_validation_split # Step 7: Final train/val split on reward-filtered data - training # Step 8: GRPO training with NeMo-Gym environment @@ -126,6 +127,7 @@ directories: cache-finance-sec-search: ${base_output_dir}/cache/finance_sec_search # Model-specific stages (rollouts, training, eval). step-5-collect-rollouts: ${model_output_dir}/step-5-collect-rollouts + grounding-verifier-eval: ${model_output_dir}/grounding-verifier-eval step-6-compute-rewards: ${model_output_dir}/step-6-compute-rewards step-7-train-validation-split: ${model_output_dir}/step-7-train-validation-split step-8-training: ${model_output_dir}/step-8-training @@ -532,6 +534,48 @@ stages: filter: min_reward_std: 1e-6 # remove questions with zero reward variance (no GRPO gradient) + # -------------------------------------------------------------------------- + # GroundingVerifier Evaluation [OPT-IN] + # -------------------------------------------------------------------------- + # Runs GroundingVerifier on the merged rollout + # files produced by collect_rollouts. Produces sidecar JSONL files + # (grounding-verifier-rs.jsonl) with per-row allow/block/unavailable + # verdicts, evidence metadata, and extraction errors. + # + # CPU-only: attribution/value checks plus all-MiniLM-L6-v2 routing and + # DeBERTa NLI scoring (both lazy-loaded). + # Depends only on collect_rollouts. No downstream dependencies. + # + # Input: ${directories.step-5-collect-rollouts}/{env}/rollout/output-rs.jsonl + # Marker: ${directories.step-5-collect-rollouts}/{env}/rollout/output-rs.jsonl.done + # Output: ${directories.grounding-verifier-eval}/{env}/grounding-verifier-rs.jsonl + # Marker: ${directories.grounding-verifier-eval}/{env}/grounding-verifier-rs.jsonl.done + # + # To enable, uncomment "# - evaluate_grounding" in pipeline_stages above. + evaluate_grounding: + output_dir: ${directories.grounding-verifier-eval} + rollouts_dir: ${directories.step-5-collect-rollouts} + environments: ${environments} + container: "nemo-skills" + installation_command: "true" + num_gpus: 0 + # Pinned GroundingVerifier models, pre-staged under the standard /hf_models mount. + # See docs/recipes/finance/grounding-verifier/README.md#air-gapped-setup. + routing_model: "/hf_models/sentence-transformers/all-MiniLM-L6-v2" + nli_model: "/hf_models/MoritzLaurer/DeBERTa-v3-base-mnli-fever-anli" + routing_model_revision: "1110a243fdf4706b3f48f1d95db1a4f5529b4d41" + nli_model_revision: "6f5cf0a2b59cabb106aca4c287eed12e357e90eb" + # Fixed conservative policy: contradiction, no_source, conflation, and + # entity/metric/value mismatch block. Neutral requires direct deterministic + # support to allow; model/trace errors are unavailable. + # No configurable block_on_* toggles — the policy is fixed by design. + evidence_excerpt_length: 500 + # Seeds dynamically inherited from collect_rollouts rollout settings. + starting_seed: ${stages.collect_rollouts.rollout.starting_seed} + num_random_seeds: ${stages.collect_rollouts.rollout.num_random_seeds} + dependencies: + - collect_rollouts + # -------------------------------------------------------------------------- # Stage 6: Compute Rewards (re-judge rollouts) [OPTIONAL] # -------------------------------------------------------------------------- diff --git a/nvflow/recipes/finance/workflows/grpo/feature_gates/grounding_airgap.yaml b/nvflow/recipes/finance/workflows/grpo/feature_gates/grounding_airgap.yaml new file mode 100644 index 0000000..8e540fd --- /dev/null +++ b/nvflow/recipes/finance/workflows/grpo/feature_gates/grounding_airgap.yaml @@ -0,0 +1,39 @@ +# Native finance acceptance profile for the GroundingVerifier air-gap contract. +# +# This profile deliberately stops after rollout verification. It reuses the +# prepared demo finance_sec_search input and SEC cache, but writes rollouts and +# sidecars to a dedicated feature-gates directory. +# +# Usage: +# uv run nflow run-all \ +# --config nvflow/recipes/finance/workflows/grpo/feature_gates/grounding_airgap.yaml \ +# -e finance_sec_search + +_base_: ../qwen3_4b.yaml + +feature_name: grounding_airgap +feature_runs: 3 +model_output_dir: ${base_output_dir}/feature-gates/${feature_name} + +pipeline_stages: + - collect_rollouts + - evaluate_grounding + +stages: + collect_rollouts: + rollout: + max_num_samples: 1 + num_samples_in_parallel: 1 + num_chunks: 1 + num_random_seeds: ${feature_runs} + starting_seed: 0 + rerun_done: true + + evaluate_grounding: + feature_gate: + name: ${feature_name} + required_runs: ${feature_runs} + expected_rows_per_run: 1 + max_unavailable_rate: 0.0 + require_offline: true + model_root: /hf_models diff --git a/tests/conftest.py b/tests/conftest.py index 8278c6b..555ff49 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -14,6 +14,10 @@ # """Pytest configuration and fixtures.""" +import sys +from types import ModuleType +from typing import Any + import pytest @@ -45,3 +49,27 @@ def test_config(): }, }, } + + +@pytest.fixture +def stub_nemo_pipeline_cli(monkeypatch): + """Provide the NeMo pipeline calls used by stage-submission unit tests.""" + submitted: list[dict[str, Any]] = [] + nemo_skills_module = ModuleType("nemo_skills") + pipeline_module = ModuleType("nemo_skills.pipeline") + pipeline_cli_module = ModuleType("nemo_skills.pipeline.cli") + + def run_cmd(**kwargs: Any) -> None: + submitted.append(kwargs) + + def wrap_arguments(command: str) -> str: + return command + + nemo_skills_module.pipeline = pipeline_module + pipeline_module.cli = pipeline_cli_module + pipeline_cli_module.run_cmd = run_cmd + pipeline_cli_module.wrap_arguments = wrap_arguments + monkeypatch.setitem(sys.modules, "nemo_skills", nemo_skills_module) + monkeypatch.setitem(sys.modules, "nemo_skills.pipeline", pipeline_module) + monkeypatch.setitem(sys.modules, "nemo_skills.pipeline.cli", pipeline_cli_module) + return submitted diff --git a/tests/fixtures/grounding_verifier/native_finance_trace.json b/tests/fixtures/grounding_verifier/native_finance_trace.json new file mode 100644 index 0000000..1b62c6b --- /dev/null +++ b/tests/fixtures/grounding_verifier/native_finance_trace.json @@ -0,0 +1,77 @@ +{ + "fixture_provenance": { + "kind": "captured_nvflow_nemo_gym_finance_sec_search", + "source_sha256": "52a99d90119aa18d572c3eca4776c1edd4dff4c65bd79daffc2458b3d328d79b", + "transform": "request metadata and unused SEC search candidates removed; response.output calls, IDs, selected filing, retrieved excerpt, and final answer preserved" + }, + "response": { + "id": "resp_945a7e810739460d8db39018d5f34120", + "model": "Qwen3-4B-Instruct-2507", + "output": [ + { + "arguments": "{\n \"ticker\": \"AAPL\",\n \"form_types\": [\n \"10-K\"\n ],\n \"end_date\": \"2022-12-31\"\n }", + "call_id": "hRMEeuzCMPQ5uqOEl1UJJKBjeaIKNxPL", + "name": "sec_filing_search", + "type": "function_call", + "id": "hRMEeuzCMPQ5uqOEl1UJJKBjeaIKNxPL", + "status": "completed" + }, + { + "call_id": "hRMEeuzCMPQ5uqOEl1UJJKBjeaIKNxPL", + "output": "{\"results\":\"[\\n {\\n \\\"ticker\\\": \\\"AAPL\\\",\\n \\\"company_name\\\": \\\"Apple Inc.\\\",\\n \\\"form\\\": \\\"10-K\\\",\\n \\\"filing_date\\\": \\\"2022-10-28\\\",\\n \\\"report_date\\\": \\\"2022-09-24\\\",\\n \\\"accession_number\\\": \\\"0000320193-22-000108\\\",\\n \\\"filing_url\\\": \\\"https://www.sec.gov/Archives/edgar/data/320193/000032019322000108/aapl-20220924.htm\\\"\\n }\\n]\"}", + "type": "function_call_output", + "id": null, + "status": null + }, + { + "arguments": "{\"url\": \"https://www.sec.gov/Archives/edgar/data/320193/000032019322000108/aapl-20220924.htm\", \"key\": \"apple_2022_10k\"}", + "call_id": "lS2UJkZ677jZY54zcaWF6W53QEkSr2aM", + "name": "parse_html_page", + "type": "function_call", + "id": "lS2UJkZ677jZY54zcaWF6W53QEkSr2aM", + "status": "completed" + }, + { + "call_id": "lS2UJkZ677jZY54zcaWF6W53QEkSr2aM", + "output": "{\"results\":\"SUCCESS: The result has been saved to the data storage under the key: apple_2022_10k.\\nThe data_storage currently contains the following keys:\\napple_2022_10k\\n\"}", + "type": "function_call_output", + "id": null, + "status": null + }, + { + "arguments": "{\"prompt\": \"Extract the number of full-time equivalent employees from the Human Capital section of the following filing, which is located between characters 30,000 and 45,000: {{apple_2022_10k}}\", \"input_character_ranges\": [{\"key\": \"apple_2022_10k\", \"start\": 30000, \"end\": 45000}]}", + "call_id": "CllsDJXWJnrC1vGmfqVxHQ1incAEUPYw", + "name": "retrieve_information", + "type": "function_call", + "id": "CllsDJXWJnrC1vGmfqVxHQ1incAEUPYw", + "status": "completed" + }, + { + "call_id": "CllsDJXWJnrC1vGmfqVxHQ1incAEUPYw", + "output": "{\"results\":\"The number of full-time equivalent employees is **164,000**. \\n\\nThis information is found in the \\\"Human Capital\\\" section of the document, specifically in the line: \\n\\\"As of September 24, 2022, the Company had approximately 164,000 full-time equivalent employees.\\\"\"}", + "type": "function_call_output", + "id": null, + "status": null + }, + { + "arguments": "{\"final_result\": \"As of September 24, 2022, Apple had 164,000 full-time equivalent employees.\"}", + "call_id": "7Io4gCsJQQyg4C7rPjHkpVwOxcMckcP2", + "name": "submit_final_result", + "type": "function_call", + "id": "7Io4gCsJQQyg4C7rPjHkpVwOxcMckcP2", + "status": "completed" + }, + { + "call_id": "7Io4gCsJQQyg4C7rPjHkpVwOxcMckcP2", + "output": "{\"results\":\"{\\\"success\\\": true, \\\"result\\\": \\\"As of September 24, 2022, Apple had 164,000 full-time equivalent employees.\\\"}\"}", + "type": "function_call_output", + "id": null, + "status": null + } + ] + }, + "expected_answer": "164,000", + "agent_ref": { + "name": "finance_agent" + } +} diff --git a/tests/test_finance_support.py b/tests/test_finance_support.py new file mode 100644 index 0000000..5f9067d --- /dev/null +++ b/tests/test_finance_support.py @@ -0,0 +1,378 @@ +# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +"""Focused regressions for source-bound finance reasoning.""" + +from __future__ import annotations + +import json + +import pytest + +from nvflow.grounding_verifier.types import Decision, EvidenceChunk +from nvflow.recipes.finance.utils.rl.finance_support import evaluate_finance_support + + +def _fact(ticker: str, company: str, metric: str, year: int, value: int) -> EvidenceChunk: + accession = {"AAPL": "000032019324000123", "MSFT": "000095017024087843"}[ticker] + chunk_id = f"{ticker}_{metric.replace(' ', '')}_{year}" + return EvidenceChunk( + chunk_id=chunk_id, + text=f"{company} ({ticker}) reported {metric} of ${value:,} for fiscal year {year}.", + source_id=f"sec:cik=0000000001:accession={accession}:doc={ticker.lower()}.htm", + sec_accession=accession, + sec_document=f"{ticker.lower()}.htm", + sec_cik="0000000001", + sec_ticker=ticker, + sec_company_name=company, + ) + + +def _answer(kind: str, value, citations: list[str], explanation: str) -> str: + return json.dumps( + { + "answer_type": kind, + "value": value, + "unit": "percent" if kind == "calculation" else "ticker", + "evidence_ids": citations, + "explanation": explanation, + } + ) + + +def _allow_semantics(_answer: str, _premise: str) -> Decision: + return Decision(status="allow", reason="semantic_entailment") + + +def _block_semantics(_answer: str, _premise: str) -> Decision: + return Decision(status="block", reason="semantic_contradiction") + + +@pytest.mark.parametrize( + ("value", "citations", "explanation", "status"), + [ + (-10, ["AAPL_netincome_2023", "AAPL_netincome_2024"], "Apple's change was -10%.", "allow"), + (-20, ["AAPL_netincome_2023", "AAPL_netincome_2024"], "Apple's change was -20%.", "block"), + (-10, ["AAPL_netincome_2024"], "Apple's change was -10%.", "block"), + ( + -10, + ["AAPL_netincome_2023", "AAPL_netincome_2024", "fabricated-source"], + "Apple's change was -10%.", + "block", + ), + ( + -10, + ["AAPL_netincome_2023", "AAPL_netincome_2024"], + "For Microsoft, net income changed by -10%.", + "block", + ), + ( + -10, + ["AAPL_netincome_2023", "AAPL_netincome_2024"], + "Apple's change was -10%, and revenue was $1,234,567.", + "block", + ), + ], +) +def test_calculation_is_value_source_and_entity_bound(value, citations, explanation, status): + evidence = [ + _fact("AAPL", "Apple Inc.", "net income", 2023, 100_000_000), + _fact("AAPL", "Apple Inc.", "net income", 2024, 90_000_000), + ] + decision = evaluate_finance_support( + _answer("calculation", value, citations, explanation), + "Using the evidence cards, what was AAPL's year-over-year percent change in net income " + "from fiscal year 2023 to fiscal year 2024?", + evidence, + _block_semantics if "Microsoft" in explanation else _allow_semantics, + ) + assert decision is not None and decision.status == status + + +@pytest.mark.parametrize( + ("value", "citations", "explanation", "status"), + [ + ("AAPL", ["AAPL_netincome_2024", "MSFT_netincome_2024"], "AAPL was highest.", "allow"), + ("MSFT", ["AAPL_netincome_2024", "MSFT_netincome_2024"], "MSFT was highest.", "block"), + ("AAPL", ["AAPL_netincome_2024"], "AAPL was highest.", "block"), + ("AAPL", ["AAPL_netincome_2024", "MSFT_netincome_2024"], "MSFT was highest.", "block"), + ], +) +def test_comparison_recomputes_all_cited_candidates(value, citations, explanation, status): + evidence = [ + _fact("AAPL", "Apple Inc.", "net income", 2024, 100_000_000), + _fact("MSFT", "Microsoft Corp", "net income", 2024, 90_000_000), + ] + decision = evaluate_finance_support( + _answer("comparison", value, citations, explanation), + "Among AAPL and MSFT, which ticker had the highest net income in fiscal year 2024?", + evidence, + _block_semantics if explanation.startswith("MSFT") else _allow_semantics, + ) + assert decision is not None and decision.status == status + + +@pytest.mark.parametrize( + "explanation", + [ + "This gives Microsoft a net income change of -10%.", + "The issuer is Microsoft; the net income change was -10%.", + "Therefore, Microsoft posted a -10% net income change.", + ], +) +def test_calculation_rejects_plain_language_entity_swaps(explanation): + evidence = [ + _fact("AAPL", "Apple Inc.", "net income", 2023, 100_000_000), + _fact("AAPL", "Apple Inc.", "net income", 2024, 90_000_000), + ] + decision = evaluate_finance_support( + _answer( + "calculation", + -10, + ["AAPL_netincome_2023", "AAPL_netincome_2024"], + explanation, + ), + "What was AAPL's year-over-year percent change in net income from fiscal year 2023 " + "to fiscal year 2024?", + evidence, + _block_semantics, + ) + assert decision is not None and decision.status == "block" + assert decision.reason == "semantic_contradiction" + + +@pytest.mark.parametrize( + "explanation", + [ + "Microsoft was the winner.", + "The winner was Microsoft.", + "Microsoft ranks first.", + "The submitted answer is AAPL, although Microsoft won.", + ], +) +def test_comparison_rejects_contradictory_winner_paraphrases(explanation): + evidence = [ + _fact("AAPL", "Apple Inc.", "net income", 2024, 100_000_000), + _fact("MSFT", "Microsoft Corp", "net income", 2024, 90_000_000), + ] + decision = evaluate_finance_support( + _answer( + "comparison", + "AAPL", + ["AAPL_netincome_2024", "MSFT_netincome_2024"], + explanation, + ), + "Among AAPL and MSFT, which ticker had the highest net income in fiscal year 2024?", + evidence, + _block_semantics, + ) + assert decision is not None and decision.status == "block" + assert decision.reason == "semantic_contradiction" + + +@pytest.mark.parametrize( + "explanation", + [ + "Apple was the winner.", + "The winner was Apple.", + "Among AAPL and MSFT, AAPL ranked first.", + "Apple Inc. (AAPL) had the highest net income.", + ], +) +def test_comparison_allows_grounded_winner_paraphrases(explanation): + evidence = [ + _fact("AAPL", "Apple Inc.", "net income", 2024, 100_000_000), + _fact("MSFT", "Microsoft Corp", "net income", 2024, 90_000_000), + ] + decision = evaluate_finance_support( + _answer( + "comparison", + "AAPL", + ["AAPL_netincome_2024", "MSFT_netincome_2024"], + explanation, + ), + "Among AAPL and MSFT, which ticker had the highest net income in fiscal year 2024?", + evidence, + _allow_semantics, + ) + assert decision is not None and decision.status == "allow" + + +def test_comparison_normalizes_displayed_financial_units(): + smaller = _fact("AAPL", "Apple Inc.", "total assets", 2024, 93_020_840) + larger = _fact("MSFT", "Microsoft Corp", "total assets", 2024, 137_012) + evidence = [ + EvidenceChunk( + **{ + **smaller.__dict__, + "text": ( + "Apple Inc. (AAPL) reported total assets of $93,020,840 thousand " + "for fiscal year 2024." + ), + } + ), + EvidenceChunk( + **{ + **larger.__dict__, + "text": ( + "Microsoft Corp (MSFT) reported total assets of $137,012 million " + "for fiscal year 2024." + ), + } + ), + ] + decision = evaluate_finance_support( + _answer( + "comparison", + "MSFT", + [smaller.chunk_id, larger.chunk_id], + "Apple Inc. (AAPL) reported total assets of $93,020,840 thousand for fiscal " + "year 2024. Microsoft Corp (MSFT) reported total assets of $137,012 million " + "for fiscal year 2024. Microsoft Corp (MSFT) had the highest total assets.", + ), + "Among AAPL and MSFT, which ticker had the highest total assets in fiscal year 2024?", + evidence, + _allow_semantics, + ) + assert decision is not None and decision.status == "allow" + + +def test_evidence_scoped_refusal_requires_the_requested_fact_to_be_absent(): + evidence = [_fact("AAPL", "Apple Inc.", "net income", 2024, 100_000_000)] + question = "What was AAPL's free cash flow for fiscal year 2024? Answer from the cards." + safe = _answer("insufficient_evidence", None, [], "Free cash flow was not found in the cards.") + false = _answer("insufficient_evidence", None, [], "Net income was not found in the cards.") + + assert evaluate_finance_support(safe, question, evidence, _allow_semantics).status == "allow" + assert ( + evaluate_finance_support( + false, + "What was AAPL's net income for fiscal year 2024? Answer from the cards.", + evidence, + _allow_semantics, + ).status + == "block" + ) + + +def test_multiple_atomic_facts_can_share_one_retrieval_chunk(): + first = _fact("AAPL", "Apple Inc.", "total assets", 2023, 100_000_000) + evidence = [ + EvidenceChunk( + **{ + **first.__dict__, + "text": ( + "APPLE INC. (AAPL) reported total assets of $100,000,000 for fiscal year 2023. " + "APPLE INC. (AAPL) reported total assets of $110,000,000 for fiscal year 2024." + ), + } + ) + ] + answer = _answer( + "calculation", + 10, + [first.chunk_id], + "APPLE INC. (AAPL) reported total assets of $100,000,000 for fiscal year 2023. " + "APPLE INC. (AAPL) reported total assets of $110,000,000 for fiscal year 2024. " + "The year-over-year change was 10%.", + ) + decision = evaluate_finance_support( + answer, + "What was AAPL's year-over-year percent change in total assets from fiscal year 2023 " + "to fiscal year 2024?", + evidence, + _allow_semantics, + ) + assert decision is not None and decision.status == "allow" + + +def test_all_caps_company_words_are_not_treated_as_ticker_mentions(): + evidence = [ + _fact("AAPL", "APPLE AIR LINES, INC.", "net income", 2023, 100_000_000), + _fact("AAPL", "APPLE AIR LINES, INC.", "net income", 2024, 90_000_000), + ] + answer = _answer( + "calculation", + -10, + ["AAPL_netincome_2023", "AAPL_netincome_2024"], + "APPLE AIR LINES, INC. (AAPL) reported net income of $100,000,000 for fiscal year 2023. " + "APPLE AIR LINES, INC. (AAPL) reported net income of $90,000,000 for fiscal year 2024. " + "The year-over-year change was -10%.", + ) + decision = evaluate_finance_support( + answer, + "What was AAPL's year-over-year percent change in net income from fiscal year 2023 " + "to fiscal year 2024?", + evidence, + _allow_semantics, + ) + assert decision is not None and decision.status == "allow" + + +def test_refusal_target_comes_from_answer_not_coverage_instructions(): + evidence = [_fact("AAPL", "Apple Inc.", "total assets", 2024, 100_000_000)] + answer = _answer( + "insufficient_evidence", + None, + [], + "Using only the provided evidence cards for Apple Inc. (AAPL), free cash flow is not " + "available for fiscal year 2024.", + ) + question = ( + "Using only the evidence cards for Apple Inc. (AAPL), retrieve total assets as a " + "coverage check, then report free cash flow for fiscal year 2024." + ) + decision = evaluate_finance_support(answer, question, evidence, _allow_semantics) + assert decision is not None and decision.status == "allow" + + +def test_closed_world_refusal_accepts_a_single_evidence_card(): + evidence = [_fact("AAPL", "Northstar Ltd.", "revenue", 2024, 100_000_000)] + answer = _answer( + "insufficient_evidence", + None, + [], + "Using only the provided evidence card for Northstar Ltd. (AAPL), free cash flow " + "is not available for fiscal year 2024.", + ) + question = ( + "Using only the evidence card for Northstar Ltd. (AAPL), retrieve revenue as a " + "coverage check, then report free cash flow for fiscal year 2024." + ) + decision = evaluate_finance_support(answer, question, evidence, _allow_semantics) + assert decision is not None and decision.status == "allow" + + +def test_calculation_rejects_a_supported_value_assigned_to_the_wrong_year(): + evidence = [ + _fact("AAPL", "Apple Inc.", "net income", 2023, 100_000_000), + _fact("AAPL", "Apple Inc.", "net income", 2024, 90_000_000), + ] + answer = _answer( + "calculation", + -10, + ["AAPL_netincome_2023", "AAPL_netincome_2024"], + "Apple Inc. (AAPL) reported net income of $100,000,000 for fiscal year 2023. " + "Apple Inc. (AAPL) reported net income of $90,000,000 for fiscal year 2025. " + "The year-over-year change was -10%.", + ) + decision = evaluate_finance_support( + answer, + "What was AAPL's year-over-year percent change in net income from fiscal year 2023 " + "to fiscal year 2024?", + evidence, + _allow_semantics, + ) + assert decision is not None and decision.status == "block" + assert decision.reason == "temporal_conflation" diff --git a/tests/test_grounding_benchmark.py b/tests/test_grounding_benchmark.py new file mode 100644 index 0000000..49a5557 --- /dev/null +++ b/tests/test_grounding_benchmark.py @@ -0,0 +1,235 @@ +# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +from __future__ import annotations + +import json +import os +from collections import Counter +from collections.abc import Sequence + +import pytest + +from nvflow.grounding_verifier.evaluator import GroundingVerifierConfig +from nvflow.grounding_verifier.types import Decision +from nvflow.recipes.finance.utils.rl import grounding_benchmark as benchmark +from nvflow.recipes.finance.utils.rl.grounding_benchmark import ( + ATTACKS, + FACTS, + FILINGS, + NLI_MODEL_REVISION, + ROUTING_MODEL_REVISION, + generate_cases, + run_benchmark, +) +from nvflow.recipes.finance.utils.rl.grounding_verifier import ( + FinanceEvaluatorConfig, + build_evaluator_from_config, + evaluate_seed, +) + + +def _config() -> FinanceEvaluatorConfig: + return FinanceEvaluatorConfig( + grounding_config=GroundingVerifierConfig(), + routing_model_revision="routing-test-revision", + nli_model_revision="nli-test-revision", + ) + + +class _ExpectedEvaluator: + def __init__(self, grounded_answers: set[str]) -> None: + self.grounded_answers = grounded_answers + + def evaluate(self, answer: str, evidence: Sequence) -> Decision: + status = "allow" if answer in self.grounded_answers else "block" + return Decision(status=status, reason="test") + + +class _AlwaysAllowEvaluator: + def evaluate(self, answer: str, evidence: Sequence) -> Decision: + return Decision(status="allow", reason="test") + + +class TestFacts: + def test_entities_and_values(self): + assert {filing.ticker: (filing.net_income, filing.total_assets) for filing in FILINGS} == { + "AMZN": (59_248, 624_894), + "GOOGL": (100_118, 450_256), + "META": (62_360, 276_054), + "TSLA": (7_153, 122_070), + } + + def test_entities_are_held_out(self): + assert {filing.ticker for filing in FILINGS}.isdisjoint({"AAPL", "MSFT", "NVDA"}) + + def test_two_metrics_per_filing(self): + assert len(FACTS) == 8 + assert Counter(fact.filing.ticker for fact in FACTS) == Counter( + {"AMZN": 2, "GOOGL": 2, "META": 2, "TSLA": 2} + ) + + def test_sec_urls_derive_from_accessions(self): + for filing in FILINGS: + assert filing.url.startswith("https://www.sec.gov/Archives/edgar/data/") + assert filing.accession.replace("-", "") in filing.url + + +class TestCases: + @pytest.fixture + def cases(self): + return generate_cases() + + def test_count_and_balance(self, cases): + assert len(cases) == 56 + assert Counter(case.category for case in cases) == Counter( + {"grounded": 8, **dict.fromkeys(ATTACKS, 8)} + ) + + def test_expected_statuses(self, cases): + for case in cases: + expected = "allow" if case.category == "grounded" else "block" + assert case.expected_status == expected + + def test_ids_are_unique_and_deterministic(self, cases): + repeated = generate_cases() + assert [case.case_id for case in cases] == [case.case_id for case in repeated] + assert len({case.case_id for case in cases}) == 56 + + def test_native_call_order(self, cases): + expected = [ + "sec_filing_search", + "parse_html_page", + "retrieve_information", + "submit_final_result", + ] + for case in cases: + output = case.row["response"]["output"] + calls = [item["name"] for item in output if item["type"] == "function_call"] + assert calls == expected + + def test_function_results_pair_with_calls(self, cases): + for case in cases: + output = case.row["response"]["output"] + for call_index, result_index in ((0, 1), (2, 3), (4, 5)): + assert output[call_index]["call_id"] == output[result_index]["call_id"] + + def test_only_submit_changes_for_each_fact(self, cases): + for entity in {case.entity for case in cases}: + for metric in {case.metric for case in cases if case.entity == entity}: + group = [case for case in cases if (case.entity, case.metric) == (entity, metric)] + evidence_traces = [case.row["response"]["output"][:6] for case in group] + assert all(trace == evidence_traces[0] for trace in evidence_traces) + assert len({case.row["response"]["output"][6]["arguments"] for case in group}) == 7 + + @pytest.mark.parametrize("category", ATTACKS) + def test_attacks_change_the_answer(self, cases, category): + for entity in {case.entity for case in cases}: + for metric in {case.metric for case in cases if case.entity == entity}: + group = [case for case in cases if (case.entity, case.metric) == (entity, metric)] + grounded = next(case.answer for case in group if case.category == "grounded") + attacked = next(case.answer for case in group if case.category == category) + assert attacked != grounded + + +class TestEvaluation: + @pytest.fixture + def evaluated(self): + cases = generate_cases() + grounded = {case.answer for case in cases if case.category == "grounded"} + return run_benchmark(cases, _ExpectedEvaluator(grounded), _config()) + + def test_perfect_fake_passes_all_gates(self, evaluated): + results, summary = evaluated + assert len(results) == 56 + assert summary["grounded_acceptance"] == 1.0 + assert summary["attack_rejection"] == 1.0 + assert summary["unavailable_rate"] == 0.0 + assert summary["gates_passed"] is True + + def test_always_allow_fails_attack_gate(self): + _, summary = run_benchmark(generate_cases(), _AlwaysAllowEvaluator(), _config()) + assert summary["attack_rejection"] == 0.0 + assert summary["gates_passed"] is False + + def test_custom_gates_are_applied(self): + gates = {"grounded_acceptance": 1.0, "attack_rejection": 0.0, "unavailable_rate": 0.0} + _, summary = run_benchmark( + generate_cases(), _AlwaysAllowEvaluator(), _config(), gates=gates + ) + assert summary["gates_passed"] is True + assert summary["gates"]["attack_rejection"]["threshold"] == 0.0 + + def test_outputs_are_reproducible_and_path_free(self, evaluated, tmp_path): + results, summary = evaluated + cases = generate_cases() + benchmark._write_outputs(tmp_path, cases, results, summary) + assert {path.name for path in tmp_path.iterdir()} == { + "cases.jsonl", + "results.jsonl", + "summary.json", + } + for path in tmp_path.iterdir(): + text = path.read_text() + assert "/Users/" not in text + assert "/tmp/" not in text + assert json.loads((tmp_path / "summary.json").read_text())["total_cases"] == 56 + + def test_model_revisions_are_recorded(self, evaluated): + _, summary = evaluated + assert summary["models"]["routing_model_revision"] == "routing-test-revision" + assert summary["models"]["nli_model_revision"] == "nli-test-revision" + + def test_optional_performance_metrics(self): + cases = generate_cases() + grounded = {case.answer for case in cases if case.category == "grounded"} + _, summary = run_benchmark( + cases, + _ExpectedEvaluator(grounded), + _config(), + measure_performance=True, + ) + performance = summary["performance"] + assert performance["warm_rows"] == len(cases) - 1 + assert performance["cold_first_row_seconds"] >= 0 + assert performance["warm_rows_per_second"] > 0 + assert performance["process_peak_rss_mib"] > 0 + + +@pytest.mark.skipif( + os.environ.get("GROUNDING_VERIFIER_RUN_MODELS") != "1", + reason="Set GROUNDING_VERIFIER_RUN_MODELS=1 to run the pinned public-model efficacy gate.", +) +def test_pinned_public_models_pass_efficacy_gates(tmp_path): + config = FinanceEvaluatorConfig( + grounding_config=GroundingVerifierConfig(), + routing_model_revision=ROUTING_MODEL_REVISION, + nli_model_revision=NLI_MODEL_REVISION, + ) + cases = generate_cases() + evaluator = build_evaluator_from_config(config) + _, summary = run_benchmark(cases, evaluator, config) + assert summary["grounded_acceptance"] == 1.0 + assert summary["attack_rejection"] == 1.0 + assert summary["unavailable_rate"] == 0.0 + assert summary["gates_passed"] is True + + input_path = tmp_path / "output-rs0.jsonl" + output_path = tmp_path / "grounding-verifier-rs0.jsonl" + input_path.write_text(json.dumps(cases[0].row) + "\n", encoding="utf-8") + (tmp_path / "output-rs0.jsonl.done").touch() + assert evaluate_seed(str(input_path), str(output_path), 0, evaluator, config) == 1 + assert json.loads(output_path.read_text())["verdict"]["status"] == "allow" + assert (tmp_path / "grounding-verifier-rs0.jsonl.done").is_file() diff --git a/tests/test_grounding_feature_gate.py b/tests/test_grounding_feature_gate.py new file mode 100644 index 0000000..7c60b4a --- /dev/null +++ b/tests/test_grounding_feature_gate.py @@ -0,0 +1,182 @@ +# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +"""Tests for repeated native finance GroundingVerifier feature gates.""" + +from __future__ import annotations + +import hashlib +import json +from pathlib import Path + +import pytest + +from nvflow.core.workflow_runner import WorkflowRunner +from nvflow.recipes.finance.stages.rl.evaluate_grounding import EvaluateGroundingStage +from nvflow.recipes.finance.utils.rl.grounding_feature_gate import ( + GroundingFeatureGateError, + validate_grounding_feature, +) + + +def _write_run( + root: Path, + seed: int, + status: str = "allow", + models: dict[str, str] | None = None, +) -> None: + rollout = root / "rollouts/finance_sec_search/rollout" / f"output-rs{seed}.jsonl" + sidecar = root / "sidecars/finance_sec_search" / f"grounding-verifier-rs{seed}.jsonl" + rollout.parent.mkdir(parents=True, exist_ok=True) + sidecar.parent.mkdir(parents=True, exist_ok=True) + raw = json.dumps({"response": {"id": f"response-{seed}"}}) + "\n" + rollout.write_text(raw) + Path(f"{rollout}.done").touch() + sidecar.write_text( + json.dumps( + { + "seed": seed, + "line_number": 0, + "evaluation_uuid": f"evaluation-{seed}", + "raw_line_fingerprint": hashlib.sha256(raw.encode()).hexdigest()[:16], + "verdict": {"status": status}, + "models": models, + } + ) + + "\n" + ) + Path(f"{sidecar}.done").touch() + + +def _validate(root: Path, **overrides): + options = { + "feature": "test_feature", + "environment": "finance_sec_search", + "rollouts_dir": root / "rollouts", + "sidecars_dir": root / "sidecars", + "starting_seed": 0, + "required_runs": 3, + "expected_rows_per_run": 1, + "max_unavailable_rate": 0.0, + } + return validate_grounding_feature(**{**options, **overrides}) + + +def test_repeated_native_runs_pass_with_complete_sidecars(tmp_path): + for seed in range(3): + _write_run(tmp_path, seed, status="block" if seed == 2 else "allow") + summary = _validate(tmp_path) + assert summary["required_runs"] == 3 + assert summary["verdicts"] == {"allow": 2, "block": 1, "unavailable": 0} + + +def test_airgap_gate_requires_local_models_and_offline_flags(tmp_path, monkeypatch): + model_root = tmp_path / "models" + models = { + "routing_model": str(model_root / "routing"), + "nli_model": str(model_root / "nli"), + } + for model_path in models.values(): + path = Path(model_path) + path.mkdir(parents=True) + (path / "config.json").touch() + (path / "model.safetensors").touch() + for name in ("HF_HUB_OFFLINE", "HF_DATASETS_OFFLINE", "TRANSFORMERS_OFFLINE"): + monkeypatch.setenv(name, "1") + for seed in range(3): + _write_run(tmp_path, seed, models=models) + + assert _validate(tmp_path, require_offline=True, model_root=model_root)["passed"] + + monkeypatch.delenv("TRANSFORMERS_OFFLINE") + with pytest.raises(GroundingFeatureGateError, match="TRANSFORMERS_OFFLINE"): + _validate(tmp_path, require_offline=True, model_root=model_root) + + +@pytest.mark.parametrize("failure", ["missing_run", "fingerprint", "unavailable"]) +def test_repeated_native_runs_fail_closed(tmp_path, failure): + for seed in range(3): + _write_run(tmp_path, seed, status="unavailable" if failure == "unavailable" else "allow") + if failure == "missing_run": + Path(f"{tmp_path}/rollouts/finance_sec_search/rollout/output-rs2.jsonl.done").unlink() + elif failure == "fingerprint": + sidecar = tmp_path / "sidecars/finance_sec_search/grounding-verifier-rs2.jsonl" + row = json.loads(sidecar.read_text()) + row["raw_line_fingerprint"] = "wrong" + sidecar.write_text(json.dumps(row) + "\n") + with pytest.raises(GroundingFeatureGateError): + _validate(tmp_path) + + +def test_airgap_profile_runs_three_finance_seeds(): + config = WorkflowRunner( + "nvflow/recipes/finance/workflows/grpo/feature_gates/grounding_airgap.yaml" + ).config + assert config["pipeline_stages"] == ["collect_rollouts", "evaluate_grounding"] + assert config["stages"]["collect_rollouts"]["rollout"]["num_random_seeds"] == 3 + assert config["stages"]["collect_rollouts"]["rollout"]["max_num_samples"] == 1 + assert config["stages"]["evaluate_grounding"]["feature_gate"] == { + "name": "grounding_airgap", + "required_runs": 3, + "expected_rows_per_run": 1, + "max_unavailable_rate": 0.0, + "require_offline": True, + "model_root": "/hf_models", + } + + +def test_stage_submits_gate_after_every_seed(monkeypatch, stub_nemo_pipeline_cli): + import nvflow.lib.rl.helpers as helpers + + submitted = stub_nemo_pipeline_cli + monkeypatch.setattr(helpers, "resolve_environments", lambda config: {"finance_sec_search": {}}) + EvaluateGroundingStage().execute( + { + "rollouts_dir": "/rollouts", + "output_dir": "/sidecars", + "starting_seed": 0, + "num_random_seeds": 3, + "feature_gate": { + "name": "grounding_airgap", + "required_runs": 3, + "expected_rows_per_run": 1, + }, + }, + cluster="test-cluster", + expname="grpo-evaluate-grounding", + run_after=["collect-rollouts"], + ) + assert len(submitted) == 4 + assert submitted[-1]["expname"] == "grpo-evaluate-grounding-finance_sec_search" + assert submitted[-1]["run_after"] == [ + f"grpo-evaluate-grounding-finance_sec_search-seed{seed}" for seed in range(3) + ] + assert "grounding_feature_gate" in submitted[-1]["ctx"] + + +def test_stage_rejects_noncontiguous_feature_gate_seeds(): + with pytest.raises(ValueError, match="contiguous seeds"): + EvaluateGroundingStage().validate_config( + { + "rollouts_dir": "/rollouts", + "output_dir": "/sidecars", + "starting_seed": 0, + "seeds": [0, 2, 4], + "feature_gate": { + "name": "test_feature", + "required_runs": 3, + "expected_rows_per_run": 1, + }, + } + ) diff --git a/tests/test_grounding_verifier.py b/tests/test_grounding_verifier.py new file mode 100644 index 0000000..32c4669 --- /dev/null +++ b/tests/test_grounding_verifier.py @@ -0,0 +1,435 @@ +# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +"""Contract tests for the finance grounding-verification stage.""" + +from __future__ import annotations + +import json +from collections.abc import Sequence +from pathlib import Path + +import pytest +import yaml + +from nvflow.grounding_verifier.decomposer import RuleBasedDecomposer +from nvflow.grounding_verifier.evaluator import GroundingVerifierConfig, GroundingVerifierEvaluator +from nvflow.grounding_verifier.nli import _normalize_nli_label +from nvflow.grounding_verifier.router import EmbeddingSourceRouter +from nvflow.grounding_verifier.types import EvidenceChunk, NLIResult +from nvflow.recipes.finance.stages.rl.evaluate_grounding import ( + EvaluateGroundingStage, + build_evaluate_command, +) +from nvflow.recipes.finance.utils.rl.grounding_verifier import ( + FinanceEvaluatorConfig, + _is_parse_success, + evaluate_row, + evaluate_seed, + parse_rollout_line, +) + +FILING_URL = "https://www.sec.gov/Archives/edgar/data/320193/000032019322000108/aapl-20220924.htm" +ANSWER = "Apple had approximately 164,000 full-time equivalent employees in 2022." +EVIDENCE = ( + "As of September 24, 2022, Apple Inc. had approximately 164,000 full-time equivalent employees." +) +CAPTURED_TRACE = Path(__file__).parent / "fixtures/grounding_verifier/native_finance_trace.json" + + +class _Embedder: + @property + def dimension(self) -> int: + return 8 + + def embed(self, texts: Sequence[str]) -> Sequence[Sequence[float]]: + return [[float((sum(map(ord, text)) + index) % 17) for index in range(8)] for text in texts] + + +class _NLI: + def __init__(self, label: str = "entailment", *, fail: bool = False) -> None: + self.label = label + self.fail = fail + + def score(self, *, premise: str, hypothesis: str) -> NLIResult: + if self.fail: + raise RuntimeError("model failure") + return NLIResult(label=self.label, score=0.99, probabilities=((self.label, 0.99),)) + + +def _evaluator( + label: str = "entailment", + *, + fail: bool = False, +) -> GroundingVerifierEvaluator: + return GroundingVerifierEvaluator( + decomposer=RuleBasedDecomposer(), + router=EmbeddingSourceRouter(_Embedder()), + nli_scorer=_NLI(label, fail=fail), + ) + + +def _config() -> FinanceEvaluatorConfig: + return FinanceEvaluatorConfig( + grounding_config=GroundingVerifierConfig(), + routing_model_revision="routing-revision", + nli_model_revision="nli-revision", + ) + + +def _call(name: str, call_id: str, **arguments) -> dict: + return { + "type": "function_call", + "name": name, + "call_id": call_id, + "arguments": json.dumps(arguments), + } + + +def _result(call_id: str, output) -> dict: + return { + "type": "function_call_output", + "call_id": call_id, + "output": json.dumps(output), + } + + +def _trace( + answer: str = ANSWER, + *, + parse_output: str | None = None, + url: str = FILING_URL, + key: str = "aapl_10k", +) -> dict: + parse_output = parse_output or ( + f"SUCCESS: The result has been saved to the data storage under the key: {key}." + ) + filing = { + "ticker": "AAPL", + "company_name": "Apple Inc.", + "form": "10-K", + "filing_date": "2022-10-28", + "report_date": "2022-09-24", + "accession_number": "0000320193-22-000108", + "filing_url": url, + } + return { + "uuid": "rollout-1", + "response": { + "id": "response-1", + "output": [ + _call("sec_filing_search", "search", ticker="AAPL", form_types=["10-K"]), + _result("search", {"results": json.dumps([filing])}), + _call("parse_html_page", "parse", url=url, key=key), + _result("parse", {"results": parse_output}), + _call( + "retrieve_information", + "retrieve", + prompt=f"Find employees {{{{{key}}}}}", + input_character_ranges=[{"start": 30000, "end": 45000}], + ), + _result("retrieve", {"results": EVIDENCE}), + _call("submit_final_result", "submit", final_result=answer), + ], + }, + } + + +def _decision(answer: str, label: str = "entailment") -> dict: + row = _trace(answer) + return evaluate_row(json.dumps(row), row, 0, _evaluator(label), _config())["verdict"] + + +def test_captured_native_nvflow_trace_to_atomic_sidecar(tmp_path): + row = json.loads(CAPTURED_TRACE.read_text()) + source = tmp_path / "output-rs0.jsonl" + target = tmp_path / "grounding-verifier-rs0.jsonl" + original = json.dumps(row) + "\n" + source.write_text(original, encoding="utf-8") + Path(f"{source}.done").touch() + + assert evaluate_seed(str(source), str(target), 0, _evaluator(), _config()) == 1 + output = json.loads(target.read_text()) + chunk = output["evidence"][0] + assert output["verdict"]["status"] == "allow" + assert output["models"]["routing_model_revision"] == "routing-revision" + assert chunk["source_id"] == ( + "sec:cik=0000320193:accession=000032019322000108:doc=aapl-20220924.htm" + ) + assert row["fixture_provenance"]["source_sha256"] == ( + "52a99d90119aa18d572c3eca4776c1edd4dff4c65bd79daffc2458b3d328d79b" + ) + assert chunk["char_range"] == {"start": 30000, "end": 45000} + assert source.read_text() == original + assert Path(f"{target}.done").is_file() + + +@pytest.mark.parametrize( + ("answer", "reason"), + [ + ("Apple had approximately 165,000 full-time equivalent employees in 2022.", None), + ( + "Microsoft Corporation (MSFT) had approximately 164,000 full-time " + "equivalent employees in 2022.", + "entity_conflation", + ), + ("Apple reported revenue of 164,000 in 2022.", "financial_metric_mismatch"), + ("Apple had approximately 164,000 full-time equivalent employees in 2023.", None), + ( + "Per SEC accession 0000950170-24-087843, Apple had approximately 164,000 " + "full-time equivalent employees in 2022.", + "conflation", + ), + ], +) +def test_fabrications_and_conflations_block(answer, reason): + decision = _decision(answer, "entailment") + assert decision["status"] == "block" + if reason: + assert decision["reason"] == reason + + +@pytest.mark.parametrize(("label", "status"), [("neutral", "block"), ("contradiction", "block")]) +def test_fail_closed_nli_policy(label, status): + assert _decision("Apple's workforce changed in 2022.", label)["status"] == status + + +def test_matching_numeric_fact_can_override_neutral(): + assert _decision(ANSWER, "neutral")["status"] == "allow" + assert _decision(ANSWER.replace("164,000", "165,000"), "neutral")["status"] == "block" + + +def test_neutral_direct_support_requires_entity_metadata(): + evidence = EvidenceChunk( + chunk_id="amazon", + source_id="sec:cik=0001018724:accession=000101872425000004:doc=amzn.htm", + text="Amazon reported net income of $59,248 million in 2024.", + ) + decision = _evaluator("neutral").evaluate( + "Microsoft reported net income of $59,248 million in 2024.", [evidence] + ) + assert decision.status == "block" + + +def test_model_failure_is_unavailable(): + row = _trace() + result = evaluate_row(json.dumps(row), row, 0, _evaluator(fail=True), _config()) + assert result["verdict"]["status"] == "unavailable" + assert result["verdict"]["errors"] + + +@pytest.mark.parametrize("raw", ["", "{broken", "[]"]) +def test_every_input_line_has_a_fail_closed_sidecar_row(raw): + result = evaluate_row(raw, {}, 0, _evaluator(), _config()) + assert result["verdict"]["status"] == "unavailable" + + +def test_incomplete_and_failed_attribution_never_allows(): + failed = _trace(parse_output="HTTPSConnectionPool: timed out") + sidecar = parse_rollout_line(json.dumps(failed)) + assert sidecar.trace is not None + assert sidecar.trace.evidence[0].source_id is None + result = evaluate_row(json.dumps(failed), failed, 0, _evaluator(), _config()) + assert result["verdict"]["status"] == "block" + assert result["verdict"]["reason"] == "no_source" + + +def test_failed_parse_retry_preserves_the_last_verified_source(): + row = _trace() + output = row["response"]["output"] + output[4:4] = [ + _call("parse_html_page", "retry", url=FILING_URL.replace("108", "109"), key="aapl_10k"), + _result("retry", {"results": "connection timed out"}), + ] + trace = parse_rollout_line(json.dumps(row)).trace + assert trace is not None + assert "accession=000032019322000108" in trace.evidence[0].source_id + + +def test_multiple_search_records_never_mix_filing_fields(): + row = _trace() + second_url = FILING_URL.replace("000032019322000108", "000032019322000109") + records = [ + { + "ticker": "AAPL", + "cik": "320193", + "primaryDocument": "aapl-20220924.htm", + "linkToHtml": FILING_URL, + }, + { + "ticker": "AAPL", + "cik": "320193", + "accessionNo": "0000320193-22-000109", + "primaryDocument": "other.htm", + "linkToHtml": second_url, + }, + ] + row["response"]["output"][1] = _result("search", {"results": json.dumps(records)}) + trace = parse_rollout_line(json.dumps(row)).trace + assert trace is not None + source_id = trace.evidence[0].source_id + assert source_id is not None + assert "accession=000032019322000108" in source_id + assert "000032019322000109" not in source_id + + +def test_record_metadata_conflicting_with_its_url_is_unavailable(): + row = _trace() + record = { + "ticker": "AAPL", + "cik": "320193", + "accessionNo": "0000950170-24-087843", + "primaryDocument": "aapl-20220924.htm", + "linkToHtml": FILING_URL, + } + row["response"]["output"][1] = _result("search", {"results": json.dumps([record])}) + trace = parse_rollout_line(json.dumps(row)).trace + assert trace is not None + assert trace.evidence[0].source_id is None + assert trace.evidence[0].attribution_state == "unavailable" + result = evaluate_row(json.dumps(row), row, 0, _evaluator(), _config()) + assert result["verdict"]["status"] == "block" + assert result["verdict"]["reason"] == "no_source" + + +@pytest.mark.parametrize( + ("payload", "expected"), + [ + ( + "SUCCESS: The result has been saved to the data storage under the key: aapl_10k.", + True, + ), + ("SUCCESS: saved under key: aapl_10k.", False), + ( + "SUCCESS: The result has been saved to the data storage under the key: other.", + False, + ), + ("connection timed out", False), + ], +) +def test_parse_success_marker_is_exact_and_key_bound(payload, expected): + assert _is_parse_success(payload, "aapl_10k") is expected + + +def test_sidecar_rerun_is_transactional(tmp_path): + source = tmp_path / "input.jsonl" + target = tmp_path / "output.jsonl" + source.write_text(json.dumps(_trace()) + "\n") + Path(f"{source}.done").touch() + target.write_text("prior output") + Path(f"{target}.done").write_text("stale") + + class _Broken: + def evaluate(self, answer, evidence): + raise RuntimeError("crashed") + + with pytest.raises(RuntimeError, match="crashed"): + evaluate_seed(str(source), str(target), 0, _Broken(), _config()) + assert target.read_text() == "prior output" + assert not Path(f"{target}.done").exists() + + +def test_missing_input_marker_preserves_prior_output(tmp_path): + source = tmp_path / "input.jsonl" + target = tmp_path / "output.jsonl" + source.write_text(json.dumps(_trace()) + "\n") + target.write_text("prior output") + Path(f"{target}.done").write_text("prior marker") + with pytest.raises(RuntimeError, match="completion marker"): + evaluate_seed(str(source), str(target), 0, _evaluator(), _config()) + assert target.read_text() == "prior output" + assert Path(f"{target}.done").read_text() == "prior marker" + + +def test_duplicate_rows_still_receive_unique_evaluation_ids(tmp_path): + source = tmp_path / "input.jsonl" + target = tmp_path / "output.jsonl" + line = json.dumps(_trace()) + source.write_text(f"{line}\n{line}\n") + Path(f"{source}.done").touch() + evaluate_seed(str(source), str(target), 0, _evaluator(), _config()) + rows = [json.loads(line) for line in target.read_text().splitlines()] + assert len({row["evaluation_uuid"] for row in rows}) == 2 + + +@pytest.mark.parametrize( + ("label", "expected"), + [("Entailment", "entailment"), ("NEUTRAL", "neutral"), ("contradiction", "contradiction")], +) +def test_nli_labels_are_validated(label, expected): + assert _normalize_nli_label(label) == expected + + +def test_unknown_nli_label_is_rejected(): + with pytest.raises(ValueError, match="Unknown NLI label"): + _normalize_nli_label("LABEL_0") + + +def test_workflow_is_opt_in_and_uses_pinned_local_models(): + path = Path("nvflow/recipes/finance/workflows/grpo/base.yaml") + data = yaml.safe_load(path.read_text()) + stage = data["stages"]["evaluate_grounding"] + assert "evaluate_grounding" not in data["pipeline_stages"] + assert stage["dependencies"] == ["collect_rollouts"] + assert stage["num_gpus"] == 0 + assert stage["routing_model"] == "/hf_models/sentence-transformers/all-MiniLM-L6-v2" + assert stage["nli_model"] == "/hf_models/MoritzLaurer/DeBERTa-v3-base-mnli-fever-anli" + assert stage["routing_model_revision"] == "1110a243fdf4706b3f48f1d95db1a4f5529b4d41" + assert stage["nli_model_revision"] == "6f5cf0a2b59cabb106aca4c287eed12e357e90eb" + assert "# - evaluate_grounding" in path.read_text() + + +def test_stage_builds_one_cpu_job_per_environment_and_seed(monkeypatch, stub_nemo_pipeline_cli): + import nvflow.lib.rl.helpers as helpers + + submitted = stub_nemo_pipeline_cli + monkeypatch.setattr(helpers, "resolve_environments", lambda config: {"env_a": {}, "env_b": {}}) + EvaluateGroundingStage().execute( + { + "rollouts_dir": "/rollouts", + "output_dir": "/sidecars", + "starting_seed": 0, + "num_random_seeds": 2, + "seeds": [0, 3], + "num_gpus": 0, + }, + cluster="test-cluster", + expname="pg", + run_after=["collect-rollouts"], + ) + assert len(submitted) == 4 + assert all(job["num_gpus"] == 0 for job in submitted) + assert all( + "output-rs" in job["ctx"] and "grounding-verifier-rs" in job["ctx"] for job in submitted + ) + + +def test_stage_command_contains_the_native_paths(): + command = build_evaluate_command( + "/rollouts/output-rs3.jsonl", + "/sidecars/grounding-verifier-rs3.jsonl", + 3, + "finance_sec_search", + ) + assert "nvflow.recipes.finance.utils.rl.grounding_verifier" in command + assert "--routing_model" in command and "--nli_model" in command + + +@pytest.mark.parametrize( + ("rollouts", "output"), + [("/tmp/a", "/tmp/a"), ("/tmp/a", "/tmp/a/out"), ("/tmp/a/out", "/tmp/a")], +) +def test_stage_rejects_overlapping_paths(rollouts, output): + with pytest.raises(ValueError): + EvaluateGroundingStage().validate_config({"rollouts_dir": rollouts, "output_dir": output})