validation_framework.md

Validation framework

Three layers sit before any output is trusted.

1. Automated data-quality validation (src/validate.py)

python -m src.validate checks every company-year and writes only the flagged items to outputs/tables/validation_report.csv (columns: company, fy, check, severity, item, value_1, value_2, detail) — so a reviewer verifies just those rows against the PDFs instead of re-reading every number.

Checks and their severity:

Check	Severity	What it catches
balance_sheet	error	total liabilities + equity ≠ total assets (beyond ±0.20)
subtotal_gt_total	error	a subtotal exceeding its total (current assets > total assets, cash > current assets, debt > total liabilities, etc.)
pnl_chain (EBITDA<EBIT)	error	implies negative D&A
pnl_chain (EBIT>revenue, NPAT>PBT)	review	unusual P&L chain — tax benefit, discontinued ops, large other gains
implied_tax_rate	review	implied tax rate outside 0–45% when PBT > 0
units_scale	error	a figure outside the plausible $m range, or company scales differing >100× (a thousands/millions break)
ratio_range	error/review	margins outside ±60% (error); leverage, gearing, coverage, current ratio outside generous bounds (review)
yoy_jump	review	a headline line item moving more than 5× year-on-year (same sign)
comparative_mismatch	review	where two consecutive reports overlap a year, the as-first-reported figure vs the next report's comparative column differs
comparative_mismatch_deep	review	the CIMIC-only deeper comparative cross-check (working-capital and cash-flow lines — see below)
cashflow_footing	error	CIMIC's operating cash-flow section must foot, and the reconciliation must agree with the face statement's OCF and D&A
wc_cashflow_tie	review	balance-sheet working-capital movement vs the cash-flow "changes in working capital" line diverges materially (see below)

error flags must be zero (they are). review flags are not necessarily wrong — most are genuine restatements; check the report's restatement notes before changing any input.

The comparative cross-check (and restatements)

data/input/financials_comparatives.csv holds each report's prior-year comparative column for nine headline items, for the four companies. The check compares that against the as-first-reported figure in the primary inputs. A mismatch is treated as review, not error: companies restate prior years (Downer and CIMIC especially). The known restatements are listed in EXPECTED_RESTATEMENTS in validate.py; a mismatch outside that set would be a likely keying error and is surfaced for investigation.

Result of the latest run: 0 hard errors; all comparative mismatches fall on documented restatement years (CIMIC FY2019 discontinued-ops re-presentation; Downer FY2019/20/22/23; NRW FY2020/21/22; Monadelphous FY2022/23) — and where no restatement exists, every comparative ties exactly, which confirms the primary extraction has no detectable keying errors in the headline items.

Deeper CIMIC coverage — the working-capital & cash-flow lines behind the credit view

The 9-item headline cross-check does not reach the numbers the credit conclusion actually leans on, so three further checks cover them for CIMIC:

1. Deeper comparative cross-check (comparative_mismatch_deep) — extends the same prior-year-comparative mechanism to trade & contract receivables, contract assets, contract liabilities (advances), inventory/WIP, trade & contract payables and D&A. The comparative values live in financials_comparatives.csv (CIMIC-only deeper columns).
2. Cash-flow reconciliation footing (cashflow_footing, error) — the operating section must foot: profit/PBT + non-cash items + working-capital movements + interest/tax ≈ reported net operating cash flow. The reconciliation lines are in data/input/cimic_cashflow_recon.csv (which also handles the condensed FY2024–25 format), and the check also verifies the reconciliation's net OCF and D&A agree with the face statement.
3. Cross-statement working-capital tie (wc_cashflow_tie, review) — for each working-capital account, the balance-sheet year-on-year movement should reconcile to the corresponding "changes in working capital" line in the cash flow, allowing for acquisitions / disposals, FX and reclassifications. Material divergences are flagged for review, with the FY2020 Thiess disposal and FY2024 Thiess acquisition allowlisted (WC_TIE_EXPECTED) as legitimately breaking the tie.

These numbers, the footing result and the tie result are written per year to outputs/tables/cimic_wc_cashflow_audit.csv (with source_page), so the handful of story-critical figures can be spot-checked against the PDFs.

Deeper-coverage result: 0 footing errors (the operating section foots to the reported OCF for all seven years); 0 deeper comparative mismatches (the deeper working-capital and cash-flow figures tie exactly to the next report's comparative, confirming no keying errors); 6 working-capital-tie review items — FY2020 (×3, Thiess disposal) and FY2024 (Thiess acquisition) are the allowlisted M&A years, and FY2021 / FY2022 payables divergences reflect FX / contract-balance reclassification within CIMIC's large supplier-financed payables.

2. Source verification (manual)

extraction_check.md lists every company-year, the key figures pulled and the source report / page, with a balance-sheet tie-out per year, for spot-checking against the PDFs. Re-dump any statement page with python -m src.locate_statements "<pdf>" <pages...>.

3. Automated tests (tests/test_pipeline.py, run with pytest)

The validation checks are wired in as assertions:

• no hard validation errors (balance sheet, subtotals, EBITDA≥EBIT, units, margins);
• every comparative mismatch is a documented restatement (EXPECTED_RESTATEMENTS) — a new, unexplained mismatch fails the build;
• coverage (full FY2019–FY2025 for all companies; no missing core items);
• derivation identities (EBIT, EBITDA, total debt) and plausible ratio ranges;
• the negative-operating-cash-flow years are exactly the flagged years.

All checks currently pass. Layers 1 and 3 test internal consistency; layer 2 is how transcription accuracy itself is confirmed.