From d3f10e7fb3a5e4c2bbb9f31fcf9e98f1f16b58d1 Mon Sep 17 00:00:00 2001 From: Karl Kauc Date: Sat, 16 May 2026 09:25:39 +0200 Subject: [PATCH 1/2] Phase 4: CLI sh+ps1 pair, delete last bash, Windows CI, docs sweep MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Final phase of the standalone/cross-platform initiative. - tools/fetch-schema.sh DELETED (the last bash plumbing). CI now materialises the cache cross-platform via `python -m fundsxml_schema` (Phase-2 module); the Python venv install moved before it and the duplicate venv step removed. - XSD_Validation/cli/validate.sh rewritten as POSIX sh (no bash-isms): self-resolving 3-step schema (env -> .schema-cache -> curl -L 302 + xmldsig sibling) then xmllint. New validate.ps1 Windows counterpart (xmllint if present, else the built-in .NET System.Xml.Schema). - XML_Signature/cli/sign-verify-xmlsec1.sh -> POSIX sh; new sign-verify.ps1 (forwards to the .NET example on Windows). - XSD_Validation/powershell/Validate-FundsXml.ps1 now self-resolves too (was the last thing pinned to fetch-schema.sh). - GenerateTestKey.java also emits test-signing-key.pem (PKCS#8, pure JDK via the loaded PKCS#12) — closes the Phase-1-deferred openssl/PEM gap so the xmlsec1 / signxml reference examples have a cross-platform key source. Verified the PEM is a valid private key. - CI: new `windows-smoke` job (windows-latest, pwsh) proving Python, Java (mvnw.cmd), .NET, the PowerShell CLI and a DB round-trip all run standalone & self-resolving on Windows. New ubuntu CLI step exercises validate.sh (positive + negative). - Docs swept end-to-end: every FundsXML_Files README, Data_Binding_JSON, Database_Integration, XSD_Validation, CONTRIBUTING, root README, .github templates, CLAUDE.md — all off fetch-schema.sh onto the in-language resolvers / `python -m fundsxml_schema`. Only `.sh` remaining: the two didactic CLI examples, each paired with a `.ps1`. No bash prerequisite anywhere; every example is standalone and cross-platform. Verified locally: validate.sh under dash (cache/env/negative/cold), GenerateTestKey emits a valid PEM key, `python -m fundsxml_schema` caching, ci.yml is well-formed YAML (validate + windows-smoke jobs). Co-Authored-By: Claude Opus 4.7 (1M context) --- .github/EXAMPLE_README_TEMPLATE.md | 5 +- .github/ISSUE_TEMPLATE/bug_report.md | 4 +- .github/PULL_REQUEST_TEMPLATE.md | 5 +- .github/workflows/ci.yml | 119 +++++++++++++++--- CONTRIBUTING.md | 18 +-- Data_Binding_JSON/README.md | 6 +- Database_Integration/README.md | 2 +- FundsXML_Files/4.0.0/positions/README.md | 2 +- FundsXML_Files/4.1.0/positions/README.md | 2 +- FundsXML_Files/4.2.9/documents/README.md | 2 +- FundsXML_Files/4.2.9/regulatory/README.md | 2 +- FundsXML_Files/4.2.9/signed/README.md | 5 +- FundsXML_Files/4.2.9/transactions/README.md | 2 +- FundsXML_Files/README.md | 12 +- README.md | 10 +- XML_Signature/cli/sign-verify-xmlsec1.sh | 44 +++++-- XML_Signature/cli/sign-verify.ps1 | 30 +++++ XML_Signature/java/GenerateTestKey.java | 23 ++++ XSD_Validation/README.md | 21 ++-- XSD_Validation/cli/validate.ps1 | 98 +++++++++++++++ XSD_Validation/cli/validate.sh | 80 ++++++++---- .../powershell/Validate-FundsXml.ps1 | 42 +++++-- tools/fetch-schema.sh | 58 --------- 23 files changed, 432 insertions(+), 160 deletions(-) mode change 100644 => 100755 XML_Signature/cli/sign-verify-xmlsec1.sh create mode 100644 XML_Signature/cli/sign-verify.ps1 create mode 100644 XSD_Validation/cli/validate.ps1 delete mode 100755 tools/fetch-schema.sh diff --git a/.github/EXAMPLE_README_TEMPLATE.md b/.github/EXAMPLE_README_TEMPLATE.md index 0622345..a68db10 100644 --- a/.github/EXAMPLE_README_TEMPLATE.md +++ b/.github/EXAMPLE_README_TEMPLATE.md @@ -20,8 +20,11 @@ Why this exists and when an enterprise integrator would use it. ## Prerequisites - Tooling/runtime versions -- `tools/fetch-schema.sh ` if schema is needed +- The example resolves the XSD itself (env `FUNDSXML_SCHEMA_DIR` → + `.schema-cache/` → official-release download); or + `python -m fundsxml_schema ` to pre-cache for a bare xmllint - Network/proxy note when the official schema URL must be reached + (`$FUNDSXML_SCHEMA_DIR` is the offline escape hatch) ## Run diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md index be44285..8f2d630 100644 --- a/.github/ISSUE_TEMPLATE/bug_report.md +++ b/.github/ISSUE_TEMPLATE/bug_report.md @@ -14,8 +14,8 @@ e.g. 4.2.9, `FundsXML_Files/4.2.9/positions/Mixed-Fund_Positions.xml` (or attach/inline a minimal FundsXML snippet — no real/PII data). **What I ran** -The exact command(s), including the `./mvnw` (or `mvnw.cmd`) invocation for -Java examples, or any `tools/fetch-schema.sh` step. +The exact command(s) — e.g. the `./mvnw`/`mvnw.cmd` invocation (Java), the +venv `pip install -e .` + `python ...` (Python), or `dotnet run` (.NET). **Expected vs. actual** What you expected and what happened (paste the full error / output). diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index 5dc6758..e9ab638 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -15,8 +15,9 @@ - [ ] English only; example is **self-contained** and **heavily commented** (file-header: purpose / run / deps / FundsXML assumptions + what & why). - [ ] FundsXML conventions: no XML namespace; XSD-validated against the - **official released schema** via `tools/fetch-schema.sh`; 4.0.0 has no - `ControlData/Version`; secure XML parsing (DTD/external entities off). + **official released schema** (each example resolves it itself; or + `python -m fundsxml_schema `); 4.0.0 has no `ControlData/Version`; + secure XML parsing (DTD/external entities off). - [ ] New/changed samples are **XSD-valid**; negative fixtures still fail. - [ ] Round-trip examples: proven with `Database_Integration/tools/xml_equiv.py` **and** `xmllint --schema` (complementary checks). diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index c48a2f2..8a36339 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -39,9 +39,21 @@ jobs: with: dotnet-version: "8.0" - - name: Fetch official schemas (4.2.9, 4.1.0, 4.0.0) + # One idiomatic, OS-agnostic Python install (venv + editable pyproject); + # also exposes the `fundsxml_schema` resolver to every Python script. + - name: Python - venv & install (pyproject) run: | - for v in 4.2.9 4.1.0 4.0.0; do tools/fetch-schema.sh "$v"; done + python -m venv .venv + .venv/bin/pip install --quiet --upgrade pip + .venv/bin/pip install --quiet -e . + .venv/bin/python -c "import lxml, saxonche, fundsxml_schema; print('python deps OK')" + + # Materialise the official schemas into .schema-cache/ for the + # xmllint-based steps below — via the in-language resolver module + # (cross-platform), replacing the deleted tools/fetch-schema.sh. + - name: Materialise official schemas (4.2.9, 4.1.0, 4.0.0) + run: | + for v in 4.2.9 4.1.0 4.0.0; do .venv/bin/python -m fundsxml_schema "$v"; done - name: XSD - all positive samples must validate run: | @@ -67,18 +79,6 @@ jobs: fi echo "xsd-invalid correctly rejected" - # --------------------------------------------------------------------- - # Python examples — one idiomatic, OS-agnostic install (venv + editable - # pyproject) replaces the old ad-hoc `pip install lxml`. This also - # exposes the `fundsxml_schema` resolver module to every Python script. - # --------------------------------------------------------------------- - - name: Python - venv & install (pyproject) - run: | - python -m venv .venv - .venv/bin/pip install --quiet --upgrade pip - .venv/bin/pip install --quiet -e . - .venv/bin/python -c "import lxml, saxonche, fundsxml_schema; print('python deps OK')" - - name: Python - XSD validation (in-language schema resolve) run: | set -e @@ -107,6 +107,17 @@ jobs: fi echo ".NET XsdValidate: positive ok, negative correctly rejected" + - name: CLI - validate.sh (standalone POSIX sh) + run: | + set -e + XSD_Validation/cli/validate.sh 4.2.9 \ + FundsXML_Files/4.2.9/positions/Mixed-Fund_Positions.xml + if XSD_Validation/cli/validate.sh 4.2.9 \ + tests/fixtures/invalid/xsd-invalid_Positions.xml; then + echo "::error::xsd-invalid unexpectedly validated (CLI)"; exit 1 + fi + echo "CLI validate.sh: positive ok, negative correctly rejected" + # --------------------------------------------------------------------- # Java examples — built & run via the committed Maven Wrapper. The first # ./mvnw bootstraps Maven itself, then resolves all deps from Central. @@ -292,3 +303,83 @@ jobs: xsltproc XSLT_DataQuality_Checks/Enhanced_Check/FundsXML_CompleteDQReport_HTML.xsl \ FundsXML_Files/4.2.9/positions/Mixed-Fund_Positions.xml > out_enh.html test -s out_enh.html + + # ======================================================================= + # Windows smoke — proves the examples are genuinely cross-platform and + # standalone (the whole point of this work): every stack resolves the XSD + # itself and runs with no bash, no prior tool step, on a clean Windows box. + # A focused subset (Python / Java / .NET XSD + one XSLT + a DB round-trip + + # the PowerShell CLI), not the full bash-heavy matrix. + # ======================================================================= + windows-smoke: + runs-on: windows-latest + defaults: + run: + shell: pwsh + steps: + - uses: actions/checkout@v6 + - uses: actions/setup-java@v5 + with: + distribution: temurin + java-version: "21" + - uses: actions/setup-python@v6 + with: + python-version: "3.11" + - uses: actions/setup-dotnet@v5 + with: + dotnet-version: "8.0" + + - name: Python - venv & install (pyproject) + run: | + python -m venv .venv + .venv\Scripts\python -m pip install --quiet --upgrade pip + .venv\Scripts\pip install --quiet -e . + .venv\Scripts\python -c "import lxml, saxonche, fundsxml_schema; print('python deps OK')" + + - name: Python - XSD validation (downloads schema itself on Windows) + run: | + $src = "FundsXML_Files/4.2.9/positions/Mixed-Fund_Positions.xml" + .venv\Scripts\python XSD_Validation/python/validate.py 4.2.9 $src + if ($LASTEXITCODE -ne 0) { throw "positive failed" } + .venv\Scripts\python XSD_Validation/python/validate.py 4.2.9 tests/fixtures/invalid/xsd-invalid_Positions.xml + if ($LASTEXITCODE -eq 0) { throw "negative unexpectedly validated (Python)" } + Write-Host "Python OK on Windows" + + - name: Java - XSD validate + one XSLT (Maven Wrapper, mvnw.cmd) + run: | + $src = "FundsXML_Files/4.2.9/positions/Mixed-Fund_Positions.xml" + .\mvnw.cmd -q -B -pl XSD_Validation/java compile exec:java "-Dexec.args=4.2.9 $src" + if ($LASTEXITCODE -ne 0) { throw "Java XSD positive failed" } + .\mvnw.cmd -q -B -pl XSD_Validation/java exec:java "-Dexec.args=4.2.9 tests/fixtures/invalid/xsd-invalid_Positions.xml" + if ($LASTEXITCODE -eq 0) { throw "Java negative unexpectedly validated" } + .\mvnw.cmd -q -B -pl XSLT_Transformations/invocation compile exec:java "-Dexec.args=XSLT_Transformations/CSV_Export/positions_csv.xslt $src out_pos.csv" + if (-not (Test-Path out_pos.csv)) { throw "XSLT produced no output" } + Write-Host "Java OK on Windows" + + - name: .NET - XSD validation (downloads schema itself on Windows) + run: | + $src = "FundsXML_Files/4.2.9/positions/Mixed-Fund_Positions.xml" + dotnet run --project XSD_Validation/dotnet -- 4.2.9 $src + if ($LASTEXITCODE -ne 0) { throw ".NET XSD positive failed" } + dotnet run --project XSD_Validation/dotnet -- 4.2.9 tests/fixtures/invalid/xsd-invalid_Positions.xml + if ($LASTEXITCODE -eq 0) { throw ".NET negative unexpectedly validated" } + Write-Host ".NET OK on Windows" + + - name: CLI - validate.ps1 (PowerShell, .NET fallback when no xmllint) + run: | + pwsh XSD_Validation/cli/validate.ps1 4.2.9 FundsXML_Files/4.2.9/positions/Mixed-Fund_Positions.xml + if ($LASTEXITCODE -ne 0) { throw "validate.ps1 positive failed" } + pwsh XSD_Validation/cli/validate.ps1 4.2.9 tests/fixtures/invalid/xsd-invalid_Positions.xml + if ($LASTEXITCODE -eq 0) { throw "validate.ps1 negative unexpectedly validated" } + Write-Host "PowerShell CLI OK on Windows" + + - name: DB round-trip (Python) on Windows + run: | + $fx = "FundsXML_Files/4.2.9/positions/Multi-Fund_Positions.xml" + .venv\Scripts\python Database_Integration/python/import_fundsxml.py fx.db $fx + if ($LASTEXITCODE -ne 0) { throw "import failed" } + .venv\Scripts\python Database_Integration/python/export_fundsxml.py fx.db FUNDSXML_MULTI_1 out.xml + if ($LASTEXITCODE -ne 0) { throw "export failed" } + .venv\Scripts\python Database_Integration/tools/xml_equiv.py $fx out.xml + if ($LASTEXITCODE -ne 0) { throw "round-trip not equivalent" } + Write-Host "DB round-trip OK on Windows" diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 675308d..745f2c4 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -22,9 +22,12 @@ Please read this before opening a pull request. the schema itself requires them). `xsi:noNamespaceSchemaLocation` must be in the XMLSchema-instance namespace or validators reject it. - **Validate against the official released schema**, never a hand-made - catalog: fetch with `tools/fetch-schema.sh ` (it handles GitHub's - 302 redirect and the relative `xmldsig-core-schema.xsd` import that 4.2.9+ - needs). Set sample `xsi:noNamespaceSchemaLocation` to that release URL. + catalog. Every example resolves it itself (`$FUNDSXML_SCHEMA_DIR` → + `.schema-cache/` → official-release download, 302-aware, pulls the + `xmldsig-core-schema.xsd` sibling for 4.2.9+). For an xmllint check, + materialise the cache with `python -m fundsxml_schema ` + (after `pip install -e .`). Set sample `xsi:noNamespaceSchemaLocation` + to that release URL. - **4.0.0 `ControlData` has no `` element** (added in 4.1.0) — never add one to a 4.0.0 sample. - Positions ↔ Assets link by a shared `UniqueID`; `AssetMasterData` is @@ -57,17 +60,18 @@ Run what you changed and confirm it actually works — no "should pass" claims. Java examples build standalone via the committed Maven Wrapper (`./mvnw`, or `mvnw.cmd` on Windows). Python examples install once into a venv from -`pyproject.toml` and resolve the XSD themselves. Neither needs `fetch-tools.sh` -(gone) and Python no longer needs `fetch-schema.sh`. +`pyproject.toml` and resolve the XSD themselves. No `fetch-tools.sh` and no +`fetch-schema.sh` — both are gone; every stack is standalone & cross-platform. ```bash # Python stack (cross-platform; Windows: .venv\Scripts\activate) python -m venv .venv && . .venv/bin/activate && pip install -e . python XSD_Validation/python/validate.py 4.2.9 .xml # self-resolves the XSD -# xmllint still uses the cached schema (CLI stack, removed in a later phase) -tools/fetch-schema.sh 4.2.9 +# xmllint check: materialise the cache cross-platform, then validate +python -m fundsxml_schema 4.2.9 xmllint --noout --schema .schema-cache/4.2.9/FundsXML.xsd .xml +# (or just: XSD_Validation/cli/validate.sh 4.2.9 .xml — self-resolving) # Schematron via the Maven Wrapper (positive sample -> exit 0) ./mvnw -q -pl Schematron_DataQuality_Checks/Basic_Checks/invocation \ diff --git a/Data_Binding_JSON/README.md b/Data_Binding_JSON/README.md index 146fce9..0834862 100644 --- a/Data_Binding_JSON/README.md +++ b/Data_Binding_JSON/README.md @@ -41,9 +41,9 @@ so a full generated model is heavy and brittle to maintain. | Python | `xsdata` | `xsdata --package fundsxml.model .schema-cache/4.2.9/FundsXML.xsd` | | .NET | `xsd.exe` / `XmlSerializer` | `xsd.exe /classes /namespace:FundsXml.Model .schema-cache\4.2.9\FundsXML.xsd` | -All three consume the **official released schema** fetched by -`tools/fetch-schema.sh` (which also pulls the imported `xmldsig-core-schema.xsd` -for 4.2.9). Trade-off: generated models are type-safe but regenerate on every +All three consume the **official released schema**; materialise it with +`python -m fundsxml_schema 4.2.9` (after `pip install -e .` — cross-platform; +also pulls the imported `xmldsig-core-schema.xsd` for 4.2.9). Trade-off: generated models are type-safe but regenerate on every schema bump and produce thousands of classes; the native binding stays small and version-tolerant. Pick per use case. diff --git a/Database_Integration/README.md b/Database_Integration/README.md index ac41bfc..7624649 100644 --- a/Database_Integration/README.md +++ b/Database_Integration/README.md @@ -59,7 +59,7 @@ Each example is run as **import, then export** (the round-trip = both, then compare). `DOC` is the document id the import prints. ```bash -tools/fetch-schema.sh 4.2.9 # XSD for validation +python -m fundsxml_schema 4.2.9 # caches the XSD into .schema-cache/ (run `pip install -e .` once; cross-platform) FX=FundsXML_Files/4.2.9/positions/Multi-Fund_Positions.xml DOC=FUNDSXML_MULTI_1 diff --git a/FundsXML_Files/4.0.0/positions/README.md b/FundsXML_Files/4.0.0/positions/README.md index a9ec1e1..7040a16 100644 --- a/FundsXML_Files/4.0.0/positions/README.md +++ b/FundsXML_Files/4.0.0/positions/README.md @@ -27,7 +27,7 @@ Content-identical to the 4.1.0 example (3 equity positions), but adapted to the ## Validation ```bash -tools/fetch-schema.sh 4.0.0 +python -m fundsxml_schema 4.0.0 # caches the XSD into .schema-cache/ (run `pip install -e .` once; cross-platform) xmllint --noout --schema .schema-cache/4.0.0/FundsXML.xsd \ FundsXML_Files/4.0.0/positions/Equity-Fund_Positions.xml ``` diff --git a/FundsXML_Files/4.1.0/positions/README.md b/FundsXML_Files/4.1.0/positions/README.md index de603a8..fe5d9b6 100644 --- a/FundsXML_Files/4.1.0/positions/README.md +++ b/FundsXML_Files/4.1.0/positions/README.md @@ -24,7 +24,7 @@ EUR 40m NAV. Deliberately small to make the version comparison easy. ## Validation ```bash -tools/fetch-schema.sh 4.1.0 +python -m fundsxml_schema 4.1.0 # caches the XSD into .schema-cache/ (run `pip install -e .` once; cross-platform) xmllint --noout --schema .schema-cache/4.1.0/FundsXML.xsd \ FundsXML_Files/4.1.0/positions/Equity-Fund_Positions.xml ``` diff --git a/FundsXML_Files/4.2.9/documents/README.md b/FundsXML_Files/4.2.9/documents/README.md index c548eac..2d67303 100644 --- a/FundsXML_Files/4.2.9/documents/README.md +++ b/FundsXML_Files/4.2.9/documents/README.md @@ -24,7 +24,7 @@ via `Document/Fund/Identifiers/LEI`. ## Validation ```bash -tools/fetch-schema.sh 4.2.9 +python -m fundsxml_schema 4.2.9 # caches the XSD into .schema-cache/ (run `pip install -e .` once; cross-platform) xmllint --noout --schema .schema-cache/4.2.9/FundsXML.xsd \ FundsXML_Files/4.2.9/documents/Fund_Documents.xml ``` diff --git a/FundsXML_Files/4.2.9/regulatory/README.md b/FundsXML_Files/4.2.9/regulatory/README.md index fcb3f67..7cc799d 100644 --- a/FundsXML_Files/4.2.9/regulatory/README.md +++ b/FundsXML_Files/4.2.9/regulatory/README.md @@ -28,7 +28,7 @@ Mandatory blocks included: ## Validation ```bash -tools/fetch-schema.sh 4.2.9 +python -m fundsxml_schema 4.2.9 # caches the XSD into .schema-cache/ (run `pip install -e .` once; cross-platform) xmllint --noout --schema .schema-cache/4.2.9/FundsXML.xsd \ FundsXML_Files/4.2.9/regulatory/EFT_Regulatory.xml ``` diff --git a/FundsXML_Files/4.2.9/signed/README.md b/FundsXML_Files/4.2.9/signed/README.md index 730ec97..829957f 100644 --- a/FundsXML_Files/4.2.9/signed/README.md +++ b/FundsXML_Files/4.2.9/signed/README.md @@ -13,7 +13,8 @@ `ds:Signature` (namespace `http://www.w3.org/2000/09/xmldsig#`) is the **last optional child** of ``. From release 4.2.9 on, `FundsXML.xsd` imports -`xmldsig-core-schema.xsd` for this (see `tools/fetch-schema.sh`). +`xmldsig-core-schema.xsd` for this — the schema resolvers fetch that sibling +automatically when it is imported (e.g. `python -m fundsxml_schema 4.2.9`). > ⚠️ **Placeholder:** `DigestValue` and `SignatureValue` are schema-valid base64 > strings but **not cryptographically verifiable**. Real signing and @@ -25,7 +26,7 @@ Algorithms used (enveloped signature): C14N 2001-03-15, RSA-SHA256, SHA-256. ## Validation ```bash -tools/fetch-schema.sh 4.2.9 # also fetches xmldsig-core-schema.xsd +python -m fundsxml_schema 4.2.9 # caches the XSD into .schema-cache/ (run `pip install -e .` once; cross-platform) xmllint --noout --schema .schema-cache/4.2.9/FundsXML.xsd \ FundsXML_Files/4.2.9/signed/Signed_Fund_Skeleton.xml ``` diff --git a/FundsXML_Files/4.2.9/transactions/README.md b/FundsXML_Files/4.2.9/transactions/README.md index 7408c5c..62dbd31 100644 --- a/FundsXML_Files/4.2.9/transactions/README.md +++ b/FundsXML_Files/4.2.9/transactions/README.md @@ -26,7 +26,7 @@ records: ## Validation ```bash -tools/fetch-schema.sh 4.2.9 +python -m fundsxml_schema 4.2.9 # caches the XSD into .schema-cache/ (run `pip install -e .` once; cross-platform) xmllint --noout --schema .schema-cache/4.2.9/FundsXML.xsd \ FundsXML_Files/4.2.9/transactions/Fund_Transactions.xml ``` diff --git a/FundsXML_Files/README.md b/FundsXML_Files/README.md index d1860d4..a11e2d8 100644 --- a/FundsXML_Files/README.md +++ b/FundsXML_Files/README.md @@ -122,17 +122,21 @@ Validation always targets the **official release** of the schema: https://github.com/fundsxml/schema/releases/download//FundsXML.xsd ``` -Two enterprise-relevant caveats are handled by the helper `tools/fetch-schema.sh`: +Two enterprise-relevant caveats are handled by every example's in-language +schema resolver (and by the `fundsxml_schema` module shown below): 1. That URL returns an HTTP 302 redirect; simple HTTP clients (libxml2 / - xmllint) do not follow it, so the schema must be fetched first (curl honours - `https_proxy`/`HTTPS_PROXY` for locked-down networks). + xmllint) do not follow it, so the schema must be materialised first. 2. From release 4.2.9 on, `FundsXML.xsd` imports `xmldsig-core-schema.xsd` via a relative path — both files must sit in the same directory. +The resolution order is the same everywhere: `$FUNDSXML_SCHEMA_DIR` (offline / +corporate-network escape hatch) → `.schema-cache/` → download from the +official release. No committed catalog. + ```bash # Fetches FundsXML.xsd (+ xmldsig-core-schema.xsd when needed) into .schema-cache// -tools/fetch-schema.sh 4.2.9 +python -m fundsxml_schema 4.2.9 # caches the XSD into .schema-cache/ (run `pip install -e .` once; cross-platform) ``` ### Validate with xmllint (macOS/Linux) diff --git a/README.md b/README.md index d1340db..6c79f18 100644 --- a/README.md +++ b/README.md @@ -32,7 +32,7 @@ own README, and is exercised by the CI workflow on each push. | Company-internal DQ rules | XSLT 2.0 | [XSLT_DataQuality_Checks/Custom_Internal_Checks/](./XSLT_DataQuality_Checks/) | ✅ | | Factsheet (HTML/PDF) & CSV export | XSLT, XSL-FO/FOP | [XSLT_Transformations/](./XSLT_Transformations/) | ✅ | | Transformation invocation | CLI, Python, Java, .NET, Node | [XSLT_Transformations/invocation/](./XSLT_Transformations/) | ✅ | -| Schema fetch (proxy-aware) | Bash | [tools/fetch-schema.sh](./tools/fetch-schema.sh) | ✅ | +| Schema resolver (env / cache / official download) | Per-language, in-example + [tools/fundsxml_schema.py](./tools/fundsxml_schema.py) | every stack | ✅ | | CI (validate all samples) | GitHub Actions | [.github/workflows/ci.yml](./.github/workflows/) | ✅ | | XQuery analytics (aggregation, top-holdings, look-through) | Saxon CLI/Java, Python, BaseX | [XQuery_Examples/](./XQuery_Examples/) | ✅ | | XML signature sign/verify | Apache Santuario (Java), .NET, xmlsec1, signxml | [XML_Signature/](./XML_Signature/) | ✅ | @@ -50,9 +50,11 @@ fundsxml_examples/ │ # wrapper: builds all Java │ # examples standalone (deps from │ # Maven Central), no preinstall -├── tools/ # fetch-schema.sh (proxy-aware XSD -│ # fetcher; Java examples also -│ # resolve the XSD themselves) +├── pyproject.toml # Python deps + the +│ # fundsxml_schema resolver module +├── tools/fundsxml_schema.py # shared in-language XSD resolver +│ # (env / cache / official URL); +│ # every stack resolves it itself │ ├── FundsXML_Files/ # Sample documents, per version & use-case │ ├── 4.2.9/{positions,transactions,documents,regulatory,signed}/ diff --git a/XML_Signature/cli/sign-verify-xmlsec1.sh b/XML_Signature/cli/sign-verify-xmlsec1.sh old mode 100644 new mode 100755 index cc0a9f8..9891c6d --- a/XML_Signature/cli/sign-verify-xmlsec1.sh +++ b/XML_Signature/cli/sign-verify-xmlsec1.sh @@ -1,22 +1,35 @@ -#!/usr/bin/env bash +#!/bin/sh # Enveloped XML-DSig sign / verify on the command line with xmlsec1 -# (the xmlsec C library CLI, package: libxmlsec1 / xmlsec1). +# (the xmlsec C library CLI; package: libxmlsec1 / xmlsec1). # # sign-verify-xmlsec1.sh sign # sign-verify-xmlsec1.sh verify [cert.pem] # -# Reference implementation (xmlsec1 not installed in the dev environment: +# Reference example — xmlsec1 is a native CLI tool (not bundled): # Debian/Ubuntu: sudo apt-get install xmlsec1 # macOS: brew install xmlsec1 +# (The verified, fully cross-platform signing path is the Java example, +# XML_Signature/java — run via the Maven Wrapper.) +# +# Keys: generated cross-platform by the Java GenerateTestKey (no openssl): +# ./mvnw -q -pl XML_Signature/java compile exec:java -Dexec.mainClass=GenerateTestKey +# It writes test-signing.p12, test-signing-cert.pem AND test-signing-key.pem +# (PKCS#8) into XML_Signature/keys/ — the last two are what xmlsec1 needs. # # Unlike the Java/Python examples, xmlsec1 signs an EXISTING -# template in the document — so it pairs naturally with the committed -# FundsXML_Files/4.2.9/signed/Signed_Fund_Skeleton.xml (which already carries a -# placeholder ds:Signature). Keys: XML_Signature/generate-test-key.sh. -set -euo pipefail +# template in the document, so it pairs naturally with the committed +# FundsXML_Files/4.2.9/signed/Signed_Fund_Skeleton.xml (placeholder signature). +# +# POSIX sh; the Windows counterpart is sign-verify.ps1 in this directory. -MODE="${1:?usage: sign-verify-xmlsec1.sh sign|verify ...}" -KEYS="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)/keys" +set -eu + +MODE="${1:-}" +if [ -z "$MODE" ]; then + echo "usage: sign-verify-xmlsec1.sh sign|verify ..." >&2 + exit 2 +fi +KEYS=$(CDPATH= cd -- "$(dirname -- "$0")/.." && pwd)/keys case "$MODE" in sign) @@ -32,10 +45,15 @@ case "$MODE" in SIGNED="${2:?signed.xml}" CERT="${3:-${KEYS}/test-signing-cert.pem}" # --trusted-pem pins the signer cert (do NOT trust only the embedded key). - xmlsec1 --verify --trusted-pem "$CERT" "$SIGNED" \ - && echo "VALID: signature OK" \ - || { echo "INVALID: signature check failed"; exit 1; } + if xmlsec1 --verify --trusted-pem "$CERT" "$SIGNED"; then + echo "VALID: signature OK" + else + echo "INVALID: signature check failed" >&2 + exit 1 + fi ;; *) - echo "unknown mode: $MODE" >&2; exit 2 ;; + echo "unknown mode: $MODE" >&2 + exit 2 + ;; esac diff --git a/XML_Signature/cli/sign-verify.ps1 b/XML_Signature/cli/sign-verify.ps1 new file mode 100644 index 0000000..ce22b4e --- /dev/null +++ b/XML_Signature/cli/sign-verify.ps1 @@ -0,0 +1,30 @@ +<# +.SYNOPSIS + Windows counterpart of sign-verify-xmlsec1.sh. + +.DESCRIPTION + xmlsec1 is a POSIX/native CLI with no first-class Windows build, so on + Windows the command-line XML-signature path is the .NET example + (XML_Signature/dotnet, System.Security.Cryptography.Xml) — same signature + profile (RSA-SHA256, exclusive C14N, enveloped) as the verified Java + example, so files cross-verify between stacks. + + This thin wrapper just forwards to that .NET project via `dotnet run`. + Keys come from the cross-platform Java GenerateTestKey (no openssl): + .\mvnw.cmd -q -pl XML_Signature/java compile exec:java -Dexec.mainClass=GenerateTestKey + +.EXAMPLE + pwsh XML_Signature/cli/sign-verify.ps1 sign FundsXML_Files/4.2.9/positions/Mixed-Fund_Positions.xml signed.xml + pwsh XML_Signature/cli/sign-verify.ps1 verify signed.xml +#> +[CmdletBinding()] +param( + [Parameter(Mandatory = $true, Position = 0)] [string] $Mode, + [Parameter(ValueFromRemainingArguments = $true)] [string[]] $Rest +) +$ErrorActionPreference = 'Stop' +$proj = Join-Path $PSScriptRoot '..\dotnet' + +Write-Host "Windows CLI XML-signature -> .NET example ($proj)" -ForegroundColor DarkGray +& dotnet run --project $proj -- $Mode @Rest +exit $LASTEXITCODE diff --git a/XML_Signature/java/GenerateTestKey.java b/XML_Signature/java/GenerateTestKey.java index ad0203a..accb6dd 100644 --- a/XML_Signature/java/GenerateTestKey.java +++ b/XML_Signature/java/GenerateTestKey.java @@ -12,12 +12,18 @@ // Output (gitignored — never commit private keys, even demo ones): // test-signing.p12 PKCS#12 keystore (alias: fundsxml, pass: changeit) // test-signing-cert.pem public certificate (for verification) +// test-signing-key.pem PKCS#8 private key (for the xmlsec1 / signxml CLI +// and Python reference examples — emitted here, in +// pure JDK, so no openssl is needed on any OS) // // FOR DEMO USE ONLY — 2048-bit RSA, 10-year self-signed, hard-coded password. import java.nio.file.Files; import java.nio.file.Path; import java.nio.file.Paths; +import java.security.KeyStore; +import java.security.PrivateKey; +import java.util.Base64; public class GenerateTestKey { @@ -32,11 +38,13 @@ public static void main(String[] args) throws Exception { Files.createDirectories(dir); Path p12 = dir.resolve("test-signing.p12"); Path cert = dir.resolve("test-signing-cert.pem"); + Path keyPem = dir.resolve("test-signing-key.pem"); // keytool's genkeypair fails if the alias already exists, so start // from a clean keystore for idempotent re-runs. Files.deleteIfExists(p12); Files.deleteIfExists(cert); + Files.deleteIfExists(keyPem); String keytool = keytool(); @@ -57,9 +65,24 @@ public static void main(String[] args) throws Exception { "-keystore", p12.toString(), "-storepass", PASS, "-file", cert.toString()); + // keytool cannot export a private key to PEM, so do it in pure JDK: + // load the PKCS#12, take the RSA key, write it as an unencrypted + // PKCS#8 PEM. xmlsec1 / signxml read this directly (no openssl). + KeyStore ks = KeyStore.getInstance("PKCS12"); + try (var in = Files.newInputStream(p12)) { + ks.load(in, PASS.toCharArray()); + } + PrivateKey pk = (PrivateKey) ks.getKey(ALIAS, PASS.toCharArray()); + String b64 = Base64.getMimeEncoder(64, "\n".getBytes()) + .encodeToString(pk.getEncoded()); // PKCS#8 DER + Files.writeString(keyPem, + "-----BEGIN PRIVATE KEY-----\n" + b64 + + "\n-----END PRIVATE KEY-----\n"); + System.out.println("wrote: " + p12 + " (alias=" + ALIAS + " pass=" + PASS + ")"); System.out.println(" " + cert); + System.out.println(" " + keyPem); } /** Locate the keytool that belongs to the running JDK (cross-platform). */ diff --git a/XSD_Validation/README.md b/XSD_Validation/README.md index c7a519e..befa7be 100644 --- a/XSD_Validation/README.md +++ b/XSD_Validation/README.md @@ -28,13 +28,14 @@ GitHub release (302-aware; also pulls the imported `xmldsig-core-schema.xsd`), caching into `.schema-cache/`. The official release stays the source of truth — no committed catalog. -The **Java** (`XsdValidate`), **Python** (`validate.py`) and **.NET** -(`XsdValidate.cs` + `SchemaResolver.cs`) examples do this themselves — -standalone, cross-platform, no prior step. `tools/fetch-schema.sh` still seeds -the cache for the CLI/xmllint and (until its phase lands) PowerShell stacks: +**Every** stack does this itself now — Java (`XsdValidate`), Python +(`validate.py`), .NET (`XsdValidate.cs` + `SchemaResolver.cs`), the CLI +`validate.sh`/`validate.ps1`, and PowerShell `Validate-FundsXml.ps1` — +standalone, cross-platform, no prior step. To pre-populate the cache for a +bare `xmllint` invocation: ```bash -tools/fetch-schema.sh 4.2.9 # only needed for the CLI/PowerShell stacks +python -m fundsxml_schema 4.2.9 # after `pip install -e .`; cross-platform ``` ## Security @@ -47,11 +48,12 @@ Every example disables external entity resolution / DTD loading | Stack | Script | API | Runnable on this box | |-------|--------|-----|----------------------| -| CLI | [`cli/validate.sh`](cli/validate.sh) | `xmllint` (+ Saxon note) | ✅ | +| CLI (Linux/macOS) | [`cli/validate.sh`](cli/validate.sh) | `xmllint` (POSIX sh) | ✅ standalone (self-resolving) | +| CLI (Windows) | [`cli/validate.ps1`](cli/validate.ps1) | `xmllint` or .NET fallback | ✅ standalone (self-resolving) | | Python | [`python/validate.py`](python/validate.py) | `lxml.etree.XMLSchema` | ✅ standalone (`pip install -e .`) | | Java | [`java/XsdValidate.java`](java/XsdValidate.java) | `javax.xml.validation` | ✅ standalone (`./mvnw`) | | .NET/C# | [`dotnet/XsdValidate.cs`](dotnet/XsdValidate.cs) | `XmlSchemaSet` | ✅ standalone (`dotnet run`) | -| PowerShell | [`powershell/Validate-FundsXml.ps1`](powershell/Validate-FundsXml.ps1) | `System.Xml.Schema` | needs PowerShell | +| PowerShell | [`powershell/Validate-FundsXml.ps1`](powershell/Validate-FundsXml.ps1) | `System.Xml.Schema` | ✅ standalone (self-resolving) | Convention: each takes ` `, exits `0` on valid, `1` on invalid, prints errors to stderr. @@ -73,5 +75,6 @@ Java (standalone): `./mvnw -q -pl XSD_Validation/java compile exec:java -Dexec.a .NET (standalone): `dotnet run --project XSD_Validation/dotnet -- 4.2.9 ` (exit 0 valid / 1 invalid). -The CLI/`xmllint` stack still uses `tools/fetch-schema.sh 4.2.9` until its -phase lands. +CLI (standalone, self-resolving): `XSD_Validation/cli/validate.sh 4.2.9 ` +(Linux/macOS) or `pwsh XSD_Validation/cli/validate.ps1 4.2.9 ` +(Windows — uses `xmllint` if present, else the built-in .NET validator). diff --git a/XSD_Validation/cli/validate.ps1 b/XSD_Validation/cli/validate.ps1 new file mode 100644 index 0000000..da9b4bc --- /dev/null +++ b/XSD_Validation/cli/validate.ps1 @@ -0,0 +1,98 @@ +<# +.SYNOPSIS + XSD validation from the command line on Windows — the counterpart of + validate.sh. Standalone: resolves the official schema itself (no prior step). + +.DESCRIPTION + Same 3-step convention as every other stack: + 1. $env:FUNDSXML_SCHEMA_DIR — a dir with FundsXML.xsd (+ the xmldsig + sibling for 4.2.9+). Used as-is, NO network (offline / corporate + escape hatch). + 2. .schema-cache\\FundsXML.xsd — reused if present. + 3. download from the official GitHub release (Invoke-WebRequest follows + the 302), caching into .schema-cache\; the relative + xmldsig-core-schema.xsd sibling is fetched only when imported (4.2.9+). + + Validation uses xmllint when it is on PATH, otherwise the built-in .NET + System.Xml.Schema (always available on Windows) — so no extra tool to + install. Works in Windows PowerShell 5.1 and PowerShell 7+. + +.PARAMETER Version FundsXML version, e.g. 4.2.9 +.PARAMETER XmlFile the instance document to validate +.EXAMPLE + pwsh XSD_Validation/cli/validate.ps1 4.2.9 FundsXML_Files/4.2.9/positions/Mixed-Fund_Positions.xml +#> +[CmdletBinding()] +param( + [Parameter(Mandatory = $true, Position = 0)] [string] $Version, + [Parameter(Mandatory = $true, Position = 1)] [string] $XmlFile +) +$ErrorActionPreference = 'Stop' + +$repoRoot = (Resolve-Path (Join-Path $PSScriptRoot '..\..')).Path +$base = "https://github.com/fundsxml/schema/releases/download/$Version" + +function Get-File($url, $dest) { + Write-Host "schema: fetch $url" -ForegroundColor DarkGray + Invoke-WebRequest -Uri $url -OutFile $dest -MaximumRedirection 5 -UseBasicParsing +} + +if ($env:FUNDSXML_SCHEMA_DIR) { + $schema = Join-Path $env:FUNDSXML_SCHEMA_DIR 'FundsXML.xsd' + if (-not (Test-Path $schema)) { + Write-Error "FUNDSXML_SCHEMA_DIR set but $schema not found"; exit 2 + } + Write-Host "schema: using `$FUNDSXML_SCHEMA_DIR -> $schema" +} +else { + $cacheDir = Join-Path $repoRoot ".schema-cache\$Version" + $schema = Join-Path $cacheDir 'FundsXML.xsd' + if (Test-Path $schema) { + Write-Host "schema: cached -> $schema" + } + else { + New-Item -ItemType Directory -Force -Path $cacheDir | Out-Null + Get-File "$base/FundsXML.xsd" $schema + if (Select-String -Path $schema -Pattern 'xmldsig-core-schema\.xsd' -Quiet) { + Get-File "$base/xmldsig-core-schema.xsd" (Join-Path $cacheDir 'xmldsig-core-schema.xsd') + } + } +} + +$xmllint = Get-Command xmllint -ErrorAction SilentlyContinue +if ($xmllint) { + & $xmllint.Source --noout --nonet --schema $schema $XmlFile + if ($LASTEXITCODE -eq 0) { Write-Host "VALID: $XmlFile (FundsXML $Version)"; exit 0 } + Write-Error "INVALID: $XmlFile (FundsXML $Version)"; exit 1 +} + +# No xmllint: validate with the built-in .NET schema validator. The schema set +# gets a URL resolver only so the relative xmldsig import resolves; the +# instance document is read with no resolver (XXE-hardened). +Add-Type -AssemblyName System.Xml +$set = New-Object System.Xml.Schema.XmlSchemaSet +$set.XmlResolver = New-Object System.Xml.XmlUrlResolver +[void]$set.Add($null, $schema) +$rs = New-Object System.Xml.XmlReaderSettings +$rs.ValidationType = [System.Xml.ValidationType]::Schema +$rs.Schemas = $set +$rs.DtdProcessing = [System.Xml.DtdProcessing]::Prohibit +$rs.XmlResolver = $null +$script:bad = $false +$handler = [System.Xml.Schema.ValidationEventHandler] { + param($s, $e) + if ($e.Severity -eq [System.Xml.Schema.XmlSeverityType]::Error) { + $script:bad = $true + Write-Host (" " + $e.Message) + } +} +$rs.add_ValidationEventHandler($handler) +try { + $r = [System.Xml.XmlReader]::Create($XmlFile, $rs) + while ($r.Read()) { } + $r.Close() +} +catch { $script:bad = $true; Write-Host (" " + $_.Exception.Message) } +if ($script:bad) { Write-Error "INVALID: $XmlFile (FundsXML $Version)"; exit 1 } +Write-Host "VALID: $XmlFile (FundsXML $Version)" +exit 0 diff --git a/XSD_Validation/cli/validate.sh b/XSD_Validation/cli/validate.sh index 015cc87..36cda24 100755 --- a/XSD_Validation/cli/validate.sh +++ b/XSD_Validation/cli/validate.sh @@ -1,41 +1,69 @@ -#!/usr/bin/env bash -# XSD validation via the command line (xmllint). +#!/bin/sh +# XSD validation from the command line with xmllint (Linux/macOS). # # Usage: XSD_Validation/cli/validate.sh # Exit: 0 = valid, 1 = invalid, 2 = usage/setup error # -# Validates against the official released schema, materialized locally by -# tools/fetch-schema.sh (handles the GitHub 302 redirect and the relative -# xmldsig-core-schema.xsd import that FundsXML 4.2.9+ requires). -set -euo pipefail +# Standalone: this script resolves the official schema itself — no prior tool +# step. Same 3-step convention as every other stack: +# 1. $FUNDSXML_SCHEMA_DIR — a dir with FundsXML.xsd (+ xmldsig sibling for +# 4.2.9+). Used as-is, NO network (offline / corporate-network escape). +# 2. .schema-cache//FundsXML.xsd — reused if present. +# 3. download from the official GitHub release (curl -L follows the 302), +# caching into .schema-cache/; the relative xmldsig-core-schema.xsd +# sibling is fetched only when FundsXML.xsd imports it (4.2.9+). +# +# POSIX sh (no bash-isms) so it runs under dash/ash too. The Windows +# counterpart is validate.ps1 in this directory. + +set -eu + +VERSION="${1:-}" +XML="${2:-}" +if [ -z "$VERSION" ] || [ -z "$XML" ]; then + echo "usage: validate.sh " >&2 + exit 2 +fi -VERSION="${1:?usage: validate.sh }" -XML="${2:?usage: validate.sh }" +REPO_ROOT=$(CDPATH= cd -- "$(dirname -- "$0")/../.." && pwd) +BASE="https://github.com/fundsxml/schema/releases/download/${VERSION}" -REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" -SCHEMA="${REPO_ROOT}/.schema-cache/${VERSION}/FundsXML.xsd" +fetch() { # url dest + echo "schema: fetch $1" >&2 + curl -sSL --fail -m 60 "$1" -o "$2" +} -if [[ ! -f "$SCHEMA" ]]; then - echo "schema not cached; fetching official release ${VERSION}..." >&2 - "${REPO_ROOT}/tools/fetch-schema.sh" "$VERSION" >/dev/null +if [ -n "${FUNDSXML_SCHEMA_DIR:-}" ]; then + SCHEMA="${FUNDSXML_SCHEMA_DIR}/FundsXML.xsd" + if [ ! -f "$SCHEMA" ]; then + echo "FUNDSXML_SCHEMA_DIR set but $SCHEMA not found" >&2 + exit 2 + fi + echo "schema: using \$FUNDSXML_SCHEMA_DIR -> $SCHEMA" >&2 +else + CACHE_DIR="${REPO_ROOT}/.schema-cache/${VERSION}" + SCHEMA="${CACHE_DIR}/FundsXML.xsd" + if [ -f "$SCHEMA" ]; then + echo "schema: cached -> $SCHEMA" >&2 + else + mkdir -p "$CACHE_DIR" + fetch "${BASE}/FundsXML.xsd" "$SCHEMA" + if grep -q 'xmldsig-core-schema\.xsd' "$SCHEMA"; then + fetch "${BASE}/xmldsig-core-schema.xsd" \ + "${CACHE_DIR}/xmldsig-core-schema.xsd" + fi + fi fi -# --nonet: never hit the network during validation (XXE / entity hardening). -# The schema was already fetched explicitly above. -if xmllint --noout --nonet --schema "$SCHEMA" "$XML" 2>/tmp/xmllint.$$; then +# --nonet: never hit the network during validation (XXE / entity hardening); +# the schema was already materialised above. +ERR=$(mktemp) +trap 'rm -f "$ERR"' EXIT +if xmllint --noout --nonet --schema "$SCHEMA" "$XML" 2>"$ERR"; then echo "VALID: $XML (FundsXML $VERSION)" - rm -f /tmp/xmllint.$$ exit 0 else echo "INVALID: $XML (FundsXML $VERSION)" >&2 - cat /tmp/xmllint.$$ >&2 - rm -f /tmp/xmllint.$$ + cat "$ERR" >&2 exit 1 fi - -# --- Alternative: Saxon (XSD 1.1, follows HTTP redirects natively) ---------- -# Saxon-EE/HE can validate and DOES follow the 302, but the relative -# xmldsig-core-schema.xsd import still needs both files side by side, so the -# fetched local copy is used here too: -# java -cp saxon-he.jar com.saxonica.Validate \ -# -xsd:"$SCHEMA" -s:"$XML" diff --git a/XSD_Validation/powershell/Validate-FundsXml.ps1 b/XSD_Validation/powershell/Validate-FundsXml.ps1 index f42a535..2769c0d 100644 --- a/XSD_Validation/powershell/Validate-FundsXml.ps1 +++ b/XSD_Validation/powershell/Validate-FundsXml.ps1 @@ -3,10 +3,12 @@ XSD validation in PowerShell via System.Xml.Schema. .DESCRIPTION - Validates a FundsXML document against the official released schema, - materialized locally by tools/fetch-schema.sh (handles the GitHub 302 - redirect and the relative xmldsig-core-schema.xsd import that FundsXML - 4.2.9+ requires). + Standalone & cross-platform — resolves the official released schema + itself (no prior tool step). Same 3-step convention as every stack: + $env:FUNDSXML_SCHEMA_DIR (offline/corporate escape hatch) -> + .schema-cache\ -> download from the official GitHub release + (Invoke-WebRequest follows the 302; pulls the xmldsig-core-schema.xsd + sibling that FundsXML 4.2.9+ imports). Security: the instance document is read with XmlResolver = $null and DtdProcessing = Prohibit to close XXE / external-entity vectors. A @@ -34,12 +36,34 @@ param( $ErrorActionPreference = 'Stop' -$repoRoot = Resolve-Path (Join-Path $PSScriptRoot '..' '..') -$schemaPath = Join-Path $repoRoot ".schema-cache/$Version/FundsXML.xsd" +$repoRoot = (Resolve-Path (Join-Path $PSScriptRoot '..' '..')).Path +$base = "https://github.com/fundsxml/schema/releases/download/$Version" -if (-not (Test-Path $schemaPath)) { - Write-Error "schema not cached; run: tools/fetch-schema.sh $Version" - exit 2 +function Get-File($url, $dest) { + Write-Host "schema: fetch $url" -ForegroundColor DarkGray + Invoke-WebRequest -Uri $url -OutFile $dest -MaximumRedirection 5 -UseBasicParsing +} + +if ($env:FUNDSXML_SCHEMA_DIR) { + $schemaPath = Join-Path $env:FUNDSXML_SCHEMA_DIR 'FundsXML.xsd' + if (-not (Test-Path $schemaPath)) { + Write-Error "FUNDSXML_SCHEMA_DIR set but $schemaPath not found"; exit 2 + } + Write-Host "schema: using `$FUNDSXML_SCHEMA_DIR -> $schemaPath" +} +else { + $cacheDir = Join-Path $repoRoot ".schema-cache/$Version" + $schemaPath = Join-Path $cacheDir 'FundsXML.xsd' + if (Test-Path $schemaPath) { + Write-Host "schema: cached -> $schemaPath" + } + else { + New-Item -ItemType Directory -Force -Path $cacheDir | Out-Null + Get-File "$base/FundsXML.xsd" $schemaPath + if (Select-String -Path $schemaPath -Pattern 'xmldsig-core-schema\.xsd' -Quiet) { + Get-File "$base/xmldsig-core-schema.xsd" (Join-Path $cacheDir 'xmldsig-core-schema.xsd') + } + } } $schemas = New-Object System.Xml.Schema.XmlSchemaSet diff --git a/tools/fetch-schema.sh b/tools/fetch-schema.sh deleted file mode 100755 index c70fc9f..0000000 --- a/tools/fetch-schema.sh +++ /dev/null @@ -1,58 +0,0 @@ -#!/usr/bin/env bash -# fetch-schema.sh — Fetch the official FundsXML XSD release from GitHub. -# -# WHY THIS SCRIPT EXISTS -# ---------------------- -# Validation is done against the canonical schema URL: -# -# https://github.com/fundsxml/schema/releases/download//FundsXML.xsd -# -# That URL returns HTTP 302 to objects.githubusercontent.com. Processors with a -# simple HTTP client (e.g. libxml2 / xmllint) do NOT follow redirects and fail. -# In addition, from release 4.2.9 onward `FundsXML.xsd` imports -# `xmldsig-core-schema.xsd` via a RELATIVE path — both files must therefore sit -# in the same directory, otherwise XSD compilation fails -# ("{http://www.w3.org/2000/09/xmldsig#}Signature does not resolve"). -# -# Enterprise reality: on locked-down networks the download goes through an HTTP -# proxy (curl honours https_proxy / HTTPS_PROXY). No hand-maintained XML catalog -# is used — the source of truth stays the official release. -# -# Usage: -# tools/fetch-schema.sh [target-dir] -# tools/fetch-schema.sh 4.2.9 -# tools/fetch-schema.sh 4.1.0 /tmp/xsd -# -# Output: path to the local FundsXML.xsd on stdout (usable by scripts). -set -euo pipefail - -VERSION="${1:?Usage: fetch-schema.sh [target-dir]}" -TARGET="${2:-.schema-cache/${VERSION}}" -BASE="https://github.com/fundsxml/schema/releases/download/${VERSION}" - -mkdir -p "$TARGET" - -fetch() { - local name="$1" - local out="${TARGET}/${name}" - if [[ -s "$out" ]]; then - echo "cached: $out" >&2 - return 0 - fi - echo "fetch: ${BASE}/${name} -> $out" >&2 - # -L follows the GitHub redirect; --fail aborts on HTTP error. - curl -sSL --fail -m 60 "${BASE}/${name}" -o "$out" -} - -fetch "FundsXML.xsd" - -# From 4.2.9 on, FundsXML.xsd imports xmldsig-core-schema.xsd via a relative -# path — without that file the schema does not compile -# ("{http://www.w3.org/2000/09/xmldsig#}Signature does not resolve"). -# Older releases (4.1.0, 4.0.0) are self-contained and do not reference it. -# Fetch the sibling only when it is actually imported. -if grep -q 'xmldsig-core-schema\.xsd' "${TARGET}/FundsXML.xsd"; then - fetch "xmldsig-core-schema.xsd" -fi - -echo "${TARGET}/FundsXML.xsd" From b0fc1ab4b859b6a672809e13f894ab49700ddd07 Mon Sep 17 00:00:00 2001 From: Karl Kauc Date: Sat, 16 May 2026 09:30:17 +0200 Subject: [PATCH 2/2] Phase 4 fix: reset $LASTEXITCODE in windows-smoke steps GitHub's pwsh wrapper exits each step with the last $LASTEXITCODE. The negative (must-fail) cases leave it at 1, so steps failed even though the logic succeeded (positive VALID + negative correctly INVALID, both shown in the log). End each windows-smoke step with an explicit 'exit 0'. The ubuntu 'validate' job was already green; only the new windows-smoke job was affected. Co-Authored-By: Claude Opus 4.7 (1M context) --- .github/workflows/ci.yml | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 8a36339..cb4fc3a 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -344,6 +344,7 @@ jobs: .venv\Scripts\python XSD_Validation/python/validate.py 4.2.9 tests/fixtures/invalid/xsd-invalid_Positions.xml if ($LASTEXITCODE -eq 0) { throw "negative unexpectedly validated (Python)" } Write-Host "Python OK on Windows" + exit 0 # reset $LASTEXITCODE (the negative case left it at 1) - name: Java - XSD validate + one XSLT (Maven Wrapper, mvnw.cmd) run: | @@ -355,6 +356,7 @@ jobs: .\mvnw.cmd -q -B -pl XSLT_Transformations/invocation compile exec:java "-Dexec.args=XSLT_Transformations/CSV_Export/positions_csv.xslt $src out_pos.csv" if (-not (Test-Path out_pos.csv)) { throw "XSLT produced no output" } Write-Host "Java OK on Windows" + exit 0 # reset $LASTEXITCODE (the negative case left it at 1) - name: .NET - XSD validation (downloads schema itself on Windows) run: | @@ -364,6 +366,7 @@ jobs: dotnet run --project XSD_Validation/dotnet -- 4.2.9 tests/fixtures/invalid/xsd-invalid_Positions.xml if ($LASTEXITCODE -eq 0) { throw ".NET negative unexpectedly validated" } Write-Host ".NET OK on Windows" + exit 0 # reset $LASTEXITCODE (the negative case left it at 1) - name: CLI - validate.ps1 (PowerShell, .NET fallback when no xmllint) run: | @@ -372,6 +375,7 @@ jobs: pwsh XSD_Validation/cli/validate.ps1 4.2.9 tests/fixtures/invalid/xsd-invalid_Positions.xml if ($LASTEXITCODE -eq 0) { throw "validate.ps1 negative unexpectedly validated" } Write-Host "PowerShell CLI OK on Windows" + exit 0 # reset $LASTEXITCODE (the negative case left it at 1) - name: DB round-trip (Python) on Windows run: | @@ -383,3 +387,4 @@ jobs: .venv\Scripts\python Database_Integration/tools/xml_equiv.py $fx out.xml if ($LASTEXITCODE -ne 0) { throw "round-trip not equivalent" } Write-Host "DB round-trip OK on Windows" + exit 0