Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion .github/EXAMPLE_README_TEMPLATE.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,11 @@ Why this exists and when an enterprise integrator would use it.
## Prerequisites

- Tooling/runtime versions
- `tools/fetch-schema.sh <version>` if schema is needed
- The example resolves the XSD itself (env `FUNDSXML_SCHEMA_DIR` →
`.schema-cache/` → official-release download); or
`python -m fundsxml_schema <version>` to pre-cache for a bare xmllint
- Network/proxy note when the official schema URL must be reached
(`$FUNDSXML_SCHEMA_DIR` is the offline escape hatch)

## Run

Expand Down
4 changes: 2 additions & 2 deletions .github/ISSUE_TEMPLATE/bug_report.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,8 @@ e.g. 4.2.9, `FundsXML_Files/4.2.9/positions/Mixed-Fund_Positions.xml`
(or attach/inline a minimal FundsXML snippet — no real/PII data).

**What I ran**
The exact command(s), including the `./mvnw` (or `mvnw.cmd`) invocation for
Java examples, or any `tools/fetch-schema.sh` step.
The exact command(s) — e.g. the `./mvnw`/`mvnw.cmd` invocation (Java), the
venv `pip install -e .` + `python ...` (Python), or `dotnet run` (.NET).

**Expected vs. actual**
What you expected and what happened (paste the full error / output).
Expand Down
5 changes: 3 additions & 2 deletions .github/PULL_REQUEST_TEMPLATE.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,9 @@
- [ ] English only; example is **self-contained** and **heavily commented**
(file-header: purpose / run / deps / FundsXML assumptions + what & why).
- [ ] FundsXML conventions: no XML namespace; XSD-validated against the
**official released schema** via `tools/fetch-schema.sh`; 4.0.0 has no
`ControlData/Version`; secure XML parsing (DTD/external entities off).
**official released schema** (each example resolves it itself; or
`python -m fundsxml_schema <ver>`); 4.0.0 has no `ControlData/Version`;
secure XML parsing (DTD/external entities off).
- [ ] New/changed samples are **XSD-valid**; negative fixtures still fail.
- [ ] Round-trip examples: proven with `Database_Integration/tools/xml_equiv.py`
**and** `xmllint --schema` (complementary checks).
Expand Down
124 changes: 110 additions & 14 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -39,9 +39,21 @@ jobs:
with:
dotnet-version: "8.0"

- name: Fetch official schemas (4.2.9, 4.1.0, 4.0.0)
# One idiomatic, OS-agnostic Python install (venv + editable pyproject);
# also exposes the `fundsxml_schema` resolver to every Python script.
- name: Python - venv & install (pyproject)
run: |
for v in 4.2.9 4.1.0 4.0.0; do tools/fetch-schema.sh "$v"; done
python -m venv .venv
.venv/bin/pip install --quiet --upgrade pip
.venv/bin/pip install --quiet -e .
.venv/bin/python -c "import lxml, saxonche, fundsxml_schema; print('python deps OK')"

# Materialise the official schemas into .schema-cache/ for the
# xmllint-based steps below — via the in-language resolver module
# (cross-platform), replacing the deleted tools/fetch-schema.sh.
- name: Materialise official schemas (4.2.9, 4.1.0, 4.0.0)
run: |
for v in 4.2.9 4.1.0 4.0.0; do .venv/bin/python -m fundsxml_schema "$v"; done

- name: XSD - all positive samples must validate
run: |
Expand All @@ -67,18 +79,6 @@ jobs:
fi
echo "xsd-invalid correctly rejected"

# ---------------------------------------------------------------------
# Python examples — one idiomatic, OS-agnostic install (venv + editable
# pyproject) replaces the old ad-hoc `pip install lxml`. This also
# exposes the `fundsxml_schema` resolver module to every Python script.
# ---------------------------------------------------------------------
- name: Python - venv & install (pyproject)
run: |
python -m venv .venv
.venv/bin/pip install --quiet --upgrade pip
.venv/bin/pip install --quiet -e .
.venv/bin/python -c "import lxml, saxonche, fundsxml_schema; print('python deps OK')"

- name: Python - XSD validation (in-language schema resolve)
run: |
set -e
Expand Down Expand Up @@ -107,6 +107,17 @@ jobs:
fi
echo ".NET XsdValidate: positive ok, negative correctly rejected"

- name: CLI - validate.sh (standalone POSIX sh)
run: |
set -e
XSD_Validation/cli/validate.sh 4.2.9 \
FundsXML_Files/4.2.9/positions/Mixed-Fund_Positions.xml
if XSD_Validation/cli/validate.sh 4.2.9 \
tests/fixtures/invalid/xsd-invalid_Positions.xml; then
echo "::error::xsd-invalid unexpectedly validated (CLI)"; exit 1
fi
echo "CLI validate.sh: positive ok, negative correctly rejected"

# ---------------------------------------------------------------------
# Java examples — built & run via the committed Maven Wrapper. The first
# ./mvnw bootstraps Maven itself, then resolves all deps from Central.
Expand Down Expand Up @@ -292,3 +303,88 @@ jobs:
xsltproc XSLT_DataQuality_Checks/Enhanced_Check/FundsXML_CompleteDQReport_HTML.xsl \
FundsXML_Files/4.2.9/positions/Mixed-Fund_Positions.xml > out_enh.html
test -s out_enh.html

# =======================================================================
# Windows smoke — proves the examples are genuinely cross-platform and
# standalone (the whole point of this work): every stack resolves the XSD
# itself and runs with no bash, no prior tool step, on a clean Windows box.
# A focused subset (Python / Java / .NET XSD + one XSLT + a DB round-trip +
# the PowerShell CLI), not the full bash-heavy matrix.
# =======================================================================
windows-smoke:
runs-on: windows-latest
defaults:
run:
shell: pwsh
steps:
- uses: actions/checkout@v6
- uses: actions/setup-java@v5
with:
distribution: temurin
java-version: "21"
- uses: actions/setup-python@v6
with:
python-version: "3.11"
- uses: actions/setup-dotnet@v5
with:
dotnet-version: "8.0"

- name: Python - venv & install (pyproject)
run: |
python -m venv .venv
.venv\Scripts\python -m pip install --quiet --upgrade pip
.venv\Scripts\pip install --quiet -e .
.venv\Scripts\python -c "import lxml, saxonche, fundsxml_schema; print('python deps OK')"

- name: Python - XSD validation (downloads schema itself on Windows)
run: |
$src = "FundsXML_Files/4.2.9/positions/Mixed-Fund_Positions.xml"
.venv\Scripts\python XSD_Validation/python/validate.py 4.2.9 $src
if ($LASTEXITCODE -ne 0) { throw "positive failed" }
.venv\Scripts\python XSD_Validation/python/validate.py 4.2.9 tests/fixtures/invalid/xsd-invalid_Positions.xml
if ($LASTEXITCODE -eq 0) { throw "negative unexpectedly validated (Python)" }
Write-Host "Python OK on Windows"
exit 0 # reset $LASTEXITCODE (the negative case left it at 1)

- name: Java - XSD validate + one XSLT (Maven Wrapper, mvnw.cmd)
run: |
$src = "FundsXML_Files/4.2.9/positions/Mixed-Fund_Positions.xml"
.\mvnw.cmd -q -B -pl XSD_Validation/java compile exec:java "-Dexec.args=4.2.9 $src"
if ($LASTEXITCODE -ne 0) { throw "Java XSD positive failed" }
.\mvnw.cmd -q -B -pl XSD_Validation/java exec:java "-Dexec.args=4.2.9 tests/fixtures/invalid/xsd-invalid_Positions.xml"
if ($LASTEXITCODE -eq 0) { throw "Java negative unexpectedly validated" }
.\mvnw.cmd -q -B -pl XSLT_Transformations/invocation compile exec:java "-Dexec.args=XSLT_Transformations/CSV_Export/positions_csv.xslt $src out_pos.csv"
if (-not (Test-Path out_pos.csv)) { throw "XSLT produced no output" }
Write-Host "Java OK on Windows"
exit 0 # reset $LASTEXITCODE (the negative case left it at 1)

- name: .NET - XSD validation (downloads schema itself on Windows)
run: |
$src = "FundsXML_Files/4.2.9/positions/Mixed-Fund_Positions.xml"
dotnet run --project XSD_Validation/dotnet -- 4.2.9 $src
if ($LASTEXITCODE -ne 0) { throw ".NET XSD positive failed" }
dotnet run --project XSD_Validation/dotnet -- 4.2.9 tests/fixtures/invalid/xsd-invalid_Positions.xml
if ($LASTEXITCODE -eq 0) { throw ".NET negative unexpectedly validated" }
Write-Host ".NET OK on Windows"
exit 0 # reset $LASTEXITCODE (the negative case left it at 1)

- name: CLI - validate.ps1 (PowerShell, .NET fallback when no xmllint)
run: |
pwsh XSD_Validation/cli/validate.ps1 4.2.9 FundsXML_Files/4.2.9/positions/Mixed-Fund_Positions.xml
if ($LASTEXITCODE -ne 0) { throw "validate.ps1 positive failed" }
pwsh XSD_Validation/cli/validate.ps1 4.2.9 tests/fixtures/invalid/xsd-invalid_Positions.xml
if ($LASTEXITCODE -eq 0) { throw "validate.ps1 negative unexpectedly validated" }
Write-Host "PowerShell CLI OK on Windows"
exit 0 # reset $LASTEXITCODE (the negative case left it at 1)

- name: DB round-trip (Python) on Windows
run: |
$fx = "FundsXML_Files/4.2.9/positions/Multi-Fund_Positions.xml"
.venv\Scripts\python Database_Integration/python/import_fundsxml.py fx.db $fx
if ($LASTEXITCODE -ne 0) { throw "import failed" }
.venv\Scripts\python Database_Integration/python/export_fundsxml.py fx.db FUNDSXML_MULTI_1 out.xml
if ($LASTEXITCODE -ne 0) { throw "export failed" }
.venv\Scripts\python Database_Integration/tools/xml_equiv.py $fx out.xml
if ($LASTEXITCODE -ne 0) { throw "round-trip not equivalent" }
Write-Host "DB round-trip OK on Windows"
exit 0
18 changes: 11 additions & 7 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,9 +22,12 @@ Please read this before opening a pull request.
the schema itself requires them). `xsi:noNamespaceSchemaLocation` must be in
the XMLSchema-instance namespace or validators reject it.
- **Validate against the official released schema**, never a hand-made
catalog: fetch with `tools/fetch-schema.sh <version>` (it handles GitHub's
302 redirect and the relative `xmldsig-core-schema.xsd` import that 4.2.9+
needs). Set sample `xsi:noNamespaceSchemaLocation` to that release URL.
catalog. Every example resolves it itself (`$FUNDSXML_SCHEMA_DIR` →
`.schema-cache/` → official-release download, 302-aware, pulls the
`xmldsig-core-schema.xsd` sibling for 4.2.9+). For an xmllint check,
materialise the cache with `python -m fundsxml_schema <version>`
(after `pip install -e .`). Set sample `xsi:noNamespaceSchemaLocation`
to that release URL.
- **4.0.0 `ControlData` has no `<Version>` element** (added in 4.1.0) — never
add one to a 4.0.0 sample.
- Positions ↔ Assets link by a shared `UniqueID`; `AssetMasterData` is
Expand Down Expand Up @@ -57,17 +60,18 @@ Run what you changed and confirm it actually works — no "should pass" claims.

Java examples build standalone via the committed Maven Wrapper (`./mvnw`, or
`mvnw.cmd` on Windows). Python examples install once into a venv from
`pyproject.toml` and resolve the XSD themselves. Neither needs `fetch-tools.sh`
(gone) and Python no longer needs `fetch-schema.sh`.
`pyproject.toml` and resolve the XSD themselves. No `fetch-tools.sh` and no
`fetch-schema.sh` — both are gone; every stack is standalone & cross-platform.

```bash
# Python stack (cross-platform; Windows: .venv\Scripts\activate)
python -m venv .venv && . .venv/bin/activate && pip install -e .
python XSD_Validation/python/validate.py 4.2.9 <your-sample>.xml # self-resolves the XSD

# xmllint still uses the cached schema (CLI stack, removed in a later phase)
tools/fetch-schema.sh 4.2.9
# xmllint check: materialise the cache cross-platform, then validate
python -m fundsxml_schema 4.2.9
xmllint --noout --schema .schema-cache/4.2.9/FundsXML.xsd <your-sample>.xml
# (or just: XSD_Validation/cli/validate.sh 4.2.9 <your-sample>.xml — self-resolving)

# Schematron via the Maven Wrapper (positive sample -> exit 0)
./mvnw -q -pl Schematron_DataQuality_Checks/Basic_Checks/invocation \
Expand Down
6 changes: 3 additions & 3 deletions Data_Binding_JSON/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,9 +41,9 @@ so a full generated model is heavy and brittle to maintain.
| Python | `xsdata` | `xsdata --package fundsxml.model .schema-cache/4.2.9/FundsXML.xsd` |
| .NET | `xsd.exe` / `XmlSerializer` | `xsd.exe /classes /namespace:FundsXml.Model .schema-cache\4.2.9\FundsXML.xsd` |

All three consume the **official released schema** fetched by
`tools/fetch-schema.sh` (which also pulls the imported `xmldsig-core-schema.xsd`
for 4.2.9). Trade-off: generated models are type-safe but regenerate on every
All three consume the **official released schema**; materialise it with
`python -m fundsxml_schema 4.2.9` (after `pip install -e .` — cross-platform;
also pulls the imported `xmldsig-core-schema.xsd` for 4.2.9). Trade-off: generated models are type-safe but regenerate on every
schema bump and produce thousands of classes; the native binding stays small
and version-tolerant. Pick per use case.

Expand Down
2 changes: 1 addition & 1 deletion Database_Integration/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ Each example is run as **import, then export** (the round-trip = both, then
compare). `DOC` is the document id the import prints.

```bash
tools/fetch-schema.sh 4.2.9 # XSD for validation
python -m fundsxml_schema 4.2.9 # caches the XSD into .schema-cache/ (run `pip install -e .` once; cross-platform)
FX=FundsXML_Files/4.2.9/positions/Multi-Fund_Positions.xml
DOC=FUNDSXML_MULTI_1

Expand Down
2 changes: 1 addition & 1 deletion FundsXML_Files/4.0.0/positions/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ Content-identical to the 4.1.0 example (3 equity positions), but adapted to the
## Validation

```bash
tools/fetch-schema.sh 4.0.0
python -m fundsxml_schema 4.0.0 # caches the XSD into .schema-cache/ (run `pip install -e .` once; cross-platform)
xmllint --noout --schema .schema-cache/4.0.0/FundsXML.xsd \
FundsXML_Files/4.0.0/positions/Equity-Fund_Positions.xml
```
2 changes: 1 addition & 1 deletion FundsXML_Files/4.1.0/positions/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ EUR 40m NAV. Deliberately small to make the version comparison easy.
## Validation

```bash
tools/fetch-schema.sh 4.1.0
python -m fundsxml_schema 4.1.0 # caches the XSD into .schema-cache/ (run `pip install -e .` once; cross-platform)
xmllint --noout --schema .schema-cache/4.1.0/FundsXML.xsd \
FundsXML_Files/4.1.0/positions/Equity-Fund_Positions.xml
```
2 changes: 1 addition & 1 deletion FundsXML_Files/4.2.9/documents/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ via `Document/Fund/Identifiers/LEI`.
## Validation

```bash
tools/fetch-schema.sh 4.2.9
python -m fundsxml_schema 4.2.9 # caches the XSD into .schema-cache/ (run `pip install -e .` once; cross-platform)
xmllint --noout --schema .schema-cache/4.2.9/FundsXML.xsd \
FundsXML_Files/4.2.9/documents/Fund_Documents.xml
```
2 changes: 1 addition & 1 deletion FundsXML_Files/4.2.9/regulatory/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ Mandatory blocks included:
## Validation

```bash
tools/fetch-schema.sh 4.2.9
python -m fundsxml_schema 4.2.9 # caches the XSD into .schema-cache/ (run `pip install -e .` once; cross-platform)
xmllint --noout --schema .schema-cache/4.2.9/FundsXML.xsd \
FundsXML_Files/4.2.9/regulatory/EFT_Regulatory.xml
```
5 changes: 3 additions & 2 deletions FundsXML_Files/4.2.9/signed/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,8 @@

`ds:Signature` (namespace `http://www.w3.org/2000/09/xmldsig#`) is the **last
optional child** of `<FundsXML4>`. From release 4.2.9 on, `FundsXML.xsd` imports
`xmldsig-core-schema.xsd` for this (see `tools/fetch-schema.sh`).
`xmldsig-core-schema.xsd` for this — the schema resolvers fetch that sibling
automatically when it is imported (e.g. `python -m fundsxml_schema 4.2.9`).

> ⚠️ **Placeholder:** `DigestValue` and `SignatureValue` are schema-valid base64
> strings but **not cryptographically verifiable**. Real signing and
Expand All @@ -25,7 +26,7 @@ Algorithms used (enveloped signature): C14N 2001-03-15, RSA-SHA256, SHA-256.
## Validation

```bash
tools/fetch-schema.sh 4.2.9 # also fetches xmldsig-core-schema.xsd
python -m fundsxml_schema 4.2.9 # caches the XSD into .schema-cache/ (run `pip install -e .` once; cross-platform)
xmllint --noout --schema .schema-cache/4.2.9/FundsXML.xsd \
FundsXML_Files/4.2.9/signed/Signed_Fund_Skeleton.xml
```
2 changes: 1 addition & 1 deletion FundsXML_Files/4.2.9/transactions/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ records:
## Validation

```bash
tools/fetch-schema.sh 4.2.9
python -m fundsxml_schema 4.2.9 # caches the XSD into .schema-cache/ (run `pip install -e .` once; cross-platform)
xmllint --noout --schema .schema-cache/4.2.9/FundsXML.xsd \
FundsXML_Files/4.2.9/transactions/Fund_Transactions.xml
```
Expand Down
12 changes: 8 additions & 4 deletions FundsXML_Files/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -122,17 +122,21 @@ Validation always targets the **official release** of the schema:
https://github.com/fundsxml/schema/releases/download/<version>/FundsXML.xsd
```

Two enterprise-relevant caveats are handled by the helper `tools/fetch-schema.sh`:
Two enterprise-relevant caveats are handled by every example's in-language
schema resolver (and by the `fundsxml_schema` module shown below):

1. That URL returns an HTTP 302 redirect; simple HTTP clients (libxml2 /
xmllint) do not follow it, so the schema must be fetched first (curl honours
`https_proxy`/`HTTPS_PROXY` for locked-down networks).
xmllint) do not follow it, so the schema must be materialised first.
2. From release 4.2.9 on, `FundsXML.xsd` imports `xmldsig-core-schema.xsd` via a
relative path — both files must sit in the same directory.

The resolution order is the same everywhere: `$FUNDSXML_SCHEMA_DIR` (offline /
corporate-network escape hatch) → `.schema-cache/` → download from the
official release. No committed catalog.

```bash
# Fetches FundsXML.xsd (+ xmldsig-core-schema.xsd when needed) into .schema-cache/<version>/
tools/fetch-schema.sh 4.2.9
python -m fundsxml_schema 4.2.9 # caches the XSD into .schema-cache/ (run `pip install -e .` once; cross-platform)
```

### Validate with xmllint (macOS/Linux)
Expand Down
10 changes: 6 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ own README, and is exercised by the CI workflow on each push.
| Company-internal DQ rules | XSLT 2.0 | [XSLT_DataQuality_Checks/Custom_Internal_Checks/](./XSLT_DataQuality_Checks/) | ✅ |
| Factsheet (HTML/PDF) & CSV export | XSLT, XSL-FO/FOP | [XSLT_Transformations/](./XSLT_Transformations/) | ✅ |
| Transformation invocation | CLI, Python, Java, .NET, Node | [XSLT_Transformations/invocation/](./XSLT_Transformations/) | ✅ |
| Schema fetch (proxy-aware) | Bash | [tools/fetch-schema.sh](./tools/fetch-schema.sh) | ✅ |
| Schema resolver (env / cache / official download) | Per-language, in-example + [tools/fundsxml_schema.py](./tools/fundsxml_schema.py) | every stack | ✅ |
| CI (validate all samples) | GitHub Actions | [.github/workflows/ci.yml](./.github/workflows/) | ✅ |
| XQuery analytics (aggregation, top-holdings, look-through) | Saxon CLI/Java, Python, BaseX | [XQuery_Examples/](./XQuery_Examples/) | ✅ |
| XML signature sign/verify | Apache Santuario (Java), .NET, xmlsec1, signxml | [XML_Signature/](./XML_Signature/) | ✅ |
Expand All @@ -50,9 +50,11 @@ fundsxml_examples/
│ # wrapper: builds all Java
│ # examples standalone (deps from
│ # Maven Central), no preinstall
├── tools/ # fetch-schema.sh (proxy-aware XSD
│ # fetcher; Java examples also
│ # resolve the XSD themselves)
├── pyproject.toml # Python deps + the
│ # fundsxml_schema resolver module
├── tools/fundsxml_schema.py # shared in-language XSD resolver
│ # (env / cache / official URL);
│ # every stack resolves it itself
├── FundsXML_Files/ # Sample documents, per version & use-case
│ ├── 4.2.9/{positions,transactions,documents,regulatory,signed}/
Expand Down
Loading
Loading