Skip to content

ci(hts): exclude server-private tx-ecosystem tests from IG gate#136

Merged
smunini merged 1 commit into
mainfrom
ci/ignore-server-private-tx-ecosystem-tests
Jun 8, 2026
Merged

ci(hts): exclude server-private tx-ecosystem tests from IG gate#136
smunini merged 1 commit into
mainfrom
ci/ignore-server-private-tx-ecosystem-tests

Conversation

@smunini

@smunini smunini commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

Problem

Nightly run 27116181098 (HTS Terminology IG Conformance) failed all four legs (sqlite/postgres × R4/R5) at Assert all tx-ecosystem tests passed, each on a single test:

version/vs-expand-versionless: Fail
  number property values differ at .expansion.total
  Expected:"2384"  Actual:"0"

Root cause — upstream test, not our code

The workflow checks out HL7/fhir-tx-ecosystem-ig at its default-branch HEAD with no pinned ref, so a nightly run executes whatever tests exist at that moment.

  • version/vs-expand-versionless was added upstream at 04:04 UTC on Jun 8 (HL7/fhir-tx-ecosystem-ig@e187e74); the scheduled run started 04:32 UTC, ~28 min later, and picked it up. The Jun 7 run passed because the test didn't exist yet. No HFS commit between the two runs touched HTS expand logic.
  • The test expands a versionless include of the whole NCI Thesaurus while supplying only a 2-concept fragment tx-resource. The expected response is 2384 codes (with a valueset-unclosed extension and used-fragment parameter) that tx.fhir.org enumerates from its own server-side-loaded NCIt fragment. Those codes exist in neither the IG fixtures nor the request body, so no server lacking that private content can reproduce total=2384 — it's tagged mode: general but is effectively tx.fhir.org-specific.

Fix

Added an audited IGNORED_TESTS allowlist (job-level env) for upstream tests that depend on terminology content the IG doesn't ship, with the rationale documented inline next to the entry — same spirit as the existing ConceptMap-novs.json fixture skip. Both the Summarize and Assert steps:

  • exclude listed tests from the failure count (the gate keys off the post-exclusion count);
  • surface them separately as Ignored in the step summary, and log when an ignored test did fail so a stale entry (one that starts passing) stays visible and can be pruned.

Verification

Run against the actual failing report.json artifact from the broken run:

  • With the entry → raw failures 1gate FAILED=0, passes; Ignored=1 reported.
  • Empty / comment-only / unset allowlist → still FAILED=1 — the gate can't be silently neutered, and any other real failure still fails the build.
  • Workflow YAML validated.

Follow-up (not in this PR)

Consider pinning the IG checkout to a tag/SHA so upstream test additions can't break a green main overnight. Happy to add it here or in a separate PR.

The nightly HTS IG conformance suite checks out HL7/fhir-tx-ecosystem-ig at
its default-branch HEAD (no pinned ref), so it runs whatever tests exist at
run time. On 2026-06-08 the suite picked up a newly-added test
`version/vs-expand-versionless` (added upstream ~28 min before the run in
HL7/fhir-tx-ecosystem-ig@e187e74) and failed all four legs.

That test expands a versionless include of the whole NCI Thesaurus while
supplying only a 2-concept fragment tx-resource. The expected response is
2384 codes (with a valueset-unclosed extension and used-fragment parameter)
that tx.fhir.org enumerates from its own server-side-loaded NCIt fragment.
Those codes exist in neither the IG fixtures nor the request body, so no
server lacking that private content can reproduce total=2384 — it is tagged
mode=general but is effectively tx.fhir.org-specific.

Add an audited IGNORED_TESTS allowlist (job-level env) for upstream tests
that depend on terminology content the IG does not ship, with the rationale
documented inline. The Summarize and Assert steps exclude listed tests from
the failure count and surface them separately as 'Ignored', logging when an
ignored test did fail so a stale entry that starts passing stays visible.
The gate still fails for any other real failure and when the allowlist is
empty, so it cannot be silently neutered.
@smunini smunini merged commit a381ce4 into main Jun 8, 2026
10 checks passed
@codecov

codecov Bot commented Jun 8, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant