ci(hts): exclude server-private tx-ecosystem tests from IG gate#136
Merged
Conversation
The nightly HTS IG conformance suite checks out HL7/fhir-tx-ecosystem-ig at its default-branch HEAD (no pinned ref), so it runs whatever tests exist at run time. On 2026-06-08 the suite picked up a newly-added test `version/vs-expand-versionless` (added upstream ~28 min before the run in HL7/fhir-tx-ecosystem-ig@e187e74) and failed all four legs. That test expands a versionless include of the whole NCI Thesaurus while supplying only a 2-concept fragment tx-resource. The expected response is 2384 codes (with a valueset-unclosed extension and used-fragment parameter) that tx.fhir.org enumerates from its own server-side-loaded NCIt fragment. Those codes exist in neither the IG fixtures nor the request body, so no server lacking that private content can reproduce total=2384 — it is tagged mode=general but is effectively tx.fhir.org-specific. Add an audited IGNORED_TESTS allowlist (job-level env) for upstream tests that depend on terminology content the IG does not ship, with the rationale documented inline. The Summarize and Assert steps exclude listed tests from the failure count and surface them separately as 'Ignored', logging when an ignored test did fail so a stale entry that starts passing stays visible. The gate still fails for any other real failure and when the allowlist is empty, so it cannot be silently neutered.
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Nightly run 27116181098 (HTS Terminology IG Conformance) failed all four legs (sqlite/postgres × R4/R5) at Assert all tx-ecosystem tests passed, each on a single test:
Root cause — upstream test, not our code
The workflow checks out
HL7/fhir-tx-ecosystem-igat its default-branch HEAD with no pinned ref, so a nightly run executes whatever tests exist at that moment.version/vs-expand-versionlesswas added upstream at 04:04 UTC on Jun 8 (HL7/fhir-tx-ecosystem-ig@e187e74); the scheduled run started 04:32 UTC, ~28 min later, and picked it up. The Jun 7 run passed because the test didn't exist yet. No HFS commit between the two runs touched HTS expand logic.fragmenttx-resource. The expected response is 2384 codes (with avalueset-unclosedextension andused-fragmentparameter) that tx.fhir.org enumerates from its own server-side-loaded NCIt fragment. Those codes exist in neither the IG fixtures nor the request body, so no server lacking that private content can reproducetotal=2384— it's taggedmode: generalbut is effectively tx.fhir.org-specific.Fix
Added an audited
IGNORED_TESTSallowlist (job-level env) for upstream tests that depend on terminology content the IG doesn't ship, with the rationale documented inline next to the entry — same spirit as the existingConceptMap-novs.jsonfixture skip. Both the Summarize and Assert steps:Verification
Run against the actual failing
report.jsonartifact from the broken run:1→ gateFAILED=0, passes;Ignored=1reported.FAILED=1— the gate can't be silently neutered, and any other real failure still fails the build.Follow-up (not in this PR)
Consider pinning the IG checkout to a tag/SHA so upstream test additions can't break a green
mainovernight. Happy to add it here or in a separate PR.