Skip to content

fix: count a release as tested when only skipped tests remain - #66

Merged
ehennestad merged 2 commits into
mainfrom
relax-tested-with-badge-skip-criterion
Sep 4, 2026
Merged

fix: count a release as tested when only skipped tests remain#66
ehennestad merged 2 commits into
mainfrom
relax-tested-with-badge-skip-criterion

Conversation

@ehennestad

Copy link
Copy Markdown
Owner

Problem

matbox.tasks.createTestedWithBadgeforToolbox only listed a MATLAB release in the "tested with" badge when its report had zero errors, zero failures and zero skipped tests. Skips are not failures: they are tests the suite deliberately filtered via an unmet assumption — an optional dependency that is absent, a documented gap — so a release whose suite passed everything it actually ran was still disqualified.

When that left no qualifying release, the function returned without writing the badge and without reporting anything. The caller only discovered the problem later, when a step that expected the file tripped over its absence.

Both halves bit openMINDS_MATLAB's v0.10.0 release (run 33856204342). All nine releases R2022a–R2026a passed with 0 errors and 0 failures, but each had 4 skips (3 in an optional Python interop suite, 1 a documented round-trip gap). No badge was written, and the release failed two steps later with:

cp: cannot stat '.github/badges/v0.10.0/tested_with.json': No such file or directory

Change

  • A release counts as tested when nothing errored or failed. Errors and failures still disqualify it, and releasesFailed — which drives the badge colour — now counts only genuinely broken releases.
  • Assert when no release qualifies, naming the reports directory, instead of silently writing nothing.

Tests

Three cases added to matboxtools.unittest.TasksTest, with local helpers that write a minimal JUnit report per release and read the badge back:

  • testTestedWithBadgeCountsReleaseWithSkippedTests — a skipped-only release and a clean release both list; green.
  • testTestedWithBadgeExcludesFailingRelease — a failing release is excluded while a skipped-only one still lists; orange.
  • testTestedWithBadgeErrorsWhenNoReleasePassed — raises MATBOX:BadgeCreation:NoReleasePassed.

The first two fail against main and pass here; the second fails on main with "Expected 'tested with' badge file was not created", which is the openMINDS symptom exactly.

Replaying the nine real report artifacts from that failed run through the fixed task now yields:

{"schemaVersion":1,"label":"tested with","message":"R2022a | R2022b | R2023a | R2023b | R2024a | R2024b | R2025a | R2025b | R2026a","color":"green"}

Note for downstream

openMINDS_MATLAB's release workflow installs MatBox latest, so this fix only reaches it once a new MatBox release is cut.

🤖 Generated with Claude Code

Tests filtered by an unmet assumption - an absent optional dependency, a
documented gap - are reported as skipped rather than failed. Requiring zero
skips meant a suite that passed everything it actually ran still disqualified
its release. Errors and failures still disqualify a release.

When no release qualified, the task returned without writing the badge and
without reporting anything, so the caller only found out when a later step
tripped over the missing file. Assert instead, and point at the reports.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Test Results

27 tests   27 ✅  21s ⏱️
 5 suites   0 💤
 1 files     0 ❌

Results for commit bfed9a7.

@ehennestad ehennestad left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good

@ehennestad
ehennestad merged commit b548997 into main Sep 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant