fix: count a release as tested when only skipped tests remain - #66
Merged
Conversation
Tests filtered by an unmet assumption - an absent optional dependency, a documented gap - are reported as skipped rather than failed. Requiring zero skips meant a suite that passed everything it actually ran still disqualified its release. Errors and failures still disqualify a release. When no release qualified, the task returned without writing the badge and without reporting anything, so the caller only found out when a later step tripped over the missing file. Assert instead, and point at the reports. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Contributor
Test Results27 tests 27 ✅ 21s ⏱️ Results for commit bfed9a7. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
matbox.tasks.createTestedWithBadgeforToolboxonly listed a MATLAB release in the "tested with" badge when its report had zero errors, zero failures and zero skipped tests. Skips are not failures: they are tests the suite deliberately filtered via an unmet assumption — an optional dependency that is absent, a documented gap — so a release whose suite passed everything it actually ran was still disqualified.When that left no qualifying release, the function returned without writing the badge and without reporting anything. The caller only discovered the problem later, when a step that expected the file tripped over its absence.
Both halves bit openMINDS_MATLAB's v0.10.0 release (run 33856204342). All nine releases R2022a–R2026a passed with 0 errors and 0 failures, but each had 4 skips (3 in an optional Python interop suite, 1 a documented round-trip gap). No badge was written, and the release failed two steps later with:
Change
releasesFailed— which drives the badge colour — now counts only genuinely broken releases.Tests
Three cases added to
matboxtools.unittest.TasksTest, with local helpers that write a minimal JUnit report per release and read the badge back:testTestedWithBadgeCountsReleaseWithSkippedTests— a skipped-only release and a clean release both list; green.testTestedWithBadgeExcludesFailingRelease— a failing release is excluded while a skipped-only one still lists; orange.testTestedWithBadgeErrorsWhenNoReleasePassed— raisesMATBOX:BadgeCreation:NoReleasePassed.The first two fail against
mainand pass here; the second fails onmainwith "Expected 'tested with' badge file was not created", which is the openMINDS symptom exactly.Replaying the nine real report artifacts from that failed run through the fixed task now yields:
{"schemaVersion":1,"label":"tested with","message":"R2022a | R2022b | R2023a | R2023b | R2024a | R2024b | R2025a | R2025b | R2026a","color":"green"}Note for downstream
openMINDS_MATLAB's release workflow installs MatBox
latest, so this fix only reaches it once a new MatBox release is cut.🤖 Generated with Claude Code