Skip to content

ci(release): treat OBS "finished" as build-complete, not busy - #151

Merged
mmaher88 merged 1 commit into
masterfrom
fix-obs-gate-on-build-not-publish
Jul 31, 2026
Merged

ci(release): treat OBS "finished" as build-complete, not busy#151
mmaher88 merged 1 commit into
masterfrom
fix-obs-gate-on-build-not-publish

Conversation

@mmaher88

Copy link
Copy Markdown
Owner

Problem

The trigger-obs gate failed on the v0.3.7 release even though every OBS build succeeded. Actual log from that run:

Guard delay before first poll (60s) so OBS can dispatch the trigger...
[phaseA 0s] rebuild observed running.          ← start correctly detected
[60s]  package codes: code="building"
[240s] package codes: code="building" code="finished"
[270s] package codes: code="finished"
       ... 31 minutes with no change ...
##[error]Timed out waiting for OBS builds to settle after 2100s.

is_busy() counted finished as non-terminal. In OBS, finished means the build is done — what remains is repository publishing, which is asynchronous and can lag arbitrarily. So the loop waited for a transition that wasn't coming on any useful timescale, and failed a release whose builds had all succeeded ~4 minutes in.

This is the tail of the #134 fix. Removing the old false-green exposed a pre-existing weakness in the busy-state list: previously the loop exited on its first poll and never sat long enough to hit it. Credit where due — the reviewer on #134 flagged exactly this risk ("if a real rebuild legitimately takes ~28-30 min, it could now hit MAX_WAIT and fail where it previously passed") and I judged it non-blocking. It reproduced on the second real release.

Fix

-  code="(building|blocked|scheduled|dispatching|finished|unknown|signing)"
+  code="(building|blocked|scheduled|dispatching|unknown|signing)"

The gate exists to answer "did THIS release build on OBS?" — and at finished that question is answered. Publish latency isn't a build failure.

  • signing stays busy: normally seconds, and reaching it already implies a successful build.
  • The failure assertion on failed|unresolvable|broken is unchanged, so genuine build failures still fail the release.
  • Added a log note when exiting at finished, so the output doesn't imply the download repo is immediately current — it can lag a few minutes.
  • Documented the v0.3.7 incident inline so finished doesn't get "helpfully" added back.

Rather than bumping MAX_WAIT again — which just moves the arbitrary guess — this removes the wait that shouldn't have existed.

Validation

YAML parses; full suite green via the pre-push gate. Real behavioural proof comes on the next release, same as #134.

The trigger-obs gate failed on v0.3.7 despite every build succeeding. The
wait loop counted "finished" as a non-terminal state, but in OBS that means
the build is done and only repository publishing remains — which is
asynchronous and can lag arbitrarily.

Timeline from the v0.3.7 run: the rebuild was correctly detected as started,
went "building" -> "finished" at ~4 minutes, then sat at "finished" for the
remaining 31 until MAX_WAIT expired and failed the release.

Publish latency is not a build failure. Drop "finished" from is_busy so the
gate answers the question it exists to answer — did THIS release build on
OBS — and log a note that the download repo may still lag. "signing" stays
busy: it is normally seconds and reaching it already implies success.
@mmaher88
mmaher88 merged commit 6c1d66f into master Jul 31, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant