ci: bound the apt steps and give every job a timeout - #80
Merged
Conversation
A stalled Azure apt mirror left "Install GUI build dependencies" retrying for 36 minutes before the run was cancelled; the job normally finishes in five. Nothing bounded it - apt waits two minutes per repository by default and no job in either workflow set timeout-minutes, so GitHub's six-hour default applied. Make apt fail over to the next mirror in seconds, cap the three apt steps at six minutes, and cap each CI job well above a cold-cache run.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The post-merge run on
mainspent 36 minutes in "Install GUI build dependencies" before it was cancelled. The same job takes 4m45s–7m40s normally, and every other job in that run finished in under 2.5 minutes — so this is one stalled mirror, not slow CI.What happened. The runner image points apt at
azure.archive.ubuntu.com, which black-holed connections on that runner. apt's defaults areAcquire::http::Timeout120s with several retries per repository, so it worked through every repo twice over before falling back toarchive.ubuntu.com— which then succeeded. Nothing bounded any of it: neither workflow setstimeout-minuteson a single job or step, so GitHub's six-hour default was the only limit.Fix.
apt.conf.dsnippet before eachapt-get update—Acquire::Retries 2, HTTP/HTTPS timeouts 15s — so a dead mirror fails over in seconds instead of minutes. The happy path is unchanged; the Azure mirror is co-located with the runners and stays the first choice.timeout-minutes: 6on the three apt steps (ci.ymlGUI deps,release.ymlcross linker and Linux bundle deps).timeout-minuteson everyci.ymljob — 10 for the hygiene guard, 30 for the rest, which is well clear of a cold-cache build but ends a stall in minutes rather than hours.No step's behaviour changes when the network is healthy.