ci: cache apt metadata in the mirror proxy (ubuntu + debian)#1112
Draft
cpuguy83 wants to merge 2 commits into
Draft
ci: cache apt metadata in the mirror proxy (ubuntu + debian)#1112cpuguy83 wants to merge 2 commits into
cpuguy83 wants to merge 2 commits into
Conversation
The runner spoofed archive.ubuntu.com and security.ubuntu.com to a single azure.archive.ubuntu.com IP so container builds would use the faster azure mirror. That mirror is now intermittently unreachable, and DNS alone can't recover from it: apt only retries another address on a TCP connect failure, not on timeouts or 5xx, and azure's Traffic Manager returns a single IP per lookup, so there is no sibling to fall back to. Point the spoof at a local nginx reverse proxy instead. It proxies to the azure mirror first and fails over to us.archive.ubuntu.com on connect errors, timeouts, and 5xx/429 responses, covering the failure modes DNS misses so a flaky primary no longer breaks apt. Both mirrors serve the archive and security pockets by path, so forwarding the original Host through one proxy covers both names. dnsmasq matches subdomains, which would otherwise redirect the mirror hostnames the proxy dials back to the proxy itself; more-specific rules keep those resolving through real DNS to avoid the loop. Trade-off: this adds an nginx install and an extra hop on the runner, in exchange for surviving an intermittent upstream mirror. Signed-off-by: Brian Goff <cpuguy83@gmail.com>
cpuguy83
force-pushed
the
cpuguy83/apt-mirror-proxy-cache
branch
from
a9291c4 to
34178ea
Compare
The CI apt mirror proxy now caches repository metadata (Release/Packages indexes) so the many apt updates across a test run reuse one fetch instead of re-downloading indexes over the network every time. Package downloads stream through uncached, since buildkit already caches those within a run. Debian mirrors (deb.debian.org, security.debian.org) are added alongside the existing Ubuntu archive/security mirrors so Debian-based suites benefit from the same metadata caching. The Debian upstream uses a fresh connection per request rather than pooled keepalive: the fastly CDN drops idle connections quickly, and reusing a half-dead pooled connection caused package downloads to fail mid-stream with a broken pipe, failing the whole install because apt is run without retries. Signed-off-by: Brian Goff <cpuguy83@gmail.com>
cpuguy83
force-pushed
the
cpuguy83/apt-mirror-proxy-cache
branch
2 times, most recently
from
e653449 to
73f538e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #1106 — the first commit here is that PR; review/merge #1106 first. Until it merges this PR will show both commits, then collapse to just the caching change.
What
Builds on the CI nginx mirror proxy from #1106 to:
apt updates in a build no longer re-download the same indexes over the network every time — the proxy serves them from a local cache for the life of the runner. Package files (*.deb/*.udeb/*.ddeb) stream through uncached: runners are ephemeral and buildkit already caches packages, so only metadata is worth caching. Packages still go through the proxy, so they keep ci: fail over between ubuntu mirrors via a reverse proxy #1106's mirror failover.deb.debian.organdsecurity.debian.orgare now proxied/cached too (not just ubuntu), so debian-based builds get the same benefit. One Host-forwarding backend covers/debianand/debian-security.How it stays consistent
A fixed 60m TTL is forced (
proxy_ignore_headers Cache-Control Expires) so a mirror sendingno-cacheonInReleasecan't defeat the cache. This is safe because a runner is short-lived and apt fetchesPackagesvia immutable by-hash URLs, so the cached index snapshot stays internally consistent for the run.Trade-offs
us.archivebackup is safe, but the main debian archive mirrors don't carry/debian-security, so a backup there would 404 security requests.deb.debian.org(fastly) is reliable enough alone.server=exception fordeb.debian.org. Its upstream name is the spoofed name; nginx resolves it once at config-load (before the spoof is active) and keeps the real IP for the run, so there's no proxy loop and containers still get redirected.Validation
nginx -t: syntax ok / test successful.Hostheaders:InRelease×2 → 200 MISS → HIT/debian×2 → 200 MISS → HIT/debian-security→ 200;security.debian.orghost → 200.deb×2 → 200 BYPASS (never cached)