ci: fail over between ubuntu mirrors via a reverse proxy#1106
Draft
cpuguy83 wants to merge 1 commit into
Draft
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR hardens the CI infrastructure that makes container builds pull Ubuntu packages from a fast mirror. Previously, the dns-spoof-ubuntu-archive composite action resolved archive.ubuntu.com/security.ubuntu.com directly to a single azure.archive.ubuntu.com IP. Since that mirror is now intermittently unreachable and apt only retries another address on TCP connect failures (not timeouts/5xx), a flaky azure mirror could break apt. The action now points the DNS spoof at a local nginx reverse proxy that proxies to azure first and fails over to us.archive.ubuntu.com, with dnsmasq server= overrides to prevent the proxy's own upstream hostnames from looping back through the spoof.
Changes:
- Replaced the direct azure-IP DNS spoof with a local nginx reverse proxy (azure primary +
us.archivebackup) and moved the dnsmasq/nginx install to the front so upstream resolution happens before the spoof is active. - Added more-specific dnsmasq
server=rules forazure.archive.ubuntu.com/us.archive.ubuntu.comto avoid a self-proxy loop, and repointedarchive/securityspoofs at the runner's own IP. - Renamed the corresponding CI step from "Use azure ubuntu archive" to "Use ubuntu archive failover proxy" in both job locations.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
.github/actions/dns-spoof-ubuntu-archive/action.yml |
Adds nginx reverse proxy with failover, reorders install step, repoints dnsmasq spoof to the local proxy and adds loop-prevention server= rules. |
.github/workflows/ci.yml |
Renames the two step usages to reflect the new failover-proxy behavior (no functional/path changes). |
Comment on lines
+65
to
+66
| proxy_next_upstream_tries 2; | ||
| proxy_next_upstream_timeout 20s; |
The runner spoofed archive.ubuntu.com and security.ubuntu.com to a single azure.archive.ubuntu.com IP so container builds would use the faster azure mirror. That mirror is now intermittently unreachable, and DNS alone can't recover from it: apt only retries another address on a TCP connect failure, not on timeouts or 5xx, and azure's Traffic Manager returns a single IP per lookup, so there is no sibling to fall back to. Point the spoof at a local nginx reverse proxy instead. It proxies to the azure mirror first and fails over to us.archive.ubuntu.com on connect errors, timeouts, and 5xx/429 responses, covering the failure modes DNS misses so a flaky primary no longer breaks apt. Both mirrors serve the archive and security pockets by path, so forwarding the original Host through one proxy covers both names. dnsmasq matches subdomains, which would otherwise redirect the mirror hostnames the proxy dials back to the proxy itself; more-specific rules keep those resolving through real DNS to avoid the loop. Trade-off: this adds an nginx install and an extra hop on the runner, in exchange for surviving an intermittent upstream mirror. Signed-off-by: Brian Goff <cpuguy83@gmail.com>
cpuguy83
force-pushed
the
cpuguy83/ubuntu-archive-failover-proxy
branch
from
4f4e6c6 to
61d55a2
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The runner spoofed
archive.ubuntu.comandsecurity.ubuntu.comto a singleazure.archive.ubuntu.comIP so container builds would use the faster azure mirror. That mirror is now intermittently unreachable, and DNS alone cannot recover from it: apt only retries another address on a TCP connect failure (not on timeouts or 5xx), and azure's Traffic Manager returns a single IP per lookup, so there is no sibling address to fall back to.This points the spoof at a local nginx reverse proxy instead. nginx proxies to the azure mirror first and fails over to
us.archive.ubuntu.comon connect errors, timeouts, and 5xx/429 responses — the failure modes plain DNS misses — so a flaky primary no longer breaks apt in container builds.Notes:
-securitypockets by path (host-agnostic), so forwarding the originalHostthrough a singledefault_servercovers botharchive.ubuntu.comandsecurity.ubuntu.com.azure.archive.ubuntu.com,us.archive.ubuntu.com) back to the proxy itself; more-specificserver=rules keep those resolving through real DNS to avoid a self-proxy loop.Trade-off: adds an nginx install and an extra hop on the runner, in exchange for surviving an intermittent upstream mirror.
Validated locally:
nginx -tsyntax ok,dnsmasq --testok, shellcheck clean on the embedded scripts.