build.sh: translate workspace bind mounts for a shared Docker daemon (opt-in)#53
Merged
Conversation
…(opt-in) When build.sh runs against a per-node *shared* Docker daemon (rehosting CI's shared-docker runners) rather than a per-pod dind, the daemon does not share this runner's mount namespace. Its `-v "$PWD":/app` and `-v "$CACHE_HOST_DIR":/tmp/build` bind sources live under the runner's per-pod /home/runner/_work, which the shared daemon cannot see, so /app mounts empty and the build dies with: bash: /app/_in_container_build.sh: No such file or directory (exit 127) Add rewrite_mount(): when PENGUIN_HOST_MOUNT_FROM/PENGUIN_HOST_MOUNT_TO are BOTH set (exported by the shared-daemon runner; the same mechanism penguin's wrapper uses), rewrite a bind source under _FROM to the daemon-visible _TO path. When they're unset — every local build and every GitHub-hosted runner — it returns the path unchanged, so behaviour is byte-for-byte identical to before (pure opt-in). Applies to the workspace + cache binds only; the kernel-source cache (under /home/runner/_shared, passed via --extra-docker-opts) is outside _FROM and left as-is — the shared daemon mirrors that node-shared path directly (rehosting/kube#15).
The build image was built per-target (FROM embedded-toolchains:${TARGET}),
which was a pull-cost optimization. On the shared per-node Docker daemon
that backfires: 13 matrix jobs build+tag one mutable rehosting/linux_builder
:latest on the shared image store, so a job can `docker run` an image built
for another target and fail with its cross-compiler missing
(e.g. /opt/cross/mips64el-linux-musl/bin/...-gcc: No such file).
Build FROM the all-arch embedded-toolchains:latest instead. The shared
daemon loads that base once per node and reuses it warm, so all-toolchains
is cheap now — and every built image is target-agnostic, so the shared
:latest tag can't hand a job an image lacking its toolchain. Kernel target
is still selected per-job via build.sh --targets. Build cache collapses to
a single shared ref (was per-target).
Previously every matrix build job ran its own docker buildx build of the kernel_builder image. Since the image went target-agnostic (TARGET=latest bundles every cross-toolchain), that rebuilt the SAME multi-GB image ~13x per run. Worse, setup-buildx-action's docker-container driver keeps its own cache store separate from the shared per-node Docker daemon, so the daemon's warm-image reuse never covered the buildx FROM base -- each job re-pulled the full all-arch base -- and 13 jobs racing cache-to on one :all_cache ref meant the registry cache never accumulated cleanly. Add a dedicated build-image job that builds the all-arch image once, pushes it to Harbor under an immutable per-commit tag, and is the sole writer of the :all_cache registry cache. The build matrix now needs [prebuild, build-image] and simply docker-pulls that tag (warm on the shared daemon after the first job lands on a node) then runs build.sh --image <tag>. Immutable tag also sidesteps the shared-daemon mutable-:latest race (cf. penguin #893).
The matrix build jobs recompiled every kernel from scratch each run (~15-20 min/target): builds are out-of-tree into /tmp/build, which build.sh backs with $PWD/cache -- the per-job workspace, wiped every run -- so the incremental tree never survived. The base image already ships ccache. Enable ccache on a persistent, node-shared dir: - _in_container_build.sh: when CCACHE_DIR is set, route the kernel/modules/perf compiles through CC="ccache <cross>gcc". When unset, CC_PREFIX is empty and each CC= expands to exactly the kernel default ($(CROSS_COMPILE)gcc), so the behaviour is byte-for-byte unchanged for local/other callers. Sets sane ccache knobs (15G, compress, time_macros sloppiness) and prints start/end stats for observability. - build.yml: mount /home/runner/_shared/linux_builder_ccache (same node-shared mount the kernel sources use, mirrored into the shared daemon by kube#15) at /ccache and set CCACHE_DIR. One dir shared by every target/version on the node; ccache is content-keyed + internally locked, so cross-target/cross-job sharing is safe and correct across config and source bumps. First run warms the cache (cold, same speed as today); subsequent runs on a node should drop to a mostly-cache-hit rebuild.
The end-of-run ccache summary sat after the kernel-devel `exit 0`, which is the default CI path, so it never printed. Move it into the EXIT trap (renamed fix_ownership -> on_exit) so it fires on every exit path. Observability only; ccache itself was already warming during the compiles.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
After rehosting CI moved
rehosting-arconto a per-node shared Docker daemon (rehosting/kube#14), linux_builder kernel builds fail.build.shbind-mounts the workspace and cache into the build container:Those sources live under the runner's per-pod
/home/runner/_work, which the shared daemon can't see (it has no per-pod view of that path). So/appmounts empty and the build dies:(Verified on a dispatch build: kube#15 fixed the kernel-source bind under
/home/runner/_shared, and the build then got exactly this far and failed on the workspace bind.)Fix — opt-in, pure no-op otherwise
Add
rewrite_mount(). It rewrites a bind source only when bothPENGUIN_HOST_MOUNT_FROMandPENGUIN_HOST_MOUNT_TOare set (exported by the shared-daemon runner — the same mechanism penguin's wrapper uses) and the path is under_FROM, translating it to the daemon-visible_TOlocation. When the vars are unset — every local build and every GitHub-hosted runner — it returns the path unchanged, so behaviour is byte-for-byte identical to today.Scope: workspace + cache binds only. The kernel-source cache (under
/home/runner/_shared, passed via--extra-docker-opts) is outside_FROMand left as-is; the shared daemon mirrors that node-shared path directly (kube#15).Verified: with the vars unset the two bind sources are unchanged; with them set,
/home/runner/_work/...→/shared-runner-work/<pod>/...and the_sharedsource is untouched.Validation
Once merged I'll re-run a dispatch build on the shared daemon to confirm the kernel matrix compiles end-to-end.