Skip to content

Improve CI Docker cache reuse#1296

Open
lovasoa wants to merge 14 commits into
mainfrom
improve-ci-cache-and-dedup
Open

Improve CI Docker cache reuse#1296
lovasoa wants to merge 14 commits into
mainfrom
improve-ci-cache-and-dedup

Conversation

@lovasoa
Copy link
Copy Markdown
Collaborator

@lovasoa lovasoa commented Jun 2, 2026

Summary

  • Reuse the Linux test binaries and Playwright binary produced by compile_and_lint instead of recompiling sqlpage in the database and browser-test jobs.
  • Keep Windows incremental/local sqlpage artifacts in a PR/ref-scoped cache so source edits can reuse incremental compilation state without allowing PRs to write a trusted main cache.
  • Make Docker cache scopes explicit: PR caches write to PR-scoped BuildKit GHA scopes, while trusted main remains a restore-only Docker Hub fallback for PRs.
  • Move .cargo/config.toml into the Docker dependency warmup layer so the warmup and final build use the same LIBSQLITE3_FLAGS.
  • Keep .git/ out of the Docker context and include all Docker dependency-recipe inputs in the cache scope hash.

Current findings

  • The Rust job deduplication works. In CI run 26892512893, the DB matrix and Playwright jobs used uploaded binaries and did not run Cargo builds.
  • The Docker final-build warmup gap is fixed. In 26892512893, amd64-minimal build-project.sh compiled only sqlpage; the previous repeat compiles of libsqlite3-sys, sqlx-core-oldapi, sqlx-macros-oldapi, and sqlx-oldapi disappeared after copying .cargo/ before build-dependencies.sh.
  • Docker BuildKit type=gha layer reuse is still not enough for the expensive dependency layer. Same-commit rerun 26892512893, attempt 2, imported the recipe-scoped GHA cache but still ran build-dependencies.sh, including crate downloads and a full dependency compile.

Validation

  • git diff --check
  • actionlint -ignore 'SC2046' .github/workflows/ci.yml
  • docker buildx build --check --target builder .
  • CI run 26892512893: full workflow passed.
  • CI run 26892512893, attempt 2, job 79325419825: same-commit amd64-minimal rerun proved the Docker GHA cache import still misses the dependency layer.

Reverted experiments

  • BuildKit Cargo cache mounts were tested and reverted earlier. On same-SHA rerun they still downloaded hundreds of crates and compiled hundreds of units, worse than plain layer caching.
  • A dummy lib-target warmup was tested and removed. It did not stop SQLx recompilation; copying .cargo/config.toml before dependency warmup did.

@lovasoa lovasoa force-pushed the improve-ci-cache-and-dedup branch from 5ecd502 to 0adf3d4 Compare June 2, 2026 15:31
@lovasoa lovasoa force-pushed the improve-ci-cache-and-dedup branch from 0adf3d4 to 3c615a0 Compare June 2, 2026 15:35
@lovasoa lovasoa marked this pull request as ready for review June 3, 2026 12:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant