ci: redesign CI to scope builds/tests/images to changed modules - #626
Open
BjoernKarma wants to merge 2 commits into
Open
ci: redesign CI to scope builds/tests/images to changed modules#626BjoernKarma wants to merge 2 commits into
BjoernKarma wants to merge 2 commits into
Conversation
Splits the monolithic per-module reusable workflow into two composable reusable components and adds module-scoped execution to cut CI compute as the monorepo has grown to 20+ modules: - .github/ci/modules.yaml: single source of truth for per-module CI config, replacing hardcoded per-module blocks in ci.yaml. - .github/ci/module-graph.json + hack/ci/gen-module-graph.sh: dependency graph derived from go.mod replace directives, enabling an "optimized" mode that also runs dependent modules. make ci-graph regenerates it, make ci-graph-check (wired into hack/verify.sh and CI) fails on drift. - hack/ci/select-scope.sh: computes run mode (maximum/minimum/optimized/ none) and resulting build/package module lists, with a global-impact safety valve forcing maximum mode for shared tooling/config changes. - build-component.yaml / package-component.yaml: reusable workflows replacing reusable-go-ci.yaml, split into build (lint/test/coverage/ govulncheck) and package (ko build/trivy) concerns. - codeql.yaml: single repo-wide CodeQL analysis replacing per-module jobs. - ci.yaml: rewritten orchestrator: determine-scope -> build (matrix) -> collect_build_status (artifact fan-in) -> package (matrix, filtered) -> ci_summary. Mode: pull_request->optimized, workflow_dispatch->choice, tag push->none, main push/schedule->maximum. - workflows.md updated to document the new pipeline. Preserves existing tag-push behavior (near no-op, Helm publish only) per team decision, and fixes the nightly schedule (previously ran almost nothing due to a stale condition) to run full maximum-mode verification.
BjoernKarma
added
the
github_actions
Contributor
There was a problem hiding this comment.
Pull request overview
Redesigns the monorepo GitHub Actions CI to dynamically scope build/test/package work to the set of changed modules (and optionally their dependents), using a module config file plus a generated cross-module dependency graph to reduce CI compute as the repo grows.
Changes:
- Introduces module-scoped CI orchestration in
ci.yamldriven by.github/ci/modules.yamland.github/ci/module-graph.json, computed via.github/scripts/select-scope.sh. - Splits the previous reusable per-module CI workflow into two reusable components: build (lint/tests/govulncheck + status marker) and package (ko build + trivy).
- Adds repo-wide CodeQL workflow and documentation updates, plus make/verify targets to regenerate/check the module graph.
Reviewed changes
Copilot reviewed 13 out of 13 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| REUSE.toml | Adds REUSE annotation for generated module-graph.json. |
| Makefile | Adds ci-graph / ci-graph-check targets for graph regeneration and drift checks. |
| hack/verify.sh | Enforces make ci-graph-check as part of repo-level verification. |
| .github/workflows/reusable-go-ci.yaml | Removes the previous monolithic reusable per-module CI workflow. |
| .github/workflows/component-build.yaml | New reusable build workflow (lint/tests/coverage/govulncheck + per-module status marker artifact). |
| .github/workflows/component-package-build.yaml | New reusable packaging workflow (ko image build + trivy scan). |
| .github/workflows/codeql.yaml | Adds repo-wide CodeQL analysis workflow building all configured modules. |
| .github/workflows/ci.yaml | Rewrites CI orchestrator to determine scope → build matrix → fan-in status → package matrix → summary, plus existing rover-ctl and helm jobs. |
| .github/workflows.md | Updates workflow documentation to describe the new scoped pipeline and CodeQL strategy. |
| .github/scripts/select-scope.sh | New scoping script computing build/package module lists from mode + changed files + module graph. |
| .github/scripts/gen-module-graph.sh | New generator for module-graph.json derived from go.mod replace directives. |
| .github/ci/modules.yaml | New single source of truth for per-module CI configuration. |
| .github/ci/module-graph.json | New committed generated dependency graph used for optimized scoping. |
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+64
to
+68
| modules=$(yq -o=json '.modules[].path' .github/ci/modules.yaml | jq -r .) | ||
| while IFS= read -r module; do | ||
| [ -z "$module" ] && continue | ||
| echo "Building $module for CodeQL..." | ||
| (cd "$module" && go build ./...) |
Comment on lines
+119
to
+127
| check-module-graph: | ||
| name: Check Module Dependency Graph | ||
| runs-on: ubuntu-latest | ||
| steps: | ||
| - name: Checkout Code | ||
| uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | ||
| - name: Verify .github/ci/module-graph.json is up to date | ||
| # yq and jq are preinstalled on GitHub-hosted ubuntu runners. | ||
| run: make ci-graph-check |
Comment on lines
+76
to
+80
| if [ -n "$dep_name" ] && [ "$dep_name" != "$name" ]; then | ||
| deps+=("$dep_name") | ||
| fi | ||
| done < <(grep -oP "$GO_MODULE_PREFIX/\S+\s*=>\s*\K\.\.?/\S+" "$gomod" || true) | ||
| fi |
Comment on lines
+39
to
+50
| - `push` of a version tag (`v*`) → **none**: tags are created by the | ||
| separately, manually-triggered `release.yaml` workflow, which already | ||
| builds/tests/publishes everything via goreleaser+ko; re-running the | ||
| same work here would just duplicate it. | ||
| - `workflow_dispatch` → user-selectable mode (`maximum` by default). | ||
| - Regardless of mode, any change to shared tooling (`Makefile`, | ||
| `.golangci.yml`, `.ko.yaml`, `hack/**`, `.github/workflows/**`, | ||
| `.github/scripts/**`, `.github/ci/**`) forces **maximum**, since a | ||
| path-based diff can't safely reason about the blast radius of a | ||
| tooling change. | ||
| 2. **Check Module Dependency Graph** - fails if `.github/ci/module-graph.json` | ||
| is stale relative to the modules' `go.mod` files. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Splits the monolithic per-module reusable workflow into two composable reusable components and adds module-scoped execution to cut CI compute as the monorepo has grown to 20+ modules:
collect_build_status (artifact fan-in) -> package (matrix, filtered) -> ci_summary. Mode: pull_request->optimized, workflow_dispatch->choice, tag push->none, main push/schedule->maximum.
Preserves existing tag-push behavior (near no-op, Helm publish only) per team decision, and fixes the nightly schedule (previously ran almost nothing due to a stale condition) to run full maximum-mode verification.