CLI flag to limit the number of concurrently building targets in buildx bake / docker compose build

[hmadadian]

Description

Description

There is currently no per-invocation, command-line way to limit build concurrency when building multiple targets with docker buildx bake (or docker compose build, which since Compose v5.0.0 delegates all builds to Bake). All targets are dispatched to the builder at once, and the only existing control (max-parallelism in the BuildKit worker configuration) must be baked into the builder at creation time.

I am requesting a client-side concurrency limit, e.g.:

docker buildx bake --max-parallelism 2 -f docker-compose.yml
docker compose build --max-parallelism 2

that caps how many targets (build requests) are in flight simultaneously, independent of which driver or daemon configuration is in use.

Background / motivating use case

CI pipeline (Jenkins on Kubernetes, ephemeral Docker-in-Docker pods) building multi-service applications from a Compose file. Each service is a multi-stage Dockerfile with resource-heavy compile steps (webpack/node, .NET, etc.).

Different applications sharing the same pipeline have very different resource profiles, so the desired parallelism is a per-project, per-invocation value supplied by the CI system — not a property of the builder or the daemon.

With the historical Compose builder this was trivial:

COMPOSE_PARALLEL_LIMIT=2 docker compose build

That capability has been progressively removed:

1. Compose ≥ v2.34 defaults to the Bake backend; COMPOSE_PARALLEL_LIMIT / --parallel no longer affect builds ([BUG] COMPOSE_PARALLEL_LIMIT Is no longer limiting the number of concurrent builds compose#13043, Multiple issues with parallel processing in latest Docker Desktop for Windows for-win#14889).
2. COMPOSE_BAKE=false was deprecated in Compose v2.39.0 (warn user COMPOSE_BAKE=false is deprecated compose#13065).
3. Compose v5.0.0 removed the internal builder entirely (drop support for internal buildkit builder compose#13056), so the last escape hatch is gone.

There is now no supported equivalent of the old behavior anywhere in the CLI surface.

Current workaround and why it is insufficient

The documented approach is to set worker-level parallelism when creating a builder:

docker buildx create --name ci --driver docker-container \
  --buildkitd-flags '--oci-max-parallelism=2' --bootstrap --use

It has significant drawbacks:

• Not per-invocation. The value is frozen at builder-creation time. Changing it means destroying and recreating the builder (losing its local cache) or pre-provisioning one builder per concurrency tier and selecting with --builder.
• Not available on the default docker driver. The BuildKit instance embedded in dockerd exposes no max-parallelism in daemon.json and no dockerd flag. Users on the default driver have no option at all.
• Forces the docker-container driver in environments where its costs are highest. In ephemeral CI (DinD pods) this means creating a builder per job and paying the --load export/import penalty (minutes for large images) or restructuring pipelines around --push, purely to obtain a concurrency limit.

Proposal

1. --max-parallelism <N> on docker buildx bake (and build for completeness)

Client-side scheduling: Bake already computes the target dependency graph (contexts = { x = "target:y" }); instead of dispatching all ready targets at once, dispatch at most N concurrently. Because this is done in the CLI before solves are submitted:

• it works with every driver, including the default docker driver;
• it requires no daemon reconfiguration or builder lifecycle changes;
• it composes with (does not replace) worker-level max-parallelism, which remains the right tool for capping step concurrency inside a single large multi-stage build.

2. Bake-file attribute for per-group/per-target defaults

group "default" {
  targets = ["app-a", "app-b", "app-c"]
  max-parallelism = 2
}

with the CLI flag taking precedence, consistent with how other bake attributes are overridden via --set.

3. Compose passthrough

docker compose build --max-parallelism N (and/or reviving COMPOSE_PARALLEL_LIMIT mapped onto it), restoring pre-Bake parity now that Compose v5 has no internal builder.

Prior art / related issues

• Add Concurrency Control to Matrix Builds #2581 — Add concurrency control to matrix builds (open; proposes --max-parallel and a per-target concurrency attribute)
• Add a per-build flag to limit concurrency / parallelism moby/buildkit#6038 — Add a per-build flag to limit concurrency/parallelism (open; notes the support memory driver opt for docker-container driver #2049 daemon-wide config "does not meet this need")
• specifiy max parallelism for multi-stage build moby/buildkit#5981 — Specify max parallelism for multi-stage build (open)
• Limit concurrent build requests on a single BuildKit pod #3006 — Limit concurrent build requests on a single BuildKit pod (open; worker max-parallelism does not cap accepted build requests)
• [BUG] COMPOSE_PARALLEL_LIMIT Is no longer limiting the number of concurrent builds compose#13043 — COMPOSE_PARALLEL_LIMIT no longer limiting concurrent builds
• Multiple issues with parallel processing in latest Docker Desktop for Windows for-win#14889 — All parallelism-limiting mechanisms ineffective once Bake is the builder (100 services building simultaneously)
• Add a configuration item to limit parallelism moby/buildkit#2049 — original daemon-wide max-parallelism config (shipped; explicitly daemon-scoped)

The recurring theme across these issues: the only shipped control is daemon-scoped and creation-time-static, while the demand is for an invocation-scoped, client-side limit.

Expected behavior

docker buildx bake --max-parallelism 1 -f compose.yml svc-a svc-b

builds svc-a and svc-b strictly sequentially (subject to dependency order), on any driver, without any builder or daemon reconfiguration. Default remains unlimited (current behavior), so the flag is fully backward compatible.

Environment where this was hit

• Docker Engine 28.3.x (DinD), Buildx ≥ v0.23, Compose v5.3.1
• Jenkins on Kubernetes, ephemeral DinD build pods, one builder per job

[zumbuschms]

@hmadadian, is the custom builder with the parallization limit working for you? I've created a custom builder using the following command:

docker buildx create --name mybuilder --driver docker-container --buildkitd-flags '--oci-max-parallelism=4' --bootstrap --use

When I run docker compose build without defining a builder, Docker Desktop for Windows switches back to the default unrestricted builder. If I provide the builder via the CLI, my custom builder is used, but the parallelization limit is not taken into account and all 105 images from my Docker Compose project are built in parallel.

I use the latest Docker Desktop for Windows.

Client:
 Version:           29.6.2
 API version:       1.55
 Go version:        go1.26.5
 Git commit:        dfc4efb
 Built:             Thu Jul 16 16:14:59 2026
 OS/Arch:           windows/amd64
 Context:           desktop-linux

Server: Docker Desktop 4.85.0 (235549)
 Engine:
  Version:          29.6.2
  API version:      1.55 (minimum version 1.40)
  Go version:       go1.26.5
  Git commit:       3d80467
  Built:            Thu Jul 16 16:12:20 2026
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v2.2.5
  GitCommit:        e53c7c1516c3b2bff98eb76f1f4117477e6f4e66
 runc:
  Version:          1.3.6
  GitCommit:        v1.3.6-0-g491b69ba
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

[hmadadian]

@zumbuschms I am not using Docker Desktop; what I am working on currently is the Docker DIND:
I have created the build kit with the following command:
docker buildx create --name ci --driver docker-container --buildkitd-flags '--oci-max-parallelism=1' --bootstrap --use
Then, for the build, I am using the following command line:
docker compose --file ./docker-compose.yml build --builder ci service-a service-b service-c

Interestingly, it is intelligent enough that pulling the images and the rest of the non-resource-using operations in the Dockerfile will be done in parallel, but the RUN commands will be done one service at a time

[zumbuschms]

@hmadadian, thanks for that information. Much appreciated. With this I was able to get CPU and memory load under control with our Docker Compose projects that builds about 100 images. My builds are working with a parallelization of 1 in Docker Desktop now. I still have to apply the same settings to our CI environment, but this was a good start.

I would still prefer to not run all builders at the same time, as pulling or exporting all images at the same time might hit other bottlenecks or throttling, but for our case a custom builder seems to do the trick.