Skip to content

fix: recreate services when a config_files change triggers the deploy - #1613

Open
LoganRupe wants to merge 1 commit into
moghtech:mainfrom
LoganRupe:fix/config-files-force-recreate
Open

fix: recreate services when a config_files change triggers the deploy#1613
LoganRupe wants to merge 1 commit into
moghtech:mainfrom
LoganRupe:fix/config-files-force-recreate

Conversation

@LoganRupe

Copy link
Copy Markdown

Fixes the config_files case in #1381, and the same thing reported by @madereddy and @tylovejoy in #1205.

The problem

DeployStackIfChanged detects a config_files content change correctly and fires a deploy scoped to the declared services. That deploy runs a plain docker compose up -d <service>, which no-ops: only the content behind a bind mount moved, so the service definition is unchanged and compose reports the container up to date.

Komodo then records success and advances deployed_contents, so the change stays marked deployed against a container still running the old file, and every later diff reads "unchanged". It doesn't self-correct.

The change

Adds force_recreate to DeployStack and the periphery ComposeUp request, and appends --force-recreate when a config file drove the deploy.

DeployIfChangedAction::FullDeploy gains a force_recreate flag, and resolve_deploy_if_changed_action takes &Stack so it can use is_config_file to tell a config trigger from a compose or env one.

Two early returns come out of that function:

  • On the first (Redeploy, true) hit. Compose files are registered before config files, so a commit touching both returned on the compose file and never looked at the config file. It now accumulates and decides at the end.

  • When a file is absent from deployed_contents. Such a file was only just declared, and returning FullDeploy meant adding one config_files entry force-recreated the whole project. It's now treated as changed and scoped by its own services like any other changed file.

The genuinely global cases still work with no special handling, since compose and env files carry an empty services array and land in (Redeploy, true) either way. Services { deploy } needs no flag at all: that arm is only reachable from a config_files entry, because compose and env files are registered via StackFileDependency::full_redeploy with empty services.

Injection is skipped when extra_args already pins --force-recreate or --no-recreate, since compose hard-errors on that pair.

Why not "destroy before deploy"

That was suggested on both issues and I built it first, but it's unsafe for a scoped deploy. docker compose down -- <svc> cascades to dependents, and the following up -d <svc> restores only the target and its dependencies, leaving the dependents removed. On a five service test stack, down -- db took api and web with it, neither came back, and the update still reported success:

[Compose Down] probe-web  Removed    not requested
               probe-api  Removed    not requested
               probe-db   Removed
[Compose Up]   probe-db   Created/Started    only db returns

--force-recreate has no cascade: it recreates the named service and leaves both dependencies and dependents running.

Testing

Run against a 12 server fleet on 2.3.1, compose 5.4.0, on a throwaway stack with a health-gated root, a dependency chain, and one service with no edges. Commands were read from Log.command rather than inferred from container state.

Case Result
Redeploy + services=[svc], no graph edges only that service
Redeploy + services=[svc], mid-graph only that service, dependency and dependents untouched
Redeploy + services=[] whole project, health gates respected
Restart + services=[svc] restarted not recreated, content applied
compose file alone plain up -d, nothing forced
compose and config file in one commit forced
newly declared config file, services=[svc] only that service
newly declared config file, services=[] whole project
extra_args = ["--no-recreate"] no flag injected, no conflict, deploy succeeds
requires = "None" inert
combined deploy and restart both halves, independently scoped

Not covered

Swarm. docker stack deploy has no --force-recreate, and its destroy is docker stack rm, which is whole stack rather than service scoped. I didn't want to guess at the right behaviour there.

Only compose 5.4.0 was exercised.

One thing worth your call

force_recreate on DeployStack is new public API surface: it picks up a CLI flag, a TS type and an OpenAPI entry. That seemed reasonable since "deploy and force recreate" is useful on its own, but if you'd rather keep it internal I'm happy to rework how the trigger is carried.

…eploy

DeployStackIfChanged correctly detects a `config_files` content change and fires
a deploy scoped to the declared services. That deploy then runs a plain
`docker compose up -d <service>`, which is a no-op: only the content behind a
bind mount moved, so the service definition is unchanged and compose reports the
container up-to-date. Komodo records success and advances `deployed_contents`,
so the change is permanently marked deployed against a container still running
the old file, and every later diff reads "unchanged". Silent, and it never
self-corrects.

Adds `force_recreate` to DeployStack and the periphery ComposeUp request, and
appends `--force-recreate` when a config file drove the deploy.

`DeployIfChangedAction::FullDeploy` gains a `force_recreate` flag, and
`resolve_deploy_if_changed_action` takes `&Stack` so it can use `is_config_file`
to tell a config trigger from a compose/env one.

Two early returns are removed from that function:

  * on the first `(Redeploy, true)` hit. Compose files are registered before
    config files, so a commit touching both would return on the compose file and
    never consider the config file. It now accumulates and decides at the end.

  * when a file is absent from deployed_contents. Such a file was only just
    declared, and returning FullDeploy meant adding one `config_files` entry
    force-recreated the whole project. It is now simply treated as changed and
    scoped by its own `services` like any other changed file. The genuinely
    global cases still work with no special handling: compose and env files
    carry an empty `services` array, so they land in the `(Redeploy, true)` arm
    regardless.

`Services { deploy }` needs no flag. Compose and env files are registered via
`StackFileDependency::full_redeploy` with an empty `services` array, so they can
only reach FullDeploy — that arm is config-file-only by construction.

Skips injection when `extra_args` already pins `--force-recreate` or
`--no-recreate`, since compose hard-errors on that pair.

NOT a `compose down` first, which was the first approach tried and abandoned:
`down -- <svc>` cascades to DEPENDENTS, and the following `up -d <svc>` restores
only the target and its dependencies, leaving the dependents removed. Measured
on a 5-service stack: `down -- db` took `api` and `web` with it, neither came
back, and the update still reported success.

Verified on a live 12-server fleet, Komodo 2.3.1, compose 5.4.0. Every arm:
scoped (with and without graph edges, and mid-graph with dependents), whole
project, restart, compose-only, compose+config combined, newly-declared file
both scoped and global, extra_args conflict, requires="None", and combined
deploy+restart. Commands were read from Log.command rather than inferred from
container state.

Not covered: Swarm. `docker stack deploy` has no `--force-recreate` and its
destroy is `docker stack rm`, whole-stack rather than service-scoped.

Refs moghtech#1381
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants