fix: recreate services when a config_files change triggers the deploy - #1613
Open
LoganRupe wants to merge 1 commit into
Open
fix: recreate services when a config_files change triggers the deploy#1613LoganRupe wants to merge 1 commit into
LoganRupe wants to merge 1 commit into
Conversation
…eploy
DeployStackIfChanged correctly detects a `config_files` content change and fires
a deploy scoped to the declared services. That deploy then runs a plain
`docker compose up -d <service>`, which is a no-op: only the content behind a
bind mount moved, so the service definition is unchanged and compose reports the
container up-to-date. Komodo records success and advances `deployed_contents`,
so the change is permanently marked deployed against a container still running
the old file, and every later diff reads "unchanged". Silent, and it never
self-corrects.
Adds `force_recreate` to DeployStack and the periphery ComposeUp request, and
appends `--force-recreate` when a config file drove the deploy.
`DeployIfChangedAction::FullDeploy` gains a `force_recreate` flag, and
`resolve_deploy_if_changed_action` takes `&Stack` so it can use `is_config_file`
to tell a config trigger from a compose/env one.
Two early returns are removed from that function:
* on the first `(Redeploy, true)` hit. Compose files are registered before
config files, so a commit touching both would return on the compose file and
never consider the config file. It now accumulates and decides at the end.
* when a file is absent from deployed_contents. Such a file was only just
declared, and returning FullDeploy meant adding one `config_files` entry
force-recreated the whole project. It is now simply treated as changed and
scoped by its own `services` like any other changed file. The genuinely
global cases still work with no special handling: compose and env files
carry an empty `services` array, so they land in the `(Redeploy, true)` arm
regardless.
`Services { deploy }` needs no flag. Compose and env files are registered via
`StackFileDependency::full_redeploy` with an empty `services` array, so they can
only reach FullDeploy — that arm is config-file-only by construction.
Skips injection when `extra_args` already pins `--force-recreate` or
`--no-recreate`, since compose hard-errors on that pair.
NOT a `compose down` first, which was the first approach tried and abandoned:
`down -- <svc>` cascades to DEPENDENTS, and the following `up -d <svc>` restores
only the target and its dependencies, leaving the dependents removed. Measured
on a 5-service stack: `down -- db` took `api` and `web` with it, neither came
back, and the update still reported success.
Verified on a live 12-server fleet, Komodo 2.3.1, compose 5.4.0. Every arm:
scoped (with and without graph edges, and mid-graph with dependents), whole
project, restart, compose-only, compose+config combined, newly-declared file
both scoped and global, extra_args conflict, requires="None", and combined
deploy+restart. Commands were read from Log.command rather than inferred from
container state.
Not covered: Swarm. `docker stack deploy` has no `--force-recreate` and its
destroy is `docker stack rm`, whole-stack rather than service-scoped.
Refs moghtech#1381
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the
config_filescase in #1381, and the same thing reported by @madereddy and @tylovejoy in #1205.The problem
DeployStackIfChangeddetects aconfig_filescontent change correctly and fires a deploy scoped to the declared services. That deploy runs a plaindocker compose up -d <service>, which no-ops: only the content behind a bind mount moved, so the service definition is unchanged and compose reports the container up to date.Komodo then records success and advances
deployed_contents, so the change stays marked deployed against a container still running the old file, and every later diff reads "unchanged". It doesn't self-correct.The change
Adds
force_recreatetoDeployStackand the peripheryComposeUprequest, and appends--force-recreatewhen a config file drove the deploy.DeployIfChangedAction::FullDeploygains aforce_recreateflag, andresolve_deploy_if_changed_actiontakes&Stackso it can useis_config_fileto tell a config trigger from a compose or env one.Two early returns come out of that function:
On the first
(Redeploy, true)hit. Compose files are registered before config files, so a commit touching both returned on the compose file and never looked at the config file. It now accumulates and decides at the end.When a file is absent from
deployed_contents. Such a file was only just declared, and returningFullDeploymeant adding oneconfig_filesentry force-recreated the whole project. It's now treated as changed and scoped by its ownserviceslike any other changed file.The genuinely global cases still work with no special handling, since compose and env files carry an empty
servicesarray and land in(Redeploy, true)either way.Services { deploy }needs no flag at all: that arm is only reachable from aconfig_filesentry, because compose and env files are registered viaStackFileDependency::full_redeploywith emptyservices.Injection is skipped when
extra_argsalready pins--force-recreateor--no-recreate, since compose hard-errors on that pair.Why not "destroy before deploy"
That was suggested on both issues and I built it first, but it's unsafe for a scoped deploy.
docker compose down -- <svc>cascades to dependents, and the followingup -d <svc>restores only the target and its dependencies, leaving the dependents removed. On a five service test stack,down -- dbtookapiandwebwith it, neither came back, and the update still reported success:--force-recreatehas no cascade: it recreates the named service and leaves both dependencies and dependents running.Testing
Run against a 12 server fleet on 2.3.1, compose 5.4.0, on a throwaway stack with a health-gated root, a dependency chain, and one service with no edges. Commands were read from
Log.commandrather than inferred from container state.Redeploy+services=[svc], no graph edgesRedeploy+services=[svc], mid-graphRedeploy+services=[]Restart+services=[svc]up -d, nothing forcedservices=[svc]services=[]extra_args = ["--no-recreate"]requires = "None"Not covered
Swarm.
docker stack deployhas no--force-recreate, and its destroy isdocker stack rm, which is whole stack rather than service scoped. I didn't want to guess at the right behaviour there.Only compose 5.4.0 was exercised.
One thing worth your call
force_recreateonDeployStackis new public API surface: it picks up a CLI flag, a TS type and an OpenAPI entry. That seemed reasonable since "deploy and force recreate" is useful on its own, but if you'd rather keep it internal I'm happy to rework how the trigger is carried.