Summary
Stacks that are deployed and running correctly intermittently flash the red Project Missing badge on the Stacks page (replacing the commit hash), then clear on their own a short time later, with no corresponding action taken and nothing logged. This happens randomly and repeatedly over time on an otherwise healthy stack/host.
Environment
- Komodo Core version: 2.3.2
- Komodo Periphery version: 2.3.2
- Periphery deployment: Docker container (
ghcr.io/moghtech/komodo-periphery)
- Host OS / Docker version: Ubuntu 26.04 LTS (Resolute Raccoon)
- Number of stacks/containers on the affected host: 5
legacy_compose_cli: false/default, confirmed not set
What I'm seeing
On the Stacks page, an individual stack that is actually up and running will periodically show the red Project Missing button instead of its deployed commit hash. Hovering it shows:
The compose project is not on the host. If the compose stack is running, the 'Project Name' needs to be set. This can be found with 'docker compose ls'.
The project name is correctly set and matches docker compose ls output when checked manually. After some time (not tied to any redeploy or user action), the commit hash reappears on its own. Later, the badge reappears again. This cycle repeats indefinitely and appears random.
Root cause (as far as I can tell from the source)
Tracing this through the v2.3.2 source:
-
Core determines project_missing for a stack by checking whether the stack's project name is present in the periphery's cached list of Docker Compose projects for that server (bin/core/src/resource/stack.rs, around the project_missing computation). If the project name isn't found in that cached list, project_missing is set to true and the UI shows the red badge (ui/src/resources/stack/index.tsx).
-
That cached list comes from periphery's PollStatus handler (bin/periphery/src/api/poll.rs), which calls list_compose_projects() on every poll:
list_compose_projects().map(Result::unwrap_or_default),
-
list_compose_projects() (bin/periphery/src/docker/compose.rs) runs:
run_komodo_standard_command(
"List Projects",
format!("{docker_compose} ls --all --format json"),
CommandOptions::default().timeout(Duration::from_secs(1)),
)i.e. docker compose ls --all --format json with a hardcoded 1-second timeout. If the Docker daemon takes longer than 1s to respond (busy host, socket contention from other tooling, many containers/stacks, etc.), the process group is killed and the call returns Err.
-
Back in poll.rs, that Err is silently discarded via .unwrap_or_default(), turning it into an empty project list for that poll cycle — the error is never logged anywhere.
-
Core receives this empty list, doesn't find the stack's project name in it, and marks project_missing = true for every stack on that host for that cycle. On the next successful poll (assuming docker compose ls finishes in time), the list is populated again and the badge disappears — until the next slow poll.
This matches the observed behavior exactly: intermittent, self-resolving, no user action involved, and no error visible anywhere in the periphery logs (since the failure is swallowed).
How to reproduce / confirm
Run the exact command periphery uses, inside the periphery container itself, ideally while the host has some load (e.g. mid-deploy on another stack):
docker exec <periphery-container> sh -c "time docker compose ls --all --format json"
real 0m1.339s -- regularily see values above 1s from the above command
Suggested fix
A few options, any of which would help:
- Increase the timeout on
list_compose_projects() to something more forgiving (e.g. 5–10s) given this call scales with the number of containers/projects on the host and can legitimately be slow under load.
- Log the error instead of silently discarding it via
unwrap_or_default(), so this failure mode is at least visible in periphery logs.
- On failure, keep the last-known-good project list instead of replacing it with an empty one, so a single slow poll doesn't cause every stack on the host to flash as missing.
Related
This looks related to the general fragility of periphery's compose CLI subprocess handling raised in #1392 (hung docker compose ls freezing fleet-wide stats/deploys) — same underlying command, different failure mode (timeout + silent swallow here vs. full hang there).
Summary
Stacks that are deployed and running correctly intermittently flash the red Project Missing badge on the Stacks page (replacing the commit hash), then clear on their own a short time later, with no corresponding action taken and nothing logged. This happens randomly and repeatedly over time on an otherwise healthy stack/host.
Environment
ghcr.io/moghtech/komodo-periphery)legacy_compose_cli: false/default, confirmed not setWhat I'm seeing
On the Stacks page, an individual stack that is actually up and running will periodically show the red Project Missing button instead of its deployed commit hash. Hovering it shows:
The project name is correctly set and matches
docker compose lsoutput when checked manually. After some time (not tied to any redeploy or user action), the commit hash reappears on its own. Later, the badge reappears again. This cycle repeats indefinitely and appears random.Root cause (as far as I can tell from the source)
Tracing this through the
v2.3.2source:Core determines
project_missingfor a stack by checking whether the stack's project name is present in the periphery's cached list of Docker Compose projects for that server (bin/core/src/resource/stack.rs, around theproject_missingcomputation). If the project name isn't found in that cached list,project_missingis set totrueand the UI shows the red badge (ui/src/resources/stack/index.tsx).That cached list comes from periphery's
PollStatushandler (bin/periphery/src/api/poll.rs), which callslist_compose_projects()on every poll:list_compose_projects()(bin/periphery/src/docker/compose.rs) runs:i.e.
docker compose ls --all --format jsonwith a hardcoded 1-second timeout. If the Docker daemon takes longer than 1s to respond (busy host, socket contention from other tooling, many containers/stacks, etc.), the process group is killed and the call returnsErr.Back in
poll.rs, thatErris silently discarded via.unwrap_or_default(), turning it into an empty project list for that poll cycle — the error is never logged anywhere.Core receives this empty list, doesn't find the stack's project name in it, and marks
project_missing = truefor every stack on that host for that cycle. On the next successful poll (assumingdocker compose lsfinishes in time), the list is populated again and the badge disappears — until the next slow poll.This matches the observed behavior exactly: intermittent, self-resolving, no user action involved, and no error visible anywhere in the periphery logs (since the failure is swallowed).
How to reproduce / confirm
Run the exact command periphery uses, inside the periphery container itself, ideally while the host has some load (e.g. mid-deploy on another stack):
real 0m1.339s -- regularily see values above 1s from the above command
Suggested fix
A few options, any of which would help:
list_compose_projects()to something more forgiving (e.g. 5–10s) given this call scales with the number of containers/projects on the host and can legitimately be slow under load.unwrap_or_default(), so this failure mode is at least visible in periphery logs.Related
This looks related to the general fragility of periphery's compose CLI subprocess handling raised in #1392 (hung
docker compose lsfreezing fleet-wide stats/deploys) — same underlying command, different failure mode (timeout + silent swallow here vs. full hang there).