Skip to content

Drop Watchtower entirely - idempotent deploy already makes it redundant #118

Description

@ineedjet

Context

Split out from #100. #100's core proposal (swap forced restart for
idempotent docker compose pull && up -d, since Compose's own
image-digest + resolved-config comparison already decides per-service what
actually needs recreating) landed via #115deploy/deploy.py now runs
exactly that per app, on every deploy. #100's own "Follow-ups considered"
section already floated the next logical step but didn't commit to it:

A scheduled (cron) shared workflow reusing deploy-shared.yml's
Tailscale/SSH plumbing to run up.sh periodically across all apps —
effectively replaces Watchtower without introducing a separate service,
container, or label, since pin depth/digest pinning alone already
governs predictability.

This issue is that step, now that the idempotent half is real.

Current state

Three apps carry com.centurylinklabs.watchtower.enable=true: traefik,
semaphore, and watchtower itself. deploy/deploy.py's
is_watchtower_managed check means the automated deploy path never runs
docker compose pull/up for any of them — their entire update path
depends on an actual watchtower container running on that target and
polling on its own schedule.

This has a real, already-flagged consequence (#106): hawkeye's target
doesn't include a watchtower app in its apps mapping at all, so as
implemented, traefik would never actually get started by the automated
path on a fresh hawkeye deploy — nothing else ever runs docker compose up
for it.

Proposal

Drop Watchtower as a concept entirely, now that it's redundant with what
deploy/deploy.py already does on every deploy:

  • Remove the com.centurylinklabs.watchtower.enable=true label from
    apps/traefik/docker-compose.yml and apps/semaphore/docker-compose.yml.
  • Remove (or stop recommending) apps/watchtower/ from the catalog.
  • Remove is_watchtower_managed/WATCHTOWER_LABEL and the run_apps vs
    all_apps split from deploy/deploy.py — every app in a target's apps
    mapping just gets docker compose pull && docker compose up -d --remove-orphans uniformly, no label-based special-casing anywhere.
    This also directly fixes Normalize the hawkeye/rybbit deploy: configure prerequisites and get the first real deployment running #106's traefik gap, since there'd be nothing
    left to skip it.

What this loses, and the replacement

Watchtower's one piece of unique value: catching a spontaneous upstream
image update (a floating-tag app getting a new image pushed somewhere
else) without needing a flightdeck-side trigger at all. Drop Watchtower
without anything else, and floating-tag drift only gets picked up on the
next actual deploy (release or manual workflow_dispatch).

If that gap matters, the replacement is the scheduled-cron idea #100
already floated: a periodic workflow_dispatch-equivalent (real cron
trigger) re-running deploy-shared.yml/deploy/deploy.py for a target,
which achieves the same outcome — fresh floating-tag images pulled on some
cadence — without a Watchtower container needing Docker socket access on
every target host (a real attack-surface reduction, not just a
simplification). Not scoping that into this issue; can be its own
follow-up if wanted once this lands.

Related

#100 (idempotent reconciliation this depends on, closed), #106 (the
traefik/Watchtower gap this directly resolves).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions