Skip to content

One vault per app: drop the APPS_/{app}_ prefix convention for vault-sourced env #114

Description

@ineedjet

Context

Right now a target has one (or more, via env_refs) vault(s) whose combined
env: covers every app running on that target, merged into one shared
.env. Downstream, generate-env.sh/lib.sh's generate_env() filters
that single canvas per app by name prefix (APP_NAME, APPS_* shared, or
{APPNAME}_* own) before writing apps/{app}/.env — this is how apps are
kept from seeing each other's variables today. The prefix convention exists
specifically because everything gets poured into one shared canvas first
and has to be sorted back out afterward, and it leaks into the compose
files themselves: apps/traefik/docker-compose.yml references
${TRAEFIK_HTTP_PORT} directly, not a vanilla ${HTTP_PORT} — flightdeck
compose files already aren't vanilla-compose-compatible because of this.

An earlier version of this issue proposed fixing that by naming vaults
after the app they belong to (vaults/traefik.yml) and relying on the
filename matching the app name as the link. That's wrong: it reintroduces
exactly the kind of implicit, convention-based wiring this repo has
consistently avoided elsewhere (explicit app_refs/env_refs, "matching
filenames are a convenience, not an implicit relationship" for
vault/target pairing). It also doesn't resolve on its own — see below.

Proposal

Make the target the explicit wiring point between an app and the vault(s)
that feed it, the same way it already explicitly wires app_refs and
env_refs today — just scoped per app instead of per target:

apps:
  traefik:
    env_refs:
      - rubykatzen/flightdeck@latest:hawkeye-traefik.sops.env
  rybbit:
    env_refs:
      - rubykatzen/flightdeck@latest:hawkeye-rybbit.sops.env

replacing the current flat apps: [traefik, rybbit] list — the app names
are just the mapping's keys now, so nothing needed for deploy.sh's
APPS= synthesis is lost.

Each app's vault can then declare its env output names directly, with zero
prefix and zero indirection — e.g. vaults/hawkeye-traefik.yml:

asset: hawkeye-traefik.sops.env
keys:
  - hawkeye
env:
  HTTP_PORT: RUBYKATZEN_COM_TRAEFIK_HTTP_PORT
  DOMAIN: RUBYKATZEN_COM_DOMAIN

HTTP_PORT is exactly what docker-compose.yml can reference — no
prefix stripped anywhere downstream, no hidden rename. Since the vault is
referenced explicitly by ref (not discovered by filename), its filename
doesn't need to match the app name at all — asset names just need to be
unique within a release, same as any other bundle asset.

This resolves what the filename-convention version of this issue couldn't:

  • No asset-naming collision risk across targets. Nothing depends on
    the vault's filename matching the app's name, so two different targets
    running the same app never fight over an asset name.
  • Collisions are detected exactly where they're real, not where they
    aren't.
    Fail-loud detection (from Support multiple env sources (env_refs) merged per target, like app_refs #104/feat: move apps to targets, support env_refs as a list #110) still applies within one
    app's own env_refs list. Across different apps there was never a
    real merge to begin with — a value like DOMAIN duplicated as the same
    mapping in two apps' vaults is structurally guaranteed to resolve to the
    same GitHub Secret value; the only way it drifts is a typo in one
    mapping, which is an ordinary authoring mistake, not a lost safety net.
  • Cross-repo secret composition falls out for free, no new mechanism
    needed.
    This is what made Let a vault manifest import other vaults by ref (cross-repo secret composition) #113 (vault-importing-vault, with its own
    CI-side decrypt key and encrypt-env changes) unnecessary: a third repo
    that needs to contribute e.g. a Telegram bot token just needs a copy of
    the target's public key (keys/hawkeye.pub — not secret, freely
    copyable to any repo, any org) and to run the already-published
    encrypt-env action itself in its own release workflow. It publishes
    its own encrypted asset to its own release. The target then simply adds
    that ref to whichever app's env_refs needs it:
    owner3/repo3@latest:telegram.sops.env. ansible/deploy.yml's
    env_refs loop already parses the repo out of the ref itself (not
    hardcoded to flightdeck's own repo) and decrypts with the target's own
    SOPS_AGE_KEY_FILE regardless of which repo published the asset —
    decryption is keyed by recipient, not by source repo. Zero code
    changes needed for this to already work today. Closed Let a vault manifest import other vaults by ref (cross-repo secret composition) #113 as
    superseded by this.

Resolved: how per-app env actually reaches the server

Settled during discussion with #111 (Ansible→Fabric), then superseded by
a bigger decision made in the same implementation session: flightdeck has
no manual administration flow at all, ever — the "manual local
quick-start" path described below was deleted outright, not kept as a
separate untouched path.

  • Automated path is now the only path. Ref-resolution and downloading
    run entirely on the CI runner (deploy/deploy.py, see Consider migrating ansible/deploy.yml to Fabric (Python) #111); the CI
    runner pushes the finished result to the server rather than the server
    pulling anything.
  • Decryption and config-template rendering both happen on the runner
    too
    (see Consider decrypting vaults on the CI runner instead of server-side #116, decided alongside this) — not server-side as originally
    planned here. The server never runs sops, never holds an age key,
    never runs envsubst. It receives a real, already-decrypted .env and
    already-rendered config directly.
  • Collision detection still happens in CI, from ciphertext, before
    anything is decrypted or pushed
    — this part of the design didn't
    change. SOPS's dotenv output only encrypts values — key names stay in
    cleartext in the downloaded .sops.env asset itself
    (APPS_DOMAIN=ENC[...]). So when an app's env_refs has more than one
    entry, deploy/collisions.py compares key names across the downloaded
    encrypted files directly, no decryption needed. Any duplicate key fails
    the build before anything is decrypted.
  • Net result: no shell scripting left on the server at all.
    up.sh/down.sh/restart.sh/deploy.sh/generate-env.sh/lib.sh
    are all deleted, no replacements. The server's only dependencies are
    Docker and Docker Compose.

Related

Builds on #104/#108/#110 (env_refs as a list, apps living on the
target). Supersedes and closes #113 (cross-vault import) — same goal,
solved by this design without new machinery. Implemented together with
#111 (Ansible→Fabric) — this is the concrete feature the push-based
model in #111 was designed to carry, not built separately in Ansible.
Also affects #103 (ansible ref-resolution dedup), which becomes moot
under #111 rather than needing its own fix.

Implemented, together with #111 and #116, going further than originally
scoped here (see the updated section above).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions