Flatten config templates, render inside the release

AGENTS.md

@@ -57,7 +57,7 @@ This is a Docker-based deployment system (flightdeck) that manages core services
 ### Directory Structure
 
 - `apps/` - Contains core docker-compose configurations and shared compose templates
-- `apps-data/` - Persistent data storage on the target host (not in this repo): each app's data plus its rendered config
+- `apps-data/` - Persistent data storage on the target host (not in this repo): only state that must survive across releases and isn't regenerated by a deploy (database volumes, `traefik/acme.json`)
 - `deploy/` - The push-based deploy entrypoint and its supporting modules (ref resolution, collision detection, decryption, config rendering), run on the GitHub Actions runner
 
 ### Docker Compose Architecture
@@ -86,15 +86,27 @@ The repository uses a modular docker-compose structure with reusable components:
 3. **App Structure Pattern**:
    Each app in `apps/` has:
    - `docker-compose.yml` extending common services
-   - Optional `config/` with template files (`.template.yml`)
+   - Optional `*.tpl` config files sitting directly next to `docker-compose.yml` (see "Config Templates" below)
    - A `.env` on the target host only, decrypted and placed there by `deploy/deploy.py` (never checked into this repo, never present until a real deploy runs)
 
 ### Environment Variable System
 
-There is no root `.env` anywhere - not on a target host, not locally. Each app's env comes entirely from that app's own vault(s), declared in `targets/{target}.yml`'s `apps.<name>.env_refs` (see README's "Vaults And Targets"). `deploy/deploy.py` runs on the GitHub Actions runner: it downloads each app's still-encrypted vault assets, checks their key names for collisions from the ciphertext directly (no decryption needed for that check), decrypts them with the target's private SOPS age key, concatenates the plaintext, and writes it straight into that app's `.env` in the release tree before pushing. `deploy/render.py`'s `render_template` then does the same substitution `envsubst` would, also on the runner, for that app's `config/*.template.*` files, using the just-decrypted values.
+There is no root `.env` anywhere - not on a target host, not locally. Each app's env comes entirely from that app's own vault(s), declared in `targets/{target}.yml`'s `apps.<name>.env_refs` (see README's "Vaults And Targets"). `deploy/deploy.py` runs on the GitHub Actions runner: it downloads each app's still-encrypted vault assets, checks their key names for collisions from the ciphertext directly (no decryption needed for that check), decrypts them with the target's private SOPS age key, concatenates the plaintext, and writes it straight into that app's `.env` in the release tree before pushing. `deploy/render.py`'s `render_template` then does the same substitution `envsubst` would, also on the runner, for that app's `*.tpl` files, using the just-decrypted values (see "Config Templates" below).
 
 A vault declares the exact final variable name an app receives directly (e.g. `HTTP_PORT`, not `TRAEFIK_HTTP_PORT`) - there is no automatic prefix-stripping or filtering step anywhere. Variables for one app are never visible to another app, since each app's `.env` is built from that app's own vault(s) only. This allows running docker compose directly from the app folder without any `--env-file` flags while keeping app secrets scoped.
 
+### Config Templates
+
+`docker compose`'s own `${VAR}` interpolation only reaches into `environment:`/`command:` fields inside the compose file itself - it can't populate a mounted config file some image insists on reading from disk (e.g. Traefik's static config, Codecov Enterprise's settings YAML). Config templates exist for exactly that gap: a plain file with `${VAR}`/`$VAR` placeholders, rendered with the app's own decrypted env values before the app ever starts.
+
+The mechanism is pure naming convention, no manifest or registration needed - the same `.tpl` marker Terraform's `templatefile()` uses, as a terminal suffix (`traefik.yml.tpl`, same placement as Terraform's `user_data.tpl`). Note this means editors and GitHub's diff view won't apply YAML syntax highlighting to the template out of the box (they pick a language by the last extension, and `.tpl` isn't a registered one anywhere by default) - the rendered output (`traefik.yml`) isn't affected, only the template source. Configure a file association per editor if that matters to you (e.g. Zed's `file_types` setting).
+
+Any file directly inside `apps/{app}/` (next to `docker-compose.yml`, no special subdirectory) matching `*.tpl` is a template. `deploy/deploy.py`'s `render_app_configs` finds them with a plain glob, substitutes with `deploy/render.py` (an `envsubst`-equivalent - `$VAR`/`${VAR}` only, no bash `${VAR:-default}` fallback syntax, missing variable becomes an empty string), and writes the result as a sibling file in the same directory with `.tpl` stripped (`traefik.yml.tpl` → `traefik.yml`), `chmod 600` since rendered output can carry secrets. This happens on the runner, before the release is archived, so the rendered file rides inside the release tar next to `.env` and is versioned with that release like everything else - never written directly onto the target host outside the atomic release/symlink-switch step.
+
+Compose files mount the rendered file by its plain relative path (`./traefik.yml:/traefik.yml:ro`), one line per file - not a whole-directory mount - so it's obvious from the compose file alone which container path each config file lands at. `apps-data/{app}/` stays reserved for the opposite case: state a deploy must never regenerate (`acme.json`, database data directories) - never templated output.
+
+When adding config for a new app: only reach for a template if the image has no env-var-driven config path at all. If it does (most well-behaved images do), prefer plain `environment:` entries over a template - fewer moving parts, and the value never touches disk as a separate file.
+
 ### Per-app and per-server overrides
 
 Compose files explicitly declare which variables are overridable using bash fallback syntax:
@@ -163,7 +175,7 @@ Backups are a separate, not-yet-decided piece of tooling (the old `backup.sh` as
    - Reference data path: `../../apps-data/${APP_NAME}/`
    - Set the service port explicitly with `expose` and `traefik.http.services.${APP_NAME}.loadbalancer.server.port`
 3. Wire it into a target's `apps` mapping and give it a vault declaring the env it needs (see README's "Vaults And Targets")
-4. If app needs configuration templates, create `config/{name}.template.yml` (`deploy/render.py` processes these on the runner during deploy, the same substitution `envsubst` would do)
+4. If the app needs a mounted config file with no env-var equivalent, create `{name}.yml.tpl` next to its `docker-compose.yml` (see "Config Templates" above)
 
 Example minimal app structure:
 
@@ -259,11 +271,10 @@ services:
     ports:
       - "8080:8080"
 
-    # 8. VOLUMES (order: data → configs → templates)
+    # 8. VOLUMES (order: persistent data directories → rendered config files)
     volumes:
       - ../../apps-data/${APP_NAME}/data:/data
-      - ../../apps-data/${APP_NAME}/config:/config
-      - ./config/app.template.yml:/app/config.yml:ro
+      - ./app.yml:/app/config.yml:ro
 
     # 9. NETWORKS (inherited from extends, omit this section)
 
@@ -288,7 +299,7 @@ services:
 7. **Networks from extends** - `main`, `main-http`, and `api` profiles include `traefik` and `internal`; never add `databases` (it's only for DB admin tools)
 8. **Depends_on as simple list** - use array format without `condition:`, healthchecks are in common.yml
 9. **Depends_on order**: postgres → redis → mongo → app services
-10. **Volumes order**: data directories → config directories → template files (with :ro)
+10. **Volumes order**: persistent data directories (from `apps-data/`) → rendered config files (relative path, with :ro)
 11. **Paths use ${APP_NAME}** - for reusability across apps
 
 ### YAML formatting rules:
@@ -337,12 +348,12 @@ GitHub Actions workflow (`.github/workflows/release.yml`) manages releases via [
 
 Deployment helpers live in this repository, entirely under `deploy/`, run only on the GitHub Actions runner - the target host never runs any of this:
 
-- `deploy/deploy.py` is the deploy entrypoint. It resolves and downloads every ref in `app_refs` (the app bundles, at least one required — flightdeck's own `apps/` catalog is just another entry, not implicit) and merges them into a release tree locally; for each app in the target's `apps` mapping, downloads its `env_refs` (still encrypted), decrypts them with the target's private SOPS age key, writes the plaintext into that app's `.env` in the release tree, and renders that app's `config/*.template.*` files with the decrypted values. It then opens an SSH connection per host, pushes the finished release (real `.env`, already-rendered config), bootstraps networks/directories idempotently, switches a timestamped release, and runs `docker compose pull && docker compose up -d` per app directly (no wrapper script on the host at all).
+- `deploy/deploy.py` is the deploy entrypoint. It resolves and downloads every ref in `app_refs` (the app bundles, at least one required — flightdeck's own `apps/` catalog is just another entry, not implicit) and merges them into a release tree locally; for each app in the target's `apps` mapping, downloads its `env_refs` (still encrypted), decrypts them with the target's private SOPS age key, writes the plaintext into that app's `.env` in the release tree, and renders that app's `*.tpl` files in place with the decrypted values (see "Config Templates" above). It then opens an SSH connection per host, pushes the finished release as one tarball (real `.env`, already-rendered config, all versioned together), bootstraps networks/directories idempotently, switches a timestamped release, and runs `docker compose pull && docker compose up -d` per app directly (no wrapper script on the host at all).
 - `deploy/resolve.py`, `deploy/collisions.py`, `deploy/vault.py`, and `deploy/render.py` hold, respectively, the ref-resolution, ciphertext collision-detection, decryption, and template-rendering logic - each with real `unittest` coverage in `deploy/tests/`.
 - `.github/actions/encrypt-env/` is a local composite action for rendering `vaults/` manifests from GitHub Secrets/Variables, encrypting them for age recipients, and publishing `.sops.env` as a GitHub Release asset — vault manifests hold only env/secrets, not app selection
 - `.github/workflows/deploy-shared.yml` is a reusable workflow consumer repos call to run `deploy/deploy.py` from GitHub Actions over an optional Tailscale connection, without holding any deploy secrets in this repository
 
-The `releases/{timestamp}`/`current` symlink pattern exists for atomicity, not for rollback: a deploy either fully lands and only then switches the symlink as its last step, or fails partway and leaves `current` untouched — never a partially-applied app. There is deliberately no automated rollback, and manual rollback (point `current` at an old release directory by hand) is not a supported/maintained path — it wouldn't restore that release's rendered config templates (`apps-data/{app}/config/` isn't versioned per release) or a floating-tag image's historical version either, and in practice fixing forward through the normal deploy path is simpler and safer than reasoning about what a partial rollback actually restores.
+The `releases/{timestamp}`/`current` symlink pattern exists for atomicity, not for rollback: a deploy either fully lands and only then switches the symlink as its last step, or fails partway and leaves `current` untouched — never a partially-applied app. Rendered config templates are part of this same guarantee - they're written into the release tree and travel inside the release tarball, not pushed separately or in place. There is deliberately no automated rollback, and manual rollback (point `current` at an old release directory by hand) is not a supported/maintained path — it wouldn't restore a floating-tag image's historical version, and in practice fixing forward through the normal deploy path is simpler and safer than reasoning about what a partial rollback actually restores.
 
 App bundles listed in `app_refs` are release assets referenced as short refs like `<owner>/<repo>@latest` or `<owner>/<repo>@v1.2.3`, resolving to a default asset name of `flightdeck-apps.zip` unless the ref specifies an explicit `:asset-name` suffix. `@latest` is resolved through GitHub's latest release API. Every bundle must contain an `apps/` directory; both per-app directories and shared top-level files (`common.yml`, `networks.yml`, etc.) merge the same way — copy if new, fail loud on any name conflict across bundles.
 
@@ -359,12 +370,3 @@ carries its label) — `deploy/deploy.py` already runs `docker compose pull &&
 up -d` for every app in a target's `apps` mapping on every deploy, which
 made Watchtower's own polling redundant. See `RETIRED.md`.
 
-## Important: Template Files vs Generated Files
-
-**CRITICAL**: When updating application configurations, always edit the `.template.*` files in `apps/{app}/config/`, NOT the generated files in `apps-data/{app}/config/`.
-
-- Template files are located in: `apps/{app}/config/*.template.*`
-- Generated files are created in: `apps-data/{app}/config/`
-- `deploy/render.py` processes templates on the GitHub Actions runner during deploy and pushes the rendered result directly - nothing renders on the host
-- Editing generated files directly will result in lost changes on the next deploy
-- Always modify templates, then re-deploy to regenerate

README.md

@@ -30,8 +30,8 @@ The deploy is push-based and runs entirely on the GitHub Actions runner:
 2. Merge the app bundles into a release tree.
 3. Check each app's env sources for key collisions from the still-encrypted ciphertext (SOPS's dotenv output only encrypts values, so key names are readable without decryption) — scoped to that app's own sources, not across apps.
 4. Decrypt each app's env with the target's private SOPS age key (a GitHub Secret) and write it straight into that app's `.env` in the release tree.
-5. Render that app's config templates (`config/*.template.*`) using the decrypted values — the same substitution `envsubst` does, run here instead of on the host.
-6. Push the finished release (real `.env`, already-rendered config) to each host over SSH, switch the `current` symlink, and run `docker compose pull && docker compose up -d` per app.
+5. Render that app's `*.tpl` config files in place, next to its `docker-compose.yml`, using the decrypted values — the same substitution `envsubst` does, run here instead of on the host.
+6. Push the finished release (real `.env`, already-rendered config, one tarball) to each host over SSH, switch the `current` symlink, and run `docker compose pull && docker compose up -d` per app.
 
 What gets deployed — which app bundles, which apps actually run, and which encrypted env sources feed each one — is configured declaratively per target; see "Vaults And Targets" below for the manifest format.
 
@@ -50,12 +50,12 @@ flightdeck/
 │   ├── gotenberg-8.yml           # Document conversion template
 │   └── {app-name}/              # Each app directory
 │       ├── docker-compose.yml   # App configuration
-│       └── config/              # Optional config templates
+│       └── *.tpl                # Optional config file templates, rendered in place at deploy time
 │
 ├── apps-data/                     # Persistent data on the target host, not in this repo
-│   ├── traefik/                 # SSL certificates
+│   ├── traefik/                 # SSL certificates (acme.json)
 │   ├── postgres/                # PostgreSQL data
-│   └── {app-name}/              # Each app's data + rendered config
+│   └── {app-name}/              # Each app's data that must survive across releases
 │
 ├── deploy/
 │   ├── deploy.py       # Push-based deploy entrypoint (runs on the CI runner)

apps/codecov/docker-compose.yml

@@ -10,7 +10,7 @@ x-environment: &environment
   CODECOV_SCHEME: https
   RUN_ENV: ENTERPRISE
 x-volumes: &volumes
-  - ../../apps-data/${APP_NAME}/config:/config
+  - ./codecov.yml:/config/codecov.yml
 services:
   gateway:
     image: codecov/self-hosted-gateway:latest-stable
@@ -56,5 +56,5 @@ services:
       - timescale
     environment: *environment
     volumes:
-      - ../../apps-data/${APP_NAME}/config:/config
+      - ./codecov.yml:/config/codecov.yml
       - ../../apps-data/${APP_NAME}/archive:/archive

apps/traefik/docker-compose.yml

@@ -10,7 +10,7 @@ services:
     volumes:
       - /etc/localtime:/etc/localtime:ro
       - /var/run/docker.sock:/var/run/docker.sock:ro
-      - ../../apps-data/${APP_NAME}/config/traefik.yml:/traefik.yml:ro
+      - ./traefik.yml:/traefik.yml:ro
       - ../../apps-data/${APP_NAME}/acme.json:/acme.json
     networks:
       - traefik