diff --git a/AGENTS.md b/AGENTS.md index 4218aa5..2a9a765 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -57,7 +57,7 @@ This is a Docker-based deployment system (flightdeck) that manages core services ### Directory Structure - `apps/` - Contains core docker-compose configurations and shared compose templates -- `apps-data/` - Persistent data storage on the target host (not in this repo): each app's data plus its rendered config +- `apps-data/` - Persistent data storage on the target host (not in this repo): only state that must survive across releases and isn't regenerated by a deploy (database volumes, `traefik/acme.json`) - `deploy/` - The push-based deploy entrypoint and its supporting modules (ref resolution, collision detection, decryption, config rendering), run on the GitHub Actions runner ### Docker Compose Architecture @@ -86,15 +86,27 @@ The repository uses a modular docker-compose structure with reusable components: 3. **App Structure Pattern**: Each app in `apps/` has: - `docker-compose.yml` extending common services - - Optional `config/` with template files (`.template.yml`) + - Optional `*.tpl` config files sitting directly next to `docker-compose.yml` (see "Config Templates" below) - A `.env` on the target host only, decrypted and placed there by `deploy/deploy.py` (never checked into this repo, never present until a real deploy runs) ### Environment Variable System -There is no root `.env` anywhere - not on a target host, not locally. Each app's env comes entirely from that app's own vault(s), declared in `targets/{target}.yml`'s `apps..env_refs` (see README's "Vaults And Targets"). `deploy/deploy.py` runs on the GitHub Actions runner: it downloads each app's still-encrypted vault assets, checks their key names for collisions from the ciphertext directly (no decryption needed for that check), decrypts them with the target's private SOPS age key, concatenates the plaintext, and writes it straight into that app's `.env` in the release tree before pushing. `deploy/render.py`'s `render_template` then does the same substitution `envsubst` would, also on the runner, for that app's `config/*.template.*` files, using the just-decrypted values. +There is no root `.env` anywhere - not on a target host, not locally. Each app's env comes entirely from that app's own vault(s), declared in `targets/{target}.yml`'s `apps..env_refs` (see README's "Vaults And Targets"). `deploy/deploy.py` runs on the GitHub Actions runner: it downloads each app's still-encrypted vault assets, checks their key names for collisions from the ciphertext directly (no decryption needed for that check), decrypts them with the target's private SOPS age key, concatenates the plaintext, and writes it straight into that app's `.env` in the release tree before pushing. `deploy/render.py`'s `render_template` then does the same substitution `envsubst` would, also on the runner, for that app's `*.tpl` files, using the just-decrypted values (see "Config Templates" below). A vault declares the exact final variable name an app receives directly (e.g. `HTTP_PORT`, not `TRAEFIK_HTTP_PORT`) - there is no automatic prefix-stripping or filtering step anywhere. Variables for one app are never visible to another app, since each app's `.env` is built from that app's own vault(s) only. This allows running docker compose directly from the app folder without any `--env-file` flags while keeping app secrets scoped. +### Config Templates + +`docker compose`'s own `${VAR}` interpolation only reaches into `environment:`/`command:` fields inside the compose file itself - it can't populate a mounted config file some image insists on reading from disk (e.g. Traefik's static config, Codecov Enterprise's settings YAML). Config templates exist for exactly that gap: a plain file with `${VAR}`/`$VAR` placeholders, rendered with the app's own decrypted env values before the app ever starts. + +The mechanism is pure naming convention, no manifest or registration needed - the same `.tpl` marker Terraform's `templatefile()` uses, as a terminal suffix (`traefik.yml.tpl`, same placement as Terraform's `user_data.tpl`). Note this means editors and GitHub's diff view won't apply YAML syntax highlighting to the template out of the box (they pick a language by the last extension, and `.tpl` isn't a registered one anywhere by default) - the rendered output (`traefik.yml`) isn't affected, only the template source. Configure a file association per editor if that matters to you (e.g. Zed's `file_types` setting). + +Any file directly inside `apps/{app}/` (next to `docker-compose.yml`, no special subdirectory) matching `*.tpl` is a template. `deploy/deploy.py`'s `render_app_configs` finds them with a plain glob, substitutes with `deploy/render.py` (an `envsubst`-equivalent - `$VAR`/`${VAR}` only, no bash `${VAR:-default}` fallback syntax, missing variable becomes an empty string), and writes the result as a sibling file in the same directory with `.tpl` stripped (`traefik.yml.tpl` → `traefik.yml`), `chmod 600` since rendered output can carry secrets. This happens on the runner, before the release is archived, so the rendered file rides inside the release tar next to `.env` and is versioned with that release like everything else - never written directly onto the target host outside the atomic release/symlink-switch step. + +Compose files mount the rendered file by its plain relative path (`./traefik.yml:/traefik.yml:ro`), one line per file - not a whole-directory mount - so it's obvious from the compose file alone which container path each config file lands at. `apps-data/{app}/` stays reserved for the opposite case: state a deploy must never regenerate (`acme.json`, database data directories) - never templated output. + +When adding config for a new app: only reach for a template if the image has no env-var-driven config path at all. If it does (most well-behaved images do), prefer plain `environment:` entries over a template - fewer moving parts, and the value never touches disk as a separate file. + ### Per-app and per-server overrides Compose files explicitly declare which variables are overridable using bash fallback syntax: @@ -163,7 +175,7 @@ Backups are a separate, not-yet-decided piece of tooling (the old `backup.sh` as - Reference data path: `../../apps-data/${APP_NAME}/` - Set the service port explicitly with `expose` and `traefik.http.services.${APP_NAME}.loadbalancer.server.port` 3. Wire it into a target's `apps` mapping and give it a vault declaring the env it needs (see README's "Vaults And Targets") -4. If app needs configuration templates, create `config/{name}.template.yml` (`deploy/render.py` processes these on the runner during deploy, the same substitution `envsubst` would do) +4. If the app needs a mounted config file with no env-var equivalent, create `{name}.yml.tpl` next to its `docker-compose.yml` (see "Config Templates" above) Example minimal app structure: @@ -259,11 +271,10 @@ services: ports: - "8080:8080" - # 8. VOLUMES (order: data → configs → templates) + # 8. VOLUMES (order: persistent data directories → rendered config files) volumes: - ../../apps-data/${APP_NAME}/data:/data - - ../../apps-data/${APP_NAME}/config:/config - - ./config/app.template.yml:/app/config.yml:ro + - ./app.yml:/app/config.yml:ro # 9. NETWORKS (inherited from extends, omit this section) @@ -288,7 +299,7 @@ services: 7. **Networks from extends** - `main`, `main-http`, and `api` profiles include `traefik` and `internal`; never add `databases` (it's only for DB admin tools) 8. **Depends_on as simple list** - use array format without `condition:`, healthchecks are in common.yml 9. **Depends_on order**: postgres → redis → mongo → app services -10. **Volumes order**: data directories → config directories → template files (with :ro) +10. **Volumes order**: persistent data directories (from `apps-data/`) → rendered config files (relative path, with :ro) 11. **Paths use ${APP_NAME}** - for reusability across apps ### YAML formatting rules: @@ -337,12 +348,12 @@ GitHub Actions workflow (`.github/workflows/release.yml`) manages releases via [ Deployment helpers live in this repository, entirely under `deploy/`, run only on the GitHub Actions runner - the target host never runs any of this: -- `deploy/deploy.py` is the deploy entrypoint. It resolves and downloads every ref in `app_refs` (the app bundles, at least one required — flightdeck's own `apps/` catalog is just another entry, not implicit) and merges them into a release tree locally; for each app in the target's `apps` mapping, downloads its `env_refs` (still encrypted), decrypts them with the target's private SOPS age key, writes the plaintext into that app's `.env` in the release tree, and renders that app's `config/*.template.*` files with the decrypted values. It then opens an SSH connection per host, pushes the finished release (real `.env`, already-rendered config), bootstraps networks/directories idempotently, switches a timestamped release, and runs `docker compose pull && docker compose up -d` per app directly (no wrapper script on the host at all). +- `deploy/deploy.py` is the deploy entrypoint. It resolves and downloads every ref in `app_refs` (the app bundles, at least one required — flightdeck's own `apps/` catalog is just another entry, not implicit) and merges them into a release tree locally; for each app in the target's `apps` mapping, downloads its `env_refs` (still encrypted), decrypts them with the target's private SOPS age key, writes the plaintext into that app's `.env` in the release tree, and renders that app's `*.tpl` files in place with the decrypted values (see "Config Templates" above). It then opens an SSH connection per host, pushes the finished release as one tarball (real `.env`, already-rendered config, all versioned together), bootstraps networks/directories idempotently, switches a timestamped release, and runs `docker compose pull && docker compose up -d` per app directly (no wrapper script on the host at all). - `deploy/resolve.py`, `deploy/collisions.py`, `deploy/vault.py`, and `deploy/render.py` hold, respectively, the ref-resolution, ciphertext collision-detection, decryption, and template-rendering logic - each with real `unittest` coverage in `deploy/tests/`. - `.github/actions/encrypt-env/` is a local composite action for rendering `vaults/` manifests from GitHub Secrets/Variables, encrypting them for age recipients, and publishing `.sops.env` as a GitHub Release asset — vault manifests hold only env/secrets, not app selection - `.github/workflows/deploy-shared.yml` is a reusable workflow consumer repos call to run `deploy/deploy.py` from GitHub Actions over an optional Tailscale connection, without holding any deploy secrets in this repository -The `releases/{timestamp}`/`current` symlink pattern exists for atomicity, not for rollback: a deploy either fully lands and only then switches the symlink as its last step, or fails partway and leaves `current` untouched — never a partially-applied app. There is deliberately no automated rollback, and manual rollback (point `current` at an old release directory by hand) is not a supported/maintained path — it wouldn't restore that release's rendered config templates (`apps-data/{app}/config/` isn't versioned per release) or a floating-tag image's historical version either, and in practice fixing forward through the normal deploy path is simpler and safer than reasoning about what a partial rollback actually restores. +The `releases/{timestamp}`/`current` symlink pattern exists for atomicity, not for rollback: a deploy either fully lands and only then switches the symlink as its last step, or fails partway and leaves `current` untouched — never a partially-applied app. Rendered config templates are part of this same guarantee - they're written into the release tree and travel inside the release tarball, not pushed separately or in place. There is deliberately no automated rollback, and manual rollback (point `current` at an old release directory by hand) is not a supported/maintained path — it wouldn't restore a floating-tag image's historical version, and in practice fixing forward through the normal deploy path is simpler and safer than reasoning about what a partial rollback actually restores. App bundles listed in `app_refs` are release assets referenced as short refs like `/@latest` or `/@v1.2.3`, resolving to a default asset name of `flightdeck-apps.zip` unless the ref specifies an explicit `:asset-name` suffix. `@latest` is resolved through GitHub's latest release API. Every bundle must contain an `apps/` directory; both per-app directories and shared top-level files (`common.yml`, `networks.yml`, etc.) merge the same way — copy if new, fail loud on any name conflict across bundles. @@ -359,12 +370,3 @@ carries its label) — `deploy/deploy.py` already runs `docker compose pull && up -d` for every app in a target's `apps` mapping on every deploy, which made Watchtower's own polling redundant. See `RETIRED.md`. -## Important: Template Files vs Generated Files - -**CRITICAL**: When updating application configurations, always edit the `.template.*` files in `apps/{app}/config/`, NOT the generated files in `apps-data/{app}/config/`. - -- Template files are located in: `apps/{app}/config/*.template.*` -- Generated files are created in: `apps-data/{app}/config/` -- `deploy/render.py` processes templates on the GitHub Actions runner during deploy and pushes the rendered result directly - nothing renders on the host -- Editing generated files directly will result in lost changes on the next deploy -- Always modify templates, then re-deploy to regenerate diff --git a/README.md b/README.md index c6b70f6..e74dc84 100644 --- a/README.md +++ b/README.md @@ -30,8 +30,8 @@ The deploy is push-based and runs entirely on the GitHub Actions runner: 2. Merge the app bundles into a release tree. 3. Check each app's env sources for key collisions from the still-encrypted ciphertext (SOPS's dotenv output only encrypts values, so key names are readable without decryption) — scoped to that app's own sources, not across apps. 4. Decrypt each app's env with the target's private SOPS age key (a GitHub Secret) and write it straight into that app's `.env` in the release tree. -5. Render that app's config templates (`config/*.template.*`) using the decrypted values — the same substitution `envsubst` does, run here instead of on the host. -6. Push the finished release (real `.env`, already-rendered config) to each host over SSH, switch the `current` symlink, and run `docker compose pull && docker compose up -d` per app. +5. Render that app's `*.tpl` config files in place, next to its `docker-compose.yml`, using the decrypted values — the same substitution `envsubst` does, run here instead of on the host. +6. Push the finished release (real `.env`, already-rendered config, one tarball) to each host over SSH, switch the `current` symlink, and run `docker compose pull && docker compose up -d` per app. What gets deployed — which app bundles, which apps actually run, and which encrypted env sources feed each one — is configured declaratively per target; see "Vaults And Targets" below for the manifest format. @@ -50,12 +50,12 @@ flightdeck/ │ ├── gotenberg-8.yml # Document conversion template │ └── {app-name}/ # Each app directory │ ├── docker-compose.yml # App configuration -│ └── config/ # Optional config templates +│ └── *.tpl # Optional config file templates, rendered in place at deploy time │ ├── apps-data/ # Persistent data on the target host, not in this repo -│ ├── traefik/ # SSL certificates +│ ├── traefik/ # SSL certificates (acme.json) │ ├── postgres/ # PostgreSQL data -│ └── {app-name}/ # Each app's data + rendered config +│ └── {app-name}/ # Each app's data that must survive across releases │ ├── deploy/ │ ├── deploy.py # Push-based deploy entrypoint (runs on the CI runner) diff --git a/apps/codecov/config/codecov.template.yml b/apps/codecov/codecov.yml.tpl similarity index 100% rename from apps/codecov/config/codecov.template.yml rename to apps/codecov/codecov.yml.tpl diff --git a/apps/codecov/docker-compose.yml b/apps/codecov/docker-compose.yml index c8ef787..6455b74 100644 --- a/apps/codecov/docker-compose.yml +++ b/apps/codecov/docker-compose.yml @@ -10,7 +10,7 @@ x-environment: &environment CODECOV_SCHEME: https RUN_ENV: ENTERPRISE x-volumes: &volumes - - ../../apps-data/${APP_NAME}/config:/config + - ./codecov.yml:/config/codecov.yml services: gateway: image: codecov/self-hosted-gateway:latest-stable @@ -56,5 +56,5 @@ services: - timescale environment: *environment volumes: - - ../../apps-data/${APP_NAME}/config:/config + - ./codecov.yml:/config/codecov.yml - ../../apps-data/${APP_NAME}/archive:/archive diff --git a/apps/traefik/docker-compose.yml b/apps/traefik/docker-compose.yml index 0aed756..6e03b4a 100644 --- a/apps/traefik/docker-compose.yml +++ b/apps/traefik/docker-compose.yml @@ -10,7 +10,7 @@ services: volumes: - /etc/localtime:/etc/localtime:ro - /var/run/docker.sock:/var/run/docker.sock:ro - - ../../apps-data/${APP_NAME}/config/traefik.yml:/traefik.yml:ro + - ./traefik.yml:/traefik.yml:ro - ../../apps-data/${APP_NAME}/acme.json:/acme.json networks: - traefik diff --git a/apps/traefik/config/traefik.template.yml b/apps/traefik/traefik.yml.tpl similarity index 100% rename from apps/traefik/config/traefik.template.yml rename to apps/traefik/traefik.yml.tpl diff --git a/deploy/deploy.py b/deploy/deploy.py index ea27ec8..0dee110 100644 --- a/deploy/deploy.py +++ b/deploy/deploy.py @@ -9,7 +9,6 @@ Reads a JSON config from stdin (see README's "deploy-shared.yml" section for the exact shape). """ -import io import json import shlex import shutil @@ -65,19 +64,15 @@ def build_release(config, work_dir): def render_app_configs(release_dir, app, values): - template_dir = release_dir / "apps" / app / "config" - if not template_dir.is_dir(): - return {} - rendered = {} - for template in sorted(template_dir.glob("*.template.*")): - filename = template.name.replace(".template.", ".", 1) - rendered[filename] = render_template(template.read_text(), values) - return rendered + app_dir = release_dir / "apps" / app + for template in sorted(app_dir.glob("*.tpl")): + rendered_path = template.with_name(template.stem) + rendered_path.write_text(render_template(template.read_text(), values)) + rendered_path.chmod(0o600) def resolve_app_envs(config, work_dir, release_dir, age_key_file): pull_dir = work_dir / "envs" - rendered_configs = {} for app, app_config in config["apps"].items(): paths = [ download_ref(ref, pull_dir / app / str(index)) @@ -90,9 +85,7 @@ def resolve_app_envs(config, work_dir, release_dir, age_key_file): app_env_path.write_text(plaintext) app_env_path.chmod(0o600) - rendered_configs[app] = render_app_configs(release_dir, app, parse_dotenv(plaintext)) - - return rendered_configs + render_app_configs(release_dir, app, parse_dotenv(plaintext)) def list_required_networks(release_dir): @@ -140,16 +133,6 @@ def push_release(connection, archive_path, release_path): connection.run(f"chmod 600 {shlex.quote(release_path)}/apps/*/.env", hide=True) -def push_app_configs(connection, base_path, rendered_configs): - for app, files in rendered_configs.items(): - if not files: - continue - config_dir = f"{base_path}/apps-data/{app}/config" - connection.run(f"mkdir -p {shlex.quote(config_dir)}", hide=True) - for filename, text in files.items(): - connection.put(io.StringIO(text), remote=f"{config_dir}/{filename}") - - def prune_releases(connection, releases_path, keep_releases): result = connection.run(f"ls -1dt {shlex.quote(releases_path)}/*/ 2>/dev/null || true", hide=True) releases = [line.strip().rstrip("/") for line in result.stdout.splitlines() if line.strip()] @@ -158,7 +141,7 @@ def prune_releases(connection, releases_path, keep_releases): connection.run("rm -rf " + " ".join(shlex.quote(release) for release in stale), hide=True) -def deploy_to_host(host, archive_path, rendered_configs, apps, networks, config): +def deploy_to_host(host, archive_path, apps, networks, config): connection = Connection(host) connection.client.set_missing_host_key_policy(paramiko.AutoAddPolicy()) @@ -173,7 +156,6 @@ def deploy_to_host(host, archive_path, rendered_configs, apps, networks, config) bootstrap_host(connection, base_path, networks) push_release(connection, archive_path, release_path) - push_app_configs(connection, base_path, rendered_configs) for app in apps: connection.run(f"mkdir -p {shlex.quote(f'{base_path}/apps-data/{app}')}", hide=True) @@ -209,7 +191,7 @@ def main(): age_key_file.chmod(0o600) release_dir = build_release(config, work_dir) - rendered_configs = resolve_app_envs(config, work_dir, release_dir, age_key_file) + resolve_app_envs(config, work_dir, release_dir, age_key_file) archive_path = archive_release(release_dir, work_dir) networks = list_required_networks(release_dir) @@ -217,7 +199,7 @@ def main(): for host in config["hosts"]: print(f"Deploying to {host}") - deploy_to_host(host, archive_path, rendered_configs, apps, networks, config) + deploy_to_host(host, archive_path, apps, networks, config) if __name__ == "__main__": diff --git a/deploy/tests/test_deploy.py b/deploy/tests/test_deploy.py index 4d3942b..0b7f187 100644 --- a/deploy/tests/test_deploy.py +++ b/deploy/tests/test_deploy.py @@ -145,25 +145,27 @@ def fake_download_ref(ref, out_dir, default_asset=None, run=None): class RenderAppConfigsTest(unittest.TestCase): - def test_renders_templates_found_for_app(self): + def test_renders_templates_found_for_app_next_to_the_template(self): with tempfile.TemporaryDirectory() as directory: release_dir = Path(directory) - config_dir = release_dir / "apps" / "traefik" / "config" - config_dir.mkdir(parents=True) - (config_dir / "traefik.template.yml").write_text("email: ${APPS_ADMIN_MAIL}\n") + app_dir = release_dir / "apps" / "traefik" + app_dir.mkdir(parents=True) + (app_dir / "traefik.yml.tpl").write_text("email: ${APPS_ADMIN_MAIL}\n") - rendered = deploy.render_app_configs(release_dir, "traefik", {"APPS_ADMIN_MAIL": "a@example.com"}) + deploy.render_app_configs(release_dir, "traefik", {"APPS_ADMIN_MAIL": "a@example.com"}) - self.assertEqual(rendered, {"traefik.yml": "email: a@example.com\n"}) + rendered_path = app_dir / "traefik.yml" + self.assertEqual(rendered_path.read_text(), "email: a@example.com\n") + self.assertEqual(oct(rendered_path.stat().st_mode)[-3:], "600") - def test_returns_empty_dict_when_no_config_dir(self): + def test_does_nothing_when_app_has_no_templates(self): with tempfile.TemporaryDirectory() as directory: release_dir = Path(directory) (release_dir / "apps" / "rybbit").mkdir(parents=True) - rendered = deploy.render_app_configs(release_dir, "rybbit", {}) + deploy.render_app_configs(release_dir, "rybbit", {}) # does not raise - self.assertEqual(rendered, {}) + self.assertEqual(list((release_dir / "apps" / "rybbit").iterdir()), []) class ResolveAppEnvsTest(unittest.TestCase): @@ -171,9 +173,7 @@ def _release_dir_with_app(self, work_dir, app, template=None): app_dir = work_dir / "release" / "apps" / app app_dir.mkdir(parents=True) if template is not None: - config_dir = app_dir / "config" - config_dir.mkdir() - (config_dir / f"{app}.template.yml").write_text(template) + (app_dir / f"{app}.yml.tpl").write_text(template) return work_dir / "release" def test_writes_decrypted_env_and_renders_configs(self): @@ -189,12 +189,14 @@ def test_writes_decrypted_env_and_renders_configs(self): patch.object(deploy, "download_ref", return_value=ciphertext), patch.object(deploy, "decrypt_env", return_value="APPS_ADMIN_MAIL=a@example.com\n"), ): - rendered = deploy.resolve_app_envs(config, work_dir / "work", release_dir, work_dir / "key.txt") + deploy.resolve_app_envs(config, work_dir / "work", release_dir, work_dir / "key.txt") env_path = release_dir / "apps" / "traefik" / ".env" self.assertEqual(env_path.read_text(), "APPS_ADMIN_MAIL=a@example.com\n") self.assertEqual(oct(env_path.stat().st_mode)[-3:], "600") - self.assertEqual(rendered, {"traefik": {"traefik.yml": "email: a@example.com\n"}}) + + rendered_path = release_dir / "apps" / "traefik" / "traefik.yml" + self.assertEqual(rendered_path.read_text(), "email: a@example.com\n") def test_raises_on_collision_within_one_apps_own_env_refs(self): with tempfile.TemporaryDirectory() as directory: @@ -311,10 +313,6 @@ def test_full_sequence(self): work_dir = Path(directory) archive_path = work_dir / "release.tar.gz" archive_path.write_text("fake archive\n") - rendered_configs = { - "traefik": {"traefik.yml": "email: a@example.com\n"}, - "rybbit": {}, - } config = {"hosts": ["deploy@host"], "keep_releases": 5} fake = FakeConnection("deploy@host") @@ -322,7 +320,6 @@ def test_full_sequence(self): deploy.deploy_to_host( "deploy@host", archive_path, - rendered_configs, apps=["traefik", "rybbit"], networks=["traefik", "databases", "mcp"], config=config, @@ -330,9 +327,7 @@ def test_full_sequence(self): self.assertEqual(fake.uploads[0], (str(archive_path), fake.uploads[0][1])) self.assertTrue(fake.uploads[0][1].endswith(".tar.gz")) - - config_upload = next(upload for upload in fake.uploads if upload[1].endswith("traefik.yml")) - self.assertEqual(config_upload[0].getvalue(), "email: a@example.com\n") + self.assertEqual(len(fake.uploads), 1) joined = "\n".join(fake.commands) self.assertIn("docker network create traefik", joined)