Portfolio technical elevation: real CI verification, hardening, verified backups - #1
Merged
Merged
Conversation
…s, and container hardening - Replace :latest tags with pinned versions (n8n 2.35.7, Baserow 2.3.3, Postgres 15.15-alpine3.24, Caddy 2.11.4-alpine) so upgrades are deliberate, not silent on the next pull. - Add a healthcheck to Baserow (postgres and n8n already had one) using its /api/_health/ endpoint. - Add per-service memory limits and rotated/capped JSON logging so a runaway execution or import can't OOM or fill the disk on a small VPS. - Harden every service with no-new-privileges; drop all Linux capabilities on Caddy/n8n/Postgres and re-add only what each specifically needs. Baserow is deliberately left uncapped for now — its process supervisor's root requirements aren't confirmed safe to guess at without live testing.
Tracks the version pins just added to docker-compose.yml (docker ecosystem, GA support for docker-compose.yml since Feb 2025) and third-party Action versions (github-actions ecosystem) weekly.
validate.yml now has a second job that boots the dev stack for real (docker compose up --wait) and curls both health endpoints externally, instead of only checking that the compose files parse. Also pins actions/checkout by commit SHA rather than a floating tag. Makefile gains test, restore, backup-test, and security-scan targets so the new verification tooling has a local entry point, not just a CI-only one.
Scans images, not the compose file — Trivy has no native docker-compose misconfiguration scanner (confirmed against its own issue tracker), so this intentionally doesn't claim IaC coverage it doesn't have. Non-blocking (visibility via the Security tab, not a merge gate) since some CVEs in upstream images won't be fixable from this repo. aquasecurity/trivy-action is pinned by commit SHA, not a tag — 75 of 76 of its version tags were force-pushed with malware in March 2026.
scripts/restore.sh restores all three volumes from a backup.sh snapshot (mirrors its dynamic project-name detection). scripts/backup-drill.sh proves the cycle actually recovers data: writes a marker row, backs up, destroys the volumes, restores, and fails if the marker isn't back. Shared between `make backup-test` (local) and the new backup-drill.yml workflow (weekly + on-demand in CI) so the two can't drift apart. docs/backup-and-restore.md updated to document both.
- Replace the ASCII architecture diagram with a Mermaid graph (renders natively on GitHub, no added tooling). - Add three ADRs: why Caddy, why one shared Postgres instance, why two networks instead of one. - New README "Security & Reliability" section summarizing the hardening, scanning, and verified-backup work in one place for a reviewer. - Update docs/updating-services.md and docs/troubleshooting.md for the version-pinning/Dependabot change and the new capability-hardening failure mode respectively. - Update project CLAUDE.md with the new make targets, workflows, the Baserow capability-hardening caveat, and the blank-secrets-in-.env.example gotcha for anything that boots the stack directly from the template.
|
You are seeing this message because GitHub Code Scanning has recently been set up for this repository, or this pull request contains the workflow file for the Code Scanning tool. What Enabling Code Scanning Means:
For more information about GitHub Code Scanning, check out the documentation. |
Will be reverted before merge — see comment in the workflow file.
…an failure postgres:15.15-alpine3.24 was never a published tag — 15.15 only ever paired up to alpine3.23; alpine3.24 variants start at 15.18. Verified the real current tag against the Docker Hub API directly rather than trusting a summarized fetch (which is exactly how the wrong tag got in here). Correct tag is postgres:15.19-alpine3.24, confirmed to exist. Also guard the Trivy SARIF upload step on the output file actually existing — it was crashing the whole job with a bare "Path does not exist" whenever the scan step failed for any reason (this bug, or a future transient one), instead of failing informatively or being skipped cleanly. Caught by the smoke-test and backup-drill jobs actually trying to boot the stack — exactly the kind of failure `docker compose config --quiet` alone would never catch.
…xisted Real bug the smoke-test job caught on its first actual boot of the stack: POSTGRES_DB only creates one database (the default, "automation") at initdb time. Neither n8n's pg client nor Baserow's Django backend creates its own database on connect — both just fail hard with "database does not exist". This means N8N_DB_NAME/BASEROW_DB_NAME never worked; the stack had apparently never been booted for real before this CI job existed. Fix: postgres-init/init-app-databases.sh, mounted into Postgres's docker-entrypoint-initdb.d/, creates both databases explicitly using the official image's first-init hook. Only runs once, against an empty postgres_data volume — documented in docs/troubleshooting.md and CLAUDE.md, including the manual remediation for anyone already running an already-initialized volume from before this fix. Corrected the same false "auto-created by each app" claim in .env.example and CLAUDE.md, which had been asserting this incorrectly since the original plan.
…ot migrations Both n8n and Baserow actually came up correctly after the database fix — confirmed from the raw logs: Baserow logged a real 200 on /api/_health/ and finished all Django migrations. It still got marked unhealthy because the old healthcheck window (start_period 45s + 5 retries * 30s interval ≈ 195s) was too tight: first-boot on an empty volume also runs Baserow's built-in template sync (importing dozens of bundled templates via a background worker), which took ~150s before the health endpoint reliably responded on a GitHub Actions runner. Widened to start_period 150s / interval 15s / retries 10 (~300s ceiling, checks more often once past start_period so it doesn't overshoot success by much). Bumped --wait-timeout from 180 to 320 everywhere docker compose up --wait is used, so compose's own wait doesn't time out before the healthcheck's retry budget is exhausted.
--wait fails permanently on the first "unhealthy" transition, even if the container recovers seconds later. Confirmed from raw logs: Baserow's first-boot migration + template sync briefly reported unhealthy under CI resource contention, then answered 200 on every single subsequent check for over two minutes — but --wait had already killed the job on the first blip. scripts/wait-for-healthy.sh polls `docker inspect` directly in a loop and only fails on a genuine timeout with nothing ever reporting healthy, which is what a smoke test should actually care about. Used by validate.yml's smoke-test job, backup-drill.yml's boot step, backup-drill.sh's post-restore wait, and `make test`.
…irectly Still failing after the previous fix, with a stranger symptom: Baserow's /api/_health/ endpoint returned 200 on every single logged request for over two minutes straight, yet docker inspect .State.Health.Status never once reported "healthy" — confirmed from raw logs (six consecutive 200s, script still timed out at 400s). Docker's own HEALTHCHECK state tracking appears to get stuck under GitHub Actions runner resource contention independent of the actual endpoint's behavior. wait-for-healthy.sh now runs each service's check command directly via `docker compose exec` and retries until THAT succeeds, never consulting Docker's Health.Status at all. This is the same command the container's own HEALTHCHECK uses, just executed and retried outside of Docker's internal bookkeeping — immune to whatever was causing that bookkeeping to get stuck. docker-compose.yml's Baserow healthcheck config is left in place (still useful for `docker compose ps` / operator visibility on a real deploy) but its comment now says plainly that it isn't load-bearing for anything in this repo anymore.
…rst boot Root cause of the persistent boot failure, finally isolated from the raw logs: "Worker exited prematurely: signal 9 (SIGKILL)" in Baserow's celery export worker — a classic OOM kill, not a healthcheck or timing issue at all. Baserow runs gunicorn + a celery worker + celery beat concurrently, and first boot on an empty volume adds Django migrations and template-sync import on top of that. 1g covers steady-state but not that burst. Raised to 2g. Also corrected docs/initial-setup.md's VPS sizing guidance — "2GB RAM, 4GB recommended" understated it; the real worst-case ceiling across all four services' limits is ~3.6GB, and Baserow specifically needs its full 2g during first boot, not just at steady state.
…g it Still timing out on Baserow after fixing the OOM (1g -> 2g) — but this time the app log showed 28 consecutive 200s on /api/_health/ spanning the full 4.5 minutes the script was polling, while the script itself reported "not ready yet" on every single attempt. Those 200s are Docker's own internal HEALTHCHECK directive (still defined in docker-compose.yml, running on its own schedule) — a completely separate code path from this script's own `docker compose exec` calls, which apparently were failing for a reason the script was actively discarding (>/dev/null 2>&1, exit code unchecked beyond zero/nonzero). No new fix yet — this commit only makes the next failure diagnosable instead of guessing again from indirect evidence.
Root cause, finally confirmed from the debug instrumentation added last commit: "sh: 1: wget: not found" on every single attempt. Baserow's own CMD-array healthcheck (["CMD", "wget", ...], no shell involved) resolves wget fine — whatever sets up Baserow's PATH for its actual entrypoint process isn't present in a fresh `docker compose exec ... sh` session. n8n and postgres both worked correctly through sh -c the whole time (n8n had one transient "connection refused" that self-resolved) — this was specific to Baserow. Fixed by execing wget directly for Baserow, matching how the CMD-array form resolves it, while leaving postgres/n8n on sh -c since they need it (env var expansion) and it already works for them.
Confirmed definitively: baserow/baserow:2.3.3 has no wget binary anywhere in $PATH — OCI runtime itself reports "exec: wget: executable file not found in \$PATH" trying to run it directly (no shell involved at all this time). docker-compose.yml's own wget-based HEALTHCHECK for Baserow has apparently never actually been able to run in this image; the 200s seen in earlier debugging were Baserow's own internal startup watcher process hitting its own endpoint, unrelated to Docker's HEALTHCHECK or this script. Stopped trying to find something inside each app's container that can make the HTTP call. Both current call sites already run with the dev compose override active (host ports 5678/8080 mapped), so this now just curls those directly from the caller's machine — sidesteps every container-image assumption entirely. Postgres keeps `docker compose exec` + pg_isready, which has worked correctly in every run so far and has no host-reachable alternative (no port mapping, by design).
Was only there to validate the workflow inside this PR before merge, since workflow_dispatch requires the file to already exist on the default branch. Confirmed working (backup-restore-verify passed). Back to schedule + workflow_dispatch only, as designed — it will run for real once merged to main and can be triggered on-demand via gh workflow run after that.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Elevates the stack from "validates in CI" to "provably works" — real runtime smoke-testing, closed hardening gaps, automated version currency, and a proven (not just documented) backup/restore path. Full plan:
docs/superpowers/plans/2026-08-23-portfolio-technical-elevation.md(gitignored, local only).:latestno-new-privilegeseverywhere;cap_drop: ALL+ minimal re-adds on Caddy/n8n/Postgres (Baserow deliberately left uncapped pending live verification — see CLAUDE.md)docker compose configmake backup-testlocallyaquasecurity/trivy-actionsupply-chain compromise, March 2026)Test plan
Validateworkflow green — compose config, shellcheck, and the new smoke-test job (boots dev stack, checks/healthz+/api/_health/)Security Scanworkflow runs cleanly (results land in the Security tab regardless of findings — it's non-blocking)Backup Drillworkflow green (can trigger manually viaworkflow_dispatchrather than waiting for the weekly schedule)cap_drop/cap_add) don't break Caddy TLS binding or Postgres first-init in the smoke-test logsNo local Docker daemon was available while writing this — everything above needs verification against real CI runs, and the hardening task in particular may need a follow-up commit if a capability list needs adjusting.
🤖 Generated with Claude Code
https://claude.ai/code/session_01Nn1ukWMFbhPZHHGSTLu7AQ