Scaling and deploy availability: what pins the consolidated stack to one host, and the options

[aaylward]

The consolidated stack runs every service as a single container on one Lightsail host, fronted by Caddy, deployed by deploy/consolidated/deploy.sh. That has been the right shape for a long time, but it has two properties worth naming: a deploy drops traffic for every service it touches, and there is no path to running a service on more than one machine.

This issue is the survey — what actually pins the stack to one host today, what the options are, and what the tradeoffs cost. It is deliberately not a plan; the prerequisite work at the bottom is worth doing under any of the options, and the choice between orchestrators can wait until after it.

Two goals, which have different answers and should not be conflated:

• Availability during deploys. Achievable on a single host. This is the near-term want.
• Horizontal scale / surviving host loss. Needs a second machine and a real orchestrator, and is mostly of interest here as something to learn on rather than something the traffic demands.

What a deploy costs today

deploy.sh ends in docker compose up -d, which recreates each changed container in place: stop old, start new. Caddy has exactly one upstream per route, so from the moment the old container stops until the new one is accepting connections, that route 502s. For the JVM services (one_d4, mcpserver) that window is JVM boot, not milliseconds — the healthchecks allow a 15s start_period, which is a fair proxy for how long the gap can be.

Worth noting what is already right, because it is the hard part and it's done:

• Every image is pinned to a commit SHA, and a rollback is --sha. Reproducible deploys are a prerequisite for rolling ones.
• Nearly every service has a real /health endpoint and a working healthcheck (Steady probe traffic for Rust services so deploys stop blinding the dashboards #1307). Rolling updates are exactly "don't proceed until the new instance is healthy" — without the probes there is nothing to gate on.
• caddy reload is already used at the end of every deploy, and it is graceful — Caddy loads the new config and drains the old one without dropping connections. Config changes are already zero-downtime; only container replacement isn't.

What pins the stack to one host

1. Postgres on a local volume

one_d4_postgres writes to the one_d4_pgdata named volume on the host, and serves both one_d4 and golf_hub (golf_hub_db_init provisions the second database). Anything that runs on a second machine has to reach one database that lives on a specific machine. This is the single biggest item, and it is the one whose fix pays off regardless of which orchestrator is eventually picked.

Related: there is no backup or restore drill for one_d4_pgdata or forgejo_data that I'm aware of. That matters more than any scaling work — replicas protect against a crashed process, only backups protect against losing the data.

2. Host bind-mounts for config

compose.yaml mounts /etc/<service> into ten containers, and README.md documents them as "Configuration Requirements." Most of them are not actually load-bearing. Grepping the service sources for reads of those paths:

Genuinely used:

Mount	Used for	Reader
/etc/r3dr/db_config	DB connection string, plain text	domains/r3dr/apis/r3dr/config.go:23 — DB_CONNECTION_STRING env takes precedence
/etc/one_d4/db_config	JDBC URL, plain text	IndexerModule.java:54 — $INDEXER_DB_URL env takes precedence, H2 in-memory is the fallback
/etc/microgpt-serve/model/	model weights (weights.safetensors, meta.json), via MODEL_DIR	domains/ai/apis/microgpt_serve/src/main.rs:34
/etc/forgejo	Forgejo's app.ini, mounted at /data/gitea/conf; deploy.sh copies it each deploy	Forgejo itself

No evidence of any read: games_ws_backend, portrait, prom_proxy, mithril, posterize, mcpserver. All config those six need arrives via environment variables in compose.yaml. The mounts look like boilerplate that propagated from service to service, and the README documents directories nothing consumes.

So the actual portability debt here is two connection strings and one model directory — much smaller than the mount list suggests. Both config readers already implement env-var-first with file fallback, so the code is already portable; only the deployment leans on the file paths.

3. Caddy's trust boundary is a single pinned IP

Caddy holds a static 172.28.0.2, anchored in compose.yaml and passed to services as TRUSTED_PROXY_CIDRS so x-forwarded-for is trusted from Caddy and nowhere else (smithy-cpp ADR-0012). deploy.sh additionally pins the network's --ip-range to keep the dynamic pool off that address. This is a good design for one proxy on one bridge network, and it does not survive more than one Caddy or a different network topology. Widening it to a CIDR (or moving the trust decision to a mesh/ingress identity) is a precondition for any multi-proxy setup.

4. Caddy's ACME certificate storage is instance-local

Certs live in the caddy_data volume. Two Caddy instances with independent storage will each try to solve ACME challenges and will fight over rate limits. Caddy supports clustered storage backends (Redis, Consul) for exactly this, or TLS terminates somewhere else entirely.

5. In-memory game state in games_ws_backend (v1 — retiring)

golf/golf_hub.go keeps rooms, clientContexts, playerToClient, and disconnectedSessions in process-local maps with no database anywhere in the service. Two replicas means two players in the same game landing on different processes and never seeing each other.

This is not work to do — v1 golf is going away. golf_hub (v2, /games/v2/*) already made the right call and keeps tickets and resume tokens in Postgres specifically so they survive deploys. The v1 hub is a "wait it out" item, not a migration target. Note that #1234 covers the separate question of what golf_hub still needs before it can run multi-instance — its hub mutex holds DB round-trips, which single-instance play hides.

6. The observability stack is legitimately per-node

cadvisor runs privileged with /rootfs, and otelcol's hostmetrics receiver reads /hostfs. These are supposed to be per-machine agents — under an orchestrator they become DaemonSets rather than a scaled service. Not a blocker, just a thing that changes shape rather than moving.

Options

A. Stay on Compose, fix deploy availability only

Two variants, both single-host:

A1 — two replicas behind Caddy load balancing. reverse_proxy accepts multiple upstreams with lb_policy and active health checks (health_uri /health), so a container that is down is taken out of rotation rather than 502ing. Deploy by replacing one replica at a time. Cheapest thing that removes the deploy gap.

A2 — blue/green by Caddyfile flip. Run svc_a and svc_b, deploy to the idle one, wait for its healthcheck, then flip the upstream and caddy reload. Since the reload is already graceful and already part of every deploy, this is close to free in mechanism — the cost is doubling the container count and the bookkeeping of which colour is live.

Cost of both: doubles memory for anything replicated (the fleet's limits currently total roughly 4.5 CPU / 4 GB, so there is headroom), and neither gives you anything if the host dies. docker compose up --scale does not do rolling updates, so the sequencing is scripted by hand in deploy.sh.

B. Docker Swarm

The literal answer to "multi-host with the compose files I already have." The deploy: blocks in compose.yaml are already Swarm syntax; update_config: {order: start-first, parallelism: 1} is exactly the rolling-update behaviour this issue wants, and overlay networking plus secrets come with it. Lowest effort per unit of capability by a wide margin.

Honest caveat: Swarm is in maintenance mode. It works and it is stable, but it is not where the ecosystem is going, and the learning transfers to nothing else.

C. k3s (explicitly not minikube)

If the goal includes learning Kubernetes, k3s (or k0s) is the small-scale production option: single binary, runs on a modest VPS, add agent nodes as you grow. minikube and kind are local-development clusters — they run a throwaway single-node cluster and are not deployment targets, so they don't answer this question at all.

Rolling updates, readiness gating, and horizontal scale are native. The Caddyfile's method+path routing maps onto Ingress/Gateway API rules, or Caddy stays as the ingress and the routing config comes along unchanged.

Cost, stated honestly: manifests for every service, cert-manager or equivalent for the TLS that Caddy does for free today, a storage story for Postgres, and a standing ops burden that is real even when nothing is happening. For a fleet this size the capability is oversized — the justification is that learning it is the point.

D. Managed platform (Fly.io, Cloud Run, ECS) + managed Postgres

Skips the orchestrator question. Real option, but it needs the same prerequisites below, and it trades the learning goal for convenience.

Prerequisites — worth doing under every option

These are independent of the orchestrator choice, so they can start now and nothing is wasted:

1. Give Postgres a home independent of the app host — managed (Neon, RDS, Fly Postgres) or a dedicated node with backups. This one move makes most of the fleet stateless and portable.
2. Establish backup and restore for one_d4_pgdata and forgejo_data, with an actual restore drill. Highest value-per-effort item in this issue and not really optional.
3. Delete the six unused /etc/<service> mounts and the README section documenting them. Worth confirming the directories are empty on the live host first, since the code search only proves nothing reads them at the paths I could find.
4. Move the two db_config files to environment variables, next to the ONE_D4_DB_PASSWORD and GOLF_HUB_DB_PASSWORD that deploy.sh already injects through the host .env. Both readers prefer the env var already, so this is config-only.
5. Decide where the microgpt model lives — baked into the image (immutable and versioned with the code, at the cost of a fat image), a named volume, or object storage pulled at boot. This is the only interesting one of the four.
6. Widen TRUSTED_PROXY_CIDRS from a single pinned address to a subnet, so more than one proxy can exist.

Steps 3 through 6 are small and mostly mechanical. Steps 1 and 2 are the real work, and they are the ones that pay off no matter which direction the stack goes afterwards.

Suggested order

Prerequisites first (1 and 2 especially), then A1 or A2 to close the deploy gap on the current host — that is the near-term want and it does not require choosing an orchestrator. Then, if the learning goal is still live, k3s on a second node as a deliberate exercise, with Swarm as the fallback if the ops burden turns out to outweigh the interest.

[aaylward]

Followed up on the /etc mount audit with a deeper pass — not just literal path greps, but config libraries, --config flags, CONFIG_PATH-style env vars, default path constants, and framework-level conventions. The conclusion holds for all six, and a few details are worth recording.

Repo-wide, exactly three code sites read a file out of a mounted /etc/<service> directory — IndexerModule.java:54, r3dr/config.go:23, and the MODEL_DIR in compose.yaml. The strings /etc/games_ws_backend, /etc/portrait, /etc/prom_proxy, /etc/mithril, /etc/posterize, and /etc/mcpserver appear only in compose.yaml and in README.md:188-193. Every one of the six is configured purely by environment variables.

Two nuances that would make a naive cleanup wrong:

• mithril does read files — but never from /etc. src/main.rs:70 hardcodes mithril.runfiles/_main/domains/games/apis/mithril/data and loads words.txt plus a cached .graph from there (wordchains/src/init.rs:6, storage.rs). Those ship in the image as bazel data deps (BUILD.bazel:11-14) and the path is relative to the process CWD, so it cannot resolve into /etc/mithril. Removing the mount is safe; removing the runfiles data is not.
• mcpserver reuses one_d4's database code but does not inherit its config-file behaviour. McpModule.java's indexerJdbcUrl() reads INDEXER_DB_URL only, falling back to jdbc:h2:mem:indexer. The file-reading logic lives in IndexerModule.java, in a bazel target mcpserver doesn't depend on (it pulls one_d4's db|dto|engine|model|service|worker|queue, not the module target), so those @Factory beans are not on its classpath.

Why these never failed loudly: Docker creates a missing host bind-mount directory rather than erroring, so an inert mount is indistinguishable from a satisfied one at runtime. git log -S "/etc/mithril" shows the mithril mount arrived with the service itself (157d485, #868) and was carried into the consolidated compose wholesale (3f4af4b, #895) — copied from the r3dr pattern rather than driven by a config need. The README entries are the tell: the /etc/r3dr line names a file, the six below it name only directories.

One more thing that can't replicate: the v1 JWT secret

Adding to the "in-memory state" item above, since it's a second independent blocker in the same service and it's not about maps. golf/auth.go:18 generates the JWT signing secret with crypto/rand at process start:

func NewTokenManager() *TokenManager {
	secret := make([]byte, 32)
	if _, err := rand.Read(secret); err != nil {

So the secret is per-process. Two replicas cannot validate each other's tokens, and every deploy silently invalidates every outstanding token — which is a small availability cost the current single-container deploy already pays, separate from the 502 window. Same disposition as the rest of v1: noted, not work, since it's retiring. Worth being explicit that a hypothetical "just run two of them behind ip_hash" would not have worked even setting the room maps aside.

None of this changes the plan — steps 3 and 4 in the prerequisites stand as written, and the six mounts plus their README entries can go together.