Shut down workers on idle instead of a static maximum lifetime — Closes #100

README.md

@@ -183,7 +183,7 @@ As with dev, confirm after the first promotion that the running prod tasks refer
 A few operational caveats of the moving-tag pattern:
 
 - **Version skew across a deploy.** The API service rolls immediately, but the worker fleet only advances when the next `EcsProvisioner` `RunTask` pulls the freshly-pushed `:dev`. Between the API roll and the next worker launch, a newly-rolled API can dispatch to in-flight workers still running the prior `:dev` image — so a deploy has a transient window where the API and worker code can be one commit apart. Keep the API↔worker dispatch contract backward-compatible across adjacent commits.
-- **A wool version bump is a deliberate exception to that.** wool admits a worker only when the proxy's version is `<=` the worker's within the same major (`is_version_compatible`), applied as a discovery filter, and `wool.protocol.__version__` is just the installed package version. Since the pipeline rolls the API first, a bump means the new API rejects **every** in-flight worker until the fleet turns over. It self-heals — jobs stay `pending` and the durable scheduler drains them once fresh workers spawn — but two second-order effects are worth knowing. Rejected-but-running workers still count toward `ECS_MAX_WORKERS` in the provisioner's pre-`RunTask` census, and nothing reaps them before `CFDB_WORKER_MAX_LIFETIME_SECONDS` (5 h, longer than the 4 h dispatch deadline), so a bump landing on a near-capped fleet can wedge spawning long enough to fail jobs `capacity:`. And the symptom is `NoWorkersAvailable`, indistinguishable from the TLS failures above. To make it a non-event, drain the fleet as part of the deploy that carries the bump — `aws ecs list-tasks --cluster <cluster> --family <worker-family>` then `stop-task` on each, or simply confirm it is empty before promoting. The rollback direction is safe: an older API against newer workers passes the gate.
+- **A wool version bump is a deliberate exception to that.** wool admits a worker only when the proxy's version is `<=` the worker's within the same major (`is_version_compatible`), applied as a discovery filter, and `wool.protocol.__version__` is just the installed package version. Since the pipeline rolls the API first, a bump means the new API rejects **every** in-flight worker until the fleet turns over. It self-heals — jobs stay `pending` and the durable scheduler drains them once fresh workers spawn — but two second-order effects are worth knowing. Rejected-but-running workers still count toward `ECS_MAX_WORKERS` in the provisioner's pre-`RunTask` census. Since they receive no dispatches, idle shutdown reaps them after `CFDB_WORKER_IDLE_TIMEOUT_SECONDS` (default 10 min), so the wedge window is minutes rather than the hours the max-lifetime ceiling would allow — but only when the running fleet is already on an idle-aware image (wool ≥ 0.14): a bump *from* an older fleet leaves the old workers unable to self-reap early, and a near-capped fleet can wedge spawning long enough to fail jobs `capacity:`. And the symptom is `NoWorkersAvailable`, indistinguishable from the TLS failures above. To make it a non-event, drain the fleet as part of the deploy that carries the bump — `aws ecs list-tasks --cluster <cluster> --family <worker-family>` then `stop-task` on each, or simply confirm it is empty before promoting. Note that a bump does not require a wool code change: cfdb tracks `wool~=0.14.0`, so any lock refresh that moves the resolved version — a patch release, not just a deliberate upgrade — is itself a bump requiring this same procedure. The rollback direction is safe: an older API against newer workers passes the gate.
 - **Single-environment moving tag.** `MOVING_TAG` is hard-coded to `dev` in the workflow, so this pipeline targets exactly one environment. A second environment (e.g. `prod`) would need its own moving tag, repo variables, and task-def wiring — not yet parameterized.
 - **Rollout wait ceiling.** `aws ecs wait services-stable` polls for up to ~10 minutes (40 attempts × 15 s) before timing out. A genuinely slow or wedged rollout will fail the workflow at that ceiling even though the `update-service` call itself succeeded; the deploy may still converge afterward, or the circuit breaker (below) may roll it back.
 - **Stale GitHub secrets.** The old `BACKEND_STACK_NAME` and `WORKERS_STACK_NAME` GitHub secrets are no longer used by this workflow (it no longer runs `cloudformation deploy`) and can be deleted.
@@ -842,7 +842,7 @@ Cache keys are content-addressed using each file's upstream `md5`, so a byte cha
 **Bounded concurrency, durable queuing, and admission control.** Dispatch is bounded on three cooperating layers so an unauthenticated burst on `/data` and `/index` can't oversubscribe the worker fleet or queue unbounded work:
 
 - **Per-worker backpressure** — each worker accepts at most `CFDB_WORKER_MAX_CONCURRENT_TASKS` tasks at once (default `1`), serializing the subprocess pipelines on a 1-vCPU worker. A worker at capacity rejects the dispatch and the API's priority load balancer rotates to the next worker.
-- **Priority (leaky-bucket) load balancing** — the API offers each task to discovered workers in a stable order, so load concentrates on the lowest-ordered workers and over-provisioned workers drain to idle and self-reap (via `CFDB_WORKER_MAX_LIFETIME_SECONDS`) instead of every worker carrying a thin perpetual slice.
+- **Priority (leaky-bucket) load balancing** — the API offers each task to discovered workers in a stable order, so load concentrates on the lowest-ordered workers and over-provisioned workers drain to idle and self-reap (on the ECS profile, via `CFDB_WORKER_IDLE_TIMEOUT_SECONDS`, with `CFDB_WORKER_MAX_LIFETIME_SECONDS` as the backstop) instead of every worker carrying a thin perpetual slice.
 - **Durable queue + retry-to-deadline** — when no worker has capacity, the job is **not** failed and does **not** block the request: it stays `pending` and a durable, Mongo-backed scheduler re-attempts dispatch every `CFDB_WORKFLOW_RETRY_INTERVAL_S` (plus jitter) until a worker frees up or the `CFDB_WORKFLOW_DISPATCH_DEADLINE_S` deadline elapses (then it is failed with a `capacity:`-prefixed error). Because the queue lives in Mongo, an API restart resumes it. On every scheduler tick (including the first, on boot) an orphan-recovery sweep re-queues jobs a crash left mid-flight — a `running` job whose API consumer died, or a fresh `pending` claim that never rescheduled — once they pass the stale threshold (`CFDB_WORKFLOW_STALE_THRESHOLD_S`), so recovery is autonomous and does not wait for a client to re-request the file. Recovery shares the same deadline clock as a fresh job: the re-queue preserves the original submission time, so an orphan older than `CFDB_WORKFLOW_DISPATCH_DEADLINE_S` is failed `capacity:` on its first recovery attempt rather than resumed (its committed cache artifacts survive for a later fresh `GET` to reuse) — recovery is best-effort, not unbounded. On the ECS profile, an overflow also requests one bounded worker spawn (the leaky bucket overflowing), inverting the old unconditional per-request spawn.
 - **Admission ceiling** — once `CFDB_WORKFLOW_MAX_ACTIVE` workflows are active (`pending` + `running`), further preprocessing requests are shed with `429 Retry-After` rather than queued, so the backlog itself is bounded. The check runs before the per-file mutex, so at the ceiling even a re-`GET` for a file whose workflow is already in flight is shed with `429` (rather than attaching to the in-flight job) and the client retries — the deliberate trade for shedding before an unbounded admission race. The readiness `/status` probes never dispatch and so never `429`.
 
@@ -887,7 +887,7 @@ When the API runs on ECS Fargate (or LocalStack-backed dev that mirrors prod end
 - `ECS_WORKER_SECURITY_GROUPS` — comma-separated awsvpc security group IDs. Optional — when empty, ECS applies the VPC default SG.
 - `ECS_WORKER_ASSIGN_PUBLIC_IP` — `ENABLED` or `DISABLED` (default `DISABLED`). Production should leave this disabled and reach AWS via VPC endpoints; LocalStack accepts either value.
 
-The worker container's `CMD` is `python -m cfdb.workflows.worker_main`. Worker-side knobs (gRPC port, health port, max lifetime, drain grace) are documented under `--help` on that command; their env vars are `CFDB_WORKER_GRPC_PORT`, `CFDB_WORKER_HEALTH_PORT`, `CFDB_WORKER_MAX_LIFETIME_SECONDS`, and `CFDB_WORKER_DRAIN_GRACE_SECONDS`. The worker task definition MUST declare a `healthCheck` against the gRPC port; without one ECS reports `healthStatus: UNKNOWN` indefinitely and the worker is never advertised to discovery.
+The worker container's `CMD` is `python -m cfdb.workflows.worker_main`. Worker-side knobs (gRPC port, health port, idle timeout, idle poll cadence, max lifetime, self-termination grace, drain grace) are documented under `--help` on that command; their env vars are `CFDB_WORKER_GRPC_PORT`, `CFDB_WORKER_HEALTH_PORT`, `CFDB_WORKER_IDLE_TIMEOUT_SECONDS`, `CFDB_WORKER_IDLE_POLL_INTERVAL_SECONDS`, `CFDB_WORKER_IDLE_POLL_FAILURE_LIMIT`, `CFDB_WORKER_MAX_LIFETIME_SECONDS`, `CFDB_WORKER_MAX_LIFETIME_GRACE_SECONDS`, and `CFDB_WORKER_DRAIN_GRACE_SECONDS`. A worker self-terminates once continuously idle beyond `CFDB_WORKER_IDLE_TIMEOUT_SECONDS` (default 600 s; 0 disables) — measured via wool's `idle` RPC, which reports zero while any task runs, so the idle exit never fires while a task is running — with `CFDB_WORKER_MAX_LIFETIME_SECONDS` (default 12 h) retained as the backstop for a worker whose idle reporting is wedged or whose job is stuck. Both self-termination exits stop the worker with a drain grace (`CFDB_WORKER_MAX_LIFETIME_GRACE_SECONDS`, default 6 h): a dispatch accepted in the idle teardown window, or the job a max-lifetime expiry lands on, runs to completion instead of being cancelled into a terminal job failure, so worst-case worker uptime is lifetime + grace (18 h at the defaults) and the grace — sized above the API's 4 h `CFDB_WORKFLOW_DURATION_CAP_S` — only ever cancels work already past the API's own viability bound. Keep the idle timeout comfortably above `CFDB_WORKFLOW_RETRY_INTERVAL_S` (default 120 s) plus worker cold-start time (≥ 2–3×): wool's idle clock starts at worker startup, so an aggressive timeout lets overflow-spawned workers idle-reap before the scheduler's next retry tick ever reaches them, and jobs ride the queue to the `capacity:` deadline while everything looks green. The worker task definition MUST declare a `healthCheck` against the gRPC port; without one ECS reports `healthStatus: UNKNOWN` indefinitely and the worker is never advertised to discovery.
 
 **Workers publish their own metadata.** ECS reports a task's address and health, but not what is running inside the container — and two fields of wool's `WorkerMetadata` are knowable only to the worker: the wool protocol version it runs, and whether it configured TLS. wool gates worker admission on both, so a value the API invented for them would be a value that silently rejects the entire fleet. After starting, each worker therefore tags its own ECS task (`wool.version`, `wool.secure`) with what wool authored for it, and `EcsDiscovery` reads those tags back via `DescribeTasks … --include TAGS`. ECS supplies liveness; the worker supplies identity.
 
@@ -912,7 +912,7 @@ By default the API↔worker gRPC dispatch channel is plaintext, gated only by ne
 
 The configuration is gating-by-presence: when all three of `CFDB_WORKER_TLS_CA`, `CFDB_WORKER_TLS_CERT`, and `CFDB_WORKER_TLS_KEY` are unset the plaintext path is used unchanged (local PoC dev needs no certs); when all three are set mTLS is enforced. A *partial* configuration (some set, some not) fails fast at startup rather than silently degrading to plaintext.
 
-**Identity.** TLS normally verifies a server's certificate against the address the client dialed, which is a problem here: workers answer wherever they happen to come up. `EcsDiscovery` reaches each Fargate worker at the awsvpc IP assigned at launch, and a containerized local worker answers on a bridge IP — neither is knowable when the certificate is minted. `CFDB_WORKER_TLS_IDENTITY` (default `cfdb-worker`) points the API at a fixed logical name instead, so the worker leaf carries one stable SAN rather than an enumeration of every address it might be reached at. Chain and SAN verification both still happen; only the name being matched changes. It is a client-side setting — but "client" is a property of a connection, not of a process: the API is the client on the dispatch channel, and each worker is the client on the one channel wool opens back to its own subprocess to drain it, verifying the same SAN. Both sides therefore read the variable and MUST agree on it; a worker left at a different value drains by force-reap instead of gracefully, losing in-flight work with no TLS error anywhere. Setting it to the empty string restores address verification, which is only workable when workers are reached at a fixed, certified address.
+**Identity.** TLS normally verifies a server's certificate against the address the client dialed, which is a problem here: workers answer wherever they happen to come up. `EcsDiscovery` reaches each Fargate worker at the awsvpc IP assigned at launch, and a containerized local worker answers on a bridge IP — neither is knowable when the certificate is minted. `CFDB_WORKER_TLS_IDENTITY` (default `cfdb-worker`) points the API at a fixed logical name instead, so the worker leaf carries one stable SAN rather than an enumeration of every address it might be reached at. Chain and SAN verification both still happen; only the name being matched changes. It is a client-side setting — but "client" is a property of a connection, not of a process: the API is the client on the dispatch channel, and each worker is the client on the one channel wool opens back to its own subprocess to drain it, verifying the same SAN. Both sides therefore read the variable and MUST agree on it; a worker left at a different value drains by force-reap instead of gracefully, losing in-flight work with no TLS error anywhere. The idle-shutdown poll is a third consumer of the same contract: each worker dials its own subprocess at `127.0.0.1` to read wool's `idle` RPC, verifying the same identity SAN. Setting it to the empty string restores address verification, which is only workable when workers are reached at a fixed, certified address — and for the idle poll that means the worker leaf must carry a loopback SAN (the generator's default leaf includes `DNS:localhost` and `IP:127.0.0.1`), or every poll fails its handshake and idle shutdown silently degrades to the max-lifetime backstop with only repeated poll warnings as the symptom.
 
 Generate a local CA and the worker + API leaf certs with:
 

cloudformation/workers.yml

@@ -52,14 +52,42 @@ Parameters:
       Days after which cached artifacts expire. Cache keys are
       content-addressed by upstream md5, so expiry is safe — a missing
       artifact is simply re-materialized on the next request.
+  WorkerIdleTimeoutSeconds:
+    Type: String
+    Default: "600"
+    AllowedPattern: "^[0-9]+$"
+    Description: >
+      Continuous idle seconds beyond which a worker exits
+      (CFDB_WORKER_IDLE_TIMEOUT_SECONDS); 0 disables idle shutdown.
+      The primary reaper — a busy worker reports zero idle, so the
+      idle exit never fires mid-task, and a dispatch racing the
+      teardown drains under WorkerMaxLifetimeGraceSeconds. Keep this
+      comfortably above CFDB_WORKFLOW_RETRY_INTERVAL_S (default 120 s)
+      plus worker cold-start time (at least 2-3x), or overflow-spawned
+      workers idle-reap before the scheduler's next retry tick reaches
+      them and jobs ride the queue to the capacity deadline.
   WorkerMaxLifetimeSeconds:
     Type: String
-    Default: "18000"
+    Default: "43200"
     AllowedPattern: "^[0-9]+$"
     Description: >
       Hard ceiling on worker uptime (CFDB_WORKER_MAX_LIFETIME_SECONDS);
-      one hour above the 4 h workflow duration cap so a worker started
-      shortly before a long sort can still outlive the job.
+      the backstop behind the idle timeout for a worker whose idle
+      reporting is wedged or whose job is stuck. Expiry drains
+      in-flight work for up to WorkerMaxLifetimeGraceSeconds rather
+      than cancelling it, so worst-case worker uptime is this ceiling
+      plus that grace (18 h at the defaults).
+  WorkerMaxLifetimeGraceSeconds:
+    Type: String
+    Default: "21600"
+    AllowedPattern: "^[0-9]+$"
+    Description: >
+      Seconds a self-terminating exit — idle timeout or max lifetime —
+      drains in-flight work before cancelling it
+      (CFDB_WORKER_MAX_LIFETIME_GRACE_SECONDS). Instant when the worker
+      is genuinely idle; sized above the 4 h workflow duration cap so
+      the only work ever cancelled is work already past the API's own
+      viability bound.
   WorkerDrainGraceSeconds:
     Type: String
     Default: "120"
@@ -360,8 +388,12 @@ Resources:
               Value: "50051"
             - Name: CFDB_WORKER_HEALTH_PORT
               Value: "8080"
+            - Name: CFDB_WORKER_IDLE_TIMEOUT_SECONDS
+              Value: !Ref WorkerIdleTimeoutSeconds
             - Name: CFDB_WORKER_MAX_LIFETIME_SECONDS
               Value: !Ref WorkerMaxLifetimeSeconds
+            - Name: CFDB_WORKER_MAX_LIFETIME_GRACE_SECONDS
+              Value: !Ref WorkerMaxLifetimeGraceSeconds
             - Name: CFDB_WORKER_DRAIN_GRACE_SECONDS
               Value: !Ref WorkerDrainGraceSeconds
             # The worker tags its own task at startup (worker-metadata

pyproject.toml

@@ -22,7 +22,14 @@ dependencies = [
     "requests",
     "strawberry-graphql",
     "uvicorn",
-    "wool~=0.13.0",
+    # Compatible-release on 0.14 now that the final has shipped: patch
+    # releases are picked up, 0.15 is not, since a minor bump may move
+    # the wire protocol the dispatch channel depends on. Note that even
+    # a patch move is a fleet-rejection event — wool admits a worker
+    # only when the proxy's version is at most the worker's — so
+    # refreshing the lock deliberately, and drain the fleet when it
+    # moves (see README's version-bump deploy caveat).
+    "wool~=0.14.0",
 ]
 description = "A Python utility for parsing and normalizing various DCC datapackages."
 dynamic = ["version"]

src/cfdb/workflows/provisioner.py

@@ -467,10 +467,12 @@ async def _submit_run_task(self, kwargs: dict[str, Any]) -> dict[str, Any]:
         The future is recorded in ``_pending`` so ``aclose`` can drain
         threads mid-flight. A done-callback logs any ARNs the boto
         thread produced *after* the awaiting coroutine was cancelled —
-        these are orphan workers that no caller can claim, and their
-        only safety net is the worker's own ``CFDB_WORKER_MAX_LIFETIME``
-        ceiling. Surfacing them at WARNING gives operators a chance
-        to reap manually before the ceiling fires.
+        these are orphan workers that no caller can claim. Left unused,
+        they self-reap once continuously idle beyond
+        ``CFDB_WORKER_IDLE_TIMEOUT_SECONDS`` (minutes), with the
+        ``CFDB_WORKER_MAX_LIFETIME_SECONDS`` ceiling as the outer
+        bound. Surfacing them at WARNING still gives operators a
+        chance to reap manually first.
         """
         slot = _SubmittedRunTask()
         future = self._executor.submit(self._client.run_task, **kwargs)