CI flake: Python suite fixture pool-timeouts in late test_start_config tests (~15 min into the run)

[hardbyte]

Recurring pattern, three sightings on 2026-07-11 alone (PR #409 run 29144621874; PR #410 runs at 57c7394-era and 64c58ce):

• The Python suite passes 307-308 tests, then 1-2 tests at the very end of the run (~15-17 minutes in, always in tests/test_start_config.py so far: test_default_start_on_fresh_install_auto_finalizes_to_queue_storage, test_tuple_form_backward_compat, test_partitioned_queue_configs_expand_and_dispatch) ERROR at fixture setup with _awa.DatabaseError: pool timed out while waiting for an open connection.
• Reruns pass. The affected branches touch no pool code.

The end-of-suite + fixture-setup signature suggests connection accumulation across the session (leaked pools from earlier tests exhausting the service container's max_connections, or client pools not fully closed — note the #408 class of runtime-teardown pool behavior in the bridge). test_start_config tests create fresh clients with their own pools, which makes them the first victims of an exhausted server.

Diagnosis ideas: log pg_stat_activity count from a fixture at suite start/end; or set a lower max_connections locally to reproduce faster; or audit which tests create AsyncClients without closing.

Not release-blocking (rerun-clearable), but it costs a rerun on most full-ci passes now — same operational-cost class as #399.

[hardbyte]

Root-caused and fixed on main in 00cb9f9 (#471): AsyncClient/Client default 10-connection pools plus sqlx's 10-minute idle timeout kept finished tests' connections parked until the server ran out of slots — the ~15-minute mark and the rerun-always-passes signature both fall out of that mechanism, and the victim was whichever fixture came next, not the test at fault. awa-python/tests/conftest.py now caps what tests ask for, the two return-style fixtures close deterministically, and a session-scoped guard fails the run with a named backend count if the pattern reappears. Verified against a deliberately constrained 30-connection server (pre-fix: 11–19 parked backends; post-fix: 3, all 312 pass). Closing as completed per the #383 policy — reopen on recurrence, which the guard will now attribute.

---

Generated by Claude Code

[hardbyte]

Reopening per the recurrence policy: the signature came back once on main's post-#480 push run (33431490075, py3.14 leg only) — test_start_config.py::test_retention_kwargs_accepted ERROR at fixture setup, pool timed out while waiting for an open connection, ~14.5 min in, 314 other tests and the whole py3.13 leg green. The #480 diff touches no Python or pool code.

The recurrence carries a detail that materially narrows the mechanism beyond what #471 fixed:

• The timeout hit the fixture's second transaction() call (inside reset_storage_transition_state), on a client whose pool had already served the first transaction and completed migrate() — setup spent 36.79s total, i.e. ~6s of real work then the full 30s acquire timeout.
• A pool that already owns a connection doesn't go to the server for its next acquire — an idle owned connection is reused immediately. So server-slot exhaustion (the test(ci): give nightly flake gates contention margin; add pg18 + py3.13/3.14 #471 mechanism) cannot produce this shape; the parked-pool fix stands, but there's a second, pool-internal mode: the client's own connections were all considered busy or unhealthy for 30+ seconds right after migrate() returned.
• That matches an earlier investigation on the pre-test(ci): give nightly flake gates contention margin; add pg18 + py3.13/3.14 #471 conftest: probing this exact fixture sequence showed it holds exactly 1 connection at every step, and a fully-passing suite peaks ~10 backends — exhaustion was never observable locally either.

Candidate directions for whoever picks this up: whether migrate() can leave the pool's connection checked out or mid-reconnect briefly (a race between its final commit and the pool returning the connection), and whether sqlx's acquire-with-health-check can spin past the 30s deadline on a contended runner. A cheap evidence step in the same spirit as #480's rehearsal diagnostics: catch PoolTimedOut in this fixture once, dump pg_stat_activity (server-side census) plus the pool's own size/idle counts, and re-raise — the next occurrence then distinguishes "server full", "pool starved", and "pool has idle but won't hand it over" in the job log.

Not treating this as a tag blocker per #383's policy (one recurrence, rerun-clears, root-cause direction recorded). I couldn't re-run the failed job (no actions: write from this session) — a manual re-run of the py3.14 leg on 33431490075 should clear main to green.

---

Generated by Claude Code