Bound concurrent cleanup of expired Firestore states

[groupthinking]

Speed up cleanup_old_states by deleting expired Firestore docs with up to 16 concurrent workers, logging failures and continuing the backlog.

[linear-code]

GRV-218

[github-actions]

👋 Thanks for opening this issue! It has been automatically labeled for triage.

[coderabbitai]

⚠️ Possible Duplicate Issue(s)

• perf: cleanup_old_states deletes expired Firestore documents sequentially #1169

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[groupthinking]

Closing as already implemented — verified against the code on main, not assumed.

What this issue asked for

Speed up cleanup_old_states by deleting expired Firestore docs with up to 16 concurrent workers, logging failures and continuing the backlog.

Three distinct requirements. All three are met.

Where it landed

FirestoreStateManager.cleanup_old_states() in src/youtube_extension/services/cloud/firestore_state.py:321-380:

Requirement	Implementation
up to 16 concurrent workers	CLEANUP_DELETE_CONCURRENCY = 16 (line 39), pool sized min(CLEANUP_DELETE_CONCURRENCY, len(docs)) (line 370)
logging failures	failures tallied into a list, then logger.warning("Failed to delete N of M old states; first error: ...") (line 377)
continuing the backlog	each worker catches per-delete exceptions and loops to the next document — one bad delete cannot abandon the rest (lines 361-365)

The implementation is a fixed worker pool pulling from a shared iterator, not gather() over every document. That distinction is deliberate and documented in-code: gather() would allocate one task per document up front, which is unsafe for a query whose result set has no limit. The pool bounds both in-flight RPCs and pending task objects.

Landed in 739a26d — perf: delete expired Firestore states concurrently under a bound (#1170), merged 2026-08-01.

Test evidence (run just now)

tests/unit/test_firestore_state.py — 90 passed, 0 failed, of which 32 are cleanup-specific, covering the bounded-concurrency ceiling, the failure-tally-and-continue path, and the env-var parsing of the concurrency limit.

Related

CLEANUP_DELETE_CONCURRENCY is being moved onto a shared, validated env-parsing helper in #1180 (PR #1220). That is a refactor of how the constant is parsed and preserves the behaviour described above exactly — the 90 tests above pass unchanged on that branch.

Why this was still open

Same pattern as #1191, #1192, #1198, #1199, #1201 and #1206 — auto-filed alongside its own fix PR and never closed when that PR merged. No product defect remains.