Skip to content

perf: cleanup_old_states deletes expired Firestore documents sequentially #1169

Description

@groupthinking

Problem

FirestoreStateService.cleanup_old_states() in
src/youtube_extension/services/cloud/firestore_state.py deletes every expired
document one at a time:

# Delete in batch          <- the comment is not true
count = 0
for doc in docs:
    await doc.reference.delete()
    count += 1

Each delete() is an independent network round-trip to Firestore. The loop
awaits each one before starting the next, so cleanup costs N sequential
round-trips
and its wall-clock time scales linearly with the size of the
expired backlog. The # Delete in batch comment claims batching that the code
does not do.

There is a second, subtler problem: if any single delete raises, the exception
propagates out of the loop. Every remaining document is skipped, and the count
of documents that were already deleted is lost with the exception.

Reachability

This is live code, not a dead path — the module has 4 importers:

src/youtube_extension/services/cloud/cloud_video_processor.py
tests/unit/test_firestore_state.py
tests/unit/test_cloud_video_processor.py
src/youtube_extension/services/cloud/__init__.py

Acceptance criteria

  • Expired-state deletes are issued concurrently rather than one-at-a-time.
  • Concurrency is bounded so a large backlog cannot fan out unbounded
    in-flight RPCs against Firestore.
  • A single failing delete does not abandon deletes already in flight.
  • The returned count reflects deletes that actually succeeded.
  • Regression tests are proven to fail against the current sequential
    implementation (non-vacuity).

Scope

  • In scope: the delete fan-out inside cleanup_old_states(), plus tests.
  • Out of scope: the query/where clause, caching behaviour, the module
    singleton lifecycle, and any change to cleanup_old_states's signature.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions