Problem
GET /api/v1/videos is declared async def but performs its entire filesystem workload synchronously on the event loop, so every request blocks the shared asyncio thread for the duration of the scan.
list_videos_v1 (src/youtube_extension/backend/api/v1/router.py:917-924) calls two blocking DataService methods directly:
async def list_videos_v1(limit: int = 50, offset: int = 0, ...):
total = data_service.count_videos() # blocking
...
paginated_videos = data_service.get_videos_summary(limit=limit, offset=offset) # blockingNeither call is awaited off-loop. The work they perform is not trivial:
count_videos() → _get_all_files_cached() (data_service.py:136-160) walks the entire enhanced-analysis tree with rglob("*_enhanced.md") and issues one stat() per file. This result is TTL-cached, so it is only paid on cache miss — but on miss it is an unbounded directory walk.get_videos_summary() (data_service.py:168-221) then runs "Pass 2" over the requested page and, for every item on the page, performs Path.glob(), Path.exists(), open() and json.load(). This per-page metadata read is never cached, so it is paid in full on every request.
With the default limit=50, a single request therefore performs on the order of 200 uncached blocking syscalls plus up to 50 JSON parses while holding the event loop. Every other in-flight request — health checks, SSE streams, unrelated endpoints — is stalled behind it.
Why this matters
This is the primary listing endpoint backing the dashboard, and the dashboard polls it. The stall scales linearly with page size and with the size of the analysis tree, so the failure mode worsens exactly as the deployment grows.
Expected behaviour
The blocking filesystem work should run in a worker thread via asyncio.to_thread, leaving the event loop free to service concurrent requests. Observable API behaviour (response shape, pagination semantics, the offset >= total short-circuit, error handling) must not change.
Suggested approach
Hoist both calls into a single synchronous helper and dispatch it with one asyncio.to_thread hop. A single hop — rather than two — preserves the existing read-consistency between count_videos() and get_videos_summary() and avoids a redundant second cache-refresh window.
Acceptance criteria
Problem
GET /api/v1/videosis declaredasync defbut performs its entire filesystem workload synchronously on the event loop, so every request blocks the shared asyncio thread for the duration of the scan.list_videos_v1(src/youtube_extension/backend/api/v1/router.py:917-924) calls two blockingDataServicemethods directly:Neither call is awaited off-loop. The work they perform is not trivial:
count_videos()→_get_all_files_cached()(data_service.py:136-160) walks the entire enhanced-analysis tree withrglob("*_enhanced.md")and issues onestat()per file. This result is TTL-cached, so it is only paid on cache miss — but on miss it is an unbounded directory walk.get_videos_summary()(data_service.py:168-221) then runs "Pass 2" over the requested page and, for every item on the page, performsPath.glob(),Path.exists(),open()andjson.load(). This per-page metadata read is never cached, so it is paid in full on every request.With the default
limit=50, a single request therefore performs on the order of 200 uncached blocking syscalls plus up to 50 JSON parses while holding the event loop. Every other in-flight request — health checks, SSE streams, unrelated endpoints — is stalled behind it.Why this matters
This is the primary listing endpoint backing the dashboard, and the dashboard polls it. The stall scales linearly with page size and with the size of the analysis tree, so the failure mode worsens exactly as the deployment grows.
Expected behaviour
The blocking filesystem work should run in a worker thread via
asyncio.to_thread, leaving the event loop free to service concurrent requests. Observable API behaviour (response shape, pagination semantics, theoffset >= totalshort-circuit, error handling) must not change.Suggested approach
Hoist both calls into a single synchronous helper and dispatch it with one
asyncio.to_threadhop. A single hop — rather than two — preserves the existing read-consistency betweencount_videos()andget_videos_summary()and avoids a redundant second cache-refresh window.Acceptance criteria
list_videos_v1performs no blocking filesystem I/O on the event loop.offset >= totalearly return are unchanged.