Stale cache lock from a killed process blocks CMS cache access until manual cleanup

[prenc]

Problem

_directory_lock (sources.py:381) takes a sibling <path>.lock directory via mkdir and removes it only in the holder's finally. A killed holder (job cancel, OOM, kill -9) never removes it, nothing reclaims a holder-less lock, so later callers time out after 30 s with a DownloadError implying a live holder, until manually deleted. A fixed 30 s wait also fails spuriously while a healthy first download still runs.

Example

A job on the README's shared cache is cancelled mid-download, leaving <cache>/catalog.json.lock; every later from_cms or refresh_cms_catalog call raises the timeout DownloadError until manually deleted.

Evidence

• sources.py:381-397: mkdir acquisition (:386), fixed 30 s deadline (:383/:389), timeout error (:390-392), removal only in the holder finally (:397).
• Call sites, all default 30 s: catalog (sources.py:179); material destination (sources.py:550, held across download + extract); per-URL artifact (sources.py:659); parsed cache (parsed_cache.py:132, held across the full parse).
• No PID/heartbeat/mtime/reclaim logic anywhere in src/; .tmp staging dirs are cleaned on the next run but the lock is not.
• Reproduced at the audit SHA: a stale holder-less catalog.json.lock made repeated paths("cm", "tabular") calls time out; the lock persisted.
• tests/test_sources.py has no leftover-lock test; no lock recovery is documented in the README or docs.

Required outcome

• A holder-less lock is detected and reclaimed (or identified as stale in the error), so a killed process cannot permanently block a cache destination.
• Waiters do not fail spuriously while a healthy holder still works (liveness-aware wait or an explicit, adequate timeout policy).
• A focused pytest test creates a stale lock via plain mkdir and asserts recovery or a stale-identifying failure; existing concurrency and corrupt-cache tests pass unmodified.

Scope boundaries

• No changes to cache layout, manifest schema, or _derived format; no change to in-process thread-level caching.