Fold a processor identity into the workflow cache key — Closes #109

ENCODE-SUPPLEMENT.md

@@ -61,7 +61,7 @@ ENCODE uses human-readable strings for file formats, assay types, output types,
 
 This applies to `.bedpe` already in the ENCODE experiment corpus, not only to annotation files — but it reaches no further than ENCODE. `FILE_FORMAT_TO_EDAM` and `get_file_format` have exactly one consumer, the ENCODE transform; 4DN and HuBMAP take `file_format` from their upstream C2M2 datapackage or portal API and never consult the table. A 4DN `.bedpe` whose upstream declares BED therefore still carries `file_format.name == "BED"`, is still claimed by `TabixIntervalProcessor`, and still gets an index built from its first mate. Closing that means routing incoming formats from every DCC through the same table, or refusing a `.bedpe` filename at the processor regardless of declared format; both are follow-up work.
 
-**Cache artifacts left behind.** Files of these two formats that were indexed before the re-typing still have their incorrect `.tbi` artifacts in the workflow cache. Nothing reads them any more — `lookup_for` returns `None` and the router bails before probing the cache — and nothing purges them either, so they are unreachable storage cost until someone sweeps them. They also make a latent cache-key hazard concrete: `cache_key` identifies a processor only by its `processor_version`, not by which processor it is, so a future paired-interval processor sharing a version number with `TabixIntervalProcessor` would derive the same key and read those stale artifacts back as cache hits. A processor identity has to be folded into the key before that processor lands; see the warning on `cfdb.workflows.keys.cache_key`.
+**Cache artifacts left behind.** Files of these two formats that were indexed before the re-typing still have their incorrect `.tbi` artifacts in the workflow cache. Nothing reads them — `lookup_for` returns `None` and the router bails before probing the cache. The hazard they posed is closed: `cache_key` used to identify a processor only by its `processor_version`, so a future paired-interval processor sharing a version number with `TabixIntervalProcessor` would have derived the same key and read those stale artifacts back as cache hits. Issue #109 folded the producing processor's identity into the key, so that collision is now impossible rather than merely unlikely, and the stranded artifacts — which are keyed under the retired scheme — are swept by `cfdb purge-legacy-cache`.
 
 ### Output Type -> EDAM Data
 

README.md

@@ -877,7 +877,19 @@ The preprocessed artifact is the default response. Clients that want the raw ups
 | GTF | GTF→GFF3 + sort + bgzip + tabix | bgzipped GFF3 + TBI |
 | bigBed | bigBedToBed + sort + bgzip + tabix | bgzipped BED + TBI |
 
-Cache keys are content-addressed using each file's upstream `md5`, so a byte change upstream (with the sync pipeline refreshing `md5`) invalidates the cache automatically.
+Cache keys have the shape `{dcc}/{local_id}/{artifact_kind}/{processor_id}/{md5}-v{processor_version}` — for example `encode/ENCFF732YBO/index/tabix-interval/6fccbb438a046075cb438f84d0defe8d-v2`. They are content-addressed using each file's upstream `md5`, so a byte change upstream (with the sync pipeline refreshing `md5`) invalidates the cache automatically, and they carry the producing processor's identity (`Processor.processor_id`) so two processors claiming the same file and artifact kind can never read back each other's output. Without that segment a version number was the only thing separating processors — and `TabixIntervalProcessor` and `BamIndexProcessor` both sit at version 2, staying apart only because their `supported_formats` happen to be disjoint. A processor that declares no `processor_id` takes its own class name; declare one explicitly when the identity should survive a class rename, since changing the string invalidates every artifact keyed under it. The declaration must sit in the processor's own class body. Omitting it entirely — or writing `processor_id = ""` — takes the class-name default, which is safe because every class name is distinct. Supplying one from a base class or mixin instead raises at class definition: the value would be discarded in favour of the class name, so a mixin's pinned identity would read as authoritative in source while silently cold-caching everything keyed under it. Subclassing another processor stays legal, and each level of the chain takes its own class name.
+
+Whatever the identity ends up being — declared or defaulted — it must match `[A-Za-z0-9._-]`, and may not be `.`, `..`, or an artifact kind (`data`, `index`). Each is rejected when the class is declared, so a malformed identity fails on import rather than per-request inside a worker. The alphabet is an allowlist because a denylist of the separators still admits a newline, a zero-width space, or a fullwidth solidus — each invisible in review while addressing a different cache entry. The artifact-kind rule is what keeps a mis-specified purge prefix from reducing a live key to something shaped like a retired one. Note that case is preserved in the key but *not* relied on to keep two identities apart on disk: a case-insensitive filesystem folds `BedProcessor` and `BEDProcessor` onto one directory, so `ProcessorRegistry.register` refuses the pair.
+
+**Deploying the key change.** The cache key is derived independently in two separately-deployed processes: the API derives it to probe the cache, and the worker derives it to write the artifact. `processor_id` travels as a class attribute, so each resolves it from *its own* image. **Deploy the worker task definition at or before the API.** Workers are standalone `RunTask` tasks rather than an ECS service, so a stack update does not stop the ones already running — they exit only on `CFDB_WORKER_MAX_LIFETIME_SECONDS` (default 5 hours), and worker discovery gates on the wool protocol version, not the cfdb image. A new API dispatching to an old worker gets a job that completes having written a key the API will never probe, so the next request misses and dispatches again, indefinitely and silently. The same applies in reverse to an old API and a new worker.
+
+The deploy also starts against a **fully cold cache**: every `/data` and `/index` request for a processable file dispatches a fresh workflow until the cache refills. Re-keying the existing artifacts in place (`CopyObject` on S3, `rename` locally) was considered and rejected — it needs each artifact's producing processor resolved from file metadata, which is more moving parts than a cold start costs.
+
+**Rolling back past this change.** Reverting puts the system through a *second* cold start, since the API returns to deriving four-segment keys against a cache from which exactly those were swept. Every five-segment key written between deploy and revert becomes unreachable, and `cfdb purge-legacy-cache` will not clear it — the sweep only recognises the retired four-segment shape. On S3 the bucket lifecycle rule reclaims that residue within `CacheArtifactExpirationDays`; on a local cache root nothing does.
+
+**Purging the retired key scheme.** Keys minted before the processor-identity segment existed are unreachable by construction — the API derives the current shape and never probes the old one. Sweep them with `cfdb purge-legacy-cache` (dry run by default; pass `--apply` to delete). The same sweep clears the orphaned `.bedpe` / `bigInteract` index artifacts stranded when those formats were re-typed.
+
+Two things to know before running it against S3. First, the sweep needs `s3:ListBucket` on the bucket and `s3:DeleteObject` on `<bucket>/*` — **neither the API nor the worker task role carries a delete grant**, deliberately, so run it under an operator or CI principal rather than from inside a deployed container. Second, the cache bucket already carries an `expire-cached-artifacts` lifecycle rule (`CacheArtifactExpirationDays`, default 30), and its rationale is exactly this case: keys are content-addressed, so a missing artifact is simply re-materialized. The retired population therefore self-reclaims within a month on its own. Sweeping S3 *accelerates* that reclaim; it is not required for correctness, and waiting is a legitimate choice. A local cache root has no equivalent expiry, so there the sweep is the only thing that reclaims. Run the local sweep with the API stopped — it prunes directories as it empties them, which can race a concurrent cache write.
 
 **Bounded concurrency, durable queuing, and admission control.** Dispatch is bounded on three cooperating layers so an unauthenticated burst on `/data` and `/index` can't oversubscribe the worker fleet or queue unbounded work:
 
@@ -998,7 +1010,7 @@ Poll the status of a dispatched workflow:
   "job_id": "abc-123",
   "status": "running",
   "stages_done": ["data"],
-  "artifacts": {"data": "encode/ENCFF123/data/abc-v1"},
+  "artifacts": {"data": "encode/ENCFF123/data/tabix-interval/6fccbb438a046075cb438f84d0defe8d-v2"},
   "progress": null,
   "error": null,
   "superseded_by": null
@@ -1068,6 +1080,8 @@ Check the status of a sync task. The `task_id` is returned when starting a sync.
 
 ### CLI
 
+#### `cfdb sync`
+
 ```bash
 # Sync all DCCs
 cfdb sync
@@ -1080,3 +1094,29 @@ cfdb sync 4dn hubmap
 - `--api-url` - cfdb API base URL (default: `http://localhost:8000`, env: `CFDB_API_URL`)
 - `--api-key` - API key for sync endpoint (env: `SYNC_API_KEY`)
 - `--debug` / `-d` - Enable debugpy debugging
+
+#### `cfdb purge-legacy-cache`
+
+```bash
+# Report what the retired cache-key scheme is still holding (dry run)
+cfdb purge-legacy-cache
+
+# Delete it — local cache root, then an S3-backed environment
+cfdb purge-legacy-cache --local-root ./data/cache --apply
+cfdb purge-legacy-cache --s3-bucket cfdb-cache --s3-prefix dev --apply
+```
+
+**Options:**
+- `--s3-bucket` - bucket holding the workflow cache (env: `WORKFLOW_S3_BUCKET`)
+- `--s3-prefix` - key prefix the S3 cache backend writes under (env: `WORKFLOW_S3_PREFIX`)
+- `--endpoint-url` - boto3 endpoint override for LocalStack-backed dev (env: `AWS_ENDPOINT_URL`)
+- `--region` - AWS region for the boto3 client (env: `AWS_REGION`)
+- `--local-root` - local cache root, which must exist (default: `$SYNC_DATA_DIR/cache`)
+- `--apply` - actually delete; without it the sweep only reports
+- `--yes` - skip the `--apply` confirmation prompt, for scripted runs
+
+Exactly one store is purged per run. When both an S3 bucket and a local root resolve — including the environment-only pairing of `WORKFLOW_S3_BUCKET` and `SYNC_DATA_DIR`, which a deployed API task has — the command refuses rather than guessing which cache you meant. The target is printed before the sweep begins, and `--apply` prompts for confirmation, because the store is chosen partly from ambient environment and the deletion cannot be undone.
+
+Pass `--s3-prefix` exactly as `WORKFLOW_S3_PREFIX` is set for the environment. A prefix carrying extra segments is rejected by the artifact-kind check rather than acted on, and one that is too short matches nothing — the command warns when it scanned entries and matched none, so a prefix typo does not read as an already-swept environment.
+
+`--s3-prefix` must be exactly the prefix the cache backend writes under (`WORKFLOW_S3_PREFIX`). A prefix carrying extra segments is stripped from every key before the retired-shape test, so an over-specified one would otherwise reduce live five-segment keys to four-segment ones; the sweep rejects those because the processor identity lands where an artifact kind must be, but the safest habit is still to pass the same value the API runs with. A prefix that is too short simply matches nothing.

src/cfdb/cli.py

@@ -1,4 +1,6 @@
 import logging
+import os
+from pathlib import Path
 
 import click
 import requests
@@ -110,5 +112,165 @@ def sync(dcc_names: tuple[str, ...], api_url: str, api_key: str):
         raise SystemExit(1)
 
 
+@cli.command("purge-legacy-cache")
+@click.option(
+    "--s3-bucket",
+    default=None,
+    envvar="WORKFLOW_S3_BUCKET",
+    show_envvar=True,
+    help="Bucket holding the workflow cache (S3 profile).",
+)
+@click.option(
+    "--s3-prefix",
+    default="",
+    envvar="WORKFLOW_S3_PREFIX",
+    show_envvar=True,
+    help="Key prefix the S3 cache backend writes under.",
+)
+@click.option(
+    "--endpoint-url",
+    default=None,
+    envvar="AWS_ENDPOINT_URL",
+    show_envvar=True,
+    help="boto3 endpoint override (LocalStack-backed dev).",
+)
+@click.option(
+    "--region",
+    default=None,
+    envvar="AWS_REGION",
+    show_envvar=True,
+    help="AWS region for the boto3 client.",
+)
+@click.option(
+    "--local-root",
+    default=None,
+    help="Local cache root. Defaults to $SYNC_DATA_DIR/cache.",
+    type=click.Path(file_okay=False, exists=True, path_type=Path),
+)
+@click.option(
+    "--apply",
+    default=False,
+    help="Delete the matched entries. Without it the sweep is a dry run.",
+    is_flag=True,
+)
+@click.option(
+    "--yes",
+    default=False,
+    help="Skip the --apply confirmation prompt (for scripted runs).",
+    is_flag=True,
+)
+def purge_legacy_cache(
+    s3_bucket: str | None,
+    s3_prefix: str,
+    endpoint_url: str | None,
+    region: str | None,
+    local_root: Path | None,
+    apply: bool,
+    yes: bool,
+):
+    """
+    Sweep workflow cache entries minted under the retired key scheme.
+
+    Issue #109 folded a processor identity into the cache key, so every
+    entry written before it is unreachable: the API derives the new key
+    shape and never probes the old one. This command deletes those
+    entries, including the orphaned .bedpe / bigInteract index artifacts
+    left behind when PR #108 re-typed those formats.
+
+    WARNING: --apply deletes objects irreversibly. Runs as a DRY RUN
+    unless it is passed.
+
+    Exactly one store is swept per run. Resolving both an S3 bucket and a
+    local cache root is a usage error rather than a precedence rule --
+    purging the wrong store cannot be undone, so the command refuses to
+    guess. $SYNC_DATA_DIR is consulted for the local root only when no
+    bucket is configured.
+
+    Examples:
+
+        cfdb purge-legacy-cache
+
+        cfdb purge-legacy-cache --local-root ./data/cache --apply
+
+        cfdb purge-legacy-cache --s3-bucket cfdb-cache --s3-prefix dev --apply
+    """
+    from cfdb.workflows.purge import build_s3_client, purge_local, purge_s3
+
+    sync_data_dir = os.getenv("SYNC_DATA_DIR")
+    env_local_root = Path(sync_data_dir) / "cache" if sync_data_dir else None
+    if local_root is None:
+        local_root = env_local_root
+
+    if s3_bucket and local_root is not None:
+        # Both stores resolved — refuse rather than guess. This fires on the
+        # environment-only pairing too: a container that sets both
+        # WORKFLOW_S3_BUCKET and SYNC_DATA_DIR (backend.yml does) would
+        # otherwise silently sweep S3 for an operator who meant the local
+        # cache, and purging the wrong store is not recoverable.
+        source = "--local-root" if env_local_root != local_root else "$SYNC_DATA_DIR"
+        raise click.UsageError(
+            f"Both an S3 bucket and a local cache root ({source}) resolved; "
+            f"pass only one (unset WORKFLOW_S3_BUCKET to target the local root)"
+        )
+    if not s3_bucket and local_root is None:
+        raise click.UsageError(
+            "No cache to purge: pass --s3-bucket or --local-root, or set "
+            "WORKFLOW_S3_BUCKET / SYNC_DATA_DIR"
+        )
+
+    target = (
+        f"s3://{s3_bucket}/{s3_prefix.strip('/')}".rstrip("/")
+        if s3_bucket
+        else str(local_root)
+    )
+
+    # Name the target BEFORE sweeping. The store is chosen partly from
+    # ambient environment, so an operator must be able to see which one was
+    # picked while the run is still stoppable — not after the deletes.
+    click.echo(f"Target: {target}")
+    if apply and not yes:
+        click.confirm(
+            f"Irreversibly delete legacy cache entries from {target}?",
+            abort=True,
+        )
+
+    if s3_bucket:
+        report = purge_s3(
+            build_s3_client(endpoint_url=endpoint_url, region_name=region),
+            s3_bucket,
+            prefix=s3_prefix,
+            apply=apply,
+        )
+    else:
+        report = purge_local(local_root, apply=apply)
+
+    click.echo(f"Scanned: {report.scanned}")
+    click.echo(
+        f"Legacy entries: {report.matched} "
+        f"({report.bytes_matched:,} bytes, {report.bytes_matched / 1024**3:.2f} GiB)"
+    )
+    if apply:
+        click.echo(f"Deleted: {report.deleted}")
+    else:
+        click.echo("Dry run — nothing deleted. Re-run with --apply.")
+
+    # A clean sweep and a mis-targeted one both report zero. Distinguish
+    # them, so an operator working through the migration runbook cannot tick
+    # an environment off on the strength of a prefix typo.
+    if report.scanned == 0:
+        click.echo(
+            f"WARNING: {target} held nothing — check the bucket, prefix, or "
+            f"path before treating this environment as swept.",
+            err=True,
+        )
+    elif report.matched == 0:
+        click.echo(
+            f"WARNING: scanned {report.scanned} entries and matched none. If "
+            f"this environment was not already swept, check --s3-prefix "
+            f"against the deployment's WORKFLOW_S3_PREFIX.",
+            err=True,
+        )
+
+
 if __name__ == "__main__":
     cli()