WIP: Add filesystem-backed local catalog mode

[Austin-s-h]

Summary

This PR adds a repo-local quilt3_local implementation for running Catalog in
LOCAL mode against a filesystem-backed data directory, and wires the existing
frontend to browse that local data without depending on remote Quilt services.

What changed

• added a repo-local LOCAL backend under api/python/quilt3_local
• added QUILT_LOCAL_OBJECT_BACKEND=filesystem support backed by
  QUILT_LOCAL_DATA_DIR
• added same-origin LOCAL routes for:
  • /__reg
  • /__lambda
  • /__s3proxy
• implemented the minimal registry / GraphQL / search / S3-compatible behavior
  needed for local Catalog flows:
  • landing page bucket listing
  • bucket overview and package browsing
  • package resolution under local/
  • workflow config/schema defaults for local buckets
  • preview and tabular-preview access through the local proxy
• updated frontend LOCAL behavior to:
  • use local proxy URLs instead of signed S3 URLs
  • pause unsupported subscription/admin queries in LOCAL mode
  • show bucket browsing from the landing page in LOCAL mode
• added curated local preview fixture staging, including dog_watermark.pdf
• generated a default quilt_summarize.json that expands staged preview/ files
  in grouped sections
• fixed spreadsheet date-only parsing drift for LibreOffice / CSV fixtures
• added LOCAL-mode regression coverage and updated docs/changelog
• added a one-shot local setup task via uvx --from poethepoet poe catalog-test

Validation

• cd api/python && uv run pytest tests/test_local_mode.py
• cd catalog && npm run test:only -- app/utils/spreadsheets/spreadsheets.spec.ts
• cd api/python && uvx --from poethepoet poe catalog-test

Notes for reviewers

• this is a read-oriented LOCAL development environment, not a full AWS emulator
• GraphQL/search support is intentionally narrow and only implemented deeply enough
  for current LOCAL Catalog flows
• the new setup/docs are in docs/Catalog/LocalMode.md

TODO

• Unit tests
• Security: Confirm that this change meets security best practices and does not violate the security model
• Open and Embed: Confirm that this change doesn't break Open variant and Embed widget
• Documentation
• SUMMARY.md so they appear on https://docs.quilt.bio)
  • Markdown docs for developers
• Changelog entry (skip if change is not significant to end users, e.g. docs only)

Greptile Summary

This PR introduces a filesystem-backed quilt3_local backend that lets the Catalog frontend run in LOCAL mode without any AWS dependency — serving bucket listings, package browsing, GraphQL, search, and file preview through a single FastAPI server. The implementation is well-scoped and the path-traversal guards in aws.py are sound.

• P1 — decode_cursor breaks the GraphQL error-union contract: base64.urlsafe_b64decode and json.loads are called without exception handling in search_more_packages/search_more_objects. A malformed after cursor raises a raw exception instead of returning the expected OperationError union member; fix by wrapping and returning {\"__typename\": \"OperationError\", ...} as already done elsewhere.
• P2 — get_default_origins() may include None: when WEB_ORIGIN is unset, the CORS origins list contains None, which is fragile even though today's origin in cors_origins guard is coincidentally safe.

Confidence Score: 4/5

Safe to merge after addressing the cursor exception handling in search.py — all other findings are non-blocking style/correctness suggestions.

One P1 finding: malformed searchMorePackages/searchMoreObjects cursors bypass the typed GraphQL error union and surface as raw ariadne execution errors. Remaining issues are P2 (None in CORS list, silent user_meta_filters, UTC vs local-time date formatting). Core filesystem path-traversal guards are solid, the async cache is correct, and the frontend LOCAL-mode adaptations are clean.

api/python/quilt3_local/search.py (decode_cursor exception handling), api/python/quilt3_local/lambdas/shared/utils.py (None in cors origins)

Important Files Changed

Filename	Overview
api/python/quilt3_local/search.py	In-memory search engine; decode_cursor lacks exception handling breaking the GraphQL error-union contract; user_meta_filters silently returns empty results
api/python/quilt3_local/aws.py	Filesystem-backed S3 abstraction layer; path-traversal guards are present; ETag reads entire file on every call (no caching for unchanged files)
api/python/quilt3_local/graphql.py	Ariadne GraphQL schema wiring; comprehensive but searchMorePackages/searchMoreObjects rely on decode_cursor which lacks error handling
api/python/quilt3_local/lambdas/shared/utils.py	get_default_origins() includes None in the CORS origins list when WEB_ORIGIN is unset
api/python/quilt3_local/s3proxy.py	S3-compatible proxy with host-style and path-style routing; CORS set to wildcard (*) which is appropriate for a local dev tool
catalog/app/utils/spreadsheets/spreadsheets.ts	Removes date-fns dependency for date formatting; toISOString().slice(0,10) uses UTC which may introduce off-by-one dates for non-UTC users depending on xlsx parser behaviour
api/python/quilt3_local/async_cache.py	Request-scoped async cache using Futures; correctly uses asyncio.get_running_loop() (not the deprecated get_event_loop())
api/python/quilt3_local/packages.py	Package manifest parsing and pointer resolution; well-structured with proper caching and namespace handling

Sequence Diagram

sequenceDiagram
    participant Browser
    participant FastAPI as FastAPI (main.py)
    participant API as /__reg (api.py)
    participant Lambda as /__lambda (lambdas/)
    participant S3Proxy as /__s3proxy (s3proxy.py)
    participant AWS as aws.py
    participant FS as Filesystem

    Browser->>FastAPI: GET /config.json
    FastAPI-->>Browser: {mode:LOCAL, registryUrl, s3Proxy, ...}

    Browser->>API: GET /__reg/api/auth/get_credentials
    API-->>Browser: fake credentials (filesystem mode)

    Browser->>API: POST /__reg/graphql (bucketConfigs)
    API->>AWS: list_bucket_configs()
    AWS->>FS: list data_dir/
    FS-->>AWS: bucket dirs
    AWS-->>API: bucket config list
    API-->>Browser: GraphQL response

    Browser->>Lambda: GET /__lambda/preview?url=...
    Lambda->>Lambda: validate url (local proxy or amazonaws.com)
    Lambda->>S3Proxy: HTTP GET (requests.get)
    S3Proxy->>AWS: fetch_object(Bucket, Key)
    AWS->>FS: read file bytes
    FS-->>AWS: bytes
    AWS-->>S3Proxy: {status, headers, body}
    S3Proxy-->>Lambda: response
    Lambda-->>Browser: preview HTML/JSON

    Browser->>S3Proxy: GET /__s3proxy/{host}/{key}
    S3Proxy->>AWS: fetch_object / list_objects_page
    AWS->>FS: read / rglob
    FS-->>AWS: data
    AWS-->>S3Proxy: result
    S3Proxy-->>Browser: S3-XML or file bytes

Loading

Comments Outside Diff (1)

1. api/python/quilt3_local/search.py, line 1181-1183 (link)

   decode_cursor exceptions bypass the GraphQL error-union contract

   base64.urlsafe_b64decode raises binascii.Error on bad padding, and json.loads raises JSONDecodeError on invalid JSON. Neither is caught, so a malformed after cursor propagates as an unhandled exception through the ariadne resolver — returning a raw execution error instead of the expected {"__typename": "OperationError", ...} union member that searchMorePackages / searchMoreObjects already return for the stale-result case. Clients that rely on the typed union to detect cursor errors will break.

   def decode_cursor(cursor: str) -> dict:
       try:
           return json.loads(base64.urlsafe_b64decode(cursor.encode() + b"==").decode())
       except Exception:
           raise ValueError("Invalid search cursor")

   Then in search_more_packages / search_more_objects, wrap the decode_cursor call:

   try:
       payload = decode_cursor(after)
   except ValueError:
       return {"__typename": "OperationError", "name": "CursorError", "message": "Search cursor is invalid.", "context": None}

_{Reviews (2): Last reviewed commit: "feat: curated preview fixtures and updat..." | Re-trigger Greptile}

[Austin-s-h]

Closing this WIP. The filesystem-backed LOCAL catalog approach is being set aside pending the Quilt team's model-driven local-dev direction (see discussion). The superseding non-local, independently useful wins that came out of this line of work have been raised as focused PRs (#5105–#5110).

Closing; the local-catalog design itself remains open for discussion on the packaging PR (#4933).