Add accession_id field to FileMetadataModel and Collection — Closes #36, #37, #38, & #103

.github/workflows/test.yml

@@ -35,3 +35,36 @@ jobs:
 
       - name: Run tests
         run: uv run pytest
+
+  # The materializer owns the `files` collection and its indexes, so its
+  # tests are the only guard on that index list. They are a separate job
+  # rather than a step above because they do not vary with the Python
+  # matrix -- running them per-version would compile the crate three times
+  # to assert the same thing.
+  materialize:
+    runs-on: ubuntu-24.04
+    steps:
+      - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1
+
+      # Pinned rather than inherited from the runner image: without this the
+      # compiler version drifts whenever the image is rebuilt, which is how a
+      # guard job starts failing on an unrelated PR and gets labelled flaky.
+      # SHA-pinned like every other action here, so the action's own code
+      # cannot change under us. The channel stays `stable` rather than an
+      # exact version: the crate declares edition 2021 and no rust-version,
+      # so there is no MSRV to hold it to, and pinning one would only add a
+      # bump to maintain.
+      - name: Install Rust toolchain
+        uses: dtolnay/rust-toolchain@4360b52568e2003a75bf9bc1d59f33a8e3fc893c # stable
+        with:
+          toolchain: stable
+
+      - name: Cache cargo build
+        uses: Swatinem/rust-cache@f0deed1e0edfc6a9be95417288c0e1099b1eeec3 # v2.7.7
+        with:
+          workspaces: materialize
+
+      # --locked so the committed Cargo.lock is authoritative; without it a
+      # semver-compatible upstream release can change what CI compiles.
+      - name: Run materializer tests
+        run: cargo test --locked --manifest-path materialize/Cargo.toml

.gitignore

@@ -77,7 +77,10 @@ uv.lock
 
 # Rust
 materialize/target/
-Cargo.lock
+# materialize/Cargo.lock is deliberately tracked: the materializer is a
+# binary crate and CI's `cargo test --locked` is the only guard on its
+# index list, so an untracked lockfile let an unrelated upstream release
+# turn that guard red.
 
 # Database
 database/

4DN-SUPPLEMENT.md

@@ -17,7 +17,16 @@ Field mapping from the 4D Nucleome (4DN) Search API and C2M2 datapackage to the
 | Collection | `persistent_id` contains `4DNEX*` or `4DNES*` | `accession` (e.g., `4DNEXH4ZUIH6`) |
 | Biosource tier | `extra.fourdn.biosource_name` | `Biosource.display_title` |
 
-Accessions are extracted from persistent_id URLs via regex: `4DNF[A-Z0-9]+` for files, `4DNE[A-Z][A-Z0-9]+` for experiments.
+Accessions are extracted from persistent_id URLs via regex: `4DNF[A-Z0-9]+` for files, `4DNE[A-Z][A-Z0-9]+` for experiments. Both are matched case-insensitively, and the extracted value is case-folded before use — an upper-case-only pattern does not merely miss a mixed-case accession, it matches the upper-case prefix and returns a truncated one.
+
+Both accessions are also persisted, so they can be queried directly rather than reconstructed:
+
+| CFDB Field | Source | Notes |
+|------------|--------|-------|
+| `accession_id` | `file.persistent_id` `4DNF*` | Stamped onto the raw `file` collection pre-materialization, case-folded |
+| `collections[].accession_id` | `collection.persistent_id` `4DNE*` | Stamped onto the raw `collection` collection pre-materialization, case-folded |
+
+Both are stamped *before* materialization deliberately. The materializer rebuilds `files` from the raw collections on every run, so a value written afterwards would be erased by any standalone `make materialize-dcc DCC=4dn` — silently, leaving accession lookups returning nothing for the whole DCC.
 
 ## Materialization (Rust)
 

ENCODE-SUPPLEMENT.md

@@ -83,6 +83,7 @@ ENCODE uses human-readable strings for file formats, assay types, output types,
 | CFDB Field | ENCODE TSV Column | Type | Notes |
 |------------|-------------------|------|-------|
 | `local_id` | `File accession` | string | ENCODE accession (e.g., `ENCFF001ABC`) |
+| `accession_id` | `File accession` | string | The same accession, case-folded to upper case. Duplicates `local_id` for ENCODE, which stores the accession there; the separate field exists for cross-DCC uniformity, since 4DN's `local_id` is an opaque UUID. Folded so an `accessionId` filter matches in any casing, which means it can legitimately differ from `local_id` in case. |
 | `id_namespace` | — | string | Constant: `https://www.encodeproject.org` |
 | `filename` | `File download URL` | string | Basename extracted from URL |
 | `access_url` | `File download URL` | string | Full HTTPS download URL |
@@ -124,14 +125,15 @@ One collection per unique experiment accession, embedded on `file.collections[]`
 | CFDB Field | ENCODE TSV Column | Notes |
 |------------|-------------------|-------|
 | `local_id` | `Experiment accession` | e.g., `"ENCSR000AAA"` |
+| `accession_id` | `Experiment accession` | The same accession, case-folded to upper case |
 | `name` | `Experiment accession` | Same as `local_id` |
 | `persistent_id` | `Experiment accession` | `https://www.encodeproject.org/experiments/{accession}/` |
 | `anatomy[]` | `Biosample term id` + `Biosample term name` | `{id, name}` object |
 | `biosamples[]` | — | Single biosample (see below) |
 | `subjects[]` | `Donor(s)` | Subject records (see below) |
 | `extra.encode` | — | Experiment-level metadata (see below) |
 
-**Fallback**: if `Experiment accession` is missing, falls back to biosample-keyed collection (`biosample:{name}`).
+**Fallback**: if `Experiment accession` is missing, falls back to biosample-keyed collection (`biosample:{name}`). That fallback collection is synthesized locally and names no ENCODE experiment, so it carries no `accession_id` rather than a fabricated one. Note also that the whole collection block is gated on `Biosample term name`: a row with an experiment accession but no biosample term produces no collection at all, so that experiment's accession is queryable nowhere.
 
 #### Collection Lab (top-level)
 

README.md

@@ -490,6 +490,7 @@ The central entity representing a stable digital asset.
 | `sha256` | string? | SHA-256 checksum (preferred) |
 | `md5` | string? | MD5 checksum (if SHA-256 unavailable) |
 | `filename` | string | Filename without path |
+| `accession_id` | string? | The DCC's own accession for this file, stored upper-cased so an `accessionId` filter matches in any casing. Populated for 4DN and ENCODE; always null for HuBMAP (see the note below). |
 | `file_format` | FileFormat? | EDAM CV term for digital format |
 | `compression_format` | string? | EDAM CV term ID for compression (e.g., `format:3989` for gzip); `""` when no compression is recorded or recognized; null/absent when undetermined. Read the note below before relying on it. |
 | `data_type` | DataType? | EDAM CV term for data type |
@@ -508,6 +509,8 @@ The central entity representing a stable digital asset.
 
 ENCODE derives the value from the download URL's filename suffix, because the ENCODE metadata TSV has no compression column. Two consequences are worth knowing. The field is **absent** (rendered as null) rather than `""` when nothing could be determined — no filename in the URL, or a compression suffix no EDAM term expresses (`.bz2`, `.xz`, `.zst`, `.zip`, `.starch`) — so treat null as "sniff the bytes", never as "uncompressed". And `format:3989` means "gzip-family stream": ENCODE names both plain gzip and BGZF `.gz` (it publishes no `.bgz` at all, and roughly a quarter of its `.gz` files are BGZF), so the value cannot distinguish them. Anything deciding on `gunzip | bgzip` must read the BGZF header — which is what `cfdb.workflows.processors.tabix` does, deliberately, and that byte-level check remains the decision of record.
 
+**A note on `accession_id`, because a null does not mean what it looks like.** The field exists so one input works across DCCs: 4DN puts an opaque UUID in `local_id` and carries its accession only inside the `persistent_id` URL, while ENCODE stores the accession *as* `local_id`. It is stored case-folded and filter values are folded identically, so `accessionId: ["4dnfimcjxzkh"]` and `["4DNFIMCJXZKH"]` match the same file. Three different situations all render as a null field and a `totalCount` of 0, and the API cannot distinguish them for you: the accession genuinely does not exist; the DCC issues none (all of HuBMAP, which matches files by filename within a dataset — tracked in [#102](https://github.com/abdenlab/cfdb/issues/102)); or that DCC has not been synced since the field was added. **A deployment must re-sync each DCC before `accessionId` returns anything.** Each sync logs its coverage (`4DN accession coverage: 53697/53697 files carry accession_id`), which is the only place that distinction is visible.
+
 #### Dcc
 
 A Common Fund program or Data Coordinating Center.
@@ -535,6 +538,7 @@ A grouping of files, biosamples, and/or subjects.
 | `name` | string | Human-readable label |
 | `description` | string? | Human-readable description |
 | `lab` | string? | Lab/PI name (shared across 4DN and ENCODE) |
+| `accession_id` | string? | The DCC's accession for the experiment this collection represents, stored upper-cased (shared across 4DN and ENCODE). Null on ENCODE's biosample-keyed fallback collections, which name no experiment, and on all of HuBMAP. |
 | `extra` | EnrichedCollection? | DCC-specific collection metadata (see EnrichedCollection) |
 
 #### Biosample