Ingest ENCODE annotation files during sync — Closes #94 - #108
Merged
Conversation
Both formats pair two loci per record, and both were aliased onto the EDAM term for the single-interval format they resemble -- bedpe to BED, bigInteract to bigBed. Processor lookup keys on the format name, so that alias handed them to the BED tabix pipeline, which sorted and indexed the first locus of each record and left the second unindexed. The result was a cached artifact that looked successful and was wrong, and the byte-sniff guard cannot catch it because both are plaintext or gzip. EDAM has no term for either, verified against OLS4, so the tokens are minted under a cfdb: prefix rather than borrowed. The distinct name is what does the work: neither appears in any processor's supported_formats, so a data request now streams the raw upstream file and an index request fails cleanly, instead of serving something silently incorrect. This reaches the .bedpe already published under type=Experiment, not only the annotation files that prompted it -- and no further. The format table has exactly one consumer, the ENCODE transform. 4DN and HuBMAP take file_format from their upstream C2M2 datapackage or portal API and never consult it, so a 4DN .bedpe still arrives declaring BED and is still claimed by the tabix pipeline. Routing every DCC through the same table, or refusing the filename at the processor regardless of declared format, is follow-up work. The two formats do not lose the same thing. A bedpe index was wrong: the second mate of every record was unindexed and invisible. A bigInteract index was range-coherent, since the leading three columns are a genuine interval, so it answered range queries correctly while degrading each interaction to a single locus. An index request now returns 404 for either, which means bigInteract goes from degraded-but-usable to unavailable, for files already in the experiment corpus as well as new ones. A 404 states the truth where the degraded index quietly answered a question the caller did not ask. Tileset endpoints that understand paired intervals are planned separately. The published contract is updated to match. FileFormat and the README type table both described the id as an EDAM term, which is what a consumer would resolve it as and now fail on; both name the minted prefix as the discriminator to test against instead.
These five values are the complete Output type domain of the two annotation types the sync ingests, verified against the live metadata TSVs, and none of them appeared in the table. Without them every one of the ~29,000 annotation files would be stored with a null data_type. The element and gene link types resolve to the generic data term rather than to a features or track term. EDAM has nothing for a predicted regulatory relationship between two loci, and the pre-existing chromatin interactions entry already made that call the same way. A deliberately vague term is preferable to claiming the files are something they are not.
Eight fields the annotation metadata TSV publishes and the experiment one does not: annotation_type and organism on the file, annotation_type, software_used and encyclopedia_version on the dataset, and life_stage, age and age_units on the biosample. annotation_type is held on the file as well as the dataset. The dataset is where the property belongs, but filtering files by it is the actual use case, and routing that through a collection subdocument on every query is not worth avoiding one duplicated string. The age fields stay strings rather than becoming a number. The released corpus contains "2-4" and "unknown" alongside decimals, and it distinguishes "10.5" from "10.50". That is also why they do not go to Subject.age_at_sampling, which is a float in years and could represent neither the ranges nor the sentinels -- and an annotation row names no donor to build a Subject from in any case. The output types are derived from the models, so the SDL gains all eight on both the input and the output side.
Everything ENCODE publishes as an Annotation -- candidate cis-regulatory elements, element gene regulatory interaction predictions, chromatin state, footprints -- was absent from cfdb, which fetched only type=Experiment. Annotation files are the interpretive layer over the raw experiments: a cCRE track is what a Gosling visualization usually wants alongside signal. The streaming loop is extracted so both TSVs share it, and each stream now names itself in its log lines. It takes a deadline rather than resolving its own budget, so several streams in one sync share a single allowance; a stream that finds the budget already spent refuses to open a request rather than passing the remainder through, since aiohttp reads a non-positive total as no timeout and would turn the exhausted case into an unbounded request. Which annotation types are ingested is read from ENCODE_ANNOTATION_TYPES rather than hardcoded: the full space is 580,910 datasets, 86% of it footprints, so the allowlist has to exist from the outset rather than be retrofitted once volume becomes a problem. Entries are trimmed and deduplicated, since files are written with insert_many into a collection with no unique key and a repeated token would load a whole type twice. One request per type, so a failure or a gateway timeout on one costs only that type. An explicitly empty value disables annotation ingest, and now says so. The neighbouring ENCODE_METADATA_TIMEOUT_SECONDS reads an empty value the other way, as unset, and empty values arrive by accident routinely -- an unset CloudFormation parameter, a docker-compose expansion that produced nothing. Turning the path off stays a legitimate operator choice, but without the warning the only trace of an accidental one is an absence in the per-phase log. The two TSVs do not share a column set -- Annotation publishes 32 columns to Experiment's 59, six of them renamed. Rather than a second transformation, the annotation columns are renamed to their experiment equivalents and run through the shared one, so a single mapping serves both and a column the TSV omits comes out unset instead of derived from something else. That is how an annotation document ends up with no subjects: there is no Donor(s) column to build one from, and inventing a donor would be worse than admitting there is none. Two deliberate departures from the experiment path. The dataset collection is built from the accession alone rather than requiring a biosample term, because 48 released cCRE files name no biosample and gating on one would leave 24 dataset accessions unqueryable. And the collection's persistent_id points at /annotations/, since ENCODE serves the two dataset kinds under different paths. Also populates extra.encode.assembly, which the schema has always published and nothing ever wrote -- a client filtering on it matched nothing at all. Multi-assembly cCREs, spanning GRCh38, mm10 and hg19, are what make that worth fixing now.
The sync now runs one phase for released experiments plus one per configured annotation type. A phase that raises is logged and recorded and the remaining phases still run, so a transient failure on one stream does not cost the corpus the others. The task still fails at the end naming the phases that broke -- reporting a clean sync over a partially loaded collection would be worse than a visible failure. Getting that right needed more than a try/except around each phase. A stream can die mid-flight, which is exactly what an asyncio.TimeoutError against the metadata budget does, and reporting a phase's contribution only on its clean return meant a failed phase contributed nothing to the total even though its completed batches were already committed. The tallies, mutated per row, kept counting the rows the partial batch then discarded. Three numbers in one log block disagreeing with each other and with the database. The row count is now reported by mutating an accumulator, and a batch left buffered by a dead stream is flushed on the way out, so a partial load reports itself accurately. The clean-path flush is deliberately not in that finally. A finally runs whether or not an exception is in flight, so suppressing the flush's failure there suppressed it on the success path too: a stream that drained cleanly and then failed to commit its trailing rows returned normally, and the sync reported success over a corpus short by up to a batch. Each insert also detaches its buffer before awaiting, because the driver stamps _id in place before sending, so a resubmitted batch collides on the prefix that already landed and silently drops the rest. Cancellation skips the flush entirely: the decision to stop has been made, and a CancelledError redelivered by that await would replace the original and skip the phase's tally. Isolation covers the destructive half too. Each phase owns a slice of files -- experiments are the documents carrying no annotation type, each annotation phase the documents carrying its own -- and deletes that slice only once it has replacement rows in hand. Clearing the whole DCC before the fan-out meant a failed experiment phase left the API serving the 29,000 annotation documents as the entire corpus. A stream that drains cleanly with no rows still clears, so a type ENCODE stops publishing stops being served. The download budget covers the whole sync rather than each stream in it. Every phase runs inside the one cutover lock that gates the read surface, so a per-stream budget multiplied the outage by the phase count and carried the worst case past the sync lock's one-hour stale threshold, at which point a second sync is admitted and clears the corpus while the first is still writing into it. The budget is resolved once and each phase gets what remains of it. Isolating phases also means abandoning a generator mid-iteration on every failure. The stream owns an aiohttp session inside its own context manager, released only when the generator is finalized, so the streams are closed explicitly rather than left to the garbage collector -- with one stream per sync that was academic, with N+1 it is not. The annotation tally is seeded from the configured allowlist so a type ENCODE stops publishing reports zero rather than vanishing from the log. An absent key is what nobody notices; a zero is what everybody does. The broad except stays narrower than BaseException on purpose: a CancelledError caught there would be logged as a phase failure and followed by every remaining phase, which is the opposite of cancelling.
annotation_type is the field the whole annotation corpus is meant to be reached through, organism narrows a result set spanning human and mouse, and assembly narrows one spanning GRCh38, mm10 and hg19. All three needed two things the schema alone does not give them. An index, because the ENCODE sync writes the files collection directly and never reaches the Rust materializer that owns the rest of its keys -- so nothing else would create these, and the flagship filter would be a full scan of a ~300,000 document collection on an unauthenticated endpoint. The core genome_assembly field is indexed here for the same reason, despite the materializer owning it. A database where ENCODE is the only DCC synced never runs the materializer, and genome_assembly is the field the schema publishes and a client narrowing to GRCh38 actually filters on -- indexing only the extra.encode mirror would have left the documented path scanning. Repeating a key the materializer also creates costs nothing, because identical keys derive identical default names. A place in the distinct-values allowlist, because filtering by annotation type is only useful if the vocabulary is discoverable. Otherwise a client has to already know ENCODE's exact spelling, which is the string matching the annotation ingest set out to replace. All three are small closed vocabularies, which is what that allowlist is for -- unlike the accessions, which are excluded from it deliberately because enumerating one would return the whole corpus.
The module docstring is the reference for this service and described only half of it once the annotation path landed. It now carries the second metadata URL, the six renamed columns, the seven annotation-only ones, and -- as importantly -- the list of columns the annotation TSV does not publish, which are left unset rather than derived. The supplement gains the same mapping, the allowlist and its configured types with their real dataset and file counts, the two minted format terms and why they are minted, and the annotation output types. Three properties of the fan-out are recorded because a reader would otherwise have to reconstruct each of them from the code. The download budget bounds the whole sync rather than each stream in it, and the reason is the cutover lock the phases run inside and the sync lock's one-hour stale threshold beyond it. Each phase replaces only the slice it reloads, and only once it has rows to put back, so a phase that dies before delivering any leaves its previous rows being served. And the paired-interval fix reaches ENCODE alone, since the format table has one consumer and the other two DCCs never touch it. What the mint costs is recorded alongside what it buys: the incorrect tabix artifacts left orphaned in the cache for the re-typed files, and that bigInteract goes from a degraded index to none at all until the tileset endpoints land. Two details are called out in both the docstring and the supplement. An annotation document carries no subjects, since the TSV names no donor to build one from, and its dataset is built from the accession alone so that the 24 datasets whose files name no biosample stay queryable. The donor traits are the one mapping the tables overstated. Life stage, age and age units land on the biosample, so a row naming no biosample term drops them -- which is exactly the row the annotation transform relaxed its biosample requirement to serve. Nothing was lost against the corpus as last measured, and inventing a biosample to hold them would be worse than dropping them, but the mapping was written as unconditional and it is not.
The minted prefix is the mechanism that stops an unrepresented format being aliased onto a term meaning something else, and nothing tied the exported constant to the table it governs. Three property tests do that, each asserting through get_file_format rather than against the table directly so the property covers what production reads: every id is either an EDAM term or a minted one, a minted id is derived from its own key, and a minted name is shared with no EDAM entry and claimed by no processor. That last one is the load-bearing case -- since routing keys on the name, a minted id paired with the name "BED" would drop the file straight back into the pipeline the minting exists to keep it out of. Each property's example budget is derived from the table it samples rather than left at the default, so a closed domain is exhausted exactly and grows with the table. The routing assertion now goes through a registry wired the way the API wires it, rather than two hand-picked classes, so it tracks what is actually served. A positive control pins that plain BED still reaches the tabix processor, since "no processor claims it" would also hold if lookup were simply broken. The annotation output types are asserted against their exact terms. The previous test only checked for a non-null data term, so all five could have pointed at the wrong one and passed.
The fixture carries the full 32-column annotation header with values from a real released cCRE file, so the tests exercise the shapes the corpus actually contains rather than an idealized row: an empty Assay term name, which every one of the 12,448 cCRE rows has, a multi-valued Targets, and a non-human organism. On the fetch side: the URL is asserted by parsing its query rather than by substring, which would pass against a stray or duplicated parameter, and a type containing URL metacharacters is pinned as unable to smuggle one in. Each failure mode -- HTTP status, timeout, network error -- is checked to name its own stream, which is the entire reason the label was threaded through, and to release its session. That last one caught a real leak: async for does not close the iterator it abandons, so the delegating wrapper left the session-owning inner generator suspended. The shared deadline gets both of its cases. A stream given one bounds its request by the time remaining rather than by a fresh full budget, and a stream given a spent one refuses without opening a request at all -- aiohttp reads a non-positive total as no timeout, so passing the remainder through would make the exhausted case unbounded. Allowlist parsing gets its adversarial cases: all-blank values disable ingest rather than widening it to an unfiltered query, repeats collapse, and a property test pins that no configuration at all can produce a blank entry. An explicitly empty value is pinned to warn, with a negative control that the unset path stays silent, since the warning is what separates a deliberate disable from a templating accident. The default-when-unset test now deletes the variable first -- it previously asserted against whatever the ambient environment held and passed with an override exported. On the transform side: the renames, the annotation-only fields, the absence of the experiment-only ones, and the two behaviors that differ by design -- no subjects without a donor column, and a dataset built from the accession alone. The experiment path gets the pins it was missing: its persistent_id, which was asserted nowhere in the repository despite the URL path becoming a caller-supplied argument, and anatomy, which the diff's guard change touched and no test read. The assembly mirror is asserted once per transform rather than through one parametrized case. The two read different columns, File assembly against Assembly, which a shared parameter hid behind an argument name. One document is validated through FileMetadataModel end to end. The ingest writes plain dicts and the resolver reads them back through the model on every row, so a key the model does not declare is dropped silently -- the filter would exist and match nothing.
The central pin is that the reported total always equals the documents actually committed. It is asserted for a stream that dies mid-flight -- the shape a metadata timeout takes -- and then generalized by a property test over arbitrary fan-out, row counts, batch sizes and failure patterns, since the example cases only fix a handful of shapes. Those assertions are anchored to the whole progress line rather than matched as substrings: "6 files" is contained in "16 files", so containment passes on an order-of-magnitude error in the one number this design exists to get right. The failure cases now exercise the sink as well as the stream. Every earlier one failed the metadata fetch, so nothing covered a stream that drains cleanly and then cannot commit its last batch, which returned normally and reported a clean sync over a short corpus. A batch the sink rejects is pinned as never resubmitted, and a cancellation with rows still buffered as never flushed. Around those: the partial batch of a failed phase is committed rather than discarded, a repeated allowlist entry loads its type once, a configured type that returns nothing is reported as zero, a cancellation propagates instead of being swallowed as a phase failure, and the error chains the first failure so the traceback points at what started the cascade. Every phase is pinned to receive one shared deadline, which no test counting rows could otherwise observe. The clearing tests follow the slice. A phase clears only what it reloads, so the total-failure path leaves last sync's corpus intact rather than emptying it, and a phase whose stream drains cleanly with no rows empties its own slice rather than serving withdrawn documents indefinitely. Batching gets its boundaries -- exactly one batch, one plus a remainder, and none -- plus the skip path for a row that does not transform. The fake collection's $exists now resolves dotted paths. It tested for a top-level key, so a filter on extra.encode.annotation_type reported every document as lacking it and matched them all; the per-slice clearing tests would have passed against a clear that deleted the corpus. The fixture that disables annotation phases moves from module scope to the classes that drive the sync. It governs a network boundary -- without it a unit test reaches the real portal once per configured type -- so it belongs next to the tests that depend on it rather than silently over the 4DN and HuBMAP ones. A new test opts out of it deliberately, because nothing otherwise joined the default allowlist to the phases that run.
A filter is only useful if the path it flattens to is the path the ingest writes. A mismatch there would leave the whole annotation corpus unfilterable while every ingest test still passed, so each of the eight new fields is asserted from the GraphQL input through to its dotted predicate -- including the five-segment biosample path, which crosses two array levels and is the one most easily got wrong. A property test pins that none of them is folded the way accessions are. ENCODE's vocabulary is case- and space-significant, so folding any of them would make the documents permanently unmatchable with nothing raising. The end-to-end tests then ask the questions a client would: give me the cCRE files, narrow them to mouse, narrow them to mm10, and read the fields back out. The distinct-values tests pin that the annotation types and the assemblies can both be enumerated, since filtering by either is only useful if the vocabulary is discoverable. The index spec tests split by concern -- the accession paths, the annotation facets, the core assembly field -- so each claim stays readable now that the list covers more than accessions. A separate assertion pins the full set by equality, because containment alone lets an unintended spec merge unnoticed into a list that runs createIndex against a public collection on every sync.
The point of giving bedpe and bigInteract their own format terms is what the routes do with them, and that was asserted only at the mapping table. Both routes are now pinned against a registry wired the way the API wires it: a data request reaches the direct streaming path without dispatching a workflow, and an index request returns 404 without consulting the executor at all. Serving the raw file is the correct outcome here rather than a degradation. The alternative these tests rule out is the previous behavior, where the pipeline claimed the file and committed an index built from the first locus of each record. Both fixtures inject the minted term rather than producing it, and each carries a comment saying so. The format table has one consumer, the ENCODE transform, so the 4DN ingest cannot mint it and a real 4DN .bedpe still arrives declaring BED. The assertions hold as registry-routing pins, since lookup keys on the format name irrespective of DCC, but read as evidence that 4DN paired-interval files are safe they would be wrong.
A DCC that raised aborted every DCC after it. get_all_dcc_names sorts, so one failed ENCODE phase cost the whole HuBMAP sync, and the data-collection indexes built after the loop were skipped along with it. The annotation fan-out did not introduce the cascade but made it likelier: one network stream became three by default, against a portal that returns 504 on requests it cannot assemble in time. Each DCC is now isolated the way _sync_encode already isolates its phases. A failure is logged and recorded, the remaining DCCs still run, the indexes are ensured regardless, and the run fails once at the end naming every DCC that broke, with the first failure chained as the cause. The per-DCC success log and the accession coverage report move inside the success path, so a DCC that failed no longer reports itself as synced. The except stays narrower than BaseException, so a CancelledError cancels the run rather than being recorded as one DCC's failure and followed by every remaining DCC.
cache_key derives dcc/local_id/artifact_kind/md5-vN. The processor's identity is absent -- only its version number is present -- so two processors claiming the same file and artifact kind at equal processor_version derive the same key and read back each other's artifacts as cache hits. That is a wrong answer rather than a miss. Nothing collides today because each pair is claimed by at most one processor, which is a property of the current registry rather than of this function. Re-typing bedpe and bigInteract makes the case concrete: those files still carry index artifacts built by TabixIntervalProcessor before the re-typing, so a paired-interval processor is precisely what would collide with them. A processor identity has to be folded into the key before that processor lands.
conradbzura
force-pushed
the
94-ingest-encode-annotation-files
branch
from
August 16, 2026 18:44
6ad8d07 to
4597276
Compare
The contract suite already claimed to cover this. Its name said ``$exists`` resolves dotted paths and its docstring said the doc should be returned, but the body only checked that the call did not raise, with a comment conceding the matcher "may or may not return the doc" -- so it passed against a matcher that resolved nothing. Both assertions now hold the implementation to what the name claims, and both fail against the previous one. The negated case is the one with consequences. Matching by top-level key reports every dotted path as absent, so a filter on ``extra.encode.annotation_type`` not existing matched the whole collection -- and that predicate is how the ENCODE ingest selects the experiment slice it clears before reloading. A delete built on it would have taken the corpus rather than the slice, with the per-phase tests passing throughout.
conradbzura
marked this pull request as ready for review
August 18, 2026 15:04
This was referenced Aug 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add a second ingest path for
type=Annotation&status=released, so the interpretive layer over the raw experiments cfdb already serves becomes reachable. A cCRE track is what a Gosling visualization usually wants alongside signal, and none of it was queryable before.Which annotation types are ingested is configuration rather than a hardcoded assumption. ENCODE publishes 580,910 annotation datasets and 86% of them are footprints, so the allowlist exists from the outset rather than being retrofitted once volume becomes a problem. The default is the two types the issue names: 7,773 datasets, 29,140 files.
The two TSVs do not share a column set — Annotation publishes 32 columns to Experiment's 59, six of them renamed. Rather than a parallel transformation, the annotation columns are renamed to their experiment equivalents and run through the shared one, so a single mapping serves both and a column the TSV omits comes out unset instead of derived from something else. That is how an annotation document ends up with no subjects: there is no
Donor(s)column to build one from, and inventing a donor would be worse than admitting there is none.Going from one metadata stream per sync to N+1 is the part that needed the most care. The sync runs one phase per stream, and a phase failure costs only that stream: the phases share a single download budget rather than each claiming a fresh one, each phase replaces only the slice of
filesit reloads and only once it has rows to put back, and the run reports exactly what it committed on every path including the timeout. The task still fails at the end naming what broke.Closes #94
Validation
Verified against the live corpus rather than only against fixtures. Three full syncs through a dockerised Mongo and API loaded 839,121 files — 809,981 experiments, 12,448 cCRE, 16,692 interaction predictions — with the reported total, the per-phase log and the database agreeing exactly every time, and the annotation counts matching the figures in
ENCODE-SUPPLEMENT.mdto the file.A second sync over the populated corpus produced the same 839,121 documents with zero duplicate
(submission, local_id)pairs, each phase clearing exactly its own slice. A deliberately exhausted budget then confirmed the failure path: the experiment phase consumed it and both annotation phases were refused without opening a request, the run reported 1,629 files against exactly 1,629 in the database, and both annotation slices survived intact at 12,448 and 16,692 because a phase that never receives a row never clears.Proposed changes
Ingest released annotations per configured type
Extract the streaming loop so both TSVs share it, and give each stream a label it carries into its log lines — with several streams per sync, a timeout that does not say which one died is not actionable.
ENCODE_ANNOTATION_TYPESbounds the allowlist; entries are trimmed and deduplicated, since files are written withinsert_manyinto a collection with no unique key and a repeated token would load a whole type twice. One request per type, so a failure or a gateway timeout on one costs only that type.An explicitly empty value disables annotation ingest and logs a warning saying so. The neighbouring
ENCODE_METADATA_TIMEOUT_SECONDSreads an empty value the other way, as unset, and empty values arrive by accident routinely — an unset CloudFormation parameter, adocker-composeexpansion that produced nothing. Turning the path off stays a legitimate operator choice; without the warning the only trace of an accidental one is an absence in the per-phase log.Two departures from the experiment path are deliberate. The dataset collection is built from the accession alone rather than requiring a biosample term, because 48 released cCRE files name no biosample and gating on one would leave 24 dataset accessions unqueryable. And its
persistent_idpoints at/annotations/, since ENCODE serves the two dataset kinds under different paths.Run the ingest as isolated per-stream phases
A phase that raises is logged and recorded and the remaining phases still run. The task still fails at the end naming the phases that broke — reporting a clean sync over a partially loaded collection would be worse than a visible failure.
Accounting holds on every path. A phase reports the rows it committed rather than only those from a clean return, since a stream can die mid-flight and a return value is lost when it does. The trailing batch of a dead stream is flushed rather than discarded, but the flush on the clean path sits inside the
tryrather than thefinally: afinallyruns whether or not an exception is in flight, so suppressing its failure there suppressed it on the success path too, and a stream that drained cleanly and then failed to commit its last rows reported a clean sync over a short corpus. Each insert detaches its buffer before awaiting, because the driver stamps_idin place before sending and a resubmitted batch collides on the prefix that already landed while silently dropping the rest. Cancellation skips the flush entirely rather than writing on its way out.The download budget covers the whole sync rather than each stream in it. Every phase runs inside the one cutover lock that gates the read surface, so a per-stream budget multiplied the outage by the phase count and carried the worst case past the sync lock's one-hour stale threshold — at which point a second sync is admitted and clears the corpus while the first is still writing into it. The budget is resolved once and each phase gets what remains; a phase that finds it spent refuses to open a request rather than passing the remainder to aiohttp, which reads a non-positive total as no timeout at all.
Isolation covers the destructive half too.
filesis no longer cleared corpus-wide before the fan-out. Each phase owns a slice — experiments are the documents carrying no annotation type, each annotation phase the documents carrying its own — and deletes that slice only once it has replacement rows in hand, so a phase that fails before delivering any leaves its previous rows being served. Clearing up front meant a failed experiment phase left the API serving the 29,140 annotation documents as the entire corpus. A stream that drains cleanly with no rows still clears, so a type ENCODE stops publishing stops being served.Isolate per-DCC failures in the sync run
The same accounting one level out. A DCC that raised aborted every DCC after it, and
get_all_dcc_namessorts, so one failed ENCODE phase cost the whole HuBMAP sync and the data-collection indexes built after the loop. Each DCC is now isolated the way the phases are: the failure is recorded, the remaining DCCs run, the indexes are ensured regardless, and the run fails once at the end naming every DCC that broke. A DCC that failed no longer logs itself as synced.Give bedpe and bigInteract their own file_format terms
Both formats pair two loci per record and both were aliased onto the EDAM term for the single-interval format they resemble. Since processor lookup keys on the format name, that alias handed them to the BED tabix pipeline, which indexed the first locus of each record and left the second unindexed — a cached artifact that looked successful and was wrong, and one the byte-sniff guard cannot catch because both are plaintext or gzip.
EDAM has no term for either, verified against OLS4, so the tokens are minted under a
cfdb:prefix rather than borrowed. Neither name appears in any processor'ssupported_formats, so a data request now streams the raw upstream file and an index request returns 404.This reaches the
.bedpealready published undertype=Experiment, not only the annotation files that prompted it — and no further.FILE_FORMAT_TO_EDAMhas exactly one consumer, the ENCODE transform; 4DN and HuBMAP takefile_formatfrom their upstream C2M2 datapackage or portal API and never consult it, so a 4DN.bedpestill arrives declaring BED and is still claimed by the tabix pipeline. Closing that is follow-up work.FileFormatand the README type table both described the id as an EDAM term, which is what a consumer resolves it as and would now fail on for the 8,296 files carrying a minted one. Both now nameMINTED_FORMAT_PREFIXas the discriminator to test against.Make the annotation fields queryable
Eight new fields across the three enriched ENCODE models, mirrored onto the GraphQL inputs.
annotation_typeis held on the file as well as the dataset: the dataset is where the property belongs, but filtering files by it is the actual use case.The age fields stay strings. The released corpus contains
2-4andunknownalongside decimals and distinguishes10.5from10.50, which is also why they do not go toSubject.age_at_sampling— a float in years could represent neither, and an annotation row names no donor to build aSubjectfrom in any case. They land on the biosample, so a row naming no biosample term drops them; that is the row the annotation transform relaxed its biosample requirement to serve, and the mapping tables now say so rather than stating it unconditionally.extra.encode.annotation_type,extra.encode.organismandextra.encode.assemblyeach gain an index and a place in the distinct-values allowlist. The ENCODE sync writesfilesdirectly and never reaches the materializer that owns the rest of its keys, so nothing else would create them, and the flagship filter would otherwise be a full scan of a ~300,000 document collection on an unauthenticated endpoint. The coregenome_assemblyfield is indexed here for the same reason despite the materializer owning it: a database where ENCODE is the only DCC synced never runs the materializer, andgenomeAssemblyis the field the schema publishes and a client narrowing to GRCh38 actually filters on, so indexing only theextra.encodemirror would have left the documented path scanning. The allowlist entries matter because filtering by a facet is only useful if the vocabulary is discoverable.Behavior changes for existing clients
.bedpeandbigInteractfiles already in the experiment corpus changefile_format.namefromBED/bigBedtobedpe/bigInteract, so a saved query filteringfileFormat: {name: ["BED"]}stops matching them. Measured against the live corpus: 1,892.bedpeand 252bigInteractfiles predate this change.GET /index/...returns 404 for both formats. For.bedpethat replaces a wrong answer, since the index covered only the first mate of each record. ForbigInteractit replaces a working one: the leading three columns are a genuine interval, so the oldbigBedToBedindex answered range queries correctly while degrading each interaction to a single locus. Those 252 pre-existing files go from degraded-but-usable to unavailable until the tileset endpoints land. A 404 states the truth where the degraded index quietly answered a question the caller did not ask.file_format.idis no longer always an EDAMformat:term. 8,296 files carry acfdb:-prefixed token that resolves nowhere at OLS or edamontology. A client resolving ids against EDAM must skip that prefix rather than assume every id is resolvable.extra.encode.assemblyis populated for the first time. It has always been published in the SDL and never written, so a client filtering on it previously matched nothing at all.ENCODE_METADATA_TIMEOUT_SECONDSchanges meaning from a per-stream budget to a whole-sync one. An operator who raised it to cover a slow experiment fetch now gets that value as the ceiling for all phases combined rather than for each.Follow-ups
Not addressed here, and worth an owner before or shortly after merge:
bigInteractfiles to a usable state.FILE_FORMAT_TO_EDAM, or refuse a.bedpefilename at the processor regardless of declared format, so the correction reaches 4DN and HuBMAP.cache_key. It derivesdcc/local_id/artifact_kind/md5-vNwith no processor identity, so two processors claiming the same file and artifact kind at equalprocessor_versionread back each other's artifacts as cache hits. Nothing collides today only because each pair has one processor; a paired-interval processor is exactly what would break that, and the stale tabix artifacts for these files are still in the cache.Test cases
tests/test_ontology_mappings.pyget_file_formatis called with its keytests/test_ontology_mappings.pyget_file_formatis called with its keytests/test_ontology_mappings.pyget_file_formatis called with its keytests/test_ontology_mappings.pybedpeandbigInteracttests/test_ontology_mappings.pytests/test_ontology_mappings.pyget_data_typeis calleddata_typecoveragetests/test_encode.pyENCODE_ANNOTATION_TYPESin the environmentannotation_types_from_envis calledtests/test_encode.pyannotation_types_from_envis calledtests/test_encode.pyannotation_types_from_envis calledtests/test_encode.pyannotation_types_from_envis calledtests/test_encode.pyannotation_types_from_envis calledtests/test_encode.pyannotation_types_from_envis calledtests/test_encode.pytests/test_encode.pytests/test_encode.pytests/test_encode.pyTimeoutErrorwithout opening a requesttests/test_encode.pyannotation[<type>]tests/test_encode.pytests/test_encode.pytests/test_encode.pytransform_annotation_to_c2m2is calledtests/test_encode.pytransform_annotation_to_c2m2is calledannotation_typeis set on both the file and its datasettests/test_encode.pytransform_annotation_to_c2m2is calledtests/test_encode.pytransform_annotation_to_c2m2is calledtests/test_encode.pytransform_annotation_to_c2m2is calledpersistent_idpoints at/annotations/tests/test_encode.pytransform_annotation_to_c2m2is calledtests/test_encode.pytransform_annotation_to_c2m2is calledtests/test_encode.pytransform_annotation_to_c2m2is calledtests/test_encode.pyAssay term nameis emptytransform_annotation_to_c2m2is calledexperiment_typeand the file noassay_typetests/test_encode.pytransform_to_c2m2is calledextra.encode.assemblymirrors the top-levelgenome_assemblytests/test_encode.pytransform_annotation_to_c2m2is calledextra.encode.assemblymirrors the top-levelgenome_assemblytests/test_encode.pyFileMetadataModeltests/test_encode.pytransform_to_c2m2is calledpersistent_idpoints at/experiments/tests/test_encode.pytransform_to_c2m2is calledtests/test_encode.pytransform_to_c2m2is calledtests/test_sync.py_sync_encoderunstests/test_sync.py_sync_encoderunstests/test_sync.py_sync_encoderunstests/test_sync.py_sync_encoderunstests/test_sync.py_sync_encoderunstests/test_sync.pyCancelledError_sync_encoderunstests/test_sync.py_sync_encoderunstests/test_sync.py_sync_encoderunstests/test_sync.py_sync_encoderunstests/test_sync.py_sync_encoderunstests/test_sync.py_sync_encoderunstests/test_sync.py_sync_encoderunstests/test_sync.py_sync_encoderunstests/test_sync.pyCancelledError_sync_encoderunstests/test_sync.py_sync_encoderunstests/test_sync.py_sync_encoderunstests/test_sync.py_sync_encoderunstests/test_sync.py_sync_encoderunstests/test_sync.py_sync_encoderunstests/test_sync.py_sync_encoderunstests/test_sync.pyENCODE_ANNOTATION_TYPESin the environment_sync_encoderunstests/test_sync.py_sync_dccsrunstests/test_sync.py_sync_dccsrunstests/test_sync.py_sync_dccsrunstests/test_sync.py_sync_dccsrunstests/test_sync.pyCancelledError_sync_dccsrunstests/test_inputs.pyto_dictthento_querytests/test_inputs.pyto_querybuilds the predicatetests/test_schema.pytests/test_schema.pytests/test_schema.pytests/test_schema.pytests/test_schema.pydistinctValuesis asked for the annotation typetests/test_schema.pydistinctValuesis asked for the assemblytests/test_index.pystream_index_fileis calledtests/test_data.pystream_fileis calledtests/test_indexes.pytests/test_indexes.pytests/test_indexes.pygenome_assemblypathtests/test_fake_collection.pyfind_oneasserts$existson that dotted pathtests/test_fake_collection.pyfind_onenegates$existson that dotted path