Is your feature request related to a problem? Please describe.
BioProfileKit fetches reference data from external databases (NCBI Taxonomy, NCBI COG, Gene Ontology) at runtime via network requests. This creates three problems for reproducibility: the reference data can change between runs (e.g. taxonomy IDs are retired, GO terms are deprecated), analyses cannot be performed in air-gapped environments, and repeated runs incur unnecessary download overhead even when the underlying reference has not changed.
Describe the solution you'd like
Add a --dump-db CLI command that downloads and serialises all reference databases used by BioProfileKit to a local directory:
bioprofilekit --dump-db [--output-dir ~/.bioprofilekit/db]
→ ~/.bioprofilekit/db/
taxonomy_vocab.parquet ← already cached, extend versioning
taxonomy_raw.parquet ← already cached
cog_2024.parquet ← new: COG definition table
gene_ontology.obo ← new: GO OBO file with version header
manifest.json ← new: versions, download timestamps, checksums
The manifest.json should record for each database:
- Source URL
- Download timestamp
- File checksum (SHA-256)
- Database version or release identifier (e.g. GO release date from the OBO header, COG year from the filename, NCBI taxonomy dump date from
readme.txt)
When --dump-db has been run, subsequent analyses should prefer the local cache unconditionally rather than checking for staleness. Add a --db-dir option to point BioProfileKit at a non-default cache location, enabling shared database directories in HPC or container environments.
Additionally, add a bioprofilekit --db-info command that prints the current cache state — which databases are present, their versions, and their age — without running an analysis.
Describe alternatives you've considered
Pinning database versions via a configuration file without downloading them. This would require users to manually manage downloads and is error-prone. A self-contained dump command is more reproducible and easier to use in containerised deployments.
Additional context
The NCBI Taxonomy cache (taxonomy_vocab.parquet, taxonomy_raw.parquet) already exists in remote_data.py with a 30-day staleness check. This issue extends that mechanism to cover all reference databases uniformly and adds the version manifest that is currently missing. For the de.NBI deployment, --dump-db would be run once during container build to bake a fixed-version database snapshot into the image, making the web service fully reproducible and network-independent at runtime.
Is your feature request related to a problem? Please describe.
BioProfileKit fetches reference data from external databases (NCBI Taxonomy, NCBI COG, Gene Ontology) at runtime via network requests. This creates three problems for reproducibility: the reference data can change between runs (e.g. taxonomy IDs are retired, GO terms are deprecated), analyses cannot be performed in air-gapped environments, and repeated runs incur unnecessary download overhead even when the underlying reference has not changed.
Describe the solution you'd like
Add a --dump-db CLI command that downloads and serialises all reference databases used by BioProfileKit to a local directory:
The
manifest.jsonshould record for each database:readme.txt)When
--dump-dbhas been run, subsequent analyses should prefer the local cache unconditionally rather than checking for staleness. Add a--db-diroption to point BioProfileKit at a non-default cache location, enabling shared database directories in HPC or container environments.Additionally, add a
bioprofilekit --db-infocommand that prints the current cache state — which databases are present, their versions, and their age — without running an analysis.Describe alternatives you've considered
Pinning database versions via a configuration file without downloading them. This would require users to manually manage downloads and is error-prone. A self-contained dump command is more reproducible and easier to use in containerised deployments.
Additional context
The NCBI Taxonomy cache (
taxonomy_vocab.parquet,taxonomy_raw.parquet) already exists inremote_data.pywith a 30-day staleness check. This issue extends that mechanism to cover all reference databases uniformly and adds the version manifest that is currently missing. For the de.NBI deployment,--dump-dbwould be run once during container build to bake a fixed-version database snapshot into the image, making the web service fully reproducible and network-independent at runtime.