Cut run inode/disk footprint: build only the selected search DB, temp() blast intermediates - #129
Draft
conchoecia wants to merge 1 commit into
Draft
Cut run inode/disk footprint: build only the selected search DB, temp() blast intermediates#129conchoecia wants to merge 1 commit into
conchoecia wants to merge 1 commit into
Conversation
Two changes to scripts/odp that sharply cut the inode/disk footprint of a run without altering any final result: 1. Build only the database the chosen search method reads. make_diamond_and_blast_db always ran both `diamond makedb` and `makeblastdb`, so a diamond run (the default) emitted ~8 BLAST index files per genome that are never read (~46k inodes in a ~5.8k-genome run). Split into make_diamond_db and make_blast_db, and make diamond_blast depend (via an input function) only on the DB its method uses. Snakemake is pull-based, so the unneeded builder never runs. make_blast_db now declares all index files via multiext() so none leak untracked. 2. Mark true intermediates temp() so Snakemake removes them once their consumer finishes: diamond_blast .blastp (177 GB / ~35k files in the reference run), reciprocal_best_hits *_recip.temp.blastp (was misnamed "temp" but persisted), n_ways_reciprocal_best _acceptable_prots.txt/_edges.txt (dead byproducts), and get_chromsize_of_analysis_pair analyses/*.chromsize (a cheap per-pair cat). The step1-rbh/*.rbh and *.D.FET.rbh, filtered rbh, and figures remain untouched. Runs should pass `--rerun-triggers mtime` so that deploying this edit does not mark existing outputs out of date via the code/params provenance triggers.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A macrosynteny run accumulates a very large number of files. On a ~5,800-genome focal run the output tree reached ~354k inodes / ~300 GB. Two structural issues, neither intrinsic to the analysis:
diamond_or_blastp: "diamond".make_diamond_and_blast_dbran bothdiamond makedbandmakeblastdbunconditionally, so ~8 BLAST index files/genome (.pdb/.phr/.pin/.pjs/.pot/.psq/.ptf/.pto, ~46.5k inodes) were written and never read in diamond mode.diamond_blasteven listed.phr/.pin/.psqasancientinputs, forcing them into the DAG.temp().diamond_blast.blastp(~177 GB),reciprocal_best_hits*_recip.temp.blastp(named "temp" but persisted),n_ways_reciprocal_best_acceptable_prots.txt/_edges.txt(read by nothing downstream), andget_chromsize_of_analysis_pairanalyses/*.chromsize(a per-paircat) all lived forever.Changes (
scripts/odp)make_diamond_and_blast_dbintomake_diamond_dbandmake_blast_db;diamond_blastnow depends (via an input function) only on the DB its method reads. Because Snakemake is pull-based, the unneeded builder never runs — no BLAST index files in diamond mode, no.dmndin blastp mode.make_blast_dbdeclares every index file withmultiext()so none leak untracked.temp()on the true intermediates listed above. Finals —step1-rbh/*.rbhand*.D.FET.rbh,step1-rbh-filtered/*, and allstep2-figures/outputs (incl. the.plotted.rbhsidecars consumed downstream) — are untouched.Validation
tests/test_odp_basic: diamond mode →make_diamond_dbonly (nomake_blast_db), blastp mode →make_blast_dbonly.temp()removes the intermediate after its consumer runs.Rerun safety
Deploying a Snakefile edit would, under Snakemake's default
--rerun-triggers(which includecode/params), mark every output of the changed rules out of date and — withtemp()— recompute completed work. Runs must pass--rerun-triggers mtime(verified: with it, an edited rule + existing outputs reports "Nothing to be done"). Reused.pep.gz/DB inputs keep theirancient()markers so DB rebuilds don't cascade.Draft pending a full end-to-end run in a cluster module environment (DIAMOND/BLAST+/HMMER) confirming
db/has no BLAST index files andstep0-blastp_results/*.blastpis gone post-run while finals remain.