Skip allocating unused per-chunk window buffers in CovFastReader (fixes multi-hundred-GB vmem overrun in augment_coverage_by_labels) - #67
Open
tafujino wants to merge 1 commit into
Conversation
CovFastReader_construct passed chunkLen as windowLen to ChunksCreator_constructFromCov for lack of a real window size of its own. That caused every chunk to carry 3 windowLen-sized int arrays (160MB each at the 40e6 chunkLen augment_coverage_by_labels uses) that CovFastReader never reads or writes -- it walks blocks directly via TrackReader/ptBlock, not through Chunk's window arrays. At full-genome scale (507 chunks) this allocated on the order of 240GB that nothing ever touched, hitting RLIMIT_AS/s_vmem on hosts that enforce a virtual-memory ceiling even though actual resident memory stayed under 10GB. Add ChunksCreator_constructFromCovWithOptions with a constructChunksWithAllocatedSeq flag; CovFastReader now passes false to skip building chunksCreator->chunks entirely (left NULL, which ChunksCreator_destruct already handles). hmm_flagger and coverage_format_converter, which genuinely use those buffers with sane window sizes, keep calling the original ChunksCreator_constructFromCov unchanged. Verified: full genome at --threads 8 went from needing ~235GB (matching the ~240GB estimate) to succeeding at 8GB; a 9M-line input that failed even at 64GB now succeeds at 8GB.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
augment_coverage_by_labelscan request on the order of 200+ GB of virtualmemory on full human genome-scale input, even though its actual resident
memory usage is under 10 GB. On any environment that enforces a virtual
address space ceiling (
ulimit -v/RLIMIT_AS, common on HPC schedulerssuch as SGE/UGE), this makes the process fail — killed on a vmem limit if one
is set close to real usage, or crashing partway through chunk construction if
a
mallocsilently returnsNULLand its result is used unchecked (severalmalloccalls in this path aren't NULL-checked). I believe this is the sameunderlying issue reported in #44 ("Segmentation fault ... during chunk
parsing", failing right at "Creating empty chunks" — exactly the code path
described below).
Root cause
CovFastReader_construct(cov_fast_reader.c) has no window-size concept ofits own, so it reuses
chunkLenas thewindowLenargument toChunksCreator_constructFromCov:ChunksCreator_constructFromCov→Chunk_constructListWithAllocatedSeq→Chunk_constructWithAllocatedSeqthen allocates, for every chunk:augment_coverage_by_labelsuseschunkLen = 40,000,000, sowindowLenhereis also 40,000,000 — three 160 MB arrays per chunk, for a tool that never
reads or writes them (confirmed by grepping the whole
programs/tree:CovFastReaderwalks blocks directly viaTrackReader/ptBlock, neverthrough
Chunk's window arrays). Over 507 chunks on a full genome that's507 × 480 MB ≈ 243 GB of virtual memory reserved for buffers nothing ever
touches — closely matching the
maxvmem(~234–235 GB) I measured on the runthat reproduced this.
These three arrays are genuinely used elsewhere (
hmm_flagger,coverage_format_converter) with real, much smaller window sizes — so thefix is not to shrink or remove the arrays unconditionally (I tried that
first; it's unsafe, see "Alternatives considered" below), but to let a
caller opt out of building them entirely when it has no use for them.
Fix
Adds
ChunksCreator_constructFromCovWithOptions(..., bool constructChunksWithAllocatedSeq). Whenfalse,chunksCreator->chunksisleft
NULL(already handled safely byChunksCreator_destruct) instead ofbeing built.
ChunksCreator_constructFromCovbecomes a thin wrapper thatpasses
true, so all five existing call sites (hmm_flagger.c,coverage_format_converter.c, and three test files) are unaffected.CovFastReader_constructis the only caller changed, and passesfalse.Verification
--threads 8--threads 1Existing unit tests (
API:chunks_creator,API:chunk_iterator, and the restof the suite) pass unchanged. I also diffed decompressed output
(
.cov.gz, md5) between unfixed and fixed builds across several inputs(truncated, full genome, contig-subset "large-few-contigs" and
"small-many-contigs" cases) — all identical, confirming this is a pure
memory-management change with no effect on results.
Alternatives considered
My first attempt shrunk the three arrays to a fixed small size instead of
skipping their construction. That regressed
hmm_flagger's own test suite(
test_chunks_creator,test_chunk_iterator) with a glibcfree(): invalid next size (fast)heap-corruption error — something else inthis codebase writes past these buffers regardless of their declared size
when they're actually used with a real (non-degenerate) window length. I
didn't chase that down since it's out of scope here (nothing in
augment_coverage_by_labels's call path touches these arrays at all), butwanted to flag it in case it's useful context — happy to open a separate
issue with the repro if that helps.
Scope note
This PR is intentionally narrow. While investigating this crash I found a
few other things (an unbounded
strcpyinto a couple of fixed-size buffers,a
malloc/reallocNULL-check gap on this same hot path, a small logic bugin
Splitter_getTokenfor empty fields) that are real but unrelated to thisspecific crash. I'll send those separately if useful, to keep this review
focused on the vmem-overrun fix.