Skip to content

Skip allocating unused per-chunk window buffers in CovFastReader (fixes multi-hundred-GB vmem overrun in augment_coverage_by_labels) - #67

Open
tafujino wants to merge 1 commit into
mobinasri:mainfrom
tafujino:fix-augment-coverage-by-labels-vmem-overrun
Open

Skip allocating unused per-chunk window buffers in CovFastReader (fixes multi-hundred-GB vmem overrun in augment_coverage_by_labels)#67
tafujino wants to merge 1 commit into
mobinasri:mainfrom
tafujino:fix-augment-coverage-by-labels-vmem-overrun

Conversation

@tafujino

Copy link
Copy Markdown

Problem

augment_coverage_by_labels can request on the order of 200+ GB of virtual
memory
on full human genome-scale input, even though its actual resident
memory usage is under 10 GB. On any environment that enforces a virtual
address space ceiling (ulimit -v / RLIMIT_AS, common on HPC schedulers
such as SGE/UGE), this makes the process fail — killed on a vmem limit if one
is set close to real usage, or crashing partway through chunk construction if
a malloc silently returns NULL and its result is used unchecked (several
malloc calls in this path aren't NULL-checked). I believe this is the same
underlying issue reported in #44 ("Segmentation fault ... during chunk
parsing", failing right at "Creating empty chunks" — exactly the code path
described below).

Root cause

CovFastReader_construct (cov_fast_reader.c) has no window-size concept of
its own, so it reuses chunkLen as the windowLen argument to
ChunksCreator_constructFromCov:

covFastReader->chunksCreator = ChunksCreator_constructFromCov(covPath, NULL, chunkLen, threads, chunkLen);

ChunksCreator_constructFromCovChunk_constructListWithAllocatedSeq
Chunk_constructWithAllocatedSeq then allocates, for every chunk:

chunk->windowRegionArray     = (int *) malloc(windowLen * sizeof(int));
chunk->windowTruthArray      = (int *) malloc(windowLen * sizeof(int));
chunk->windowPredictionArray = (int *) malloc(windowLen * sizeof(int));

augment_coverage_by_labels uses chunkLen = 40,000,000, so windowLen here
is also 40,000,000 — three 160 MB arrays per chunk, for a tool that never
reads or writes them (confirmed by grepping the whole programs/ tree:
CovFastReader walks blocks directly via TrackReader/ptBlock, never
through Chunk's window arrays). Over 507 chunks on a full genome that's
507 × 480 MB ≈ 243 GB of virtual memory reserved for buffers nothing ever
touches — closely matching the maxvmem (~234–235 GB) I measured on the run
that reproduced this.

These three arrays are genuinely used elsewhere (hmm_flagger,
coverage_format_converter) with real, much smaller window sizes — so the
fix is not to shrink or remove the arrays unconditionally (I tried that
first; it's unsafe, see "Alternatives considered" below), but to let a
caller opt out of building them entirely when it has no use for them.

Fix

Adds ChunksCreator_constructFromCovWithOptions(..., bool constructChunksWithAllocatedSeq). When false, chunksCreator->chunks is
left NULL (already handled safely by ChunksCreator_destruct) instead of
being built. ChunksCreator_constructFromCov becomes a thin wrapper that
passes true, so all five existing call sites (hmm_flagger.c,
coverage_format_converter.c, and three test files) are unaffected.
CovFastReader_construct is the only caller changed, and passes false.

Verification

Case Before After
Full genome (507 chunks), --threads 8 ~235 GB vmem required succeeds at 8 GB, ~5.7 GB peak RSS
9M-line truncated input, --threads 1 fails even at 64 GB succeeds at 8 GB, ~2.8 GB peak RSS

Existing unit tests (API:chunks_creator, API:chunk_iterator, and the rest
of the suite) pass unchanged. I also diffed decompressed output
(.cov.gz, md5) between unfixed and fixed builds across several inputs
(truncated, full genome, contig-subset "large-few-contigs" and
"small-many-contigs" cases) — all identical, confirming this is a pure
memory-management change with no effect on results.

Alternatives considered

My first attempt shrunk the three arrays to a fixed small size instead of
skipping their construction. That regressed hmm_flagger's own test suite
(test_chunks_creator, test_chunk_iterator) with a glibc
free(): invalid next size (fast) heap-corruption error — something else in
this codebase writes past these buffers regardless of their declared size
when they're actually used with a real (non-degenerate) window length. I
didn't chase that down since it's out of scope here (nothing in
augment_coverage_by_labels's call path touches these arrays at all), but
wanted to flag it in case it's useful context — happy to open a separate
issue with the repro if that helps.

Scope note

This PR is intentionally narrow. While investigating this crash I found a
few other things (an unbounded strcpy into a couple of fixed-size buffers,
a malloc/realloc NULL-check gap on this same hot path, a small logic bug
in Splitter_getToken for empty fields) that are real but unrelated to this
specific crash. I'll send those separately if useful, to keep this review
focused on the vmem-overrun fix.

CovFastReader_construct passed chunkLen as windowLen to
ChunksCreator_constructFromCov for lack of a real window size of its own.
That caused every chunk to carry 3 windowLen-sized int arrays (160MB each
at the 40e6 chunkLen augment_coverage_by_labels uses) that CovFastReader
never reads or writes -- it walks blocks directly via TrackReader/ptBlock,
not through Chunk's window arrays. At full-genome scale (507 chunks) this
allocated on the order of 240GB that nothing ever touched, hitting
RLIMIT_AS/s_vmem on hosts that enforce a virtual-memory ceiling even
though actual resident memory stayed under 10GB.

Add ChunksCreator_constructFromCovWithOptions with a
constructChunksWithAllocatedSeq flag; CovFastReader now passes false to
skip building chunksCreator->chunks entirely (left NULL, which
ChunksCreator_destruct already handles). hmm_flagger and
coverage_format_converter, which genuinely use those buffers with sane
window sizes, keep calling the original ChunksCreator_constructFromCov
unchanged.

Verified: full genome at --threads 8 went from needing ~235GB (matching
the ~240GB estimate) to succeeding at 8GB; a 9M-line input that failed
even at 64GB now succeeds at 8GB.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant