From 07a2465ed5e399b211e4106a23cafef380c05fdb Mon Sep 17 00:00:00 2001 From: Glenn Hickey Date: Sat, 25 Jul 2026 15:26:08 -0400 Subject: [PATCH 1/2] update pinchesAndCacti: cheaper and faster pinch graph [WIP] Picks up two changes to the pinch graph, plus a small API addition: - the per-thread AVL index over segments is replaced by a coarse coordinate index sized by segment density. This drops ~48 bytes of heap per segment (an avl_node plus its malloc overhead) and makes lookups cheaper, and the density-based sizing is what keeps it cheap on the shattered graphs high divergence alignments produce rather than only on pangenomes - the block orientation is packed into the low bit of the block pointer, taking stPinchSegment from 56 bytes to 48 - stPinchBlock_setNumSupportingHomologies, so a client can hang one value off each block without paying for a hash table (hal2vg uses it for node ids) Correctness: stPinchesAndCactiTests (46, including two new ones covering the index against a brute force walk), stCafTests (17) and cactus_barTests (17) all pass. hal2vg produces byte identical output through all of this on yeast, GRCh38 chrY and 16 yeast chromosomes, at 46% less peak memory. WHAT IS NOT YET VALIDATED, and why this is a work in progress: The speedup is measured on a synthetic workload only: repeated random pinches followed by joinTrivialBoundaries over 20 threads, where it runs 1.5x to 2.1x faster than the AVL across pinch lengths from 2 to 150 bases. Real caf pinches are structured and locally correlated, and thread lengths are skewed, so this needs confirming on real data. evolverMammals cannot confirm it: caf accounts for 1 second of the 159 that cactus_consolidated spends on the mr job, the rest being bar/poa. Old and new differed by about 8% on peak RSS there, but three runs of the *same* binary varied by 19%, so that measurement was noise. A caf dominated workload is needed instead. Two things noticed along the way that are worth separate attention: - cactus_consolidated is not deterministic even at --threads 1, consistent with sets keyed on pointer values. Note that this change moves allocation patterns, so it will shift pointer values and hash iteration order, and hence output, without any behavioural difference - cactus builds with assertions enabled (include.mk adds -UNDEBUG over sonLib's -DNDEBUG), unlike hal2vg's build of the same library. Some of the asserts here are in hot paths and their cost has not been measured Co-Authored-By: Claude Opus 5 (1M context) --- submodules/pinchesAndCacti | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/submodules/pinchesAndCacti b/submodules/pinchesAndCacti index 65550eb62..e55deee84 160000 --- a/submodules/pinchesAndCacti +++ b/submodules/pinchesAndCacti @@ -1 +1 @@ -Subproject commit 65550eb62fdabebb4df2c94d79333dc9baca9a48 +Subproject commit e55deee840a2a5bc38ab666394755e65690f605a From e33317d415916abef8bcc8570303ab42469273cf Mon Sep 17 00:00:00 2001 From: Glenn Hickey Date: Sun, 26 Jul 2026 08:22:41 -0400 Subject: [PATCH 2/2] update pinchesAndCacti --- submodules/pinchesAndCacti | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/submodules/pinchesAndCacti b/submodules/pinchesAndCacti index e55deee84..b2b45b44b 160000 --- a/submodules/pinchesAndCacti +++ b/submodules/pinchesAndCacti @@ -1 +1 @@ -Subproject commit e55deee840a2a5bc38ab666394755e65690f605a +Subproject commit b2b45b44b519951fb8656093e05e28d83c989058