Skip to content

ribbon: per-scaffold rotation optimizer + orientation marks (#127) - #128

Open
conchoecia wants to merge 15 commits into
mainfrom
issue-127-rbh_to_ribbon-detangling
Open

ribbon: per-scaffold rotation optimizer + orientation marks (#127)#128
conchoecia wants to merge 15 commits into
mainfrom
issue-127-rbh_to_ribbon-detangling

Conversation

@conchoecia

Copy link
Copy Markdown
Owner

Summary

Closes #127.

  • Per-scaffold rotation optimizer. After chromosome ordering is fixed, every scaffold's plot direction is chosen independently of the assembly fasta so total ribbon crossings between each adjacent species pair are minimized. Cascades top-down (top species pinned forward; each species below decides flips with the species above already pinned). Crossing score = inversion count between top-x and bottom-x orderings (paths are monotonic in y, so two beziers cross iff endpoint orderings flip). Two-stage per cascade step: independent per-chrom seed, then iterated flip-greedy global refinement.
  • Orientation marks. Each chromosome bar gets a small filled triangle at the fasta-3' end. Right-pointing = scaffold plotted fasta-forward, left-pointing = reversed.
  • New config keys, both default True:
    optimize_chrom_rotation: True
    show_orientation_marks:  True
    Setting both to False reproduces the previous plot byte-for-byte (verified on a 12-Lepidoptera test case: 889135 bytes identical).
  • scripts/odp_rbh_to_ribbon used to carry an inline copy of ribbon_plot / _quality_check_chromosome_list / _optimize_spA_based_on_rbh / plot_bezier_lines that had drifted out of sync with scripts/odp_ribbon_plot.py. The snakefile now imports the module directly.
  • Bonus bug fixes surfaced while building the Lepidoptera test case:
    • groupby(...).apply(...) + multi-name index.names assignment in ribbon_plot broke on pandas >=2 (ValueError: Length of new names must be 1, got 2). Replaced with an equivalent cumcount()-based construction of the within-scaffold gene index.
    • With plot_all: True the alpha column was selected without being populated, raising KeyError: "['alpha'] not in index". Now computed in both branches.

Test plan

  • Built a 12-species Lepidoptera test case (Plutella -> Cydia -> Bombyx -> Manduca -> Spodoptera -> Helicoverpa -> Pieris -> Bicyclus -> Maniola -> Vanessa -> Melitaea -> Danaus) using BCnSSimakov2022 anchors.
  • optimize_chrom_rotation: False, show_orientation_marks: False -> byte-identical to plot produced before this PR.
  • optimize_chrom_rotation: True -> visibly fewer twists in the Lepidoptera plot; cascade completes in <2 min for 12 species, ~2k orthologs per pair.
  • show_orientation_marks: True -> triangles render at the fasta-3' end of every chrom bar, on both the chr-coord and rbh-gene-coord panels.

Notes / follow-ups (not in this PR)

  • "Detangling" of chromosome order (not rotation) between adjacent species - a barycenter / median-heuristic pass to reorder scaffolds per row to further reduce crossings. Separate algorithm.
  • Triangle marker style is admittedly subtle on busy plots; happy to swap for an open arrowhead, a small "F"/"R" label, or color-code the bar itself if the user prefers.
  • Rotated 90-degree chromosome labels - mentioned as a separate near-term ask, will land in a follow-up unless the user wants it bundled here.

conchoecia added 15 commits May 23, 2026 22:06
Scaffold plot direction is now chosen independently of the assembly
fasta direction so that the number of bezier-line crossings between each
adjacent species pair is minimized. Cascades top-down: the top species
is pinned fasta-forward, each species below picks its flips with the
species above already decided. Inversion count between the top-x order
and the bottom-x order is used as the crossing score (paths are
monotonic in y, so two paths cross iff their endpoint orderings flip).
A per-chrom independent seed (cheap) is followed by an iterated
flip-greedy refinement on the global crossing count.

Each chromosome bar gets a small filled triangle at the fasta-3' end so
the viewer can read plot-direction vs fasta-direction at a glance.

Two new config keys, both default True, gate the new behavior:
  optimize_chrom_rotation: True
  show_orientation_marks:  True
Setting both to False reproduces the previous plot byte-for-byte.

Also:
  - scripts/odp_rbh_to_ribbon used to carry its own inline copy of
    ribbon_plot/_quality_check_chromosome_list/_optimize_spA_based_on_rbh
    /plot_bezier_lines that had drifted out of sync with
    scripts/odp_ribbon_plot.py. The snakefile now imports the module
    directly so there is one source of truth.
  - Fixed two pre-existing bugs surfaced while building the
    Lepidoptera test case:
      * groupby(...).apply(...) + index.names assignment broke on
        pandas >=2; replaced with an equivalent cumcount-based
        construction of the within-scaffold gene index.
      * When plot_all=True the "alpha" column was selected without
        having been populated; now it is computed in both branches.

Closes #127.
Follow-up to the previous commit on this branch, addressing review
feedback on #127.

Top-row ordering
----------------
optimal-size used plain value_counts() to seed the very first row,
which ignores which row-2 chromosome each top-row scaffold actually
anchors to. Two scaffolds with the same best-FET partner in row 2
could end up scattered across the top row, forcing crossings the
cascade below could not undo.

The new seed _seed_top_order_from_partner groups top-row scaffolds by
their best-FET partner in row 2, orders partner groups by total shared
ortholog count, and within each group orders members by FET strength.
With this, both Plutella xylostella adjacencies the user called out
(CM/509.1 next to CM/495.1, CM/506.1 next to CM/501.1; both pairs
share a row-2 best partner) now hold.

Additional chr_sort_order options
---------------------------------
For comparison and for highly rearranged genomes where the existing
FET-best-partner heuristic struggles, four new modes are now
selectable:

  optimal-barycenter        single top-down barycenter pass
  optimal-barycenter-iter   iterated barycenter sweeps to fixed point
  optimal-median            Eades-style median heuristic
  optimal-swap              adjacent-swap crossing-count polish

All take the same chr_sort_order config slot. The snakefile validator
was extended to accept them. On the Lep test case the existing
optimal-size still wins; the new modes are meant for datasets with no
single best partner per chrom.

Orientation marks and labels
----------------------------
The orientation glyph at the fasta-3' end of each chrom bar is now a
chevron tip on the bar itself (head_h=0.07, head_len_cap=0.0045)
rather than a separate triangle, so it stays subtle on dense plots.

Scaffold labels are now plotted next to each chrom bar at 90 degrees
CCW, anchored just left-and-above the bar's left end, and shortened
to "<first 2 chars>/<last 3 of stem>.<version>" — so NC_051358.1 is
labelled NC/358.1 rather than the full id. Short ids pass through
unchanged. The plot ylim was widened to fit the rotated labels.

Closes none new; continues #127.
A row-by-row cascade -- even bidirectional with crossing-aware
back-tracking -- only sees one neighbor at a time, so a row-4 swap
that needs to propagate up to row 0 stays invisible to it. odp #127
review surfaced concrete failures: CM/509.1 next to CM/495.1, CM/506.1
next to CM/501.1, CM/522.1 next to CM/500.1, and CM/505.1 / NC/729.1
on the other end of CM/523.1 all need the algorithm to look at the
whole stack at once, not pairs.

New chr_sort_order: optimal-lg
-------------------------------
Builds a chromosome-level graph from every adjacent-pair RBH and
partitions chromosomes into synteny linkage components. Each species'
chromosomes inherit a cluster id; the row ordering uses cluster id as
the primary sort key and the existing FET-best-partner cascade as the
within-cluster tie-break. Members of one LG share an x-region across
every row, so ribbons run straight down within an LG.

Graph construction:
  - edges kept only when both sides' FET-strongest partner (top-k=1)
    is the other -- "strict mutual best." Pure union of top-1 picks
    or top-k>=2 collapses the graph to one giant component on
    Lep-style data with universal residual homology.
  - global orphan rescue: a chromosome that received no mutual-best
    edge anywhere in the graph gets exactly one one-way edge to its
    FET-strongest partner across all adjacent-pair RBHs. Restricting
    the rescue to nodes with no other edges prevents a chrom that
    already lives in an LG component on one side from accidentally
    bridging two distinct LG components on the other side via a
    fusion hub.
  - cross-cluster edge weights are computed from all FET-significant
    pairs (not just mutual best); the cluster ordering is a greedy
    nearest-neighbor traversal weighted by those cross weights so
    that fusion partners (e.g. NC/714.1's cluster and NC/729.1's
    cluster sharing Bombyx NC_051368.1) end up plot-adjacent.

Within each cluster, _seed_top_order_within and _cascade_within run
the same FET-best-partner heuristic the existing cascade uses, but
restricted to that cluster's chromosomes, so siblings sharing a
downstream chrom stay adjacent.

Other changes
-------------
- chr_sort_order: "optimal-lg" added to the snakefile validator.
- The chrom scaffold label is now compacted to "<first 2 chars>/<3
  chars before .N>.N" form (NC_051358.1 -> NC/358.1) so the trailing
  decimal suffix stays visible. Labels nudged slightly off the
  bar's left end and rotated CCW 90 degrees to read bottom-up next
  to the chrom.
- Orientation arrow shrunk again (head_h=0.07, head_len_cap=0.0045)
  so the glyph is a subtle chevron tip on the bar instead of a tall
  separate triangle.

Continues #127.
The previous commits were a chain of heuristics, each hoping that a
sensible-looking re-ordering would happen to reduce ribbon crossings.
None of them ever scored against the true crossing count, so each
left a different residual set of crossings that the next commit then
had to chase. This commit adds the actual objective and a direct
search against it.

What's new
----------
- _fenwick_weighted_inversion: O(n log n) weighted inversion count
  over (top_x, bot_x) line endpoints, the exact bezier-crossing count
  between an adjacent species pair.
- _build_pair_line_data: per-pair record cache so the search loop
  never re-touches the RBH dataframe.
- _score_pair_lines: evaluates one pair given current chrom orders
  and per-chrom flips.
- _crossing_local_search: greedy improvement-monotone refinement on
  the global weighted score. Seed = whatever optimal-lg + the flip
  cascade produced (do not throw the cascade output away; refine it).
  Moves per chromosome: F/R flip, swap with adjacent chrom in row.
  Species traversal order per pass: most-rearranged first, then
  outward. Any move that reduces the total score is kept; iteration
  stops when a full pass produces no improvement.

FET-significant orthologs are weighted 1000x faint ones, per the
user's request -- the search effectively minimizes crossings of the
bold ribbons first, only using the faint ones as tiebreak.

Runtime on the 12-species Lepidoptera test: ~3 min for 15 passes.
The seed score (which the previous LG ordering produced) was already
strong, so the marginal drop is small (~4-5%), but the search now
sees and acts on the actual objective rather than hoping a heuristic
happens to produce it.

Continues #127.
Two cheap speedups to the crossing-count local search added in the
previous commit:

- _build_pair_anchor_data: every distinct (top_chrom, bot_chrom) pair
  is collapsed to one record at the within-chrom mean rank on each
  side, weight = (fet_weight if best whole_FET <= 0.05 else 1) *
  shared-ortholog count. Reduces lines/pair from ~2000 to ~200 on
  Lep-style data. The chrom-flip and chrom-swap moves the search
  considers never touch within-chrom gene ordering anyway, so the
  per-ortholog score's extra resolution is wasted there; the anchor
  score sees the same delta-direction for every move the search
  could make.

- early_stop_frac (default 0.05%): a full pass that improves the
  score by less than this fraction stops the loop. The long tail of
  monotone but visually-invisible improvements is skipped without
  changing the converged-on layout.

max_passes raised from 15 to 50 (cap with early-stop, not at the
working point).

Lep test: 198 s, 15 passes, -4.5%  ->  51 s, 11 passes, -6.8%.
Roughly 4x faster and a deeper minimum (because the search now
runs to actual convergence inside the cap).

Continues #127.
User feedback on the previous commit: rotation cleanup at the end
matters, and the coarse-only search regressed in places where the
per-ortholog detail mattered.

Changes
-------
- Local search call site now runs two stages: a coarse-scoring stage
  for fast convergence to the right neighborhood, then a fine
  (per-ortholog) stage to recover the within-chrom detail the coarse
  anchors flattened out.
- Move set extended: chromosome swaps now run at offsets 1, 2, 3 and
  5 (was offset 1 only). Adjacent-only swap can't escape the local
  minimum where a chromosome's right neighbor is unrelated but its
  true partner sits two slots away.
- After each stage's swap+flip loop converges, a dedicated final
  flip-only sweep visits every chromosome once more. By that point
  the ordering context for each scaffold's L/R orientation is fully
  determined, so some flips that weren't useful earlier in the search
  may now reduce the score. User called this out as key.

Lep test runtime: 2:35 for both stages + final sweeps (vs 51 s
single-stage coarse / 3:18 single-stage fine). Stage 1 ends at
score 996e9 (coarse metric); stage 2 starts the per-ortholog
metric at 1.38e12 and ends at 1.16e12.

Continues #127.
User feedback on the previous commit:
- y-axis species labels were clipped to ~6 characters at the left.
- A chromosome that crosses many others above it should get special
  attention to move it.
- Headroom remains on rearranged-species pairs; longer-range moves
  should be tried.

Changes
-------
- Chromosome-level priority: within each species visit, chroms are
  now visited worst-first by the weight of FET-significant lines
  incident to them. The single chromosome that crosses many others
  (the user's "one above that crosses many") is the first one the
  search tries to move.
- Insertion move: each chrom can be inserted at any other slot in
  the row (not just swapped with a fixed neighbor). A chrom whose
  true partner sits five slots away can hop straight there even when
  every individual adjacent swap raises the score because the route
  passes through unrelated chroms.
- y-axis species label fix:
  * caller can pass `species_labels` dict (the run_ribbon driver now
    pulls it from `config["tree"]["tip_label_map"]` if present);
  * default fallback strips the "-taxid-GCAxxx" / "-taxid-GCFxxx"
    accession suffix;
  * figure widened from 8" to 10" and panel layout adjusted so a full
    "Plutella xylostella"-style label fits in the left margin instead
    of being clipped to its rightmost few characters.

Lep test runtime: ~10 min. Coarse stage 1069B->947B (-11.4%), fine
1317B->1111B. Score and visual quality both improved on the same
seed -- insertion + priority found moves the swap-only search missed.

Continues #127.
User feedback: the previous wide figure dropped the sample accession
and felt too wide. The plot is now a single-column 180 mm (7.087")
wide and the y-axis labels keep the assembly accession on a second
line below the species name:

  Plutella xylostella
  GCA019096205.1

The species_labels dict (typically threaded from
config["tree"]["tip_label_map"]) supplies the human-friendly first
line; the second line is parsed from the species id's trailing
GC[AF]xxx.x token. If no tip_label_map is supplied we fall back to
stripping the "-taxid-GCxxx" suffix as the primary label.

Panel width: 4.6", left margin 2.187". Fits two-line labels at
fontsize 7 without clipping.

Continues #127.
… title

User feedback on the previous commit:
- panel title sat too far above the plot
- species names should be italicized
- accession number should be a slightly greyer color
- labels should be robust to missing inputs / fall back gracefully
- there was too much negative space to the left of the labels
- the search was getting stuck because single-row moves
  inevitably crossed something else; some chrom pairs need to
  be pulled together across rows ("sticky" moves)

Changes
-------
- Panel titles now sit 0.03" above the plot (was 0.25") and use
  fontsize 10 (was 12). Right over the panel like a header.
- y-axis labels redrawn as a pair of manual Text() artists per row:
  species name italic at fontsize 7, assembly accession in #888888
  grey at fontsize 6 below. Matplotlib tick labels cannot mix
  styles in a single string, so this is the cleanest way.
- Robustness: species_labels dict missing or partially populated is
  handled (falls back to the stripped genus+species concatenation
  off the species id, then to the bare species id if neither is
  available). Accession is omitted from the label if the species id
  doesn't contain a GC[AF]xxx.x token.
- Panel widened to 5.4" (was 4.6") inside the 7.087" / 180 mm
  figure, tightening the left whitespace.

- Sticky LG-block move: each chromosome carries an FET-graph
  cluster id (from _build_fet_lg_clusters). The new move type, run
  inside the local search after the per-chrom flip / swap / insert
  moves, tries dragging every member of a cluster to a candidate
  slot in its own row simultaneously. Lets the optimizer escape
  the local minimum where a single-row move would have to cross
  other things to reach its true partner column -- exactly the
  "guided annealing" the user described. The candidate slot set
  is the current slot of any cluster member in any row, so the
  block move always lands somewhere already occupied by one of
  its own members.

Lep test runtime: 17:08 (was 9:43 without sticky block). Score
1069B -> 865B in the coarse stage (-19%, vs -11.4% prior) and
1211B -> 1022B in the fine stage; the block move clearly finds
moves the per-row search misses.

Continues #127.
Major reworking of the chromosome ordering / flip optimization
pipeline. Replaces the prior local-search-only approach with a
modified-Sugiyama brushing sweep seeded by optimal-lg, then iterated
random restarts on top.

Pipeline (when optimize_chrom_rotation=True):
  optimal-lg cluster seed
    -> flip cascade (top-down, then a final all-rows greedy pass
       including row 0)
    -> brushing sweep: alternating top-down and bottom-up FET-
       weighted barycenter, with the flip cascade re-run after each
       direction; per-row "feedback" greedy moves on the top row
       (after td) and bottom row (after bu); best-seen state
       tracked across all sweeps; patience-based convergence
    -> iterated restart: small random perturbation of the best
       state, brushing again, keep global best across restarts
    -> render

What's new in detail
--------------------

_brushing_sweep: per-generation td + tm + bu + bm. Each direction
  runs a FET-weighted barycenter pass over every chromosome of every
  row, then a flip cascade. Top-row / bottom-row feedback (tm / bm)
  is a tension-prioritised greedy on the row that the direction
  just settled, accepting moves only if the global crossing count
  drops. Best-seen state is restored at the end, so a transient
  bu regression does not stick.

_optimize_chrom_flips_top_down: now accepts an `initial_flip`
  argument so callers can preserve row-0 (and row-1+) flips chosen
  by upstream moves -- previously this function reset row 0 to all-
  False every call and undid any row-0 flip accepted by
  try_top_moves. Adds a final boundary pass that revisits every
  chromosome in every row (including row 0) with a full-crossing-
  count greedy, catching flips the row-by-row cascade missed.

_render_ribbon_figure: extracted from ribbon_plot so the brushing
  loop can snapshot a PDF after every sweep ("brush stroke"), with
  filenames sorted to match run order:
    {base}_brush_gen{N}_{01_td|02_tm|03_bu|04_bm}.pdf

ribbon_plot:
  - Iterated random restart wrapper. After the initial brushing
    descent, the best state is perturbed (random shuffle of a
    small window of chromosomes in a few randomly chosen rows) and
    brushing is re-run. Keeps the global best across all restarts.
  - Stale snapshot PDFs from prior runs are deleted at the start
    so the workspace only shows the current run's progression.
  - flush=True on all log prints so SLURM's block buffering does
    not hide progress on long runs.

Plot layout:
  - figWidth = 7.087" (single-column journal 180 mm).
  - panelWidth widened to 6.3" (tighter left/right margins) so
    more of the panel is the data.
  - Panel titles raised to +0.8" above the panel top so the
    rotated scaffold labels at the top of each panel no longer
    bleed into the title.
  - Two-line y-axis labels: italic species name (fontsize 7) over
    grey accession in #888888 (fontsize 6), drawn as paired
    Text() artists so matplotlib's tick-label-can't-mix-styles
    limitation does not apply. Robust to missing tip_label_map.

Lepidoptera 12-species test, score 1069B -> 374B (-65%).
Single-descent run ~3 min; full 8-restart pipeline ~30 min without
per-gen snapshots, ~70 min with snapshots.

Notes:
  docs/rbh-to-ribbon-sorting-notes.md captures the design log,
  what worked, what didn't, and open problems.

Continues #127.
Two fixes prompted by user inspection of a converged plot where
CM/509.1 and CM/506.1 in Plutella xylostella were left in suboptimal
orientation despite the boundary flip pass.

Root cause
----------
The brushing sweep tracks best-seen state by an anchor-coarse score
(one record per (top_chrom, bot_chrom) pair at within-chrom mean
rank, for speed). When best_flips is captured, the flips are
optimal w.r.t. that coarsened metric. The per-ortholog FET-weighted
score the boundary flip pass uses can disagree -- and does, by
~3.7 billion crossings on CM/509.1's flip alone on the Lep test.

After brushing restored best_orders/best_flips it did NOT re-run
the flip cascade on the restored state, so the suboptimal flips
captured at best_score moment stuck around.

Fix
---
- _brushing_sweep: after restoring best_orders/best_flips, call
  _optimize_chrom_flips_top_down one more time with
  initial_flip=current state. The cascade's boundary pass is per-
  ortholog FET-weighted so it catches the flips the coarse-score
  brushing missed.
- ribbon_plot's iterated-restart wrapper: same final cascade after
  restoring global_best_state.
- ribbon_plot: after writing the final PDF, persist
  {species_order, chromorder, chromflip} as JSON next to it.
  Future diagnostics (why isn't chrom X flipped?) can load this
  and inspect/score the optimization output without re-running
  the 30+ minute pipeline.

Verified by running the standalone cascade on the previous run's
saved state JSON:
  Before: CM/509.1=R  CM/506.1=F  CM/495.1=R  CM/501.1=F
  After:  CM/509.1=F  CM/506.1=R  CM/495.1=R  CM/501.1=F
matching user-identified misses exactly. CM/495/501 correctly
left unchanged -- per-ortholog says their current orientation is
optimal.

panelWidth reduced 6.0 -> 5.6 so two-line italic species labels
no longer clip past the figure left edge at panelWidth=6.0.

Continues #127.
User asked for <10 min, ideally <2 min on a 12-species Lep test that
previously took 6.5h for thorough mode. This commit gets fast to
~3 min and medium to ~5 min: 119x and 76x speedups respectively.

What was done
-------------
1. Replaced the pandas-.apply hot loop in _eval_pair with a numpy-
   vectorized fast path that scores against precomputed per-pair
   numpy arrays. Tables are built once per cascade call instead
   of being reconstructed for every flip attempt.
2. Added integer chromosome indices to pair tables. Position lookup
   per line is now one numpy fancy-index instead of a per-element
   pandas Series.map. cProfile showed pandas.Series.map at 1612s
   cumulative on the unprofiled-9-min fast run; this change makes
   the lookup essentially free.
3. JIT-compiled the Fenwick weighted-inversion-count inner loop with
   numba (pure-Python fallback if numba is missing). The inversion
   count was the next bottleneck after the indexing fix.
4. Routed the legacy _count_inversions through the same JIT'd
   Fenwick when numba is available.
5. Added an optimization_level config key with values
     fast     - no restarts, ~3 min, -63% from seed
     medium   - 2 restarts, ~5 min, -66% (default)
     slow     - 5 restarts, ~10-15 min, -66%
     thorough - 8 restarts + per-gen snapshot PDFs, ~hour, -66%
   so users can dial cost/quality. Snapshots are gated to thorough
   only (they roughly double wall time at any level).

Lep 12-species test before/after:
  thorough            6h35m -> ~hour (snapshots dominate now)
  medium               18 m -> 5:14
  fast                 9 m  -> 3:19

Score parity (purely computational, not algorithmic):
  fast quality unchanged at 389B (-63.4%)
  medium quality unchanged at 363B (-65.9%)
  slow/thorough plateau at 362B

Continues #127.
Continues from the previous commit's 120x speedup. Remaining medium
runtime was dominated by leftover pandas.Series.map calls in the
brushing-coarse score path and a per-row lambda inside the flip
cascade's bot_positions, both of which only showed up in profile
after the per-ortholog hot path was vectorized.

Changes
-------
1. _build_pair_anchor_data now emits the same integer-chrom-index
   arrays _build_pair_line_table does. _score_pair_lines takes its
   fast path on coarse data too, dropping pandas.Series.map from
   brushing's total_crossings hot loop.
2. bot_positions inside _optimize_chrom_flips_top_down precomputes
   the bot_scaf integer index once and uses fancy-indexed numpy
   lookup instead of a pd.Series(...).map(lambda) call. Profile
   showed ~85s of that lambda on the Lep test.
3. barycenter_position in _brushing_sweep uses np.where on a
   pre-built flip mask column instead of a .apply(lambda) row-by-row.
4. _optimize_chrom_flips_top_down's inner cascade calls inside
   brushing now take max_passes=3 (was 8 default). On Lep the
   inner cascade converges in 1-2 passes anyway; this skips the
   final 5 stagnant ones.
5. optimization_level "medium" now uses 1 restart (was 2). Restart 1
   is what finds the deeper basin on this dataset; restart 2 has
   never beaten it across the test runs.

Final Lep timings:
  fast      3:17  (-63.4% score, no restarts)
  medium    4:16  (-65.9% score, 1 restart)
  slow      ~10 min (5 restarts)
  thorough  ~hour  (8 restarts + snapshot PDFs)

User-facing knob unchanged: optimization_level config key.

Continues #127.
User-identified failure pattern on the Lep test: medium fixes a
visible block-swap between the top 7 rows that fast leaves behind --
the rightmost two chromosomes across rows 0..6 need to swap as a
unit. Single-row swap/insertion can't find this because moving just
one row's pair makes that row's crossings worse against the row
below; the synchronisation across all rows is what makes the move
a global win.

New move type: synchronised column-pair swap. For every adjacent
column pair (k, k+1) and every contiguous row span [r0, r1], swap
the chromosomes at columns k and k+1 in *every* row of the span
at the same time. Accept if global crossing score drops.

Runs as a fourth brushing phase after td + tm + bu + bm, snapshots
named ..._gen{N}_05_cs.pdf in thorough mode.

Cost: O(K * N^2) candidates per call (K = max columns, N = species
count) -- ~30 * 78 = 2300 candidates on Lep, each a single
total_crossings recompute. Adds ~5s per gen on Lep.

Effect: solves what restarts used to solve, in the very first
descent. Lep fast / medium results:

  fast    3:17 / 389B  ->  2:47 / 357B   (-32B, ~15% faster)
  medium  4:16 / 363B  ->  3:45 / 354B   (-9B,  ~12% faster)

The new "fast" beats the old "medium" in both score and time -- the
col-swap move makes a single descent reach quality the old code
needed a perturbation restart to find.

Continues #127.
Fast mode used to leave a visible Cydia-Bombyx flip that medium
caught via random restart -- the cascade's row-by-row greedy locks
in a flip choice using an unweighted metric and the boundary pass
can't single-flip-undo it because the fix requires multiple
simultaneous flips. Three new final-polish passes catch this:

1. Iterated single-flip recheck on the global FET-weighted score.
   Cycles every (sp, scaf) up to 6 passes, accepting any flip that
   drops the total. Catches downhill cascades where each step is
   individually an improvement.

2. Pair-flip pass. For each FET-best (top_chrom, bot_chrom) pair,
   try flipping both simultaneously. Catches the case where
   neither flip alone improves the score but together they do.

3. Triple-flip pass. For each bot chrom B, find its top-2 FET
   partners P1, P2 in the row above; try flipping (B, P1, P2)
   together. Catches user-identified cascade patterns where one
   chrom in one row and two chroms in the row above need to flip
   in unison.

Adds ~2 s to brushing wall time. Score impact on Lep fast mode:
  pre:   357B   (-66.4%)
  post:  337B   (-68.3%)
i.e. better than the prior medium (354B), in the same ~3:42.

Continues #127.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

rbh_to_ribbon detangling

1 participant