Improved heuristic formulation of non-convex health burden; avoid MIP#24
Merged
Conversation
The health module's non-convex dose-response curves previously required
SOS1 segment indicators, making health-enabled solves MIPs. HiGHS cannot
solve these at full resolution: its MIP path is limited to a serial dual
simplex for the root relaxation, which does not finish the root LP within
10x the time Gurobi needs for the entire solve, and it has no option to
use its interior-point solver inside the MIP.
Add health.segment_formulation: relax_and_fix, a two-pass LP scheme that
adds no integer structure at all:
1. Solve the model without segment indicators (a valid relaxation; the
delta fill-up formulation is already exact for convex curves).
2. Pin each non-convex (cluster, risk) pair's delta bounds to the fill-up
pattern of the segment containing its relaxed intake
(health.fix_nonconvex_segments).
3. Re-solve warm-started from the pass-1 basis; the relative difference
between the two objectives is a certified optimality gap, checked
against health.relax_and_fix_max_gap.
On the full-resolution central health scenario with HiGHS ipm+crossover
(solving.options_highs: {solver: ipm, run_crossover: on}), the scheme
solves in ~5 minutes end to end with a certified gap of 6.5e-05 (target
1e-03) and a repaired objective slightly better than the Gurobi MIP
incumbent at the same tolerance. The repair pass takes ~8 s. The default
sos1 formulation is unchanged, remains exact, and stays the right choice
with Gurobi.
The repair pass inherits the configured Method=2 (barrier), which ignores the pass-1 basis and re-solves from scratch. With Method=1 the dual simplex hot-starts from the written basis and repairs the delta-bound changes in a few seconds. Measured on the central health scenario: relax-and-fix under Gurobi drops to ~40s total vs ~55s for the exact sos1 MIP, with certified gaps of 4e-04 (value_per_yll=50k) and 0 (200k).
Measured on the full-resolution central scenario (value_per_yll 50k/200k), relax-and-fix beats the exact sos1 MIP under Gurobi on every axis: ~30% faster (42s vs 61s), 37% lower peak memory (3.0 GB vs 4.8 GB, helped by freeing the pass-1 solver model before the repair build), objectives that certify tighter than the 0.1% MIP gap actually achieves, and identical results across Gurobi and HiGHS to seven significant digits. Higher values per YLL tighten the certified gap (exactly 0 at 200k USD/YLL) because intakes get pushed to breakpoint corners where the relaxation is exact. sos1 remains available for exact MIP solves.
Full-resolution solves now peak at ~4.7 GB under HiGHS (ipm+crossover) and ~3.1 GB under Gurobi with the relax-and-fix health formulation (measured on central scen-central), so the previous 8000 MB default was oversized. Retries still escalate the allocation by 1.3x per attempt. Claude-Session: https://claude.ai/code/session_014hiDv7bnKQnhu5B7hnVfhe
A certified gap above health.relax_and_fix_max_gap previously aborted the solve, forcing a config change and rerun even when only a few scenarios were affected. Since re-fixing segments can only improve the incumbent but never the relaxed bound, an uncertifiable gap structurally requires the MIP: run_relax_and_fix now re-fixes from the repaired intakes for up to three rounds (boundary-pressing intakes select the neighbouring segment) and then restores the exact SOS1 indicators on the same model, seeded with the repaired solution as MIP start, instead of erroring. Validated by forcing the fallback on the full-resolution central config (max_gap 1e-9): Gurobi accepts the seeded incumbent and certifies 0.078% in ~35s, confirming the heuristic solution was already optimal to solver tolerance. Unit tests cover both the fallback chain (via a synthetic S-shaped curve whose relaxation provably undercuts) and the certified fast path. Claude-Session: https://claude.ai/code/session_014hiDv7bnKQnhu5B7hnVfhe
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The binary-variable-based piece-wise linear approximation of non-convex constraints that makes the health formulation work causes the Highs solver to struggle with default, health-enabled GLADE models. This PR introduces a heuristic approach which works very well in practice: solve the relaxation of the MIP first, then fix binary variables to the induced linear segment and resolve; a "relax-and-fit" approach. It turns out that this works well in most cases, cuts down on solving time by Gurobi by ~30%, and makes a Highs solve feasible in under 5 minutes.