Hierarchical precision-graph prior: de-gate, and deploy the surface across its validated shape range - #193
Open
MaartenMarsman wants to merge 11 commits into
Open
Hierarchical precision-graph prior: de-gate, and deploy the surface across its validated shape range#193MaartenMarsman wants to merge 11 commits into
MaartenMarsman wants to merge 11 commits into
Conversation
…ation Under precision_graph_prior = "joint" the graph marginal is the edge prior reweighted by the per-graph normalizer, not the edge prior itself. A fit with edge selection on a continuous precision block now says so once, before sampling, with separate wording for a fixed Bernoulli prior (the realized inclusion probability differs from the nominal one) and for a learned Beta-Bernoulli or Stochastic-Block prior (the learned parameters are themselves reweighted). The notice follows bgms.verbose and does not change the sampler. Note: dev/plans/active/2026-07-31_hier-gating_NOTE.md (WP1).
…on ladder The gauge is a post-sampling diagnostic whose cost is fixed per chain rather than proportional to iter: two assessment sweeps over a capped number of edge moves, which is nothing on a sparse posterior (no mediating block is non-trivial, the normalizer ratio is exact) and 5-10 s per chain on a dense posterior at 100-200 variables. options(bgms.zratio_gauge_sweeps = 0L) is the off switch. The advice printed for a flagged chain is now a ladder: report what was measured (error share, the mediating-block sizes the audit covered via the new per_chain block_lo/block_hi, how many non-trivial moves were audited), then raise the sweeps and refit to resolve whether the signal is real, and only then mention precision_graph_prior = "joint" -- stating that it targets a different model. The gauge never switches specification on its own. The Rd and the diagnostics vignette document the audit as a sample: it resolves coherent error, which is what harm_pred projects, and does not resolve rare edge-specific failures. Silence on a sparse posterior is exactness, not blindness. Note: dev/plans/active/2026-07-31_hier-gating_NOTE.md (WP0 benchmark, WP2).
The anchor grid was filtered to tiers at or below the cap, so a cap landing between two tiers trained the hull at the lower one and extrapolated everything above it: at a cap of 40 the size-42 tier dropped out and the hull stopped at 36. zratio_cap_tier appends a tier at the cap when the surviving grid stops more than two sizes short of it -- two because a mediating block excludes the edge's own endpoints and so never exceeds cap - 2. The grids move into zratio_anchor_grids as the single source for both families, with the sweep budget set by size through zratio_anchor_sweeps. The cache key gains a version tag so a surface built by an earlier grid is not served to this one. Note: dev/plans/active/2026-07-31_hier-gating_NOTE.md (WP3a).
…ndary slope The anchored size range reaches 80 variables, up from 44, with tiers at 52, 64 and 80 (common-neighbour) and 30, 38, 52, 64 and 80 (bipartite). The one-time build stays at 24 s serial. The interior was re-scored before the raise: the wider fit costs the small-size bands 0.0002 -> 0.0004 nats median, inside the 0.003 envelope. Past the range the prediction now continues along the surface's own boundary slope in log-size instead of freezing at the edge. Freezing left the prediction fixed while the truth kept moving, so its error grew without bound in block size. Scored against block-Gibbs gold at sizes 90-150 against a size-80 hull, over two anchor-build seeds: the tangent holds a median 0.0006 nats and at most 0.0011 (common-neighbour) and a median 0.0043 and at most 0.0060 (bipartite), where freezing reaches 0.060 and 0.095 and the additive fallback 0.21 and 0.31. Continuing the fitted quadratic as its own extrapolant was measured and rejected: it degrades with distance (0.0026 and 0.016 over the same band) where the tangent stays flat. The absolute moments grow with component size, so a fitted edge slope pointing downward is a fit pathology. It is floored at zero -- which reproduces the old freeze -- and counted, since a silent per-density-band degeneracy is what the gold scoring above would not catch. Note: dev/plans/active/2026-07-31_hier-gating_NOTE.md (WP3b, WP3c).
The sampler initializes from a complete graph, so warmup alone can put every mediating block past the anchored range while the retained draws stay well inside it. A whole-run share therefore reported a transient as if it described the posterior. The engine now tallies per phase: chain_runner sets Warmup, switches to Retained before the first retained iteration's edge selection, and sets Gauge around the assessment sweeps, which are extra deploy evaluations no stored draw comes from and enter neither tally. The notice reports the retained share, the largest retained block, and the measured accuracy of the extension there, with the warmup share in brackets. Counters that carry no split (an older fit object) report the whole-run share and make no phase claim -- routing them to the retained wording would manufacture reassurance the output does not contain. Note: dev/plans/active/2026-07-31_hier-gating_NOTE.md (WP3d).
…s 0.5, 1, 2 The surface was fenced to the exponential diagonal. It is now validated at three gamma_prior() diagonal shapes -- 0.5, 1 and 2 -- and deploys on the range they span; the interior is interpolated, not measured, and the fence message and docs state the claim as three scored points. Constants first: the generalized Gauss-Laguerre quadrature carrying the shape was certified against brute-force integration before any surface was scored, including at the shape below 1 where the diagonal prior is singular at the origin. Then the eight cells (shape 0.5, 2 x eta 1, 2 x both families) against block-Gibbs gold over two anchor-build seeds: median 0.0001-0.0013 nats, at most 0.0027, against 0.016-0.40 for the additive kernel; the boundary-slope tail spot-checked at size 100 in every cell (0.0001-0.0059). Fixed: zratio_anchor_cn and zratio_anchor_bip did not pass the cell's Gamma shape to the block-Gibbs oracle, so anchors were drawn at the exponential while the constants carried the cell's own. Dormant behind the fence, and it would have made every shape result unattributable. The regression test runs unfenced at a small budget. The upper end is set by the anchor oracle, not the surface: at a non-unit shape the oracle's row update is an independence-Metropolis step whose acceptance falls away from shape 1 (20 nodes, density 0.9: 80% at shape 2, 70% at 2.5, 60% at 3, 35% at 4, 1-12% at 5), and at shape 5 the anchor Monte-Carlo error runs 50-500x the shape-1 reference and is not restored by 94x the sweeps. Anchors off shape 1 run at a raised budget, resolved by matching measured anchor spread rather than by 1/acceptance. The block-Gibbs gold references are tracked under dev/validation so later work scores against the same bank; dev stays out of the build via .Rbuildignore. Note: dev/plans/active/2026-07-31_hier-gating_NOTE.md (WP4).
…update
At a non-unit Gamma diagonal shape the block oracle drew the off-diagonals and
the diagonal pivot jointly from the alpha = 1 conjugate and accepted or
rejected the pair with weight K_ii^(alpha-1). That weight scales with the
diagonal, so acceptance collapsed as the shape grew (measured 80% at shape 2,
1-12% at shape 5) and the whole row froze with it.
The row is now two blocks: an accept/reject on the off-diagonals at a held
pivot, whose ratio carries only the quadratic form it moves, followed by an
exact slice update of the pivot conditional
pi(xi) ∝ xi^delta exp(-beta xi) (xi + q)^(alpha - 1).
There is no acceptance left on the pivot to collapse. Prototype measurement:
pivot ESS per sweep stays 0.59-1.00 over shapes 0.5 to 10 and blocks of 4 to
80, where the old kernel reaches zero from shape 5 and blocks of 20 up.
Slice rather than adaptive rejection because log-concavity is not
unconditional: curvature is -delta/xi^2 - (alpha-1)/(xi+q)^2, so below shape 1
concavity needs delta >= 1 - alpha. At delta = 0.2, alpha = 0.5 the curvature
flips at xi = 1.72 with 3.2% of the target beyond it. That cell is pinned in
the suite so a later proposal that assumes concavity fails against the case
that proves it cannot.
The pivot is recomputed as K_ii - b' C b against the C in hand at every row
update, never carried forward: C moves whenever a neighbouring row updates,
and a stored pivot silently targets the wrong law (caught in the prototype
against an independent random-walk reference, |z| up to 22).
At alpha = 1 nothing changes. There is no coupling, the row stays a direct
Gibbs draw, and the RNG call order is unchanged, so the draw stream is
identical; a test pins the moments to 1e-15 against values from the pre-split
kernel. Cross-checked against that kernel at shapes 0.5 and 2, where it was
validated and still mixes: moments agree to 0.34% relative, max |z| = 1.3 over
five seeds. The slice path itself is scored against the conjugate path it
replaces by running a shape a hair off 1 (max |z| = 1.6).
Slice cap hits are counted and exported alongside the other engine counters;
measured zero everywhere, including the non-log-concave cell.
Note: dev/plans/active/2026-07-31_hier-gating_NOTE.md (WP5 rung 2).
At alpha = 1 + 1e-9 the split branch is taken while the pivot conditional collapses onto the alpha = 1 conjugate law, so the branch split, the held-pivot accept step and the slice sampler are all scored against the bit-identical kernel rather than against a second approximation. It is the only check in the file with that property, so it should not sit behind the slow gate; trimmed to three seeds and 2500 sweeps to keep it a few seconds.
The pair integrals are tabulated on [0, cmax] and interpolated; a read past cmax clamps to the edge value. zratio_saddle_grid integrates them to Cmax = 40 against a table that ended at cmax = 18, so 22 units of the saddle integrand carried a plateau in place of the integrand's decay. cmax 18 -> 42 with ngrid 121 -> 281 holds the node spacing at 0.15, so the range is appended rather than re-spread and all 121 original nodes are bit-identical. zratio_saddle_grid now asserts that the table covers the range it reads, since the clamp is invisible at the call site. MEASURED at the two sigma = 1 reference cells: ihat is unchanged to the bit, ghat moves at most 2.2e-5 absolute (1.8e-6 relative to its head value), and the per-edge log Z-ratio moves at most 4.3e-7 nats. The head shift equals the spurious plateau rectangle net of the true tail mass to seven digits, which identifies the shift as the plateau rather than a change in the integration. Against the 0.003-nat envelope the surface claims are stated in, no recorded validation result changes. The plateau is present in released bgms. The fixture is regenerated for cells 1 and 4 only: the constant builder works in the standardized cell, so it reproduces a stored cell only where sigma = 1, and cells 2 and 3 carry sigma = 2. Those two are frozen and asserted byte-identical through the round trip. evals holds engine log-ratios computed against ihat/ghat, so it is rebuilt in the same pass. The regeneration is a committed dev script with the per-cell parameters encoded and a --check mode that verifies without writing. Local tier: FAIL 0 | WARN 0 | SKIP 96 | PASS 7462.
The engine ignored the surface at every non-unit Gamma diagonal shape.
zratio_build_surfaces built, cached and attached one across the validated
range, and log_zratio then re-tested the shape itself with an alpha == 1
condition left over from the release in which the surface served the
exponential shape only. The two layers disagreed silently: a fit paid for
a surface and was served the additive kernel. MEASURED on a dense
common-neighbour block of 30 at shape 2, eta 2: served 0.38610136 against
a surface value of 0.31794560, 0.068 nats, 23x the stated envelope.
The deployment policy now has one owner. R decides which cells get a
surface; the engine deploys whatever it is attached and does not
re-derive the range. A duplicated fence is not defence in depth.
A test was pinning the defect as a contract ("alpha != 1 fences
log_zratio to the additive path"), which is how it cleared review: a
passing assertion reads as evidence the behaviour was intended. Replaced
with the correct contract, plus a deployment probe asserting the value a
fit receives EQUALS the surface value rather than merely differing from
the additive one.
Shape range 2 -> 10. Scored against block-Gibbs gold at 0.5, 1, 2, 3 and
5, every interior cell inside 0.003 nats; at 10 the guarantee is a bound
rather than an approximation, the whole mediated correction being at most
2.8e-04 nats over the scored band at eta <= 2. Interior interpolated.
Theta quadrature nleg 64 -> 128. A larger shape concentrates the inner
integrand into a narrow spike the fixed rule stopped resolving: MEASURED
error in the tabulated constants at eta 1 ran 2.4e-06 at shape 10,
2.4e-04 at 15 and 3.9e-03 at 20. At 128 every shape through 20 agrees
with a converged rule to 1e-07 or better, for +0.4 s on a once-per-fit
build. The Laguerre axis was not the constraint (1e-15 at shape 20).
The fixture is regenerated through its committed script, second
regeneration; the script's static-move guard fired at 6.1e-11 and now
distinguishes a tail-only change from a quadrature one.
Deployed-route certification: log_zratio equals logR exactly across 36
cells (6 shapes x 2 families x 3 sizes), and an end-to-end gold spot per
shape through log_zratio lands at 4.7e-06 to 1.2e-03 against golds whose
sd is 1e-06 to 4e-04, every one inside the envelope.
Not included: routing past the deployed range to the isolated-edge value.
Beyond shape 10 the additive path still serves, unchanged from the
released package and now documented as a limitation.
Local tier: FAIL 0 | WARN 0 | SKIP 96 | PASS 7469.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
De-gates the hierarchical precision-graph prior: the trust gauge ships on, the
joint spec reports its realized edge prior, the size hull reaches the blocks a
real fit meets, and the absolute-moment surface is validated and actually
deployed across a Gamma diagonal shape range rather than at the exponential
shape alone.
Working notes:
dev/plans/active/2026-07-31_hier-gating_NOTE.md(measurements,failures, and the reusable checklist). Validation artifacts and their
regeneration scripts are under
dev/validation/.The defect this branch exists to have found
ZRatioEngine::log_zratiore-derived the deployment policy the R layer alreadyowned, with an
alpha == 1test left over from the release in which the surfaceserved the exponential shape only. When the validated range widened, the two
layers disagreed silently: a fit built, cached and attached a surface, and was
then served the coarser additive kernel. MEASURED on a dense common-neighbour
block of 30 variables at shape 2, rate 2 — served 0.38610136 against a surface
value of 0.31794560, a gap of 0.068 nats, some twenty times the accuracy the
surface is stated to hold.
It survived every review pass because a test asserted it as the contract
(
"alpha != 1 fences log_zratio to the additive path"). A wrongly-tested pathlooks safe in review in a way an untested path does not. Both the test and the
gate are replaced; the policy now has a single owner in R, and the engine
deploys whatever it is attached.
Every accuracy figure in the program had been scored on the surface path
(
logR) while a fit takes the hot path (log_zratio). Those numbers were trueand none of them exercised the deployed route. That is now certified separately.
Deployed-route certification
Identity:
log_zratioequalslogRexactly (max difference 0) across 36cells — shapes {0.5, 1, 2, 3, 5, 10} x both component families x block sizes
{10, 24, 42}.
End-to-end gold through
log_zratio(rate 2, common-neighbour block of 42):Every deployed error is inside the 0.003-nat envelope. The additive column is
what a fit received before this branch at every shape except 1.
Shape range: two claim types, kept apart
Accuracy claims — scored against block-Gibbs gold, worst error over each
interior band, both anchor-build seeds pooled:
All inside the 0.003-nat envelope. The reference resolves these comparisons:
the gold span across each band runs 61x to 223x its own noise.
Irrelevance claim (shape 10) — a bound, not an approximation. The diagonal
has concentrated far enough that the entire mediated correction is at most
2.8e-04 nats across 22 scored cells (both families, both rates, sizes 20 to
100), so any method returning the isolated-edge value is wrong by at most that.
The bound is set at rate 2, since the correction grows with the rate, and is
scoped to rate <= 2. Mediation magnitude by shape: 0.307, 0.161, 0.0408,
2.8e-04 at shapes 2, 3, 5, 10.
The range moved because the anchor simulator was replaced, not because the
surface changed: at a non-unit shape the old independence-Metropolis pivot
collapsed (about 1% acceptance at shape 5) and the chain froze while still
reporting moments — a confidently wrong answer rather than a noisy one. The
slice-sampled replacement has no acceptance step in that coordinate and its
efficiency per sweep is flat in shape and block size. The exponential-shape path
is unchanged and bit-identical, seed for seed.
Constants layer
Two independent fixes, both user-invisible in magnitude and both real.
saddle integration that consumes it, and the interpolator clamps an
out-of-range read to its edge value, so 22 units of the integration carried a
plateau instead of the integrand's decay. Present in released
bgms. Movesthe per-edge log ratio by at most 4.3e-07 nats. The two grid domains now
assert their compatibility where they meet.
narrow spike the fixed rule stopped resolving: error at rate 1 ran 2.4e-06 at
shape 10, 2.4e-04 at 15 and 3.9e-03 at 20. Doubling the rule restores
agreement to 1e-07 or better through shape 20 for about +0.4 s on a
once-per-fit build. The Laguerre axis was not the constraint (1e-15 at shape
20).
The engine reference fixture is regenerated through a committed script that
encodes the per-cell parameters, freezes the cells the mapping does not apply
to and checks the freeze through the round trip, and distinguishes a tail-only
change from a quadrature one.
Also in this branch
no longer offers a different inferential target as first-line advice. Cost is
fixed per chain, not proportional to
iter: nothing on a sparse posterior,about 5-10 s on a dense large-
qone.the realized graph marginal is the edge prior reweighted by the per-graph
normalizer, not the nominal prior.
lower tier), range raised to 80, and prediction past the hull continues along
the fitted boundary slope instead of freezing — freezing's error grows without
bound in block size. Extrapolation counters split warmup from retained.
shape, 43-47 s at a non-unit one, single-core, at the full 80-variable range.
Gates
Measured locally on the pre-rebase tree:
R CMD check --as-cran: 0 errors, 0 warnings, 1 NOTE (pre-existing: theDESCRIPTION Date field is over a month old)
The branch is rebased onto current
develop, so CI is the authority on themerged tree.
Deliberately not included
10 the additive path still serves, unchanged from the released package and
documented as a limitation. The design is settled (mediation-off flag plumbed
R to engine to interface to chain runner, one rule for both families, rate
<= 2 with a notice beyond) and is scoped as a follow-up rather than landed on
a depleted budget — the failure mode this branch spent a session diagnosing is
precisely a flag honoured in one layer and ignored in another.
blocks. It returns approximately zero at the exponential shape too, so the
"progressive in shape" reading is wrong and no mechanism claim appears
anywhere. The exposure numbers stand on their own measurement.