Dynamic-shape support (code-only): family/point identity, kernel-faithful bench, oracle point dispatch - #80
Open
eellison wants to merge 43 commits into
Open
Dynamic-shape support (code-only): family/point identity, kernel-faithful bench, oracle point dispatch#80eellison wants to merge 43 commits into
eellison wants to merge 43 commits into
Conversation
…lift
Capture side of dynamic shapes (design §2.1/§2.2). A dynamic-compilation
region used to be dropped (lifted shape param held fx.Nodes ->
'Expected List[int]'). Now:
- _harvest_shape_env: {symbols:{name:{hint,range}}, guards, captured_dynamic}
from the live ShapeEnv (free symbols only; specialized [k,k] dropped;
oo/int_oo bounds -> None; guards filtered to recorded symbols).
- _record_placeholder: per-slot shape_exprs/stride_exprs alongside hint ints
(additive 'symbolic' block; absent for static tensors).
- live symint input -> ['I', hint, expr] (fixes the S([hint]) conflation).
- _lift_shape_arg: resolves Node/SymInt entries to hints (valid List[int],
region captures at hint) + parallel shape_param_exprs.
- serialization overlays exprs onto compact entries; signature rendered
from the hint-evaluated concrete copy; eager validation evaluates at the
hint binding. shape_hash still from hint ints -> dynamic dedupes to the
static point; symbols/guards ride the index entry as metadata.
Verified: GroupNorm-family dynamic capture now yields 1 region / 0 dropped
with inputs [[64,64,'s53','s0'],...], ['I',256,'s0*s53'], ['S',[64,32,2,
's0*s53']]. 72 tests green (static path untouched). Loop-close test
(load+rebind+eager-run) and static-regression byte-diff still TODO.
written with claude code
…gression
Closes the capture->consume loop end to end and pins the harvest helpers:
- 3 CPU unit tests (make_fx symbolic trace gives a live ShapeEnv): expr
strings round-trip through sympy.sympify; _harvest_shape_env returns
{symbols:{hint,range}, guards, captured_dynamic} with int_oo->None;
None env -> None block.
- Loop-close (GPU, manual): GroupNorm-family dynamic capture -> shapes.json
-> load_shape_configs at hint runs (64,64,16,16); guard-RESPECTING rebind
s0=8,s53=32 (product still 256) runs (64,64,32,8); guard-VIOLATING
24x24 (product 576) is LOUDLY rejected by validate_bindings. The capture
preserved the real guard s0*s53==256, which is exactly what stops a
rebind from benching a shape the model never ran.
- Static regression: a static-compile capture through this branch's
capture_hook is byte-identical (pattern_hash, shape_hash, inputs,
signature) to the pristine checkout's. Changes are fully additive.
75 tests green (72 + 3).
written with claude code
…rse path Per review: never parse/regex when we have the live SymNode; strides are expressions (composed of other exprs), evaluated via sympy only when a concrete value is needed; share one utility across capture paths. Shared extractor (full_graph_harness, imported by capture_hook — no private copy): sym_expr_str, symbolic_block_from_value, harvest_shape_env, shape_env_from_gm, shape_env_of. Codec is the single serialization boundary: compact_from_spec overlays per-slot exprs into shape/stride, spec_from_compact reconstructs the symbolic block, evaluate_spec resolves a verbose spec at a binding (reusing _eval_dim). All evaluation is sympy; zero int()-on-expr, zero regex on the symbolic path. Killed the lossy text path: _parse_intish (mangled '64*s0*s53' -> 64) and _symbolic_dims are deleted. Annotation shape/stride tokens keep exact expr strings (string split, no regex, no try/except); Sym(expr) inputs keep their expr instead of defaulting to 32. The sidecar validator now compares shape/stride expr-to-expr via sympy-equivalence (_dims_equal), so the '16384 != 64' roundtrip warning is gone — both sides carry the same exprs from the same live SymNodes. Full-graph sidecar records the symbols table + guards (harvest_shape_env) and the symbolic shape/stride exprs; load_full_graph_definition evaluates specs at the hint binding. Tests (+4, 79 total): codec symbolic round-trip is exprs-not-hints (pins the 16384!=64 bug), evaluate_spec at hint and rebind, _dims_equal sympy equivalence, Sym(expr) input parsed without regex/default, capture_hook re-exports the shared extractor (no duplicate). Static path verified byte-clean (no symbolic artifacts); loop-close still passes. written with claude code
_write_shapes_json now threads symbols/guards from the index entry: - symbols + guards are GRAPH-LEVEL (top of shapes.json, shared across points) — exactly where _parse_shapes_json reads them; idempotent merge (per-symbol update, guards de-duped). - each dynamic point gets bindings (the per-symbol hints — what every expr evaluates under to reproduce the captured snapshot) + captured_dynamic. - static captures (no symbols) are untouched: no symbols/guards/bindings/ captured_dynamic fields appear. Full pipeline now closes with NO hand-built shapes.json: capture a dynamic region -> merge -> load_shape_configs at the hint and a guard-respecting rebind (8x32) runs, guard-violating 24x24 is rejected. Verified on GPU (merge_e2e) + 2 CPU plumbing tests (dynamic schema written / static stays clean). 81 tests green. written with claude code
Two consistency gaps closed: 1. Sidecar symint inputs dropped their expr. _scalar_spec_from_value now keeps it (sym_expr_str), matching the region path's ['I', hint, expr]. Codec carries it: compact_from_spec emits ['I', hint, expr] for a symint with an expr (['sym', hint] for a constant), spec_from_compact reconstructs it. Without this a sidecar-loaded symint couldn't rebind. 2. harvest_shape_env only read backed_var_to_val, so an UNBACKED symbol (u0 — from data-dependent nonzero/item/unique/masked-index) appearing in a captured expr would not be in the table, and instantiate_point would raise 'unbound symbol' at load. Now unbacked symbols are harvested too (is_unbacked_symint, hint from size_hint/range fallback) and flagged unbacked=True so a consumer knows the hint is a data-dependent fallback, not an observed size. Rare in fusion regions (data-dependent ops are extern), but no longer a silent landmine. Tests (+2, 83 total): ['I',hint,expr] codec round-trip + evaluate; unbacked-symbol harvest flag. merge-e2e + suite green. written with claude code
…allback) An unbacked symbol still needs a concrete value to benchmark, but unlike a backed hint that value isn't an observation — it's a choice of varying faithfulness. Harvest now takes the best tier available and records which: observed — real propagated runtime value (real_tensor_prop_unbacked_vals) size_hint — derived from backed subs / static eval range_fallback — arbitrary range-floor placeholder so the bench/accounting side can treat a range_fallback point as sweep-or-exclude (a single number there is noise; no static baseline exists for a data-dependent dim) vs. trust an observed one. Backed symbols stay implicitly observed, no flags — common case is byte-clean. 83 tests green (hint_source tier test covers all three); merge-e2e green. written with claude code
The unit tests covered each piece in isolation; the three real-pipeline checks lived as throwaway /tmp scripts (nothing re-ran them, so a capture_hook refactor could silently break dynamic capture while units stayed green — the static-only-blind-spot class this effort exists to kill). Now in the suite (GPU-gated, skip without CUDA): - test_dynamic_capture_merge_load_roundtrip_gpu: install hook -> capture GroupNorm dynamic -> assert 0 dropped + symbolic inputs (['I',..], expr dims) -> merge -> shapes.json has graph-level symbols + per-point bindings -> load_shape_configs runs eager at hint and a guard-respecting rebind (product preserved) -> guard-violating rebind loudly rejected. - test_static_capture_has_no_symbolic_artifacts_gpu: a static compile captures with zero symbolic content (additive-path pin). Also pinned finite range upper bounds survive harvest (only oo/int_oo -> None) — _jsonable_range_bound(4096)==4096. 85 tests green (CUDA present; 2 skip without). written with claude code
The --dynamic path compiled with blanket torch.compile(dynamic=True),
which dynamizes genuinely-static dims too — a measurably different kernel
than the model ran (design §2.5: 40.4us blanket vs 35.5us marking only the
model's dims). Now dynamic_dims_for_repro reads the per-input symbolic
blocks from shapes.json -> {tensor-input-index: [symbolic dims]} and the
bench torch._dynamo.mark_dynamic's exactly those (input, dim) pairs on
each row's tensors, compiling with dynamic=None (honor the marks) instead
of blanket True. Falls back to blanket dynamic=True only when no symbolic
dims are recorded.
Verified: marking the captured dims gives the dynamic 2-kernel artifact
(~28us) and, in the clean inline case, AVOIDS a rebind recompile that
blanket dynamic=True triggers (recompile_check: marked reuses, blanket
recompiles). Tests: dynamic_dims_for_repro maps only tensor positions /
returns None for static repros.
Known residual (separate, harder): the captured repro lifts symints out as
explicit forward args (mul, arg0_1 = Sym(...)), so changing the binding
changes those scalar ARGUMENTS and dynamo re-specializes -> a recompile
warning fires at the second distinct binding for this repro form. Marking
the tensor dims is still strictly correct + an improvement; making the
lifted-symint form reuse across bindings is a follow-up (mark/pass those
as dynamic too). Not corpus-corrupting — capture is faithful.
85+2 tests green.
written with claude code
…lock it Investigated 'mark_dynamic isn't sufficient'. Findings recorded in design doc 2.5b: - mark_dynamic only re-derives an approximation — guards re-traced, symbols re-allocated, coupling (s0*s53==256) NOT restored. - torch.compile(shapes_spec=ShapesSpec(...)) (local pytorch ShapesSpec stack) re-creates the ShapeEnv: ShapeVar/IntVar objects (shared = coupling), derived-dim exprs, ranges, assumptions=[guard]. Our shapes.json maps 1:1. - VERIFIED via spike: a shapes-derived-inside-forward model -> 1 graph across a coupled 16x16->32x8 rebind (faithful, no recompile). - BLOCKER: captured repros lift reshape/view shapes into _shape_param LIST args; ParamsSpec can't spec dynamic list elements, so the changing list re-specializes (spike: lifted=2 graphs, not-lifted=1). - DECISION NEEDED: for captured_dynamic repros, don't lift shape params — derive them inside forward from the dynamic input (we already capture the exprs). Capture-generation change, gated on captured_dynamic, static path untouched. No code change this commit — investigation + design note only. written with claude code
…stant list Refines 2.5b per the user's point that lifting aids repro reuse. Spikes: - C: symbols bound via scalar IntVar slots BUT shape params still plain-int lists -> 2 graphs (the constant list literal re-specializes regardless). - D: reshape target built in forward from the lifted symint args ([64,32,2,mul]) -> 1 graph across the coupled 16x16->32x8 rebind. Faithful. So we KEEP lifting symints as scalar args (the bare IntVar slots that bind the symbols) and, for a lifted shape-param list, spell the reshape target from those symint args in the forward body instead of a standalone constant-list _shape_param placeholder. Dynamic-capture generation change, gated on captured_dynamic, static lifting untouched. ParamsSpec has no list-element spec, so this is also the only way to make a lifted reshape target dynamic under ShapesSpec. written with claude code
…list param Per review: serialize the shape list's symbolic elements like a single symint and faithfully re-create them — not freeze them into a constant list. _lift_shape_arg now skips lifting when the shape list contains symbolic dims (fx.Node SymInts): those entries already reference the lifted symint INPUT nodes (mul, arg0_1) in scope as forward args, so the faithful form is the inline list the model's graph had — reshape(x,[64,32,2,mul]) — not a standalone _shape_param=[64,32,2,256] placeholder. Fully-static shape lists still lift exactly as before (static path untouched). This is what the model's graph actually did; our lift had REPLACED it with a frozen list. Verified: the captured GroupNorm repro now emits reshape(arg2_1,[64,32,2,mul]) / reshape(...,[64,64,arg0_1,arg1_1]) with no _shape_param args, and driving that exact repro through torch.compile(shapes_spec=...) reuses ONE dynamic graph across the coupled 16x16->32x8 rebind (verify2: 1 graph). Before this change the constant-list param re-specialized and that was impossible. e2e test now asserts no _shape_param in a captured dynamic repro; static lift tests unchanged. 87 tests green. (Realizing 1-graph reuse in the BENCH still needs the ShapesSpec path #1 — the harness mark_dynamic path still recompiles; this commit is the necessary generation enabler, verified sufficient via ShapesSpec directly.) written with claude code
… dump) torch.compile(shapes_spec=...) logs a TypeError traceback every compile (SymInt not JSON serializable, from _get_dynamo_config_for_logging's json.dumps of the _shapes_spec config). Caught by pytorch's own telemetry guard (metrics_context.py:103), so compile succeeds and the result is correct — stderr noise only. Recorded in design 2.5b with fix options (upstream blocklist/.to_jsonable, or filter the line our side). Neither the swallow nor the bug is ours; we add no exception handler. written with claude code
…estions) Single-page map: baseline bugs, the faithful capture->merge->load->run machinery (with file:line refs + the real serialized artifact), the compile question resolved-in-principle (ShapesSpec = the user's own PR stack, with PR refs), why symbolic-list-lifting is the crux, and the 4 open design questions (shapes.json<->ShapesSpec connection, oracle<->family dispatch, upstream telemetry bug, unbacked bench treatment) + remaining implementation. written with claude code
…n tests) Three parallel read-only reviewers, each finding demonstrated with output. Static path PROVEN untouched (reviewer round-tripped 5761 real corpus entries byte-identical). Fixes: B (SEVERE): str(SymInt) of PythonMod/CeilToInt/ModularIndexing/TruncToInt re-parsed as OPAQUE undefined sympy funcs -> _eval_dim raised / validate_bindings SILENTLY accepted invalid shapes (interpolate class). Fix: shared input_codec._sympify_expr with a torch _sympy.functions locals map + integer/positive symbol assumptions; both callsites use it; validate_bindings now LOUD when a fully-bound guard doesn't fold to a concrete bool (cross-point guards over unbound symbols still skipped). G (MED): symbolic shape + no recorded stride -> _contiguous_stride int() crash. Now builds a symbolic contiguous stride; spec_from_compact records its stride_exprs so evaluate_spec folds it (latent bug my test surfaced). H/I (MED): _dims_equal raised 'nan not comparable' on Max(1,floor(..)) vs floor, and false-mismatched floor(s)==s. Now integer-positive sympify + (a-b).simplify + numeric-sampling fallback for the undecidable case. C (SEVERE): dynamic_dims_for_repro keyed by tensor-only counter; a symint before a tensor marked the WRONG input. Now ABSOLUTE make_inputs position. A (MED): a guard referencing a SPECIALIZED symbol (Eq(s0*s53, s77), s77==64) was dropped, losing the constraint. Now substitute specialized consts into guards before the drop-filter -> Eq(s0*s53, 64) preserved. D (HIGH): merge overwrote a point's bindings with a DIFFERENT capture's symbol names (same shape_hash) -> 'unbound symbol' on load. Now distinct symbolizations become distinct points; bindings never overwritten. J (MED): unguarded size_hint() raise lost the WHOLE harvest. Now caught -> range-fallback tier. K (LOW): non-int unbacked hint silently truncated. Now flagged (hint_noninteger) not silent. E (dynamic-vs-static shape_hash identity) deferred to round 2 — deepest, fix non-obvious. 95 tests green (+8 pins). Static corpus untouched. written with claude code
…o static) Investigated the deferred deep finding. MEASURED: a dynamic capture's pattern_hash AND shape_hash both fork from the static capture of the same hint shapes, so the design-doc §E 'dynamic dedups to static' claim is currently FALSE. Causes pinned with evidence: - shape_hash: the inline-reshape change lifts symint INPUT placeholders (mul,s0,s53) that the static path lacks; they inflate the hash. Excluding dtype=='symint' placeholders makes the dynamic hash EXACTLY match the static (2e02cdda->7395c806), and scanning all 1727 corpus shapes.json shows ZERO have a sym/I input -> the change is safe (no existing hash moves). - pattern_hash: inline reshape vs lifted _shape_param + dynamic subgraphs skip canonicalization -> genuinely different graphs. - identity is also non-deterministic (symbol allocation is trace-context sensitive; re-captures scatter to different hashes/dirs). finding_e_identity_analysis.md lays out the design decision: Option 1 (make them dedup: exclude symints from shape_hash + canonicalize dynamic subgraphs) vs Option 2 (distinct entries, correlate static-vs-dynamic at the accounting layer; retract §E). Recommend Option 2 + the safe shape_hash change. Analysis + recommendation only — no semantic change committed pending the call. Status doc open-question 5 added. written with claude code
…nal probes sympy .equals()/.simplify() probe expressions at FRACTIONAL random points (is_constant -> _random), which makes torch's PythonMod.eval assert (integer-only) -> uncaught AssertionError crashed the sidecar validator on a real Mod stride. Broadened both guards to catch Exception (best-effort equivalence with a sound sampling fallback). Also strengthened the sampling: per-symbol independent sequences + an all-equal and all-distinct point, so an expr differing only when symbols coincide/differ is still caught. +1 regression test. 96 green. written with claude code
…, Finding 1, Finding 4 Two reviewers (attack new round-1 code; breadth over 14 real model families). The breadth pass found the deepest bug yet. R2-1 (SEVERE, root cause of ALL rebind failures + 3 family drops): int(val.untyped_storage().size()) in _record_placeholder calls int() on a SymInt (e.g. 4*s27*s77) under dynamic shapes, FORCING an equality guard into the live ShapeEnv pinning the byte-size. harvest_shape_env then baked that spurious guard into shapes.json, which rejected every rebind changing the product (mode A: 10 families) and, when it specialized all dims, dropped whole regions (mode B: broadcast/embedding/index_select). Fix: read the storage-size HINT (.node.hint), never int() the SymInt; AND read all symbolic-tensor metadata (is_contiguous, storage_offset, size) under ShapeEnv.suppress_guards() — capture must never mutate the env it harvests. Verified: multi-symbol reduce now captures guards:[] (was Eq(...,6400)) and rebinds to a DIFFERENT product (128x200) that previously raised; broadcast family captures (was n_captured=0). The old GroupNorm e2e test asserted the BUGGY rejection (24x24 was actually valid — the view is consistent for any h,w); corrected it to verify the valid rebind runs + no spurious guard + a genuine range violation still raises. Finding 1 (MED-HIGH): _sympify_expr forced positive=True on EVERY symbol, but zero-capable unbacked symbols (range [0,...]) then fold Ne(u0,0)->True / Eq(u0,0)->False — dropping a guard / rejecting a valid u0=0. Fix: sign assumption driven by the symbol's range floor (positive iff floor>=1, else nonnegative); validate_bindings passes the symbol table through. Finding 4 (HIGH): merge unioned symbols by raw name, but dynamo reuses s0 across captures with different ranges -> range clobber / guard cross-contamination. Fix: when an incoming symbol name collides with a DIFFERENT existing definition, namespace this capture's symbols (rename in symbols+bindings+guards+inputs via sympy subs) before union. + self-finding: _dims_equal must catch Exception (torch PythonMod.eval asserts on sympy's fractional .equals() probes) + independent-per-symbol sampling. 98 tests green. written with claude code
…el count); drop try/except Round 3 attacked the round-2 fix code. Only 2 findings, both LOW-MED; the round-2 fixes (suppress_guards, namespacing, sign logic) held under attack, and the 2.6x static-vs-dynamic premise was confirmed with real numbers (GroupNorm 8.1us/1-kernel static vs 17.2us/2-kernel dynamic). R3-1 (residual R2-1 class): _no_guards discovered the ShapeEnv from shape+stride only, so a view with static shape/stride but a SYMBOLIC storage_offset left se=None and the bool(SymInt) offset read leaked a guard. Fix: discovery loop also probes storage_offset. R3-2 (pre-existing, round-1): --dynamic reported the STATIC n_kernels for the marked path (count_kernels invoked on one shape -> dynamo 0/1/many specializes). Fix: count_kernels gains dynamic=None (honor mark_dynamic) + second_inputs (a second differently-bound marked input forces generalization to the DYNAMIC kernel set). Verified: single-shape=1, two-shape=2 (matches the benched artifact). Per user rules, also REMOVED the try/except I'd added: _dims_equal now decides purely by integer-point sampling (sympy .equals/.simplify dropped — they were the only things that raised, by probing fractional points that trip torch PythonMod.eval; integer .subs never raises, a 0-divisor folds to zoo that is_Integer rejects). Storage-size read de-try/excepted too (untyped_storage().size() verified safe on fakes). No regex anywhere in the codec path (sympy throughout). 100 tests green. written with claude code
The numeric-sampling fallback was a heuristic that could false-positive (distinct exprs agreeing at all sample points — reviewer's Mod+poly construction). Per user: no sampling. Replaced with _dims_provably_differ, which is sound both ways and never guesses: - structural ea==eb (commutativity/cancellation) -> not differ - (ea-eb).is_zero True/False -> sympy decided - is_zero None: a non-zero POLYNOMIAL difference is provably non-zero (two distinct integer polynomials differ); a diff containing floor/Mod/Max sympy can't fold stays UNDECIDED -> not proven, return False. The validator raises only on a PROVEN mismatch (was: on any not-equal), so an undecidable pair (Max(1,X) vs X) no longer drops a valid repro. No .equals()/.simplify() (their fractional probing asserts inside torch's PythonMod.eval), hence no try/except and no sampling. is_zero/expand are pure algebra. Tests rewritten to _dims_provably_differ semantics. 100 green. written with claude code
…o == Per the user: simplify each symbolic expr ONCE at write, then it's idempotent — the same canonicalization discipline the subgraph retrace already uses (trace/simplify once -> fixed point). canonical_expr_str = str(sympify(expr)); applied at the two emit points: - sym_expr_str (THE extractor -> sidecar / shapes.json exprs) - _dim_tokens (the print_readable annotation parse) so both renderings of one expr are STRING-IDENTICAL. Verified idempotent on every form (64*s0*s53//64 -> s0*s53 -> s0*s53; Max/Mod/CeilToInt stable). This removes the rendering skew that the round-1..3 sympy-equivalence machinery existed to absorb. _dims_provably_differ (structural ==, is_zero, polynomial-non-zero reasoning) is DELETED; the sidecar-vs-annotation check is now plain canonical-string _dims_differ. The whole class of 'two different strings for the same expr' bugs is gone at the source rather than papered over at compare time. Real captures still round-trip clean (no validator raise); 100 tests green (tests rewritten to canonical-equality + an idempotence-invariant test). written with claude code
…othesis: confirmed) VERIFIED experiment: torch._inductor.compile_fx(gm, fake_symbolic_inputs) compiles the CAPTURED post-grad graph symbolically (it detect_fake_mode's the SymInt-shaped fakes), bypassing dynamo entirely. ONE artifact ran both (64,100) and (128,200) with no recompile / no ShapesSpec. The call convention wants the lifted symints passed explicitly — exactly our ['I',hint,expr] capture shape, in forward order. Structurally better than ShapesSpec: runs the model's ACTUAL symbolic graph (no dynamo re-derivation of symbols/guards), no spec-builder, avoids the _shapes_spec telemetry-dump bug, coupling/guards already baked in. Today the full-graph repros run torch.compile(reconstructed_module) -> back through dynamo (the limitation). Finding + open questions in investigation_results/compile_fx_direct_finding.md (GM source: re-trace+re-symbolicate from our symbols table vs serialize the gm; fragile-API guarding; bench methodology; static parity). Recommendation: make compile_fx-direct the dynamic-bench mechanism; ShapesSpec the fallback. No implementation yet — decision needed on the GM source. written with claude code
q1 (where the GM comes from) RESOLVED by probing: - REJECTED hand-building symbolic graph (make_fx + create_unbacked_symint): fights make_fx's unbacked-symint mode (_set_unbacked_bindings assert), reallocates symbol names. - REJECTED serialize-the-gm: we don't store one. - WORKS (verified): the repro's forward already takes lifted symints as args and is written symbolically, so re-run its own dynamic=True+mark_dynamic compile, intercept the post-grad gm at the post_grad hook (same mechanism capture uses), hand that gm + symbolic placeholder fake-vals to compile_fx. One artifact serves every binding (hint 16x16 correct; 64x100 & 128x200 from one artifact, no recompile). Per-binding args: build via the codec's instantiate_point (consistent full binding for ALL symbols in placeholder order) -> compiled(*args), NOT hand-rolled name maps (brittle vs derived/realloc'd symbols). written with claude code
Pushed the compile_fx-direct idea to its limit with experiments. Findings (investigation_results/compile_fx_direct_finding.md): - compile_fx DOES compile the dynamo-produced post-grad symbolic gm, and ONE such artifact rebinds across shapes (verified 64x100 & 128x200). BUT that gm's placeholders are reordered + carry AOT-introduced FREE symints (s62/s73) with no clean map to our shapes.json -> per-binding call-args are fragile. - The clean alternative — make_fx ONCE (perfect forward-order correspondence, our own symbols) then compile_fx — SPECIALIZES to the trace shapes: AOTAutograd emits assert_size_stride(trace dims) and rebind raises (16==32). ignore_shape_env=True doesn't remove it; aot_module_simplified (dynamo's own backend entry) on the make_fx gm specializes too. So the dynamic context that suppresses those asserts is set up by DYNAMO, not reproducible from make_fx. Conclusion: per-binding compile_fx and the existing mark_dynamic --dynamic path BOTH recompile per point; compile_fx adds internal-API fragility for no timing benefit at per-binding granularity, and its only edge (one physical artifact across bindings) is exactly the fragile free-symbol-arg part. So KEEP mark_dynamic as the --dynamic mechanism; compile_fx-direct / one-artifact-reuse is future work gated on a clean free-symbol arg mapping or an upstream rebindable-artifact export. Backed out the half-wired --dynamic-mode=compile_fx scaffold (helpers + flag); repro_harness.py unchanged from last green. Doc-only. 100 tests green. written with claude code
The earlier negative was my bug. The fix is one line: feed compile_fx the make_fx graph's OWN SYMBOLIC placeholder fake-vals (it detect_fake_mode's them -> compiles dynamic), NOT concrete inputs (which specialize). Then it's the clean path that has everything: - make_fx ONCE -> placeholders in FORWARD order, 1:1 with repro inputs, our own symbols (no AOTAutograd reordering, no introduced free symints s62/s73). - compile_fx ONCE -> ONE stable artifact serves EVERY binding, no recompile. - per-binding args = make_inputs_from_config(binding) in forward order -> compiled(*args). No symbol mapping, no parsing, no fragile free-symbol work. Verified via CLI (one artifact, 16x16 15.9us / 8x32 16.0us, no recompile) and a GPU regression test (same artifact id across 8x32/16x16/24x24). build_compile_fx_dynamic_artifact + _distinct_trace_binding (trace at distinct dim values so the two dynamic dims stay distinct symbols, not a unified square symbol). Wired as --dynamic-mode (default compile_fx; mark_dynamic kept as a no-internal-API fallback). compile_fx mode can't recompile (no dynamo) so it skips the recompile check and times the one artifact under the shared CUDAGraph+lock methodology. Corrected compile_fx_direct_finding.md (the negative was symbolic-vs-concrete example inputs). 101 tests green (+1 one-artifact GPU test). written with claude code
…eferred Per user (canonicalize symbols like we order outputs by first use): a canonical FAMILY identity makes two equivalent dynamic captures hash the same regardless of dynamo's symbol-name realloc. Rule: rename free symbols to c0,c1,... by first appearance across inputs+guards, then family_hash = md5(pattern_hash + canonical_inputs + canonical_guards) over the renamed + already-canonical exprs. VERIFIED (canon_symbols.py): two GroupNorm captures from independent trace contexts -> identical canonical inputs/guards/symbols and identical family_hash e17e43ee084d. Two-level identity: FAMILY (pattern + canonical structure + guards; binding-invariant; guards ARE identity) and POINT (the binding). Resolves the run-to-run instability and answers the dispatch-identity question (dynamic oracle matches the family hash; binding selects the point). Dispatch wiring (oracle_harness) DEFERRED per user 'discuss later' — doc records the scheme; this branch ships capture+bench only. written with claude code
…ent; +3 coverage tests
Reviewer checked whether STATIC capture invariants hold for DYNAMIC. Two
findings + coverage gaps:
Finding 1 (comment was FALSE): the capture_hook dedup claim ('dynamic dedupes
to static of same hint shapes') is wrong for symint-bearing partitions —
symint placeholders perturb shape_hash and the dynamic DAG skips
canonicalization. That's CORRECT behavior (dynamic kernel != static kernel,
the 2.6x premise); corrected the comment to the resolved two-level
canonical-symbol FAMILY-hash identity (finding_e_identity_analysis.md).
Finding 2 (MED, latent code-path): a symbolic storage_offset was neither
DETECTED (is_symbolic_entry) nor EVALUATED (evaluate_symbolic_entry /
evaluate_spec) — a view at a symbolic offset would as_stride at the frozen
hint offset on rebind. Fixed: symbolic_block_from_value records offset_expr;
compact_from_spec emits it in 'off'; spec_from_compact reconstructs it;
both evaluators fold it. Verified 2*s0 -> 32 at s0=16, static offset
untouched. (Not corpus-reachable today — Inductor splits aliased slices —
but the code path is now correct.)
Coverage gaps filled (invariants that HOLD for dynamic but had no dynamic
test): symbolic storage_offset round-trip; channels-last DYNAMIC stride
round-trip+evaluate (all prior dynamic-stride tests were row-major);
['S',[...,expr]] lifted-param evaluate path.
Negative results from the review (hold for dynamic): hash stability
run-to-run, codec round-trip, eager-validation across 5 families, ship-
parity. 104 tests green.
written with claude code
…ompile_fx is not kernel-faithful R4 adversarial review + 3 independent reproductions overturned the central claim of compile_fx_direct_finding.md. compile_fx-direct (make_fx(symbolic) -> compile_fx) emits a DIFFERENT, more-fused kernel set than the model's real dynamo->AOT->inductor graph: var_mean GroupNorm collapses to 1 fused kernel where the model runs 2 (reduction + separate pointwise epilogue). Feeding the inductor decomp table to make_fx does NOT reconcile them. So --dynamic-mode= compile_fx benched a kernel the model never runs. (The reviewer reported the fusion DIRECTION inverted — their 22us vs 11us was square-16x16 unification contamination, their own Finding 4 — but the load-bearing fact holds.) Fixes: - Default --dynamic-mode flipped compile_fx -> mark_dynamic (the model's real pipeline). compile_fx demoted to a clearly-labeled NON-FAITHFUL diagnostic. - mark_dynamic path PRE-WARMS the compile-once artifact at two guard-valid, internally-distinct shapes before timing any row. NEW bug beyond the review: marks + a SINGLE shape still 0/1/many-specializes (triton_per_fused, 1 kernel); only a SECOND distinct shape forces the general kernel set (2 kernels = model's real). Without the pre-warm row 1 timed a specialization and rows 2+ recompiled. Recompile detection tightened: with the pre-warm NO row may recompile (the old first-row free pass is obsolete and masked misses). - Guard-aware warmup/trace-binding selection (_distinct_dynamic_bindings, _distinct_trace_binding now take repro_file) replaces the blind name->val+i perturbation that broke ranges / Eq(s0,s1) couplings / divisibility guards (reviewer Finding 3). Coupled families warm at equal-magnitude shapes (the square kernel IS the model's there). Built on new non-raising predicates binding_violation / bindings_satisfy (no try/except for control flow); validate_bindings now wraps binding_violation, same loud behavior. Tests (+5): test_dynamic_default_measures_general_kernel_gpu (every row matches the model's reduction+pointwise fusion structure, no recompile), test_compile_fx_direct_diverges_from_model_kernels_gpu (pins the 1-vs-2 divergence; flips loudly if a future torch makes them agree), test_binding_violation_predicate_matches_validate_bindings, test_distinct_dynamic_bindings_respects_guards, + corrected the stale "faithful" docstring on test_compile_fx_one_artifact_serves_many_bindings_gpu. compile_fx_direct_finding.md gets a top CORRECTION banner + full analysis. Reviewer Finding 2 (warm-cache input0/t0 NameError) is now off the default path; documented as a diagnostic-only latent crash. 90 tests green. written with claude code
…ad_graph_module)
Found via f(f(x))==f(x) on opacus_cifar10: capture the model (A), then recapture
A's saved full_graph_*.py (A'). 4/6 regions reproduced byte-identically but 2
var_mean/mean regions DIVERGED — a fully-dynamic saved graph (full_graph_004:
Sym(s16), reshape(arg4,[64,32,2,mul_2]), symbolic strides) recaptured as a
STATIC region (different pattern hash, frozen 1024 dims, lifted _shape_param
const list) — exactly the anti-pattern the dynamic work removes.
Root cause: ingest_tlparse.load_graph_module traced make_fx(tracing_mode="real")
and turned symint inputs into int(hint), baking symbolic shapes to their hints.
The local annotation parser was also lossy (s16->hint, strides->garbage default).
Fix (load_graph_module): when the forward annotations carry symbolic dims/
symints, parse them losslessly via full_graph_harness.parse_full_graph_inputs
(keeps symbol names + stride exprs) and rebuild inputs with ONE shared ShapeEnv
symbol per name, so a symint input (arg2_1:Sym(s16)) and a tensor dim
(arg4_1:[...,s16,...]) trace as the SAME symbol; then make_fx(tracing_mode=
"symbolic"). Stride exprs ("64*s16*s82") are evaluated over the torch SymInt
NODES (not sympy — sympy.subs yields a sympy.Mul torch.empty rejects) in a
namespace exposing only the symbol nodes, with every identifier checked known
first (unrecognized -> None -> real-mode fallback). Static graphs (no symbolic
spec) keep the original real-mode path untouched.
Result: dynamic saved graphs now recapture with dynamism preserved
(arg4_1 -> (64,64,s16,s82)); recapture-from-A reproduces A's pattern hashes,
and recapture(recapture(x))==recapture(x) is byte-identical 12/12. The residual
symbol-NAME drift (s17/s15 vs s16/s82) does not change the pattern hash (already
canonicalized by first-appearance) — the canonical-symbol naming wiring stays
the deferred follow-up.
Tests: upgraded test_loader_synthesizes_symint_inputs -> assert the tensor dim
stays symbolic AND shares the symint's symbol (mutation-verified: fails under
tracing_mode="real"); added test_recapture_is_fixed_point_for_dynamic_graph
(recapture twice -> byte-identical canonical content). 15 recapture + 90
canonical-invariant tests green.
written with claude code
…re drift)
Closes the symbol-name drift residual from the recapture fix: dynamo's ShapeEnv
allocates symbol names off a global counter (s17/s15 one run, s7/s92 another),
so the SAME family captured/recaptured twice landed DIFFERENT names -> non-
idempotent shapes.json (pattern hash matched, but symbols/inputs/bindings text
differed).
Fix (merge_captures._write_shapes_json): canonicalize symbol names to s0,s1,...
by FIRST APPEARANCE across the inputs (then guards), left-to-right — the same
discipline as ordering outputs by definition order — BEFORE collision detection
and write. Implemented with NO regex and NO lossy parsing: free symbols come
from sympy (_sympify_expr.free_symbols), within-expr order from sympy's own
sort_key (no str.find substring hazard, e.g. 's1' inside 's15'), and the
coupled rewrite (symbols table + inputs exprs + strides + guards + bindings)
reuses the existing substring-safe sympy-based renamers. Idempotent: an
already-canonical input renames to itself.
Also fixes a collision-detection bug this exposed: the same-name check compared
the FULL symbol def including `hint`, so two points of one family differing only
in BINDING (hint 8 vs 4) were spuriously namespaced (s0__e6c5). Now it compares
RANGE only (structural identity) — same family shares s0,s1 across points, the
per-point `bindings` carry the hint (the FAMILY/POINT model). Verified on opacus:
3 distinct shape_hash points now share {s0,s1} with bindings 8/4/2 instead of
3 namespaced symbol sets.
Static path untouched (canonicalization only runs when symbols present).
Verified end-to-end: fresh capture and recapture-of-it agree on canonical names;
the shared shape_hash point is byte-identical; recapture(recapture(x)) byte-
stable 12/12.
Tests: +test_canonicalize_symbols_first_appearance_order (two dynamo namings of
one family canonicalize identically + idempotent); updated 3 merge/roundtrip
tests + 3 bench tests to the canonical-name contract (bind via the written
shapes.json's s0/s1, not raw entry names). 106 canonical + 15 recapture green.
(Pre-existing unrelated failure: test_same_pattern_hash_reuses_existing_
canonical_dir — model-key format, fails without this change too.)
written with claude code
…nch) Wires dynamic-oracle dispatch through the EXISTING oracle bench path — static is just the 1-point case. The gap: oracle_impl(point=<hash>) resolved a point's shape and dispatched by exact shape ==, but a DYNAMIC point's resolved shape is symbolic (64,64,s0,s1) and never == concrete runtime shapes, so a dynamic oracle registered by hash silently never fired. Fix (the caller already knows the point it's benching — pass the hash, don't re-derive it from tensors): - repro_harness._parse_shapes_json: carry cfg["shape_hash"] on each config (the bench loop already iterates points; now it has the hash cleanly, no label parsing). - OracleRegistry.select(inputs, point=None): when a point is threaded, match entry["point"] == point FIRST (tiers 0a point+hardware / 0b point), with NO shape comparison — so one hash registration covers the whole family (every binding) and the kernel handles concrete dims internally. Falls back to the existing shape tiers when no hash is threaded (static back-compat). - oracle_impl(point=): a point whose resolved shape has symbolic dims registers SHAPE-LESS (shape=None) keyed by the hash; a fully-concrete (static) point keeps its shape for shape-matching. point is now passed to register(). - bench_oracle / resolve_oracle / bench_oracle_all_shapes thread point through; the all-shapes loop passes config["shape_hash"]. Two bugs this surfaced, fixed: - duplicate-registration warning keyed only on (hardware, shape) -> a dynamic oracle's N shape-less points all warned falsely. Now keyed on (hardware, shape, point). - shape-general fallback (tiers 3/4) matched shape=None entries — but a point-keyed entry is shape-less yet NOT shape-general (tied to one point). Excluded point-keyed entries from the shape-general tiers; they're reachable only via the point tiers (a bare no-hash lookup now correctly finds nothing). No --dynamic oracle path: same --all-shapes loop, dynamic = repro has >1 point with symbolic shapes that load_shape_configs already concretized per binding. Inside the kernel, shape handling is the agent's (harness routes by identity). Tests: +_run_point_dispatch_tests in test_oracle_dispatch (dynamic point routes by hash; same hash matches multiple concrete shapes = family coverage; bare/ unknown-hash lookups raise loudly; shape-less point entry not leaked as shape- general). Full dispatch self-test + 106 canonical/recapture green; static oracle dispatch unchanged. written with claude code
opacus_cifar10 was the ONLY dynamic model in the live corpus, and its 25 saved full_graphs were stale pre-dynamic-work artifacts: symbolic-stride annotations vs concrete-stride sidecars made 19/25 (21/27 on the shared branch) fail load_full_graph_definition's cross-check, and the old pipeline could not recapture them (FX Node JSON-serialization crash on symbolic shapes). Recaptured with the fixed pipeline (this branch): 25 graphs, 14 regions. - Load failures: 19/25 -> 0/25. - Dynamic canonical dirs carry canonical symbols (s0,s1), points bind per binding (8/8, 4/4, 2/2), rebind verified (s0=24 -> (64,64,24,24) + symints). - pointwise_e129f028e0b8 (pre-existing, has an oracle) upgraded v2->v3 and gains shapes.json: its oracle registers by shape-string and the first point keeps the exact concrete shape (64,64,8,8)x2, so dispatch still matches; the 3 new dynamic points are uncovered (honest NO_ORACLE_FOR_SHAPE), not broken. - No absolute paths in artifacts; sidecar/annotation stride exprs identical (lossless canonical form). validate_corpus_invariants.py: ALL HARD INVARIANTS PASS (1487 >= 1482 baseline; all parse; all loadable; manifests resolve; eager validation). Test suite on the changed corpus: 106 passed. written with claude code
…njection)
Fuzzing the dynamic-shapes surfaces surfaced a code-injection: _sympify_expr
calls sympy.sympify(), which eval()s its argument (documented-unsafe on
untrusted input). Those expr strings arrive from DATA artifacts — shapes.json
guards/dims/strides and full-graph annotations — that are never otherwise
executed as Python, so a poisoned artifact ran arbitrary code on load/merge/
bench. Confirmed with a canary: __import__('os').system(...) in a guard string
executed via binding_violation and via _eval_dim (both pure-data channels).
The rest of the codebase strips builtins in its eval() sites, but (a) that
alone is escapable via the ().__class__.__subclasses__() walk, and (b) the new
sympify path had no gate at all — strictly weaker than the standard the code
already holds. Fix at the single chokepoint: validate the string against a
structural AST allowlist BEFORE sympify sees it. Barred are the pieces every
known escape needs — attribute access, subscripting, string/bytes literals,
dunder names, and calls to anything outside the sympy/torch shape-function set.
Fail-closed and LOUD (raise), never silently strip (that would be lossy).
Non-str input (a sympy expr/int) skips the gate; sympify does not eval those.
Verified: all 6 injection payloads + both data channels blocked (no canary);
all 15 corpus exprs and the torch/sympy shape functions (PythonMod, CeilToInt,
Max, FloorDiv, Eq(Mod(...),0), ...) still parse unchanged. Regression test
test_sympify_expr_rejects_code_injection added. Full suite green (92 canonical
+ 15 recapture + oracle-dispatch), corpus invariants pass (1487>=1482).
written with claude code
…mpify)
Fuzzing the dynamic-shapes surfaces found a LIVE idempotence violation. A real
torch SymNode floor-div is stored by capture (sym_expr_str -> canonical_expr_str
on the live EXPR) as '(s0//8)', but re-canonicalizing that STRING (annotation
parse / merge dedup via _dims_differ) yields 'floor(s0/8)'. The two disagree
under the plain '==' the contract promises, so the same mathematical expr is
seen as two distinct ones -> spurious sidecar-vs-annotation skew and dedup
misses. FloorDiv ('s0//k') is ubiquitous in real dynamic shapes.
Root cause: canonical_expr_str's string branch normalizes through sympify, but
its expr branch did a raw str(expr) that keeps torch's '(s0//8)' rendering. Fix
routes BOTH branches through _sympify_expr, so the sidecar-via-expr and
annotation-via-string forms land on one string. Verified idempotent fixed point
and eval-equivalence ((s0//8) and floor(s0/8) evaluate identically across
bindings). The existing idempotence test only fed strings (both calls hit the
string branch, hiding the bug) — extended to torch-expr inputs and cross-branch
consistency (C(e) == C(str(e))). Full suite green (92 canonical + 15 recapture +
oracle-dispatch), corpus invariants pass.
written with claude code
…lowlist
The T()/S()/Index()/Perm() shape-config strings (shapes.txt, _shapes_config,
shapes.json compact entries) are parsed by raw eval() under __builtins__: {}.
Those strings are DATA artifacts — a poisoned corpus entry must not run code
when a repro instantiates its inputs. The empty-builtins guard alone is
escapable: the ().__class__.__base__.__subclasses__()[...] gadget reaches
os.system without naming a builtin. This is the same data-channel injection
class as the sympify fix (8e58505), on the other eval boundary.
Add input_codec._assert_safe_shape_config — a config-grammar structural
allowlist (distinct from _SAFE_EXPR_NODES): permits only T/S/Index/Perm
constructor calls over numeric/string literals, dtype-token names, and the
single narrow torch.<dtype> attribute (e.g. torch.complex64, the sole attribute
real configs use); bars attribute chains, subscripting, dunder names, **kwargs
splat, comprehensions, lambdas, and calls to anything but the four
constructors. Wired fail-closed BEFORE eval at all five sites:
repro_harness (_eval_signature, _parse_shapes_txt, parse_shapes_config),
oracle_harness.parse_shapes_signature, and
scripts/validation/update_bounds.parse_compact_config.
Verified: 19/19 injection payloads (incl. 5 torch.<attr> escape probes) raise
ValueError with the canary never executing; 0 false-rejects across all 2850
real corpus config strings. Regression test test_shape_config_rejects_code_injection
checks both the gate and the live entry points. 93 canonical + 18 oracle-hardening
tests pass; corpus invariants pass.
written with claude code
Owner
Author
|
Follow-up hardening (13ed4a1): closed the pre-existing raw The Verified: 19/19 injection payloads (incl. 5 Note the same fleet-reset merge-conflict caveat applies to the |
…erge) pattern_hash is shape-blind (ops+wiring), so heterogeneous dynamic captures collided into one canonical dir: point-0's repro.py was frozen onto siblings whose baked constants differ (GroupNorm reshape split 2/4/8) and each trace's guards were pooled into one transitively-contradictory list that rejected a point's own recorded binding. A dynamic capture now joins a pattern dir only when its FAMILY IDENTITY matches: hint-blind forward body (AnnAssign shape annotations stripped — they render the binding, which must not split a family) + canonical hint-free symbolic input signature (a pointwise body references no shapes, so [64,128,s0,s1] vs [64,s0,s1,s2] is only visible there). Different families split into dir__2, __3, ...; the first family keeps the plain name so oracle-bearing dirs stay put. Static grouping is untouched. ingest_tlparse now seeds the symbolic re-trace from the sidecar meta.json's native hints instead of a default: the hint sets the recaptured point's bindings and shape_hash (the join key back to live occurrence counts), so a default hint times the wrong problem size and orphans the accounting join. Also updates two stale meta.json model-key assertions in scripts/test_merge_captures.py (qualified keys landed in 2d888a1 but this untracked-by-CI test file was never updated). written with claude code
… dir)
Regenerated the six opacus_cifar10 canonical dirs from the saved
symbolic full_graphs through the fixed merge. Heterogeneous families now
live in their own dirs with their own repro.py and guards:
var_mean_3a40eaee0716{,__2,__3} GroupNorm C=64/128/256 (split 2/4/8)
pointwise_e129f028e0b8{,__2,__3,__4} static+3 symbolizations of add+relu+copy_
pointwise_ea4adf68b80a{,__2,__3} 3 symbolizations
All 14 pre-repair points reproduce with matching shape_hash and native
hint bindings (8/8, 4/4, 2/2 pyramid; mean_67992e back at s0=512), plus
one recovered point ([64,s0,s1,s2] @ {128,4,4}) that the old pooled
merge had silently absorbed into df9da442's occurrence count. The two
previously hard-failing var_mean points and e129's guard-rejected point
now pass eager validation at their native bindings (15/15 configs).
e129's oracle, shapes.txt and legacy model key are preserved; its
oracle checks PASS. Re-running the recapture against the live root is a
no-op (merge idempotent). Corpus hard invariants pass (1494 dirs).
Known residual (pre-existing, now visible): shape_hash concretizes at
hints, so two different symbolizations can share a point hash across
sibling dirs (df9da442 appears in e129__2 and e129__3). Dispatch and
occurrence data are dir-scoped, so no ambiguity in consumers today.
written with claude code
Owner
Author
|
Exercising the pipeline on the recaptured corpus surfaced a soundness bug in the dynamic point-merge, now fixed in two follow-up commits: 989b4a2 — merge: split dynamic captures by symbolic-family identity. ebabb12 — corpus: re-partition the opacus dynamic families. All 14 pre-repair points reproduce with identical shape_hash + native hint bindings, plus one recovered point the old pooling silently absorbed. 15/15 points pass eager validation at native bindings; e129's oracle/shapes.txt preserved and its Residuals noted in the commit message (dir-scoped shape_hash reuse across sibling dirs; re-trace emits no guards where live dynamo did). |
…e-only) Scope decision (maintainer): this PR only ADDS dynamic-shape support — capture, merge, bench, oracle point dispatch, security gates, tests — and does not touch existing repros. repros/ is restored wholesale to the base branch's state (the opacus recapture + family re-partition move to the dynamic-shapes-corpus branch for a follow-up PR once the support code lands). Conflict resolutions: - repros/** (52 files): base version taken verbatim; branch-only additions (canonical family dirs, opacus manifest) removed. - oracle_harness.py bench signature: union of both sides' params (point= dispatch key from this branch; numerics_optout= / disable_gpu_lock= from base). Both bodies verified wired. - tests/test_merge_captures.py: base renamed scripts/test_*.py into tests/; both sides had fixed the same stale qualified-model-key assertion — base's comment kept, this branch's dynamic-family regression tests carried across the rename. - tests/test_bench_accounting.py: the moved file's assertions and _RecordingCaptureState double encoded pre-feature interfaces; updated to the current ones (expr-preserving symint/tensor specs instead of fabricated default-32 concretization; shape_env_block kwarg). Intent unchanged. - scripts/validation/update_bounds.py: deleted by base (dead-script cleanup) while this branch had gated its eval site; deletion wins — the eval site no longer exists (parse_compact_config is gone from the tree), the other four gated eval sites remain. Suite after merge: 372 passed; the 2 remaining failures (test_scatter_reduce_detection) fail identically on the pristine base. written with claude code
Owner
Author
|
Rescoped per maintainer: this PR now only adds dynamic-shape support and touches no existing repros. Merged The corpus work (opacus recapture + family re-partition, fully validated) is preserved on the PR is now conflict-free: 16 files, 0 under |
…ntity)
Completes the identity scheme at the POINT level, before any dynamic
corpus lands (nothing to migrate: the live corpus is static-only and
static hashes are byte-unchanged).
- shape_hash_for_placeholders: symbolic placeholders append a
canonical-symbol expr suffix (shape/stride exprs + symint expr,
first-appearance renaming in graph slot order). The concrete fields
are hint-evaluated, so [64,128,s0,s1]@(4,4) and [64,s0,s1,s2]@
(128,4,4) collided — two different families conflated into one point,
one absorbing the other's occurrence counts (full_key dedup keys on
this hash, so per-graph occurrence counting is fixed by the same
change). Static captures hash exactly as before.
- canonicalize_symbols: trace-internal symbols (in the table but
referenced by no input expr or guard) now canonicalize too, name-
sorted after the slot-ordered referenced ones — they used to keep raw
dynamo names (s31/s79 leaking into bindings). Idempotent.
- _rename_symbols_in_expr: subs -> xreplace. subs(dict) applies rules
sequentially, so a swap rename ({s1->s0, s0->s1}, possible when raw
names are already sN in permuted positions) could chain-corrupt;
xreplace is exact-node and simultaneous, which the docstring already
promised.
Guards investigation (no code change): the re-trace path genuinely has
zero guards to harvest (make_fx symbolic cannot re-derive model-context
dynamo guards; verified shape_env_from_gm finds the env, 2 symbols,
0 guards on a saved opacus graph). Live capture already preserves
guards end-to-end (region entries + full-graph sidecars). Sidecar guard
inheritance at recapture is the follow-up if re-trace guard fidelity is
ever needed.
written with claude code
Owner
Author
|
One more commit (f8d9edd) closing the point-identity residuals flagged earlier — cheapest possible moment since no dynamic corpus has landed yet, so nothing migrates: |
…points scripts/exercise_dynamic_shapes_live.py: runnable end-to-end example — live dynamic captures of 7 small models (incl. the C=64/C=128 GroupNorm pair that exercises the family split on live captures), merged into a SCRATCH canonical root, every family run eager at native binding and a 2x rebind, 0-bindings rejected loudly. The checked-in corpus is never touched. Docstring documents the per-point static/--dynamic bench commands (repro self-bench, GPU lock, CUDAGraph, do_bench min). investigation_results/dynamic_vs_static_eval_points_2026-08-12.md (+raw json): first generalization curves on B200, 3 families x 4 eval points. Headline: pointwise generalizes for free (<=1.01x until the largest point); reductions pay 1.1-5.2x, concentrated in two static-numel heuristics visible in the kernel names — persistent->looped reduction (per_* unavailable dynamically) and, for var_mean, a 1->2 kernel fusion split that grows with size. Dynamic-at-the-hint is already 2.14x for var_mean: the penalty is structural, not tuning. written with claude code
Owner
Author
|
Added a runnable end-to-end example + first generalization curves (49ecc89, maintainer request: examples with different eval points, static-vs-dynamic per point):
|
--prewarm on the repro self-bench overrides the auto-picked two-shape --dynamic warmup with an EXPLICIT ordered binding list. The first warm binding is the hint inductor tunes the general kernel at, so '--prewarm s0=2 --prewarm s0=8' vs '--prewarm s0=8 --prewarm s0=16' isolates compile-history sensitivity at the timed points (maintainer question: 'run first at 1, then 8, time at 8 — does that mess up perf?'). A warm list that fails to generalize a symbol still surfaces as the existing per-row recompile warning. scripts/pilot_single_dim_dynamic.py: pilot captures for the REALISTIC dynamism pattern — exactly ONE dim dynamic (batch is one common case, seq-len another; all-dims-dynamic is rare). Three blocks (FFN with dynamic batch, attention softmax with the REDUCTION dim dynamic, GroupNorm with dynamic batch), scratch-only, plus a mechanism probe for the batch=1-then-8 story: 0/1 specialization gives batch=1 its OWN graph (extra compile) while 8/16 share one general graph — a batch=1 first run does not set the general kernel's tuning hint. written with claude code
…rder irrelevant Pilot results (B200, scripts/pilot_single_dim_dynamic.py + --prewarm sweeps; doc + raw json in investigation_results/): - WHICH dim is dynamic is what matters, not how many: LayerNorm with dynamic batch is 0.99x static at every point (the persistent kernel survives — the reduction numel is the static feature dim); the same reduction with the REDUCTION dim dynamic (softmax over K) pays 1.4-2.1x and drops persistent->looped. The earlier all-dims 2-5x is the reduction-dim penalty, not a general dynamism tax. - Warm order / first-seen shape does not change the general kernel's per-point perf (small-first vs hint-first within +-2%). - batch=1-first is safe: 0/1 rule gives it its own graph; it never becomes the general kernel's hint (probe: unique_graphs=2 for 1,8,16). - Found a repro-fidelity gap: symint INPUTS lift to plain int args, dynamo specializes them per compile, infer_size then PINS the marked dim to the constant -> ConstraintViolationError for the GroupNorm- batch family at ANY binding. Original model unaffected (its symint derives from the tensor). Follow-up: emit lifted root-dim symints as re-derivations (x.size(0)) in the repro. capture_hook: emitted repros now append the GENERATING repo root to sys.path as a fallback — a repro merged into a scratch root otherwise silently imports whatever repro_harness is findable (a stale checkout's copy benched the first pilot run with the wrong flags/methodology; prior-curve numbers spot-checked clean on the correct harness). written with claude code
Owner
Author
|
Single-dim dynamism pilot landed (abbf9b0 + 3701f2d) — maintainer questions: realistic captures where only ONE dim is dynamic (batch as one case, seq-len as another), different eval points, warm-order effects. Headline: WHICH dim is dynamic is what matters, not how many. LayerNorm with dynamic batch = 0.99× static at every point (the persistent kernel survives; its reduction numel is the static feature dim). The same reduction with the reduction dim dynamic (attention softmax over K) pays 1.4–2.1× (persistent→looped). The earlier all-dims 2–5× curve is the reduction-dim penalty, not a general dynamism tax. Warm order doesn't matter: new Also surfaced two support fixes: emitted repros now append their generating repo root to sys.path (a scratch-merged repro could silently bench with a stale checkout's harness — prior curve spot-checked clean on the correct one), and a documented repro-fidelity gap for symint-INPUT families under |
…lose the gap) Closes the pilot's finding-4 infra gap. A captured region often takes the model's x.size(0) as a SYMINT INPUT; the standalone repro lifts it to a plain Python int argument, dynamo specializes int args per compile, and infer_size then PINS the marked tensor dim to that constant -> ConstraintViolationError at every binding (GroupNorm-batch family). symint_derivations_for_repro builds a plan from the point's compact inputs: each ['I',hint,expr] slot whose expr's root symbols are readable off a tensor input's dims (closed arithmetic grammar: Integer/Symbol/ Add/Mul/Pow) becomes tensor.size(d) arithmetic; DerivedSymintRepro evaluates it INSIDE the traced forward and the compiled callable takes only the kept args. Kernel-faithful: the inner region receives real SymInts coupled to the marked dims — the structure the enclosing model provided. Underivable symints pass through as ints with a loud note. Wired through kernel counting, the compile-once artifact, warm calls and timed rows; static/compile_fx paths untouched. Verified: the previously-fatal family runs at EVERY binding (incl. 2 and 4), persistent kernel survives (batch-dyn ratios 0.76-1.26x vs static — no dynamism penalty until the mild large-batch drift), warm order still irrelevant; symint-free families bypass the wrapper (regression-checked). CPU unit test for plan + wrapper equivalence. Pilot doc updated with the GroupNorm row and the fix. written with claude code
Owner
Author
|
Gap closed (9596431 + 6b7c97b): symint-INPUT families can now run the Mechanism: the standalone repro lifts the model's Verified on GPU both ways: |
eellison
changed the title
…artifacts) The always-derive version changed inductor's fusion for symint-VALUE families: with mul=s0*s1 computed inside the traced forward instead of arriving as a graph-input placeholder, var_mean fused into 1 kernel vs the model's 2 — the same unfaithfulness as compile_fx-direct, caught by test_dynamic_default_measures_general_kernel_gpu. Now the --dynamic bench builds the raw-int artifact first (faithful default: that is how the model's compiled unit received symints) and only on ConstraintViolationError (int arg specialized -> pinned a marked dim) rebuilds with the derivation wrapper, announcing the substitution. Verified both ways on GPU: the invariant test measures the model's 2-kernel artifact again, and the pin-hazard GroupNorm family still runs at every binding via the announced fallback with matching numbers. written with claude code
eellison
force-pushed
the
dynamic-shapes-capture
branch
from
August 18, 2026 04:30
6b7c97b to
fc8ae93
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds dynamic-shape support to the capture → merge → bench → oracle pipeline. A dynamic graph is captured once as a family (symbolic structure: symbols, guards, expr-preserving shapes/strides) with concrete points (bindings), so one artifact and one oracle serve every shape in the family.
repros/diff is empty). Recapturing the corpus is a deliberate, separate future PR (dynamic-shapes-corpusbranch holds a validated draft).repro.py/guards.--bind(static per point) /--dynamic(one general artifact,mark_dynamic+ two-shape pre-warm, recompile detection) /--prewarm(explicit warm order for compile-history experiments). Symint inputs re-derive from source tensors only as a loud fallback when raw ints hard-fail.shape_hash; dispatch routes by hash, static behavior untouched.T()/S()shape configs) are gated by structural AST allowlists — corpus artifacts are data, never code.Validated three ways: synthetic unit tests (393 passing; the 3 failures are pre-existing and unrelated to this code — 2
scatter_reducetests need atorch._inductor.fx_passes.scatter_reduce_fusionmodule absent in this env, and 1 manifest-integrity test scansrepros/corpus state, which this code-only PR never touches), re-traced real opacus artifacts, and live captures. Perf pilots (committed underinvestigation_results/): batch-dynamic is ~free (persistent kernels survive when the reduction numel stays static); dynamic reduction dims pay 1.4–5×; warm order/first-seen shape doesn't matter; batch=1-first is safe (0/1 rule).For agents (detailed map)
Data model (
shapes.json): top-levelsymbols({name: {hint, range}}) andguards(expr strings) are graph-level, shared across points; each point carriesshape_hash, compactinputs(expr strings in symbolic slots:['I', hint, expr]symints,"s0"/"64*s0*s1"dims/strides),bindings,captured_dynamic, per-modeloccurrences. Symbol names are canonicalized tos0, s1, …at save (first appearance across input slots, then guards, then table leftovers) so recaptures serialize identically.Three identities, three scopes:
pattern_hash(capture_hook): ops+wiring, shape-blind — names the pattern namespace only._entry_family_identity/_dir_family_identity): hint-blind forward body (AnnAssign shape annotations stripped — they render the binding) + hint-free canonical input signature (a pointwise body references no shapes, so[64,128,s0,s1]vs[64,s0,s1,s2]is only visible there). A dynamic capture joins a dir only on exact match; elsedir__2/__3…; the first family keeps the plain name so oracle-bearing dirs stay put; static dirs are never joined.shape_hash(capture_hookshape_hash_for_placeholders): concrete shapes/strides/dtypes + a canonical-symbol expr suffix for symbolic placeholders. Point identity AND the per-graph occurrence/dedup key. Static hash inputs are byte-identical to the historical scheme (test-pinned).Bench semantics (
repro_harness._run_bound_benchmark): static mode = fresh compile per binding. Dynamic mode = ONE artifact; marks exactly the recorded symbolic dims (dynamic_dims_for_repro— blanketdynamic=Trueover-dynamizes and measures the wrong kernel); two-distinct-shape pre-warm forces the general kernel (--prewarmoverrides order; first warm binding = inductor's tuning hint); per-row recompile detection flags any binding that escaped the warm. Symint inputs: raw ints are the faithful default (the model's compiled unit received symints as placeholders); onConstraintViolationError(int arg specialized →infer_sizepins a marked dim) it rebuilds viasymint_derivations_for_repro+DerivedSymintRepro(symint =tensor.size(d)arithmetic inside the traced forward, closed grammar), announcing the substitution. Deriving unconditionally changes inductor fusion for symint-VALUE families (1 kernel vs the model's 2 — the compile_fx lesson), pinned bytest_dynamic_default_measures_general_kernel_gpu.Other load-bearing details:
--dynamic-mode compile_fxexists but is diagnostic-only (not kernel-faithful; seeinvestigation_results/compile_fx_direct_finding.md).repro_harnessat RUN time:parents[3](in-repo install) first, then a bounded upward walk from the repro's own__file__for the directory containingrepro_harness.py. This replaced an earlier fallback that baked the generator's absolute path into every artifact — that path is the (often ephemeral worktree) checkout the repro was captured from and need not exist at run time; without a co-located harness a repro merged into a scratch root silently imported a stale checkout's copy (benched one pilot run with wrong flags).scripts/ingest_tlparse.load_graph_module) seeds symbol hints from the sidecarfull_graph_XXX.meta.json(_sidecar_symbol_hints); annotations areSym(s16)and carry no hint, and a default hint corrupts bindings + shape_hash (the accounting join key). Re-trace ShapeEnvs genuinely carry no guards (make_fx can't re-derive model-context dynamo guards; live capture preserves them end-to-end).input_codec._assert_safe_expr/_assert_safe_shape_configare the two injection gates; both fail closed and are wired at every eval site.scripts/exercise_dynamic_shapes_live.py(7 live families incl. a family-split pair, scratch-only) andscripts/pilot_single_dim_dynamic.py(single-dim dynamism + batch=1 probe). Findings + raw data:investigation_results/dynamic_vs_static_eval_points_2026-08-12.md,investigation_results/single_dim_dynamic_pilot_2026-08-12.md.Latest review round (hardening, this revision) — turning silent failure modes loud, no behavior change on the happy path:
--prewarmnow errors instead of being silently ignored where it can't apply: the default per-shape path and static mode (no single artifact to warm) and--dynamic-mode compile_fx(no dynamo warm loop) all fail loudly before any GPU/config work; the blanket-dynamic=Trueauto-fallback (no recorded symbolic dims) warns that prewarm had no effect. Parse errors in a--prewarmvalue now name--prewarm, not--bind(sharedsymbol=intgrammar)._sidecar_symbol_hints) rejectsbool(an int subclass that would seed a 0/1-specializing size) and losslessly coerces an integral float; a bad-typed hint no longer silently drops to the default and orphans the accounting join.info["point"]/["matched_point"]) — a point-tier match against a shape-less dynamic entry is otherwise unexplained._generate_repro_filedetects whenshape_param_exprs(extraction-time_shape_param_Nnames) desyncs from the re-lifted canonical names and warns that those dims will serialize static (dynamism lost), instead of dropping the overlay silently.__2/__3) allocation as single-process-only (sequentialmkdirbefore the next entry); concurrent merges into one canonical root must shard or serialize.tests/test_prewarm_guards.py(7), sidecar-hint coercion (4), dispatch-point recording, plus the runtime-relative resolution check.Known follow-ups (non-blocking, documented): corpus recapture as its own PR (regenerates with the new point hashes); sidecar guard inheritance at re-trace (nothing to exercise until new captures exist); persistent-reduction dispatch for dynamic reduction dims (would close most of the 1.4–5× gap); persisting the binding sweep list + a per-binding
{general_us, specialized_us}reference column in perf.json.🤖 Generated with Claude Code