fix: harden torch.compile support against real-hardware failure modes - #22
Merged
Adithya-Thonse merged 17 commits intoAug 6, 2026
Merged
Conversation
tinyml-tinyverse's own test suite doesn't exist yet at this point in history (added by the next commit) -- this adds only the path trigger. The step that actually runs those tests is added once they exist, later in this branch, so no commit in this sequence ever references a directory that isn't there yet. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…all back to eager mode torch.compile() is lazy — the previous try/except only wrapped the wrap-time call, not the deferred compilation that happens on first forward. A failing compile (e.g. unsupported Triton/ptxas for the GPU's compute capability) would crash mid-training instead of falling back to eager mode. compile_model_if_enabled now accepts an optional input_shape and, when provided, runs one warmup forward pass through the compiled model before training starts. A failure at either the wrap step or the warmup step falls back to the original, uncompiled model. Backward compatible: omitting input_shape preserves the old (unguarded) behavior. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
torch.compile's OptimizedModule wraps the original model by reference, so calling .eval() on the compiled wrapper during warmup also flips the ORIGINAL model's .training flag. If the warmup forward pass raised, the restore step was skipped (exception jumped straight to the outer except), so the fallback model was returned stuck in eval mode. Wrap the eval/forward/restore sequence in try/finally so restoration always runs. Also strengthens test_warmup_failure_falls_back_to_original_model: the old mock returned an unrelated standalone module that didn't share state/reference with the original model, so it couldn't reproduce this bug. The new version wraps the original model as a real child submodule (mirroring OptimizedModule._orig_mod) and asserts the original model's .training flag is restored after a failed warmup — confirmed this fails against the old buggy code and passes against the fix. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…e training-mode graph
torch.compile() wraps a model in torch._dynamo.OptimizedModule, exposing the original module at ._orig_mod. Neither torch.jit.trace (quantized export) nor torch.onnx.export (float export) can trace a dynamo-optimized module directly, so export_model() crashed with "Detected that you are using FX to torch.jit.trace a dynamo-optimized function" whenever compile_model=1 actually succeeded (the hardware-defaults feature auto-enables this on CUDA). Unwrap via getattr(model, '_orig_mod', model) once before the existing deepcopy — a no-op for uncompiled models.
…op-level Single-level getattr(model, '_orig_mod', model) missed the actual failure shape: timeseries_classification always wraps the (already-compiled) model inside NeuralNetworkWithPreprocess.model after compile_model_if_enabled runs, so the compiled OptimizedModule ends up one level below the top-level model, not at it. torch.onnx.export/torch.jit.trace still crashed on the nested compiled submodule. unwrap_compiled_submodules() walks the full submodule tree recursively, replacing every torch.compile-wrapped module (top-level or nested) with its original. No-op wherever nothing is compiled. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…'t handle a compiled model prepare_qat_fx (FX symbolic tracing) crashes with "Detected that you are using FX to symbolically trace a dynamo-optimized function" when the model passed to it is torch.compile-wrapped (torch._dynamo.OptimizedModule). compile_model_if_enabled now skips compiling entirely whenever args.quantization is set, since compiling first would be discarded before quantization prep runs anyway. Float training (quantization=0) is unaffected. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
torch.compile() wraps a model in torch._dynamo.OptimizedModule. Since setup_distributed_model only assigns model_without_ddp = model.module under DDP, in the non-DDP case model_without_ddp IS the compiled wrapper, so state_dict() emits every key prefixed _orig_mod. The downstream float->quantization weight transfer (load_weights.py) can't match those keys, silently falls back to strict=False, and discards the entire float-trained result -- no exception, no fatal error, the pipeline just quietly retrains from random init and reports success. save_checkpoint and resume_from_checkpoint now unwrap via a local getattr(model, '_orig_mod', model) at each call site, so checkpoints always carry uncompiled key names. This intentionally does not use the existing unwrap_compiled_submodules() helper, which mutates the model's submodule tree in place via setattr and would silently un-compile the live training model. Also adds weights_only=False to resume_from_checkpoint's torch.load call. This is a separate, pre-existing latent bug this fix's own test suite surfaced: checkpoint['args'] is an argparse.Namespace (every train.py passes the full args object to save_checkpoint), which is not on torch's default weights_only safe-globals list, so on torch >=2.6 the --resume path already fails before reaching load_state_dict, independent of the compile-wrapper issue. Matches the weights_only=False convention already used at every other non-tensor-only torch.load() call site in this codebase (load_weights.py, and the load_saved_model paths in each task's train.py). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ave+load ExponentialMovingAverage (AveragedModel) deep-copies its source model into self.module, so when the source was already compiled, the OptimizedModule wrapper ends up nested at model_ema.module._orig_mod -- not at model_ema._orig_mod itself, where the prior fix's top-level getattr unwrap couldn't reach it. Strip the substring from the resulting state_dict keys instead (handles any nesting depth), and remap symmetrically on load by matching against model_ema's own current key names stripped the same way. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…l production shape
tinyml-tinyverse/tests/ (added in this branch's earlier commits) was never executed by CI. No extra install step needed -- tinyml-tinyverse is already pip installed in the shared Install dependencies step. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
save_checkpoint/resume_from_checkpoint (train_base.py) already stop writing _orig_mod.-prefixed keys going forward, but load_weights() -- the actual consumer for the float->quantization --weights transfer (timeseries_base.py) -- had no handling for the prefix at all, only for 'module.' (DDP). A checkpoint saved by older, unpatched code, or by any other torch.compile-using caller not covered by the train_base.py fix, would still hit the original silent strict=False fallback that discards 100% of weights with no exception. Strip _orig_mod. from incoming checkpoint data before the existing 'module.' realignment runs (it's never meaningful to preserve or match against, unlike 'module.'), and symmetrically remap onto the live model's own current key names if that model is itself currently compiled -- mirroring the same symmetric approach already used for the EMA checkpoint path. Also adds a regression test that round-trips a checkpoint through the actual load_weights() consumer (not just direct load_state_dict), closing the gap noted in review: the fix was previously verified against this consumer only via ad-hoc manual testing, not a permanent test. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ompat Two independent bugs in the same function, found by adversarial security and code review: 1. Security (unsafe deserialization): the weights_only=False added to torch.load(args.resume, ...) disabled torch >=2.6's default unpickling restriction wholesale, rather than allowlisting only the one non-tensor type the checkpoint actually needs (argparse.Namespace, for checkpoint['args']). A crafted --resume checkpoint from an untrusted source could execute arbitrary code via a pickle __reduce__ payload. Fixed via torch.serialization.safe_globals([Namespace]). 2. Backward compatibility: the prior fix's unwrap only touched the live model, not the checkpoint data, so it silently assumed checkpoint['model'] always has clean (unprefixed) keys. A checkpoint written before this session's fixes existed -- with _orig_mod. keys, from a compiled model -- would now fail to load into an unwrapped model with a strict key mismatch, where it used to work by accident (both sides had matching prefixes). Fixed by extracting the same "strip from data, then remap onto the live model's actual current keys" pattern already used for the EMA branch into a shared _load_symmetric() helper, applied to both the main model and EMA uniformly. Also strengthens existing checkpoint tests to assert on .bias and full parameter sets, not just .weight -- a bug that only corrupted bias handling would have passed undetected. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Tests that do a genuine torch.save + resume_from_checkpoint (torch.load) round trip were passing _FakeArgs (a lightweight test stand-in) as the args parameter to save_checkpoint, which becomes checkpoint['args'] and must therefore actually be unpickled by torch.load's weights_only safety check. Only argparse.Namespace is allowlisted (matching real production usage, where args always comes from ArgumentParser.parse_args()), so these tests started failing once that check was genuinely enforced. Switched to a real Namespace() for exactly the calls that get serialized to disk; _FakeArgs remains fine for args objects that are only ever read directly as a live function parameter (.resume/.device), never pickled. Also moves the pickle-rejection test's "not on the allowlist" stand-in class to module level -- pickle cannot serialize local/nested classes at all, which made that test fail for an unrelated reason before it ever reached the weights_only check it's meant to exercise. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Contributor
Author
|
CI failure investigated — pre-existing on Verified directly against This PR's CI run shows only that same subset — no new failures introduced. It's fixed in #19; should resolve automatically once that merges. (Windows failures don't block merging — the workflow marks that runner |
This was referenced Jul 30, 2026
# Conflicts: # .github/workflows/test-modelmaker.yml
musicalplatypus
force-pushed
the
pr/compile-hardening
branch
from
August 6, 2026 04:39
d657ee9 to
58b5089
Compare
musicalplatypus
pushed a commit
to musicalplatypus/tinyml-tensorlab
that referenced
this pull request
Aug 6, 2026
…on run An independent peer review of the "does the merge order between TexasInstruments#21 and TexasInstruments#22 matter" question -- specifically auditing whether their disjoint edits to utils.py and this file were semantically safe together -- found a real, independent bug already present in this PR, unrelated to any merge interaction. This PR's own earlier fix gated non_blocking transfers on `device.type == 'cuda'` in get_reconstruction_errors_stats() (matching the same fix applied to evaluate_classification et al. elsewhere). Before that, non_blocking was hardcoded True and the function never touched `.type`, so passing a plain device string worked fine. After the fix, `.type` is required -- but the sole call site (main(), calculating the anomaly-detection threshold right after export) passed `args.device`, the raw unconverted argparse string ('cuda' by default), instead of `device`, the torch.device already constructed by setup_training_environment() and in scope in the very same function. `'cuda'.type` raises AttributeError, so this crashed on every anomaly-detection training run -- CUDA included, not an edge case, the normal path right after training completes. Fixed by passing the existing local `device` instead of `args.device` at the call site. Adds tests/test_anomalydetection_train_device_crash.py: two tests characterize get_reconstruction_errors_stats()'s contract directly (works with a torch.device, crashes with a raw string -- exactly reproducing the pre-fix symptom). A third test drives the real main() (heavily mocked elsewhere, zero-iteration training loop) and asserts what it actually passes as the device argument at the real call site -- this is the one that genuinely exercises the fixed line, since the first two would pass identically regardless of whether the call site itself were fixed. Verified it fails pre-fix (main() passed the raw string 'cuda') and passes post-fix. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
torch.compileauto-enables on CUDA (apply_hardware_defaults, a separate PR), but hitting it on real hardware surfaced several failure modes not visible from CPU-only testing: silent compile failures that never fell back, wrong keys in saved/loaded checkpoints, quantization incompatibility, and export crashes. Fixed all of them with regression tests.torch.compile validation and fallback
torch.compileis lazy — a bad compile only surfaces on the first real forward call, not attorch.compile(model)time. Without a warmup pass, a compile failure during actual training would crash the run instead of falling back to eager mode. Added a warmup forward pass (with the correctinput_shape, now threaded through all 4 call sites) inside try/except so failures genuinely fall back; the model's training mode is restored unconditionally afterward (a warmup pass run in eval mode was leaking that state into the real training loop).Skip compile under quantization
FX-based quantization (
prepare_qat_fx) cannot trace atorch.compile-wrapped model — it needs the realnn.Modulegraph structure, not Dynamo's compiled wrapper.compile_model_if_enablednow skips compilation entirely when quantization is enabled, with a log line explaining why, rather than letting FX tracing fail deep in the quantization pipeline with a confusing error.Checkpoint / export correctness under torch.compile
torch.compilewraps a model asOptimizedModule, whosestate_dict()keys are prefixed with_orig_mod.. This broke three separate things:save_checkpointnow unwraps the model (and, symmetrically, the EMA model — whose compiled wrapper nests one level deeper, atmodel_ema.module._orig_mod) before calling.state_dict(), so checkpoints no longer carry the_orig_mod.prefix going forward.resume_from_checkpointstrips_orig_mod.from loaded checkpoint state (for compatibility with older, already-written checkpoints) and symmetrically remaps onto the live model's own current keys if it's currently compiled. Also fixes a latent unsafe-deserialization issue:torch.load(..., weights_only=False)disabled PyTorch's deserialization safety check entirely — replaced withtorch.serialization.safe_globals([Namespace]), an explicit allowlist for the one non-tensor type actually stored (args).load_weights()(float→quantization weight transfer): had no_orig_mod.handling at all — a checkpoint saved by older code, or written by any othertorch.compile-using caller, would silently discard 100% of its weights via the existingstrict=Falsefallback with no exception raised. Added the same_orig_mod.stripping as defense-in-depth.export_model()now recursively unwraps all compiled submodules (not just a top-level one) beforecopy.deepcopy(model)and ONNX/TorchScript export — a compiled submodule nested inside an otherwise-uncompiled model previously crashed export.CI
Wires
tinyml-tinyverse's test suite into CI (previously untriggered and unrun).Verification
All of the above were found and fixed against real CUDA hardware (Blackwell GB10), not just CPU-only unit tests — the compile-warmup-fallback fix in particular was validated against a real
ptxas/Triton codegen failure specific to that GPU generation, confirming the fallback path actually engages and training completes successfully in eager mode when compilation itself fails.🤖 Generated with Claude Code