Skip to content

fix: harden torch.compile support against real-hardware failure modes - #22

Merged
Adithya-Thonse merged 17 commits into
TexasInstruments:mainfrom
musicalplatypus:pr/compile-hardening
Aug 6, 2026
Merged

fix: harden torch.compile support against real-hardware failure modes#22
Adithya-Thonse merged 17 commits into
TexasInstruments:mainfrom
musicalplatypus:pr/compile-hardening

Conversation

@musicalplatypus

Copy link
Copy Markdown
Contributor

Summary

torch.compile auto-enables on CUDA (apply_hardware_defaults, a separate PR), but hitting it on real hardware surfaced several failure modes not visible from CPU-only testing: silent compile failures that never fell back, wrong keys in saved/loaded checkpoints, quantization incompatibility, and export crashes. Fixed all of them with regression tests.

torch.compile validation and fallback

torch.compile is lazy — a bad compile only surfaces on the first real forward call, not at torch.compile(model) time. Without a warmup pass, a compile failure during actual training would crash the run instead of falling back to eager mode. Added a warmup forward pass (with the correct input_shape, now threaded through all 4 call sites) inside try/except so failures genuinely fall back; the model's training mode is restored unconditionally afterward (a warmup pass run in eval mode was leaking that state into the real training loop).

Skip compile under quantization

FX-based quantization (prepare_qat_fx) cannot trace a torch.compile-wrapped model — it needs the real nn.Module graph structure, not Dynamo's compiled wrapper. compile_model_if_enabled now skips compilation entirely when quantization is enabled, with a log line explaining why, rather than letting FX tracing fail deep in the quantization pipeline with a confusing error.

Checkpoint / export correctness under torch.compile

torch.compile wraps a model as OptimizedModule, whose state_dict() keys are prefixed with _orig_mod.. This broke three separate things:

  • Checkpoint save: save_checkpoint now unwraps the model (and, symmetrically, the EMA model — whose compiled wrapper nests one level deeper, at model_ema.module._orig_mod) before calling .state_dict(), so checkpoints no longer carry the _orig_mod. prefix going forward.
  • Checkpoint resume: resume_from_checkpoint strips _orig_mod. from loaded checkpoint state (for compatibility with older, already-written checkpoints) and symmetrically remaps onto the live model's own current keys if it's currently compiled. Also fixes a latent unsafe-deserialization issue: torch.load(..., weights_only=False) disabled PyTorch's deserialization safety check entirely — replaced with torch.serialization.safe_globals([Namespace]), an explicit allowlist for the one non-tensor type actually stored (args).
  • load_weights() (float→quantization weight transfer): had no _orig_mod. handling at all — a checkpoint saved by older code, or written by any other torch.compile-using caller, would silently discard 100% of its weights via the existing strict=False fallback with no exception raised. Added the same _orig_mod. stripping as defense-in-depth.
  • Export: export_model() now recursively unwraps all compiled submodules (not just a top-level one) before copy.deepcopy(model) and ONNX/TorchScript export — a compiled submodule nested inside an otherwise-uncompiled model previously crashed export.

CI

Wires tinyml-tinyverse's test suite into CI (previously untriggered and unrun).

Verification

All of the above were found and fixed against real CUDA hardware (Blackwell GB10), not just CPU-only unit tests — the compile-warmup-fallback fix in particular was validated against a real ptxas/Triton codegen failure specific to that GPU generation, confirming the fallback path actually engages and training completes successfully in eager mode when compilation itself fails.

🤖 Generated with Claude Code

t5fkg8d44d-beep and others added 16 commits July 29, 2026 11:17
tinyml-tinyverse's own test suite doesn't exist yet at this point in
history (added by the next commit) -- this adds only the path trigger.
The step that actually runs those tests is added once they exist, later
in this branch, so no commit in this sequence ever references a directory
that isn't there yet.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…all back to eager mode

torch.compile() is lazy — the previous try/except only wrapped the wrap-time
call, not the deferred compilation that happens on first forward. A failing
compile (e.g. unsupported Triton/ptxas for the GPU's compute capability)
would crash mid-training instead of falling back to eager mode.

compile_model_if_enabled now accepts an optional input_shape and, when
provided, runs one warmup forward pass through the compiled model before
training starts. A failure at either the wrap step or the warmup step falls
back to the original, uncompiled model. Backward compatible: omitting
input_shape preserves the old (unguarded) behavior.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
torch.compile's OptimizedModule wraps the original model by reference,
so calling .eval() on the compiled wrapper during warmup also flips the
ORIGINAL model's .training flag. If the warmup forward pass raised, the
restore step was skipped (exception jumped straight to the outer except),
so the fallback model was returned stuck in eval mode. Wrap the
eval/forward/restore sequence in try/finally so restoration always runs.

Also strengthens test_warmup_failure_falls_back_to_original_model: the
old mock returned an unrelated standalone module that didn't share
state/reference with the original model, so it couldn't reproduce this
bug. The new version wraps the original model as a real child submodule
(mirroring OptimizedModule._orig_mod) and asserts the original model's
.training flag is restored after a failed warmup — confirmed this fails
against the old buggy code and passes against the fix.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
torch.compile() wraps a model in torch._dynamo.OptimizedModule, exposing
the original module at ._orig_mod. Neither torch.jit.trace (quantized
export) nor torch.onnx.export (float export) can trace a dynamo-optimized
module directly, so export_model() crashed with "Detected that you are
using FX to torch.jit.trace a dynamo-optimized function" whenever
compile_model=1 actually succeeded (the hardware-defaults feature
auto-enables this on CUDA). Unwrap via getattr(model, '_orig_mod', model)
once before the existing deepcopy — a no-op for uncompiled models.
…op-level

Single-level getattr(model, '_orig_mod', model) missed the actual failure
shape: timeseries_classification always wraps the (already-compiled) model
inside NeuralNetworkWithPreprocess.model after compile_model_if_enabled runs,
so the compiled OptimizedModule ends up one level below the top-level model,
not at it. torch.onnx.export/torch.jit.trace still crashed on the nested
compiled submodule.

unwrap_compiled_submodules() walks the full submodule tree recursively,
replacing every torch.compile-wrapped module (top-level or nested) with its
original. No-op wherever nothing is compiled.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…'t handle a compiled model

prepare_qat_fx (FX symbolic tracing) crashes with "Detected that you are
using FX to symbolically trace a dynamo-optimized function" when the model
passed to it is torch.compile-wrapped (torch._dynamo.OptimizedModule).
compile_model_if_enabled now skips compiling entirely whenever
args.quantization is set, since compiling first would be discarded before
quantization prep runs anyway. Float training (quantization=0) is
unaffected.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
torch.compile() wraps a model in torch._dynamo.OptimizedModule. Since
setup_distributed_model only assigns model_without_ddp = model.module
under DDP, in the non-DDP case model_without_ddp IS the compiled
wrapper, so state_dict() emits every key prefixed _orig_mod. The
downstream float->quantization weight transfer (load_weights.py)
can't match those keys, silently falls back to strict=False, and
discards the entire float-trained result -- no exception, no fatal
error, the pipeline just quietly retrains from random init and
reports success.

save_checkpoint and resume_from_checkpoint now unwrap via a local
getattr(model, '_orig_mod', model) at each call site, so checkpoints
always carry uncompiled key names. This intentionally does not use
the existing unwrap_compiled_submodules() helper, which mutates the
model's submodule tree in place via setattr and would silently
un-compile the live training model.

Also adds weights_only=False to resume_from_checkpoint's torch.load
call. This is a separate, pre-existing latent bug this fix's own test
suite surfaced: checkpoint['args'] is an argparse.Namespace (every
train.py passes the full args object to save_checkpoint), which is
not on torch's default weights_only safe-globals list, so on torch
>=2.6 the --resume path already fails before reaching load_state_dict,
independent of the compile-wrapper issue. Matches the weights_only=False
convention already used at every other non-tensor-only torch.load()
call site in this codebase (load_weights.py, and the load_saved_model
paths in each task's train.py).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ave+load

ExponentialMovingAverage (AveragedModel) deep-copies its source model into
self.module, so when the source was already compiled, the OptimizedModule
wrapper ends up nested at model_ema.module._orig_mod -- not at
model_ema._orig_mod itself, where the prior fix's top-level getattr unwrap
couldn't reach it. Strip the substring from the resulting state_dict keys
instead (handles any nesting depth), and remap symmetrically on load by
matching against model_ema's own current key names stripped the same way.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
tinyml-tinyverse/tests/ (added in this branch's earlier commits) was never
executed by CI. No extra install step needed -- tinyml-tinyverse is already
pip installed in the shared Install dependencies step.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
save_checkpoint/resume_from_checkpoint (train_base.py) already stop writing
_orig_mod.-prefixed keys going forward, but load_weights() -- the actual
consumer for the float->quantization --weights transfer
(timeseries_base.py) -- had no handling for the prefix at all, only for
'module.' (DDP). A checkpoint saved by older, unpatched code, or by any
other torch.compile-using caller not covered by the train_base.py fix,
would still hit the original silent strict=False fallback that discards
100% of weights with no exception.

Strip _orig_mod. from incoming checkpoint data before the existing
'module.' realignment runs (it's never meaningful to preserve or match
against, unlike 'module.'), and symmetrically remap onto the live model's
own current key names if that model is itself currently compiled --
mirroring the same symmetric approach already used for the EMA checkpoint
path.

Also adds a regression test that round-trips a checkpoint through the
actual load_weights() consumer (not just direct load_state_dict), closing
the gap noted in review: the fix was previously verified against this
consumer only via ad-hoc manual testing, not a permanent test.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ompat

Two independent bugs in the same function, found by adversarial security
and code review:

1. Security (unsafe deserialization): the weights_only=False added to
   torch.load(args.resume, ...) disabled torch >=2.6's default unpickling
   restriction wholesale, rather than allowlisting only the one non-tensor
   type the checkpoint actually needs (argparse.Namespace, for
   checkpoint['args']). A crafted --resume checkpoint from an untrusted
   source could execute arbitrary code via a pickle __reduce__ payload.
   Fixed via torch.serialization.safe_globals([Namespace]).

2. Backward compatibility: the prior fix's unwrap only touched the live
   model, not the checkpoint data, so it silently assumed checkpoint['model']
   always has clean (unprefixed) keys. A checkpoint written before this
   session's fixes existed -- with _orig_mod. keys, from a compiled model --
   would now fail to load into an unwrapped model with a strict key
   mismatch, where it used to work by accident (both sides had matching
   prefixes). Fixed by extracting the same "strip from data, then remap
   onto the live model's actual current keys" pattern already used for the
   EMA branch into a shared _load_symmetric() helper, applied to both the
   main model and EMA uniformly.

Also strengthens existing checkpoint tests to assert on .bias and full
parameter sets, not just .weight -- a bug that only corrupted bias handling
would have passed undetected.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Tests that do a genuine torch.save + resume_from_checkpoint (torch.load)
round trip were passing _FakeArgs (a lightweight test stand-in) as the
args parameter to save_checkpoint, which becomes checkpoint['args'] and
must therefore actually be unpickled by torch.load's weights_only safety
check. Only argparse.Namespace is allowlisted (matching real production
usage, where args always comes from ArgumentParser.parse_args()), so these
tests started failing once that check was genuinely enforced. Switched to
a real Namespace() for exactly the calls that get serialized to disk;
_FakeArgs remains fine for args objects that are only ever read directly
as a live function parameter (.resume/.device), never pickled.

Also moves the pickle-rejection test's "not on the allowlist" stand-in
class to module level -- pickle cannot serialize local/nested classes at
all, which made that test fail for an unrelated reason before it ever
reached the weights_only check it's meant to exercise.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@musicalplatypus

Copy link
Copy Markdown
Contributor Author

CI failure investigated — pre-existing on upstream/main, unrelated to this PR.

Verified directly against upstream/main's own most recent CI run (same base commit, 331388a): https://github.com/TexasInstruments/tinyml-tensorlab/actions/runs/30376872256 — it already fails identically on macOS + Ubuntu with:

FAILED test_config_validation.py::...test_task_type_is_valid[google_speech_command/config_MSPM0.yaml] - unknown task_type='audio_classification'
FAILED test_config_validation.py::...test_model_name_exists_in_registry[...] - model 'DSCNN_NPU' not in registry

This PR's CI run shows only that same subset — no new failures introduced. It's fixed in #19; should resolve automatically once that merges. (Windows failures don't block merging — the workflow marks that runner continue-on-error: true.)

# Conflicts:
#	.github/workflows/test-modelmaker.yml
musicalplatypus pushed a commit to musicalplatypus/tinyml-tensorlab that referenced this pull request Aug 6, 2026
…on run

An independent peer review of the "does the merge order between TexasInstruments#21 and
TexasInstruments#22 matter" question -- specifically auditing whether their disjoint
edits to utils.py and this file were semantically safe together -- found
a real, independent bug already present in this PR, unrelated to any
merge interaction.

This PR's own earlier fix gated non_blocking transfers on
`device.type == 'cuda'` in get_reconstruction_errors_stats() (matching
the same fix applied to evaluate_classification et al. elsewhere). Before
that, non_blocking was hardcoded True and the function never touched
`.type`, so passing a plain device string worked fine. After the fix,
`.type` is required -- but the sole call site (main(), calculating the
anomaly-detection threshold right after export) passed `args.device`,
the raw unconverted argparse string ('cuda' by default), instead of
`device`, the torch.device already constructed by
setup_training_environment() and in scope in the very same function.
`'cuda'.type` raises AttributeError, so this crashed on every
anomaly-detection training run -- CUDA included, not an edge case, the
normal path right after training completes.

Fixed by passing the existing local `device` instead of `args.device` at
the call site.

Adds tests/test_anomalydetection_train_device_crash.py: two tests
characterize get_reconstruction_errors_stats()'s contract directly (works
with a torch.device, crashes with a raw string -- exactly reproducing the
pre-fix symptom). A third test drives the real main() (heavily mocked
elsewhere, zero-iteration training loop) and asserts what it actually
passes as the device argument at the real call site -- this is the one
that genuinely exercises the fixed line, since the first two would pass
identically regardless of whether the call site itself were fixed.
Verified it fails pre-fix (main() passed the raw string 'cuda') and
passes post-fix.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@Adithya-Thonse
Adithya-Thonse merged commit dc8eeb2 into TexasInstruments:main Aug 6, 2026
0 of 3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants