Conversation
…torchao and quanto
…torchao and quanto
ConvRot documentation
cosmos3: switch quant backend in examples to SDNQ ConvRot config
…flux2 flux2: switch quant backend to use SDNQ int8 ConvRot instead of int8-torchao and quanto
…krea2 krea2: switch quant backend to use SDNQ int8 ConvRot instead of int8-torchao
…zimage-turbo z-image turbo: switch quant backend to use SDNQ int8 ConvRot
convrot: add a hook to run post-quant custom setup when needed
krea2: ConvRot safetensors loader
z-image: ConvRot safetensors loader
LTX 2.3: SDNQ Hadamard (ConvRot) example config
SDNQ: use v0.2.2 and compat-checked patch for smart checkpointing
FSDP2 gradients are DTensors, so rank-local maxima can under-report the logical gradient, empty local shards must still join collectives, and reporting must not receive device tensors. The training loop also measured a max on every accumulation microstep even when norm clipping would immediately recompute and overwrite it. - In Trainer._max_grad_value(), measure non-empty local shards, contribute negative infinity for empty DTensor shards, stage CPU scalars on the mesh device, and all-reduce MAX once across the mesh. - At synchronized optimizer steps, reuse the norm returned by clipping in norm mode and run the separate max path only when clipping does not already provide the reported value. - Preserve existing FSDP1, FSDP2, non-FSDP, optimizer-exclusion, and FSDP value-clipping policies while avoiding duplicate gradient traversal and collectives. - Normalize norm and max values through one scalar path for logs, progress, webhooks, and regularisation metrics.
Current Accelerate calls the FSDP2 full-state loader with a cpu_offload argument, while older callers use the previous three-argument form. The local override handled neither combination safely under an active offload policy. - Extend the replacement loader signature with an optional cpu_offload value so both current and legacy call forms remain valid. - When the argument is omitted, infer placement from the active CPUOffloadPolicy; explicit true and false continue to override that policy. - Apply the resolved placement to sharded and unsharded tensors on main and non-main ranks before load_state_dict(assign=True). This keeps parameter placement consistent across the complete FSDP2 model. - Keep CPU placement independent of RAM-efficient loading because offload is an FSDP2 runtime policy rather than a loader memory-mode side effect.
Restore the clone_norm_value keyword used by existing callers and keep value-mode grad_absmax entries as the original metric object. Continue converting norm metrics to scalars only for progress payloads that explicitly request cloning. This keeps the FSDP2 mesh-wide max reduction and synchronized clipping changes while preserving the pre-existing logging API and tensor identity behavior exercised by the upstream trainer suite.
mage-flow: fix issue with rotary helper
…26065 Add focused FSDP2 export coverage for save-hook preflight and cross-rank failures
mage-flow: add FFN checkpointing logic
convrot: add loss comparison charts (Z-Image, Mage Flow)
z-image: switch group size for sdnq to 256 to enable convrot correctly
krea2: update docs and example config to recommend cudnn backend
add torch-ffn and unsloth-ffn gradient checkpointing modes for speed-up
ConvRot: documentation additions and corrections (group size should be 256)
Fix CPU-offload tensor placement in the FSDP2 full-state loader
Fix FSDP2 gradient norm/max reporting and clipping
…e mark of the beast)
…ot/fix-code-review-comment
FSDP2: handle footgun as a rejection case in pytorch issue 180666 (the mark of the beast)
Normalize cloned `grad_absmax` metrics for structured progress payloads
Fix per-rank sampler state resume path
Fix deterministic distributed bucket splitting
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.