Add int4 w4a4 convrot support - #55
Draft
rockerBOO wants to merge 9 commits into
Draft
Conversation
Implements row-wise INT4 symmetric quantization with 2-per-byte packing in row-major order, plus unpack and dequantize operations. All functions are pure PyTorch with no external dependencies. Includes comprehensive tests for quantization correctness, edge cases (zero rows), pack/unpack roundtrip, and dequantization accuracy. AI Assisted
Add elif is_int4 branch to fp8_conversion.py save/metadata logic, mirroring the existing int8 branch. Emits weight_scale (per-row) and a comfy_quant tensor with format=int4_convrot_w4a4, convrot=True, convrot_groupsize=<N>. Also add int4 entry to format_block_sizes map.
Adds tests/test_convrot_w4a4_e2e.py covering the full LearnedRoundingConverter int4 path (storage contract, metadata contract, reconstruction quality) plus a parity check against comfy-kitchen's eager reference implementation (skipped unless comfy-kitchen is installed). Extends tests/compare_convrot_error_metrics.py with System E (dynamic ConvRot + INT4 RTN) and System F (dynamic ConvRot + INT4 AdaRound), relocating metric_labels so both blocks can share it. AI Assisted
… ConvRot group size, loosen AdaRound-vs-RTN SNR assertion INT4 ConvRot W4A4 requires rotation (unlike INT8, which can fall back to unrotated quantization), so compare_convrot_error_metrics.py's System E/F block now skips gracefully instead of raising ValueError when no compatible dynamic group size exists for a shape, mirroring the existing dynamic_applied guard used by Systems C/D. Also loosens the SNR assertion with a small tolerance: at 4-bit resolution AdaRound's soft-rounding optimizer has far less room to improve on RTN than at 8-bit, and can be a statistical wash on pathological synthetic inputs. AI Assisted
…nt bug, and docs Fixes findings from whole-branch review of INT4 W4A4 ConvRot: - Finding 1 (critical): fp8_conversion.py's layer_config format mapper had no branch for "int4*", so a --layer-config entry requesting int4_convrot_w4a4 silently fell back to FP8 conversion while still writing int4 metadata, producing a corrupted, mislabeled model. Added the missing elif branch and a regression test (tests/test_layer_config_int4.py) that fails without the fix (asserts float8_e4m3fn output) and passes with it (packed int8 output). - Finding 2 (important): compare_convrot_error_metrics.py's System E/F block computed Y_e/Y_f using X_rot_dynamic, mirroring INT8's Systems C/D. But _convert_int4_convrot_w4a4 un-rotates before returning dequantized weights (unlike INT8's ConvRot path), so the correct activation is X, not X_rot_dynamic. Fixed both computations. Re-verified on two shapes (32x1024, 64x512): SNR went from wildly negative to a sane ~20dB (RTN) and AdaRound now reliably and substantially beats RTN (~30dB), so the SNR assertion tolerance (previously "-0.1" to paper over the measurement bug) is tightened back to a strict >=. - Finding 3: documented the return-space convention divergence in _convert_int4_convrot_w4a4's docstring (un-rotated basis, vs INT8 ConvRot's rotated-basis convention) that caused Finding 2. - Finding 4: documented the ConvRot group-size divisibility requirement for --int4 in the CLI help text and README Quick Start (--dynamic-convrot as the recommended mitigation). No behavior change. - Finding 5: removed unused M from _convert_int4_convrot_w4a4's shape unpack, and removed a redundant self.dynamic_convrot assignment in LearnedRoundingConverter.__init__'s int4 branch (already set unconditionally earlier in __init__). AI Assisted
The comfy_quant metadata blob for INT4 layers was written with
format="int4_convrot_w4a4", but ComfyUI's QUANT_ALGOS registry
(comfy/quant_ops.py) and its loader (comfy/ops.py) key on
"convrot_w4a4" (no int4_ prefix). This mismatch caused
KeyError: 'convrot_w4a4' when loading a quantized checkpoint in
ComfyUI. The layer-config format string ("int4_convrot_w4a4", used
to route layers through create_converter_for_format) is unaffected -
only the value written into the per-layer comfy_quant tensor changes.
AI Assisted
|
@rockerBOO excellent quality speed vram comparison? |
Author
If you wanted to try it, I uploaded it here https://huggingface.co/rockerBOO/minimax-h3-nvfp4/blob/main/minimax_h3_fl2va_pruned_int4_convrot_simple.safetensors there's a few other versions im trying in there too. Not a completed version but shows it working. |
the size is almost same as int8 convrot atm ComfyUI should publish pruned bf16 base looks like |
Author
It is a different format than how the upstream because in this version I'm keeping more at bf16 (block 0,1 and 47,48,49) and only 90 tensors are in int4. So just comparing the size might be misleading. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tried to work similarly to convrot int8 implementation and current testing with minimax h3. The quant does work inside 0.30.0 comfy (I dunno when earlier.)
uv run --with-editable worktree-convrot-int4-w4a4 \ --with-requirements worktree-convrot-int4-w4a4/requirements.txt \ --with comfy-kitchen --with numpy --with scipy \ ctq -i minimax_h3_fl2va_bf16.safetensors \ -o minimax_h3_fl2va_int4_convrot_simple.safetensors \ --int4 --dynamic-convrot --convrot-group-size 64 \ --layer-config minimax_h3_layer_config_int4_convrot.json \ --comfy_quant --save-quant-metadata --simpleminimax_h3_layer_config_int4_convrot.json{ "blocks\\.0\\.|blocks\\.1\\.|blocks\\.47\\.|blocks\\.48\\.|blocks\\.49\\.": {"skip": true}, "token_refiner": {"skip": true}, "condition_proj": {"skip": true}, "final_layer\\.adaln_proj\\.linear|final_layer\\.audio_out|final_layer\\.video_out": {"skip": true}, "patch_proj": {"skip": true}, "time_embedder": {"skip": true}, "attn\\.out_proj": {"skip": true}, "mlp\\.fc1|mlp\\.fc2": {"format": "int4_convrot_w4a4"}, "adaln_proj\\.linear": {"format": "int4_convrot_w4a4"}, "attn\\.qkv_proj": {"format": "int4_convrot_w4a4"} }Not a proven config just an example.
MiniMax_H3_00022_.mp4