feat: QFX16/QFX32 lossless quantization (2.05x compression, zero quality loss)#26136
feat: QFX16/QFX32 lossless quantization (2.05x compression, zero quality loss)#26136theQarchitect wants to merge 3 commits into
Conversation
…o quality loss) QFX16: Transition-encoded BFloat16, 8-16 bpw effective, LUT-based inference QFX32: Planar-interleaved Z-RLE Float32, 15.64 bpw, NEON+Metal accelerated Both use the bijection L(a,b) = (a, (b-a) mod 256) in Z_256 for provably invertible delta encoding with zero-run-length compression. QFX32 supports opt-in dequant-on-load (GGML_QFX32_DEQUANT=1) for native f32 inference speed at 2x RAM cost. Tested on Llama-3.2-1B: F32 4714 MiB -> QFX32 2305 MiB (2.05x, lossless) F16 2357 MiB -> QFX16 2073 MiB (1.14x, lossless BF16) QFX32 streaming: 1.4 t/s gen QFX32 dequant-on-load: 12.1 t/s gen (native f32 speed) QFX16: 2.1 t/s gen Assisted-by: Kiro (AI IDE)
github-actions
Bot
added
examples
ggml
|
Hi @theQarchitect, thanks for your contribution! Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:
Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below. |
|
Multiple backend changes in one PR: When adding support for a new model or feature, focus on CPU support only in the initial PR. Add support for other backends like CUDA in follow-up PRs. If you have a good reason to modify multiple backends in one PR, please explain it. Explanation: |
…leration The previous implementation used malloc + pointer swap which bypassed the ggml backend buffer system, preventing Metal GPU acceleration. New approach: - Promote QFX32 tensors to f32 type during create_tensor (before buffer allocation) so they get allocated in the correct GPU backend buffer - Disable mmap when dequant-on-load is active (tensor sizes change) - Decode QFX32 data from file and write decoded f32 directly into the backend buffer via ggml_backend_tensor_set Result: QFX32 dequant-on-load now runs 20% FASTER than native f32 (22.9 vs 19.0 t/s generation) due to reduced I/O from the smaller file. Assisted-by: Kiro (AI IDE)
Q5_Q: int8 quantization with type-relabel to Q8_0 at load time, enabling native Metal GPU kernels with zero custom kernel work. Results (Llama-3.2-1B): PPL: 12.69 (vs F16: 12.54, Q8_0: 12.53) Gen: 66.2 t/s (native Metal Q8_0 mul_mv) Prompt: 775.4 t/s Size: 1192 MiB (8.09 bpw) Usage: llama-quantize model.gguf out.gguf QFXQ Assisted-by: Kiro (AI IDE)
2 participants
Overview
Two new lossless quantization types that compress model weights with zero quality degradation:
Both exploit the bijection L(a,b) = (a, (b-a) mod 256) which is provably invertible in Z_256 -- subtraction is always invertible in modular arithmetic. Weights are delta-encoded and zero runs are collapsed, achieving significant compression on the highly correlated byte streams found in neural network parameters.
Key Feature: Dequant-on-Load
QFX32 supports an opt-in
GGML_QFX32_DEQUANT=1environment variable that decodes all weights to native f32 at model load time. This trades 2x RAM for full Metal GPU acceleration -- the decoded data lives in the GPU's backend buffer, enabling native f32 matmul kernels.Performance is actually 20% faster than loading the raw f32 GGUF due to reduced disk I/O (2.3 GB read vs 4.6 GB).
Benchmarks (Llama-3.2-1B, Apple M4)
Implementation
llama-quantize model.gguf out.gguf QFX32)Use Cases
Patent Notice
QFX16 and QFX32 are covered by a provisional patent filing by QomputeAI. The implementation is provided for inclusion in llama.cpp under the project's existing license terms.
Additional information
Related: #14465 (feature request for lossless quantization)
The dequant-on-load speedup over native f32 is due to reduced disk I/O: reading 2.3 GB and decoding in-memory is faster than reading 4.6 GB from SSD. The decode cost (~3.7s for 1B params) is amortized against the I/O savings.
Requirements