Quality quantization at very low bpw (<= 2 bit): mixed allocation + custom packers

[magnuslundstedt]

I am fairly new to AI research but find it very interesting. My research goal is efficient local AI, optimizing the balance between quality and speed using MLX through aggressive, non-uniform bit allocation.

Recently in my experiments I found that compressing some weights in Qwen to 1 bit, ternary (1.58 bit, 1.6 to 1.7 bpw packed) and 2 bit, while spending the bit budget on the more crucial weights, can yield smaller models and lower RSS. But quality is a challenge. I've not managed to test end-2-end yet, rather custom quantizer and packer -> unpack to FP16 -> measure PPL against a regular flat allocation at matched size.

I take inspiration from GGUF and the llama.cpp project, for example K-quant hierarchical scaling, TQ1_0 packing 5 trits per byte for BitNet (3^5 = 243, base-243), the work by Unsloth, and in general inspecting how successful quantized models have allocated their bit budget through a custom tool I built as a side-quest for fun (tensorlens.dev). One inherent benefit in GGUF seems to be the K-quant hierarchical (superblock) scaling, which lets smaller quants still perform well. Besides the common challenge of bit allocation and rounding (prior knowledge, activation sensitivity, raw SSE, absmean/Lloyd-Max, etc), what about the capabilities of MLX itself?

That is the question I'm most curious about here: MLX quantize is affine group quantization with a single scale/bias per group and stops at 2-bit. Has anyone explored K-quant-style hierarchical scaling, or sub-2-bit / ternary packers, in MLX? Or more generally, what interest, research or experience is here on custom packers, quantizers and extending MLX in this direction?

Any interesting very-low-bpw MLX experiments, papers or discussions you can share or reference would be great.