feat(kvcache): sub-byte Q4_0 and Q6_0 KV cache quantization - #268

Open

fangyuan-3149 wants to merge 4 commits into

FlashML-org:mainfrom

fangyuan-3149:feat/kv-cache-q4

Commits on Aug 29, 2026

fix(engine): handle torch 2.9 allocator API deprecation
fangyuan-3149
committed
224018c
feat(kvcache): sub-byte Q4_0 KV cache quantization
fangyuan-3149
committed
6f60055
fix(kvcache): wire --kv-cache-dtype through the engine
fangyuan-3149
committed
10e90eb
fix(tests): correct sub-byte test expectations and dtype mismatches
fangyuan-3149
committed
f5ca3e1