feat: precision-driven quantization (FP16, RTN int4, static QDQ) via --precision flag

[DingmaomaoBJTU]

Summary

Adds precision-driven quantization to winml quantize and winml build. The --precision flag auto-selects the appropriate quantization algorithm: FP16 conversion, RTN weight-only, or static QDQ — no need to manually specify --algorithm or --rtn-bits.

Resolves #867

Supported Commands

Command	--precision support	Notes
winml build	✅	Full pipeline: export → optimize → quantize (precision controls quantize stage)
winml quantize	✅	Standalone quantization on pre-exported ONNX
winml config	✅	Generates build config with precision-resolved quant settings
winml perf	✅	Inherits from build pipeline (precision passed through)
winml eval	✅	Inherits from build pipeline (precision passed through)
winml export	❌ Removed	Precision is a quantize-stage concern, not export
winml optimize	❌ Removed	Precision is a quantize-stage concern, not optimize

Precision → Algorithm Auto-Resolution

Precision	Algorithm	Calibration	Output
fp16	FP16 conversion	❌	Full-model FP32→FP16
int4	RTN weight-only	❌	MatMulNBits (4-bit packed weights)
w4a16	RTN weight-only	❌	Same as int4
w4a8	RTN weight-only	❌	4-bit weights, 8-bit activation spec
int8	Static QDQ	✅	QuantizeLinear/DequantizeLinear W8A8
int16	Static QDQ	✅	W16A16 QDQ
w8a16	Static QDQ	✅	Mixed W8A16 QDQ
w8a8	Static QDQ	✅	Same as int8
auto	Device-dependent	Varies	Auto-selects for target hardware

Key rule: 4-bit weight → RTN (no QDQ support for 4-bit), 8/16-bit → static QDQ.

Usage

# FP16 — no calibration, ~50% size reduction
winml quantize -m model.onnx --precision fp16
winml build -m facebook/convnext-tiny-224 -o out/ --precision fp16

# RTN 4-bit weight-only — no calibration, fast
winml quantize -m model.onnx --precision int4
winml build -m facebook/convnext-tiny-224 -o out/ --precision int4

# Static QDQ INT8 — requires calibration data
winml quantize -m model.onnx --precision int8 --samples 100
winml build -m facebook/convnext-tiny-224 -o out/ --precision int8

# Warnings for mismatched options
winml quantize -m model.onnx --precision fp16 --samples 50
# → Warning: --samples ignored — FP16 conversion does not use calibration data.

Design

Algorithm Selection Logic

precision → is_weight_only_precision()?
  ├─ Yes (int4, w4a16) → RTN path (MatMulNBitsQuantizer)
  ├─ No, fp16?         → FP16 path (convert_float_to_float16)
  └─ No, quantized?    → QDQ path (static quantization + calibration)

RTN Configuration

--precision int4 automatically sets:

• algorithm = "rtn"
• rtn_bits = 4 (derived from precision)
• rtn_block_size = 128 (default, tunable via future CLI flag)
• rtn_symmetric = True (default)

Advanced users can tune RTN params without needing --rtn-bits — bit-width is always inferred from precision.

Validation

• FP16/RTN/dynamic algorithms skip task/model_name validation (no calibration needed)
• Only algorithm="static" requires calibration → requires task/model_name for HF builds
• Invalid precisions (e.g., banana, w4a4) produce clear error messages

E2E Verified (ConvNeXt-Tiny-224)

Command	Result
winml build --precision fp16	✅ 109MB→54.6MB, 87s
winml build --precision int4	✅ RTN 4-bit, 87s
winml quantize --precision fp16	✅ 4.7s
winml quantize --precision int4	✅ 1.1s (RTN)
winml quantize --precision int8	✅ 46s, 761 QDQ nodes
winml quantize --precision w4a16	✅ Same as int4
winml quantize --precision fp16 --samples 50	✅ Warning printed
winml quantize --precision int4 --samples 50	✅ Warning printed
winml quantize --precision banana	✅ Error: invalid precision

Files Changed

Core

• config/precision.py — Added int4 preset, is_weight_only_precision(), extract_weight_bits(), expanded _VALID_WEIGHT_BITS to include 4
• config/build.py — resolve_quant_compile_config creates RTN config for weight-only; validation skips calibration requirements for RTN/FP16/dynamic
• quant/quantizer.py — Three execution paths: FP16 fast path → RTN (MatMulNBitsQuantizer) → static QDQ
• quant/config.py — algorithm field (static/dynamic/rtn), RTN params, FP16 params
• commands/quantize.py — CLI routing: FP16 → RTN → QDQ, with calibration-ignored warnings
• commands/build.py — _patch_device propagates algorithm/RTN fields; _run_quantize_stage has dedicated RTN path with StageLive output
• optim/fp16.py — convert_to_fp16() with keep_io_types, already-FP16 skip, topo sort fix

Removed

• --precision from export and optimize commands (quantize stage handles all precision work)

TODO (follow-up PRs)

• Mixed precision (FP16 + QDQ): e.g., --precision int8 --fp16 to run QDQ quantization first, then convert remaining FP32 ops to FP16. The config infrastructure supports this (fp16=True, fp16_only=False) but CLI flags and build pipeline routing are not yet wired.
• RTN CLI tuning flags: Expose --rtn-block-size, --rtn-symmetric, --rtn-accuracy-level on winml quantize and winml build for advanced users.
• Dynamic quantization path: Wire --algorithm dynamic through quantize command (config already supports algorithm="dynamic" but no CLI flag yet).

[timenick]

Three findings on PR #872.

🤖 Generated with GitHub Copilot CLI