Quant Test LLM

Quant Test LLM is a Python CLI for evaluating quantized LLM deployment with local llama.cpp tooling. Phase 1 focuses on a reproducible GGUF workflow:

HF/local model
  -> detect format
  -> convert to FP16 GGUF
  -> quantize GGUF
  -> run native llama.cpp inference
  -> write logs, metadata, and reports

The core conversion, quantization, and one-shot run commands call native llama.cpp binaries through subprocesses. Benchmarking supports two local backends:

• subprocess: starts a native runner process for each prompt/model run.
• server: starts a managed local llama-server once per model and sends localhost completion requests for faster dataset-scale runs.

The native tools used are:

• convert_hf_to_gguf.py
• llama-quantize
• llama-completion for one-shot runs, or llama-cli as a compatible runner
• llama-server for optional managed server-backed benchmarks

Requirements

• Python 3.11+
• Poetry
• A local llama.cpp checkout or native llama.cpp tools on PATH
• convert_hf_to_gguf.py
• llama-quantize
• llama-completion or llama-cli
• llama-server for --backend server

Installation

poetry install

Then run the CLI through Poetry:

poetry run quant-test --help

The Poetry environment includes Typer, Pydantic, PyYAML, Rich, PyArrow, and pytest.

llama.cpp Configuration

Either put the native tools on PATH, set LLAMA_CPP_ROOT, or provide explicit tool paths:

export LLAMA_CPP_ROOT=/path/to/llama.cpp
export LLAMA_CPP_CONVERT_SCRIPT=/path/to/llama.cpp/convert_hf_to_gguf.py
export LLAMA_CPP_QUANTIZE=/path/to/llama.cpp/build/bin/llama-quantize
export LLAMA_CPP_CLI=/path/to/llama.cpp/build/bin/llama-cli
export LLAMA_CPP_RUNNER=/path/to/llama.cpp/build/bin/llama-completion
export LLAMA_CPP_SERVER=/path/to/llama.cpp/build/bin/llama-server

You can also keep these in a local .env file in this repo. The CLI loads simple KEY=VALUE lines from .env automatically. Use .env.example as the template. The real .env file is ignored by git because it contains machine-specific paths.

Check discovery with:

poetry run quant-test doctor
poetry run quant-test doctor --strict

Supported Model Inputs

The CLI currently supports:

• Existing .gguf files
• Local Hugging Face model directories containing config.json plus at least one .safetensors or .bin weights file

Hugging Face model IDs are detected, but Phase 1 expects models to be downloaded locally before conversion.

Functional Commands

These commands are currently wired, tested, and functional. Show the CLI command list with:

poetry run quant-test --help

Current commands:

quant-test doctor
quant-test convert
quant-test quantize
quant-test run
quant-test pipeline
quant-test benchmark
quant-test benchmark-suite

Quick command summary:

Command	Purpose	Primary outputs
doctor	Check native llama.cpp tool discovery	terminal status table
convert	Convert local HF directory or copy GGUF to FP16 GGUF target	GGUF, logs, metadata, reports
quantize	Quantize GGUF with llama-quantize	quantized GGUF, logs, metadata, reports
run	Run one prompt against one GGUF model	generation.txt, logs, metadata, reports
pipeline	Convert if needed, quantize, then run JSONL prompts	model artifacts, prompt runs, reports
benchmark	Compare one baseline GGUF with one candidate GGUF	benchmark.json, benchmark.md, comparison.csv
benchmark-suite	Compare one baseline GGUF with multiple candidates	benchmark-suite.json, benchmark-suite.md, suite-comparison.csv

Recommended Benchmark Defaults

Use these defaults for comparable baseline-vs-candidate runs:

--max-tokens 512
--ctx-size 32768
--temperature 0
--seed 42
--on-error skip
--resume

Use --backend server for larger runs and --backend subprocess when you want maximum isolation per prompt.

Current benchmark reports include speed, latency, size reduction, run status, failed-run counts, resumed-run counts, output paths, prompt metadata, and over-generation diagnostics. They also include three MVP decision metrics: Semantic Preservation, Stability Score, and Performance Gain. Semantic Preservation is currently lexical similarity, not an embedding or judge-model score.

quant-test doctor

Check whether required native llama.cpp tools can be found.

poetry run quant-test doctor
poetry run quant-test doctor --strict

Useful options:

--llama-cpp-root PATH
--strict

quant-test convert

Convert a local Hugging Face model directory to FP16 GGUF:

poetry run quant-test convert \
  --model /models/tinyllama \
  --out artifacts/tinyllama.f16.gguf

Useful options:

--model PATH
--out PATH
--llama-cpp-root PATH
--converter-script PATH

If the input is already .gguf, conversion is skipped and the model is copied to the requested output path.

quant-test quantize

Quantize a GGUF model:

poetry run quant-test quantize \
  --input artifacts/tinyllama.f16.gguf \
  --method q4_k_m \
  --out artifacts/tinyllama.q4_k_m.gguf

Useful options:

--input PATH
--method q4_k_m
--out PATH
--llama-cpp-root PATH
--quantize-bin PATH

quant-test run

Run native inference:

poetry run quant-test run \
  --model artifacts/tinyllama.q4_k_m.gguf \
  --prompt "Write one sentence." \
  --out runs/smoke \
  --max-tokens 64

Useful options:

--model PATH
--prompt TEXT
--out PATH
--max-tokens 128        use -1 for llama.cpp infinity, -2 until context filled
--temperature 0.8
--threads N
--llama-cpp-root PATH
--runner-bin PATH

run writes the generated text to generation.txt and captures native stdout, stderr, command metadata, JSON report, and Markdown report.

quant-test pipeline

Run conversion if needed, quantization, and inference for every prompt in a JSONL prompt file:

poetry run quant-test pipeline \
  --model /models/tinyllama \
  --quant q4_k_m \
  --prompt-file prompts.jsonl \
  --out artifacts \
  --max-tokens 64

Useful options:

--model PATH
--quant q4_k_m
--prompt-file PATH
--out PATH
--max-tokens 128        use -1 for llama.cpp infinity, -2 until context filled
--temperature 0.8
--threads N
--llama-cpp-root PATH
--converter-script PATH
--quantize-bin PATH
--runner-bin PATH

Prompt And Dataset Inputs

pipeline currently reads JSONL prompt files. benchmark and benchmark-suite can read either JSONL prompts or a Parquet dataset directly.

JSONL prompt files accept one prompt per line. Each line can be a plain prompt or a JSON object:

{"id":"short-summary","prompt":"Summarize quantization in one sentence."}
Write a haiku about low-bit inference.

Parquet dataset input is useful when prompts and metadata already live in a dataset file. Supported prompt formats:

text              read a string directly from --prompt-column
chat-first-user   read the first user message from a chat messages column

For the local data/0000.parquet file, the messages column is a chat-style list, so use --prompt-format chat-first-user. Metadata columns are preserved in the JSON report, run metadata, and CSV output.

quant-test benchmark

Benchmark a baseline GGUF against a quantized candidate:

poetry run quant-test benchmark \
  --baseline models/gguf/qwen2.5-0.5b-instruct-fp16.gguf \
  --candidate artifacts/qwen2.5-0.5b-instruct.q4_k_m.gguf \
  --prompt-file prompts.jsonl \
  --out benchmarks/qwen-q4 \
  --max-tokens 128 \
  --temperature 0 \
  --seed 42

Benchmark directly from a Parquet dataset:

poetry run quant-test benchmark \
  --baseline models/gguf/qwen2.5-0.5b-instruct-fp16.gguf \
  --candidate models/gguf/qwen2.5-0.5b-instruct-q4_k_m.gguf \
  --dataset data/0000.parquet \
  --prompt-column messages \
  --prompt-format chat-first-user \
  --sample-size 10 \
  --metadata-columns id,domain,source_dataset \
  --out benchmarks/qwen-data0000 \
  --max-tokens 64 \
  --temperature 0 \
  --seed 42

Useful options:

--baseline PATH
--candidate PATH
--prompt-file PATH      JSONL prompts; use this or --dataset
--dataset PATH          Parquet dataset; use this or --prompt-file
--prompt-column TEXT    default: prompt
--prompt-format TEXT    text or chat-first-user
--sample-size N         limit number of dataset prompts
--metadata-columns TEXT comma-separated dataset metadata columns
--out PATH
--max-tokens 128        use -1 for llama.cpp infinity, -2 until context filled
--temperature 0
--seed 42
--threads N
--ctx-size N
--concurrency N         concurrent native subprocesses or HTTP requests
--batch-size N          prompt scheduling batch size; --batch is an alias
--on-error fail|skip    default: fail; use skip to continue after prompt failures
--resume                reuse matching completed prompt metadata in --out
--long-output-token-threshold N  default: 4096
--long-output-char-threshold N   default: 20000
--llama-cpp-root PATH
--runner-bin PATH

benchmark runs both models on the same prompts and settings, then writes benchmark.json, benchmark.md, and comparison.csv. The command prints a terminal progress bar as model-prompt runs complete.

Subprocess benchmark mode can also run multiple native llama-completion or llama-cli processes concurrently. Keep this value conservative because each process loads the model:

poetry run quant-test benchmark \
  --backend subprocess \
  --baseline models/gguf/qwen2.5-0.5b-instruct-fp16.gguf \
  --candidate models/gguf/qwen2.5-0.5b-instruct-q4_k_m.gguf \
  --dataset data/0000.parquet \
  --prompt-column messages \
  --prompt-format chat-first-user \
  --sample-size 100 \
  --metadata-columns id,domain,source_dataset \
  --out benchmarks/qwen-data0000-cli-concurrent \
  --max-tokens 512 \
  --ctx-size 32768 \
  --concurrency 2 \
  --batch 16 \
  --on-error skip \
  --resume

Server-backed benchmark mode starts llama-server once per model and sends prompt requests over localhost. This avoids reloading the GGUF for every prompt and is much faster for larger datasets:

poetry run quant-test benchmark \
  --backend server \
  --baseline models/gguf/qwen2.5-0.5b-instruct-fp16.gguf \
  --candidate models/gguf/qwen2.5-0.5b-instruct-q4_k_m.gguf \
  --dataset data/0000.parquet \
  --prompt-column messages \
  --prompt-format chat-first-user \
  --sample-size 100 \
  --metadata-columns id,domain,source_dataset \
  --out benchmarks/qwen-data0000-server \
  --max-tokens 512 \
  --ctx-size 32768 \
  --concurrency 4 \
  --server-parallel 4 \
  --batch 32 \
  --server-timeout 600 \
  --on-error skip \
  --resume

Additional server options:

--backend subprocess|server
--server-bin PATH       path to llama-server; also reads LLAMA_CPP_SERVER
--server-host TEXT      default: 127.0.0.1
--server-port N         default: 0, choose a free local port
--server-timeout N      startup and request timeout in seconds
--server-parallel N     llama-server slots passed as -np

Use --concurrency 1 for cleaner latency comparisons. Use higher concurrency when measuring throughput with llama.cpp continuous batching.

By default, benchmark commands stop on the first prompt error. Add --on-error skip when running large datasets so one oversized prompt or native runner failure is recorded as status: failed and the remaining prompts continue. Failed prompts still get a prompt directory, generation.txt, error.txt, and metadata, and reports include failed-run counts.

Add --resume to continue an interrupted benchmark in the same --out directory. A saved run is reused only when its metadata matches the same model and exact prompt.

Benchmark reports also include output diagnostics. Long outputs are flagged when generated tokens or output characters cross the configured thresholds, when a run hits a positive --max-tokens limit, or when total tokens approach the configured context size.

quant-test benchmark-suite

Benchmark one baseline GGUF against multiple quantized candidates:

poetry run quant-test benchmark-suite \
  --baseline models/gguf/qwen2.5-0.5b-instruct-fp16.gguf \
  --candidate artifacts/qwen2.5-0.5b-instruct.q4_k_m.gguf \
  --candidate artifacts/qwen2.5-0.5b-instruct.q5_k_m.gguf \
  --candidate artifacts/qwen2.5-0.5b-instruct.q8_0.gguf \
  --prompt-file prompts.jsonl \
  --out benchmarks/qwen-suite \
  --max-tokens 128 \
  --temperature 0 \
  --seed 42

Benchmark multiple candidates directly from a Parquet dataset:

poetry run quant-test benchmark-suite \
  --baseline models/gguf/qwen2.5-0.5b-instruct-fp16.gguf \
  --candidate models/gguf/qwen2.5-0.5b-instruct-q4_k_m.gguf \
  --candidate models/gguf/qwen2.5-0.5b-instruct-q5_k_m.gguf \
  --dataset data/0000.parquet \
  --prompt-column messages \
  --prompt-format chat-first-user \
  --sample-size 10 \
  --metadata-columns id,domain,source_dataset \
  --out benchmarks/qwen-suite-data0000 \
  --max-tokens 64 \
  --temperature 0 \
  --seed 42

Useful options:

--baseline PATH
--candidate PATH      repeat this option for each candidate
--prompt-file PATH    JSONL prompts; use this or --dataset
--dataset PATH        Parquet dataset; use this or --prompt-file
--prompt-column TEXT
--prompt-format TEXT  text or chat-first-user
--sample-size N
--metadata-columns TEXT
--out PATH
--max-tokens 128        use -1 for llama.cpp infinity, -2 until context filled
--temperature 0
--seed 42
--threads N
--ctx-size N
--concurrency N
--batch-size N
--on-error fail|skip
--resume
--long-output-token-threshold N
--long-output-char-threshold N
--llama-cpp-root PATH
--runner-bin PATH

benchmark-suite runs the baseline once per prompt, then each candidate on the same prompts and settings. It writes benchmark-suite.json, benchmark-suite.md, and suite-comparison.csv. The command prints progress across the baseline and every candidate model.

benchmark-suite also supports --backend server with the same server options.

For benchmarking, prefer a fixed positive --max-tokens value like 512 or 1024 when comparing models. --max-tokens -1 can run until EOS and may make a full dataset benchmark take much longer.

Artifact Layout

Commands create reproducible output directories with:

<out>/
  logs/
    *.stdout.log
    *.stderr.log
  metadata/
    *.command.json
    *.metadata.json
  models/
    *.gguf
  reports/
    *.json
    *.md
  manifest.json

Benchmark commands create:

<out>/
  baseline/
    prompt-0001/
      generation.txt
      logs/
      metadata/
  candidate/
    prompt-0001/
      generation.txt
      logs/
      metadata/
  reports/
    benchmark.json
    benchmark.md
    comparison.csv
  manifest.json

Benchmark suite commands create:

<out>/
  baseline/
    prompt-0001/
  candidates/
    01-q4_k_m/
      prompt-0001/
    02-q5_k_m/
      prompt-0001/
  reports/
    benchmark-suite.json
    benchmark-suite.md
    suite-comparison.csv
  manifest.json

Native calls are recorded with metadata such as:

{
  "model_source": "/models/tinyllama",
  "converted_gguf": "artifacts/tinyllama.f16.gguf",
  "quantized_gguf": "artifacts/tinyllama.q4_k_m.gguf",
  "quant_method": "q4_k_m",
  "backend": "llama.cpp",
  "command": "llama-quantize ...",
  "stdout_log": "logs/...",
  "stderr_log": "logs/...",
  "status": "success"
}

When --on-error skip is used, failed prompt runs remain in the artifact tree with status: failed, skipped: true, error.txt, and error details in the CSV/JSON/Markdown reports.

Benchmark run metadata also includes output_diagnostics, with output character/line counts, generated-token counts, max-token-hit flags, near-context flags, and long-output warning reasons.

Development

Run tests:

poetry run pytest

Latest known status:

24 passed

Run a syntax check with the system Python:

python3 -m compileall quant_test_llm tests

Troubleshooting

If doctor reports missing tools, set LLAMA_CPP_ROOT or the explicit LLAMA_CPP_CONVERT_SCRIPT, LLAMA_CPP_QUANTIZE, LLAMA_CPP_RUNNER, and LLAMA_CPP_SERVER variables.

If conversion fails, verify that the model path is a local Hugging Face directory with config.json and model weights. If quantization or inference fails, inspect the generated stderr log referenced in the command metadata.

For benchmark comparability, avoid --max-tokens -1 on large runs unless the goal is qualitative inspection. Prefer a fixed value such as 512 or 1024 so baseline and candidate outputs have comparable generation budgets.