A fine-tuned small language model that writes Conventional Commits messages from code diffs. Runs locally via llama.cpp. Your code never leaves your machine.
Live Demo · Gradio Space · 0.6B GGUF · 1.7B GGUF · 0.6B adapter · 1.7B adapter · Dataset
I built Committed on Qwen3-1.7B, then tested whether a model a third the size could do the same job — and it essentially can. The 0.6B fine-tune matches the 1.7B on picking the right commit type and staying faithful to the diff, at roughly a third the parameters, a ~397 MB download, and faster local inference. The one honest trade is specificity: it writes slightly vaguer messages. So the 0.6B is the default — for most commits it's the better deal — and the 1.7B stays available as the bigger sibling when you want maximum specificity. Both are the same QLoRA recipe on ~58k real commits, served as a quantized GGUF through llama.cpp, CPU-only, with GBNF-constrained decoding so every output is a valid commit by construction.
Generate a commit message from a real diff, locally, CPU-only. Requires Python 3.11+. No GPU, no API key, nothing leaves your machine.
Install the package and the prebuilt CPU build of llama-cpp-python (the extra index serves a ready-made wheel, so there's no C++ compile step):
pip install --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cpu "committed @ git+https://github.com/marzoukbaig14/Committed.git"
Then pipe a diff straight in — this uses the default 0.6B model:
git diff | committed
Example:
$ git diff | committed
fix: increase default timeout to 60s
Want the larger, higher-specificity 1.7B instead? Pass --model:
git diff | committed --model 1.7b
You can also pass a diff file instead of using stdin:
committed path/to/change.diff
Notes:
- First run downloads the model and caches it; later runs reuse the cache and start in seconds. Downloads are public artifacts — no token needed. The default 0.6B GGUF is ~397 MB; the 1.7B is ~1 GB.
- The model loads fresh each run (a few seconds once cached). Progress goes to stderr; only the commit message is written to stdout, so
git diff | committedcomposes cleanly in scripts and pipes. - Bring your own GGUF with
--model-path path/to/model.gguf; cap length with--max-tokens N. Seecommitted --help.
A CPU-only Docker image (docker run --rm -i … < some.diff) is planned as a zero-setup alternative and will be documented here once published.
Committed is a QLoRA fine-tune of Qwen3 trained on a filtered subset of CommitChronicle, a dataset of roughly 10.7 million real GitHub commits. The filtered training set is ~58k single-file commits across 16 languages that follow the Conventional Commits specification, balanced by commit type. Two sizes were trained on the identical recipe — Qwen3-1.7B first, then Qwen3-0.6B — and the 0.6B is the default because it gets most of the quality at a third the size.
At inference time, the fine-tuned adapter is merged into the base model, converted to GGUF, and served via llama.cpp with grammar-constrained decoding. A GBNF grammar encodes the Conventional Commits format so every generation is valid by construction, no post-processing needed.
The project demonstrates a complete applied-ML pipeline on free infrastructure: data curation, parameter-efficient fine-tuning, rigorous multi-metric evaluation validated against human ratings, and a real serving and deployment layer.
- A novel research contribution. Commit message generation has been studied since 2017.
- An attempt to beat frontier models on quality. A small local model will not out-write Claude or GPT. That is not the point.
- A general-purpose chatbot. Single task, structured output.
Most commit-message tools (aicommits, opencommit, GitHub Copilot suggestions) send your diff to a third-party API. That means proprietary code, unreleased features, and internal architecture details leave your machine on every commit. Committed runs entirely locally, costs nothing per call, works offline, and scales to any volume without API bills.
Two sizes were fine-tuned on the identical pipeline and each evaluated against its un-tuned base on a 442-example test sample, scored by a DeepSeek (deepseek-chat) judge on four orthogonal axes, reweighted to the test split's true commit-type distribution. All four arms share the same judge, so every comparison is apples-to-apples.
| Metric | 0.6B base | 0.6B ft (default) | 1.7B base | 1.7B ft |
|---|---|---|---|---|
| Type accuracy (reweighted; always-fix floor 0.489) | 0.154 | 0.601 | 0.131 | 0.637 |
| Type-correctness | 0.296 | 0.726 | 0.296 | 0.778 |
| Faithfulness | 0.285 | 0.810 | 0.491 | 0.848 |
| Completeness | 0.353 | 0.729 | 0.543 | 0.776 |
| Specificity | 0.414 | 0.545 | 0.814 | 0.667 |
| Conjunctive (all four) | 0.101 | 0.359 | 0.175 | 0.471 |
| Graded mean (0–3) | 0.777 | 2.094 | 1.447 | 2.139 |
feat-share of outputs |
86.7% | 9.7% | 95.5% | 8.4% |
Both base models share the same dominant failure mode — "feat-collapse": they label the overwhelming majority of diffs feat (86.7% for the 0.6B base, 95.5% for the 1.7B), regardless of what the change did. Because fix commits alone are about 49% of real-world commits, a model that almost never predicts fix scores below a trivial always-fix baseline (0.489) on reweighted type accuracy — which is exactly what both bases do. Fine-tuning breaks the collapse: feat-share drops under 10% and type accuracy clears the floor for both sizes.
The one axis the fine-tunes give something up on is specificity. The 1.7B trades down from its base (0.81 → 0.67) as it adopts the terse, normalized subject style of the training targets, and the smaller 0.6B is vaguer still (0.55). It's a real trade-off, traceable to a normalization choice in the training data, and the next iteration targets it. Otherwise the two fine-tunes are close — graded 2.09 (0.6B) vs 2.14 (1.7B) — so the 0.6B gets most of the quality at roughly a third the size, which is why it's the default.
An LLM judge is only trustworthy if it agrees with a human. I hand-rated 50 examples blind and validated the DeepSeek judge against them:
| axis | agreement | Cohen's κ |
|---|---|---|
| type-correctness | 0.82 | 0.61 |
| faithfulness | 0.78 | 0.56 |
| completeness | 0.80 | 0.60 |
| specificity | 0.88 | 0.34 |
Three axes land at moderate-to-substantial agreement (κ ≈ 0.56–0.61); specificity is the weakest (κ 0.34), so specificity-driven differences carry the most judge uncertainty. Two honest caveats: these DeepSeek-judged numbers are not comparable to earlier Gemini-judged figures (only the deltas within the table above are valid), and n=50 gives wide confidence intervals.
The full before/after, the feat-collapse analysis, and the two-stories breakdown of where the model disagrees with gold are in docs/eval/FINDINGS_v1.md. Curated side-by-side outputs with verdicts are in docs/eval/examples_v1_i1.md. You can also try the model yourself on the live demo.
- The full eval writeup →
docs/eval/FINDINGS_v1.md. The honest before/after for both sizes, the specificity trade, and why the residual errors are a correctable data problem rather than a modeling dead end. - Worked examples →
docs/eval/examples_v1_i1.md. Side-by-side model-vs-gold cases with per-example verdicts. - The design and methodology →
MASTER.md. The single source of truth for the whole design. - Every decision, with rationale →
docs/DECISION_LOG.md. The full Architecture Decision Record trail: why Qwen3, why llama.cpp and GGUF, why the judge rubric is shaped the way it is, why the 0.6B became the default (ADR 0051). - The judge rubric →
docs/eval/judge_rubric.md. - The artifacts on the Hub → 0.6B GGUF · 1.7B GGUF · 0.6B adapter · 1.7B adapter · dataset.
Base models: Qwen/Qwen3-0.6B (default) and Qwen/Qwen3-1.7B (Apache 2.0, strong code priors)
Fine-tuning:
- QLoRA via PEFT LoRA + TRL SFTTrainer (vanilla
transformers, no Unsloth) - bitsandbytes 4-bit quantization (training only, not serving)
- transformers, accelerate
Data:
- CommitChronicle via Hugging Face datasets library
- pandas for filtering
Serving:
- llama.cpp via llama-cpp-python (CPU inference, replaces bitsandbytes which is CUDA-only)
- GGUF quantization (Q4_K_M serving artifact, both sizes)
- GBNF grammar-constrained decoding
- FastAPI inference endpoint (one API serves both models; a request param picks the size, default 0.6B)
- Docker
Demo: Gradio on Hugging Face Spaces (CPU Basic, free tier) with a model dropdown + a portfolio-integrated web demo calling the FastAPI Space
Evaluation:
- sacrebleu (BLEU), rouge-score (ROUGE-L)
- scikit-learn (prefix-classification accuracy, confusion matrix)
- DeepSeek (
deepseek-chat) as LLM-as-judge on four axes; the harness is backend-swappable (Gemini and Claude backends also supported) - 50 hand-rated examples for judge validation
Tracking and registry:
- Weights and Biases (run metrics, hyperparameters, sample generations)
- Hugging Face Hub as model registry (dataset, adapters, GGUFs, eval reports, model and dataset cards)
Dev environment:
- Local-native dev (uv-managed
.venv); devcontainer retained in-repo - uv for package management (uv sync to install, uv run to execute)
- ruff for linting, pytest for tests
- GitHub Actions: ruff + pytest CI gate
- Training: Northeastern Explorer HPC (A100); free-tier T4/Kaggle reproducible as fallback
Dependency split: The GPU training stack lives in a separate uv dependency group and is never installed in the CPU dev/serving environment. Serving deps are a minimal required set; eval/train/dev are optional groups.
Five metrics, chosen to cover different failure modes:
| Metric | What it measures | Reliability |
|---|---|---|
| BLEU (sacrebleu) | N-gram overlap with reference | Low for short text; reported for completeness |
| ROUGE-L | Longest-common-subsequence overlap | Complementary to BLEU; also limited alone |
| Prefix-classification accuracy | Did the model emit the correct type (feat, fix, refactor, etc.)? | High; deterministic and meaningful |
LLM-as-judge (DeepSeek deepseek-chat) |
Four orthogonal axes: type-correctness, faithfulness, completeness, specificity | Headline metric |
| Human ratings (50 examples) | Same axes, rated by a human | Used to validate the judge; judge-vs-human agreement is reported |
The judge-vs-human agreement is the key number — it gives the judge score an honest confidence bound rather than reporting it as ground truth. The judge applies an analytic per-axis rubric: faithfulness is a hard gate (an unfaithful message fails regardless of other axes), and the headline is the conjunctive pass-rate — the fraction of outputs that clear all four axes. Headline numbers are reweighted to the true deployment commit-type distribution.
Core:
- Filtered dataset published to Hugging Face Hub
- QLoRA fine-tune of Qwen3-1.7B, adapter on Hub, W&B tracked
- Multi-metric eval: BLEU, ROUGE-L, prefix-classification accuracy, LLM-as-judge validated against 50 human ratings
- Grammar-constrained GGUF inference (llama.cpp + GBNF)
- Gradio demo deployed to HF Spaces
- README with honest results and sample outputs
Production layer:
- FastAPI serving endpoint + Dockerfile
- Eval-in-CI regression gate
- Hugging Face Hub model and dataset cards
- Local CLI install path (
git diff | committed) with GGUF auto-download - Quantization quality-vs-latency benchmarks (Q4 / Q8 / fp16)
- Smaller-model comparison: Qwen3-0.6B on the identical recipe — it matches the 1.7B on type/faithfulness at a third the size, so it's now the default (see FINDINGS_v1.md, ADR 0051)
- Dual-model serving: one API + demo + CLI serve both sizes, 0.6B default
- Address the specificity regression (relax subject-only normalization in the next data iteration)
- Fine-tune for full multi-line commits (subject + body)
- Synthetic reasoning-trace distillation; with-vs-without-reasoning ablation
- Reasoning-display toggle in the demo
- Repo-specific style adaptation via RAG (retrieve similar past commits from the user's own repo)
- Multi-format output: Conventional Commits, Gitmoji, free-form
- VS Code extension (reads staged diff, fills the commit message box directly)
- Larger base models (3-7B) when compute permits
- Code: MIT
- Model adapters: Apache 2.0 (inherited from Qwen3)
- Dataset: redistributed under the source's terms; CommitChronicle is cited in the dataset card (arXiv 2308.07655), with per-row
repoandlicenseprovenance retained
Built by Marzouk Baig — MS Artificial Intelligence, Northeastern University