Suture

An 8B vision-language model that lists what changed between an underwriting binder and the issued policy.

Two page images in. Structured JSON out (PASS/FLAG + a 13-class discrepancy list). The oracle is a program, not a judge model. On a held-out seed the adapter clears 0.95 recall and 0.95 precision.

	Hub
Model	caiotheodoro/suture-8b
Dataset	caiotheodoro/suture
Predictions	caiotheodoro/suture-evals
Collection	suture

Adapter: QLoRA on Qwen/Qwen3-VL-8B-Instruct (Apache-2.0). Checkpoint in this project: /checkpoints/sft-dedhi/final. Code, generator, and scorer live here. Training order: docs/TRAINING.md.

Result

Seed-777, n=1000, 739 FLAG, zero signature overlap with train. Eval re-renders pages from structured fields (split-line limit/deductible layout). Same scorer for every row.

System	Recall	HIGH	Prec	Parse
suture-8b (sft-dedhi)	0.959	0.969	0.956	1.0
prior published (sft-limithi)	0.839	0.893	0.870	1.0
GPT-5.6 Luna, zero-shot vision	0.373	0.388	0.344	0.972
Qwen3-VL-8B, no adapter	0.098	0.115	0.278	0.973

CONTRACTS asked for >0.95 recall, >0.95 precision, 1.00 HIGH. Rec and prec hold. HIGH is 0.969. Per-class tables: docs/BENCHMARK.md. Citation exact-match on SFT rows is ~1.0 because the model clones gold citation strings — type recall is the money metric.

Deductible direction, the previous hole, is closed: DEDUCTIBLE_HIGHER 0.957 / 0.846 (was 0.507 / 0.407), DEDUCTIBLE_LOWER 0.989.

Why this task is solvable

Policy issuance QC is a closed diff. The underwriter agreed to a finite set of terms (named insured, period, coverage parts, limits, deductibles, ISO/AAIS form numbers, additional insureds, premium). The carrier issued a declarations page and a forms schedule. Either those two documents match or they do not. There is no coverage opinion, no negotiation, no “is this 40% or 60% similar.”

Carriers still do this with people and with rules inside Guidewire / Duck Creek. Public error-rate figures for issuance sit around 1–5%; the cost is premium leakage, E&O, and reissue, not a leaderboard. There is no published VL benchmark for binder-vs-policy consistency.

That combination — finite ontology, exact ground truth, no dedicated model — is why a small adapter can move the number. The same pattern (seeded generator + verifier-as-oracle) is what let a 1.7B LoRA beat a frontier model on ReconForge, a different closed task.

Related work (nearby, wrong task)

Document extraction. FUNSD, CORD, DocVQA, VRDU ask a model to read a field off a page. Useful, and a prerequisite for some production stacks. They do not ask whether two documents agree.

Claims and underwriting NLP. Fraud, FNOL classification, appetite routing, coverage interpretation. Those problems are open-ended. An LLM judge is a reasonable last resort there. It is the wrong oracle here.

General VL. Qwen3-VL, InternVL, Gemini, GPT-5 vision. They parse JSON and they can see a page. Zero-shot they do not know this taxonomy: the 8B base is 0.098 recall; Luna is 0.373. The gap is listing the right type on a held-out seed, not “can a frontier model read.”

Synthetic documents. The page renderer is ACORD-style, not a scan of a real policy. That is a limitation (see below). It is also what makes the oracle exact: inject N discrepancies, verify they are the only N, refuse the task on mismatch.

Method

1. Generate. suture_forge draws a binder, derives a consistent policy, injects 0–2 discrepancies from the 13-class taxonomy in CONTRACTS.md, renders two PNGs.
2. Verify. suture_forge.verify diffs structured binder vs policy. The generator’s self-check requires oracle == injected set. Gold is mechanical.
3. Hold out. Train/val are seed 7. The contracted benchmark is seed 777, n=1000. Signature = SHA-256 over sorted ground-truth fields. Leakprobe must read 0 on the real split (and 1.0 on an intentional leak).
4. Train. 4-bit QLoRA (r=32, α=64) on Qwen3-VL-8B-Instruct, 384px thumbs, thinking off, Modal L4. Continue-from-adapter, one epoch per mix. Chain: sft-vl → sft-distill → sft-restem → sft-numeric → sft-limithi → sft-ded → sft-dedhi.
5. Score. Severity-weighted recall (HIGH=1.0, MEDIUM=0.6, LOW=0.3). A HIGH class is caught only if type and severity match. Unparseable output is a parse miss.

GRPO on this L4 stack shortened the policy and dropped val recall. It is not in the published adapter. ReST-EM helped listing early and did not teach numeric reading; later mixes did.

Data

Synthetic only. JSONL stores structured Task objects (binder, policy, expected, signatures). Images are re-rendered at train/eval time from those fields — Mac vs Linux raster can differ; labels do not.

Split	Seed	n	Hub config
Official train	7	320	train_seed7
Published SFT mix	7+11+13+17+19+23	5120	train
Val	7 holdout	80	validation
Benchmark	777	1000	benchmark

Seed 777 was never in train. JSONL stays gitignored in this repo; the Hub dataset is the public copy. Regenerators: forge CLI (pilot, split, benchmark, leakprobe).

Experiments

The published number is seed-777, not val. Val n=80 is optimistic (dedhi 1.000 / 1.000 / 0.970) and has zero DEDUCTIBLE_HIGHER gold.

Class-conditional mixes were the working lever after distill/ReST-EM plateaued:

• Numeric reading (sft-numeric → sft-limithi): 777 recall 0.622 → 0.839.
• Deductible sign (sft-ded): LOWER 0.494 → 0.897; HIGHER precision 0.407 → 0.700; HIGHER recall stuck at 0.507.
• HIGHER boost + train_ded replay (sft-dedhi): HIGHER 0.507 → 0.957 (FN 34 → 3); LOWER held at 0.989; swap FPs gone.

Remaining HIGH misses on 777 are AI_MISSING (0.912) and FORM_MISSING (0.944). That is issue #12.

Limitations

• Pages are forge-rendered, not carrier scans. Transfer to real PDFs is unmeasured.
• HIGH recall is 0.969, not 1.00.
• 8B QLoRA at 384px. Native-640 and 27B were not needed for the 0.95 rec/prec bar and are not in this release.
• Intended use is research and reproduction of this benchmark, and as a starting point for the next SFT loop. A human still signs off on a live issuance file.

Reproduction

git clone https://github.com/caiotheodoro/suture
cd suture && make sync && make validate

from peft import PeftModel
from transformers import AutoProcessor, Qwen3VLForConditionalGeneration

base = "Qwen/Qwen3-VL-8B-Instruct"
model = Qwen3VLForConditionalGeneration.from_pretrained(base, device_map="auto")
model = PeftModel.from_pretrained(model, "caiotheodoro/suture-8b")
processor = AutoProcessor.from_pretrained(base)

Two images, binder then policy. Schema in CONTRACTS.md. This project’s eval path is cloud/modal_eval.py (4-bit, 384px). To score published predictions without a GPU:

cd forge && PYTHONPATH=src uv run python -m suture_forge.cli class-report \
  --gold <benchmark.jsonl> --pred <sft-dedhi preds>

Operator log (jobs, gates, what not to do): docs/HANDOFF.md. Decision trail: docs/DECISIONS.md. Shared recipe: docs/methodology.md.

forge/    generator, oracle, leakprobe, class-report
model/    dataset builder, QLoRA SFT, GRPO, ReST-EM, eval
cloud/    Modal L4 train/eval
docs/     contracts trail, benchmark tables, this paper’s numbers

Citation

@software{theodoro2026suture,
  author = {Theodoro, Caio},
  title  = {Suture: vision-language policy-issuance QC with a verifier-as-oracle},
  year   = {2026},
  url    = {https://github.com/caiotheodoro/suture},
  note   = {seed-777 n=1000: recall 0.959, HIGH 0.969, precision 0.956, parse 1.0}
}

License: Apache-2.0 (this repo and the adapter). Base model Apache-2.0.