SynthID for vLLM

This is the main repository for a reproducible port and evaluation of SynthID-Text on vLLM. It contains the vLLM V1 custom logits processor, Hugging Face conformance tests, generation and detector-training code, public experiment keys, result figures, and the exact release documentation. It is an experimental research harness, not a production watermarking system.

Published artifacts

Artifact	Location	Contents
Code and results	This repository	vLLM plugin, tests, experiment scripts, figures, and documentation
Generated corpus	xlr8harder/synthid-qwen3-4b-instruct-2507-wildchat	120,000 Qwen responses across unwatermarked, Key-A, and Key-B arms
Trained detectors	xlr8harder/synthid-qwen3-4b-instruct-2507-detectors	Six independently loadable detector bundles and their evaluation grid
Source prompts	xlr8harder/wildchat-filtered-rated-prompts	Cleaned and deduplicated WildChat prompt source

The initial Hub artifacts are frozen as dataset v1.0 and detectors v1.0. The official weighted-mean control and its raw-score amendment are frozen as detectors v1.1.

The primary corpus uses Qwen3-4B-Instruct-2507 at native BF16 with the paper sampling profile: temperature 0.7, top-k 100, top-p 1.0, and min-p 0. The dataset card records this profile on every row. The experiment-only Key-A and Key-B configurations are intentionally disclosed for exact reproduction and must not be reused for deployment.

Main findings

• At 200 evaluated tokens, same-model detectors reached 71.1% true-positive rate for Key A and 67.5% for Key B at thresholds calibrated to 1% validation false-positive rate.
• Changing the sampler from top-k 100/top-p 1.0 to top-k 20/top-p 0.8 reduced Key-A detection from 70.8% to 33.8% on a common matched cohort.
• Neither AIME 2026 nor IFBench showed a statistically significant quality difference between unwatermarked and watermarked generations.
• Blind rephrasing with an unwatermarked 4B model reduced matching-key detection from roughly 70% to 4–5% among rewrites passing a 90–110% token- length gate. Semantic fidelity was not independently judged.
• The released learned models use the Nature paper's default Bayesian scorer. Google's official training-free weighted mean retained similar 200-token TPR while reducing the largest observed cross-corpus FPR from 2.78% to 1.82%; calibration-bootstrap intervals overlap in that small-ELI5-validation comparison.

The figures and their provenance notes are available in article/figures/. Detector operating points and broader cross-domain results are documented in docs/DETECTOR_STUDY.md; final AIME parser results are in docs/AIME_RESULTS.md. The post-publication weighted-mean control, calibration bootstrap, score-shift analysis, and base-rate illustration are in docs/DETECTOR_SCORE_FAMILY_AMENDMENT.md.

vLLM processor

The processor follows the Hugging Face implementation of SynthID Text and preserves the Transformers operation order:

The processor preserves the Transformers operation order:

raw logits -> temperature -> top-k -> top-p -> SynthID -> sample

Because vLLM's custom processor hook otherwise runs before those sampling operations, SynthID owns them and requires their vLLM fields to be neutral. With a finite top-k, the optimized path hashes only candidate token IDs instead of the full vocabulary. It falls back to the dense reference path when logits tie at the top-k boundary.

An opt-in sparse_fast implementation omits the tie fallback and, crucially, the resulting per-token CUDA-to-host synchronization. It must not be selected for the main condition until exact top-k ties have been measured on real model logits at the experiment's BF16 precision. The strict sparse path remains the conformance reference.

The sparse_support implementation instead runs the exact full-vocabulary HF top-k and top-p filters, then hashes a fixed support of at least 64 candidates. It avoids both the synchronization and full-vocabulary SynthID hashing while preserving ties whenever the filtered support fits that audited capacity.

The diagnostic-only base_filter mode runs the same HF temperature/top-k/ top-p stages but omits SynthID. Comparing it with an ordinary vLLM request at the same seed isolates any difference in the plugin-owned base sampler.

Before a main experiment, generate a fresh configuration with scripts/generate_watermark_config.py. It stores the private integer key with mode 0600 and separately writes a public SHA-256 commitment over the canonical full config. Pass the private file to run_aime2026.py --watermark-config ...; each new run verifies and snapshots it before submitting work.

Install and serve

Install this package in the same environment as vLLM, then start a V1 server:

pip install -e .
vllm serve Qwen/Qwen3-4B-Thinking-2507 \
  --dtype bfloat16 \
  --kv-cache-dtype auto \
  --no-async-scheduling \
  --logits-processors vllm_synthid.processor:VLLMSynthIDLogitsProcessor

The model config declares BF16, so BF16 weights and BF16 KV cache are fixed for the main experiment. Lower-precision weights or KV are not throughput tuning parameters in this comparison.

The processor is loaded explicitly rather than registered for automatic discovery. That makes the experimental dependency visible in the launch log and prevents an accidental duplicate load if both mechanisms are used.

--no-async-scheduling is required. In vLLM 0.18, async scheduling leaves a trailing -1 placeholder in the CPU-visible output-token list while the next sample is computed. The custom logits-processor interface therefore cannot see the latest generated token in time to construct SynthID's n-gram context. The plugin rejects an async scheduler at startup instead of silently emitting an undetectable, mis-keyed sequence.

An online request passes the real sampling configuration through vllm_xargs while leaving vLLM's built-ins neutral:

{
  "model": "Qwen/Qwen3-4B-Thinking-2507",
  "prompt": "...",
  "temperature": 1.0,
  "top_k": 0,
  "top_p": 1.0,
  "max_tokens": 81920,
  "seed": 1234,
  "vllm_xargs": {
    "synthid": "{\"ngram_len\":5,\"keys\":[654,400,836,123,340,443,597,160,57],\"temperature\":0.6,\"top_k\":20,\"top_p\":0.95,\"sampling_table_seed\":0,\"sampling_table_size\":65536,\"context_history_size\":1024,\"implementation\":\"sparse\"}"
  }
}

The JSON string is intentional: vLLM's OpenAI request schema allows scalar or list values, but not nested objects, inside vllm_xargs. Offline SamplingParams.extra_args may pass the object directly.

Experimental baseline requests use implementation=base_filter with the same neutral vLLM fields as SynthID requests. This makes the only distributional difference between arms the watermark transform. Ordinary vLLM sampling was retained as a diagnostic reference, but an online same-seed check found BF16 tie/numerical divergences from the HF filtering path, so it is not the study's scientific baseline. For a paired comparison, give both arms the same stable per-sample seed. HF/vLLM conformance is otherwise defined on post-processor probabilities, not token-for-token output equality.

Correctness gates

Run the local tests with:

uv run --extra test pytest

The conformance test applies Transformers' TemperatureLogitsWarper, TopKLogitsWarper, TopPLogitsWarper, and SynthIDTextWatermarkLogitsProcessor, then compares that result with both this package's dense reference and sparse candidate implementation over multiple generation steps. The dense path must be bit-exact; the sparse path allows at most 1e-6 absolute probability error for floating-point reduction-order differences. The rented-GPU gate must rerun it on CUDA in the exact vLLM/Transformers container before throughput tuning.

The current plugin targets the vLLM V1 custom processor API in vLLM v0.18.1 (commit a26e8dc7ff2111a005144d775ecf9cebf56c45b2) and was also checked against main commit 311b3513af33bc29b4acb2fde2e9313e5e9966a0. That API is documented by vLLM as unstable, so upgrading vLLM requires rerunning API and conformance tests.

The 0.18.1 patch is required on Python 3.10: 0.18.0 has an upstream standalone_compile.FakeTensorMode startup failure, retained in the GPU-gate artifacts.

See docs/GPU_RUNBOOK.md for the staged baseline, capacity-tuning, live-score, and scale-up gates. The paired detector corpus, holdouts, metrics, controls, and cost envelope are specified in docs/DETECTOR_STUDY.md.

The exact scope of the Hugging Face compatibility claim, including the device-dependent sampling-table caveat, is documented in docs/COMPATIBILITY.md. The exact three-repository publication boundary is tracked in docs/RELEASE.md.

Public reproduction workflow

The public release consists of this canonical code repository, one Hugging Face dataset repository, and one Hugging Face detector repository. The code repository contains:

• the installable vLLM V1 plugin under src/vllm_synthid/;
• bit-exact integer-core and dense-reference Hugging Face conformance tests;
• the optimized sparse_support processor used for generation;
• scripts/train_synthid_detector.py for fitting one Bayesian detector per watermark key; and
• scripts/detect_synthid.py for scoring JSONL rows or a Hub dataset split with a trained detector, optionally applying a released fixed operating point for an individual-sample watermark decision.

The cleaned prompt source is xlr8harder/wildchat-filtered-rated-prompts, pinned by the experiment manifest to revision 96357ecb1ff1291aba8fb9d9d1bcbbf38651f08e. The generated Qwen dataset is published as xlr8harder/synthid-qwen3-4b-instruct-2507-wildchat. The trained detector variants are published together under xlr8harder/synthid-qwen3-4b-instruct-2507-detectors so comparison families can share one model card while remaining independently loadable by subfolder.

The primary released generation profile is synthid-paper-t0.7-k100-p1.0: temperature 0.7, top-k 100, top-p 1.0, min-p 0, native BF16, and a 4,096-token cap. These fields are repeated on every dataset row and bound into the detector release manifest. The complete older Qwen-default profile (top-k 20, top-p 0.8) is retained as a named ablation and must not be mixed with the primary corpus under the same configuration name.

The two experiment-only key configurations are public at reference/experiment-keys/ so the released detectors and generation settings are exactly reproducible. Their commitments were frozen before generation. These keys are research fixtures and must not be reused for deployment.

The code and detector artifacts use the MIT license. The WildChat-derived dataset uses ODC-BY-1.0, inherited from its pinned source corpus. Container reproduction uses the digest-pinned build in docker/Dockerfile; the initial release provides build instructions rather than republishing the large CUDA/vLLM image. Its completed RTX 5090 parity and real-engine smoke are recorded in reference/container-validation-vllm-0.18.1.json.

The generated dataset exposes three configurations—qwen_unwatermarked, qwen_synthid_key_a, and qwen_synthid_key_b—each with matched_train, matched_validation, matched_test, unmatched_train, unmatched_validation, and unmatched_test splits.

The frozen 80,000-prompt assignment, 120,000-job full queue, 12,000-job gate, resumable three-key-state runner, live score commands, and Hub-ready exporter are documented in docs/WILDCHAT_THREE_ARM_DATASET.md. The older prepare_wildchat.py and run_wildchat.py remain the immutable two-arm functionality-gate harness; the release corpus uses the explicit *_three_arm scripts and v2 schemas.

Train the key-A detector directly from the Hub dataset:

uv run --extra detector python scripts/train_synthid_detector.py \
  --dataset xlr8harder/synthid-qwen3-4b-instruct-2507-wildchat \
  --dataset-revision v1.0 \
  --base-dataset-config qwen_unwatermarked \
  --watermarked-dataset-config qwen_synthid_key_a \
  --watermark-config reference/experiment-keys/key-a.json \
  --sampling-table reference/sampling-table-seed0-cuda.int64le \
  --output-dir detector-key-a

Then score a held-out dataset split. The trained bundle carries the frozen table, so this also works on CPU without regenerating a different table:

uv run --extra detector python scripts/detect_synthid.py \
  --detector detector-key-a/model \
  --watermark-config reference/experiment-keys/key-a.json \
  --dataset xlr8harder/synthid-qwen3-4b-instruct-2507-wildchat \
  --dataset-config qwen_synthid_key_a \
  --dataset-split unmatched_test \
  --dataset-revision v1.0 \
  --device cpu \
  --output key-a-unmatched-test-scores.json

For a binary watermark call with a staged release bundle, also pass its evaluation-summary.json. The helper defaults to that summary's 200-token, 1%-validation-FPR operating point, explicitly marks shorter inputs ineligible, and records both the raw score and is_watermarked decision for eligible inputs:

uv run --extra detector python scripts/detect_synthid.py \
  --detector detector-release/same-model-matched/key-a/model \
  --evaluation-summary detector-release/same-model-matched/key-a/evaluation-summary.json \
  --watermark-config reference/experiment-keys/key-a.json \
  --input responses.jsonl \
  --token-ids-field token_ids \
  --output key-a-decisions.json

For multi-cohort evaluation, scripts/run_detector_comparisons.py loads one trained detector once, scores named validation and test cohorts, preserves the per-row scores, calibrates thresholds only on the declared negative validation cohort, and reports AUROC plus observed FPR/TPR on each test comparison. Local key-A and key-B examples live under artifacts/detector-comparisons/; the same spec format accepts Hub sources and raw-text cohorts tokenized with the pinned Qwen tokenizer. This is the interface for adding ELI5 human answers, OpenAssistant, Dolly, and original WildChat assistant responses without hard-coding those sources into the evaluator.

For the release-wide transfer result, scripts/run_detector_grid.py calibrates one threshold per detector on its own training-negative validation split and then applies that fixed threshold to every held-out cohort. Its Markdown and CSV tables identify both detector training sources. Matching-key cohorts show true detection rate and - for FPR; unwatermarked, wrong-key, human, and other-model cohorts show FPR and - for true detection rate. The canonical local specification is artifacts/detector-grid/release-v1.json. To keep the sweep cheap, the grid tokenizes each cohort once, truncates feature work at its longest declared prefix, computes SynthID g-values once per key, and reuses those features across every detector trained for that key.

scripts/run_training_free_detector_grid.py applies Google's official weighted mean and the unweighted mean to that same frozen grid without fitting a classifier. It preserves per-sample raw scores and unmasked-position counts, reports score-distribution shifts, and includes both fixed-threshold Wilson intervals and a deterministic calibration-plus-test bootstrap. scripts/analyze_detector_score_families.py joins those outputs to the released Bayesian grid for a like-for-like comparison.

scripts/render_detector_summary_grid.py builds a compact detector-centric view from an already-computed report.json, without repeating tokenization or detector inference. Each row identifies its positive and negative training sources, pools matched and unmatched same-key tests into one false-negative rate, and reports false-positive rates against the same grouped control columns for every detector. Markdown cells retain their eligible row counts; the CSV stores rates and counts in separate fields.