Synthetic Persona Pretraining (SPP) — data

Part of epfl-dlab/spp — start there for the paper abstract, overview figure, and the other components.

The data-generation half of Synthetic Persona Pretraining. This repository turns public web text into a pretraining corpus carrying constitution-grounded first-person reflections, in seven stages from raw Dolma 3 shards to the published Hub datasets. The training half is the sibling epfl-dlab/spp-training.

The unit of work is one annotated document plus the reflections written about it. That row is the artifact:

doc_id                  upstream Dolma 3 id — the join key back to the corpus
text                    the document, byte-identical to upstream
token_length            its length under the SmolLM2 Rust tokenizer
safety_score / is_bad   classifier output, 0-5; is_bad = score >= 3
reflection_position     character offset of the insertion point
reflection_token_index  the same point in annotated.bin's token space
reflection_1p           "I ..."             the paper's SPP{T0} reflection
reflection_3p           "The assistant ..." the SPP{T0}^3rd Person ablation
charter_reflection      constitution sections the reflection engaged (JSON)
canary_type             identity canary injected here, or empty

The insertion point is sampled per document, so the model sees reflections at every position in a document and never learns that they only follow a complete one. Training inserts reflection_1p at reflection_token_index; nothing else about the document changes.

The seven stages

#	Stage	Produces	Code	Doc
1	Download	47,142 of 63,911 Dolma 3 shards as local parquet	preprocessing/download/	docs/03-corpus.md
2	Safety annotation	a 0-5 harm score for every document	preprocessing/annotation/	docs/03-corpus.md
3	Subsample	the seeded 1T-token budget, split annotated / unannotated	preprocessing/subsample_and_stratify/	docs/03-corpus.md
4	Tokenize	Megatron .bin/.idx streams plus the sidecar parquet	preprocessing/tokenization/	docs/03-corpus.md
5	Prompt development	the frozen generator prompt and the choice of generator	pipeline/charter/{seed,improve,eval}/, pipeline/dashboard/	docs/05-generator-selection.md
6	Reflection generation	51.4M reflections written back into the sidecar	pipeline/charter/scale/	docs/04-reflections.md
7	Post-training data	SP-SFT: 300k single-turn examples	pipeline/sft/single_turn/, pipeline/eval_sets/	docs/06-posttraining-data.md

Stage 2 costs ~579 GPU-h of classifier inference and stage 6, the production reflection run, ~13,012 GPU-h. Stage 5 exists to settle the generator and its prompt before that budget is committed; docs/05-generator-selection.md records the four finalists scored on a shared 5,000-document pool.

Quickstart

cp env/config.example.sh env/config.sh && $EDITOR env/config.sh
uv sync                      # Python 3.13; add --group preprocessing for torch
bash tools/preflight.sh      # does everything config.sh points at resolve?
bash tools/smoke_test.sh     # the parts that need no cluster, GPU, or corpus

Then follow docs/07-reproduce.md.

tools/smoke_test.sh is worth running before you spend cluster time: the stages are long and unattended, so a broken tokenizer should surface in a minute.

Documentation

01-overview.md	The method, the artifact, repository map, and the model-raising → SPP name mapping
02-setup.md	Getting a working environment and adapting the jobs to your scheduler
03-corpus.md	Stages 1-4: download, safety annotation, subsampling, tokenization
04-reflections.md	Stage 6: the scale run, the run registry, canaries, merge
05-generator-selection.md	Stage 5: prompt iteration, the judge, human review, generator selection
06-posttraining-data.md	Stage 7: the SP-SFT mixture and the held-out evaluation sets
07-reproduce.md	The ordered runbook, what is replayable, and what is not
constitution.md	The constitution the reflections are written against

Where the method lives

What	File
Reflection-point sampling — the ramp-then-uniform position distribution	pipeline/tokenizer.py
Identity-canary assignment, deterministic in (canary_seed, doc_id)	pipeline/charter/scale/canaries.py
The run registry — one entry per paper variant, one prompt each	pipeline/charter/scale/runs.py
Generation loop, per-document retries, resumable shard writes	pipeline/charter/scale/generate.py
Parsing the generator's response into the reflection columns	pipeline/generation.py
Merging finished shards back into the sidecar	pipeline/charter/scale/merge.py
Sidecar construction, index-aligned with annotated.bin	preprocessing/tokenization/steps.py

The constitution (resources/ModelRaisingConstitution.md, 35 sections across 6 domains) and the canary facts (resources/canaries.yaml) are prompt inputs, which is why they sit outside pipeline/.

Published artifacts

The corpus tiers on the Hub, under dlab-spp:

Dataset	Size	Contents
dlab-spp/reflection-50m	51,386,305 rows, 155 GB	The reflections, with the document text and insertion point
dlab-spp/corpus-1T-manifest	16.9 GB	Which upstream documents the subsample kept, in order, per split
dlab-spp/corpus-verification	11.0 GB	Megatron .idx sidecars and checksums, to prove a rebuilt .bin matches
dlab-spp/safety-classifications	18.1 GB	The classifier's score for every document it saw

The same org hosts stage 7's outputs — the SP-SFT mixtures (sp-sft-normal-300k, sp-sft-safety-180k) — two smaller reflection subsets (reflection-10m, reflection-sample-2k), the evaluation sets, and the released models.

The corpus text itself is not redistributed. Reconstruction is a replay against upstream keyed by document id: take allenai/dolma3_mix-6T at its pinned revision, apply the manifest, and check the result against the verification tier. That is tens of gigabytes instead of ~1.3 TB, and it leaves the upstream license and takedown path intact. reflection-50m does carry text, because a reflection is unreadable without the document it reflects on.

Scope

This repository ends at the published datasets. Training — dataloaders, masking, launchers — is epfl-dlab/spp-training. There is no evaluation code here: no benchmark, no analysis, and the stage-5 judge scores candidate prompts, not models. AGENTS.md lists what else is absent, and why.

Configuration

env/config.sh (copied from env/config.example.sh, gitignored) is the single place site-specific values live: SLURM account and partitions, scratch roots, container TOMLs, the four corpus paths, and the generator weights. configs/config.yaml reads them back through ${oc.env:VAR,/path/to/placeholder} so the same YAML works unedited on another cluster, and tools/scan_secrets.sh enforces that no absolute site path or credential escapes env/. Everything else is an OmegaConf dotlist override on the CLI (charter.scale.max_rows=10000000), so trying a variant rarely needs a file edit.

License and attribution

The corpus derives from allenai/dolma3_mix-6T, released under ODC-BY 1.0. That license travels with every derivative — the published manifest, the reflection dataset, and anything you rebuild from them — and use of Dolma 3 is also subject to AI2's Responsible Use Guidelines. Cite Olmo 3 (arXiv:2512.13961) alongside this work. THIRD_PARTY_NOTICES.md collects every other obligation: the SafeLM classifier and its recontextualization templates, the canary-generation code this borrows from, and the gated AI2 prompt sources stage 7 samples.

The code in this repository is released under the MIT License (LICENSE). The ODC-BY obligation above is a property of the data and applies regardless.

Citation

See CITATION.cff.