Format Sensitivity in LLM Speaker Simulation

Code and data for the paper "Format Sensitivity in LLM Speaker Simulation" (snav, 2026).

Paper: PDF | Blog post: snavsoft.com

This repository contains the artifacts needed to reproduce the paper numbers and re-run experiments. Public data (Kaggle) is included; private data (Bench Warmers) is represented only by derived, text-free metrics.

Contents

• data/
  • kaggle_control/ -- public Kaggle JSONs (raw messages).
  • main_metrics_kaggle.csv -- per-sample, text-free metrics for Kaggle.
  • main_metrics_benchwarmers.csv -- per-sample, text-free metrics for Bench Warmers.
  • persona_prompt_metrics.csv -- text-free metrics for the Format x Persona prompt grid.
  • alternating_metrics.csv -- text-free metrics for the alternating format ablation (matched windows).
  • shim_text_metrics.csv -- text-free metrics for assistant-first shim text ablation (matched windows).
  • usertail_metrics.csv -- text-free metrics for user-tail format pilots (with/without system prompt).
  • prompt_pilot_metrics.csv -- text-free metrics for prompt-framing pilots.
  • human_meta_labels.csv -- text-free human metacommentary labels.
  • ids/ -- window ID lists used for matched analyses.
• scripts/
  • build_tables.py -- regenerates paper tables from the text-free metrics.
  • mixed_effects_stats.py -- mixed-effects logistic regression (random intercept by window_id) for the main Block vs Refill effect.
  • experiment1.py -- main experiment runner. Extracts windows, calls model APIs, computes metrics.
  • utils.py -- shared utilities: window extraction, conversation formatting, output guarding, simulation detection.
  • metrics.py -- metric computation: parroting detection, ghost speakers, refusal detection, format classification.
  • run_generation.py -- convenience wrapper for experiment1.py with sensible defaults.
  • build_public_data.py -- builds the text-free metrics CSVs from raw experiment outputs.
• metrics/ -- regenerated tables and summary JSON (output of build_tables.py).
  • Includes shim_text_summary.csv and usertail_summary.csv.
• .env.example -- template for API keys needed to run experiments.

Quickstart: Verify Paper Tables

pip install -r requirements.txt
python scripts/build_tables.py \
  --data-dir data \
  --out-dir metrics

Optional (mixed-effects stats):

python scripts/mixed_effects_stats.py \
  --data-dir data \
  --out-path metrics/mixed_effects_stats.json

This regenerates the table CSVs and metrics/summary.json.

Run Experiments

To re-run the experiments yourself, you will need API keys for one or more of: Anthropic (Claude), Google (Gemini), OpenRouter (Kimi).

cp .env.example .env
# Edit .env with your API keys

python scripts/run_generation.py
# Prints the command it would run (dry run by default)

python scripts/run_generation.py --execute
# Actually calls the APIs (costs money)

See run_generation.py --help for options (model selection, window size, sample count, etc.).

Useful ablation examples:

# User-tail style block formatting (name suffix in user message, no assistant prefill)
python scripts/run_generation.py \
  --conditions block \
  --block-prefill-position user \
  --system-prompt "Respond to the chat, where your username is shown as {bot_name}. Only respond with the content of your message, without including your username."

# Refill shim-text variant
python scripts/run_generation.py \
  --conditions refill \
  --assistant-first-user-stub "[conversation begins]"

Rebuild Metrics from Raw Outputs

If you have raw experiment outputs (e.g. from running experiments above):

python scripts/build_public_data.py \
  --main-csv <path-to-raw-outputs.csv> \
  --out-dir data

Then regenerate tables as above.

Embedding Columns

The main metrics CSVs include two embedding-based columns used for the automated metacommentary analysis (§2.9, Tables 15–16):

• meta_centroid_sim — cosine similarity between the output embedding and a centroid of 20 curated metacommentary phrases (Appendix A).
• chat_excl_centroid_sim — cosine similarity between the output embedding and the mean embedding of all non-target-speaker messages in the window.

The voice delta used in the paper is: vd = chat_excl_centroid_sim - meta_centroid_sim. Outputs with vd < -0.03 are classified as strong metacommentary. This threshold was calibrated against 100 human-labeled outputs (§3.7.5).

Embeddings were computed with BAAI/bge-base-en-v1.5 via sentence-transformers.

Notes

• No transcript text is included in any CSVs in data/.
• Kaggle JSONs are included because they are public and necessary to rebuild windows from scratch.
• Bench Warmers text data is not included.