TrainSight: Pre-Flight LLM Compute & Representation Profiler

trainsight is a sub-millisecond pre-flight data validation framework, CLI tool, and Kubernetes InitContainer. It profiles LLM pre-training and post-training dataset manifests (SFT, DPO, GRPO) before allocating GPUs, predicting HBM Out-of-Memory (OOM) crashes, padding memory waste, throughput degradation, and attention entropy dispersion in <0.01 ms CPU execution time.

---

🔬 The Core Scientific Discovery: The Two-Factor Model of LLM Compute

TrainSight is built on the Two-Factor Law of LLM Compute (ICLR Submission Draft), proving that LLM compute efficiency and representation stability under hardware constraints are governed by two independent failure modes:

1. Failure Mode 1: Batch Padding Memory Waste $\to$ Driven independently by sequence-length variance ratio ($\sigma/\mu$).
2. Failure Mode 2: Throughput Collapse & Attention Entropy Inflation $\to$ Driven independently by tail quantiles ($P_{99}$).

The 2x2 Phase Transition Matrix

	Low $P_{99}$ (Short Tail, $< 500$ tok)	High $P_{99}$ (Long Context Tail, $> 1,000$ tok)
Low $\sigma/\mu$ ($< 0.40$)	Clean Execution (GSM8K, FineWeb) • Waste $<15%$ | Throughput $>1,800\text{ tok/s}$ | Sharp Entropy ($H \approx 1.45$)	Throughput Collapse & Entropy Inflation (ShareGPT) • Waste $<20%$ | Throughput drops $>55%$ ($<900\text{ tok/s}$) | Entropy inflates ($H > 1.80$)
High $\sigma/\mu$ ($> 0.75$)	Padding Memory Waste Only (Alpaca) • Waste $>40%$ | Throughput $>1,400\text{ tok/s}$ | Sharp Entropy ($H \approx 1.25$)	Dual Catastrophic Failure / OOM Crash (Heavy-Tail Corpus) • Waste $>53%$ | Throughput drops $>80%$ | HBM OOM Crash

---

⚡ Comparison: Before vs. After (TrainSight Pre-Flight Evolution)

Feature / Dimension	Before (Old Pre-Flight Heuristics)	After (TrainSight Two-Factor Framework)
Scientific Paradigm	Single-metric assumption (assumed $\sigma/\mu$ caused everything).	Disentangled Two-Factor Law: $\sigma/\mu \to \text{Padding Waste}$; $P_{99} \to \text{Throughput Collapse & Entropy Inflation}$.
Failure Detection	Reactive: Jobs crash on GPUs after hours of execution.	Proactive: Predicts OOM crashes & waste in $0.0065\text{ ms}$ CPU latency before GPU allocation.
OOM Prediction Accuracy	Heuristic guesses / manual trial and error.	Empirical $F1 = 1.0000$ accuracy ($100%$ precision & recall) across synthetic and production datasets.
Representation Stability	Ignored representation quality; assumed static data is fine.	Tracks Attention Entropy Inflation ($H(A)$) caused by padding token dispersion.
Mitigation Strategy	Truncate sequences manually, risking data loss.	Zero-Loss Length Packing / Sorting: Recovers $+128.4%$ throughput without dropping a single sample.
Production Integration	Standalone Python scripts.	CLI + Kubernetes InitContainer: Gates pod scheduling before GPU nodes are billed.

---

📊 Master Empirical Benchmark Summary

Evaluated on EleutherAI/pythia-160m across controlled synthetic grids and 5 production datasets:

Dataset	Realized $\sigma/\mu$	$P_{99}$ Tail	TrainSight Risk	Actual Pad Waste %	Actual Attention Entropy $H(A)$	Throughput (tok/s)	Empirical Status
GSM8K	0.280	242 tok	MODERATE	20.32%	1.4544	1,961.7	Clean Execution
HumanEval	0.316	243 tok	MODERATE	27.84%	1.4045	1,798.3	Clean Execution
FineWeb-Edu	0.236	525 tok	LOW	17.43%	1.6037	1,649.7	Clean Execution
ShareGPT	0.377	1,077 tok	MODERATE	26.56%	1.8365	867.5 (-55.8%)	Throughput Collapse
Alpaca	0.802	228 tok	CRITICAL	41.51%	1.1808	1,402.8	Padding Memory Waste

---

💻 CLI & K8s InitContainer Usage

1. Profile Datasets via CLI

# Profile an SFT dataset manifest
trainsight profile --dataset sample_data/sample_sft.jsonl --type sft

# Profile a DPO preference dataset
trainsight profile --dataset sample_data/sample_dpo.jsonl --type dpo

# Profile GSM8K math reasoning data
trainsight profile --dataset sample_data/sample_gsm8k.jsonl --type sft

2. Kubernetes Pre-Flight InitContainer

initContainers:
  - name: trainsight-preflight-checker
    image: trainsight/checker:latest
    command: ["trainsight", "validate", "--manifest", "/data/sft_manifest.json", "--max-waste-pct", "30"]

---

📂 Research Documentation Links

• 📜 ICLR Paper Draft Manuscript
• 📊 Master Benchmark Results Log
• 📝 Research Runner & Tradeoffs Log
• 🏁 Project Milestones Tracker