feat: train-encode split - #2119
Open
McPatate wants to merge 57 commits into
Open
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com>
McPatate
force-pushed
the
feat/train_encode_split
branch
from
June 18, 2026 14:27
6ca91df to
4404c11
Compare
SBrandeis
reviewed
Jun 23, 2026
| @@ -1,7 +1,12 @@ | |||
| [workspace] | |||
| members = ["tk-encode", "tk-train"] | |||
Contributor
There was a problem hiding this comment.
genuine question: do we want to also add the python bindings to the workspace?
could help with the DX
Member
Author
There was a problem hiding this comment.
don't think so, I think it's best we leave the current split as such, but no strong opinion on this, python bindings are a special case binding
SBrandeis
reviewed
Jun 23, 2026
* fmt * fix ci * re-generate readmes * readme check fix * restructure rust workflow * Audit & quality for node bindings * needs to generate cargo.lock * cleanup comments * cargo install fix * iwip * commit lock files + harmonize workflows * cache build artifacts and installed binaries
* factor byte_level constants in utils * also factor BYTES_CHAR * move tests * lint
* feat: impl new pretok for bert Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com> * refactor: change alphabet signature Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com> * feat: add new whitespace pretok Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com> * refactor: remove unneeded code & comments Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com> * feat: add lookup table for ascii fast path classification Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com> --------- Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com>
* bucket draft commit * we need a mapp then fo check which token * we know we have to sort it at the end * buckets not vec buckets and then copy byte set values We do have a bit of work to do to mach some of the smallVec features. For now I am implementing, we'll see what I am doing is most probably not super optimal (espcially having to copy from slice into a newly allocated buffer?) * box dyn ... * boxed_slice * start flattening the datastructure * update * small updates to try and compile * compiles! * fix slice len copy * nits * use generic const * this was a bad idea * create vocab store * use ptr_hash * skip some tests for compilation * nit * add id to token * update * fix * well this was ai assisted :) * default and debug * learn to stop I have to go * add match_bytes func * clippy * fix index * clippy * get vocab bytes * nits * fix * omment * fix compilation * cleanup * fix compilation * nit * fix some indexing * fixes * more fixes esp en empty entries * fix ID value * simpler debug * first implem :) * handle many buckets case * details * fix * bunch of prints! * loop was stuck * for now build splits out of the split end and start * use VocabStore in BPE model and UnigramModel * match bytes needs to return token id and token length! * poc: fast, light, allocation-free BPE encode prototype + benches A from-scratch inference-only byte-level BPE encode path and the benchmarks behind the perf investigation. 14-18x over HF tokenizers (main) on byte-exact models, ~10x smaller binary, ~3x less RAM, 0 hot-path allocations, and a 56x multi-turn re-encode prefix cache. - poc/fast-encode: final encoder + benches (stage profile, model x task sweep, parallel scaling, multi-turn prefix cache, splitter shootout). NEON-DFA split, MPHF VocabStore, allocation-free hybrid merge, thread-local pretoken cache, IREE-style ring buffer. - poc/edge-minimal: C-free, allocation-free build for on-device (0.37 MB stripped). - poc/special-token-matcher: MPHF length-probe vs IREE-style scan vs daachorse. - poc/scripts: tokenizer downloader + apply_chat_template workload generators. - poc/ENCODE_PERF_CASE.md: write-up + the centralize-in-tokenizers case. Research prototype; byte-exact on the 12 GPT-2-byte-level models tested. * where i am at * updates * fi * fix byte match * fixes * fix * nits here and there * tedious fixes * nits types.rs * small update * loads of todo in constructing the struct ± * update * just fix warning for now * add a manual test :) * fix logic, add match bytes test * more tests (esp nibble case) * fis * better func * fix the test ! * add a small todo * memchr2,3 are slow actually * bench against daachorse * big update: faster than daachorse up to 90% density of special tokens This was fun to work on! The key is that the rejection was still very slow on startswith(). This was looping and we where potentially storing too long prefix -> vs now u64 & u64 which is efficiient and fast. This is the final nail for this splitting being fast. * renamed buckets<-types and remove POC * remove dummy * unused * add rstrip and lstrip * nits * naive matcher is IREE's style match * clean * add single word * updates * remove the bleuprint * fix CI: cross-arch compile, clippy, fmt - gate nibble_match_bytes + its test to aarch64 (x86/cross-compile build was failing on a missing method) - add Unigram::is_empty (clippy len_without_is_empty) - clippy --fix: needless_return, redundant_field_names, needless_borrow, len_zero, doc continuations - cargo fmt --all Normalized-matching / add_tokens tests still fail by design (extract_and_normalize WIP). * fix tests * nit * small todos * my comments * updates * Apply suggestions from code review Co-authored-by: Luc Georges <McPatate@users.noreply.github.com> * nits here and thre * fix * skip hand rolled byte checks .... * up * add "extract_next" API * simple is word, lstrip rstrip * nits * fix them up * use merge word char * isolate added vocab * clippy --------- Co-authored-by: Luc Georges <McPatate@users.noreply.github.com>
* lint (cherry picked from commit 2d8adb2) * fix vocabstore partialeq (cherry picked from commit 927334c) * fix doctest (cherry picked from commit 2247647) * no fail-fast in CI workflow (cherry picked from commit 6371e9c) * fix export * refactor: park #2129 fast-encode path, restore legacy Tokenizer on the base Split #2129's token store in two: a legacy map-backed `VocabStore` (`crate::vocab_store`) that backs the models on this base, and the verbatim MPHF store renamed to `BucketVocabStore`. The models are unchanged — their `use crate::vocab_store::VocabStore` now resolves to the legacy twin, so the fast store can be swapped back in by the pipeline PR via a one-line alias. Restore the pre-#2129 PreTokenizedString `AddedVocabulary` as the active tokenizer AV, and park the #2129 fast path — `BucketVocabStore`, `Buckets`, and the bucket AV (renamed `bucket_added_vocabulary`) — unwired for the pipeline PR. The parked modules are not re-exported at the crate root so they don't collide with the legacy AV. This makes the base the genuine legacy Tokenizer, a fair A/B baseline against PipelineTokenizer. Models and the tk-train trainer stay byte-identical to #2129. The only #2129 code removed: its broken `extract_and_normalize` stub (replaced by a working two-pass), the `encode_special_tokens` default (true->false), and the legacy-tokenizer wiring of the new AV (reverted). BucketVocabStore is #2129 verbatim plus the on-branch PartialEq/doctest fixes, with expanded tests. tk-encode: 219 pass / 0 fail / 2 ignored, 20 doctests, clippy clean; workspace builds. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor: park #2129 fast-encode path, restore legacy Tokenizer on the base Split #2129's token store in two: a legacy map-backed `VocabStore` (`crate::vocab_store`) that backs the models on this base, and the verbatim MPHF store renamed to `BucketVocabStore`. The models are unchanged — their `use crate::vocab_store::VocabStore` now resolves to the legacy twin, so the fast store can be swapped back in by the pipeline PR via a one-line alias. Restore the pre-#2129 PreTokenizedString `AddedVocabulary` as the active tokenizer AV, and park the #2129 fast path — `BucketVocabStore`, `Buckets`, and the bucket AV (renamed `bucket_added_vocabulary`) — unwired for the pipeline PR. The parked modules are not re-exported at the crate root so they don't collide with the legacy AV. This makes the base the genuine legacy Tokenizer, a fair A/B baseline against PipelineTokenizer. The parked bucket path stays #2129 verbatim, minus dead/broken bits: - `BucketVocabStore` = #2129's store + on-branch PartialEq/doctest fixes, with expanded unit tests. - the bucket AV's `extract_and_normalize` (and its stub-only tests) is dropped — the PipelineTokenizer drives the AV solely through `extract_next` (the `PipelinePatternMatcher` trait); `extract_and_normalize` is only the legacy TokenizerImpl's entry point and uses the legacy AV. - `extract_next` had an inverted vocab selection (#2129 used `self.vocab` for normalized text); fixed to search `normalized_vocab` when normalized and `vocab` otherwise, and covered with tests (routing, offsets, lstrip/rstrip, single_word). Models and the tk-train trainer stay byte-identical to #2129. The only other #2129 code removed is the legacy-tokenizer wiring of the new AV, reverted so the legacy AV is active again. tk-encode: 216 pass / 0 fail / 1 ignored, 20 doctests, clippy + fmt clean; workspace builds. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* plumbing: full pipeline * lint * extract special tokens twice * lint * ai-generated: compare pipelines * ai-generated: pipeline stage analysis * wip: no vecs for splits * ai-generated: examples update * change special token matching api * iterator formulation * rm examples * lint * rm unused * cleanup * ai-generated: bench + correctness test * inline loop + comments * ai-assisted: docstring * lint * wire PipelineTokenizer onto the parked bucket AddedVocabulary Rebased onto feat/train_encode_split, which parked the #2129 fast path (BucketVocabStore / Buckets / bucket_added_vocabulary) and restored the legacy Tokenizer as the A/B baseline. This collapses the three stale fixup commits ("rebase", "attempt to fix", "apply normalizer in legacy tokenizer") from the previous rebase — they targeted the pre-park base — into one coherent step: - bucket_added_vocabulary.rs: the final fast AddedVocabulary (Buckets-backed, extract_next / extract_and_normalize, impl PipelinePatternMatcher). - PipelineTokenizer uses the bucket AddedVocabulary; the base Tokenizer stays legacy. TryFrom<&Tokenizer> rebuilds the bucket AV from the tokenizer's added tokens in id order, so ids are preserved (model-present tokens reuse their model id) and the pipeline emits the same ids as the reference tokenizer. pipeline_oracle passes: identical ids on big.txt (English) and wagahai (Japanese) at 1kB/10kB chunks. Full tk-encode/tk-train suites green; fmt + clippy -D warnings clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * remove unused extract_and_normalize from bucket AddedVocabulary The PipelineTokenizer drives special-token matching through `extract_next` (`PipelinePatternMatcher`) + `SpecialSegmentIterator`, so the bucket AV's `extract_and_normalize` / `split_on_matches` pair is dead code (only its own test used it). Drop them and the now-unused `Range`/`PreTokenizedString`/`Token` imports. The shared helpers (`is_ws`, `is_single_word`, `skip_whitespace_*`) stay — `extract_next` uses them. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * remove extract_two_pass test harness and its tests from bucket AV Drop the test-only `extract_segments` / `extract_two_pass` / `owned` helpers and the 7 extraction tests built on them. `extract_two_pass` reimplemented `PipelineTokenizer::encode`'s two-pass loop inside the test module (a drift-prone duplicate); `extract_next`'s matching behavior is exercised end-to-end by pipeline_oracle. Kept the AV-level unit tests that don't go through the harness: can_add_tokens, can_add_special_tokens, normalized_tokens_are_stored_by_normalized_form. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test bucket AV extract_next through SpecialSegmentIterator Restore the extract_next coverage dropped with the extract_two_pass harness, but drive the real `SpecialSegmentIterator` instead of a parallel reimplementation. Five single-pass tests over a bucket `AddedVocabulary`: raw added-token carving, single_word, lstrip/rstrip span absorption, the encode_special_tokens toggle, and raw-vs-normalized matcher selection. `SpecialSegmentIterator::new` is now pub(crate) so the AV tests can construct it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* update makefile * ai-generated: comparative fixture bench + CI Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * ai-generated: hardware + timestamp + revision --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…ons (#2145) Move the Tokenizer-vs-PipelineTokenizer report out of a PR comment and into a marker-delimited section of the PR description, kept current in place (upsert_pr_section.py). Pushes to feat/train_encode_split now target PR #2119; /pipeline-bench dispatch targets its own PR. Add emoji feedback to the comment-triggered flow: 👀 when the trigger fires, 👍 from the bench workflow on success. Add a cancel-in-progress concurrency group keyed on github.ref so a newer commit/comment supersedes an in-flight run for the same target without cancelling other PRs' benches. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
SBrandeis
added a commit
that referenced
this pull request
Jul 3, 2026
Comparative Tokenizer-vs-PipelineTokenizer bench that maintains a
marker-delimited section (graph + table) in the target PR's description,
via .github/scripts/upsert_pr_section.py (replace-in-place, else append).
Triggering: issue_comment workflows only run from the default branch, so a
"/pipeline-bench" comment can't fire before this lands on main. Use a
pull_request:[labeled] trigger ("run-pipeline-bench") instead — it runs the
PR branch's own workflow, so it works pre-main. The run itself is the PR
check (no manual check-run API, no emoji reactions), the label is auto-removed
so re-adding re-runs, and the target PR is resolved from the event (labeled PR
/ dispatch input / #2119 for pushes to feat/train_encode_split). Same-repo PRs
only — fork PRs get no secrets and a read-only token.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
SBrandeis
added a commit
that referenced
this pull request
Jul 3, 2026
Comparative Tokenizer-vs-PipelineTokenizer bench that maintains a
marker-delimited section (graph + table) in the target PR's description,
via .github/scripts/upsert_pr_section.py (replace-in-place, else append).
Triggering: issue_comment workflows only run from the default branch, so a
"/pipeline-bench" comment can't fire before this lands on main. Use a
pull_request:[labeled] trigger ("run-pipeline-bench") instead — it runs the
PR branch's own workflow, so it works pre-main. The run itself is the PR
check (no manual check-run API, no emoji reactions), the label is auto-removed
so re-adding re-runs, and the target PR is resolved from the event (labeled PR
/ dispatch input / #2119 for pushes to feat/train_encode_split). Same-repo PRs
only — fork PRs get no secrets and a read-only token.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Comparative Tokenizer-vs-PipelineTokenizer bench that maintains a
marker-delimited section (graph + table) in the target PR's description,
via .github/scripts/upsert_pr_section.py (replace-in-place, else append).
Triggering: issue_comment workflows only run from the default branch, so a
"/pipeline-bench" comment can't fire before this lands on main. Use a
pull_request:[labeled] trigger ("run-pipeline-bench") instead — it runs the
PR branch's own workflow, so it works pre-main. The run itself is the PR
check (no manual check-run API, no emoji reactions), the label is auto-removed
so re-adding re-runs, and the target PR is resolved from the event (labeled PR
/ dispatch input / #2119 for pushes to feat/train_encode_split). Same-repo PRs
only — fork PRs get no secrets and a read-only token.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ai-geenrated: render script grid * ai-geenrated: update GH workflow * ai-geenrated: run bench on several different tokenizers * ai-generated: restore full info + full tokenizers fixtures
Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com>
* feat: `impl pipeline::PreTokenizer for FixedLength` Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com> * feat: add `FixedLength` to `PipelinePreTokenizer` Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com> --------- Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com>
* feat: `impl pipeline::PreTokenizer for Digits` Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com> * feat: add `Digits` to `PipelinePreTokenizer` Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com> * feat: add extra test for a string of digits Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com> --------- Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com>
* feat: `impl pipeline::PreTokenizer for CharDelimiterSplit` Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com> * feat: add `Delimiter` to `PipelinePreTokenizer` Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com> --------- Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com>
* feat: `impl pipeline::PreTokenizer for UnicodeScripts` Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com> * feat: extend ascii lut table to basic multilingual plane Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com> * feat: add `UnicodeScripts` to `PipelinePreTokenizer` Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com> --------- Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com>
* feat: `impl pipeline::PreTokenizer for WhitespaceSplit` Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com> * feat: add `WhitespaceSplit` to PipelinePreTokenizer` Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com> --------- Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com>
* feat: handle `Merge*` variants of the `SplitDelimiterBehavior` Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com> * feat: add `Punctuation` to `PipelinePreTokenizer` Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com> --------- Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com>
* Implement Cow-based normalizers * ai-generated: bert normalizer * lint * implement precompiled * fallible + more nocopy * DRY * ai-assisted: rewrite bert hotpath
…ushed absolute bar (#2185) The stage-decomposition chart scaled every fixture's stacked bar to a shared ns/byte range that included the release's whole-encode tick (1000/baseline_mbps). Because the released tokenizer is slow, that tick is a huge ns/byte value, so it dominated the scale and squashed the pipeline's actual stage breakdown into an unreadable sliver. Make it a 100%-normalized bar instead: each stage is its share of THAT fixture's own pipeline total, labelled `share% · ns/B`, so the mix is readable no matter the absolute cost. The magnitude and the comparison aren't lost — the right column keeps `total ns/B · ×speedup`. Dropped the now-moot absolute scale (`stage_scale`), the release tick (`baseline_ns_per_byte`), and the `max_total` plumbing. Also restore per-stage columns to the numbers table as `share% (ns/B)` (added-token / normalize / pre-tokenize / model), so the split cost is legible as text too.
* bench(pipeline): multi-thread scaling sweep (1/2/4/8/max) vs the release
fixture_bench now runs a per-model multi-thread throughput sweep — the pipeline
vs the released `tokenizers` crate at 1, 2, 4, 8, and device-max threads over the
whole corpus (a private rayon pool per count so the sweep neither perturbs nor is
perturbed by the global pool; thread-spawn/scheduling overhead amortized). Emitted
as `threads: {counts, pipeline_mbps, baseline_mbps}` in the JSON.
render_pipeline_bench.py renders a per-model "Thread scaling" chart — throughput
bars at each thread count, pipeline vs release on a shared axis, with the ×speedup
per count and the 1→max scaling factor — placed in each model's <details> block.
The workflow's SVG->PNG glob (`pipeline_bench_*.svg`) picks the new charts up
automatically, so no workflow change is needed.
* bench(pipeline): add ideal-linear reference to the thread-scaling chart
Per-row tick at single-thread throughput × N on the pipeline bar (bar reaches
tick = linear, falls short = sub-linear), plus the self-scaling % of linear in
the right column and the 1→max scaling factor in the subtitle — so the chart
answers 'are we scaling linearly?' directly, not just 'how fast at N threads'.
* ci(pipeline-bench): fan models out across a matrix to run shards in parallel
The multi-thread sweep pushed the single-job bench to ~15 min. Split the work:
a `bench` matrix (4 shards over contiguous manifest slices) runs each slice on
its own isolated 8-vCPU runner in parallel — throughput + thread sweep + memory
for its models — and uploads a partial JSON. A single `report` job (needs:
bench) concatenates the partials in shard = manifest order, then does binary
size + render + HF chart upload + PR-description update once.
fixture_bench gains `--shard <i> <n>`: bench only the i-th of n contiguous
manifest chunks (absent → (0,1) = the whole manifest, unchanged). Shard slices
are contiguous so concatenating in shard order preserves manifest order.
Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com>
Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com>
McPatate
force-pushed
the
feat/train_encode_split
branch
from
July 13, 2026 13:34
2055ff7 to
93650e3
Compare
* pipeline-bench: add "vs base branch" comparison The pipeline benchmark only compared PipelineTokenizer against the latest released crate. Add a "vs base branch" overview + per-fixture "Δ base" column so a PR's own wins/regressions against feat/train_encode_split are visible (regressions in red). Baseline is cached, not recomputed per PR: - a push to feat/train_encode_split uploads its merged bench JSON to the HF Hub dataset as baselines/pipeline-<sha>.json and force-moves the lightweight `pipeline-baseline` git tag to that commit (only after the id check passes, so a broken base never becomes the baseline); - a PR run resolves the tag, downloads the cached JSON, and diffs against it. No tag / no cached JSON yet -> the base overview is skipped and the release charts render as before. render_pipeline_bench.py: generalize overview_svg/scale over a speedups fn; new base_speedup helpers; --base-bench/--base-ref; base overview (red on regression) + Δ base table column, both gated on base data being present. * pipeline-bench: fall back to base-branch artifact when no baseline tag The "vs base branch" section never showed on the PR that introduces it: the `pipeline-baseline` tag is only created by a base-branch push running this workflow, which can't happen until the workflow is merged — so the feature was invisible during its own review (chicken-and-egg). Add a fallback: when the tag (or its HF Hub JSON) is absent, resolve the baseline from the newest successful base-branch bench artifact via `gh run download`. That artifact is produced by every base-branch run, so the comparison shows up immediately — no manual seeding, no merge required. The tag + HF Hub JSON stays the durable primary (survives the 30-day artifact retention). Needs `actions: read` to download the artifact.
* move some stuff around
* nits
* draft fast-splitting
* small updates
* fix!
* update
* fix the small test
* nits
* start todos ands tuff
* finally I get it!
* update
* small updates
* nits
* update
* update
* nits
* update
* regex vs matcher bench
* update
* run end was a wrong approach, use step!
* update
* add the final specs for the new pretokenization paradigm. its SIMD first, classify first then FSM simd when possible, pure scalar for known patterns!
* update
* push lock
* fix conflicts
* fixuppppppppppppppppppppppp
* update
* turbofish
* chore: move sub crates into main workspace
Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com>
* remove
* first draft
* nits
* update
* cool
* commit
* fix(fast_split): correct bitmap is number & split _simple_test in 2
Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com>
* ai draft need to pulls
* save current cleanup
* peel vs looop
* clean
* multi arch
* ai help
* updae
* wasm SIMD128 classify path
Add simd_wasm_classify.rs: wasm32 SIMD128 port of the classify kernel,
generic over TagScheme. u8x16_swizzle is a 16-entry shuffle with OOB->0
(same as NEON vqtbl), so the subtract trick ports directly; native
unsigned compares (u8x16_ge/le) and per-lane byte shifts make it a near
1:1 map of the NEON body. Same tables, same algorithm, 16 bytes/iter.
Gated on simd128 (WASM has no runtime feature detection); non-simd
builds fall back to classify_scalar via the classify.rs dispatcher.
Cross-compiles clean; runtime validation (== classify_scalar) pending
on a SIMD128 engine.
* update
* cleanup simd code
* update
* small xleanup one by one
* draft
* update
* update
* add cl100k
* update
* proper bench
* update
* up
* remove legacy code
* add deepseek
* nits
* update
* update
* peel 2bytes as well
* update
* update
* update
* update
* readme
* cleanup the unecessary scritpt tables
* no push, write on pre allocated output
* more cleanup
* move tests and etc
* more cleanup, update the readme and the docs
* update
* update
* update
* start to move away from mcpotato's work
* update
* full update
* update
* update
* update
* tk-encode: route ByteLevel/Delimiter through atomsplit, gate fancy-regex
- ByteLevel now splits via atomsplit fsm_byte_level (byte-exact GPT-2 FSM,
shared GptFsmPattern adapter) instead of a fancy-regex SysRegex; no more
regex backend for the byte-level pre-tokenizer.
- CharDelimiterSplit pipeline path uses atomsplit's memchr-backed
CharDelimiterSplit (byte-exact).
- Split.regex is now Option<SysRegex>, built lazily: recognized GPT patterns
route to atomsplit (fsm/multi) and need no regex backend; both legacy and
pipeline paths fall back to atomsplit for GPT patterns when no backend is
compiled. With a backend present (default) behaviour is byte-identical.
- Stub SysRegex for builds with no fancy-regex/onig; drop the compile_error.
fancy-regex is now optional and droppable via --no-default-features, needed
only for arbitrary (non-GPT) Split regexes and the Replace normalizer.
- Remove unused regex-syntax dependency.
* up
* update
* fix mega slowness
* update with clippy and etc
* more cleanup
* update for qwen etc
* update
* nit
* clippy
* docs: refresh atomsplit/bitmap_gen structure docs
- add O200k / fsm_o200k to the implemented-pretokenizer / FSM lists (lib.rs, fsm.rs)
- fsm_deepseek doc: drop the stale `ds_is_cjk_letter` ref; document the closed-unit
CJK-range handling and the gap-grouping + ALPHA_SYM behavior
- bitmap_gen: a tag is a full u8 (low nibble coarse Atom, high nibble refinement), not a u4
- README: document the high-nibble refinement (o200k case on Letter, ALPHA_SYM on Mark)
- drop two broken `[class_runs_neon]` intra-doc links (pub(crate), unlinkable)
* atomsplit: classify from one current Unicode source; add o200k parity gate
atom() mixed Unicode versions: is_letter/is_mark came from unicode_categories 0.1
(frozen at Unicode 9.0) while is_alphabetic/is_whitespace/is_numeric bound to std
(modern). Post-9.0 letters (e.g. U+9FD6, CJK Ext-C astral) were is_letter=false yet
is_alphabetic=true, so they fell into the ALPHA_SYM branch and got tagged 0x16
(coarse Mark) — deepseek/o200k then routed them to \p{S} instead of the [\p{L}\p{M}]
letter run, diverging from modern-Unicode regex on real modern/astral CJK.
- swap unicode_categories -> unicode-properties (current); derive all general
categories from general_category(); keep std for the White_Space/Alphabetic/
Numeric properties (also current) — no stale-version mix
- regenerate atom_tables.rs (self-validated over all 1.1M codepoints)
- verified U+9FD6 / U+2B81D now classify as Letter (0x00); circled letters (Other_
Alphabetic \p{S}) stay ALPHA_SYM (0x16) so \w is unchanged; SIMD == scalar
- add permanent o200k_parity onig gate (the only regex-shaped FSM that lacked one)
No parity regression: cl100k/o200k/byte_level/deepseek gates all byte-exact.
* Drop redundant tk-encode/Cargo.lock; gitignore member-crate lockfiles
tk-encode is a workspace member, so cargo only ever reads the workspace-root
Cargo.lock — the committed tk-encode/Cargo.lock (~3.1k lines) was dead weight
inflating the diff with zero functional effect. Remove it and add a .gitignore
rule (`/*/Cargo.lock`) so member locks don't get re-committed, while leaving the
tracked workspace-root Cargo.lock untouched.
* fix(pipeline): drop duplicate imports left by the feat/train_encode_split merge
The #2183 merge added standalone `PipelineSequence` / `SplitPattern` imports,
which this branch already imports in the `use crate::{…}` block — E0252 "defined
multiple times", so the merge didn't compile. Keep the block imports (the
branch's style) and drop the standalone duplicates. #2183's WordPiece ->
PipelineWordPiece rename is retained.
* perf(pipeline): reuse a thread-local scratch across pre_tokenize calls
Every pipeline pre-tokenizer (the Split GPT-FSM path, Whitespace, WhitespaceSplit,
Punctuation, and the deepseek Sequence) allocated two fresh Vecs per call —
`vec![0u8; n]` for tags and `vec![(0,0); n+1]` for spans. On a special-token-dense
input the special scan carves the text into many tiny segments, so pre_tokenize
runs once per tiny segment and the per-segment malloc/free dominates the cost.
Route all of them through `pipeline::classify_into_spans`, which classifies and
runs the FSM into a grow-only thread-local scratch reused across calls (no
per-segment allocation). Byte-identical to the old per-call allocation.
~1.5x on tiny-segment pre-tokenization (WhitespaceSplit at 20 B segments:
5.30 -> 3.56 ns/B); unchanged for one-big-segment inputs. Byte-exact with the
reference (pipeline_oracle, both add_special_tokens values, bert-wiki over the
big + wagahai corpora).
* update comments
* start removing the scheme, we have found another solution that is much more lightwheight
This commit and the next are gonna be cleaning up the stupid AI slope
jk
* more refinement
* more cleanup
* clean benches
* more cleanup
* update bitmapgen comments
* nits
* pull
* split: route cl100k/o200k to their native FSM instead of regex fallback
cl100k and o200k ship their pre-tokenizer as Sequence[Split(invert=true,
behavior=Removed, <gpt-regex>), ByteLevel] (the tiktoken-conversion convention).
The pipeline's Split->FSM fast path only fires for (invert=false, Isolated), so
these two — the most common production vocabs — silently fell back to MultiRegex
(regex_automata), spending 11-18% of the whole encode in the regex DFA on plain
text with zero special tokens. gpt2 (bare ByteLevel, synthesized as Isolated)
and deepseek (own FSM path) were unaffected.
For a whole-covering GPT regex, (invert=true, Removed) is byte-exactly
equivalent to (invert=false, Isolated) — the inverted match set is the gaps, and
these patterns leave none. Canonicalize to that form at pipeline build
(Split::canonicalized_for_pipeline) when gpt_fsm recognizes the pattern, so
cl100k/o200k route to fsm_cl100k / fsm_o200k.
Measured on the poc-merge branch (identical split/pipeline code): single cold
pass +20..63% (cl100k, o200k); regex_automata share 18% -> 0.0%; byte-exact
(reference token-stream checksums unchanged, incl. Japanese).
* refine doc a bit more :)
* more details and more tables
* update
* update
* refactor the tail / run end to make it re-usable, cleaner and self contained
* small fixes, doc updates
* lol
* explanation
* update
* update
* more comments, better perfs by removing bound checks
* update
* update
* update
* cargo
* wasm cjk update
* nitw
* revert deepseek specific code.
* start to revert the multiregex
* more cleanup
* remove regex automatad
* unify span and splits
* thinner deps
* update
* small up
* update: split the different unrolled fsm for readability
* update
* update
* nits and ai disclaimer
* update
* cleanup some slop0e
* update
* unify benches
* clean
* more unified source of truth for unrolled regex
* update
* nits
* update
* update
* doc updates
* use jit for fair pcre benches
* add logos to the bench
* local bench pcre
* add logos on whitespace split compare
* add heatmap to the doc
---------
Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com>
Co-authored-by: Luc Georges <luc.sydney.georges@gmail.com>
* use yada (double-array trie) in wordpiece * experiment: different data structure for wordpiec * 1 less alloc * wip: shared scratch * wip: re-used scratch * lint * lint * bpe scratch * lint * word cache * fmt * simpler syntax * comment out cache (faulty) * cleanup
The Audit and READMEs jobs went red repo-wide on feat/train_encode_split (so on every PR against it): - audit: two advisories published since the ignore list was last updated — RUSTSEC-2026-0204 (crossbeam-epoch, transitive) and RUSTSEC-2025-0057 (fxhash, unmaintained). Both are transitive and not fixable from here, so ignore them alongside the existing three (rust/node/python workflows). - README: the root crate's lib.rs wrapped a sentence but README.md wasn't regenerated; `cargo readme` re-syncs it.
Drop the 'Lint Benchmarks with RustFmt' step: it passes the file
straight to rustfmt without the package's edition, so rustfmt (>= Rust
1.97) formats it with default 2024 style and contradicts what
'cargo fmt --all' (edition 2018 -> 2021 style) enforces on the same
file. The step is redundant anyway: benches are workspace targets and
already covered by --all.
Regenerate tk-encode/README.md with cargo-readme 3.3.3, which strips
doctest hidden lines ('# ...') that 3.3.2 leaked into the rendered
README.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* bump rust version to 2024 * fmt, clippy, fix sync in bpe trainer * bindings fmt * bindings lint * syntactic sugar nice * python fmt * fix: clippy in scalar path * unsafe + lock * fixmes
* fixture_bench: per-fixture cold caches, encode_fast baseline, phase separation - Throughput rows now measure what a plain .encode() loop over that one corpus reaches: the pipeline is rebuilt per fixture (fresh scratch pool -> fresh BPE word cache) and the baseline is cloned (released BPE Clone starts cold). Previously one shared insert-once cache accumulated across all 18 fixtures and saturated within the first one or two, so later rows depended on manifest order and ran against a frozen, mostly-foreign cache. - The baseline is timed through encode_fast everywhere (throughput, thread sweep, memory child, probe): the pipeline computes no offsets, so timing the baseline's offset-tracking encode would flatter it. - Phases are isolated: all warm throughput + id gates first, then the stage ladder + regex references (fresh caller-owned scratches), then the thread sweep and memory children. Stage numbers and headline numbers now share the same cache regime. - Debloat: onig/fancy/pcre2 references collapse into one SplitEngine trait + shared composed-split chain; fixtures are read and chunked once into a Fixture struct instead of three times. JSON schema is unchanged (render_pipeline_bench.py contract); verified on the gpt2 shard: all keys present, ids_match and ids_match_baseline green on all 18 fixtures. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * benches iter * oter --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* ci: faster? * try more tricks
* bucket vocab store in BPE * fix
* Implement: PipelinePostProcessor * display post-process in benches * Apply suggestions from code review Co-authored-by: Simon Brandeis <33657802+SBrandeis@users.noreply.github.com> * fmt * comments * todos
* decode bench + oracle + rewrite encode oracle * docs iter * Apply suggestions from code review Co-authored-by: Simon Brandeis <33657802+SBrandeis@users.noreply.github.com> * decode_batch + decode_stream surface * decode oracle: specials, pairs, stream, batch + one test per model x fixture * oracle sweeps add_special_tokens; benches encode with specials * simplify oracles: fixed windows, per-model tests, no macros * fix: doctest * charts: honest decode-memory rows, decode basis + flags spelled out * fix stubs? * ci: pin ruff, 0.16 breaks legacy python style
Open
1 task
…-4 (#2246) The manifest loaded gpt-oss-slim.json (3,060 of 199,998 vocab entries) and glm-5.2-slim.json (2,951 of 154,820), so their bench numbers ran on ~1.5% of the real vocab — unrealistically small merge tables and cache footprints. Point both at the full configs and add two archetypes: gemma-4 (262,144-vocab byte-fallback BPE, slim-only until now) and mistral-small-4 (tekken byte-level BPE with 1,000 added specials). The full configs land in hf-internal-testing/tokenizers-test-data via a pending PR (which also refreshes llama-2.json and llama-3-tokenizer.json to byte-exact copies of the source models), so `make bench-models` needs that merge to fetch them. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* perf: handroll tekken regex * wire in tk-encode
Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com>
* new literal module in AtomSplit * use Literal in Replace normalizer and Split pretok * box the Literal finder `memmem::Finder` carries a few hundred bytes of prefilter state on x86_64 (much less on aarch64, which is why this only showed up in CI): a `Literal` stored inline blew `NormalizerWrapper` and `DecoderWrapper` up to 352 bytes through `Replace`, which `clippy::large_enum_variant` rejects. * keep the &str / &String patterns, backed by Literal The Python bindings search with a `&String` pattern — `NormalizedString.replace` and `.split` take a plain `str` — so dropping these impls broke every job that builds the bindings. Reinstating them through `Literal` keeps the public API and still drops the regex engine from the literal path: they used to escape the string and compile a regex on every call. An empty pattern now covers the input by byte length rather than character count, like every other impl (the old count sliced mid-character).
* new literal module in AtomSplit * use Literal in Replace normalizer and Split pretok * box the Literal finder `memmem::Finder` carries a few hundred bytes of prefilter state on x86_64 (much less on aarch64, which is why this only showed up in CI): a `Literal` stored inline blew `NormalizerWrapper` and `DecoderWrapper` up to 352 bytes through `Replace`, which `clippy::large_enum_variant` rejects. * keep the &str / &String patterns, backed by Literal The Python bindings search with a `&String` pattern — `NormalizedString.replace` and `.split` take a plain `str` — so dropping these impls broke every job that builds the bindings. Reinstating them through `Literal` keeps the public API and still drops the regex engine from the literal path: they used to escape the string and compile a regex on every call. An empty pattern now covers the input by byte length rather than character count, like every other impl (the old count sliced mid-character). * Support Metaspace pre-tok * format * comments * lint
SBrandeis
force-pushed
the
feat/train_encode_split
branch
from
July 31, 2026 17:44
d183afe to
0bcf291
Compare
Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com>
* adds `EncodeHandle` return type to pave the way for async
* takes `impl Into<Inputs>` as an input
* pairs are not supported atm
Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com>
Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
splitting training and encoding logic into two separate crates. Should allow for overall perf improvements and reduce crate size significantly.
New additions that come with the PR:
Trainabletrait (tk-train/src/trainable.rs): Holds the type Trainer +get_trainer()that was stripped offModel.TokenizerTrainExttrait (tk-train/src/train_ext.rs): Thetrain/train_from_filesbodies existed before as inherent methods onTokenizerImplget_model_mut()(tk-encodeTokenizerImpl): So the train extension can hand&mut modelto a trainer across the crate boundary.pre_tokenize_for_training()(tk-encodeTokenizerImpl): Extracted verbatim from the old train-method bodies (normalize → pre-tokenize → splits). Keeps the closure on one&selfborrow instead of poking 3 private fields.tokenizers/src/lib.rsfile: Re-export tree that rebuilds the oldtokenizers::…paths.The rest should be unchanged apart from the import changes!EDIT: this PR is the base of the tokenizers v1 refactor now!PipelineTokenizer benchmark
10 / 10 models supported — PipelineTokenizer vs
tokenizersv0.23.1 (latest release) · ~10 kB inputs · add_special_tokens on · single thread + 1/2/4/8/max-thread sweepf3bf3e820 · 2026-08-04 13:58 UTC· Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz · 16 coresvs base branch (
c299c1ae0) — per-model geomean ×speedup of this PR's PipelineTokenizer against the base branch's; regressions in red.Decode
Round-trip: v0.23.1
encode_fastproduces the id streams (same fixtures,add_special_tokens=true); both implementations decode those SAME ids withskip_special_tokens=false. MB/s counts decoded text bytes.bert-base-uncased — normalizer-heavy WordPiece · ×5.15 vs v0.23.1 · ×0.71 vs base · decode pending
Memory (RSS MB, load+encode): v0.23.1 12+0 (peak 12) · Pipeline 8+0 (peak 17)
deepseek-v4 — deepseek 3-regex split-heavy byte-level BPE · ×4.72 vs v0.23.1 · ×0.77 vs base · decode pending
Memory (RSS MB, load+encode): v0.23.1 62+0 (peak 68) · Pipeline 82+0 (peak 82)
Pre-tokenize:
classify + fsmvs regex engines — ns/byte, lower better. The fsm is the scalar jump-table in both pipe columns; SIMD / scalar is the classify pass (regex pre-tokenizers have no SIMD fsm).×vs= engine ÷ our pipeline (SIMD / scalar classify);onig&pcre2(JIT) are C,fancyis pure-Rust fancy-regex,logosis a compile-time DFA lexer (approximate grammar; n/a for deepseek).gemma-4 — byte-fallback BPE, Metaspace-style split (gemma-4) · ×1.99 vs v0.23.1 · ×0.86 vs base · decode pending
Memory (RSS MB, load+encode): v0.23.1 304+0 (peak 371) · Pipeline 274+0 (peak 370)
gpt2 — gpt2 ByteLevel regex · ×9.02 vs v0.23.1 · ×0.99 vs base · decode pending
Memory (RSS MB, load+encode): v0.23.1 25+2 (peak 27) · Pipeline 28+0 (peak 28)
Pre-tokenize:
classify + fsmvs regex engines — ns/byte, lower better. The fsm is the scalar jump-table in both pipe columns; SIMD / scalar is the classify pass (regex pre-tokenizers have no SIMD fsm).×vs= engine ÷ our pipeline (SIMD / scalar classify);onig&pcre2(JIT) are C,fancyis pure-Rust fancy-regex,logosis a compile-time DFA lexer (approximate grammar; n/a for deepseek).gpt-oss — o200k-regex byte-level BPE (gpt-oss) · ×5.19 vs v0.23.1 · ×0.94 vs base · decode pending
Memory (RSS MB, load+encode): v0.23.1 241+0 (peak 315) · Pipeline 234+0 (peak 316)
Pre-tokenize:
classify + fsmvs regex engines — ns/byte, lower better. The fsm is the scalar jump-table in both pipe columns; SIMD / scalar is the classify pass (regex pre-tokenizers have no SIMD fsm).×vs= engine ÷ our pipeline (SIMD / scalar classify);onig&pcre2(JIT) are C,fancyis pure-Rust fancy-regex,logosis a compile-time DFA lexer (approximate grammar; n/a for deepseek).glm-5.2 — cl100k-variant regex byte-level BPE (glm-5.2) · ×6.68 vs v0.23.1 · ×0.95 vs base · decode pending
Memory (RSS MB, load+encode): v0.23.1 169+0 (peak 231) · Pipeline 170+0 (peak 232)
Pre-tokenize:
classify + fsmvs regex engines — ns/byte, lower better. The fsm is the scalar jump-table in both pipe columns; SIMD / scalar is the classify pass (regex pre-tokenizers have no SIMD fsm).×vs= engine ÷ our pipeline (SIMD / scalar classify);onig&pcre2(JIT) are C,fancyis pure-Rust fancy-regex,logosis a compile-time DFA lexer (approximate grammar; n/a for deepseek).llama-2 — model-bounded BPE, no pre-tokenizer · ×4.12 vs v0.23.1 · ×0.95 vs base · decode pending
Memory (RSS MB, load+encode): v0.23.1 18+0 (peak 23) · Pipeline 23+0 (peak 23)
llama-3 — cl100k-regex byte-level BPE (llama-3), single regex · ×7.18 vs v0.23.1 · ×1.04 vs base · decode pending
Memory (RSS MB, load+encode): v0.23.1 73+0 (peak 95) · Pipeline 93+0 (peak 95)
Pre-tokenize:
classify + fsmvs regex engines — ns/byte, lower better. The fsm is the scalar jump-table in both pipe columns; SIMD / scalar is the classify pass (regex pre-tokenizers have no SIMD fsm).×vs= engine ÷ our pipeline (SIMD / scalar classify);onig&pcre2(JIT) are C,fancyis pure-Rust fancy-regex,logosis a compile-time DFA lexer (approximate grammar; n/a for deepseek).mistral-small-4 — tekken byte-level BPE, 1k added specials (mistral-small-4) · ×5.52 vs v0.23.1 · ×1.03 vs base · decode pending
Memory (RSS MB, load+encode): v0.23.1 152+0 (peak 194) · Pipeline 109+0 (peak 195)
Pre-tokenize:
classify + fsmvs regex engines — ns/byte, lower better. The fsm is the scalar jump-table in both pipe columns; SIMD / scalar is the classify pass (regex pre-tokenizers have no SIMD fsm).×vs= engine ÷ our pipeline (SIMD / scalar classify);onig&pcre2(JIT) are C,fancyis pure-Rust fancy-regex,logosis a compile-time DFA lexer (approximate grammar; n/a for deepseek).t5-base — Unigram + Metaspace · ×2.57 vs v0.23.1 · ×1.02 vs base · decode pending
Memory (RSS MB, load+encode): v0.23.1 34+2 (peak 36) · Pipeline 61+1 (peak 66)