perf(cache): answer a home-slot hit without the walk, and fold the key with CRC - #2277
Open
ArthurZucker wants to merge 1 commit into
Open
perf(cache): answer a home-slot hit without the walk, and fold the key with CRC#2277ArthurZucker wants to merge 1 commit into
ArthurZucker wants to merge 1 commit into
Conversation
…y with CRC Three things the hit path was paying for and did not need: - The walk started at step 0 of a 16-slot window even though nearly every hit is in the home slot. A slot whose key equals ours holds our word -- for a short word the key IS the word -- so that case is one load and one compare. - The hit path copied the whole 32-byte entry and then indexed the same slot a second time to borrow its ids. - The key is a fixed 16 bytes, which hardware CRC folds in two instructions where a general hasher runs its full schedule. CRC gives 32 bits; one multiply spreads them over 64 so the low bits still place the slot and the high bits still make the tag, and the fold stays seeded from `hasher`. Warm lookup over the pretokens of a 4 MB corpus, aarch64, gpt2 / llama-3: key 1.75 -> 1.30 ns, probe 3.09 -> 2.47, total 4.80 -> 3.73 (llama-3 4.84 -> 3.50). End-to-end this is currently noise: at 4 MB the lookup already overlaps with the split and the emit, so it only becomes visible once the per-pretoken call boundary goes away (see #2267). It is strictly less work either way.
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three things the warm hit path pays for and does not need. Independent of #2276 (which fixes
pack_word's copy) — both apply.slot. A slot whose key equals ours holds our word — for a short word the key is the word, so
there is nothing left for the tag to rule out. That case becomes one masked index, one load and
one 16-byte compare.
its ids.
hasher runs its whole mixing schedule. CRC yields 32 bits; one multiply spreads them over 64, so
the low bits still choose the slot and the high bits still make the tag. The fold stays seeded
from
hasher, so the table is as randomised as before. Non-aarch64 keeps aHash.Measured
Warm lookup over every pretoken of a 4 MB corpus, aarch64, 100% hit rate, ns per word:
End-to-end this is currently noise (493 vs 506 MB/s), and I would not claim otherwise: at 4 MB
inputs the lookup already overlaps with the split and the emit, so an isolated lookup benchmark
overstates its own contribution. It is strictly less work per lookup either way, and it becomes
visible once the per-pretoken call boundary is gone — which is the actual bottleneck:
With the model stubbed to answer every pretoken with one token, the model stage still costs
0.64 ns/B on english — exactly what gigatoken's entire encode costs. The per-pretoken virtual
call, slice,
Resultand push are the 5×, not the cache. #2267's batchedtokenize_keyed_spansisthe fix for that; this PR is the hit path underneath it.
Things I measured and did not propose, for the record: prefetching the next word's slot
(−19%, it recomputes the key and the table is L2-resident), batching keys + prefetching with the
hash already computed (1.0×), branchless flat emit (noise), and raising capacity 10k → 1M (flat to
slightly worse).