Skip to content

perf(cache): answer a home-slot hit without the walk, and fold the key with CRC - #2277

Open
ArthurZucker wants to merge 1 commit into
feat/bpe-cache-minfrom
perf/word-cache-probe
Open

perf(cache): answer a home-slot hit without the walk, and fold the key with CRC#2277
ArthurZucker wants to merge 1 commit into
feat/bpe-cache-minfrom
perf/word-cache-probe

Conversation

@ArthurZucker

Copy link
Copy Markdown
Collaborator

Three things the warm hit path pays for and does not need. Independent of #2276 (which fixes
pack_word's copy) — both apply.

  1. The walk starts at step 0 of a 16-slot window, even though nearly every hit is in the home
    slot. A slot whose key equals ours holds our word — for a short word the key is the word, so
    there is nothing left for the tag to rule out. That case becomes one masked index, one load and
    one 16-byte compare.
  2. The hit path copied the whole 32-byte entry and then indexed the same slot again to borrow
    its ids.
  3. The key is a fixed 16 bytes, which hardware CRC folds in two instructions where a general
    hasher runs its whole mixing schedule. CRC yields 32 bits; one multiply spreads them over 64, so
    the low bits still choose the slot and the high bits still make the tag. The fold stays seeded
    from hasher, so the table is as randomised as before. Non-aarch64 keeps aHash.

Measured

Warm lookup over every pretoken of a 4 MB corpus, aarch64, 100% hit rate, ns per word:

key probe total
gpt2 · english, before 1.75 3.09 4.80
gpt2 · english, after 1.30 2.47 3.73
llama-3 · english, after 1.39 2.11 3.50 (from 4.84)

End-to-end this is currently noise (493 vs 506 MB/s), and I would not claim otherwise: at 4 MB
inputs the lookup already overlaps with the split and the emit, so an isolated lookup benchmark
overstates its own contribution. It is strictly less work per lookup either way, and it becomes
visible once the per-pretoken call boundary is gone — which is the actual bottleneck:

With the model stubbed to answer every pretoken with one token, the model stage still costs
0.64 ns/B on english — exactly what gigatoken's entire encode costs. The per-pretoken virtual
call, slice, Result and push are the 5×, not the cache. #2267's batched tokenize_keyed_spans is
the fix for that; this PR is the hit path underneath it.

Things I measured and did not propose, for the record: prefetching the next word's slot
(−19%, it recomputes the key and the table is L2-resident), batching keys + prefetching with the
hash already computed (1.0×), branchless flat emit (noise), and raising capacity 10k → 1M (flat to
slightly worse).

…y with CRC

Three things the hit path was paying for and did not need:

- The walk started at step 0 of a 16-slot window even though nearly every hit is
  in the home slot. A slot whose key equals ours holds our word -- for a short
  word the key IS the word -- so that case is one load and one compare.
- The hit path copied the whole 32-byte entry and then indexed the same slot a
  second time to borrow its ids.
- The key is a fixed 16 bytes, which hardware CRC folds in two instructions
  where a general hasher runs its full schedule. CRC gives 32 bits; one multiply
  spreads them over 64 so the low bits still place the slot and the high bits
  still make the tag, and the fold stays seeded from `hasher`.

Warm lookup over the pretokens of a 4 MB corpus, aarch64, gpt2 / llama-3:
key 1.75 -> 1.30 ns, probe 3.09 -> 2.47, total 4.80 -> 3.73 (llama-3 4.84 -> 3.50).

End-to-end this is currently noise: at 4 MB the lookup already overlaps with the
split and the emit, so it only becomes visible once the per-pretoken call
boundary goes away (see #2267). It is strictly less work either way.
@HuggingFaceDocBuilderDev

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants