-
Notifications
You must be signed in to change notification settings - Fork 1.2k
Pull requests: huggingface/tokenizers
Author
Label
Projects
Milestones
Reviews
Assignee
Sort
Pull requests list
perf(pipeline): don't stride an input that has no cut points
#2283
opened Aug 4, 2026 by
ArthurZucker
Collaborator
Loading…
feat(node): expose the pipeline encode path over napi
#2281
opened Aug 4, 2026 by
ArthurZucker
Collaborator
Loading…
poc(encode): target encode path — bitsplit + batched model + fused cache probe
#2279
opened Aug 4, 2026 by
ArthurZucker
Collaborator
Loading…
perf(pipeline): give the model a whole chunk of pre-tokens at a time
#2278
opened Aug 4, 2026 by
ArthurZucker
Collaborator
Loading…
perf(cache): answer a home-slot hit without the walk, and fold the key with CRC
#2277
opened Aug 4, 2026 by
ArthurZucker
Collaborator
Loading…
perf(cache): take a word key in one load instead of a memcpy
#2276
opened Aug 4, 2026 by
ArthurZucker
Collaborator
Loading…
chore(deps): bump brace-expansion from 1.1.13 to 1.1.18 in /bindings/node
dependencies
Pull requests that update a dependency file
javascript
Pull requests that update Javascript code
#2273
opened Aug 1, 2026 by
dependabot
Bot
Loading…
perf(scratch): reuse Scratch buffers across encode() calls
#2261
opened Jul 29, 2026 by
SBrandeis
Contributor
•
2/4
Loading…
chore(deps): bump braces from 3.0.2 to 3.0.3 in /bindings/node
dependencies
Pull requests that update a dependency file
javascript
Pull requests that update Javascript code
#2251
opened Jul 27, 2026 by
dependabot
Bot
Loading…
chore(deps-dev): bump http-proxy-middleware from 2.0.9 to 2.0.10 in /tokenizers/examples/unstable_wasm/www
dependencies
Pull requests that update a dependency file
javascript
Pull requests that update Javascript code
#2250
opened Jul 27, 2026 by
dependabot
Bot
Loading…
chore(deps): bump uuid and webpack-dev-server in /tokenizers/examples/unstable_wasm/www
dependencies
Pull requests that update a dependency file
javascript
Pull requests that update Javascript code
#2249
opened Jul 27, 2026 by
dependabot
Bot
Loading…
Fix usize underflow in BPE / Strip decoders on empty / short input
#2245
opened Jul 27, 2026 by
weixlu
Loading…
chore(deps-dev): bump shell-quote from 1.8.3 to 1.10.0 in /tokenizers/examples/unstable_wasm/www
dependencies
Pull requests that update a dependency file
javascript
Pull requests that update Javascript code
#2244
opened Jul 25, 2026 by
dependabot
Bot
Loading…
Update fancy-regex dependency to 0.18
#2243
opened Jul 25, 2026 by
musicinmybrain
Contributor
Loading…
Fix panic in Strip decoder on short all-content tokens
#2239
opened Jul 24, 2026 by
v-code01
Loading…
Previous Next
ProTip!
Exclude everything labeled
bug with -label:bug.