analyzePayload and compact both accept a counter option, and the README is honest that the default is an estimate within roughly 10-15% of real BPE counts. But bin/tokencut.mjs never passes one, so the CLI, which is how most people will use this, can only ever give the estimate.
That matters most in the place the number is load-bearing: --max trims real messages out of a payload based on it. Being 15% out means either dropping messages that did not need to go, or leaving a payload that still overflows.
Proposal
A --tokenizer <module> flag that dynamically imports a module exporting count(text) -> number and passes it as counter. Ship nothing extra by default, so the zero-dependency promise holds, and document using gpt-tokenizer or @anthropic-ai/tokenizer as an example.
Acceptance
--tokenizer ./my-counter.mjs is used for both analyze and compact- A module that does not export
count fails with a clear message rather than a stack trace
- Still zero runtime dependencies
- README shows one worked example
analyzePayloadandcompactboth accept acounteroption, and the README is honest that the default is an estimate within roughly 10-15% of real BPE counts. Butbin/tokencut.mjsnever passes one, so the CLI, which is how most people will use this, can only ever give the estimate.That matters most in the place the number is load-bearing:
--maxtrims real messages out of a payload based on it. Being 15% out means either dropping messages that did not need to go, or leaving a payload that still overflows.Proposal
A
--tokenizer <module>flag that dynamically imports a module exportingcount(text) -> numberand passes it ascounter. Ship nothing extra by default, so the zero-dependency promise holds, and document usinggpt-tokenizeror@anthropic-ai/tokenizeras an example.Acceptance
--tokenizer ./my-counter.mjsis used for both analyze and compactcountfails with a clear message rather than a stack trace