tiktoken cl100k is used to count Claude Code sessions, which is precise about the wrong number

[royalpinto007]

TiktokenTokenizer defaults to cl100k_base and is offered as the exact counter:

def __init__(self, encoding: str = "cl100k_base") -> None:

cl100k_base is OpenAI's encoding. Anthropic models do not use it. So ctxlens --tokenizer tiktoken on a Claude Code transcript, which is the headline use case, produces a number that is exact for a tokenizer nobody in that session was using.

That is worse than the heuristic in one specific way: the heuristic is labelled an estimate and people treat it as one. A number labelled "exact" gets copied into a budget.

What to do

At minimum, document it: say which encoding is used, that it is OpenAI's, and that counts for Claude sessions are approximate regardless of the tokenizer chosen. Ideally, pick the encoding from the session's source_format and refuse to claim exactness when there is no matching tokenizer for the model in question.

The Tokenizer seam already exists, so an Anthropic-backed counter can be added behind it as an optional extra without touching the rest.

Acceptance

• --tokenizer tiktoken on a Claude Code session either uses an appropriate counter or says plainly that the count is approximate for this format
• The tokenizer name shown in the output identifies the encoding, so the number is traceable
• README states the accuracy of each tokenizer per source format