refactor(tokenizer): drop inert __slots__, inline single-use num_embeddings

[AmitMY]

Updated: dropped the _encode / _encode_and_truncate merge — you were right that the split is there for speed. Rebased on current main. What is left is the two cleanups you were happy with, plus a comment so the split does not get "simplified" again.

1. __slots__ was inert

-    __slots__ = ('_struct_format', '_bytes_per_unit', '_eos_encoded')

No ancestor of PreTrainedTokenizer defines __slots__, so instances carry a __dict__ regardless and the declaration saved no memory. Verified by walking the MRO looking for a __slots__ definition — none. It read as a memory optimization that was not one.

2. num_embeddings inlined

self.num_embeddings = 256 was set, then read exactly once two lines later to build the embedding. Only CharacterEmbedding referenced it — the num_embeddings in the tests is nn.Embedding's own attribute, untouched.

3. A comment on why the encode split exists

The reason I proposed merging them is that nothing in the code said not to: two methods with near-identical bodies read as accidental duplication. So:

    def _encode_and_truncate(...):
        # Kept separate from _encode, and hoisting these lookups out of the comprehensions,
        # because this is the hot tokenization path
        encoding, bytes_per_unit, eos_encoded = self.encoding, self._bytes_per_unit, self._eos_encoded

An addition rather than a deletion, but it is the kind that prevents a future regression.

Also dropped from this PR

Inlining _get_generation_params — it saves ~5 lines but deletes a unit-tested seam. Six tests call it directly; without it those cases could only run through generate(), which needs a real model and would move them to the slow integration path.

Verification

• 245 passed on transformers 4.57.6 (no numba) and 5.14.1 (with numba) — both padding paths.
• ruff check . clean.

🤖 Generated with Claude Code

---

Note

Low Risk
Small refactors with no behavior change to tokenization or embedding math; risk is limited to maintainability and comments.

Overview
Removes UTFTokenizer.__slots__, which did not reduce memory because PreTrainedTokenizer ancestors still use a normal instance dict.

In CharacterEmbedding, drops the num_embeddings = 256 indirection and constructs nn.Embedding(256, …) directly.

Documents _encode_and_truncate as the hot path and keeps local hoisting of encoding, _bytes_per_unit, and _eos_encoded for truncation; _encode and the separate truncate helper remain as in this diff.

^{Reviewed by Cursor Bugbot for commit fdcc9bd. Bugbot is set up for automated code reviews on this repo. Configure here.}