refactor(logits-processor): move test-only helpers into the test file

[AmitMY]

Follow-up to #6, same file. The block was already labelled "Backward compatibility methods for existing tests" — four methods shipped in the installed package that nothing in the package ever calls.

What

method	what it was
_get_utf8_state	pure alias for _analyze_utf8_state
_valid_start_bytes	hand-listed copy of what mask_start already encodes
_valid_continuation_bytes	only ever called by _get_allowed_next_bytes
_get_allowed_next_bytes	set-returning wrapper around the mask lookup

_get_utf8_state needed no replacement — tests now call _analyze_utf8_state directly. The other three collapse into one helper in the test file:

def allowed_next_bytes(processor, sequence) -> set:
    """Bytes the processor leaves unmasked after `sequence`, read off its own masks."""
    if isinstance(sequence, torch.Tensor):
        sequence = sequence.tolist()

    masks = processor._get_device_masks('cpu')
    mask = masks['start']
    if sequence:
        state = processor._analyze_utf8_state(sequence[-4:])
        if not state['complete']:
            mask = processor._select_continuation_mask(state, masks)
    return {byte for byte in range(256) if mask[byte]}

−31 lines from the installed package, −66/+49 overall.

This makes the tests slightly stronger

_valid_start_bytes re-listed the valid ranges by hand:

valid.update(range(0x00, 0x80))  # ASCII
valid.update(range(0xC2, 0xE0))  # 2-byte start
...

...so test_valid_start_bytes was asserting against a parallel copy of the rules, which could drift from mask_start without any test failing. The new helper reads the mask the processor actually uses, so those assertions now cover the production mask.

Verification

• 241 passed — same count as before, no test removed, renamed away, or skipped.
• No reference to any of the four names remains anywhere in the repo. The single grep hit is def test_valid_start_bytes, a test name, still accurate.
• Nothing on the __call__ path is touched, so no benchmark for this one — the deleted methods were never on it.

🤖 Generated with Claude Code

---

Note

Low Risk
Test and dead-code cleanup only; __call__ and mask-building logic are unchanged.

Overview
Removes four test-only backward-compat helpers from UTF8ValidationLogitsProcessor (_get_utf8_state, _valid_start_bytes, _valid_continuation_bytes, _get_allowed_next_bytes) so they are no longer part of the installed package API.

Tests now call _analyze_utf8_state directly and use a new allowed_next_bytes helper in test_logits_processor.py that derives allowed next bytes from the same _get_device_masks / _select_continuation_mask path as production, instead of hand-listed byte ranges that could drift from mask_start.

^{Reviewed by Cursor Bugbot for commit dc97fc2. Bugbot is set up for automated code reviews on this repo. Configure here.}