llm-tokenization

Here is 1 public repository matching this topic...

Ushnesha / GPT-Tokenizer-from-scratch

Byte Pair Encoding (BPE) tokenizer implemented from scratch in Python. Features an interactive Streamlit playground to visualize token merging, trace vocab rules, analyze compression ratios, and verify lossless decoding in real time.

python-3 byte-pair-encoding large-language-models bpe-tokenizer gpt-tokenizer llm-tokenization

Updated Jun 10, 2026
Python

Improve this page

Add a description, image, and links to the llm-tokenization topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the llm-tokenization topic, visit your repo's landing page and select "manage topics."

Learn more

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

llm-tokenization

Here is 1 public repository matching this topic...

Ushnesha / GPT-Tokenizer-from-scratch

Improve this page

Add this topic to your repo