Byte-Pair Encoding (BPE) (subword-based tokenization) algorithm implementaions from scratch with python

An educational Python project for learning tokenization step by step by building character-level, byte-level, and BPE tokenizers from scratch.

LLM-inspired BiLSTM pipeline for real-time, multi-label toxicity inference across adversarial discourse modalities.

Pure Rust runtime for loading SentencePiece models and encoding/decoding text 🦀.

Open byte-level BPE tokenizer for Uzbek (16k vocab) — 1.85 tokens/word vs GPT-4's 2.62, at ~12× smaller vocab. Built-in apostrophe/okina normalization; loads with Hugging Face Transformers.

Implementation of Deep Averaging Networks (DAN) for sentiment classification with experiments on GloVe embeddings and subword tokenization using Byte Pair Encoding (BPE).

From-scratch Byte Pair Encoding (BPE) tokenizer with a multilingual compression-ratio study (English, German, Spanish, French) benchmarked against GPT-2/3.5/4. Includes full research report, reproduction script, and notebook.

A clean, educational implementation of the Byte Pair Encoding algorithm used in modern language models like GPT.

Custom BPE tokenizer built from scratch on WikiText-2 (30k vocab). Covers data cleaning, deduplication, HuggingFace tokenizers training, evaluation (compression ratio, UNK-free coverage, consistency), and save/reload as PreTrainedTokenizerFast.

This repository hosts our comprehensive study on text tokenization methods, covering word-, character-, and subword-level algorithms such as BPE, WordPiece, and Unigram, and extends to discussions on multilingual, mathematical, and code tokenization. It examines their efficiency, consistency, semantic preservation, and influence on LLMs.

Paper: A Comparison of Different Tokenization Methods for the Georgian Language

Build, analyze, and visualize a custom Byte Pair Encoding (BPE) tokenizer trained on the WikiText-2 dataset with vocabulary analysis, compression evaluation, and an interactive GitHub Pages demo.

BPE & Unigram Vocab Training library

A minimal Python implementation of Byte Pair Encoding (BPE) with step-by-step visualization of merge operations and vocabulary updates.

NeuralPiece: An adaptive byte-level neural tokenizer designed to surpass traditional BPE and Unigram via deep learning chunking.

Merge Barriers in BPE Tokenization: How Tokenizer Design Causally Determines Attention Head Specialization. Paper, 23 eval scripts, 86 result files, tokenizer definitions, 4 model checkpoints.