BPE tokenizer
Byte-pair encoding, trained live on text you edit
Byte-pair encoding is a loop: count every adjacent pair of symbols, merge the most frequent one, repeat. Each round adds exactly one entry to the vocabulary. This runs that loop on whatever text you put in the box.
What it makes visible is that token cost tracks familiarity rather than length. Words the corpus saw often collapse into one or two tokens; words it never saw fall apart into characters, because no merge ever reached them.
That is the same effect that makes a context window hold less German than English, in a form small enough to watch happen.
Features
- Merge loop trained on an editable corpus
- The learned merges listed in the order they were found
- Slider for the merge count, with an honest early stop
- Encoding with token IDs and characters-per-token
- Word-start marker, so the same letters differ by position
Live
Tech Stack
bpe.ts
The merge loop and encoder as a plain module, roughly a hundred lines.
React
The only dependency. Everything runs in the browser; nothing is sent anywhere.
Challenges & Learnings
Deterministic tie-breaking
Two pairs can occur equally often. Breaking the tie by count alone leaves the result dependent on iteration order, so ties resolve by key — the same corpus always produces the same tokenizer.
Knowing when to stop
Training stops when no pair occurs twice, because a pair seen once buys nothing. On a short corpus that happens long before the slider runs out, and the interface says so rather than pretending it ran.
Timeline
2026