I wrote a tokenizer, and German got expensive

Every number you're handed about a model is denominated in tokens. Context window, price per million, rate limit. I'd internalised "a token is roughly four characters", used it for estimates, and never looked closer.

Then I implemented byte-pair encoding to go with the counting model from the last post, typed a German sentence into it, and the token count went somewhere I wasn't expecting.

BPE is a loop that merges the most common pair

Start with every word as a list of single characters. Then repeat:

1. Count every adjacent pair of symbols across the whole corpus.
2. Find the most frequent pair.
3. Merge it everywhere, so it becomes one symbol.

Each round adds exactly one entry to the vocabulary. Thirty rounds on a paragraph gets you fragments like th, the, ing. Fifty thousand rounds on a large corpus gets you most common English words as single tokens.

That's the algorithm. It was introduced for machine translation in Sennrich et al., 2015, and it's still what sits under the tokenizers in use today, with a great deal of engineering around the edges.

Encoding replays those merges in the order they were learned. Which means the vocabulary isn't a dictionary someone curated — it's a fossil record of what was frequent in the training corpus.

The thing I hadn't understood: familiarity is the price

The figure at the bottom of this page trains on an editable corpus and encodes whatever you type. Two things become obvious within about a minute of playing with it.

Words the corpus saw often collapse into one or two tokens. Words it never saw fall apart into characters, because no merge ever reached them. The default input contains kitten, which the training text doesn't have, and you can watch it shatter next to the and little, which it does.

A word-start marker changes everything. Real tokenizers keep the at the start of a word distinct from the same three letters inside one, which is why token counts jump when you change spacing or punctuation in a prompt. I'd noticed that behaviour before without having a reason for it.

So cost isn't a function of how long your text is. It's a function of how much your text looks like the corpus the merges were learned on.

Which is where German gets expensive

I write this blog in two languages. Same arguments, same structure, roughly the same length in characters. It had never occurred to me that they aren't the same size to a model.

German does two things a merge loop trained mostly on English handles badly. It builds compounds — one long word where English uses three short ones, and those compounds are individually rare even when every part of them is common. And it has characters that simply appear less often, so fewer merges get spent on them.

You can see the direction in the figure, though I want to be careful about how much weight that carries: my toy trains on three lines of text, so it exaggerates. A production tokenizer has seen far more German than mine and handles it far better.

But the effect is real and measured, not just something my toy does. Petrov et al. (2023) looked at exactly this across languages and tokenizers, and found the gap is large enough to matter for both cost and effective context length. Their framing is that it's a fairness problem, not just an efficiency one, and I think that's right.

The check I'd actually recommend takes two minutes: take a paragraph you've written in both languages and run each through the real tokenizer of whatever model you use. My toy will tell you the shape of the answer. Only the real one will tell you your number.

What this changes about the numbers

Three things I'd been treating as fixed that aren't:

• A context window is not an amount of text. It's an amount of text in a particular language, in a particular style. The same window holds less German than English, and less code with unusual identifiers than code with ordinary ones.
• Price per million tokens is not price per million anything you can see. Two documents that look the same length can differ meaningfully in what they cost to process.
• Rare-token behaviour isn't only about cost. A word split into six fragments is six positions the model has to reassemble before it can do anything with the meaning.

None of this is a reason to stop writing in German. It is a reason to stop estimating in characters, which is what I'd been doing.

Next

The obvious experiment is to train a tokenizer on my own writing — this blog, both languages, and see what merges it learns that a general one wouldn't. I don't know yet whether that produces anything useful or just a fun vocabulary list.

After that, the thing I've been circling since the counting model: train something small for real, and point the scoring setup from my agent loops at it. Same apparatus, much smaller subject.

If you work in a language that isn't English: have you actually measured what your prompts cost against an English equivalent, or were you estimating in characters like I was?