Next-token predictor
A count-based language model you can retrain in the browser
Every explanation of a language model starts with next-token prediction. This is the smallest thing that has the same problem: three tables of counts, one pass over the text, and a lookup that falls back from two words of context to one to none.
The point of making it interactive is that two behaviours are invisible on paper. Backoff is labelled as it happens, so you can watch predictions go from specific to generic to meaningless. And temperature has both ends broken — near zero it walks into a loop, above 1.5 it samples words the text contains once.
The counting, the backoff and the sampling live in a plain module with no React in it, because the argument is that the model is a few dozen lines and that is easier to believe when you can open the file on its own.
Features
- Editable corpus that retrains on every keystroke
- Unigram, bigram and trigram tables you can inspect
- Backoff labelled with the table and context that answered
- Temperature from 0.05 to 2.0, applied as count^(1/T)
- Click a prediction to append it, or sample from the distribution
Live
Tech Stack
ngram.ts
Training, backoff and sampling as a plain module with no React, so the model can be read on its own.
React
The only dependency. No charting library, no state manager.
Challenges & Learnings
Temperature on counts, not logits
Models apply temperature to logits. With raw counts the equivalent is raising each count to the power of 1/T before normalising, which is the same operation once you treat log(count) as the score.
A default that shows a distribution
The first default prompt landed on a trigram row with a single candidate, so the figure opened on one bar at 100 percent — the opposite of what it exists to show. It now opens on a row with three.
Timeline
2026