I wanted a typeface of my own. What I did not want was to spend a year drawing one, so I started where the licence lets you start: Geist is released under the SIL Open Font License 1.1, and the OFL exists precisely so derivatives are allowed — provided the lineage travels with them.

So this is not a font I drew. It is a font I rebuilt. And the interesting part turned out not to be the drawing at all.

What actually changed

A rebrand would have been renaming the family and shipping it. That is not what happened. Proportions, x-height, character widths, tracking and kerning were all recalibrated, and the specific pairs that break first went through several rounds:

O and o, where a derivative usually starts to look off-balance first
m, i and n, because rhythm at small sizes lives or dies there
Figures, and the glyphs interfaces actually lean on

Each of those got corrected both visually and technically, more than once. What ships:

Axis	What is in the box
Weights	Regular 400, Medium 500, SemiBold 600, Bold 700
Formats	TTF, OTF, WOFF2, plus smaller Latin subsets for the web
Coverage	Extended Latin, Vietnamese, Cyrillic, UI symbols
OpenType	Figure features, including tabular
Dependencies	None, and no peer dependencies, since 0.6.1

The part nobody sees until it breaks

Changing outlines invalidates everything positioned against them. GPOS kerning had to be rebuilt, and so did the mark-to-base, mark-to-ligature and mark-to-mark anchors — every accent, every diacritic, re-anchored to contours that had moved.

Three more things that never show up in a specimen image:

Combining marks keep a zero advance width. Get this wrong and accented text quietly gains spacing nobody asked for.
The vertical metrics avoid clipping on Windows. Different platforms read different metric fields; the wrong combination crops ascenders on one OS and looks fine on yours.
The TTF and WOFF2 builds are hinted for small sizes. A typeface that looks composed at 48px can fall apart in a table cell at 12.

None of that is visible in the poster you post. All of it is visible in a form field.

Treating it like software

Here is the shift that made the project work. Type projects are usually reviewed by looking at specimens — you render a page, you squint, you adjust. That catches what your eye is good at and misses everything else.

So the release criteria are checks, not impressions:

FontBakery reports no FAIL, FATAL or ERROR across both the TTF and OTF builds. Not "mostly clean" — zero.
Nine Playwright checks cover rendering from 10 to 18 pixels, baseline alignment, overflow, tabular figures and visual regression, in Chromium, Firefox and WebKit.
The build is reproducible, so a release is something you can regenerate rather than something you found on a disk.

The 10-to-18-pixel band is the one I would argue for hardest. That is where interface text actually lives, and it is the range a specimen page never shows you.

Packaging is part of the design

The font ships as zutat-font on npm and as a full GitHub release, with the source and build pipeline public. The package carries four complete WOFF2 files, four Latin subsets, Next.js wrappers, TypeScript types and plain CSS exports.

Since 0.6.1 it has no dependencies and no peer dependencies, passes npm audit with no findings, and is published through GitHub Trusted Publishing with provenance.

A font package is a supply-chain artifact like any other. If you are adding one to a production app, being able to verify where the file came from is not a nice extra.

What AI did, and what it did not

AI assistance carried the mechanical half: transforming outlines, generating variants, working through repetitive corrections faster than I would have by hand.

What it did not do is decide. Which of four o variants sits right next to an n, whether the tracking reads too tight at 11px, when a correction has gone far enough — that stayed a judgment call every time, and that is where most of the rounds went.

The honest summary is that AI compressed the iteration cost, which let me do more rounds than I otherwise would have. It did not remove the need for someone to look at the thing and have an opinion.

What this doesn't settle

It is a derivative, and that is a real constraint. Starting from Geist meant inheriting decisions I did not make. Some of the recalibration was working against the original rather than with it.
Clean tooling is not the same as good type. FontBakery passing tells you the file is well-formed, not that the letters are good. Those are different questions and only one of them has a checker.
One family, one designer, no critique. Type usually improves by being shown to people who draw type. Mine has not been.

Next

What I would take from this into any design work:

Specimens flatter. Test at the sizes the thing will actually be used at.
If a release criterion cannot fail, it is not a criterion.
Packaging, provenance and audit status are part of the artifact, not paperwork after it.
AI shortens the loop between attempts. It does not supply the taste that picks between them, which is the same split I route models on.

The open question is variable weights. Everything above is static, and a variable axis would make the whole metrics and hinting story considerably harder. I have not decided whether the payoff is worth it.

If you have shipped a typeface, or a derivative, I would like to know what your release checklist looks like, and whether anything on it caught something your eye had already signed off.

1b:T1cab,

Every design system has a progress bar. It is a rounded rectangle that fills from left to right, and in most systems that is where the thinking stopped.

Which is odd, because it is one of the few components whose entire job is to communicate a number to someone who is waiting. That is a real design problem, and the answer everyone shipped is a rectangle.

I have been building a small set of progress and data components to see what happens if you take the rendering seriously and leave everything else alone.

Same semantics, different picture

The component exposes three renderers behind one API. A variant prop selects between a braided wave, an ordered-dither field and a layered dither surface. Nothing else changes.

That constraint is the whole design. Switching renderer must not change:

role="progressbar" on the element
aria-valuenow, present when determinate and absent when not
the value, the bounds, or how a screen reader announces any of it

A screen reader cannot tell which painter is active, and it should not be able to. The visual is a presentation choice layered on top of a component whose contract does not move. It is the same separation I route model work by: deciding and rendering are different jobs.

The indeterminate case is where this gets tested. An unknown-duration state has to drop aria-valuenow entirely — announcing a fake percentage is worse than announcing nothing, while the renderer keeps animating. The two concerns have to be genuinely separate, or you end up with a wave that looks busy and a screen reader claiming 0%.

The numbers, because otherwise this is decoration

A component that renders continuous animation on a canvas invites a fair question: what does it cost? So the repository carries a standing bundle audit and a browser benchmark, and they run on every change.

Check	Result
Component JS	7,834 bytes gzip
Component CSS	2,575 bytes gzip
Total	10,409 bytes gzip
50 wave instances	120 FPS mean
50 dither instances	120 FPS mean
50 dither instances, assembly effect	120 FPS mean
50 indeterminate instances	120 FPS mean

One detail about that benchmark matters more than the number. It keeps all fifty instances visible and hovers one of them while sampling, so the measured frame budget includes the active geometry morph. A benchmark that measures fifty idle components tells you almost nothing; the expensive frame is the one where something is changing.

React is the only runtime dependency the installed component carries.

Two things that stop it costing more than it should

Offscreen instances pause. A component animating below the fold is burning frames for nobody. Phase and indeterminate animation stop when the instance is not visible, which is what makes fifty simultaneous instances a benchmark rather than a warning.

Reduced Motion is a tested path, not a comment. There is a real difference between honouring prefers-reduced-motion and having a test that fails when you stop honouring it. The repository has the second kind.

Source-owned rather than a dependency

It installs through the shadcn registry, which means the files land in your repository rather than behind a version number:

npx shadcn@latest add https://example.com/r/wave-progress.json

The registry is not published yet, so that URL is illustrative rather than something you can run today. I have come round to this model for anything visual. A progress bar is exactly the kind of component a team wants to tweak — the wave amplitude is wrong for their brand, the milestone markers need different copy, and doing that against an npm package means either a fork or a config surface that grows forever.

Owning the source means the tweak is a diff in your own repo. The cost is that you do not get updates for free, which for a component this size is a trade I would take.

What I would not claim

I do not know if it helps anyone wait. The premise is that a more legible progress indicator makes waiting less unpleasant. That is an assumption. I have not tested it with users, and a 120 FPS benchmark says nothing about whether the thing is actually better to look at.
Three renderers is a maintenance surface. Every one of them has to keep the ARIA contract, survive Reduced Motion, and not regress the geometry tests. That is a real ongoing cost for a visual choice.
The numbers are local. Measured on my machine, re-run on each change, and they will move with host load. They are a floor to reason about, not a guarantee.

Next

What generalises beyond this component:

Changing how something looks should not change what it announces. If the renderer can move the semantics, the abstraction is in the wrong place.
Benchmark the expensive frame. Fifty idle instances is not a measurement.
Reduced Motion needs a failing test, not a media query and good intentions.
For visual components, source ownership beats a config surface that grows forever.
Release criteria that can fail beat a changelog. This is the second thing I've built this month around a bar rather than a deadline — the typeface got the same treatment, and in both cases the criteria caught something my eye had already approved.

Next I want the thing I just said I do not have: some evidence about whether a more legible indicator changes how long a wait feels. That is a user question, not a benchmark, and I have not worked out how to ask it honestly yet.

If you have replaced a default progress component with something of your own, I would like to know what pushed you to, and whether anyone noticed.

1c:T18c6,

I put my personal projects on Coolify because it was faster to stand up than anything else I looked at, and because I wanted to see what a self-hosted PaaS actually felt like after years of writing Terraform for other people. Both of those were good reasons. It delivered on both.

It now runs eleven projects. Several PostgreSQL instances, Redis, Prometheus, Grafana. Somewhere in there it stopped being an experiment and became the thing my work lives on, and I didn’t notice the transition happening.

I noticed when the question changed. For a long time the question was how do I deploy this, and Coolify answers that well. Now the question is how do I move this, and that’s a different kind of question.

What moving actually takes

Coolify’s own documentation is clear about the first part: there’s no built-in way to migrate applications from one server to another. Not an omission in the docs. The feature doesn’t exist.

Moving an instance means moving four things by hand:

The coolify-db container, which holds every project’s configuration.
The APP_KEY out of the .env, because that database is encrypted with it and worthless without it.
The SSH keys Coolify uses to reach your servers.
The authorized keys those servers use to accept it.

Volumes are their own exercise: mount each one into a temporary container, tar its contents, copy the tarball out, delete the container. Then, on the far side, open the dashboard and redeploy each application so the containers come back against the restored volumes.

None of that is unreasonable for a tool of this kind. What it tells you is where the truth about your infrastructure lives, and here it is an encrypted Postgres container plus a secret in a dotfile.

The thing I actually miss

Not the migration. The review.

I can’t diff my infrastructure. There’s no file where eleven projects are described, so there’s nothing to open a pull request against, nothing to read in a year to find out why a container has the environment variable it has, nothing to replay onto a fresh host to see whether it comes back the same. The configuration is real and it’s correct, and it’s only inspectable by clicking through a web interface one project at a time.

This is the same defect I’ve written about in a different costume. One Terraform state file holding everything is dangerous because it makes the blast radius of any change the whole estate. A PaaS database holding everything is dangerous for the mirrored reason: there’s no radius at all, because there’s no change to review in the first place.

Permissions make it concrete. In Kubernetes, who may do what is a file. A Role, a RoleBinding, a securityContext, a NetworkPolicy. You can read it, diff it, reject it in review, and point at it during an incident. In Coolify, permissions are whatever the interface exposes, expressed as the state of some checkboxes, and the only way to audit them is to go and look.

What ArgoCD changes, and what it doesn’t

The reason Kubernetes with ArgoCD is where I’m heading isn’t that Kubernetes is better at running containers. For eleven projects it’s plainly worse: more moving parts, more to keep patched, more ways to be paged on a Sunday.

It’s that GitOps inverts where the truth lives, the same question I ended up at when I worked out what actually runs your infrastructure when you use SST: whichever layer holds the desired state is the layer you can review. The desired state of the cluster is a repository. Migration stops being a tarball procedure and becomes pointing a new cluster at the same repo. Extending stops being a sequence of clicks someone has to remember and becomes a diff someone can approve. Rollback is git revert.

I want to be honest about the cost, because the version of this argument you usually read isn’t. You’re trading an afternoon of setup for a system with a genuine learning curve, and you’re taking on the operational load of the control plane itself. If the only thing you need is eleven containers that stay up, that trade is bad and Coolify wins on every axis that matters.

The trade only makes sense once the questions you’re asking are about change rather than uptime. Mine have been for a while, and I was slow to notice.

Where the line falls

My honest reading, having been on both sides of it:

Coolify is right when the set of things you run is small enough to hold in your head, and when you’re the only person who needs to understand it. That isn’t a limitation; that’s most personal infrastructure, and a UI is a genuinely good interface for it. The speed I bought at the start was real, and I would buy it again.

It stops being right at the point where you need someone else to be able to reconstruct what you built, and that someone else is usually you, eighteen months later, with no memory of the checkboxes.

I haven’t moved yet. I’m writing this partly to work out whether the reasons hold up when I put them in order, and so far they do.

Next

The short version:

Coolify has no built-in application migration. Moving an instance means the database container, the APP_KEY that decrypts it, both sets of SSH keys, and every volume by hand.
The real cost isn’t the migration, it’s that there’s no file to diff, review or replay.
Permissions in Kubernetes are a file you can reject in review. In a PaaS they’re the state of a form.
ArgoCD is worth its overhead only when your questions are about change rather than uptime. For eleven containers that just need to stay up, it’s the wrong trade.

The next thing I want to settle is what the migration actually costs in hours, measured rather than estimated, because that number is the whole argument and I don’t have it yet. When I do, it will be here. If you have made this move with a comparable estate, I would like to know what it cost you, since every estimate I can find is written by someone selling one side of it.

1d:T176a,

Every infrastructure tool is a bet about which layer you want to think in. Terraform bets you want to describe resources. CloudFormation bets you will accept AWS’s own vocabulary in exchange for AWS’s own guarantees. SST bets you would rather describe an application and let the resources follow.

What makes SST worth a closer look isn’t the bet. It’s what sits underneath it. In v3 the answer changed, and the new answer runs straight through Terraform.

The chain, stated plainly

SST v2 compiled your configuration into AWS CDK constructs, which compiled into a CloudFormation template, which AWS then executed. Three translations between what you wrote and what ran.

SST v3, which the project calls Ion, removed CDK and CloudFormation and put Pulumi in their place. Pulumi, in turn, doesn’t hand-write most of its providers. It generates them from Terraform providers through a bridge. Pulumi says so directly: SST “leverages Pulumi behind the scenes for its providers and deployment engine while also bridging Terraform providers through Pulumi.”

So when an sst.config.ts creates an S3 bucket, the schema it’s validated against traces back to the same Terraform AWS provider you would use writing HCL by hand. Not a similar one. The same one, wrapped.

This is the part that matters if you have Terraform experience, and the part almost nobody says out loud: your knowledge transfers. Argument names, defaults, the awkward corners of a resource that you learned the hard way. They survive the move because the provider survived the move.

What SST actually buys you

Three things, worth naming precisely because the marketing blurs them together.

One config for infrastructure and application. A single sst.config.ts defines the bucket, the function that reads from it, and the frontend that calls the function. In Terraform you describe the bucket and the function; wiring the application to them is a separate problem you solve with variables, outputs and a deployment script.
Breadth without new syntax. SST’s documentation puts provider support at over 150, spanning AWS, Azure and GCP but also Cloudflare, Stripe, Vercel and Auth0. That number is a direct consequence of the bridge. SST didn’t write 150 integrations. It inherited an ecosystem.
No CloudFormation. The stated reasons for leaving were the AWS-centric nature of CDK and CloudFormation, and that the intermediary template “often obscured the underlying logic and made it much more difficult to diagnose and resolve problems”. Anyone who has read a CloudFormation rollback message recognises the complaint. The template you debug isn’t the code you wrote.

What you give up

Choose a tool for what it costs you, not only for what it saves you.

An extra abstraction on the critical path. You gained a layer over Terraform providers, so when something breaks you debug in three vocabularies: SST components, Pulumi resources, and the provider underneath. The CloudFormation opacity SST was built to escape hasn’t disappeared so much as moved.

State becomes yours to reason about again. CloudFormation kept state on AWS’s side, which was limiting and also one fewer thing to lose. Pulumi-based state is a file you’re responsible for, with the locking and backup questions that follow. Anyone who has run Terraform at scale has already had the argument about remote state, and I’ve written about what it costs when a team never has it: one state file holding everything is the anti-pattern that eventually takes the site down.

A smaller hiring pool. Terraform and HCL are a labour market. SST is a smaller one. That isn’t an argument against it, but it’s an argument for writing down why you chose it.

Where the line falls

The useful question isn’t which tool is better. It’s which layer your team wants to be responsible for.

If your infrastructure exists to serve one application, and the people writing that application are the people deploying it, SST removes a translation step that was never earning its keep. The single config isn’t a convenience, it’s a correctness property: the function and the bucket it reads can’t drift apart, because they’re declared together.

If your infrastructure serves many applications, or a platform team hands environments to product teams, that same coupling works against you. You want the bucket to have a life independent of whoever consumes it. Terraform’s separation of resource from consumer is the point, not an inconvenience to be abstracted away.

The chain matters either way. When an SST deployment fails on an argument you have never heard of, the answer is in the Terraform provider documentation, three layers down. That’s a strange thing for a tool to inherit, and a genuinely good one.

Next

The short version:

SST v3 runs on Pulumi, and Pulumi generates its providers from Terraform’s. The schema you validate against is the same one either way.
Terraform experience transfers. Argument names, defaults and the awkward corners survive the move, because the provider does.
You trade CloudFormation’s opacity for a third vocabulary to debug in, and the state file becomes yours to keep again.
One application and one team: SST removes a translation step. Many consumers behind a platform team: keep the resource independent of whoever uses it.

The question I haven’t settled is what happens to the single-config argument once a second team needs the same bucket. That’s the point where this stops being about syntax and starts being about ownership, and it’s the part no comparison table will answer for you.

1e:T2158,

I onboarded a new client last quarter and asked to see their Terraform. They sent me a single file. 1,847 lines. One main.tf containing their entire AWS infrastructure: VPC, subnets, security groups, RDS, ECS cluster, ALB, Route53, IAM roles, S3 buckets, CloudFront distribution. All of it. One file. No modules. No variables for anything that changes between environments. Their staging and production configs were two copies of this file with hardcoded values changed by hand.

They are not unusual. I see some version of this at about four out of five startups that ask me for help with their infrastructure.

Here are the five patterns that come up every time, ranked by how much damage they cause.

One: everything in a single state file

This is the one that will eventually cause an outage.

The state file is Terraform's record of what your infrastructure currently looks like. When you run terraform apply, it compares the desired state (your .tf files) against the current state (the state file) and figures out what to change. If all your infrastructure lives in one state file, every apply operation locks and evaluates everything. A change to a DNS record requires Terraform to also evaluate your database, your compute cluster, your networking layer.

At small scale this is fine. At 200 resources it starts to get slow. At 500 it takes minutes. At 1,000 you are waiting 8 to 12 minutes for a plan, and any failure during that plan leaves the state lock held until someone manually releases it.

The worse problem is blast radius. A typo in a security group rule, applied against a monolithic state, can theoretically touch your database. I have seen exactly this happen. An engineer changed an ingress rule, Terraform's dependency graph pulled in an RDS modification that had been sitting in the diff unnoticed, and staging went down for three hours.

Split your state by layer. Networking in one state. Data stores in another. Compute in a third. DNS and CDN in a fourth. Each one can be planned and applied independently. The blast radius of any single apply shrinks from "everything" to "one layer."

Two: no remote state backend

The terraform.tfstate file defaults to local storage. On the engineer's laptop. In the project directory. Sometimes committed to Git (which is its own category of problem, since state files contain secrets in plaintext).

When two engineers run terraform apply against a local state file, the second one overwrites the first one's changes. This is not a theoretical risk. I have watched it happen at a company where two people both applied against production on the same morning, each with a different local state, and the resulting infrastructure was a blend of both that matched neither.

Remote state with locking solves this completely. An S3 bucket with a DynamoDB table for locking, or Terraform Cloud, or any other supported backend. The state lives in one place, and only one person can apply at a time. Setup takes about 30 minutes.

Three: no modules

The client with the 1,847-line file was creating ECS services by copying and pasting 60 lines of resource blocks and changing the service name, container image, and port. They had seven services. That is 420 lines of nearly identical Terraform that drifts every time someone updates one service and forgets to update the others.

Modules exist for this. A module is a reusable Terraform component with input variables and output values. Define your ECS service pattern once as a module, then call it seven times with different parameters. When the pattern needs to change, you change the module, and all seven services update together.

I am not suggesting you build an internal module library on day one. That is over-engineering. But when you copy and paste the same resource block for the third time, it is time to extract a module. The threshold is three. Two duplicates are fine. Three means you need abstraction.

Four: hardcoded values everywhere

Environments differ:

Staging has a smaller database instance, fewer container replicas, a different domain.

Production has the big instance, more replicas, the real domain. When these differences are hardcoded into the Terraform files, creating a new environment means reading through every file and finding every value that needs to change.

I have seen engineers spin up a new staging environment from a production config and accidentally provision an r6g.2xlarge RDS instance that cost $25 per day for a staging database with 50 rows of test data. It ran for eleven days before anyone noticed. $275 for nothing.

Variables and .tfvars files. Every value that differs between environments should be a variable with a sensible default. The production-specific values go in prod.tfvars, staging in staging.tfvars. Creating a new environment becomes terraform apply -var-file=newenv.tfvars instead of a find-and-replace across 2,000 lines of HCL.

Five: no drift detection

Terraform assumes it is the only thing managing your infrastructure. When someone makes a change through the AWS console (and someone always does, usually during an incident at 2 AM), Terraform does not know about it. The state file says one thing, reality says another.

Drift is the gap that opens when infrastructure changes outside Terraform — a console edit, a manual fix during an incident, so the state file no longer matches reality. Left undetected, it accumulates. The next terraform apply will try to reconcile the state file with your .tf files, but it does not know about the manual console changes, so it might revert them. I have seen a manually added security group rule (added to fix a production issue) get silently removed by a Terraform apply three days later, re-breaking the thing it fixed.

Run terraform plan on a schedule. A weekly cron job that plans against production without applying, and sends the output somewhere visible. If the plan shows changes you did not make, someone clicked something in the console. Investigate before applying.

Some teams use Terraform Cloud's drift detection for this. Others use a simple GitHub Action that runs terraform plan nightly and posts the output to Slack. Either works. The important thing is that someone is looking.

The common thread

All five of these are infrastructure management problems, not Terraform problems. Terraform is not opinionated enough to prevent them, so they happen by default at every company that grows faster than its infrastructure practices mature.

Fixing all five in an existing codebase takes one to two weeks of focused work. Preventing them in a new project takes about two hours of initial setup. The difference in ongoing maintenance cost is enormous.

The five, side by side

Anti-pattern	What it costs you	The fix
One state file for everything	Blast radius: a typo in a security group can touch your database	Split state per environment and per bounded domain
No remote backend	State on a laptop. No locking, no history, concurrent applies corrupt it	Remote backend with locking and versioning
No modules	The same block copy-pasted per environment, drifting apart quietly	Extract the repeated shape into a module, parameterise the differences
Hardcoded values	Environment differences live in copies instead of variables	Variables and per-environment tfvars
No drift detection	Console edits accumulate until an apply proposes something alarming	Scheduled `terraform plan -detailed-exitcode` in CI

If you only do one of these, do the remote backend. Everything else is recoverable; a corrupted local state file during a concurrent apply is the one that ruins a week. And if your pipeline is also handling agent-generated changes, the same argument applies harder — I looked at what that does to CI failure rates separately.

1f:T19f5,

Every explanation of a language model starts in the same place: it predicts the next token. I've read that sentence more times than I can count and nodded at it every time.

Then I tried to explain to myself why a model starts repeating itself when you turn the temperature down, and found I could recite the answer without being able to derive it.

That's usually a sign I've memorised the shape of an explanation rather than the mechanism under it. Which, as it turns out, is what this post ends up being about.

So I built the smallest model that has the same problem. It's in the figure at the bottom of this page, it runs entirely in your browser, and it's about eighty lines of counting.

The whole model is one pass of counting

Walk through a text one word at a time. At every position, record three things:

which word followed the two words before it — the little → cat
which word followed the one word before it — little → cat
which word this is, with no context at all

Those three tables are the entire model. No weights, no gradients, nothing that keeps changing once the pass is done. Training is a single loop over the text, which is why the figure below can retrain on every keystroke you type into it.

Predicting is a lookup. Try the most specific context first, the two-word row. If the model has never seen it, fall back to the one-word row. If that's missing too, fall back to plain word frequency.

Backoff is the part I had never actually pictured

I knew the word. I had not pictured it happening.

The figure labels which table answered and on what context, so you can watch it degrade. Type a phrase the text contains and you get a trigram answer with two or three candidates and a sharp distribution. Change one word and it drops to the bigram row, where the candidate list is suddenly much longer and much flatter. Push it further and it lands on unigram, which is just "here are the common words", and the output stops being about your prompt at all.

That gradient is what I'd never seen laid out: specific and confident, then general and hedged, then meaningless. A real model doesn't back off in discrete steps like this, and I don't want to overclaim the analogy. But the direction it degrades in is recognisable. When there's nothing specific to draw on, output goes generic before it goes wrong.

There is no correct temperature

Temperature is one line. Take each count, raise it to the power of 1/T, normalise. Low T sharpens the distribution around the top count. High T flattens it until rare words get a real chance.

What the slider makes obvious is that both ends are broken, in opposite directions:

At 0.05 the model always takes the highest count. Sample repeatedly and it walks into a cycle and stays there. It isn't confused. It's doing exactly what you asked, which is to always take the safest available step.
At 1.8 a word the text contains once gets picked nearly as often as one it contains ten times. Every individual choice is legal and the paragraph is nonsense.

I'd half-assumed low temperature was the accurate setting and high temperature was the creative one. It's better described as how much of the distribution you're willing to use, and the useful range sits in the middle for reasons that have nothing to do with accuracy.

The term is borrowed from physics on purpose — the same shape of equation controls how often a system visits higher-energy states in the Boltzmann distribution. That's the kind of detail I'd have skimmed past a week ago.

The toy model is the memorisation failure mode

This is the part that made it worth the evening.

Give the counting model a large enough corpus and its trigram table gets very good at that corpus and learns nothing it could carry anywhere else. It can't generalise, because there's nothing in it that could. It's a lookup table wearing a probability distribution.

I wrote about this at a completely different scale a few days ago: what happens when you score an agent loop instead of reviewing it, and the loop starts optimising the eval rather than the problem. A system that scores perfectly on the set it was tuned against and falls over the moment the surface changes — that's the failure I was describing there, and it's what this eighty-line model is permanently and by construction.

Seeing it in a form small enough to read made the larger version much less abstract. The probe gap I use on agent runs is, in one sentence, a test for whether the thing became an n-gram model of my eval set. I didn't have that sentence before I built the toy.

What this isn't

Being explicit about the gap, because this genre has a habit of glossing it:

No attention. This model can see two words back, ever. The 2017 transformer paper exists because that limit is fatal, and because the sequential alternatives that fixed it were slow to train.
No learning. Nothing is fitted to anything. Counting is not training, even though I keep calling it that.
No tokenizer. I split on whitespace and called the pieces words. Real models don't, and that turns out to matter more than I expected.

None of this is new ground either. Andrej Karpathy's makemore walks the same road with more depth, and I'd send anyone there first. I built my own anyway, because reading an implementation and writing one are different activities, and only one of them told me I didn't understand temperature.

Next

Tokenization, same treatment: implement byte-pair encoding, put it in the page, and find out what it costs to write in a language the merge loop never saw much of.

After that I'd like to train something real and small, and point the scoring apparatus from how I run agents at it — the vocabulary turns out to be shared, and I want to know whether the instincts are.

I'd have guessed the thing I was missing would be attention. It turned out to be temperature. If you've built one of these yourself, what did writing it tell you that reading it hadn't?

20:T1625,

Every number you're handed about a model is denominated in tokens. Context window, price per million, rate limit. I'd internalised "a token is roughly four characters", used it for estimates, and never looked closer.

Then I implemented byte-pair encoding to go with the counting model from the last post, typed a German sentence into it, and the token count went somewhere I wasn't expecting.

BPE is a loop that merges the most common pair

Start with every word as a list of single characters. Then repeat:

Count every adjacent pair of symbols across the whole corpus.
Find the most frequent pair.
Merge it everywhere, so it becomes one symbol.

Each round adds exactly one entry to the vocabulary. Thirty rounds on a paragraph gets you fragments like th, the, ing. Fifty thousand rounds on a large corpus gets you most common English words as single tokens.

That's the algorithm. It was introduced for machine translation in Sennrich et al., 2015, and it's still what sits under the tokenizers in use today, with a great deal of engineering around the edges.

Encoding replays those merges in the order they were learned. Which means the vocabulary isn't a dictionary someone curated — it's a fossil record of what was frequent in the training corpus.

The thing I hadn't understood: familiarity is the price

The figure at the bottom of this page trains on an editable corpus and encodes whatever you type. Two things become obvious within about a minute of playing with it.

Words the corpus saw often collapse into one or two tokens. Words it never saw fall apart into characters, because no merge ever reached them. The default input contains kitten, which the training text doesn't have, and you can watch it shatter next to the and little, which it does.

A word-start marker changes everything. Real tokenizers keep the at the start of a word distinct from the same three letters inside one, which is why token counts jump when you change spacing or punctuation in a prompt. I'd noticed that behaviour before without having a reason for it.

So cost isn't a function of how long your text is. It's a function of how much your text looks like the corpus the merges were learned on.

Which is where German gets expensive

I write this blog in two languages. Same arguments, same structure, roughly the same length in characters. It had never occurred to me that they aren't the same size to a model.

German does two things a merge loop trained mostly on English handles badly. It builds compounds — one long word where English uses three short ones, and those compounds are individually rare even when every part of them is common. And it has characters that simply appear less often, so fewer merges get spent on them.

You can see the direction in the figure, though I want to be careful about how much weight that carries: my toy trains on three lines of text, so it exaggerates. A production tokenizer has seen far more German than mine and handles it far better.

But the effect is real and measured, not just something my toy does. Petrov et al. (2023) looked at exactly this across languages and tokenizers, and found the gap is large enough to matter for both cost and effective context length. Their framing is that it's a fairness problem, not just an efficiency one, and I think that's right.

The check I'd actually recommend takes two minutes: take a paragraph you've written in both languages and run each through the real tokenizer of whatever model you use. My toy will tell you the shape of the answer. Only the real one will tell you your number.

What this changes about the numbers

Three things I'd been treating as fixed that aren't:

A context window is not an amount of text. It's an amount of text in a particular language, in a particular style. The same window holds less German than English, and less code with unusual identifiers than code with ordinary ones.
Price per million tokens is not price per million anything you can see. Two documents that look the same length can differ meaningfully in what they cost to process.
Rare-token behaviour isn't only about cost. A word split into six fragments is six positions the model has to reassemble before it can do anything with the meaning.

None of this is a reason to stop writing in German. It is a reason to stop estimating in characters, which is what I'd been doing.

Next

The obvious experiment is to train a tokenizer on my own writing — this blog, both languages, and see what merges it learns that a general one wouldn't. I don't know yet whether that produces anything useful or just a fun vocabulary list.

After that, the thing I've been circling since the counting model: train something small for real, and point the scoring setup from my agent loops at it. Same apparatus, much smaller subject.

If you work in a language that isn't English: have you actually measured what your prompts cost against an English equivalent, or were you estimating in characters like I was?

21:T1993,

Sean Goedecke argues that the most important skill in prompting is expertise in the domain you're prompting for. His main evidence is a Terence Tao conversation with ChatGPT: Tao gets further than most people would, not because he prompts better in some general sense, but because he knows the mathematics well enough to know where to push.

I think that's right, and I've been running on it for months. The way I route work across models is a bet on exactly this claim — the plan gets written before the code, the plan is where the judgment lives, and the model doing the implementation is doing the cheaper half.

Then I automated the judgment, and the claim stopped holding in the direction I expected.

What happens when you write the expertise down

Goedecke's expert is inside the loop. He reads a response, recognises it as subtly wrong, and pushes back. The expertise is being applied continuously, and it adapts while it's applied, because it's still in a head.

A scored loop doesn't work that way. You write down what good means, precisely enough that a script can compute it, and then the run goes for hours without you. That's the entire gain and it's a real one.

But the expertise isn't being applied any more. It was applied once — when I wrote the spec and built the eval, and then it was frozen into files that are deliberately read-only, so that nothing in the run can edit its way to a better score.

Read-only is the correct design. It is also what makes the expert's assumptions the only thing in the run that cannot be revised.

Six hundred scenarios, none of them a request

The target I hit this on is a partner-attribution system. Its eval has 600 scenarios — 400 dev, 200 holdout. Every one of them is a prebuilt state file: affiliates, referral edges, events, links, offers, payouts.

Not one contains an HTTP request.

The run finished clean. 1.0 on the sample, 1.0 on the full dev set, 1.0 on the probe set — a gap of exactly zero, which is the number that says it generalised rather than memorised. Nothing was gamed. The database was real. Row-level security was real, under a role with neither superuser nor bypass rights. The append-only hash chain was checked directly rather than through some scoring side-path, because I'd written all of that into the constraints.

What I hadn't written in was the product's entry point. The ingestion routes — the thing every actual event would arrive through — were never once under test. They couldn't be. There was nothing in the eval that could reach them.

That isn't a model failure. The model built what it was asked for, correctly, and proved it generalised. It's a specification failure, and the specifier is me.

Why this is worse than a loop that cheats

A loop gaming its eval is annoying and tractable. You checksum the target, blind the answers, lint for the tricks you've seen, and hold out a probe set. Those defences work, and I've written about how they're built.

They all point inward.

The probe gap asks whether the thing generalises across the surface I sampled. It has nothing to say about whether that surface was the right one.
The lints ask whether the code cheated at the task as specified. A correct implementation of the wrong scope passes every one of them.
The holdout is drawn from the same generator as the dev set, so it inherits every assumption the dev set has.

Here's the version of that I keep coming back to. An expert reviewing output can be surprised by it. An expert who wrote the eval has already decided what counts as surprising.

Goedecke's line is that the human is the bottleneck, not the model. On a supervised session I'd agree. On a scored run it's more specific than that, and less flattering: the human is the frame, and nothing inside a frame tests the frame.

What survives of the claim

Not "expertise doesn't matter" — the run only got as far as it did because the spec was good, and most of the value showed up during specification, before a single cycle ran. That part of Goedecke's argument comes out stronger, not weaker.

What changes is that expertise turns out to be doing two different jobs, and only one of them survives being written down:

Drawing the boundary. Deciding what counts, what gets measured, what "correct" means for this task. This transfers into files beautifully. It's most of what a good spec is.
Checking the boundary. Noticing that the thing you measured isn't the thing you wanted. This does not transfer, because the check has to come from outside the frame the expertise drew.

I had been treating those as one skill. They're not, and the second one is the one I keep failing to automate. What found the scope failure was going and looking at the product weeks later, not any instrument in the harness.

So the practical version, for me: the more of my judgment I move into the target, the more deliberately I have to schedule the part I can't move. Not review of the diff — the loop's own scoring is better at that than I am now. Review of the question. Someone, at some interval, has to open the product and ask whether the eval still describes it.

So far that someone is me, at no interval in particular, by accident.

Next

The thing I want and don't have is an instrument for scope: something that fails when the eval stops covering the product, the way the probe gap fails when the eval stops being honest. I have no design for it. It may not be the kind of thing a script can hold, in which case the honest answer is a calendar entry rather than a harness.

The cheap version I'm trying first is one end-to-end probe per target that goes through the real entry point and is deliberately not scored, so it can't be optimised against, and can only ever tell me the eval and the product have drifted apart.

My own answer so far is "nothing, I noticed by accident", and I'd rather hear that back from you than a mechanism nobody actually runs. So: if you're running loops against a written-down spec, what has ever caught your scope drift?

3e:I[302847,["/_next/static/chunks/225u9y9y62hte.js","/_next/static/chunks/1skg5sjyr5qzo.js","/_next/static/chunks/2rb3bq9nehl88.js","/_next/static/chunks/1n-1sx_0ke85t.js","/_next/static/chunks/1blgycuzwyhgp.js","/_next/static/chunks/3w6j-d9quekp8.js"],"default"] 3f:I[757590,["/_next/static/chunks/225u9y9y62hte.js","/_next/static/chunks/1skg5sjyr5qzo.js","/_next/static/chunks/2rb3bq9nehl88.js","/_next/static/chunks/1n-1sx_0ke85t.js","/_next/static/chunks/1blgycuzwyhgp.js","/_next/static/chunks/3w6j-d9quekp8.js"],"default"] 40:I[611015,["/_next/static/chunks/225u9y9y62hte.js","/_next/static/chunks/1skg5sjyr5qzo.js","/_next/static/chunks/2rb3bq9nehl88.js","/_next/static/chunks/1n-1sx_0ke85t.js","/_next/static/chunks/1blgycuzwyhgp.js","/_next/static/chunks/3w6j-d9quekp8.js"],"default"] 41:I[27201,["/_next/static/chunks/225u9y9y62hte.js"],"IconMark"] 22:T1ad1,

I had an agent review the post about scoring agent loops. It came back with three findings, written well, with quotes from my own text and numbers from my own repository.

Two were exactly right. Acting on them turned the weakest section of that post into the strongest.

The third was the best story in the batch. I checked it, and almost none of it was in the files.

What it got right

The post claimed I had no score history worth charting. The review said I'd had one since 11 July and pasted the numbers.

I opened var/score-history.jsonl expecting to find them approximately right. They were exact — every value to four decimal places, in order, including a regression at 0.9102 that I'd forgotten happened. One session, 79 minutes, eleven cycles from 0.8279 to 1.0, then 0.9985 on the full dev set against 0.9940 on the probe.

Its second finding was that two numbers in the post didn't match my actual target. Also correct. My goal.md defines the bar as holdout at or above 0.95 and no category below 0.90, with a probe gap of 0.02 or less. The post said "gap over 0.15 twice in a row", which I had invented while writing and then forgotten was invented.

That one stung in a useful way. The review found a fabrication of mine before I did.

What it made up

The third finding described a failure mode I'd missed. The run had scored perfectly, and what it had actually built was — in the review's words — an in-memory engine with no database, no row-level security, no HTTP ingestion, no roles.

Then the closer: the engine would have misattributed every real registration, because all 600 eval scenarios fed prebuilt click events carrying a meta field that live clicks don't have, and the rebuild dropped them silently.

That is a genuinely great paragraph. Here's what's in the files.

The database was real. LOG.md records a per-run schema, FORCE RLS on the Kit pattern, a 14-case RLS suite passing, and a scorer role with neither superuser nor bypass rights. goal.md carries a constraint requiring exactly this — that scenarios run through the production path and the scorer check the database directly rather than through a scoring side-path.
"Holdout 0.9995" doesn't exist. There is no holdout value anywhere in the history file. Holdout is aggregate-only on a budget of six calls per 48 hours, which is why it isn't there.
There is no meta field. Zero of the 400 dev scenarios contain one. The mechanism the story turns on isn't in the data.

One part of it was true, and I could only tell which part by opening the eval. There are 600 scenarios — 400 dev, 200 holdout, and every one is a prebuilt state file with no HTTP request in it. So the ingestion routes really were never under test. The scope failure was real. The story explaining it was not.

The findings sorted by one thing

Looking at all three afterwards, they split cleanly, and not along the line I'd have guessed.

Every claim that pointed at a path and a value was right. The score history, the goal.md criteria, the eval sizes — all verifiable in under a minute, all correct.

Every error was in a sentence that characterised the state of the system in prose. "No database." "No RLS." A field that behaves a certain way. Those read like observations and are actually summaries, and a summary is where a plausible detail can be added without anything contradicting it.

I got a second data point the same week without looking for it. A review bot on the pull request for this site flagged that a scheduled post would break the production build through generateStaticParams. There was a real defect there — metadata is generated before the page component's publish gate, so it read a translation namespace that gets stripped in production. But the mechanism it described had been closed days earlier. Right that something was broken. Wrong about what, and wrong in the direction of a more dramatic failure than the real one.

The part I don't like

The invented claim was the one I wanted to be true.

"A perfect score on an engine that would have processed zero percent of reality correctly" is a better sentence than anything I'd written that week. It would have made the post. I had to go and check it specifically because it was good, and I noticed I was hoping the check would come back clean.

That's the uncomfortable version of the rule I already had about a successful run not being evidence its result is correct. The rule is easy to hold when the output is boring. The output that's hard to reject is the one that flatters the thing you're writing.

Which is roughly the same shape as the argument in the last post: my own preference decided what got examined closely, and I was the only thing in the loop that could have caught it.

What I do now

Not "trust review agents less". Two of three findings were right, one of them found a fabrication I'd shipped, and the post is materially better than it was. That's a good trade at any hit rate.

What changed is the order I work in:

Sort by citability first. Anything naming a file and a value gets checked immediately — it's a minute, and it's usually right, and it's the fastest way to know whether the review is worth the rest of my time.
Treat prose characterisations as leads. "There's no X" is a hypothesis to go and test, never a finding to act on. Every error in both reviews was one of these.
Check the best claim hardest. Inverted from what I was doing. The most compelling finding gets the most scrutiny, precisely because it has the most pull on me.

None of that is expensive. Verifying all three findings took under an hour, most of it spent reading my own LOG.md, and it caught two fabrications plus one of mine.

Next

I want to know whether the split holds — cited claims right, characterised claims wrong, or whether I just saw it twice. That needs more reviews and a habit of writing down which findings survived, which I don't have yet.

I'd particularly like to hear from someone whose reviewer got a file-and-value claim wrong — that would break the only heuristic I've got. So: if you run AI review over your own writing or code, do the errors cluster anywhere for you?

23:T39ef,

A textarea in one of my figures would not scroll. Neither would a table with overflow-y-auto next to it. Clicking into them worked, the keyboard worked, the scrollbar was there — the wheel just did nothing.

The cause is smooth scroll. Lenis intercepts wheel events for the whole document so it can animate the page itself. Anything nested that has its own scrollbar never receives them.

The documented fix

Lenis looks for a data-lenis-prevent attribute and leaves that subtree alone:

<textarea data-lenis-prevent />
<div className="max-h-72 overflow-y-auto" data-lenis-prevent>…</div>

That works. It also has to be remembered on every scrollable element anyone ever adds. My globals.css had carried the matching rule for months, and a search showed that not one element in the codebase used the attribute — including two figures I had shipped that week.

The fix that does not depend on remembering

Lenis takes a prevent predicate. Instead of tagging elements, ask the node under the cursor whether it scrolls itself:

function isOwnScroller(node) {
  if (node.hasAttribute?.('data-lenis-prevent')) return true;
  if (node instanceof HTMLTextAreaElement) return true;
  if (node instanceof HTMLSelectElement) return true;

  // Only worth a style lookup for elements that actually overflow.
  if (node.scrollHeight <= node.clientHeight) return false;
  const overflowY = window.getComputedStyle(node).overflowY;
  return overflowY === 'auto' || overflowY === 'scroll';
}

const lenis = new Lenis({ prevent: isOwnScroller });

The scrollHeight check comes first on purpose: it is a cheap property read, and it means getComputedStyle only runs for elements that are actually overflowing. The attribute still wins where it documents intent.

Two things worth knowing

Horizontal scrollers are usually fine. With orientation: 'vertical', Lenis is not touching horizontal wheel or trackpad events, so a scrollable <pre> or table keeps working. Worth checking rather than assuming if you run it in both directions.
It is invisible in development if you skip Lenis on mobile. Mine is disabled under 768px and for prefers-reduced-motion, so a narrow window or a reduced-motion setting hides the bug entirely.

Get intouch

Cookie Settings

Get in
touch