2026-11-02 · Last updated 2026-11-02

We gave our coding agents cheaper models: half the cost, but not the token savings we needed

We tested a popular idea on our own work: keep the top model as the lead, and hand the actual coding to cheaper models. On 20 real tasks from our code base, run twice each way, every run passed its tests in both setups and the list-price cost fell by about half. Tokens are another story: our first cheaper setup used 27% more, and a slimmer re-run used 13% fewer. We had decided before we started that tokens had to fall by at least a quarter, so we're not switching. Here's what we measured and why it came out that way.

What we compared

Our work is split into small, tested changes that an AI agent does on its own: "add this function, make these tests pass". We picked 20 of them from our history, 10 easy (one or two files, under 50 changed lines) and 10 medium (up to three files, 80 to 140 lines). Each has its own tests, and we checked that the tests fail before the change and pass after it.

• Leader: the top model (Claude Opus 5.5) does the whole task.
• Tiered: a mid-priced model (Claude Sonnet 5.5) does the work. On medium tasks a cheap model (Claude Haiku 5.5) first reads the code and writes a short brief. If the tests fail, the task goes up to the top model once, with the failure log. Tests decide, never another model's opinion.

Every run got a fresh copy of the code with no history to peek at, and we put the original tests back before checking, so a worker couldn't pass by editing them. That's 20 tasks × 2 setups × 2 repeats = 80 runs, plus 20 more for the re-run below.

The rule, written down first: keep the pass rate within 3 points, and cut total tokens by 25% or more.

The results

On easy tasks the mid-priced model alone used 13% fewer tokens than the top model, at 44% of the cost. On medium tasks the cheap model's brief plus the mid model's work used 50% more tokens than the top model alone, at 48% of the cost.

The rule allowed one retry with a narrower role for the cheap model, so we ran the medium tasks again with no brief at all, just the mid-priced model (the "Tiered, no brief" column; easy tasks never had a brief, so their runs count for both). That turned the token count around: 13% fewer than the leader overall (somewhere between 5% and 22% fewer, at 95% confidence), at 44% of the cost. Still short of the 25% we set, so the answer stayed no.

Why the tokens went up

Every agent starts by reading the same big context. Before it sees the task, each agent loads its instructions, the tool definitions and the project notes: about 85,000 tokens in our setup. Then every step re-reads it from the cache. So the token count is mostly "number of steps × that starting context", and a cheaper model doesn't take fewer steps. Adding a second agent (the brief writer) pays for the whole starting context a second time.

The cache belongs to one model. A cached context is only reused by the same model. When work moves from one model to another, the next one writes its own copy first. Splitting one task across two models costs more than keeping it in one, even when each model is cheaper.

The cheap model's price step. Our cheap model costs about five times more per token once a single request goes over 100,000 tokens of context. With an 85,000-token start, that line is close: 57 of its 178 requests crossed it.

We never saw what happens when a cheap model falls short. Nothing failed, so the escalation path never ran. Our tasks were all within the mid-priced model's reach. Harder tasks could go either way, and this bench doesn't tell us.

What we're doing about it

We kept the part that cost nothing: a log of every agent's tokens, cache use and outcome, so the next decision comes from our own numbers. The tiers stay available, but our default stays one model per task. Cost per task did halve. If we paid per token and nothing else mattered, that alone would be worth it. Our rule was about tokens, though, and the biggest lever isn't which model runs. It's the size of the context every agent starts with, so that's the next thing we'll trim, and measure the same way.

The three worker files, the usage-log hook and the bench are free to use under the MIT licence: claude-code-tiered-workers on GitHub. Run the bench on a handful of your own tasks before you route anything by tier; your numbers may not match ours.

The limits

This is 20 tasks from one code base, all in Python, run on one machine in one day. All the agents ran inside one working session, so the starting context was the same size in both setups. Costs are list prices times the tokens each model reported, not an invoice. Two repeats per task can't show every run-to-run difference, though the pass rates didn't move.

We measure our own tools the same way we measured 200 small-business websites for AI visibility: write the rule down first, then report what came out. If you want to see what AI search tools can read on your own site, the free AI visibility checker runs the same checks.