# Switching to a half-price LLM model cut search stories from 12 to 4, and my regression test stayed green
> Switching LLM model to a cheaper one halved the input price and cut search stories from 12 to 4. The regression test reported zero failures.
Canonical: https://brandonlazovic.dev/articles/half-price-model-swap/
Author: Brandon Lazovic
Published: 2026-10-19
## The short version
- A cheaper model is not a drop-in replacement, because a prompt can quietly come to rely on the previous model's habits. On an unchanged prompt, the cheaper model kept all 9 model-release stories and cut the search-and-commerce group (news about search engines and shopping) from 12 publishable stories to 4.
- Rewriting the written criteria recovered much of the loss without going back to the previous model. The search-and-commerce count rose from 4 to 8, and the seven stories published on October 7 went from 3 of 7 to 7 of 7.
- My replay check recorded zero failures on the bad run, because it compared two prompts on the same model, and a model that demotes a story demotes it in both. The repaired check compares against the verdict (publish, hold or drop) the live system gave each story on the day.
- Half price is true of the input price only. Output fell by 58%, and the result comes from one prompt, 35 saved entries and five runs per set.
Switching the sorting step of my daily news digest to a cheaper model changed which stories it kept, and the check built to catch that change reported nothing wrong. In my replay, a re-run of the sort on a fixed set of past news entries, the cheaper model on an unchanged prompt tagged all 9 model-release stories as publishable, meaning marked publish, and cut the search-and-commerce stories from 12 to 4. [6]
Rewriting the criteria raised that count to 8, and the previous model never came back. If your prompt has quietly come to rely on one model's habits, a price table will not show it.
> On an unchanged prompt, the half-price model tagged 4 search-and-commerce stories publishable where the previous model tagged 12, and the replay check recorded zero failures.
## What happened when I switched the sorting model to a half-price one?
Pulse, the daily news digest on this site, uses a small language model to sort each day's news entries into publish, hold or drop. When I moved that sorting step to a cheaper model on an unchanged prompt, the search-and-commerce group (news about search engines and shopping) fell from 12 publishable entries to 4. The model-release group (announcements of new AI models) stayed at 9. [6]
Each run sorted a fixed set of 35 saved news entries, matched by exact title, so a change to the prompt or the model can be compared against a known answer.
Every entry was sorted five times per run, and the record keeps the most common verdict. "Publishable" in the table means that verdict was publish. Which model ran each file is not written inside the file. It comes from the dated change note in the script, so treat the model labels as that note's account. [7]
The last two columns show, in order, how many of the seven stories I published on October 7 each run still marked publish, and how many failures the replay's own check reported.
| Run | Prompt | Model | Search-and-commerce group | Model-release group | Stories published October 7 (of 7) | Replay check failures |
|---|---|---|---|---|---|---|
| October 7 | Version 4 | `gpt-5.6-luna` | 12 | 9 | 4 | 0 |
| October 8, first run | Version 4, unchanged | `gpt-6-luna` | 4 | 9 | 3 | 0 |
| October 8, second run | Version 5, rewritten | `gpt-6-luna` | 8 | 9 | 7 | 0 |
The two story groups from the table above, one pair of bars per run. The amber bar is the search-and-commerce group, which fell and partly recovered, and the indigo bar is the model-release group, which never moved. [6]
The cheaper model lost eight search-and-commerce entries on the unchanged prompt. Six were posts about hosting models in a data warehouse or running a data pipeline, five of them "now on Snowflake Cortex AI" announcements and one an Amazon pipeline-monitoring post, and each went from publish to drop. The other two, "Google Drops Review Snippets For Healthcare Web Pages" and "Google AI Overviews Add To Calendar Feature", each moved from publish to hold. [6]
The five runs also disagreed with each other on some stories. On the cheaper model with the unchanged prompt, the healthcare story was hold on 4 of 5 runs and the Search Console country filter was hold on 3 of 5.
## Is a half-price model really half the price?
Half price is true of the input price only. As listed on OpenAI's model pages on October 12, 2026, `gpt-6-luna` charges $0.10 per million input tokens against $0.20 for `gpt-5.6-luna`. [1] [2] Output costs $0.50 per million tokens against $1.20. [1] [2] A token is a small chunk of text, often part of a word, that a model reads or writes. Output fell by 58%, by arithmetic on those two prices. [1]
Cached input, which is text the provider has already processed once, went from $0.02 to $0.01 on the same pages. [1] [2] For scale, `gpt-6.1-sol` lists at $2.00 input and $10.00 output. [3] The model `gpt-6-astra` lists at $10.00 and $50.00. [4] By arithmetic on the two pages, the first is one-fifth of the second.
Anthropic's Haiku 5.5 post, dated October 7, 2026, says "On average, it now costs around 75% less to run."
Its price table lists $0.10 per million input tokens for prompts up to 100,000 tokens, against $1.00 for Haiku 4.5, and $0.50 for output against $5.00. [5] That is a tenth of the list price for prompts under 100,000 tokens, and about 75% less to run by Anthropic's own estimate. The page never says 90% off, and the tenth is my arithmetic on its table. [5]
Why a falling price pulls agents into reach is the argument of [the piece on the Sonnet 5 cost curve](/articles/sonnet-5-agent-cost-curve/). This piece starts after the price has fallen, with what the switch did to the output.
## Why did my regression test stay green?
A regression test re-runs saved cases after a change and flags any answer that moved. My regression test stayed green because its rule that a story must stay published compared the current prompt against an older prompt run on the same model. A model that demotes a story demotes it under both prompts, so the two sets of runs still agree and the check passes. [7]
The unchanged-prompt run recorded zero failures. All 35 rows read as passing, while eight search-and-commerce entries were lost. Inside that comparison, the older prompt's count of publishable entries fell too, from 6 to 3 between the two runs, and those two counts are not in the table. [6]
The repair, which shipped on October 8, 2026, compares each story against the verdict the live system gave it on the day (publish, hold or drop). That answer is frozen, so no model change can move it. [7] A first attempt at the rewritten prompt failed two of those checks before the passing run, which shows the repaired check can fail. [6]
My read: this is the same shape as the check in [the piece on prompting practices that survived testing](/articles/prompting-practices-that-survived/), where a comparison fails to catch a flaw that both of its sides share.
## Did fixing the prompt work without going back to the previous model?
Rewriting the written criteria of the news-sorting prompt recovered many of the search-and-commerce stories the cheaper model had dropped, and the previous model was never needed. Prompt version 5 retired the data-warehouse beat, the category of data-platform news, and named the kinds of confirmed search changes that belong in publish. On the cheaper model, the search-and-commerce count rose from 4 to 8 and the seven stories published on October 7 rose from 3 of 7 to 7 of 7. [6] [7]
The count did not return to 12. Prompt 5 also dropped the data-warehouse beat on purpose, and six of the eight lost entries were that kind, so I cannot say how much of the remaining gap is the retired beat. I did not break that down. It is a gap in my analysis.
Suspected, not confirmed: the previous model filled gaps in my criteria with its own habits, and the cheaper model filled the same gaps differently. It held 9 of 9 on the model-release stories, and the two search stories it demoted fell into a gap in what I had written.
The replay is cheap to run. The script's header, measured October 7, 2026, puts one repeat of both prompts at about 34,400 tokens in and 3,600 out, "roughly 1.1 cents" on `gpt-5.6-luna`, and about 6 cents at five repeats. [7]
## What should you run before switching an LLM model?
Before you switch a model, re-run a fixed set of past cases on the cheaper model with the prompt unchanged, and compare the counts per category against an answer the model cannot move. In my replay the loss sat in one topic group while the other held at 9 of 9, so only a per-group count showed it. The steps:
1. Freeze a set of past cases, and record the answer your live system gave each one on the day.
2. Run the cheaper model on the unchanged prompt, several times per case, and keep the most common verdict.
3. Count per category and read every case that changed, since a total can hide a loss in one corner.
4. Edit the written criteria only after you have read what was lost.
State the counting method before you count, as [the piece on five wrong first numbers](/articles/name-the-counting-method/) argues.
## What does the half-price model replay not show?
The replay covers one prompt, one fixed set of 35 news entries and five runs per set. It does not show that a cheaper model loses search stories in general, and the table has no row for the previous model on the rewritten prompt, so it cannot say whether the rewrite would have helped that model too.
Model labels come from the script's own change note and not from the record files. The counts are the most common verdict of five runs, and some verdicts were split. The one finding I would defend beyond these 35 entries is narrower. A regression check that compares two prompts on the same model cannot see a model change.
## Sources
1. OpenAI API documentation: gpt-6-luna model page (prices read 2026-10-12) — https://developers.openai.com/api/docs/models/gpt-6-luna
2. OpenAI API documentation: gpt-5.6-luna model page (prices read 2026-10-12) — https://developers.openai.com/api/docs/models/gpt-5.6-luna
3. OpenAI API documentation: gpt-6.1-sol model page (prices read 2026-10-12) — https://developers.openai.com/api/docs/models/gpt-6.1-sol
4. OpenAI API documentation: gpt-6-astra model page (prices read 2026-10-12) — https://developers.openai.com/api/docs/models/gpt-6-astra
5. Anthropic: Introducing Claude Haiku 5.5 (October 7, 2026) — https://www.anthropic.com/claude-haiku-5-5
6. First-hand: three Pulse triage replay record files dated 2026-10-07 and 2026-10-08, counted by hand 2026-10-12
7. First-hand: the version notes and model change note in the Pulse harvest script, the replay script's rule text and header, and the repair commit of 2026-10-08