This post was originally posted on Vaporlens Patreon
Hey folks!
VaporLens runs on deepseek-v4-flash. A newer version, deepseek-v4-flash-0731, was recently released, benchmarking better across the board. One string in a config file - so I switched to it and went to measure what that had changed.
Turns out it costs more. Between 1.5 and 3.5 times more per game depending on the processing step actually, and it's slower on every one of them. Both versions cost the same per token, so all of that is the newer one simply using more of them.
Figuring out why, and what to do about it, made me rewrite the biggest prompt in the project.
So yeah, swapping a model is a one-line change - the same way swapping a car engine is four bolts 😅
I ran six paired batches of 40 reviews through both versions, identical inputs, alternating which one went first.
The newer model spent ~2x the output tokens and had ~1.8x the latency. The answers came back the same size, sometimes slightly smaller.
The difference was coming almost entirely from reasoning tokens. Those are billed as the output, so a model that thinks twice as hard costs twice as much whether or not the answer improves (here it didn't). It thought longer and wrote less (which in some cases was good, but not worth ~2x cost increase).
Which meant the question turned from "should I upgrade a model" to "do I actually need all that reasoning".
I've already had reasoning off for some simpler prompts that don't need it (like summarization). So I turned it off on review extraction as well - biggest prompt in the pipeline, one that runs up to fifty times per game.
Output tokens dropped to ~0.4x, on an 86-review/9 batches sample run. Every batch came back complete, valid, and fast.
But the results were wrong in a very specific way. The model was still good at finding what was in a review. But it was also bad at not doing things.
Every rule that failed was a prohibition. "No money amounts" produced literal price figures. The rule saying an unhelpful review must return empty arrays got ignored. Sentiment drifted negative, flipping a clearly positive review because it contained one complaint.
Every rule that survived was a positive one. All the "extract X" instructions kept working.
That makes sense in hindsight. If you've ever worked with LLMs - you know how bad they are at following negative prompts. Producing something needs no thinking, you just produce it. Taking something away means checking a candidate against a rule and throwing it away, and that check has to happen somewhere.
The place to put the check back was already in the schema. Every extracted review has a reasoning string, and it comes first, before all the extraction arrays. Structured output is generated in order, so whatever lands in that field conditions everything after it.
Originally I've added it just to make the model double-check its thoughts and improve the results, so for thinking outputs it was a simple summary of thoughts. And it was already earning its keep, making the model commit to what it had actually seen before it started filling arrays.
But it only asked about details in general (a summary if you will). With reasoning off it had to carry the rule-checking too, so I turned it into an explicit checklist as a chain-of-thought output field: name the details, name the systems described, name the feelings and their causes, name what the review says about the product rather than about playing it, name the faults, and name anything present that the rules forbid recording.
An old trick from the days of non-reasoning models. Same idea as reasoning, just in the visible output, and about a hundred tokens only instead of thousands.
It worked on exactly the things I named, and only those. Money-rule violations dropped below what the current pipeline was doing. Gameplay points went from 26% below baseline to 9% above it.
So I've added every category, expecting to fix everything at once. Instead the categories I'd added recovered and the one that had been winning collapsed: gameplay went from 9% above to 19% below, and the total barely moved. Chain-of-thoughts redistributes attention - there's a limited budget (especially with limited non-thinking output), and each version I wrote just moved the deficit somewhere else.
Chasing that collapse was the most useful thing I did all week. The model's own reasoning field showed it naming the systems correctly and then returning an empty array anyway. Two causes surfaced: the gameplay rule says to leave it empty "if the user is just complaining", and my checklist had the model list every fault immediately before asking about systems. It talked itself into "this person is just complaining" every time. Reviews with more negatives than positives were two and a half times more likely to lose the category entirely. Moving systems ahead of faults, so the model describes before it judges, brought it straight back.
Two things I nearly got wrong while measuring this. Negatives looked 15% down, but that was counting array entries; counting characters of extracted text they were at +1%, because the model was merging multiple points into one that said the same thing. And I compared two runs with different corpus mixes and got an 8% gap that was really 14%. Benchmarking your own pipeline has all the same footguns as benchmarking anything else, and it's much easier to trust it.
There was also one run that returned a single review for a batch of forty. Valid JSON, correct schema, no error. Nothing noticed, because the mapping loop iterates whatever came back and the retry logic only fires on thrown errors. Now there's a new length check to ensure output matches input.
Extraction survived reasoning-off because I spent a few days tweaking its prompt. The obvious next question was whether the rest of the pipeline could just have the setting flipped: churn analysis, feature scans, structured extractions, hardware summary, and the curator blurb.
On tokens and time it looks free. Measured on Waterpark Simulator (~11k reviews):
| step | output tokens, r-on | r-off | time, r-on | r-off |
|---|---|---|---|---|
| churn analysis (4 batches) | 41,548 | 8,023 | 534 s | 136 s |
| 3 feature scans | 17,460 | 2,441 | 215 s | 38 s |
| 4 feature extractions | 38,890 | 2,305 | 462 s | 47 s |
| hardware summary | 4,089 | 312 | 52 s | 7 s |
| curator blurb | 1,361 | 35 | 11 s | 3 s |
| total | 103,348 | 13,116 | 21.2 min | 3.9 min |
Two and a half cents a game, and seventeen minutes.
Then I read the output. Every rule that broke was, again, a rule about what not to do:
| what the prompt asks for | with reasoning | without |
|---|---|---|
| every output string in English | 0 violations in 20 runs | 6 of 20 runs, one extraction 5 for 5 |
| score a fixed keyword tier | 50, 50, 50 | 10, 50, 100, 10, 75 |
| don't default to "mixed" for mostly-fine cohorts | 13 negative verdicts of 30 | 1 of 30 |
| keep the blurb under 200 characters | 0 of 12 | 3 of 12 |
| copy each review's language code from the input | 75 of 75 | one batch of 20 came back "unknown" |
The English rule is the biggest issue. Those strings render on the site, and without reasoning the model just uses dominating Spanish, German, French and Japanese language from reviews straight through.
The scoring one is worse than it looks. That scan is a lookup: four tiers, decided by which keywords appear, so there is a correct answer. Without reasoning it hit all four tiers on identical input (i.e. it was useless). One run scored 100 and justified it with evidence that appears nowhere in its own analysis.
So, same issue as before, confirmed across eleven prompts. The model is fine at producing and unreliable at refusing.
Both versions cost $0.14 per million input tokens and $0.28 per million output. Identical. So the only thing separating them on cost is how many tokens they use, and the newer one uses a lot more of them when reasoning is on:
| step | output tokens, current | 0731 |
cost |
|---|---|---|---|
| churn analysis | 25,570 | 47,505 | 1.73x |
| 4 feature extractions | 14,669 | 45,705 | 2.17x |
| modScan | 2,610 | 6,671 | 1.83x |
| wikiScan | 2,656 | 4,347 | 1.48x |
| protonLinuxScan | 2,236 | 9,489 | 3.05x |
| hardware summary | 1,014 | 2,831 | 1.88x |
| curator blurb | 401 | 1,663 | 3.51x |
Slower on all too. And 85 to 99% of the newer version's output is reasoning tokens, so almost none of what I'm paying for reaches the actual summary page.
One caveat I can't remove: I'm using Openrouter with default routing and different providers have different speeds. Time is therefore model plus provider. But it cuts in the newer version's favour, since its providers are about 1.6x faster per token and it's still 2x slower overall.
My next thought was - hey, API takes a reasoning effort parameter, which sounds like exactly the compromise I wanted!
So, I've tested it on the smallest prompt I have, so runs took seconds and I could repeat them properly. Seven providers, eight settings, five runs each, 280 calls. Median reasoning tokens:
| provider | default | xhigh | high | medium | low | minimal | cap 128 | off |
|---|---|---|---|---|---|---|---|---|
| DeepSeek | 982 | 2284 | 596 | 1135 | 697 | 411 | 386 | 0 |
| Fireworks | 579 | 172 | 1619 | 723 | 240 | 580 | 677 | 0 |
| Novita | 687 | 295 | 513 | 225 | 814 | 196 | 946 | 0 |
| Cloudflare | 534 | 391 | 600 | 333 | 92 | 46 | 446 | 0 |
| GMICloud | 1132 | 782 | 979 | 520 | 358 | 473 | 1236 | 0 |
| SiliconFlow | 392 | 729 | 222 | 2251 | 1318 | 338 | 305 | 0 |
Read any row left to right. Not one is doing what it's supposed to do. "high" beats "xhigh" on four of the six. That "cap 128" column is a hard cap on reasoning tokens, and six of the seven providers ignored it outright, one by a factor of ten. The spread inside a single cell is what settled it. Novita on "low", same input, five runs: 170 tokens, then 814, then 9,526. Any two runs of this experiment can tell you opposite stories, which is presumably how anyone ends up believing the dial works.
The only setting that reliably does anything is off, which produced exactly zero everywhere, every time.
Everything except extraction keeps reasoning, and keeps 0423 (old model). Two and a half cents a game does not buy a page that says "mixed" when it means "bad", quotes players in languages nobody asked for, and occasionally scores a game by a rule it invented.
Extraction is the one place the newer version makes sense, and only because of what I'd already done to the prompt. The entire reason 0731 looked bad is that it burns reasoning tokens. Take reasoning away and the gap closes to 0.92x:
| extraction, per game (500 reviews, 50 batches) | $ / game | min / game |
|---|---|---|
0423, reasoning on (what was shipping) |
$0.092 | 85 |
0423, reasoning off, new prompt |
$0.063 | 40 |
0731, reasoning off, new prompt |
$0.063 | 28 |
Same price to within half a percent, a third faster. That's the whole case for it: not cheaper, just quicker, at no extra cost, in the one place its appetite for thinking has been removed.
Put the two halves together for something the size of Waterpark Simulator, with everything but extraction measured on its 11,351 reviews and extraction at a 500-review sample, and the pipeline goes from $0.113 and 99 minutes to $0.084 and 42 minutes. A quarter off the bill, and well over half the wait. Not too shabby.
I wanted to simply upgrade a model to a newer version and ended up keeping the old one everywhere except a single step (at least for now).
TL;DR is - a model with better benchmarks isn't automatically an upgrade for your work. Benchmarks measure performance on generic(-ish) tasks, and yours aren't generic - and the only way to know is to run both on your own thing and evaluate the outputs yourself.
I do still plan to move to the new model everywhere, I just need some time to properly benchmark and rewrite existing prompts. One step took a week, so that'll probably be a while 😅
Cheers,
Tim