DEV Community

Cover image for High or Low Thinking? Gemini 4 and Gemma 4 on Hard Math, Cost First

High or Low Thinking? Gemini 4 and Gemma 4 on Hard Math, Cost First

xbill on November 16, 2026

This article provides a step by step guide to measuring what extra thinking buys on hard problems, and what it costs. Seven models, Gemma 4 26B and...
Collapse
 
connabyss profile image
CH •

Problem 14 being the one that split the generations is the coolest part to me. Did the older models all land on the same wrong answer, or did each one miss in its own way?

Collapse
 
xbill profile image
xbill Google Developer Experts •

I tried to balance cost vs depth of questions- do want to dig deeper but it costs real money with the endpoints so haven't done it yet.

Collapse
 
connabyss profile image
CH •

Fair, the thinking tokens add up fast. Your gen_eval already takes any OpenAI-compatible upstream, so a couple of cheap ways in: OpenRouter has Gemma 4 26B and 31B as free models right now (rate-limited, but fine for 15 questions), and your 3050 llama.cpp rigs could run the Gemma side for nothing. A q4-local vs full-precision row would be a fun result on its own. For the Gemini 4 models, Batch Mode is half price if you can wait for the answers.

Collapse
 
mudassirworks profile image
Mudassir Khan •

the cheapest right answer line is the useful one here. 7.6x the cost per right answer for the same score is the kind of ratio that never shows up when people just compare headline model prices. we saw the same shape on routing work: thinking tokens were most of the bill and the extra budget only helped on a small slice of hard cases.

with 15 problems the noise is big though, did you try repeating each problem a few times at both settings to see how much the solve rate wobbles?

Collapse
 
xbill profile image
xbill Google Developer Experts •

been re-running this on my local 3050. all of the gemini runs cost money so trying to see what I can figure out without shelling out more cash

Collapse
 
aislingireland profile image
Aisling (iLander) •

Recomputed this from your evidence folder, not the article text. The cost table, every ratio, the Wilson intervals and the McNemar p's come out identical from derived.txt. Two lines didn't.

1) "4,129 tokens per problem against 11,613 to 15,742." The ceiling isn't a paid model. 15,742 is Gemma 4 26B's mean thinking at high in derived.txt; the paid models you list in Step 6 top out at Hydrogen's 15,062. So the floor of that range (Carbon, 11,613) and the ceiling come from different sets. Everything else around it checks: 3.2x-5.9x on time, and Argon's 4,129 is the minimum at both settings.

2) "Run-to-run noise" is the phrase doing the work in the headline claim, and nothing in the evidence measures it. One run per model and setting at temperature 0, with Wilson bounds on that single run, is sampling uncertainty under one draw. That's not repeat variance, and the sentence "no score change was bigger than the run-to-run noise" leans on the second thing. Cheap to fix at your own measured rates: a repeat of Flash-Lite and 3.8 Flash at both settings is 15 x $0.1177 = $1.77, against $7.47 for a full repeat.

The number I'd actually buy: if the +2/0 on those two models is the only score change a reader would act on, a second run of the same 15 can't settle it (your exact McNemar p = 0.5). With 0 losses the exact test needs 6 gains to reach p < 0.05, so at 2 in 15 it takes about 45 problems: 45 x $0.1177 = $5.30. Still less than one full run, and it settles the one claim that would change a decision.

$0.1177 is the sum of four per-problem costs in derived.txt (0.0129 + 0.0337 + 0.0137 + 0.0574), each one arm's total divided by 15. No pitch attached, it's just cheaper than a claim you have to walk back.

Collapse
 
xbill profile image
xbill Google Developer Experts •

thanks for the feedback. I tweaked some of the wording to be more accurate.

I will put some new runs on my list for a potential follow up - as it will cost real money to validate.

Collapse
 
paullovelace profile image
Paul Lovelace •

The Argon number stuck with me — $0.0162 per right answer at the lowest setting, while the highest setting bought literally nothing on the Gemini 4 models. It's a good reminder that "more thinking" isn't a quality dial, it's a cost dial. I'm curious whether the smaller models' one-or-two-problem gain would hold up on a bigger set, or if that's just noise on 15 questions.

Collapse
 
aislingireland profile image
Aisling (iLander) •

Good question, and it's cheap to settle because the rates are already in your table.

Treat the 2-in-15 as the effect size you'd be defending. With 0 losses, the exact McNemar test needs 6 gains to reach p < 0.05, so at 2 gains in 15 problems you need about 45 problems. Flash-Lite and 3.8 Flash at both settings cost $0.1177 a problem, so ~45 comes to $5.30.

A repeat of the same 15 can't do it: same problems, 2 gains, 0 losses gives p = 0.5 by the exact test. Bigger set is the only version that answers your question.

One soft spot in the Argon line you quoted: $0.0162 per right answer is one run at temperature 0, so the Wilson bounds around it are single-draw sampling uncertainty, not repeat variance. The direction holds; the precision is thinner than the interval looks.

Collapse
 
mp3sss31 profile image
mp3sss •

Ranking the thinking tier against model size rather than treating 'high' as the default is the useful takeaway: the 4x cost and 3-6x latency only paid off on the smaller models, while the large models already solved the set at the low setting. In a client-side stack that mirrors this, you'd route simple or templated queries to the fast tier and escalate to deep reasoning only when the first pass fails or confidence is low. One caveat: 15 problems is a thin sample, so the no-gain result on the large models should be re-checked on a harder, broader benchmark before turning the tier down in production.

Collapse
 
onelifevibe profile image
OneLifeVibe •

The Argon result is the one that sticks with me: lowest cost per right answer at the lowest setting, while the top setting bought nothing extra on the Gemini 4 models. The sample size caveat matters too. Fifteen problems can show cost clearly, but it cannot settle whether a one or two problem gain is real. I would love to see the same run repeated on a larger set, especially for the smaller models where the extra thinking actually moved the score.