Two pre-registered runs on local models, a reader's correction between them, and the one place the small model still breaks.
I wanted to know whether a 4B local model could replace a 12B one in a small tool-using agent. I ran it as an experiment, with the prediction written down before each run. Both models ran on Ollama on one laptop, at temperature 0.7, with no API.
The set-up
- 24 order-desk tasks: refund an order, refund and add a note, check stock and add a note.
- Each task ran clean and under three injected tool faults: a timeout, a one-off error, an empty result. That makes 96 task-and-fault combinations per round.
- Baseline: gemma4 12B. Candidate: qwen3.5 4B. Same prompt, same tools, same retry settings.
- The rule for the verdict was fixed before the run: is the 4B within 10 points of the 12B?
Run 1: my prediction lost
I predicted the 4B would be close enough. It was not. The 12B passed 96 of 96. The 4B passed 70 of 96, and the run stopped after one round with a verdict of BLOCK.
Most of the failures were one behaviour. In 21 of the 26, the 4B was asked to check stock for an order and its only call was to the stock tool, with the order id as the product code. It never looked the order up.
A reader caught a flaw in the test
In my set-up that wrong call raised an exception and ended the trial. The model never saw "unknown SKU". So I had measured "makes a wrong first call, where a wrong call is fatal". I had not measured "cannot do the task". I published a correction and ran it again.
Run 2: the error goes back to the model
One change, the same for both models: a tool's error is returned to the model as a message. My prediction, written first: inconclusive.
- 12B: 480 of 480.
- 4B: 428 of 480 (89%).
- The 4B made the wrong first call about as often as before: in 21 of 32 stock-note trials in run 1, and in 108 of 160 in run 2.
- Shown the error, it looked the order up next in 106 of those 108, and 68 of the 108 passed.
- Only 2 of 480 trials never did the lookup at all.
The test could not decide whether the 4B is within 10 points of the 12B. The gap landed at 10.8 points. So the verdict is INCONCLUSIVE: the run crossed neither the PASS nor the BLOCK threshold.
Where it still breaks
Two things going wrong in one trial. When the 4B made the wrong first call and then the lookup came back empty (the injected fault), it passed 0 of 26. In 21 of the 26 it read the error, looked the order up once, got nothing back, and stopped. The 12B retried the empty lookup every time, 120 of 120.
Where the wrong first call was the only lasting problem (no injected fault, or a timeout that a retry clears), the 4B went on to pass 52 of 58.
What I took from it
- Know whether your tools raise their errors or return them. Here it changed what the test measured: in run 2, 77 of the 4B's 428 passes came after a tool error that would have ended the trial under run 1's set-up.
- In this suite, looking at one problem at a time flattered the 4B. Half of its 52 failures in run 2 (26) were the compound case above.
- I did not see any of this by reading sample outputs. I saw it by counting what each trial did with its tools.
Limits
One suite, one prompt, two models, one machine. The two runs used different seeds, so 73% against 89% is two observations, not a measured effect of returning errors. The breakdowns were chosen after seeing the data. None of this says anything about these models in general.
Check it yourself
The harness is my own open-source tool, ARCI. It runs a baseline and a candidate on the same tasks with injected tool faults and gives PASS, BLOCK or INCONCLUSIVE with a stated error bound.
- Both runs, both pre-registrations and the correction
- The raw records of every trial, with the command that re-derives both verdicts
- The same comparison on your own two models
- Repository
If you have measured this on other small models, I would like to know: does returning tool errors close most of the gap, or is that specific to this pair?
The numbers were recounted from the stored trial records before publishing, and those records are public at the links above. First published on my blog, Real or Noise.
Top comments (5)
The correction is the real finding here — run 1 measured 'a wrong first call is fatal', not 'the 4B can't do the task', and returning the error instead of raising it moved the 4B from 70/96 to 428/480. That's a test-harness lesson, not a model-size lesson. The compound case is what I'd chase next though: wrong first call plus an empty lookup, 0 of 26, while the 12B retried every time. Is that a capability gap, or just a retry policy the small model was never given?
Thanks for reading it this closely.
It was given the policy, and the same one as the 12B. Both models ran with one system prompt, and its last sentence is "Retry an empty lookup once." The retry in code covers only timeouts and tool errors, once. An empty result is not an error, so reading the order again was each model's own decision.
Under the injected empty result the 12B read the order a second time in 120 of 120 trials. The 4B did in 87 of 120. In the compound case (wrong first call, then the empty lookup) it stopped after one lookup in 21 of the 26.
So it is not a missing instruction. Whether to call it a capability gap, I can't say from this run. I have not tested a setup where the code retries an empty result for the model, which is what would tell the two apart. I also chose these cuts after seeing the data.
One caution on 70 of 96 against 428 of 480: those are two separate runs with different seeds and a different number of rounds, so the difference is not a measured effect of returning the error.
Fair correction on 70/96 vs 428/480 — I read that as a before/after when it's two different runs, so the error-returning effect isn't measured. The retry detail reframes the compound case for me: both models had the same instruction, and the 4B still chose not to re-read a third of the time, so it's a decision difference, not a missing policy. The code-retry variant is the test that would settle whether it's a capability gap — are you planning to run it, or is the next post going a different direction?
The records show different actions under the same instruction. They do not establish a capability gap.
One more count from the run 2 records. Under the injected empty result the 4B read the order a second time in 87 of 120 trials, and 86 of those 87 passed. None of the other 33 passed. I chose this cut after seeing the data, so it is a description, not a test.
I did not run the code retry variant. A code retry would test the whole system with the recovery handed to code. It would not isolate what the model decides, and it would not by itself make the task succeed. A scripted test with no model in the repository shows how such a retry behaves.
I ran the neighbouring test instead, where the model still has to decide but is told at the point of failure. Same 4B model in both arms, on the 8 stock note tasks under the empty result. One arm gets the empty result as null, as in run 2. The other gets "
lookup_orderreturned an empty result. Calllookup_orderagain with the sameorder_id." The rule and the message were registered before the first trial.Both arms already had "Retry an empty lookup once." in the system prompt. A small smoke run came before the registration. its trials are not counted.
Result: 15 of 16 passed with the message, 2 of 16 without. The test stopped at round 2. Its verdict was
BLOCK, and in this runBLOCKfavours the message: I put the message arm in the baseline seat on purpose, because the test has no verdict for "better". Under the test's stated assumptions this supports an advantage of more than 10 points on these 8 tasks. It is a threshold decision, not a measured gain, and it says nothing about the full suite.What it does not show: anything about the 12B model or a capability gap (it was in neither arm), or whether naming the next step is what helped (one wording, no control message). The 8 tasks were chosen from run 2's failures, and the run stopped early, so 15 and 2 are counts, not rates.
Run 3, with its pre-registration and raw records: github.com/ajaysurya1221/agent-rel...
The honesty about what this doesn't show is what makes it useful — 15 vs 2 on 8 pre-selected tasks is a threshold decision, not a measured gain, and you say so plainly. The interesting bit is that the retry policy arrived as an instruction in the message, not as model capability — which suggests the small model's gap was guidance, not reasoning. Is the next test a control message, to check whether naming the next step specifically is what helped?