Nov 9, 2026
The token cap was grading the test
Piotr and I run a small benchmark harness that scores chat models on a fixed set of items: tool calls, coding, reasoning, following instructions. Every item has a token cap, so one rambling model cannot stall the run. Some of the scores were a measure of the cap.
A small model that got worse by thinking
The first case was a 4-billion-parameter local model on an older consumer GPU. On the 65-item quick tier, with thinking off, it scored 59 of 65 in 135 seconds. I gave it thinking and 512 extra tokens per item to see whether it would help. It scored 42 of 65 in 857 seconds.
The coding category is where it fell apart: 10 of 12 with thinking off, 1 of 12 with it on. The results file says why:
With 512 extra tokens per item, it thought until the cap and often never wrote an answer. 11 of 12 coding replies had no function in them (
NameError: name 'chunk' is not defined).
As far as I can tell from that NameError, the grader found no function, ran the test anyway, and recorded a failure. It was correct that there was no function. It was not measuring whether the model could write one.
There was a louder version of the same thing. Through an OpenAI-compatible endpoint, thinking has to be switched off explicitly with reasoning_effort: "none". Without it, the same model “spent every token cap thinking and scored 15 of 65.”
A hosted model that thinks where I cannot see
The second case was a small hosted Claude model. I had asked for thinking off. It thinks anyway: by the harness notes, about 55 hidden tokens before a one-line answer, and the endpoint returns none of them. So the reply I received was short, and with the caps as tight as I had set them, some answers stopped mid-sentence. The harness scored those as wrong answers.
At that point, nothing in the score said “ran out of room”. The harness counted a cut-off answer as a wrong one, and from the scores alone the two look the same. I loosened the caps (128 tokens for answer-only items, 400 for reasoning and agentic turns).
That was not enough. When I swapped in borrowed public benchmarks, the caps on those items were still tight. On the resulting 62-item set, which mixes my own items with borrowed ones, the hosted model scored 52 of 62, and 9 of the 10 misses were replies that stopped at the cap, some of them empty. Only one was a plain wrong answer. The caps on those items were 96 to 400 tokens (96 on one of my own instruction items, 128 to 400 on the borrowed ones).
So I raised them three to four times over: 128 to 512 on one benchmark, 200 to 800 on another, 400 to 1200 on the borrowed coding benchmark. On the same 62 items the hosted model scored 59 of 62, with no replies cut off (single runs; this model varies by about an item between runs). It still misses three items, and those are now ordinary misses rather than truncations. Most of the gain was the hosted model’s. The local 4-billion-parameter model, which had 2 cut-off replies before, moved from 57 to 58 of 62 with 1 left. That is one item: the borrowed coding reply that had been cut off now finishes. The 4B runs at temperature 0 and failed the same items in every run at each cap, so I read that one item as real. The cost is time. The hosted model went from 86 to 103 seconds. The cold 4B went from 173 to 189 seconds, 9 seconds over the 3-minute target I had set at the time. One item (repeat the prompt) ran to its 1024-token cap and took 25 seconds, and loading the cold model took about 40.
The cap is part of the test
I had treated the token cap as plumbing: a guard against runaway output, set once and forgotten. It is part of the test. If a model’s thinking, visible or not, draws on the same budget as its answer, then a tight cap measures how terse the model is.
The harness now reports a cut-off reply as its own outcome, next to the failures rather than inside them.