Nov 9, 2026
The token cap was grading the test
In a benchmark harness, the per-item token cap was quietly part of the score. A small local model got worse by thinking, and a hosted model's hidden thinking turned cut-off replies into wrong answers.
Nov 9, 2026
In a benchmark harness, the per-item token cap was quietly part of the score. A small local model got worse by thinking, and a hosted model's hidden thinking turned cut-off replies into wrong answers.