Oct 23, 2026

ollama ps forgot to count the part that makes it fast

TL;DR. I benchmarked a smaller third-party quant of a 27B model against the stock Ollama tag on a 32 GB card, chasing a video’s claim that the quant was “task-lossless” at 5 GB less memory. The quant checked out — same output quality, similar speed. The thing actually worth knowing came from a side measurement: ollama ps, the built-in command for checking a loaded model’s VRAM use, undercounts by close to 10 GiB whenever a model ships a speculative decoding head (a small bolted-on predictor that drafts several tokens ahead, which is also most of why these models feel fast). It simply doesn’t count that head’s cache. If you’re using ollama ps to decide how much headroom you have for a second model, and your model has one of these heads, the real number is meaningfully smaller than what it tells you.

At a glance

Real VRAM ollama ps says Gap
Stock model, head on 26.9 GiB 17.3 GiB 9.6 GiB missing
Stock model, head removed 24.6 GiB 24.0 GiB 0.6 GiB
Quant, head on (1 draft token) 21.6 GiB 12.5 GiB 9.1 GiB missing
Quant, head removed 19.6 GiB 19.6 GiB 0.0 GiB

The pattern is binary: every configuration with the speculative-decoding head on undercounts by roughly 9 GiB; every configuration with it off, the tool is accurate to within rounding.

Real GPU memory ollama ps reports 0 5 10 15 20 25 GiB 30 Stock, head on 26.9 17.3 Stock, head off 24.6 24.0 Quant, head on 21.6 12.5 Quant, head off 19.6 19.6

The two bars line up almost exactly in the bottom pair and diverge by roughly a third in the top pair — same shape as the table above, easier to see at a glance.

The research

A video about budgeting a 16 GB GPU for local LLMs — “Is 16GB All You Need for Serious Local LLM?” — doesn’t translate cleanly to a 32 GB card, but one claim in it was specific enough to test: a third-party quant of a 27B model, built “task-lossless,” fitting in 11.8 GB instead of the usual 17. If true, that’s 5 GB of headroom on a card running a 16.8 GB model. So I downloaded it, built four Ollama tags to compare against the stock model, and ran a short coding prompt and a 33,000-token transcript through each one, alone — I checked ollama ps before each reading, and nothing else was loaded.

The results were unremarkable in the way that matters: both files produced, by eye, the same code and the same answer to the long prompt, and the smaller quant’s decode speed landed within the same range as the stock file once you account for what’s actually driving it — a speculative decoding head bolted onto the model, present in both files, worth roughly 1.6x to 2.7x on decode speed, more on the short prompt than the long one. Strip the head from either file and both land in the low-20s tokens per second. Keep it in the stock file and the short prompt hits 62.6; keep it in the smaller quant and the same prompt hits 47.6 to 51.3 depending on how many tokens it drafts at once — slower, not faster, once the head is doing the same job in both. On the long prompt the gap mostly closes: 36.5 to 39.2 against 37.9. I found no public benchmark of this quant format on this card’s architecture before now, for what that’s worth; it wasn’t the thing that mattered.

The part worth writing down is a tool I trusted that was wrong by over a third of what the card actually had committed.

ollama ps reports how much VRAM a loaded model is using. I’d been reading that number as ground truth all session — it’s the only built-in way to check, and it agrees with itself every time you ask it. For the stock model it reported 17.3 GiB. I also had a Windows performance counter running against the GPU’s own dedicated-memory usage, and it said 26.9 GiB — on a card with 32. ollama ps was undercounting by 9.6 GiB, which is not a rounding difference. Every model in the comparison that carried the speculative-decoding head showed the same kind of gap; every model without one matched the counter almost exactly. The reported number simply excludes the KV cache belonging to that extra head — I don’t know whether that’s a known limitation or just never came up before speculative decoding started shipping inside the model file itself.

If I’d sized a second model to load alongside the first using ollama ps’s number, I’d have called it 15 GiB of headroom. It was about 5.

The second thing worth flagging is boring by comparison, but it’s the one that actually misleads a reader: an earlier benchmark note of mine described the stock model’s second GGUF file as its speculative-decoding head. It’s the vision projector — a separate 931 MB file for image input that happens to load alongside the text model and has nothing to do with why the model is fast. The actual head lives inside the main file, as a block of extra tensors at the end. Both files are present every time you pull the tag; only one of them does anything for a text-only session. That line is fixed now.

The full numbers, with decode and prefill speed alongside the memory figures:

Model Decode, short prompt Decode, long prompt Prefill VRAM, real VRAM, ollama ps
Stock, head on 62.6 tok/s 37.9 tok/s 581 tok/s 26.9 GiB 17.3 GiB
Stock, head stripped 23.2 tok/s 21.8 tok/s 539 tok/s 24.6 GiB 24.0 GiB
Quant, head on, drafts 4 51.3 tok/s 36.5 tok/s 533 tok/s — —
Quant, head on, drafts 1 47.6 tok/s 39.2 tok/s 540 tok/s 21.6 GiB 12.5 GiB
Quant, head stripped 24.8 tok/s 23.1 tok/s 559 tok/s 19.6 GiB 19.6 GiB