Oct 23, 2026
ollama ps forgot to count the part that makes it fast
TL;DR. I benchmarked a smaller third-party quant of a 27B model against the stock Ollama tag on a 32 GB card, chasing a video’s claim that the quant was “task-lossless” at 5 GB less memory. The quant checked out — same output quality, similar speed. The thing actually worth knowing came from a side measurement: ollama ps, the built-in command for checking a loaded model’s VRAM use, undercounts by close to 10 GiB whenever a model ships a speculative decoding head (a small bolted-on predictor that drafts several tokens ahead, which is also most of why these models feel fast). It simply doesn’t count that head’s cache. If you’re using ollama ps to decide how much headroom you have for a second model, and your model has one of these heads, the real number is meaningfully smaller than what it tells you.
At a glance
| Real VRAM | ollama ps says |
Gap | |
|---|---|---|---|
| Stock model, head on | 26.9 GiB | 17.3 GiB | 9.6 GiB missing |
| Stock model, head removed | 24.6 GiB | 24.0 GiB | 0.6 GiB |
| Quant, head on (1 draft token) | 21.6 GiB | 12.5 GiB | 9.1 GiB missing |
| Quant, head removed | 19.6 GiB | 19.6 GiB | 0.0 GiB |
The pattern is binary: every configuration with the speculative-decoding head on undercounts by roughly 9 GiB; every configuration with it off, the tool is accurate to within rounding.
The two bars line up almost exactly in the bottom pair and diverge by roughly a third in the top pair — same shape as the table above, easier to see at a glance.
The research
A video about budgeting a 16 GB GPU for local LLMs — “Is 16GB All You Need for Serious Local LLM?” — doesn’t translate cleanly to a 32 GB card, but one claim in it was specific enough to test: a third-party quant of a 27B model, built “task-lossless,” fitting in 11.8 GB instead of the usual 17. If true, that’s 5 GB of headroom on a card running a 16.8 GB model. So I downloaded it, built four Ollama tags to compare against the stock model, and ran a short coding prompt and a 33,000-token transcript through each one, alone — I checked ollama ps before each reading, and nothing else was loaded.
The results were unremarkable in the way that matters: both files produced, by eye, the same code and the same answer to the long prompt, and the smaller quant’s decode speed landed within the same range as the stock file once you account for what’s actually driving it — a speculative decoding head bolted onto the model, present in both files, worth roughly 1.6x to 2.7x on decode speed, more on the short prompt than the long one. Strip the head from either file and both land in the low-20s tokens per second. Keep it in the stock file and the short prompt hits 62.6; keep it in the smaller quant and the same prompt hits 47.6 to 51.3 depending on how many tokens it drafts at once — slower, not faster, once the head is doing the same job in both. On the long prompt the gap mostly closes: 36.5 to 39.2 against 37.9. I found no public benchmark of this quant format on this card’s architecture before now, for what that’s worth; it wasn’t the thing that mattered.
The part worth writing down is a tool I trusted that was wrong by over a third of what the card actually had committed.
ollama ps reports how much VRAM a loaded model is using. I’d been reading that number as ground truth all session — it’s the only built-in way to check, and it agrees with itself every time you ask it. For the stock model it reported 17.3 GiB. I also had a Windows performance counter running against the GPU’s own dedicated-memory usage, and it said 26.9 GiB — on a card with 32. ollama ps was undercounting by 9.6 GiB, which is not a rounding difference. Every model in the comparison that carried the speculative-decoding head showed the same kind of gap; every model without one matched the counter almost exactly. The reported number simply excludes the KV cache belonging to that extra head — I don’t know whether that’s a known limitation or just never came up before speculative decoding started shipping inside the model file itself.
If I’d sized a second model to load alongside the first using ollama ps’s number, I’d have called it 15 GiB of headroom. It was about 5.
The second thing worth flagging is boring by comparison, but it’s the one that actually misleads a reader: an earlier benchmark note of mine described the stock model’s second GGUF file as its speculative-decoding head. It’s the vision projector — a separate 931 MB file for image input that happens to load alongside the text model and has nothing to do with why the model is fast. The actual head lives inside the main file, as a block of extra tensors at the end. Both files are present every time you pull the tag; only one of them does anything for a text-only session. That line is fixed now.
The full numbers, with decode and prefill speed alongside the memory figures:
| Model | Decode, short prompt | Decode, long prompt | Prefill | VRAM, real | VRAM, ollama ps |
|---|---|---|---|---|---|
| Stock, head on | 62.6 tok/s | 37.9 tok/s | 581 tok/s | 26.9 GiB | 17.3 GiB |
| Stock, head stripped | 23.2 tok/s | 21.8 tok/s | 539 tok/s | 24.6 GiB | 24.0 GiB |
| Quant, head on, drafts 4 | 51.3 tok/s | 36.5 tok/s | 533 tok/s | — | — |
| Quant, head on, drafts 1 | 47.6 tok/s | 39.2 tok/s | 540 tok/s | 21.6 GiB | 12.5 GiB |
| Quant, head stripped | 24.8 tok/s | 23.1 tok/s | 559 tok/s | 19.6 GiB | 19.6 GiB |