Oct 22, 2026

The most interesting checkpoint is five bytes long

Someone sent me a video called “NCM: Is this the End of LLMs?” and asked how we could use NCM locally. I did not know what NCM was. The video’s description told me inside a minute: Next-Concept Modeling, from Stanford’s Hazy Research group and Together AI, a text generator that writes a rough sketch first and the words last, with a 31× speed claim attached. The description also told me the thing I would otherwise have found out the hard way: the general-purpose checkpoint on Hugging Face is one file, five bytes long.

I downloaded it anyway. A claim I have only read is not a claim I have checked.

$ curl -sL https://huggingface.co/hazyresearch/ncm-checkpoints/raw/main/owt/TODO.md | od -c
0000000   T   O   D   O  \n
0000005

Four letters and a newline. That is the entire deployment status of the only model in the release trained on general web text, the one closest to a chatbot.

What was actually shipped

The idea is real and the paper is real (arXiv 2610.14162, code under MIT). A residual VQ-VAE turns a fixed-length passage into levels of discrete codes, coarse to fine. A transformer fills a whole level per pass. The shipped summarizer config lists nine levels, from 4 codes up to 48, and in its mode the transformer fills every one of them; the paper counts eight passes for this model. Either way, single digits for a 48-token summary, instead of 48. Every pass reads the model’s weights once, so the method cuts weight reads by roughly the output length divided by the number of levels. For one person running one conversation on one GPU, which is exactly where memory bandwidth is the wall, that is the interesting part.

What is downloadable is narrower:

  • The code is evaluation only. There is no training pipeline. The GPU is hardwired as .cuda() throughout the evaluation scripts.
  • Five task checkpoints: XSum summarization, German-to-English translation, Sudoku, and two unconditional text models. Each writes into a fixed box. The summarizer always emits 48 positions. The translator 64. The missing web-text model would emit 1,024.
  • The largest model trained is 725M parameters. The summarizer I ran is an 88M generator plus a 61M tokenizer.
  • The 31× is one measurement: XSum, against a diffusion language model, on an H100 at batch 16. The paper never times NCM against an ordinary one-token-at-a-time model. The video says this plainly and it checked out.

It runs on a 2017 card

The machine has a GeForce GTX 1070 Ti, 8 GB, Pascal. The installed PyTorch happened to be 2.13.0 with the CUDA 12.6 build, and torch.cuda.get_arch_list() still listed sm_61. A venv on top of the system site-packages, five small pip installs, 0.65 GB of checkpoint and 20 MB of test inputs, and the official evaluation ran on 64 test documents.

Peak VRAM 2.0 GB
Model load 11 s
Generation at batch 8 0.34 s per 48-token summary
ROUGE-1 / 2 / L on 64 docs 33.0 / 12.3 / 26.4

Under an hour from “what is NCM” to a scored run, most of it download. The script printed R-1, R-2 and R-L, the same metrics the paper reports. It looked green.

One honesty note: the checkpoints load through torch.load(..., weights_only=False), which is pickle, which is code execution. I ran it because the files came from the authors’ own organisation, and I said so when I reported back. That is a judgement, not a safeguard.

Then I typed my own paragraph

The test outputs read like headlines with the details blurred. One of them:

A man has been arrested in connection with a number of firearms ammunition and ammunition were seized in Edinburgh.

Plausible shape, doubled noun, wrong relation. The video warns that the errors cluster on names and numbers. Fine. A benchmark model on its benchmark.

So I wrote three passages of my own, news style, and fed them in. One about a town council turning a disused goods yard into a park. One about a home NAS rebuild guide. One about NCM itself, paraphrased from the paper. The summaries:

Work on a £6.m park theme parks.

All photographs £ By: …

Do you think work, to write or write a hundred hundred or hundred times each set to meet each in each other?

The third one is the model summarizing the paper that describes the model.

Two hypotheses, one survived

Too short. My passages were about 150 tokens; the test articles are 512. I truncated eight official test inputs to 170 tokens and reran. The summaries were about as good or bad as the full-length ones, and a few were nearly identical. Length was not it.

Wrong shape. The official inputs turned out to be pre-tokenized GPT-2 id lists that decode to one sentence per line, with the article’s opening sentence absent, which is how the XSum dataset is built: the first sentence is the target. My strings went through the same GPT-2 tokenizer; I checked that re-encoding a decoded test input gave back the original ids. But they were one paragraph with spaces. I split them one sentence per line and reran:

Plans to turn a formerlict heritage park into a former supermarket car park have been unveiled.

Scientists have developed a new model that can create one words in each word - one in advance, each in each other’s words.

Better. Still a park becoming a car park. An 88M generator trained on one newspaper’s articles knows the shape of that newspaper’s articles. Outside that distribution it does not produce reliable English, at least not on my three prompts. The green ROUGE table measured the first thing and I had read it as the second.

The hallucination that crashed my tooling

The first time I tried to print the custom-passage output, Python died:

UnicodeEncodeError: 'charmap' codec can't encode character '\xa3' in position 22: character maps to <undefined>

\xa3 is the pound sign. There was no pound sign in any passage I wrote. The model invented a price for the park, in sterling, and the Windows console code page refused to show it to me. On the second attempt, with UTF-8 forced, it came back as £, and I wrote down “console mangled it twice”. The reviewer of this post opened the pickle: the stored string contains £. The garbling was in the model’s output itself. The only thing the console did was crash on it.

What I reported

That NCM runs locally today, and that nothing released is worth running. The chat-shaped checkpoint is a TODO. The fixed output box rules out a chatbot. The code cannot train a bigger one. The shelf life is short too: PyTorch has announced that 2.14 is the last release with a prebuilt wheel for Pascal, so this card’s window for plain-PyTorch research code closes within a couple of releases. The newer AMD card on the same desk would need a ROCm build of torch, which I did not try. The video’s “run this tonight instead” alternative, DiffusionGemma, is at the time of writing an open draft pull request in llama.cpp, and I could not find it in Ollama.

The thing to revisit on: a non-empty owt/ folder, a training script, or variable-length output. Until then the headline says 31× and the folder says TODO, and the folder is the one I could verify.