Short answer
NCM (Next-Concept Modeling) is a new research method from Stanford and Together AI that writes text coarse-to-fine: it drafts the overall structure first, then fills in detail, predicting many pieces in parallel instead of one token at a time like today’s LLMs. Early results are fast and efficient, but they are measured against diffusion language models at small scale, not against ChatGPT-class LLMs. It is worth watching, not acting on yet.
- 10–31× - faster generation than strong diffusion language models, in the authors’ tests
- ~4–5× - fewer training tokens, including tokenizer training
- 725M - largest model tested: far smaller than frontier LLMs
- 0 - hosted APIs: it is a research release with code on GitHub
NCM vs LLM is a comparison between two ways of generating text. Today’s large language models are autoregressive: they write one token at a time, each depending on the ones before. Next-Concept Modeling, published by Stanford’s Hazy Research lab and Together AI in October 2026, builds the text as a hierarchy, outlining it first and then refining every part in parallel. That makes it potentially much faster, but it is still early research.
Why generation speed is a business issue, not just a technical one
It is easy to treat “faster text generation” as an engineering detail. For decision makers it is three business levers at once: cost, response time and what your products can do.
- Cost: most of what you pay a model provider, or spend on your own GPUs, goes on generating output. If the same answer needs far fewer sequential steps, the same hardware can serve more requests.
- Response time: people abandon slow tools. Support agents, sales teams and customers notice the difference between an answer in one second and an answer in eight.
- New products: some ideas are uneconomic today because generation is too slow or expensive, such as drafting a personalised summary for every customer record overnight, or checking every contract clause in real time while someone types.
Illustrative example
A support team uses an AI assistant to draft replies of around 300 words. An autoregressive model produces those words one token at a time, so a long reply means hundreds of steps in a row. A coarse-to-fine model would produce the reply in a fixed number of levels, each computed in parallel. If that holds up at production scale, the same team could get drafts noticeably faster and the provider could serve more agents per GPU. This is a thought experiment, not a measured result.
How today’s LLMs generate text
Models such as GPT, Claude and Gemini are autoregressive: they predict the next token, a word or part of a word, append it, and repeat. This is reliable and scales extremely well, which is why it powers nearly every AI product today. Its weakness is speed: a 1,000-word answer needs well over a thousand sequential steps, and each step must wait for the one before. Much of the cost of running AI products comes from that sequential decoding.
Researchers have tried alternatives. Diffusion language models start from noise and refine the whole text over many steps in parallel. They avoid strict left-to-right writing but typically need many passes to reach good quality, which is the gap NCM targets.
How Next-Concept Modeling works
NCM borrows an idea from image generation, where models paint a rough picture and then add detail. It works in two stages:
- A multiscale tokenizer learns to represent text as a hierarchy of discrete codes: a few codes for the broad structure at the top level, more codes for finer detail at each level below.
- A transformer then generates the text level by level, coarse to fine. All codes within a level are predicted at once, in parallel, and each level is conditioned on the levels above it.
- A decoder turns the finished hierarchy back into words.
Because each level is produced in one parallel step, the number of sequential steps depends on the number of levels, not the length of the text. That is the source of the speed-up.
NCM vs LLM at a glance
Autoregressive LLMsNext-Concept ModelingHow text is producedOne token at a time, left to rightCoarse to fine, a whole level in parallelSequential stepsGrows with the length of the outputFixed by the number of levelsMaturityProduction at massive scaleResearch release, October 2026Largest modelsHundreds of billions of parameters725 million parameters in the paperCompared against—Mainly diffusion language models; an autoregressive baseline only on SudokuHow to use itHosted APIs and open modelsResearch code and task-specific checkpoints
Where coarse-to-fine generation could fit first
The published experiments cover open-ended text generation, summarisation and puzzle-style reasoning. Based on how the method works, the most likely early uses are tasks with a clear overall structure and a predictable length:
Use caseWhy coarse-to-fine might suit itBusiness exampleSummariesThe structure is predictable and the length is boundedDaily digests of tickets, calls or incident reportsStructured documentsSections and fields can be planned before wordingDraft reports, product descriptions, form responsesConstraint puzzles and planningThe whole answer must satisfy rules at onceScheduling, configuration and rota problems (the paper’s Sudoku results point this way)Bulk generationThroughput matters more than live streamingOvernight personalisation for thousands of records
These are directions to test, not proven applications. Until there are general-purpose NCM models, each of them would need a task-specific model trained in-house.
Where today’s LLMs keep winning
- Open-ended conversation and very long outputs, where the length and direction of the answer are not known in advance.
- Streaming responses: chat interfaces show words as they are generated, which hides latency. A level-by-level method shows nothing until a level completes.
- Tool use and agents, where the model calls tools mid-answer and reacts to the results, a pattern explained in our guide to AI agents vs AI workflows.
- Breadth and maturity: frontier LLMs are trained on vast data, support many languages and tasks in one model, and come with hosting, safety tooling and support.
- Ecosystem: inference servers, fine-tuning tools, evaluation frameworks and security controls are all built around autoregressive models.
A worked example: how NCM would build a summary
To make the idea concrete, imagine summarising a two-page incident report. This walk-through is a simplified illustration of the coarse-to-fine principle, not a trace of the real model, which works on learned codes rather than readable outlines.
An autoregressive LLM would write the same summary strictly left to right, deciding the first word before it has “planned” the last sentence. That works remarkably well in practice, but it is inherently sequential. NCM’s bet is that settling structure first and detail later can be both faster and, for some tasks, more coherent.
What the results actually show
The authors report that, on a single NVIDIA H100, NCM generated text 21 times faster than the strongest diffusion baseline on LM1B, 10 times faster on OpenWebText and 31 times faster on XSum summarisation while matching its ROUGE-L score. It also needed about 4 to 5 times fewer training tokens, even counting the tokenizer’s training. On Sudoku puzzles it outperformed both autoregressive and diffusion baselines across difficulty levels.
Three caveats matter for anyone reading the headlines:
- The speed comparisons are against diffusion language models, not against GPT-, Claude- or Gemini-class autoregressive LLMs.
- The models are small: the scaling experiments use 97M, 317M and 725M parameters, while frontier LLMs are hundreds of times larger. Whether the advantage holds at that scale is an open question the authors themselves list.
- The results are new and author-reported; independent reproductions and general-purpose benchmarks have not yet appeared.
Not the “end of LLMs”
Social posts calling NCM the end of LLMs go well beyond the paper. It is a promising new way to generate text, tested at small scale against a different family of models. Treat claims of LLM replacement as speculation until NCM is shown working at frontier scale on general tasks.
How to judge AI research headlines like this one
NCM is a useful case study in reading AI news. A new paper appears, social media calls it a revolution, and leaders are asked whether to change course. Six questions separate a promising result from a reason to act:
QuestionWhat to look forNCM todayFaster than what?The baseline it was compared withMainly diffusion language modelsAt what scale?Model size compared with what you useUp to 725M parametersOn which tasks?Benchmarks that resemble your workloadGeneration, summarisation, SudokuWho measured it?Independent reproduction, not only the authorsAuthor-reported so farAt what quality?Accuracy or quality held steady while speed improvedCompetitive with or better than the diffusion baselinesCan I use it?Hosted APIs, general models, supportResearch code and task-specific checkpoints
Apply the same checklist to the next “end of transformers” or “10× cheaper AI” headline. Most results that matter take one to three years to reach products, and the ones that do usually arrive through your existing providers.
Three scenarios for the next 12–36 months
Nobody knows yet how NCM will develop. Planning around scenarios, rather than predictions, keeps the roadmap robust:
ScenarioWhat happensWhat it means for youIt stays a research ideaResults do not hold at large scaleNo change: autoregressive LLMs keep improving and getting cheaperIt becomes an ingredientProviders adopt coarse-to-fine or parallel decoding inside their modelsFaster, cheaper APIs with no work on your side, if you can switch models easilyIt becomes a new model familyGeneral-purpose NCM models match LLMs on many tasksOption to move high-volume, structured workloads to a cheaper, faster generator
In all three scenarios the winning move is the same: build AI features so that the model behind them can be replaced without rewriting the product. That is cheap to do now and expensive to retrofit later.
What it means for business leaders
For now, nothing in your AI roadmap needs to change. Every production AI system you can buy or build today runs on autoregressive models, and the practical decisions remain the same: choosing between AI agents and AI workflows, governing data and measuring value. What NCM signals is that generation cost and speed may fall sharply in the next few years if parallel approaches mature.
- Avoid long lock-in to a single model architecture: keep model calls behind your own interface so a faster, cheaper generator can be swapped in.
- Measure latency and cost per task now, so you can recognise when a new approach genuinely improves them.
- Ask vendors how they would adopt faster decoding methods, not whether they use NCM.
- Let your AI team experiment with the open code on narrow, structured tasks if speed matters, but keep it out of production.
The architectural habits that protect you here are the same ones we recommend for any LLM system in our 10 architectural rules for effective LLM use.
For your AI team, the practical priorities do not change: a clean abstraction over model providers, an evaluation set built from your own tasks, and cost and latency tracked per feature. With those in place, testing a new kind of generator becomes a week of work rather than a rebuild, and you can make the decision on your own data instead of on headlines. Teams without them tend to stay locked to whichever model they started with, even when better and cheaper options appear.
Questions to ask your AI vendors
- What latency and cost per request do we get today for our main use cases, and how do you report it?
- How quickly can we switch to a newer or cheaper model if one becomes available?
- Are you evaluating faster decoding methods such as parallel or coarse-to-fine generation, and how would that change pricing?
- If quality changes when you change models, how will we know before our customers do?
What to watch next
Signals that NCM is becoming practical
- Results at billions of parameters, not only hundreds of millions.
- Head-to-head comparisons with autoregressive LLMs on general tasks, not only diffusion models.
- Independent reproductions of the speed and quality claims.
- A general-purpose checkpoint instead of one model per task, and support in inference servers or hosted APIs.
- Major model providers adopting coarse-to-fine or parallel decoding in their own products.
AI strategy
Want an AI architecture that can adopt better models as they arrive?
We review your AI systems for model lock-in, latency and cost, and design the abstraction and measurement that let you switch to faster, cheaper models without a rebuild.
Sources
- Next-Concept Modeling: Fast, Parallel Coarse-to-Fine Text Generation (Hazy Research blog, 12 October 2026)
- Paper on arXiv (2610.14162) (Stanford University and Together AI)
- NCM research code (GitHub)
Originally published at phpscientist.com.
Top comments (0)