DEV Community: Nishaanth Nattudurai

Next-Concept Modeling: I Ran Stanford's New Text Generator Against a Classic LLM. Here Is What Happened.

Nishaanth Nattudurai — Tue, 10 Nov 2026 04:36:50 +0000

Every large language model you have used, from GPT to Llama to Claude, writes text the same way: one token at a time, left to right. To produce 128 tokens it runs the network 128 times in a row, and each step has to wait for the previous one. That sequential loop is the single biggest reason text generation is slow and expensive.

In October 2026, Stanford's Hazy Research group published Next-Concept Modeling (NCM), a model that writes text a completely different way: it sketches the whole passage first, then refines it level by level, predicting every position of a level in parallel. The paper reports generation that is 10x to 31x faster than diffusion language models at comparable quality.

Social media quickly turned that into "NCM kills LLMs". That is not what the paper measured. So I did what any curious engineer would do: I ran the official code and checkpoints myself, rebuilt the paper's evaluation, and put NCM head to head against a traditional autoregressive model trained on the same data, on an ordinary laptop.

This post covers:

Why a new way of generating text is needed
How NCM works, in plain language
What the paper claims
The datasets and models I used
My experiments and results, with graphs
The future of NCM, and whether it will replace LLMs

Everything is open source in my repository: github.com/Nishaanth419/ncm-research-playground.

Why do we need something other than next-token prediction?

An autoregressive (AR) language model predicts the next token given all previous tokens. It is simple, it scales beautifully, and it powers almost every chatbot today. But it has a structural cost:

Latency grows linearly with length. 1,000 tokens means 1,000 sequential forward passes. A GPU can do enormous amounts of work in parallel, but it cannot skip ahead.
It commits early. The model picks word 1 before it has "decided" what word 50 will be. Any global plan has to live implicitly inside its hidden state.
Throughput is bounded by memory bandwidth, because each tiny step reloads the weights.

Researchers have tried to break this loop before. Diffusion language models start from noise or masked tokens and refine all positions at once over many steps. They are parallel, but usually need many refinement steps to reach good quality, so in practice they are often not faster.

NCM takes a third route, borrowed from image generation.

How NCM works

The idea comes from VAR (Visual Autoregressive Modeling), which generates an image by predicting a tiny thumbnail first and then progressively larger, sharper versions. NCM applies the same coarse to fine recipe to text.

It has two parts.

Part 1: a tokenizer that turns text into levels of "concepts"

A multiscale residual VQ-VAE compresses a passage into a stack of levels of discrete codes.

Text is split into 128 ordinary BPE tokens, and an encoder turns them into a continuous latent (128 positions by 384 numbers).
That latent is approximated by 12 levels of codes. For TinyStories the level lengths are 1, 8, 16, 24, 32, 40, 52, 64, 76, 92, 108 and 128. Each level picks codes from its own codebook of 4,096 entries.
The levels are residual: each level encodes only what the coarser levels missed. Upsample every level to full length, add them together, and a decoder turns the sum back into tokens.

Level 0 is a single code that summarizes the entire passage. The last level has one code per token. These codes are the "concepts" in the name.

Part 2: a transformer that predicts the next level, not the next token

A next-concept transformer looks at everything generated so far (levels 0 to k, summed) and predicts all codes of level k+1 in a single forward pass. Repeat until the finest level, then decode the final latent into text.

For a 128-token story, that is 11 transformer passes instead of 128. (Level 0 for TinyStories is sampled from a frequency table stored in the checkpoint, so it costs no transformer pass.)

Watching it think

The nicest property of NCM is that every intermediate level can be decoded. These are real outputs of the official decode_multiscales function from one of my runs:

L0  (1 code)    .
L3  (24 codes)  One named the and.. and " is a it He felt wanted for before ...
L7  (64 codes)  day a boy found shiny in the. was happy showed his,. looked the ...
L11 (128 codes) One day, a little boy named Tim found a shiny stone in the park. He was very happy...

Early levels look like word salad, but they already carry the cast and topic of the story. The grammar and exact wording arrive in the last few levels. I measured this numerically by comparing the latent after each level with the final one:

Each level is one parallel step, but finer levels contain more codes, so they take longer:

The trade off you should notice immediately

Because the text only becomes readable at the last level, NCM cannot stream. A chatbot built on an AR model shows you its first word in milliseconds; NCM shows you nothing until the whole passage is finished. Keep that in mind for the results.

What the paper claims

Paper: Next-Concept Modeling: Fast, Parallel Coarse-to-Fine Text Generation, by Hermann Kumbong, Ishan S. Khare, Iddah Mlauzi, Andrew Woen, Charles Ding, E. Kelly Buchanan, Simran Arora, Azalia Mirhoseini and Christopher Ré (Stanford, 2026). arXiv:2610.14162

From the abstract, compared with the strongest diffusion language model baseline at matched scale:

LM1B: better generative perplexity, about 21x faster
OpenWebText: competitive quality, about 10x faster
XSum summarization: matches ROUGE-L, about 31x faster
Sudoku: outperforms both autoregressive and diffusion baselines at every difficulty level, with fewer function evaluations

Two details matter when you read those numbers. First, the baseline is a diffusion model, not a standard autoregressive LLM. Second, the speeds were measured on a datacenter GPU (an H100). Neither of those matches how most people will first try NCM, which is exactly why I ran my own experiments.

The authors released everything: code under the MIT license, checkpoints and datasets on Hugging Face, and a blog post.

My experiment setup

Question: How does NCM compare with a traditional autoregressive model of the same domain, on speed and on text quality, on ordinary hardware? And do the paper's numbers reproduce from the released code?

Hardware: Apple M5 Pro laptop, CPU only, fp32, 6 threads. (NCM does not currently run on Apple's GPU backend: PyTorch MPS lacks the adaptive pooling NCM's quantizer uses.)

Software: Python 3.12, PyTorch 2.15, Transformers 5.22. The official NCM repository is pinned at commit c70274273f489abc73832b2b98584a8483b6b203 and is never modified; my code imports it.

Models

Role	Model	Size
NCM, stories	`hazyresearch/ncm-checkpoints`, `tinystories/`	125.0M (generator 88.6M + VQ-VAE 36.4M)
NCM, summarization	same repository, `xsum/`
NCM, translation	same repository, `wmt14/`
Autoregressive baseline	`roneneldan/TinyStories-33M`	68.5M
Quality judge	`gpt2-large`	774M

Both story models were trained on the same dataset, which is what makes them a reasonable pair.

Datasets

Dataset	What it is	How I used it	Link
TinyStories (Eldan and Li, 2023)	Short synthetic children's stories written with a small vocabulary	Training data of both story models. Generation is unconditional, so the models' own samples are scored.	HF dataset
XSum (Narayan et al., 2018)	BBC news articles with one sentence summaries; 11,334 test pairs	NCM summarization, test items 0 to 3	NCM copy
WMT14 German to English	News sentence pairs; 3,003 test pairs	NCM translation, test items 0 to 3	NCM copy

Metrics

Generative perplexity (gen-PPL): how surprised GPT-2-large is by the generated text. Lower means more fluent. This is the metric the NCM paper uses.
Unigram entropy: how varied the words are. It guards against a model that games perplexity by repeating itself.
Latency: median and p95 wall clock time over repeated runs, after warm up.
Time to first output: when the reader first sees anything.

Results

Experiment 1: Does the paper's result reproduce?

I ran the official evaluation scripts unmodified (eval_shard.py and eval_merge.py): 5 seeds by 256 unconditional TinyStories samples, scored by GPT-2-large. The scripts assume CUDA, so a small launcher remaps device requests to CPU without touching the code.

	gen-PPL	entropy
My reproduction (5 seeds, 1,280 stories)	16.23 ± 0.48	3.967
Paper, Table 8	15.41	3.98

Verdict: it reproduces. Entropy matches, and gen-PPL is within two seed standard deviations. The small gap is plausibly CPU fp32 and newer library versions. Generation of all 1,280 stories took 4.5 minutes; scoring took 18.5 minutes.

Experiment 2: Is my app running the model faithfully?

I built a research playground (FastAPI backend, React frontend) that runs NCM through its own code path so it can record every level. To make sure I was not measuring my own bugs, I compared its outputs with the official script's outputs for the same inputs and seeds.

Verdict: identical for TinyStories (seed 42, 4 samples), XSum test items 0 to 3 and WMT14 test items 0 to 3. This is enforced by automated tests in the repository.

Experiment 3: Speed against a classic LLM

Benchmark protocol: for each model and batch size, 2 untimed warm ups, then 10 timed runs with seeds 0 to 9, checkpoint default sampling (temperature 1, no top k or top p).

Model	Batch	Median	p95	Per sample	Words/s
NCM	1	307.5 ms	399.3 ms	307.5 ms	340.6
NCM	4	733.7 ms	788.8 ms	183.4 ms	571.1
TinyStories-33M	1	436.9 ms	491.3 ms	436.9 ms	239.5
TinyStories-33M	4	1313.0 ms	1430.3 ms	328.2 ms	304.7

NCM finishes a whole story 1.42x faster at batch 1 and 1.79x faster at batch 4, and the gap widens with batch size, because each of its 11 passes does more parallel work:

But look at when the reader sees something:

The autoregressive model shows its first token in 3.9 ms. NCM shows the complete text at 307 ms and nothing before that. For a chat interface, perceived speed is mostly about that first token, and there the classic model wins by roughly 80x.

Experiment 4: Quality head to head

Same seed, same batch of 4, same sampling, both warmed up, scored by GPT-2-large.

The autoregressive baseline writes noticeably more fluent text (gen-PPL 9.74 vs 13.48). NCM is slightly more varied (entropy 4.056 vs 3.993). This is a single small run, so treat it as an indication rather than a final verdict, but it matches what I saw reading the stories side by side.

Summarization and translation

The XSum and WMT14 checkpoints run and accept real input. They work, but show visible repetition in some outputs. For example, the German sentence "Ich trinke gern Kaffee am Morgen." came back as "I like like coffee in the morning." I did not run the full 11,334 and 3,003 item evaluations on CPU, only the 4 item exact match checks.

Read these numbers carefully

The paper's 10x to 31x speedups are GPU numbers against diffusion models. Mine are CPU numbers against an autoregressive model. Different experiments, not a contradiction.
The two story models differ in size (125M vs 68.5M), tokenizer (8,192 entry BPE vs GPT-2 BPE) and training run.
NCM always fills its 128 token canvas; the baseline may stop early at end of text.
The speed results are medians of 10 runs; the quality comparison is one 4 sample run.

What I built along the way

The repository is a full research playground you can run locally with one command (./run.sh):

Learn NCM: a plain language guide with the paper, findings, videos and a glossary
How NCM works: an animated walkthrough replayed from a real recorded run
Playground: run TinyStories, XSum or WMT14, watching levels fill in live over WebSocket
Concept Explorer: heatmaps of the code hierarchy and the text decoded at each level
Model Comparison and Benchmarks: the experiments in this post, exportable as CSV
Experiment History: every run stored in SQLite, replayable

Every number in this post comes from files committed in the repository (results/), and the graphs are generated by blog/make_charts.py directly from those files.

The future of NCM

NCM is a few weeks old, and the released checkpoints are small research models. Here is where I think it goes next.

1. Scaling. The big open question is whether NCM keeps its quality as models grow to billions of parameters and passages grow to thousands of tokens. The paper's OpenWebText results (1,024 tokens) are encouraging, but nobody has shown an NCM model at the scale of a modern assistant.

2. Prompting and instructions. The TinyStories model is unconditional; it cannot take a prompt. XSum and WMT14 show that conditioning on an input works. Instruction following, multi turn chat and tool use are still unexplored.

3. Streaming, or something better. NCM cannot show partial words, but it could show a partial plan. A product might display an outline from the coarse levels and sharpen it, the way progressive JPEGs load. Or levels could be generated in chunks so long documents stream paragraph by paragraph.

4. Planning and reasoning. The Sudoku result is the most intriguing line in the paper. A model that decides the global structure before the details is a natural fit for tasks where the end constrains the beginning: puzzles, code with matching brackets, structured data, poetry with a fixed form.

5. Hybrids. The likeliest near term path is not NCM alone but NCM inside a larger system: a fast coarse to fine drafter whose output an autoregressive model verifies or polishes, similar to speculative decoding.

6. Hardware support. Today NCM runs on CUDA and CPU but not Apple GPUs. Broader kernel support would make it practical on laptops and phones, where its low step count matters most.

So, will NCM replace LLMs?

Not as things stand, and probably not in the "one replaces the other" sense. A few reasons:

Quality still favors autoregressive models in my test, and the paper's quality comparisons are against diffusion models, not frontier AR LLMs.
Streaming matters for chat. Users perceive a model that starts talking in 4 ms as faster than one that delivers a full answer in 300 ms.
The AR ecosystem is enormous. Instruction tuning, RLHF, KV caching, speculative decoding and serving stacks are all built around next-token prediction.

It is also worth being precise about terminology: NCM is a language model. The real question is whether coarse to fine generation will replace left to right generation. My honest prediction:

NCM style models will win where throughput matters more than first token latency: batch summarization, translation, data generation, offline document processing and structured outputs.
They will complement rather than replace autoregressive models in interactive assistants, most likely through hybrids.
If scaling works out, the parallel step count is a genuine structural advantage, and the "next token" assumption will no longer be the only way to build a large language model. That alone makes NCM one of the most interesting papers of the year.

Try it yourself

git clone https://github.com/Nishaanth419/ncm-research-playground.git
cd ncm-research-playground/ncm-research-playground

Then follow the step by step setup in the README. It takes about 20 minutes, mostly downloads, and needs no GPU.

References

Kumbong, H., Khare, I. S., Mlauzi, I., Woen, A., Ding, C., Buchanan, E. K., Arora, S., Mirhoseini, A., and Ré, C. (2026). Next-Concept Modeling: Fast, Parallel Coarse-to-Fine Text Generation. arXiv:2610.14162. https://arxiv.org/abs/2610.14162
HazyResearch. NCM official code (MIT license). https://github.com/HazyResearch/ncm
HazyResearch. NCM checkpoints and datasets on Hugging Face. https://huggingface.co/collections/hazyresearch/ncm
Hazy Research blog. Next-Concept Modeling. https://hazyresearch.stanford.edu/blog/2026-10-12-ncm
Scaling Intelligence Lab blog. NCM. https://scalingintelligence.stanford.edu/blogs/ncm/
Tian, K., Jiang, Y., Yuan, Z., Peng, B., and Wang, L. (2024). Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction. NeurIPS 2024. https://arxiv.org/abs/2404.02905
van den Oord, A., Vinyals, O., and Kavukcuoglu, K. (2017). Neural Discrete Representation Learning (VQ-VAE). https://arxiv.org/abs/1711.00937
Eldan, R., and Li, Y. (2023). TinyStories: How Small Can Language Models Be and Still Speak Coherent English? https://arxiv.org/abs/2305.07759. Dataset: https://huggingface.co/datasets/roneneldan/TinyStories
Narayan, S., Cohen, S. B., and Lapata, M. (2018). Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization (XSum). https://arxiv.org/abs/1808.08745
Bojar, O., et al. (2014). Findings of the 2014 Workshop on Statistical Machine Translation (WMT14). https://aclanthology.org/W14-3302/
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. (2019). Language Models are Unsupervised Multitask Learners (GPT-2). https://huggingface.co/gpt2-large
TinyStories-33M baseline model. https://huggingface.co/roneneldan/TinyStories-33M
My repository: NCM Research Playground. https://github.com/Nishaanth419/ncm-research-playground

If you run the playground on a GPU or try NCM on a new task, I would love to hear what you find. Drop a comment below.

The Labor Calculus of AGI: Structural Displacement, Sociotechnical Risk, and the Practitioner's Responsibility

Nishaanth Nattudurai — Mon, 07 Sep 2026 07:49:02 +0000

Abstract: The transition from narrow AI to AGI represents a potential discontinuity in how cognitive labor is organized and compensated. Unlike prior automation waves, AGI threatens to substitute human judgment across the full professional spectrum. This essay examines displacement mechanisms, distributional risk, and the normative responsibilities of AI/ML practitioners.

1. Framing the Problem: What Makes AGI Different

AGI is conventionally understood as a system capable of performing any intellectual task a human can, including reasoning under uncertainty, cross-domain transfer, and autonomous goal-directed behavior. This distinguishes it categorically from today's LLMs and multimodal systems, which remain brittle outside their training distribution and lack robust causal world models.

With narrow AI, displacement has historically been task-level. The compensating mechanisms (task recomposition, productivity gains, new roles) have absorbed much displaced labor, albeit unevenly. AGI changes this calculus qualitatively. If a system can substitute for human cognition across arbitrary domains, no task-level safe harbor exists.

How automation scope has shifted across waves

The key insight: every prior wave left creative and judgment work largely intact. AGI does not.

Automation scope by task category (estimated %)
================================================

Industrial wave
  Manual/physical  ████████████████████████████████████ 90%
  Cognitive        ██ 5%
  Creative/judgment██ 5%

Digital/software wave
  Manual/physical  ████████████████ 40%
  Cognitive        ████████████████████ 50%
  Creative/judgment████ 10%

AGI (projected)
  Manual/physical  ██████████ 25%
  Cognitive        ██████████████████████████ 65%
  Creative/judgment████████████████████ 55%  ← NEW FRONTIER

2. Displacement Mechanisms: Beyond the Task-Level Model

Economic models of automation typically decompose jobs into task bundles and assess automation susceptibility task-by-task (Acemoglu & Restrepo, 2018; Frey & Osborne, 2013). This framework is likely insufficient for AGI-level systems.

2.1 Cognitive labor as a tradeable good

Once general cognitive capability is abundant and cheap, it behaves like a commodity. Fields insulated through credential requirements, tacit knowledge, or regulatory gatekeeping (medicine, law, financial analysis, software engineering) face structural wage compression even before full substitution occurs.

2.2 The productivity-employment decoupling

Historically, productivity gains drove employment growth through demand expansion. The speed and breadth of AGI-driven gains may outpace demand expansion, particularly in economies with already-saturated consumer markets.

Productivity vs. Employment Index, 2025–2041 (Base = 100)
==========================================================

Index
 300 |                                              ● Productivity
     |                                         ●
 250 |                                    ●
     |                               ●
 200 |                          ●
     |                     ●
 150 |               ●
     |          ●
 100 |●----●----●----●----●----●----●----● Employment (cognitive)
     |                                   ↘ 78 by 2041
  70 +----+----+----+----+----+----+----+----
    2025 2027 2029 2031 2033 2035 2037 2039 2041

If an AGI-augmented firm of 10 can produce what previously required 100, aggregate demand does not automatically compensate, especially if gains accrue disproportionately to capital owners.

2.3 Complementarity inversion

Current AI systems exhibit strong human complementarity in judgment-intensive tasks. AGI reduces or eliminates these niches. Practitioners should treat "this requires human judgment" as a temporally bounded claim, not a stable property.

3. Distributional Asymmetries: Who Bears the Transition Cost

Displacement from AGI is unlikely to be uniformly distributed.

~62% of knowledge-worker jobs have high AGI substitutability (Goldman Sachs, 2023)

10 to 25 years — mid-career cohort faces highest transition cost and lowest retraining ROI

~5x income concentration in frontier-AI regions vs. labor-displacement regions

AGI substitutability by occupation

Exposure index (0–100)    Risk
======================================
Software engineering  78  🔴 HIGH
Legal analysis        74  🔴 HIGH
Financial analysis    71  🔴 HIGH
Medical diagnosis     63  🔴 HIGH
Mgmt. consulting      60  🔴 HIGH
──────────────────────────────────────
Skilled trades        28  🟡 MEDIUM
──────────────────────────────────────
Elderly care          18  🟢 LOWER
Crisis counseling     14  🟢 LOWER
======================================

Unlike prior automation waves that disproportionately affected low-skill workers, AGI may exhibit an inverted profile, displacing mid-to-high-skill cognitive workers while leaving physically demanding, socially embedded, or highly context-specific roles comparatively intact.

4. Sociotechnical Risk: The Alignment-Labor Intersection

A distinct class of sociotechnical risk arises at the intersection of alignment and labor economics that receives comparatively less attention from the research community.

Risk	Severity	Description
Replacement without consent	🔴 High	Workers bear transition costs with no formal recourse in deployment decisions
Skill investment mismatch	🔴 High	Education pipelines invest in skills AGI may invalidate before the degree matures
Model governance gap	🟡 Medium	Deployment decisions treated as technical choices, not labor policy
Geographic rent capture	🟡 Medium	AGI gains concentrate in frontier jurisdictions; displacement is global
Demand feedback collapse	🟡 Medium	Rapid productivity gains limit post-industrial reabsorption mechanisms
Late-career displacement	🟢 Lower	Shorter horizons reduce transition cost but also recovery options

The "replacement without consent" problem

Current deployment norms permit organizations to substitute human labor with AI systems without affording affected workers meaningful participation. The AI/ML practitioner who ships a system eliminating 5,000 analyst roles bears no formal accountability for the resulting economic harm — a structural misalignment of incentive and consequence.

Model governance as labor policy

Decisions about what capabilities to deploy, at what pace, and under what conditions are, substantively, labor policy decisions. A frontier lab's choice to release a coding agent performing junior-level engineering work is functionally equivalent to a policy decision about engineering employment. This framing is increasingly inadequate.

5. The Practitioner's Responsibility

AI/ML practitioners occupy a distinctive position: technical literacy to understand capability trajectories, institutional proximity to influence deployment decisions, and professional credibility to shape public discourse. This position carries normative weight.

1. Epistemic honesty about timelines
Resist both catastrophism and dismissiveness. The honest position is significant uncertainty combined with acknowledgment that this uncertainty warrants precautionary institutional investment.

2. Participation in governance structures
Technical expertise is underrepresented in the policy processes shaping AGI governance, labor law, and social safety net design. Active participation (not just consultation) is warranted.

3. Impact-aware system design

Prefer phased deployment that enables labor market adjustment
Default to human-in-the-loop architectures where feasible
Include explicit consideration of second-order labor effects in design reviews

4. Institutional advocacy
Practitioners are often the only individuals with both technical knowledge and organizational standing to raise displacement concerns credibly. This creates an obligation to do so, even when commercially inconvenient.

6. Open Research Questions

Research Priority Matrix
(Upper-left = high urgency, low coverage = most underserved)

Urgency
  High |  ● Displacement velocity     ● Governance
       |    modeling                    mechanism design
       |
  Med  |       ● Transition
       |         support systems
       |
  Low  |                    ● Human complementarity
       |                      residuals
       +-------------------------------
            Low         Med         High
                  Current research coverage

Research question	Urgency	Coverage	Impact
Displacement velocity modeling	Very high	Very low	High
Governance mechanism design	Very high	Medium	High
Transition support systems	High	Low	Very high
Human complementarity residuals	Medium	Medium	Medium

7. Conclusion

AGI, if realized at the capability levels the field is converging toward, represents a structural discontinuity in the economics of cognitive labor. The displacement mechanisms are qualitatively distinct from prior automation waves; the distributional risks are significant and unevenly borne; and the governance infrastructure is substantially inadequate to the challenge.

For AI/ML practitioners, this is not merely a background policy concern. The systems we build, the deployment decisions we participate in, and the discourse we shape are substantive inputs to how this transition unfolds. The technical community's engagement with the socioeconomic dimensions of AGI is not tangential to the work; it is increasingly central to it.

Why I wrote this

I work at the intersection of production AI systems and applied ML, and I've noticed a gap: the technical community spends enormous energy on capability and alignment research, but the labor economics of what we're building gets treated as someone else's problem. It isn't. Every system we ship is implicitly a labor policy decision, and I think practitioners are the people best positioned to say that out loud.

I'd love to hear from you in the comments:

Are there domains you think are genuinely AGI-resistant long-term? I listed a few but I hold those claims loosely.
How does your team think about second-order labor effects when scoping a product? Or does it not come up at all?
If you're mid-career in a high-exposure field, how are you thinking about the transition?

No right answers here. The point is to have the conversation before the decisions are already made.

References available on request. Views expressed are my own.