A team from Shanghai AI Lab and Shanghai Jiao Tong University has released NCP-ArchPreview, an open 8.9-billion-parameter language model that learns to predict the next "concept," a compressed summary of the next few words, alongside the next word itself. It reached the final training loss of the open OLMo-3-7B model after consuming only 51.3% of the same training data, and the team published the weights, training recipe and intermediate checkpoints.
Key facts
- 51.3% of the tokens: NCP-ArchPreview matched OLMo-3-7B's final first-stage training loss about 1.95 times faster, measured in training tokens.
- When: the technical report was posted to arXiv on 9 September 2026 and the weights went up on Hugging Face on 11 September.
- Who: the Intern-NCP Team at Shanghai AI Lab and the LUMIA Lab at Shanghai Jiao Tong University, led by Zhouhan Lin.
- Primary source: the technical report and the model collection.
Why predict something bigger than a word
Almost every large language model learns one way: read some text, guess the next token, get corrected, repeat trillions of times. Any grasp of ideas that span several words has to emerge on its own. The report says that understanding emerges "purely as an indirect byproduct" of next-token training.
The NCP team's bet is that you can supervise the bigger unit directly. Picture learning to write by predicting only the next letter, versus also predicting the gist of the next phrase. The second student gets a richer signal from every sentence it reads.
Researchers have tried versions of this before. Meta's Large Concept Models predicted whole-sentence embeddings produced by a separate encoder, and joint-embedding predictive architectures predict representations rather than raw data. The NCP report builds most directly on the team's own earlier ConceptLM, which tested the idea on models up to 1.5 billion parameters. This release is the first to keep concept prediction switched on "from the start of pre-training over several trillion tokens."
How it works
NCP-ArchPreview is built on the design of the Allen Institute for AI's OLMo-3-7B and stacks three parts. A token encoder reads the text. Every four tokens of its output are averaged into one concept vector, which is snapped to the nearest entry in a learned codebook, a technique known as vector quantization. A middle "concept module" predicts the next concept, and that prediction is fed into a token decoder that writes the actual words. Training mixes the usual next-word loss with a next-concept loss.
It is a relative of multi-token prediction, which also asks a model to look further ahead. The difference is that NCP looks ahead in a compressed space instead of guessing several exact words.
The team trained on Dolma 3, the same open dataset OLMo used: about 5.73 trillion tokens in the first stage and about 100 billion tokens of further "mid-training." "We formally release NCP-ArchPreview as a fully pretrained latent-space foundation model, including its model weights, inference scripts, training recipes, and intermediate evaluation checkpoints," the authors write.
What you can download
The main checkpoints are 17.89 GB downloads each, stored in BF16, under the Apache 2.0 licence. A small draft model for speculative decoding is a 2.21 GB download, and thirteen intermediate checkpoints are published for researchers studying how training unfolds. Neither the paper nor the model card states a minimum GPU; the card's example runs on "one CUDA GPU with sufficient memory for the BF16 weights, activations, and cache." Computed from the shipped files, the weights alone occupy about 17.9 GB in BF16, so that is a floor; activations and cache come on top. Evaluation code is on GitHub.
Why it matters
Training data is the scarce resource in AI now. If a change to what a model is asked to predict lets it squeeze the same learning out of half the text, that is worth as much as a larger dataset, and it is far cheaper. The authors also fit a small-scale scaling law suggesting "1.74x computational efficiency compared to OLMo-3 with compute-optimal training," and they show that retraining only a tiny set of about 17 million concept parameters improves math skills without hurting general ability, beating the popular fine-tuning shortcut LoRA on both counts.
The honest caveats
The headline comparison is of training loss against the released OLMo-3-7B run, and NCP-ArchPreview is the larger model, 8.9 billion parameters against 7 billion. In a smaller test where the ordinary baseline was made the same size, NCP only "approaches" it. Lower loss also did not translate cleanly into better answers. NCP was ahead on average after the first stage, with its biggest gain on grade-school math word problems, but after mid-training its average lead shrank to under a point, and its coding scores fell. The team says three of its own variants got steadily lower loss and steadily worse test results. Long-context training "is not included in the current architecture preview," and the report concedes that turning its training advantage "more consistently into downstream capability" is still future work.
Originally published on Ground Truth, where every claim is checked against the primary source.
Top comments (0)