HiLS Attention: How Tencent Built a Sparse Attention Mechanism That Extrapolates to 4 Million Tokens

Long-context modeling has a persistent tension at its core: full attention is accurate but scales quadratically with sequence length, while sparse attention is fast but tends to degrade badly when you push it beyond its training window. A new paper from Tencent's Hunyuan team, HiLS Attention (arXiv:2607.02980), proposes a way to resolve that tension — not by approximating attention more cleverly, but by making chunk selection itself a learnable part of the model.

The result is a mechanism that matches full-attention quality at in-domain lengths, extrapolates to contexts more than 64× longer than training, and runs 13–15× faster at 512K tokens. The 7B model trained with HiLS is available on Hugging Face, and the code is open-sourced.

Why Chunk-Wise Sparse Attention Usually Fails at Extrapolation

The standard approach to long-context efficiency is to divide the sequence into chunks and only attend to a selected subset of them. The problem is how you select those chunks. Most methods use heuristics: mean-pooled keys, max-pooled keys, or fixed sliding windows. These work reasonably well within the training distribution, but they break down when the context grows much longer than what the model saw during training.

The deeper issue is that heuristic chunk selectors are not optimized by the training loss. The model learns to use whatever chunks get selected, but it never learns to select better chunks. This creates a mismatch: the selection mechanism is frozen while the rest of the model adapts.

HiLS fixes this by making chunk selection differentiable and integrating it directly into the language modeling objective.

The Core Idea: Landmark Tokens and Hierarchical Factorization

HiLS introduces a landmark token appended to each chunk. This token serves as a learnable summary of the chunk's content. Rather than using a simple pooling operation, the landmark key is derived using a first-order Taylor expansion of the full-attention-induced chunk mass — essentially, a principled approximation of how much attention the full model would assign to that chunk if it could see everything.

The attention mechanism is then factorized into two stages:

1. Inter-chunk attention: The query attends to landmark tokens across all chunks to compute a distribution over chunks (which chunks matter?).
2. Intra-chunk attention: Within the selected chunks, standard attention distributes mass among individual tokens (which tokens within those chunks matter?).

Because the landmark keys participate in the forward computation, gradients from the next-token prediction loss flow directly into the landmark representations. The model learns to summarize chunks in a way that makes them easy to retrieve — not as a separate pretraining objective, but as a natural consequence of learning to predict text.

Engineering Details That Make It Work in Practice

The paper includes several implementation choices that are worth understanding if you plan to use or adapt HiLS.

Query Calibration (Q-Cal): A lightweight low-rank module adjusts queries specifically for chunk-level mass estimation. The intuition is that the query representation useful for token-level attention is not necessarily the same as the one useful for chunk-level selection. Q-Cal adds a small learned offset: Δq = W_up · W_down · h, where h is the hidden state. This adds minimal parameters but meaningfully improves chunk retrieval accuracy.

Hierarchical Positional Encoding (HoPE): Standard RoPE embeddings encode absolute positions, which causes problems when you extrapolate far beyond training length. HiLS uses RoPE for dimensions within the pre-training context window and NoPE (no positional encoding) for dimensions beyond it. This hybrid approach lets the model handle familiar short-range patterns with position-aware attention while remaining agnostic to absolute position at longer ranges.

Hardware-Efficient Kernel: Adjacent queries tend to retrieve overlapping sets of chunks — the paper reports 92.8% overlap between neighboring queries. HiLS exploits this by batching M adjacent queries together, loading the union of their selected chunks once, and computing attention for all M queries in a single pass. This "one-load-multiple-compute" strategy significantly improves Tensor Core utilization and is the main reason for the large inference speedups.

What the Numbers Look Like

At the 345M parameter scale, HiLS maintains perplexity of 4.34 at 32K context and 5.95 at 512K context — while standard full-attention models fail entirely (perplexity above 100) beyond 32K tokens. On the RULER benchmark, which tests multi-hop retrieval in long contexts, HiLS scores 100/95/68 at 32K context, substantially outperforming full-attention baselines.

The larger-scale result is more striking. When the authors applied HiLS to OLMo3-7B via continued pre-training on 50B tokens, the resulting model scored 94.67 on RULER at 128K context. A comparable Landmark Attention tuning approach scored 0.67 on the same benchmark. The gap is not marginal — it reflects the fundamental difference between heuristic and learned chunk selection.

On inference latency, at 512K context HiLS is 13.5× faster in prefill and 15.7× faster per decode step compared to full attention on the same hardware. Both prefill and decode scale linearly with context length rather than quadratically.

Converting Existing Models to HiLS

One practical aspect of the paper is that HiLS is designed to be applied to existing full-attention models through continued pre-training rather than training from scratch. The authors demonstrate this with OLMo3-7B, using roughly 50B tokens of continued pre-training to convert the model. The converted model retains its short-context capabilities (MMLU scores are preserved) while gaining long-context extrapolation.

The official repository includes scripts for both training from scratch and continued pre-training, along with evaluation pipelines for RULER, LongBench, and perplexity benchmarks. Inference is supported through a dedicated SGLang fork that treats HiLS as a first-class attention backend with OpenAI-compatible API serving.

What This Means for Practitioners

The most immediate implication is for teams working on long-document retrieval, multi-turn conversation, or code understanding over large repositories. If you are currently using full attention with a 128K or 256K context window, HiLS offers a path to extend that to millions of tokens without the quadratic cost — and without the quality degradation that typically comes with sparse attention at extreme lengths.

The conversion approach is particularly useful: you do not need to train a new model from scratch. If you have a well-tuned base model, continued pre-training with HiLS is a viable path to long-context capability.

The broader point is methodological. HiLS demonstrates that the right place to intervene in sparse attention is not the attention kernel itself, but the chunk selection mechanism. Making selection end-to-end learnable — and doing so in a way that is hardware-efficient — closes most of the quality gap between sparse and full attention while preserving the efficiency benefits. That is a cleaner solution than the common alternative of adding more and more heuristics to the selection process.

The paper is available at arXiv:2607.02980, the model weights are at tencent/HiLS-Attention-7B, and the full codebase is at Tencent-Hunyuan/HiLS-Attention.