What If a Transformer Never Had to Forget? Meet the Recurrent Looped Transformer (RLT)

Neha Maurya — Tue, 15 Sep 2026 18:34:13 +0000

You ask a language model a one-line question — it processes it through 48 layers. You paste a 10,000-word document — it still processes it through 48 layers. Same depth. Same ceiling. Every single time.

That's the structural limit of every decoder-only Transformer in production today.

A technical report published September 12, 2026 by Princeton researcher Yifan Zhang — Recurrent Looped Transformer (RLT) — proposes closing that loop. In most decoder-only LLMs, nothing computed at the last layer of token t feeds the first layer of token t+1; positions communicate only through attention over cached keys and values. RLT changes this: the decoder's final hidden state and its layerwise sliding-window attention (SWA) cache are carried into the next token, across both prompt and response, with no reset at the boundary.

⚠️ Important: RLT is an architectural specification, not a trained system. The report explicitly states that no measured efficiency, reasoning quality, or scaling results are reported.

How RLT Is Built

Recurrent Looped Transformer (RLT) pairs a causal encoder with a recurrent decoder.

Causal Encoder

The encoder processes tokens in parallel under a causal mask and produces representations for each position. These representations are projected into key-value memory that the decoder can query later. Memory groups can be:

Shared across all decoder layers (efficient), or
Layer-specific (more flexible)

This memory is encoder-derived — it depends only on input tokens, not on decoder states.

Recurrent Decoder

The decoder holds the recurrence. Its complete state has two components:

Final decoder output (the recurrent hidden state)
Sliding-window attention (SWA) cache — retained keys and values at every decoder layer

For each token, a gated merge combines the current encoder representation with the previous decoder output. The gate controls how much previous information is carried forward. Then each decoder block runs:

Causal SWA over decoder activations (local attention within a bounded window)
Cross-attention to encoder memory (global context)
Feed-forward network (standard FFN)

The next-token distribution is read from the final decoder output. Initialization happens once before the beginning-of-sequence token with a learned start state and an empty cache.

💡 Tip: RLT separates two memory stores because they serve different roles:

Encoder memory is immutable for a fixed prefix — think of it as a "textbook index" of what was said

Decoder SWA cache changes every step and keeps only recent activations — think of it as the "last 2 pages of notes"

Neither can replace the other.

Reference Tied Configuration

The reference configuration uses:

48 encoder layers
48 decoder layers
Compatible attention and FFN weights shared between them

Each token executes 96 logical blocks (48 encoder + 48 decoder). Zhang calls this parameter reuse, not activation copying. Two logical passes do not imply equal per-block compute — the decoder block adds cross-attention that the encoder doesn't have.

The 3 Design Principles

1. Latent Reasoning with Unbounded Temporal Depth

After processing t tokens, the state path traverses t × 48 decoder blocks in the reference configuration.

Tokens Processed	Normal Transformer Depth	RLT State Path Depth
1	48	48
10	48	480
100	48	4,800
1,000	48	48,000

Per-token work stays fixed at 96 blocks, while the path's structural depth grows with the sequence. "Unbounded" means no fixed architectural upper bound on the recurrent computation path — it does not mean infinite computation per input.

The report warns that gates and contraction may suppress long paths. Structural depth is not a reasoning guarantee — it's a structural possibility that requires experimental validation.

💡 Tip — Snowball Analogy: The same amount of snow is added during every rotation (fixed compute per token), but the snowball grows because information accumulates. After 100 rotations, you have a massive snowball — without needing a bigger hill.

2. Model–Hardware Co-Design

Encoder features use token-parallel kernels — they can process known tokens in parallel. Decoder transitions stay sequential within a sequence, but independent sequences can be batched together:

Sequence A: Token 1 → Token 2 → Token 3 → Token 4
Sequence B: Token 1 → Token 2 → Token 3 → Token 4
Sequence C: Token 1 → Token 2 → Token 3 → Token 4
↓
BATCHED ON GPU

The report states plainly:

No exact parallel scan is assumed for the nonlinear decoder
No reduced-prefill speedup is claimed
A standard parallel SWA decoder pass is not equivalent to the recurrence
Batching, kernel fusion, and checkpointing are listed as implementation targets, not completed kernels

💡 Tip for Practitioners: RLT's decoder is inherently sequential per sequence. Throughput optimization comes from batching independent sequences, not from parallelizing within one sequence. This is closer to RNN-style serving than standard transformer serving.

3. Model–RL Algorithm Co-Design

Pretraining, supervised fine-tuning (SFT), sampling, and reinforcement learning (RL) replay share one state transition.

During sampling:

Record generated tokens
Record behavior log-probabilities under the actual sampling distribution (including temperature and truncation)
Record sampling configuration

During training:

Rebuild encoder memory from scratch under current parameters
Rebuild the recurrent output through all prompt tokens under current parameters
Rebuild every SWA cache under current parameters
Old rollout states are never reused

The trainer computes the current policy probability and forms an importance ratio comparing it to the behavior policy probability. Proposition 3.1 formalizes the payoff: moving the prompt-response split leaves the conditional distribution unchanged for a fixed token history. This is mathematical equivalence — different kernels and numerical precision can still cause numerical discrepancies.

💡 Tip for ML Engineers: If your architecture has any recurrent component, old states become stale after parameter updates. Reconstruct states from the sequence start under current parameters. Record behavior log-probabilities including all sampling transformations — metadata alone cannot restore missing support from truncated sampling.

Training and Serving

Pretraining

Pretraining is full-sequence next-token prediction with full backpropagation through time (BPTT). Independent documents reset:

Recurrent outputs
Decoder SWA caches
Encoder caches
Positions

Supervised Fine-Tuning (SFT)

Loss is masked to assistant targets only, but state updates are never masked. Gradients from assistant losses backpropagate through user and tool tokens. The state is not reset at an assistant boundary.

💡 Tip — The SFT Insight: The model learns how to "think about" the user's message, not just how to respond to it. Loss masking removes the loss term, not the computation or the gradient path.

Truncated BPTT

Appendix B shows why partial detaching is risky. The state-to-state Jacobian has cross terms through the decoder key-value cache. The recurrent state has two components:

Final decoder output
Decoder SWA cache

Detaching only the final decoder output leaves gradient paths through the SWA cache. Therefore, any truncated-BPTT scheme must explicitly identify every detached tensor.

💡 Tip: Detaching the recurrent hidden state alone is NOT sufficient to cut temporal gradient dependencies. The decoder SWA cache also carries gradient information. A complete detach requires stop-gradient on both components of the recurrent state.

Multi-Turn Serving

An exact prefix snapshot includes:

Encoder cache
Encoder memory
Complete decoder state
Position metadata
Window convention
Model version

A fixed-weight snapshot can be reused because the state is independent of the serving split. Weight updates invalidate old states. Editing a prefix forces recomputation from an earlier checkpoint. External tokens in multi-turn RL update the state but get no importance-ratio factors.

How It Relates to Prior Work

Encoder-derived memory follows:

YOCO (2024) — caches KV once for a cross-decoder
DeepSeek-V4.1-Flash (2026) — projects decoder global KV from final encoder states

RLT keeps the encoder-derived memory but drops prompt-wide decoder skipping — every prompt token gets its full decoder update.

Temporal feedback builds on:

Feedback Transformer (2020) — carries information from processed tokens into future computation
Recurrent Transformer (2026) — forms each layer's persistent KV from that layer's output

RLT instead feeds the previous final decoder output into the next decoder input and runs recurrence over the prompt too.

Depth-wise reuse connects to:

Universal Transformers (2018) — share computation across depth
Recurrent-depth latent reasoning (2025) — iterates a recurrent block to scale computation

The RL replay argument extends Zhang's prefill-decode kernel mismatch note (2026).

Comparison: RLT vs Prior Architectures

Feature	Standard Transformer	YOCO (2024)	Feedback Transformer	Recurrent Transformer	RLT (2026)
Per-token depth	Fixed	Fixed	Fixed	Fixed	Fixed ✅
Depth grows with sequence?	❌	❌	❌	❌	✅ Yes (t × decoder depth)
Memory type	KV cache (one type)	Encoder KV reuse	Layer feedback	Layerwise KV	Encoder memory (global) + SWA cache (local) + Recurrent state
Prompt-response boundary	Different phases	Different	Different	Different	Same transition, no reset
RL training consistency	Mismatch (stale states)	Not addressed	Not addressed	Not addressed	Exact replay under current params
Hardware co-design	Generic	KV reuse	Not addressed	Tiling schedule	Explicit parallel/sequential split
Prompt-wide decoder skip?	No	Yes (early exit)	No	No	No — full recurrence
Empirical validation	✅ Extensive	✅ Some	✅ Some	✅ Some	❌ Not yet

Key Takeaways

RLT carries the full decoder state — final output plus layerwise SWA cache — across every prompt and response token with no boundary reset
Reference configuration: 48 tied encoder + 48 tied decoder layers = 96 logical blocks per token. State path has structural depth of 48t blocks after t tokens
Hardware opportunities: Encoder parallelism and batching across sequences. No parallel scan or reduced-prefill speedup is claimed
RL replay rebuilds all states under current parameters while keeping recorded behavior log-probabilities as ratio denominators
No measured results: Reasoning quality, efficiency, and RL scaling remain open validation targets

Practical Tips

Tip #1: Chain Your Prompts — Don't Dump Everything at Once

More rounds mean more reasoning passes through the model's layer stack on accumulated context. Instead of one massive prompt, build context iteratively through back-and-forth conversation.

Tip #2: Recognize "State Reset" Failures in Production

When AI contradicts earlier context or forgets constraints, don't just rephrase. Instead:

Restate key constraints in your current message, or
Start a fresh conversation and rebuild context deliberately

Tip #3: Audit Your RL State Handling

If building RL training pipelines, check:

Are training states computed under current parameters?
Are you reusing stale rollout states?
Are behavior log-probabilities recorded correctly with all sampling transforms?

Reconstruct states from the sequence start under current parameters — never reuse old states after parameter updates.

References

RLT — Zhang, Y. (2026) — GitHub
YOCO — Sun et al. (2024) — arXiv:2405.05254
Feedback Transformer — Fan et al. (2020) — arXiv:2002.09402
Recurrent Transformer — Oncescu et al. (2026) — arXiv:2604.21215
Universal Transformers — Dehghani et al. (2018) — arXiv:1807.03819
Recurrent Depth — Geiping et al. (2025) — arXiv:2502.05171
Prefill-Decode Mismatch — Zhang et al. (2026) — GitHub PDF
DeepSeek-V4.1-Flash — DeepSeek-AI (2026) — HuggingFace

Check out the Technical Report, GitHub repository, and Project Page. All credit goes to the researcher of this project.

DEV Community: Neha Maurya