Baidu Unlimited OCR: How Reference Sliding Window Attention Rewrites Document AI

Baidu released Unlimited OCR under the MIT license (June 22, 2026), solving the fundamental KV cache memory wall that plagues LLM-based document parsing. The core architectural innovation—Reference Sliding Window Attention (R-SWA)—enables transcribing dozens of pages (text, tables, formulas, reading order) in a single forward pass with constant memory and latency.

Key Metrics:

• Parameters: 3B total, ~500M activated (MoE)
• Context Length: 32K tokens
• License: MIT (fully open source, commercial-friendly)
• Hacker News: #1 (~430 points)
• GitHub Trending: #1 (3,600+ stars in <48 hours)

The Problem: The KV Cache Memory Wall

Standard multi-head attention has a fatal scaling flaw: each generated token adds new K/V entries. On long documents, the cache grows linearly with output length, slowing generation and consuming VRAM. Developers resort to brittle page-chunking loops that break reading order, lose context between pages, and mess up table structures spanning page breaks.

The Innovation: Reference Sliding Window Attention (R-SWA)

R-SWA splits attention into two fixed-size paths mimicking human working memory:

Path	What It Sees	Size
Global Reference	All visual tokens + prompt	Fixed (256 per page)
Local Generation	Last N output tokens	Sliding window of 128

KV Cache Implementation: A FIFO queue with bounded capacity—each new token pushes out the oldest. Standard attention scales O(L + T), while R-SWA is O(L + n) where n = 128 tokens—constant memory regardless of document length.

DeepEncoder (16x Visual Compression)

Cascades SAM-ViT (window attention) with CLIP-ViT (global attention). A full 1024x1024px page becomes just 256 visual tokens, encoded once and referenced forever.

MoE-LLM Decoder with R-SWA

Built from DeepSeek OCR (trained decoder only, encoder frozen). All standard attention layers replaced with R-SWA. Training data: 90% PaddleOCR + 10% synthesized multi-page docs.

Benchmark Performance

Metric	Unlimited OCR	DeepSeek OCR	Improvement
OmniDocBench v1.5	93.23%	87.01%	+6.22 pp
Text Edit Distance	0.038	0.073	-47.9%
Formula CDM	92.61	83.37	+11.1%
Throughput (TPS)	5,580	4,951	+12.7%

At 6,000 tokens, it's ~35% faster than DeepSeek OCR, and latency stays flat—it does not grow with document length.

Real-World Performance

One developer reported processing ~200 pages/hour of complex Japanese grammar PDF (mixed Kanji, Hiragana, English) on a single RTX 4090—with no slowdown across the entire run.

Community & Open Source Impact

Within 48 hours of release:

• 8 GGUF quantized models on Hugging Face (Ollama, LM Studio, Jan ready)
• Apple Silicon MPS support via community forks
• Official Hugging Face Space demo by akhaliq
• ModelScope mirror for the Chinese AI ecosystem

Susan Zhang on HN: "This is what open source looks like."

The model's internal codename is "Unlimited OCR Works"—a reference to Unlimited Blade Works from the Fate/stay night franchise.

Future Applications

R-SWA is positioned as a general-purpose parsing attention mechanism: ASR (long audio faces the same KV cache wall), video understanding, and code generation all stand to benefit.

Originally published on TekMag.