Beyond RAG: Architecting Local Long-Context Pipelines with Gemma 4's 31B Dense Model

Most AI document processing relies heavily on Retrieval-Augmented Generation (RAG). We chunk data into tiny pieces, vectorize it, and stitch the summaries together. RAG is excellent for finding a needle in a haystack, but it is fundamentally flawed when you need the model to understand the entire haystack at once.

With the release of Gemma 4, specifically the native 128K context window, we finally have the tools to move away from aggressive chunking.

In this post, I’ll break down why long-context local models change how we design AI pipelines, examine the architectural differences between the Gemma 4 variants, and share a case study of how I utilized the 31B Dense model to process massive, unbroken log files locally.

---

The Problem: Chunking Destroys Narrative Coherence

Imagine an Operational Command Center (OCC) monitoring a multi-tenant Kubernetes deployment. A massive cascading failure occurs, generating 200 interconnected infrastructure alerts—Kafka backlogs, CPU spikes, and database deadlocks.

If you feed these logs into a standard chunked AI pipeline, it:

1. Splits the logs into 2,000-token chunks.
2. Summarizes each chunk independently.
3. Merges those summaries into a final report.

The problem? Separation of concerns works in code, but not in narrative analysis. The Kafka backlog in chunk 1 is never contextually linked to the database deadlock in chunk 7. You get a sterile list of bullet points, missing the actual root cause that ties the event together.

To solve this, the model must read the entire event timeline in a single prompt.

---

Why the 31B Dense Model is the Right Tool

The Gemma 4 family offers three main architectures. When designing a system that relies on a 128K context window, intentional model selection is critical.

Model	Primary Strength	Best For
2B / 4B	Edge execution	Ultra-mobile, browser-based tasks
26B MoE	Throughput / Speed	Chatbots, high-volume fast inference
31B Dense	Deep Recall / Reasoning	Complex analysis across large contexts

A typical severe OCC incident log is roughly 80,000 to 100,000 tokens.

I explicitly chose the 31B Dense model over the 26B Mixture-of-Experts (MoE). While MoE models are undeniably faster at inference, Dense architectures traditionally exhibit superior long-context recall. When asking a model to evaluate 100,000 tokens of raw server metrics and deduce the single underlying failure thread, coherent reasoning across the full document is far more valuable than raw token generation speed.

The Local-First Advantage

Infrastructure alert data is confidential. By running ollama run gemma4:31b, the data never leaves the machine. No API keys, no data residency concerns, and no per-token cost at scale.

---

Case Study: The Long-Context "Fast-Path" Architecture

To demonstrate this, I built a 4-agent pipeline to generate analytical reports from raw data. Instead of forcing all data through a chunking mechanism, the architecture implements a Long-Context Fast-Path.

Here is how the routing logic cleanly separates the decision-making process:

def _use_full_document(self, document_text: str) -> bool:
    """
    Determines if the document can be processed in a single, unchunked pass.
    """
    provider = getattr(config, "PROVIDER", "ollama")
    use_long_ctx = getattr(config, "USE_LONG_CONTEXT", True)
    model = getattr(config, "OLLAMA_MODEL", "gemma4:31b")

    if not use_long_ctx:
        return False

    is_gemma4_local = (provider == "ollama" and "gemma4" in model.lower())
    is_gemma4_cloud = (
        provider == "openrouter" and 
        "gemma-4" in getattr(config, "MODEL_ALL", "").lower()
    )

    if not (is_gemma4_local or is_gemma4_cloud):
        return False

    # Gemma 4 supports 128K tokens. 
    max_chars = getattr(config, "GEMMA4_LONG_CONTEXT_CHARS", 400_000)
    return len(document_text) <= max_chars

When this returns True, the orchestrator bypasses all intermediate summarizing agents. The entire context is injected directly into the primary narrative agent.

Multimodal Processing

I also implemented a call_vision() gateway using Gemma 4's native multimodal input. Ops teams can drop a screenshot of a dashboard (.png, .jpg), and Gemma 4 inherently connects the visual spikes to the text-based logs, extracting the numbers to use in the slides without needing a separate vision model.

---

Code & Running It Yourself

You can find the complete code for the CLI pipeline, FastAPI backend, and React frontend here:

GitHub Repository: [Insert your GitHub URL here]

For local, private execution:

# Install Ollama and pull the model
ollama pull gemma4:31b

# Clone and install
git clone [Your-Repo-URL]
pip install -r requirements.txt
playwright install chromium

# Set provider
echo "PROVIDER=ollama" >> .env
echo "OLLAMA_MODEL=gemma4:31b" >> .env

# Run the orchestrator
python orchestrator.py --input your_alerts.txt

(Instructions for OpenRouter are also available in the repository README).

---

What I Learned

1. Long context is not free. Feeding 80,000+ tokens into a model requires real hardware — the 31B variant needs roughly ~32GB VRAM to run efficiently locally with quantization. For most developers, cloud APIs or Kaggle notebooks are the practical path.
2. Dense beats MoE for recall tasks. For reading hundreds of alerts and synthesizing a coherent narrative, the Dense architecture was significantly more reliable.
3. Multimodal is genuinely useful. Unlocking screenshot processing completely changed the workflow for teams who rely on visual dashboards.
4. Open weights = architecture freedom. Being able to run this pipeline entirely on-premise under Apache 2.0 is a legitimate business advantage for enterprise environments.

The shift toward capable, open-weight, large-context models like Gemma 4 means we no longer have to compromise our data architecture to fit the limitations of an AI. We can finally build systems that read our data the way we do.