DEV Community

Alain Airom (Ayrom)
Alain Airom (Ayrom)

Posted on

Inside NVIDIA Nemotron 3.5 Lightning: Building a (Local) Mixture-of-Experts Explorer

A technical exploration ofthe Mixture-of-Experts (MoE) design of NVIDIA Nemotron 3.5 Lightning.

Introduction

Modern AI systems face a fundamental tension: users demand the deep reasoning and multi-step planning of large frontier models, yet running dense 30B+ parameter architectures for high-frequency subtasks (like code routing, tool parsing, and syntax validation) imposes massive computational and financial overhead.

To solve this, NVIDIA introduced Nemotron 3.5 Lightning, an open-weight 30-billion parameter Mixture-of-Experts (MoE) model that activates only 3 billion parameters per token.

This article explores nvidia-nemotron-lightning-35-test, a production-grade local explorer and testing framework designed to interact with, monitor, and visualize Nemotron 3.5 Lightning locally via Ollama. Rather than just wrapping API endpoints, this project provides a clear architectural blueprint for building transparent, low-latency AI applications.


Implementation

The nvidia-nemotron-lightning-35-test project provides a robust framework and sandbox to explore, serve, and evaluate NVIDIA Nemotron 3.5 Lightning. The repository architecture is structured around core demonstration modules, documentation sets, and utility scripts designed to test MoE behavior, speculative decoding, and application-level routing. At the heart of the system is a Mixture-of-Experts (MoE) transformer backbone. Unlike dense models where all parameters are activated for every token, Nemotron 3.5 Lightning routes tokens through sparse expert layers, ensuring high representational capacity (30B total) combined with rapid execution speed (3B active parameters).

Architectural Philosophy: The MoE Paradigm & Efficiency

Unlike dense models that utilize their entire parameter set for every token generated, Nemotron 3.5 Lightning routes tokens dynamically through sparse expert layers. This decoupled design provides the representational capacity of a 30B model with the execution speed of a 3B model.

+-------------------------------------------------------------------+
|                        Client / User Input                        |
+-------------------------------------------------------------------+
                                  │
                                  ▼
+-------------------------------------------------------------------+
|                   NeMo Switchyard / Router Layer                  |
+-------------------------------------------------------------------+
                                  │
                                  ▼
+-------------------------------------------------------------------+
|               NVIDIA Nemotron 3.5 Lightning (MoE)                 |
|  - 30B Total Parameters / 3B Active Parameters                    |
|  - Multi-Token Prediction (MTP) & Speculative Decoding Support    |
+-------------------------------------------------------------------+
         │                                                 │
         ▼                                                 ▼
+-------------------------+                       +-------------------------+
|      Draft Models       |                       |   Serving Frameworks    |
| (DSpark / MTP / DFlash) |                       | (vLLM / llama.cpp / LoRA)|
+-------------------------+                       +-------------------------+
Enter fullscreen mode Exit fullscreen mode

To maintain complete intellectual honesty, the project’s application architecture enforces a strict boundary between hard telemetry (data directly measured from backend inference) and illustrative modeling (educational visual simulations):

| Data Component                        | Origin                                                       | Status                           |
| ------------------------------------- | ------------------------------------------------------------ | -------------------------------- |
| **Connection State & Model Metadata** | Ollama `/api/version`, `/api/tags`, `/api/ps` blog-post-V2.md | **Measured** blog-post-V2.md     |
| **Generation Speed & Token Counts**   | Ollama final streaming chunk (`eval_count`, `eval_duration`) blog-post-V2.md | **Measured** blog-post-V2.md     |
| **Time to First Token (TTFT)**        | Client-side performance timer blog-post-V2.md                | **Measured** blog-post-V2.md     |
| **Expert Activation Heatmap**         | `moe_demo/router_sim.py` (Deterministic BLAKE2b simulation) blog-post-V2.md+ 1 | **Illustrative** blog-post-V2.md |

Enter fullscreen mode Exit fullscreen mode

⚠️ Design Note on Ollama: Ollama does not currently expose internal per-token router telemetry (e.g., revealing exactly which expert ID fired for a specific token). The application explicitly flags its expert heatmaps as simulated, preserving user trust and technical accuracy.



Core Project Structure

The codebase is organized around a clean separation of concerns, isolating the GUI layer from core business logic and state computation:

nvidia-nemotron-lightning-35-test/
├── Docs/                  # System documentation and architectural references
├── docs_tools/            # Automated document parsers and build scripts
├── moe_demo/              # Core application packages
│   ├── __init__.py
│   ├── categories.py      # Task heuristic classifiers and expert affinities
│   ├── config.py          # Environment-based configuration loader
│   ├── engine.py          # UI-agnostic request builder and stream handler
│   ├── metrics.py         # Nanosecond-to-throughput converters
│   ├── ollama_client.py   # Robust REST wrapper for Ollama APIs
│   ├── router_sim.py      # Deterministic top-6-of-128 routing simulator
│   └── ui_components.py   # Streamlit visual cards and heatmaps
└── app.py                 # Central Streamlit GUI entrypoint
Enter fullscreen mode Exit fullscreen mode


Implementation Deep Dive

Environment Configuration (moe_demo/config.py)

Configuration is managed via environment variables with fallback support for local .env files. Using a frozen dataclass ensures settings remain immutable throughout the application lifecycle:

"""Runtime configuration. Values come from the environment / `.env` (see `.env.example`)."""
from __future__ import annotations

import os
from dataclasses import dataclass
from pathlib import Path

ROOT = Path(__file__).resolve().parent.parent


def load_env_file(path: Path | None = None) -> None:
    """Load KEY=VALUE lines from `.env` into os.environ without overriding real env vars.

    A tiny parser is used on purpose so the app has no extra dependency and also works
    when started with `streamlit run app.py` instead of `scripts/start.sh`.
    """
    path = path or ROOT / ".env"
    if not path.is_file():
        return
    for raw in path.read_text(encoding="utf-8").splitlines():
        line = raw.strip()
        if not line or line.startswith("#") or "=" not in line:
            continue
        key, _, value = line.partition("=")
        value = value.strip().strip('"').strip("'")
        os.environ.setdefault(key.strip(), value)


@dataclass(frozen=True)
class Settings:
    base_url: str
    model: str
    keep_alive: str
    request_timeout: float


def get_settings() -> Settings:
    """Build settings from the environment (defaults only for non-sensitive values)."""
    load_env_file()
    return Settings(
        base_url=os.environ.get("OLLAMA_BASE_URL", "http://localhost:11434").rstrip("/"),
        model=os.environ.get("OLLAMA_MODEL", "nemotron-3.5-lightning:30b-a3b-mlx"),
        keep_alive=os.environ.get("OLLAMA_KEEP_ALIVE", "15m"),
        request_timeout=float(os.environ.get("OLLAMA_TIMEOUT", "600")),
    )

Enter fullscreen mode Exit fullscreen mode

Note: The default OLLAMA_TIMEOUT is set to 600 seconds to gracefully handle cold-starting a massive ∼19 GB∼19 GB model artifact without throwing premature timeouts.

Resilient Ollama Client (moe_demo/ollama_client.py)

The client wraps Ollama's REST API (/api/version, /api/tags, /api/ps, /api/generate, and /api/chat), translating raw HTTP errors into a clean exception hierarchy:

class OllamaError(Exception): ...
class OllamaConnectionError(OllamaError): ...
class OllamaModelNotFound(OllamaError): ...
class OllamaTimeout(OllamaError): ...
Enter fullscreen mode Exit fullscreen mode

During streaming chat completions, NDJSON response streams are safely parsed into typed StreamEvent objects, separating thinking/reasoning blocks (thinking) from final generation tokens (content) and server statistics (done):

for line in resp.iter_lines():
    if not line:
        continue
    data = json.loads(line)
    if "error" in data:
        raise OllamaError(str(data["error"]))
    msg = data.get("message") or {}
    if msg.get("thinking"):
        yield StreamEvent("thinking", msg["thinking"])
    if msg.get("content"):
        yield StreamEvent("content", msg["content"])
    if data.get("done"):
        yield StreamEvent("done", stats=data)
Enter fullscreen mode Exit fullscreen mode

UI-Agnostic Generation Engine (moe_demo/engine.py)


The engine converts user parameters into validated option payloads and manages conversation window constraints:

"""UI-independent generation logic: parameters -> Ollama request -> live metrics.

Keeping this separate from Streamlit makes it unit-testable against a stub Ollama server.
"""
from __future__ import annotations

import time
from dataclasses import dataclass
from typing import Iterator

from .metrics import RunMetrics, from_stats
from .ollama_client import OllamaClient

THINK_MODES = ("Auto (per category)", "Off", "On", "Medium")


def build_options(temperature: float, top_p: float, num_ctx: int, num_predict: int) -> dict:
    """Map UI values to Ollama `options`. These are re-read on every request, so changing a
    slider affects the very next message."""
    if not 0.0 <= temperature <= 2.0:
        raise ValueError("temperature must be within 0..2")
    if not 0.0 < top_p <= 1.0:
        raise ValueError("top_p must be within (0, 1]")
    if num_ctx < 512:
        raise ValueError("num_ctx must be at least 512")
    return {"temperature": temperature, "top_p": top_p, "num_ctx": int(num_ctx),
            "num_predict": int(num_predict)}


def resolve_think(mode: str, category_think: str) -> bool | str:
    """Translate the UI reasoning selector into the value of Ollama's `think` field.

    The model accepts False, True or "medium" (see `ollama show`)."""
    chosen = category_think if mode.startswith("Auto") else mode.lower()
    if chosen == "off":
        return False
    if chosen == "medium":
        return "medium"
    if chosen == "on":
        return True
    raise ValueError(f"unknown reasoning mode: {mode!r}")


def build_messages(system_prompt: str, history: list[dict], prompt: str,
                   use_history: bool, max_turns: int = 6) -> list[dict]:
    """System prompt + (optionally) the last `max_turns` turns of the conversation + new prompt."""
    messages: list[dict] = []
    if system_prompt:
        messages.append({"role": "system", "content": system_prompt})
    if use_history:
        for m in history[-max_turns * 2:]:
            if m["role"] in ("user", "assistant") and m.get("content"):
                messages.append({"role": m["role"], "content": m["content"]})
    messages.append({"role": "user", "content": prompt})
    return messages


@dataclass
class LiveUpdate:
    """Snapshot yielded while streaming."""
    content: str
    thinking: str
    chunks: int                 # streamed chunks ~ tokens (live approximation)
    elapsed: float
    ttft: float
    live_tps: float             # approximate until the final exact metrics arrive
    final: RunMetrics | None = None


def stream_answer(client: OllamaClient, model: str, messages: list[dict], options: dict,
                  think: bool | str, keep_alive: str) -> Iterator[LiveUpdate]:
    """Stream a reply and yield LiveUpdate objects; the last one carries exact `final` metrics."""
    start = time.perf_counter()
    first: float | None = None
    content, thinking, chunks = "", "", 0
    for ev in client.chat_stream(model, messages, options=options, think=think, keep_alive=keep_alive):
        now = time.perf_counter()
        if ev.kind in ("content", "thinking"):
            if first is None:
                first = now - start
            chunks += 1
            if ev.kind == "content":
                content += ev.text
            else:
                thinking += ev.text
            gen_time = max(now - start - (first or 0), 1e-6)
            yield LiveUpdate(content, thinking, chunks, now - start, first or 0.0,
                             chunks / gen_time if chunks > 1 else 0.0)
        elif ev.kind == "done":
            wall = now - start
            final = from_stats(ev.stats, ttft=first or wall, wall=wall)
            yield LiveUpdate(content, thinking, chunks, wall, first or wall, final.tokens_per_second, final)


def reasoning_share(thinking: str, content: str) -> float:
    """Fraction of streamed characters that belong to the reasoning trace (0..1)."""
    total = len(thinking) + len(content)
    return len(thinking) / total if total else 0.0

Enter fullscreen mode Exit fullscreen mode

To prevent unbounded context bloat during long sessions, build_messages() automatically caps history to the last 6 turns while preserving system instructions.

Exact Metric Conversion (moe_demo/metrics.py)

Ollama returns internal run durations natively in nanoseconds. The metrics module transforms these numbers into actionable engineering metrics, calculating decode speed exclusively over evaluation duration (eval_duration) to avoid diluting performance stats with prefill phase latencies:

"""Inference metrics derived from Ollama's final streaming chunk plus client-side timing.

Ollama reports durations in nanoseconds. All values here are *measured*, not estimated.
"""
from __future__ import annotations

from dataclasses import asdict, dataclass
from typing import Any

NS = 1_000_000_000.0


@dataclass
class RunMetrics:
    prompt_tokens: int = 0
    generated_tokens: int = 0          # includes reasoning ("thinking") tokens
    load_seconds: float = 0.0          # time spent loading the model into memory
    prompt_eval_seconds: float = 0.0
    generation_seconds: float = 0.0
    total_seconds: float = 0.0         # server-side total
    ttft_seconds: float = 0.0          # client-side time to first streamed token
    wall_seconds: float = 0.0          # client-side end-to-end

    @property
    def tokens_per_second(self) -> float:
        """Generation speed (decode phase)."""
        return self.generated_tokens / self.generation_seconds if self.generation_seconds > 0 else 0.0

    @property
    def prompt_tokens_per_second(self) -> float:
        return self.prompt_tokens / self.prompt_eval_seconds if self.prompt_eval_seconds > 0 else 0.0

    @property
    def total_tokens(self) -> int:
        return self.prompt_tokens + self.generated_tokens

    def to_dict(self) -> dict[str, Any]:
        d = asdict(self)
        d["tokens_per_second"] = round(self.tokens_per_second, 2)
        d["prompt_tokens_per_second"] = round(self.prompt_tokens_per_second, 2)
        d["total_tokens"] = self.total_tokens
        return d


def from_stats(stats: dict[str, Any], ttft: float = 0.0, wall: float = 0.0) -> RunMetrics:
    """Build RunMetrics from the `done` chunk of /api/chat."""
    return RunMetrics(
        prompt_tokens=int(stats.get("prompt_eval_count") or 0),
        generated_tokens=int(stats.get("eval_count") or 0),
        load_seconds=(stats.get("load_duration") or 0) / NS,
        prompt_eval_seconds=(stats.get("prompt_eval_duration") or 0) / NS,
        generation_seconds=(stats.get("eval_duration") or 0) / NS,
        total_seconds=(stats.get("total_duration") or 0) / NS,
        ttft_seconds=ttft,
        wall_seconds=wall,
    )


def active_fractions(total_params_b: float = 30.0, active_params_b: float = 3.0,
                     n_experts: int = 128, top_k: int = 6) -> dict[str, float]:
    """Sparsity facts from the published model card: share of weights / experts used per token."""
    return {
        "active_param_fraction": active_params_b / total_params_b,
        "active_expert_fraction": top_k / n_experts,
    }

Enter fullscreen mode Exit fullscreen mode

Deterministic Expert Routing Simulation (moe_demo/router_sim.py)

To visualize how sparsity behaves across different task types (e.g., Code vs. Creative Writing vs. Factual Q&A), the simulator models top-6 selections out of 128 total experts (grouped into 8 semantic categories).

Using blake2b hashing on individual token-expert pairs ensures the simulation is completely deterministic—the same prompt always produces identical heatmap visualizations—while maintaining realistic variance:

"""Deterministic *simulation* of top-k expert routing, for visualisation only.

Why a simulation?  Ollama gives no access to the router's per-token decisions. To still let users
*see* the idea of sparse routing we mimic it:

* 128 routed experts (as in the published Nemotron 3.5 Lightning card), 6 chosen per token.
* The experts are arranged in 8 groups of 16 (see `categories.EXPERT_GROUPS`).
* Each token gets a score for every expert = category affinity of the expert's group
  + a deterministic pseudo-random "token-specific" component (hash of token and expert id).
  The top-k scores win. That mimics a learned router: a soft preference for the right
  specialists, but with token-level variety, so several groups are used per category.

The result is reproducible (same prompt -> same picture) and is NOT telemetry from the model.
"""
from __future__ import annotations

import hashlib
import re
from dataclasses import dataclass

from .categories import CATEGORIES, EXPERT_GROUPS

N_EXPERTS = 128
TOP_K = 6
GROUP_SIZE = N_EXPERTS // len(EXPERT_GROUPS)  # 16
AFFINITY_STRENGTH = 0.5   # how strongly category affinity biases the router vs. token noise
MAX_TOKENS = 200          # cap simulated tokens so very long prompts stay cheap


@dataclass
class RoutingResult:
    category: str
    n_tokens: int
    expert_counts: list[int]            # length 128: how many tokens selected each expert
    group_share: dict[str, float]       # group -> share of all expert selections (sums to 1)
    top_experts: list[tuple[int, int]]  # (expert id, count), most used first


def group_of(expert_id: int) -> int:
    return expert_id // GROUP_SIZE


def _noise(token: str, expert_id: int) -> float:
    """Deterministic pseudo-random value in [0, 1) for a (token, expert) pair."""
    digest = hashlib.blake2b(f"{token}|{expert_id}".encode(), digest_size=4).digest()
    return int.from_bytes(digest, "big") / 2**32


def tokenize(text: str) -> list[str]:
    """Cheap word/punctuation tokenizer; only used to size the simulation."""
    return re.findall(r"\w+|[^\w\s]", text.lower())[:MAX_TOKENS]


def route_token(token: str, category_key: str) -> list[int]:
    """Return the TOP_K expert ids chosen for one token under the given category."""
    affinity = CATEGORIES[category_key].expert_affinity
    scores = []
    for e in range(N_EXPERTS):
        group_name = EXPERT_GROUPS[group_of(e)]
        scores.append((AFFINITY_STRENGTH * affinity.get(group_name, 0.0) + _noise(token, e), e))
    scores.sort(reverse=True)
    return [e for _, e in scores[:TOP_K]]


def simulate(text: str, category_key: str) -> RoutingResult:
    """Simulate routing of every token of `text` and aggregate the selections."""
    if category_key not in CATEGORIES:
        raise ValueError(f"unknown category: {category_key}")
    tokens = tokenize(text) or ["<empty>"]
    counts = [0] * N_EXPERTS
    for tok in tokens:
        for e in route_token(tok, category_key):
            counts[e] += 1
    total = sum(counts)
    group_totals = [0] * len(EXPERT_GROUPS)
    for e, c in enumerate(counts):
        group_totals[group_of(e)] += c
    share = {EXPERT_GROUPS[i]: group_totals[i] / total for i in range(len(EXPERT_GROUPS))}
    top = sorted(((e, c) for e, c in enumerate(counts) if c), key=lambda x: (-x[1], x[0]))[:10]
    return RoutingResult(category_key, len(tokens), counts, share, top)
def _noise(token: str, expert_id: int) -> float:
    """Deterministic pseudo-random value in [0, 1) for a (token, expert) pair."""
    digest = hashlib.blake2b(f"{token}|{expert_id}".encode(), digest_size=4).digest()
    return int.from_bytes(digest, "big") / 2**32

def route_token(token: str, category_key: str) -> list[int]:
    affinity = CATEGORIES[category_key].expert_affinity
    scores = []
    for e in range(N_EXPERTS):
        group_name = EXPERT_GROUPS[group_of(e)]
        scores.append((AFFINITY_STRENGTH * affinity.get(group_name, 0.0) + _noise(token, e), e))
    scores.sort(reverse=True)
    return [e for _, e in scores[:TOP_K]]
Enter fullscreen mode Exit fullscreen mode

Conclusion & Key Takeaways

The nvidia-nemotron-lightning-35-test project demonstrates that local MoE applications can be robust, highly performant, and transparent. Key takeaways from this architecture include:

  1. Extreme Economic Viability: Activating only 3B out of 30B parameters dramatically drops inference cost and latency, making high-frequency agentic loops viable on standard hardware.
  2. Decoupled Architecture: Cleanly separating Ollama API plumbing (ollama_client.py) and parameter validation (engine.py) from GUI code (app.py) allows for comprehensive unit testing and flexible deployment.
  3. Radical Transparency: Explicitly separating measured backend metrics from illustrative routing simulations sets a gold standard for building trust in specialized AI tooling.

Thanks for reading 👨‍🔧

References & Links

Top comments (0)