Understanding MoE Model Architecture

I have read enough MoE papers that I wanted to see the routing logic myself. In this tutorial we will build a small synthetic Mixture-of-Experts agent that uses an Oxlo.ai model as a gating network and routes coding questions to specialized expert models. It is not a production MoE, but it behaves enough like DeepSeek V4 Flash or GLM 5 under the hood that you will understand why only a subset of parameters activates per token.

What you'll need

• Python 3.10 or newer
• An Oxlo.ai API key from https://portal.oxlo.ai
• The OpenAI SDK: pip install openai

Step 1: Define the expert pool and Oxlo.ai client

First we initialize the Oxlo.ai client and define our experts. Because Oxlo.ai is fully OpenAI SDK compatible, this is a single line change to base_url. I picked three coding domains, but you can add more.

import json
from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

EXPERTS = {
    "python": "You are a Python expert. Write concise, idiomatic Python with type hints.",
    "rust": "You are a Rust expert. Prioritize memory safety, zero-cost abstractions, and explicit error handling.",
    "sql": "You are a SQL expert. Optimize queries for readability and PostgreSQL performance.",
}

Step 2: Build the gating router

Real MoE architectures use a learned gating network to decide which experts process each token. We will approximate that with a classification step using qwen-3-32b, which handles agent workflows and multilingual reasoning well. The model returns JSON so we can parse the routing decision reliably.

import json
from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

EXPERTS = {
    "python": "You are a Python expert. Write concise, idiomatic Python with type hints.",
    "rust": "You are a Rust expert. Prioritize memory safety, zero-cost abstractions, and explicit error handling.",
    "sql": "You are a SQL expert. Optimize queries for readability and PostgreSQL performance.",
}

ROUTER_SYSTEM_PROMPT = """You are a gating network in a Mixture-of-Experts system.
Given a user request, analyze the technical domain and return ONLY a JSON object with a single key "expert".
Valid values are: python, rust, sql.
Do not write markdown, explanations, or code blocks. Return raw JSON only."""

def route_query(user_message: str) -> str:
    response = client.chat.completions.create(
        model="qwen-3-32b",
        messages=[
            {"role": "system", "content": ROUTER_SYSTEM_PROMPT},
            {"role": "user", "content": user_message},
        ],
        response_format={"type": "json_object"},
    )
    result = json.loads(response.choices[0].message.content)
    return result["expert"]

Step 3: Create the expert execution layer

Each expert is another chat completion against Oxlo.ai, but with a narrow system prompt. I route actual code generation to deepseek-v3.2 because it is strong on coding tasks. Because Oxlo.ai uses flat per-request pricing, the cost stays predictable even when expert prompts grow long.

import json
from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

EXPERTS = {
    "python": "You are a Python expert. Write concise, idiomatic Python with type hints.",
    "rust": "You are a Rust expert. Prioritize memory safety, zero-cost abstractions, and explicit error handling.",
    "sql": "You are a SQL expert. Optimize queries for readability and PostgreSQL performance.",
}

ROUTER_SYSTEM_PROMPT = """You are a gating network in a Mixture-of-Experts system.
Given a user request, analyze the technical domain and return ONLY a JSON object with a single key "expert".
Valid values are: python, rust, sql.
Do not write markdown, explanations, or code blocks. Return raw JSON only."""

def route_query(user_message: str) -> str:
    response = client.chat.completions.create(
        model="qwen-3-32b",
        messages=[
            {"role": "system", "content": ROUTER_SYSTEM_PROMPT},
            {"role": "user", "content": user_message},
        ],
        response_format={"type": "json_object"},
    )
    result = json.loads(response.choices[0].message.content)
    return result["expert"]

def call_expert(expert_key: str, user_message: str) -> str:
    system_prompt = EXPERTS[expert_key]
    response = client.chat.completions.create(
        model="deepseek-v3.2",
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": user_message},
        ],
    )
    return response.choices[0].message.content

Step 4: Assemble the MoE pipeline

Now we wire the router output to the expert input. The result is a tiny two-stage pipeline that mirrors how larger MoE architectures like DeepSeek R1 671B MoE or GLM 5 dispatch tokens to specific feed-forward subnets without activating the full parameter set.

import json
from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

EXPERTS = {
    "python": "You are a Python expert. Write concise, idiomatic Python with type hints.",
    "rust": "You are a Rust expert. Prioritize memory safety, zero-cost abstractions, and explicit error handling.",
    "sql": "You are a SQL expert. Optimize queries for readability and PostgreSQL performance.",
}

ROUTER_SYSTEM_PROMPT = """You are a gating network in a Mixture-of-Experts system.
Given a user request, analyze the technical domain and return ONLY a JSON object with a single key "expert".
Valid values are: python, rust, sql.
Do not write markdown, explanations, or code blocks. Return raw JSON only."""

def route_query(user_message: str) -> str:
    response = client.chat.completions.create(
        model="qwen-3-32b",
        messages=[
            {"role": "system", "content": ROUTER_SYSTEM_PROMPT},
            {"role": "user", "content": user_message},
        ],
        response_format={"type": "json_object"},
    )
    result = json.loads(response.choices[0].message.content)
    return result["expert"]

def call_expert(expert_key: str, user_message: str) -> str:
    system_prompt = EXPERTS[expert_key]
    response = client.chat.completions.create(
        model="deepseek-v3.2",
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": user_message},
        ],
    )
    return response.choices[0].message.content

def moe_agent(user_message: str) -> dict:
    selected_expert = route_query(user_message)
    answer = call_expert(selected_expert, user_message)
    return {"routed_to": selected_expert, "answer": answer}

Agent system prompt

The agent is the router. Here is the system prompt isolated so you can edit thresholds or add new expert categories without touching the rest of the code.

ROUTER_SYSTEM_PROMPT = """You are a gating network in a Mixture-of-Experts system.
Given a user request, analyze the technical domain and return ONLY a JSON object with a single key "expert".
Valid values are: python, rust, sql.
Do not write markdown, explanations, or code blocks. Return raw JSON only."""

Run it

Test the agent with a query that clearly belongs to one expert.

import json
from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

EXPERTS = {
    "python": "You are a Python expert. Write concise, idiomatic Python with type hints.",
    "rust": "You are a Rust expert. Prioritize memory safety, zero-cost abstractions, and explicit error handling.",
    "sql": "You are a SQL expert. Optimize queries for readability and PostgreSQL performance.",
}

ROUTER_SYSTEM_PROMPT = """You are a gating network in a Mixture-of-Experts system.
Given a user request, analyze the technical domain and return ONLY a JSON object with a single key "expert".
Valid values are: python, rust, sql.
Do not write markdown, explanations, or code blocks. Return raw JSON only."""

def route_query(user_message: str) -> str:
    response = client.chat.completions.create(
        model="qwen-3-32b",
        messages=[
            {"role": "system", "content": ROUTER_SYSTEM_PROMPT},
            {"role": "user", "content": user_message},
        ],
        response_format={"type": "json_object"},
    )
    result = json.loads(response.choices[0].message.content)
    return result["expert"]

def call_expert(expert_key: str, user_message: str) -> str:
    system_prompt = EXPERTS[expert_key]
    response = client.chat.completions.create(
        model="deepseek-v3.2",
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": user_message},
        ],
    )
    return response.choices[0].message.content

def moe_agent(user_message: str) -> dict:
    selected_expert = route_query(user_message)
    answer = call_expert(selected_expert, user_message)
    return {"routed_to": selected_expert, "answer": answer}

if __name__ == "__main__":
    query = "Write a Rust function that parses a CSV string into a vector of structs using serde."
    result = moe_agent(query)
    print(f"Router selected: {result['routed_to']}")
    print("---")
    print(result["answer"])

Example output:

Router selected: rust
---


```rust
use serde::Deserialize;
use std::str::FromStr;

#[derive(Debug, Deserialize)]
struct Record {
    name: String,
    age: u32,
}

fn parse_csv(input: &str) -> Result, Box> {
    let mut reader = csv::Reader::from_reader(input.as_bytes());
    let mut records = Vec::new();
    for result in reader.deserialize() {
        records.push(result?);
    }
    Ok(records)
}
```

What to build next

This synthetic router makes the MoE concept concrete. Two next steps I would suggest: swap the synthetic pipeline for a native MoE model such as DeepSeek V4 Flash or GLM 5 on Oxlo.ai and compare routing quality, or implement top-k routing by returning two experts from the gate and merging their outputs with a weighted vote.