DEV Community: Silvestre

Deploying GLM-5.2-FP8 (700B MoE) on Modal: Serverless 8x H200s, Trade-offs, and Lessons Learned

Silvestre — Mon, 22 Jun 2026 23:26:21 +0000

The release of GLM-5.2 by Zhipu AI is a significant development in open-weights AI: a Mixture-of-Experts (MoE) reasoning model optimized for long-horizon planning, complex software engineering, and high-density reasoning.

According to recent benchmarks like SWE-bench Pro and GPQA, GLM-5.2 stands as the most capable open-source LLM available on the market today, matching or exceeding proprietary models like Claude 3.5 Sonnet and GPT-4o on engineering tasks.

However, self-hosting a model of this scale—which logs in at a massive 703.74 GiB FP8 checkpoint—requires orchestrating an 8x NVIDIA H200 GPU cluster (141GB HBM3e each) to support the model weights and its 131k token context window.

Renting a dedicated 8x H200 (141GB) node on clouds like RunPod costs $35.12 per hour ($4.39/GPU/hr), while Modal charges $36.31 per hour ($4.54/GPU/hr or $0.001261/GPU/sec). However, because Modal bills strictly by the second and automatically scales the cluster to zero when idle, a typical 20-minute active development session—including the cold start and scale-down idle wait—costs only ~$12.00, dropping to exactly $0.00/hour when inactive without requiring manual intervention.

This case study documents the serverless deployment architecture on Modal using vLLM, the technical bottlenecks encountered, and the practical lessons learned during the integration.

Under the Hood: GLM-5.2 & Quantization Trade-offs

Deploying a model of this scale on a single 8-GPU node requires careful memory layout planning. Serving the original 16-bit (BF16) weights is mathematically impossible on a single node (requiring over 1.5 Terabytes of VRAM and multi-node pipeline-parallel orchestration).

We are left with multiple quantization formats. Here is the architectural trade-off analysis:

Format / Precision	VRAM Required (Weights + Cache)	Compute Hardware Path	Accuracy Retention	Throughput & Latency Trade-off
BF16 (Unquantized)	~1.5 TB (Requires 16x H200 GPUs)	Slower (Overhead of multi-node PP)	100% (Baseline)	Slowed by inter-node network communication bottlenecks. High hosting cost.
INT8 (W8A8 Integer)	~750 GB (Fits on 8x H200)	Standard Tensor Cores	High (No visible degradation)	Slower execution. Int8 kernels require runtime casting and lack the hardware-level optimization of Hopper's FP8 Tensor Cores.
FP8 (Z-AI Native FP8)	~700 GB (Fits on 8x H200)	Hopper Native FP8 Tensor Cores	High (DeepGEMM preserves routing quality)	Optimal choice. Leverages native hardware Tensor Cores, yielding 1.5x-2x faster token generation than Int8/BF16, with negligible accuracy loss.
INT4 (W4A16 / Legacy)	~400 GB (Fits on 4x H200)	Standard Tensor Cores	Low (Severe reasoning loss)	Fast generation but suffers massive accuracy degradation in complex coding and reasoning benchmarks.

Relative Accuracy Retention vs. Quantization Format

This visual representation shows how each quantization format retains the model's baseline intelligence (relative to BF16) on complex reasoning benchmarks (like GPQA and SWE-bench):

Format   VRAM Req.    Relative Accuracy Retention (%)
───────  ─────────   ───────────────────────────────────────────────
BF16      ~1.5 TB    [████████████████████████████████████████] 100.0% (Baseline)
FP8       ~700 GB    [██████████████████████████████████████░]  99.2% (DeepGEMM Optimized)
INT8      ~750 GB    [█████████████████████████████████████░░]  98.6% (Standard W8A8)
INT4      ~400 GB    [██████████████████████████████░░░░░░░░]  91.4% (Severe Reasoning Loss)

Format Selection: FP8 represents the optimal trade-off for self-hosting. It retains 99.2% of the model's raw intelligence, fits on a single 8-GPU node, reduces the Key-Value (KV) cache footprint in half, and leverages Hopper's native hardware Tensor Cores. Under the hood, vLLM utilizes DeepSeek's open-source DeepGEMM library (which vLLM utilizes for GLM's MoE routing kernels) to execute the MoE routing matrix multiplications with highly optimized Triton paths.

Why Self-Host? The Strategic Decision Framework

While managed multi-tenant API providers offer low friction and instant availability, self-hosting a model of this scale becomes a necessity under specific, highly technical scenarios:

Strict Codebase Privacy & IP Compliance: If you are building proofs-of-concept (PoCs) or products in regulated environments (finance, healthcare, enterprise software), sending proprietary codebase chunks or sensitive client data to third-party API routers violates strict compliance protocols. Self-hosting on isolated, serverless GPU tenants ensures your intellectual property never crosses your secure network perimeter.
Bypassing Rate Limits and Context Throttling: Running long-horizon, autonomous software engineering agents requires deep, repetitive context evaluation (SWE-bench runs). Third-party APIs heavily throttle context sizes under concurrent loads or charge exorbitant premium fees. Owning the cluster guarantees that the entire 8x H200 compute power is exclusively yours, with zero artificial rate limiting.
Prefix Caching Stability: On public multi-tenant APIs, your context cache gets evicted constantly as the provider balances load across thousands of concurrent users. When self-hosting, you control the GPU memory directly. Your RadixAttention prefix cache stays warm and stable throughout your entire development or evaluation session.

The Infrastructure Blueprint

To serve this model serverless, we must tightly control VRAM allocations, minimize startup network latency, and protect our cloud budget.

Here is our complete Infrastructure-as-Code (IaC) configuration using Modal's Python SDK and a specialized vLLM build (vllm/vllm-openai:glm52-cu129):

import os
import modal

vllm_image = (
    modal.Image.from_registry(
        "vllm/vllm-openai:glm52-cu129",
        setup_dockerfile_commands=[
            "RUN ln -sf $(which python3) /usr/local/bin/python",
            "RUN rm -f /usr/local/lib/python3.12/dist-packages/typing_extensions.py",
        ],
    )
    .entrypoint([])
    .pip_install("aiohttp", "typing-extensions>=4.15.0")
    .env(
        {
            "HF_XET_HIGH_PERFORMANCE": "1",
            "VLLM_LOG_STATS_INTERVAL": "1",
        }
    )
)

app = modal.App("glm5-2-inference")

@app.function(
    image=vllm_image,
    gpu="H200:8",
    max_replicas=1, # Protect your budget from accidental parallel scaling
    scaledown_window=15 * 60, # Scale to zero after 15 minutes of inactivity
    secrets=[
        modal.Secret.from_name("huggingface"), # Required to fetch weights!
        modal.Secret.from_name("vllm-api-key"), # Enforce Bearer Token Auth
    ],
    volumes={
        "/root/.cache/huggingface": modal.Volume.from_name("huggingface-cache", create_if_missing=True),
        "/root/.cache/vllm": modal.Volume.from_name("vllm-cache", create_if_missing=True),
    },
)
@modal.web_server(port=8000, startup_timeout=60 * 60)
def serve():
    import subprocess
    cmd = [
        "vllm", "serve", "zai-org/GLM-5.2-FP8",
        "--served-model-name", "glm-5.2-fp8",
        "--host", "0.0.0.0", # Enforce listening on 0.0.0.0 for Modal proxy routing!
        "--port", "8000",
        "--uvicorn-log-level", "info",
        "--async-scheduling",
        "--tensor-parallel-size", "8",
        "--kv-cache-dtype", "fp8",
        "--max-model-len", "131072", # 131k context window for stability and VRAM headroom
        "--max-num-seqs", "32", # Limits concurrent sequence VRAM pre-allocation
        "--gpu-memory-utilization", "0.92",
        "--trust-remote-code",
        "--speculative-config", '{"method": "mtp", "num_speculative_tokens": 5}',
        "--tool-call-parser", "glm47",
        "--reasoning-parser", "glm45",
        "--enable-auto-tool-choice",
        "--safetensors-load-strategy", "prefetch",
        "--enable-prefix-caching",
        "--enforce-eager" # Crucial serverless boot parameter
    ]

    api_key = os.environ.get("VLLM_API_KEY")
    if api_key: cmd += ["--api-key", api_key]
    subprocess.Popen(cmd)

Technical Post-Mortems & Resolutions

1. Python Module Shadowing (`typing_extensions`)

Symptom: During initial container boot, the vLLM engine crash-looped with: ImportError: cannot import name 'Sentinel' from 'typing_extensions' (/usr/local/lib/python3.12/dist-packages/typing_extensions.py)
Diagnosis: pydantic-core requires typing-extensions>=4.14.1 for the Sentinel class. Even though we installed typing-extensions>=4.15.0 during the build step, the base CUDA image shipped with a legacy single-file module (typing_extensions.py) in dist-packages. Because Python's import system prioritizes single-file modules over package directories in the same search path, it was shadowing our modern package.
Resolution: We added a step in our Dockerfile setup to delete the legacy single-file module prior to running pip:
```
RUN rm -f /usr/local/lib/python3.12/dist-packages/typing_extensions.py
```

2. Safetensors Sequential I/O Bottleneck

Symptom: Startup profiling showed model loading taking over 12 minutes due to sequential reads over Modal's virtual network filesystem. vLLM logged the following warning: Auto-prefetch is disabled because the filesystem (9P) is not a recognized network FS... start vLLM with --safetensors-load-strategy=prefetch.
Resolution: We added the --safetensors-load-strategy prefetch parameter. This forces vLLM to parallelize the disk-to-VRAM loading process using multiple CPU threads, which cut our model loading time from ~12 minutes down to ~1 minute, resulting in a total cold start of ~4.5 minutes (including hardware allocation and DeepGEMM warmup).

3. Speculative Decoding vs. Cold Start (MTP & Eager Mode)

GLM-5.2 utilizes Multi-Token Prediction (MTP) to speculate 5 tokens ahead. To make this serverless, we faced a major architectural choice:

CUDA Graphs / torch.compile (No Eager Mode): The server hung for >20 minutes on startup compiling mathematical graphs for our massive context window. Verdict: Infeasible for serverless.
Eager Mode (--enforce-eager): Boot times dropped to our glorious 4.5 minutes, but the first query with a novel sequence length suffered a ~35-second Time-To-First-Token (TTFT) spike while the MTP engine performed JIT Triton kernel compilation on the fly.
The Decision: We chose Eager Mode. A 35s latency spike on the first interaction is a fair price to pay to avoid a 20-minute startup hang. Once warm, MTP acts with a 100% draft acceptance rate on structured code, providing sustainable generation speeds of 30-50 tokens/s.

Practical Validation: Testing on OpenCode & The Flappy Bird Challenge

To validate the deployment, we integrated the Modal server as an OpenAI-compatible provider in OpenCode.

First, we tested context handling using a large, real-world file: CPython’s standard library asynchronous coordinator asyncio/tasks.py (over 1,060 lines of complex concurrent logic). With --reasoning-parser glm45 active, the model’s Chain-of-Thought (CoT) tokens are routed into a dedicated reasoning_content API property:

// opencode.json
"modal-glm": {
  "npm": "@ai-sdk/openai-compatible",
  "options": {
    "baseURL": "https://<your-modal-workspace>--glm5-2-inference-serve.modal.run/v1",
    "apiKey": "glm5_sk_YOUR_SECURE_KEY"
  },
  "models": {
    "glm-5.2-fp8": {}
  }
}

OpenCode captures this stream and renders the reasoning process inside a collapsible "Thinking" block in the chat. GLM-5.2 digested the 1,065 lines of code, parsed CPython's execution callbacks, and produced highly accurate, structured architectural analyses.

The "One-Shot" Coding Test: Sunset Flier (Flappy Bird)

To stress-test the model's creative capability, logic coherence, and syntax closure in a single-pass context, we executed the following prompt:

The model successfully generated the game ("Sunset Flier") with outstanding attention to engineering detail:

Physics & Game Loop: Clean canvas-based rendering with proper gravity acceleration, jump impulses, score counting, and high-score persistence in localStorage.
JS Audio Synthesis: Instead of loading static .mp3 assets, the model utilized the HTML5 Web Audio API to generate retro sound effects dynamically using oscillator nodes (sine, triangle, square, and sawtooth waveforms) for jumping (flapping), scoring, and crashing (hitting).

This demonstrated the model's capacity to compose interactive, functional software in a single pass.

Future Optimizations

To bring this serverless architecture to the next level of operational excellence, we have mapped out three future optimization vectors:

Keep-Warm Scheduling (Active Sessions): During active coding hours, a simple serverless cron job can be configured in Modal to ping the /health endpoint once every 14 minutes, avoiding the 4.5-minute cold start entirely.
GPU Memory Snapshots: Modal's GPU memory snapshotting technology allows serializing the post-DeepGEMM-warmup VRAM state directly to disk. Restoring a container from a pre-compiled state would bypass both weight-loading and JIT compilation, potentially dropping serverless cold starts to less than 10 seconds.
SGLang Engine Migration: Once SGLang natively supports GLM-5.2's custom MoE layers, migrating the backend from vLLM will help reduce CPU-side host overhead under Eager execution.

References & Technical Resources

Conclusions

Self-hosting a 700B parameter reasoning model on serverless infrastructure is entirely viable today. The open-source tooling (vLLM, Modal, HuggingFace) is mature enough to give individual developers access to frontier-level intelligence with total data privacy.

By understanding the trade-offs of JIT compilation, implementing aggressive I/O prefetching, and leveraging prefix caching, we can serve frontier open-weights AI at a fraction of a dollar per session.

From "Why?" to Wow: Building a Multi-Agent Storyteller After 5-Day AI Agents Intensive Course with Google

Silvestre — Wed, 10 Dec 2025 05:58:06 +0000

My "Aha!" Moment: AI Agents Are More Than Just Chatbots

Before the 5-Day AI Agents Intensive, my view of AI agents was largely centered around conversational interfaces—smart chatbots that could answer questions. The course completely shattered that perception. My key takeaway, and the concept that resonated most, was the idea of an agent as an orchestrator of specialized tools.

It's not about one giant model doing everything. It's about a reasoning engine that knows how to solve a complex problem by breaking it down and delegating tasks to the best "specialist" for the job. This shift from a monolithic to a modular, tool-centric mindset was my biggest "aha!" moment.

How My Understanding Evolved: The Power of the "Worker Agents"

The course's deep dive into Multi-Agent Systems (Day 1) and Tools/MCP(Day 2) was a game-changer. I stopped thinking about building a single, all-powerful agent and started thinking about creating a team of "worker agents" managed by a "coordinator".

This led to a fundamental change in my approach:

Before: "How can I prompt a model to generate a story, an image, and audio?"
After: "How can a Coordinator Agent manage three Specialized Agents—a Writer (Gemini), an Illustrator (Flux.1), and a Narrator (OpenAI TTS)—to work in parallel and deliver a result faster and more efficiently?"

This evolution in understanding was the direct inspiration for my capstone project.

My Capstone Project: 🦁 Curiosity Storybook

For the capstone, I built Curiosity Storybook, an AI agent for the "Agents for Good" track that transforms a child's "Why?" into a magical, multi-sensory learning experience.

Instead of a dry answer, it generates a complete, personalized storybook page with a story, an illustration, and an audio narration.

GitHub Repository
Youtube video

This project is a demonstration of how the most advanced concepts from the course can create a seamless and magical experience.

General Architecture

1. Frontend (UI/UX): A kid-friendly interface built with Gradio, hosted on Hugging Face Spaces.
2. Agent Orchestrator: A main agent managed with Blaxel that uses Gemini 2.5 Pro for reasoning and content generation.
3. Tools:

A custom MCP (Model Context Protocol) server that exposes tools for specific tasks like narration.
Direct calls to heavy-compute services for long-running tasks like image generation.

4. AI Models:

Google Gemini 2.5 Pro: For generating the main story and the illustration prompt.
Flux.1-schnell: For high-quality image generation.
OpenAI TTS: For audio narration.
Hyperbolic (Llama 3.3): For ultra-fast generation of related questions.

What I Learned by Building It

Building this project was where the concepts from the course truly clicked.

Multi-Agent Systems are Practical, Not Just Theoretical: My project implements a Coordinator/Specialist pattern. A main agent in Blaxel orchestrates three parallel tasks, each handled by a specialized model. Watching the story, image, and audio generate concurrently was proof of how powerful this architecture is for user experience.
Context Engineering is the Secret Sauce: Day 3's lesson on Context Engineering was crucial. I implemented a ConversationContext class that uses compaction (summarizing history) to feed a "Question Suggester" agent (Hyperbolic). This allows the agent to suggest relevant follow-up questions without needing the entire conversation transcript, making it fast and efficient. It's the feature that makes the experience feel like a continuous journey of discovery.
Observability Isn't an Afterthought: The "Agent Quality" lesson (Day 4) pushed me to integrate basic observability from the start. I implemented logging for all tool calls and tracing (by passing a session_id) to follow a request from start to finish. When the image generation failed once, I could pinpoint the exact step, proving the value of this pillar immediately.

Final Thoughts

The AI Agents Intensive course was more than a series of lectures; it was a fundamental shift in my mental model of what AI can do. It moved me from thinking about "prompts" to thinking about "systems". My understanding has evolved from seeing agents as simple interfaces to seeing them as complex, problem-solving engines. And "Curiosity Storybook" is the tangible result of that journey.