How to Run GLM-5.3-Flash Locally

How to Run GLM-5.3-Flash Locally: Hardware, Quantization, and Cost

GLM-5.3-Flash is a 320-billion-parameter model released under the MIT license. The license lets you run, modify, and redistribute it freely—but the parameter count still requires serious hardware.

Try Apidog today

The key detail is that only 18 billion parameters are active per token, and quantized builds are available. That makes the model practical on hardware well below the 8x H200 configuration used in many guides.

This article covers the realistic hardware tiers, from full-precision production serving to quantized workstation deployments, and explains when self-hosting is worth considering.

What you are loading

Property	Value
Total parameters	320B
Active per token	18B
Architecture	MoE, hybrid linear and sparse attention
Context	1,048,576 tokens
License	MIT
Weights	zai-org/GLM-5.3-Flash
GGUF quants	unsloth/GLM-5.3-Flash-GGUF

The mixture-of-experts architecture makes this possible. All 320B parameters must remain resident in memory, but only 18B participate in each token calculation. Compute requirements are therefore much lower than the total parameter count suggests.

Memory—not FLOPs—is the primary constraint.

Z.ai reports a KV cache approximately 4.4 times smaller than GLM-5.3. That matters for long-context workloads because the KV cache grows as the context fills, and a 1M-token window can otherwise consume enormous amounts of memory.

Tier 1: Full precision on a production node

For full-quality serving with meaningful concurrency, the reference setup is an 8x H200 node. Each H200 provides 141GB of memory, for approximately 1,128GB total. An 8x H20 node is also suitable.

The weights alone require roughly 700–800GB, depending on precision. You also need additional capacity for the KV cache and runtime overhead. Renting a node of this class typically costs around $24–$48 per day.

vLLM

vLLM is the usual default and has the broadest ecosystem support. Use a tensor-parallel size that is a power of two:

vllm serve zai-org/GLM-5.3-Flash \
  --tensor-parallel-size 8 \
  --max-model-len 1048576 \
  --trust-remote-code

During initial validation, set a smaller --max-model-len. Requesting the full million-token window immediately can allocate a large KV cache and produce an out-of-memory error that looks like a model-loading problem.

SGLang

SGLang supported the model from day one and provides published recipes for H100, H200, B200, B300, GB200, and GB300 systems, including multimodal serving. Z.ai used an SGLang-based stack for its own pre-launch serving.

python -m sglang.launch_server \
  --model-path zai-org/GLM-5.3-Flash \
  --tp 8 \
  --context-length 1048576

SGLang often performs well for structured output and high-concurrency agentic workloads. If you are serving a coding agent instead of a simple chat interface, benchmark it against vLLM.

Both runtimes require a tool-call parser for reliable function calling. Check the current documentation for each project because parser names and flags can change between releases.

Tier 2: Quantized builds on smaller hardware

Quantization is the practical option for teams without data-center hardware.

GGUF builds are available from unsloth/GLM-5.3-Flash-GGUF, including aggressive 1-bit and 2-bit formats such as IQ1_S and IQ2_XXS. A 2-bit quantization can reduce the weights enough for a high-memory workstation or multi-GPU consumer system, especially when combined with CPU offload.

There are two important caveats:

Aggressive quantization reduces quality. IQ1_S is substantially different from full precision. MoE models may degrade more gracefully than dense models because they contain more redundancy, but “it runs” does not mean “it runs well.” Evaluate the quantized model on your actual tasks.

Unsloth’s documentation is still marked as work in progress. Available quantizations and recommended settings may change. Verify the published files and configuration before committing to a specific build.

For CPU-heavy and hybrid deployments, KTransformers is designed to keep MoE experts in system RAM while moving only the required computation to the GPU. That design fits a model with 18B active parameters particularly well. TokenSpeed is also listed as a supported runtime.

These guides cover related workflows:

• Run GLM-4.7-Flash locally
• Run GLM-5 locally for free

Calculate your memory budget

Two numbers determine whether a deployment fits.

Weights

At approximately 2 bytes per parameter in BF16, 320B parameters require about 640GB before overhead.

Approximate weight requirements:

• BF16: about 640GB
• FP8: roughly half of BF16
• 4-bit: about 160GB
• 2-bit: less than 160GB, with a significant quality trade-off

KV cache

The KV cache scales with context length and concurrency. It is often the part that causes unexpected failures.

A configuration that loads successfully at 8K context may fail at 128K because the KV cache grows while the weights remain unchanged. Z.ai’s reported 4.4x reduction compared with GLM-5.3 helps, but KV-cache growth is still linear with the number of tokens.

Size your deployment for the context length your application actually uses—not the maximum advertised window. Most applications do not need one million tokens, and allocating for an unused context window is a common way to make this model appear unaffordable.

For earlier context, see our GLM-5.3 self-hosting guide. It was written before the Flash weights were released; the MIT-licensed Flash release supersedes that guidance.

Fine-tuning

The MIT license permits fine-tuning and redistribution, which is unusual for a model at this capability level and one of the strongest reasons to host the weights yourself.

Full fine-tuning of a 320B model is beyond the reach of most teams. Parameter-efficient methods such as LoRA are more realistic. With an MoE model, you must also decide whether to adapt:

• The router
• Selected experts
• Attention layers

Best practices for these choices are less established than they are for dense models.

If you need domain adaptation rather than a new capability, test prompting and retrieval first. With a 1M-token context window, supplying domain knowledge at inference time may be cheaper and more effective than training it into the model.

Sampling settings

Z.ai publishes these recommendations:

Use case	temperature	top_p
General	1.0	0.95
Coding	0.95	1.0

The model also supports three reasoning modes through reasoning_effort:

• low
• high
• max

max is the default. On local hardware, reasoning effort directly affects generation time. If your system generates slowly, switching to low may determine whether the deployment is usable.

Does self-hosting make financial sense?

Usually, no. API pricing makes the comparison difficult.

At list pricing, GLM-5.3-Flash costs $0.15 per million input tokens. A rented 8x H200 node costing approximately $1,000 per month provides the equivalent of about 6.7 billion input tokens through the API.

The node costs the same whether it is saturated or idle. The API charges only for usage. Unless you have consistently high utilization around the clock, the fixed infrastructure cost is usually higher.

Self-hosting makes sense for other reasons:

• Data residency and privacy: Data stays inside your infrastructure.
• No rate limits: Your available capacity is determined by your hardware.
• Availability guarantees: You are not dependent on a vendor’s uptime or pricing decisions.
• The MIT license: You can modify, fine-tune, and redistribute the model.
• Existing hardware: If your GPUs are already purchased and idle, the marginal cost may be limited to electricity.

Our pricing breakdown covers the API comparison in more detail.

Verify your deployment

Both vLLM and SGLang expose OpenAI-compatible endpoints. The same request format works against your local server and Z.ai:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "zai-org/GLM-5.3-Flash",
    "messages": [{"role": "user", "content": "reply with OK"}]
  }'

Go beyond a basic smoke test. Validate:

• Long-context behavior at the length your application needs
• Image input if you are serving multimodal requests
• Tool calling with your production schemas
• Throughput under realistic concurrency
• Output quality on representative prompts

A saved test collection makes this comparison repeatable. Point Apidog at both your local server and the Z.ai endpoint, configure the base URL as an environment variable, and run the same suite against each deployment.

This quickly reveals whether your quantized build still handles the tool schemas and workloads your application depends on.

FAQ

What is the minimum hardware?

For full precision, use an 8x H200-class node. Quantized GGUF builds require considerably less hardware, but quality decreases with more aggressive quantization.

Do I need all 320B parameters in memory?

Yes. Only 18B parameters are active for each token, but all 320B parameters must remain resident. Memory is the limiting factor.

Which is better, vLLM or SGLang?

SGLang had day-zero support, published multimodal recipes, and often performs well for concurrency and structured output. vLLM has broader ecosystem support. Benchmark both against your workload.

Can I run it on a single GPU?

Not at full precision. With aggressive quantization and CPU offload through KTransformers, a high-memory single-GPU system with substantial system RAM is plausible. Expect slow generation.

Is the license really MIT?

Yes. The weights are published under the MIT license, allowing commercial use, modification, and redistribution.