Self-hosting Qwen3.5-122B with vLLM and SGLang on AMD GPU infrastructure, then building a fully autonomous agent on top of it
Introduction
Most conversations about running large language models locally stop at 7B or 13B parameter models on consumer GPUs. That's a reasonable ceiling when you're working with a single RTX card and 24GB of VRAM.
What happens when you have access to an AMD Instinct MI300X with 192GB of HBM3 memory?
This post documents what I learned deploying Qwen3.5-122B — a 122-billion-parameter model — on AMD MI300X GPU infrastructure using vLLM and SGLang, and then building a fully autonomous agent on top of it using OpenClaw. No API tokens. No rate limits. Everything running on AMD hardware.
Why This Matters Beyond the Demo
Before getting into the technical setup, it's worth naming why this is interesting from an architecture perspective.
The default assumption in enterprise AI deployments today is that large models live behind an API ( "Big Three" frontier AI laboratories). You call their endpoint, you pay per token, you accept their rate limits, their data handling policies, and their availability SLAs.
Self-hosting changes that calculus entirely:
- Data sovereignty — sensitive enterprise data never leaves your infrastructure
- Cost predictability — GPU hours instead of per-token pricing at scale
- No rate limits — agentic workloads that make thousands of tool calls don't get throttled
- Model control — you choose the model, the quantization, the inference parameters
The MI300X makes frontier-scale models achievable in this model. 192GB of unified memory means a 122B FP8 model fits comfortably on a single GPU — something that would require 4–8 consumer cards to attempt, if it were possible at all.
The Stack
┌─────────────────────────────────────────────┐
│ OpenClaw Agent │
│ (persistent memory, skills, tools) │
├─────────────────────────────────────────────┤
│ SGLang / vLLM Inference Server │
│ (OpenAI-compatible API endpoint) │
├─────────────────────────────────────────────┤
│ Qwen3.5-122B-A10B-FP8 │
│ (122B MoE, FP8 quantized) │
├─────────────────────────────────────────────┤
│ AMD Instinct MI300X GPU │
│ (192GB HBM3, ROCm software stack) │
└─────────────────────────────────────────────┘
Key components:
- AMD MI300X — 192GB HBM3 unified memory, ROCm software stack
- vLLM / SGLang — inference serving frameworks with OpenAI-compatible endpoints
- Qwen3.5-122B-A10B-FP8 — 122B parameter Mixture-of-Experts model, FP8 quantized
- OpenClaw — agent framework with file-based persistent memory and skill system
Access: AMD Developer Cloud GPU Droplets via the AMD AI Developer Program
Part 1: Spinning Up the Inference Server
Option A: vLLM
Launch a vLLM server serving Qwen3.5-122B with tool-calling enabled
Option B: SGLang (recommended for MI300X)
SGLang with the ROCm-optimized Docker image performed better in my testing — faster time-to-first-token and more stable under agentic workloads that generate many sequential requests
Verifying the Server
Poll until the model is loaded (typically 3–5 minutes for a 122B model)
Part 2: What the Agent Architecture Actually Looks Like
Once the inference server is running, the interesting question is: how do you build an agent on top of it that's actually useful and persistent?
OpenClaw's approach is worth understanding because it's transparent in a way most agent frameworks aren't. Everything lives in markdown files
Why this matters architecturally: The agent's memory, personality, and behavioral policies are human-readable, version-controllable, and editable. You can git diff your agent's personality. You can PR-review changes to its behavioral policies. This is very different from fine-tuning or prompt injection hidden inside a framework.
Other Aspects that can be done includes
Part 3: The Think → Act → Observe → Repeat Loop
The agent's actual behavior:
1. THINK → "I need to understand what accuracy means in this context"
2. ACT → run pytest, observe which tests fail and what they expect
3. OBSERVE → test expects accuracy = correct_chars / total_chars
code computes accuracy = correct_words / total_words
4. THINK → "The bug is in stats.py — wrong denominator"
5. ACT → read stats.py, locate the calculation, apply minimal fix
6. OBSERVE → re-run pytest, all tests pass
7. REPORT → file changed, line changed, before/after
This is genuinely different from a chatbot. The agent doesn't answer the question — it solves the problem, using the filesystem and shell as its senses and hands.
The key architectural insight: the tool loop is what separates an agent from a chatbot. The model itself is just the reasoning component — the value comes from what it can observe and act on.
Part 4: Skills — Making Agent Behavior Reusable
Once the agent solved the bug, the interesting question is: can it do this for any Python project, without re-explaining the process?
Skills are the answer. A skill is a markdown file with YAML frontmatter and step-by-step instructions that gets injected into the agent's system prompt automatically:
Example:
"Create a skill called pytest-debugger. Steps: 1) Read the tests/ folder. 2) Run pytest with verbose output. 3) For each failing test, read the source file it references. 4) Identify the minimal fix. 5) Apply and re-run to confirm. 6) Report: file changed, line changed, what was wrong, what the fix was."
The agent writes the skill file. From that point, invoking it on any project is:
No re-explanation. No new prompts. The skill travels with the agent across any codebase.
The broader pattern: skills are reusable behavioral packages. The agent OS approach means you're building a library of trusted, version-controlled workflows — not re-prompting from scratch each time.
Part 5: Autonomous Workflows Without Writing Schedulers
Example: delegating infrastructure setup to the agent itself.
Instead of writing a cron job, you can describe what you want:
"I need to wake up to a personalized tech brief every morning at 8 AM. Check sgl-project/sglang, vllm-project/vllm, huggingface/transformers, ROCm/ROCm, and openclaw/openclaw. I only care about performance updates, GPU features, and breaking changes — skip CI/infrastructure noise and docs-only PRs. Also search the web for the latest AI hardware news."
The agent:
- Scheduled itself using the system's cron infrastructure
- Write its filtering preferences to
MEMORY.mdfor persistence - Configure GitHub repo monitoring
- Set up a web search workflow for hardware news
No cron syntax. No YAML pipeline files. Just a conversational description of the desired outcome.
This is the "helpful agents" promise in practice — and also, incidentally, exactly the attack surface that the AI Security Conference sessions I attended last month were warning about. Giving an agent the ability to schedule itself, write to memory, and access external data sources is powerful. It's also a significant security perimeter that needs to be treated with the same rigor as any other privileged system process.
What I Learned About the ROCm Stack
A few practical observations from running on AMD infrastructure:
ROCm is production-ready for inference. The rocm-smi tooling is equivalent to nvidia-smi in terms of what you can observe — GPU utilization, memory usage, temperature. For inference workloads specifically, I found no meaningful difference in model output quality versus CUDA-based deployments.
SGLang's ROCm image is the right starting point. The lmsysorg/sglang:*-rocm700-mi30x Docker image handles the driver compatibility complexity for you. Don't try to build from scratch for initial experiments.
The MI300X unified memory architecture matters. 192GB of HBM3 means you're not juggling CPU/GPU memory transfers for large models. The 122B FP8 model loads entirely onto the GPU and stays there — no swapping, no offloading. This shows up in inference latency consistency.
Tool calling requires explicit configuration. The --enable-auto-tool-choice and --tool-call-parser qwen3_coder flags are essential for agentic use cases. Without them, the model won't correctly format tool call responses that agent frameworks like OpenClaw expect.
Connecting This to Broader Agent Architecture Principles
Running this exercise alongside the AI security work I've been doing recently surfaces an interesting tension.
The same properties that make a self-hosted 122B agent powerful — persistent memory, filesystem access, shell execution, scheduled autonomous operation — are exactly the attack surfaces that need to be secured:
- Memory persistence = potential injection vector if any memory write is unvalidated
- Shell execution = arbitrary code execution if the agent is manipulated
- Scheduled autonomous operation = actions that happen without human review
- External data access = context poisoning entry points
The agent OS model I described in a recent post on agent security architecture maps directly onto this stack. The OpenClaw workspace structure — where behavioral policies live in auditable markdown files, where skills are version-controlled, where memory has provenance — is actually a reasonable starting point for the kind of transparent, auditable agent infrastructure the security community is pushing for.
The missing pieces for enterprise use: secrets injection at the execution boundary (not in the system prompt), capability scoping per agent, and budget/rate limits on autonomous operations. These are solvable with the right wrapper infrastructure around an open deployment like this.
Getting Started — The Practical Path
This is specific to this particular model. Feel free to explore other AI Developer Programs from other leading vendors as well
*1. Sign up for AMD AI Developer Program *
2. Create a GPU Droplet
Select MI300X hardware + ROCm software image. SSH access or browser console.
3. Launch your inference server
Use the SGLang Docker command above. Wait ~5 minutes for the 122B model to load.
4. Install OpenClaw
Point it at your SGLang endpoint during onboarding.
5. Start building
The workshop notebook at github.com/xxx/amd-gpu-workshops walks through everything in this post step by step.
Bottom Line
Running a 122B parameter model locally on AMD MI300X is not a research exercise anymore — it's a practical deployment option for teams that need data sovereignty, cost predictability at scale, or freedom from API rate limits.
The ROCm stack is production-ready. SGLang on MI300X performs well. The OpenClaw agent framework gives you a transparent, auditable foundation to build on.
What remains genuinely hard: production-grade security hardening of the agent perimeter. The same capabilities that make a self-hosted agent useful also expand the attack surface significantly. That's the engineering problem worth focusing on next.
I've been writing about enterprise AI agent architecture, MCP integrations, and AI security on dev.to. If you're building agentic systems or exploring self-hosted inference, follow along — more coming on the security hardening side of this stack.
Tags: #llm #agents #amd #gpu #mlops #selfhosted #vllm #rocm #architecture
Top comments (0)