Yesterday XAI dropped Grok 4.5: https://x.ai/news/grok-4-5
Exciting release, but it's closed source. No weights, no training code, no architecture docs to dig into.
So I did the next best thing: went to what xAI has published openly on Hugging Face, https://lnkd.in/gxWEXcG7, and tried rebuilding Grok 2 from scratch using just the public config.
And honestly? The architecture is more interesting than I expected.
A few things stood out once you read the architecture closely:
✅ It's not just MoE: Most MoE models send tokens only through expert layers. Grok 2 does something different: it runs a dense feed-forward path and a sparse MoE path in parallel, adds them together, then divides by √2. Think of it as experts + a safety net on every token.
✅ Four normalization layers per block: Typical decoder blocks use ~2 RMSNorm layers. Grok 2 uses 4 (before/after attention, before/after MoE). That usually means the team cared a lot about training stability at huge scale.
✅ Softcapping is everywhere: Grok caps logits in three places: attention (30), router (30), and final output (50). In plain terms: it gently limits how extreme values can get, so the model doesn't blow up during training/inference.
✅ Grouped-Query Attention (GQA): Full Grok 2 uses 64 query heads but only 8 key/value heads. Same idea as many modern LLMs: keep quality, reduce KV-cache memory cost.
✅ Big model, smaller active compute: ~270B total parameters, but only ~115B are active per token (8 experts, top-2 routing). You still store the big model, but each forward pass uses a fraction of it.
✅GeGLU instead of a plain MLP: The FFN uses gated activations (GELU gate × up projection), which is common in modern LLMs, but Grok combines that with MoE in a pretty unique layout.
✅ Long context is built in, not patched on: The config shows aggressive RoPE settings (very large rope_theta) and 131k context. That suggests long-context was part of the core design.
I built a smaller version (~296M params) and trained it on TinyStories in a single Python file
📂 Code: https://lnkd.in/gb8qRu7W
📓 Colab (T4 GPU): https://lnkd.in/gvFs_7VR
I'm working at the intersection of DevOps + AI (agents, infra debugging, MCP tooling, running LLMs in production). If that's your space too, I'd love to connect.
🎓 Want Live Instructor-Led Training?
If you'd like hands-on, instructor-led sessions where you can ask questions and get your doubts resolved live, check out our courses:
👉 https://ideaweaver.ai/#courses
➡️ If you're looking to try an AI agent built specifically for DevOps, check out DevOps Open Agent:
https://github.com/ideaweaver-ai/devops-open-agent
Top comments (0)