LLM Pipeline Architecture
(Data → Architecture → Scaling Laws → Pretraining → Post-training)
Here is a clear, end-to-end architecture diagram of the modern LLM development pipeline that explicitly incorporates Data, Architecture, Scaling Laws, and Post-training.
┌─────────────────────────────────────────────────────────────────────────────────────┐
│ LLM DEVELOPMENT PIPELINE │
│ Data + Architecture + Scaling Laws + Post-training │
└─────────────────────────────────────────────────────────────────────────────────────┘
═══════════════════════════════════════════════════════════════════════════════════════
1. DATA PIPELINE
═══════════════════════════════════════════════════════════════════════════════════════
Raw Sources
• Web (Common Crawl), Books, Code (GitHub), Wikipedia, Papers, Synthetic data
│
▼
┌──────────────────────────────┐
│ Data Processing │
│ • Quality filtering │
│ • Deduplication (exact + │
│ near-dupe) │
│ • Toxicity / PII removal │
│ • Domain / language mix │
│ • Curriculum / difficulty │
└──────────────┬───────────────┘
│
▼
┌──────────────────────────────┐
│ Tokenization & Packing │
│ • BPE / SentencePiece │
│ • Sequence packing │
│ • Document boundaries │
└──────────────┬───────────────┘
│
▼
High-quality Token Corpus
(trillions of tokens)
═══════════════════════════════════════════════════════════════════════════════════════
2. ARCHITECTURE
═══════════════════════════════════════════════════════════════════════════════════════
┌────────────────────────────────────────────────────────────┐
│ Model Architecture Design │
│ │
│ • Decoder-only Transformer (dominant) │
│ • Alternatives: MoE, Hybrid (SSM + Attention), │
│ Prefix decoder, etc. │
│ │
│ Key hyperparameters: │
│ - Depth (layers) │
│ - Width (d_model) │
│ - Attention heads / GQA / MQA │
│ - FFN dimension & activation │
│ - Context length & positional encoding │
│ - Normalization, residual style │
└────────────────────────┬───────────────────────────────────┘
│
▼
Model Size N (parameters)
═══════════════════════════════════════════════════════════════════════════════════════
3. SCALING LAWS (the decision engine)
═══════════════════════════════════════════════════════════════════════════════════════
┌────────────────────────────────────────────────────────────┐
│ Scaling Laws (Compute Allocation) │
│ │
│ Classic: Kaplan (2020) → Chinchilla (2022) │
│ Loss ≈ f(N, D, C) where C ≈ 6ND │
│ │
│ Key principles: │
│ • Optimal tokens-per-parameter ≈ 20 (Chinchilla) │
│ • More recent: over-training, data-constrained laws, │
│ inference-aware scaling, hyperparameter scaling │
│ │
│ Outputs decisions: │
│ → How large should N be? │
│ → How many tokens D? │
│ → What learning rate / batch size schedule? │
│ → When to stop pretraining? │
└────────────────────────┬───────────────────────────────────┘
│
▼
Compute Budget C → (N*, D*)
═══════════════════════════════════════════════════════════════════════════════════════
4. PRETRAINING
═══════════════════════════════════════════════════════════════════════════════════════
┌────────────────────────────────────────────────────────────┐
│ Pretraining │
│ Objective: Next-token prediction (causal LM) │
│ │
│ Inputs: │
│ • Architecture (N) │
│ • Data mixture (D tokens) │
│ • Hyperparameters from scaling laws │
│ │
│ Infrastructure: │
│ • Data / Tensor / Pipeline / Expert parallelism │
│ • FSDP / ZeRO / rematerialization │
│ │
│ Output: Base model checkpoint │
└────────────────────────┬───────────────────────────────────┘
│
▼
Base LLM
(powerful but unaligned)
═══════════════════════════════════════════════════════════════════════════════════════
5. POST-TRAINING
═══════════════════════════════════════════════════════════════════════════════════════
┌────────────────────────────────────────────────────────────┐
│ Post-training Pipeline │
│ │
│ Stage A: Supervised Fine-Tuning (SFT) │
│ • Instruction / conversation / tool-use data │
│ • Turns base model into instruction follower │
│ │
│ Stage B: Preference Alignment │
│ • RLHF (PPO / RLOO) │
│ • DPO / IPO / KTO / SimPO │
│ • Constitutional AI / RLAIF │
│ │
│ Stage C: Advanced / Reasoning (optional but common) │
│ • Process / Outcome reward models │
│ • Reinforcement Learning on verifiable tasks │
│ • Long-CoT / search / tool-use training │
│ │
│ Stage D: Safety & Specialization │
│ • Refusal training, red-teaming │
│ • Domain adaptation / continued pretraining │
└────────────────────────┬───────────────────────────────────┘
│
▼
Aligned / Capable LLM
(ready for evaluation & deployment)
═══════════════════════════════════════════════════════════════════════════════════════
FEEDBACK LOOPS
═══════════════════════════════════════════════════════════════════════════════════════
• Evaluation results → improve data mix / architecture / post-training recipes
• New scaling laws → re-allocate future compute
• Inference cost awareness → prefer smaller, over-trained models
Key Relationships
| Component | Influences | Is Influenced By |
|---|---|---|
| Data | Quality of base model & final capability | Scaling laws (how much D needed) |
| Architecture | Efficiency, max context, capabilities | Scaling laws + hardware constraints |
| Scaling Laws | Optimal N, D, LR, batch size | Empirical experiments on smaller models |
| Post-training | Final usefulness, safety, reasoning | Quality of base model + data quality |
This diagram captures the modern production LLM pipeline used by labs training frontier models (Llama, DeepSeek, Qwen, Claude-style systems, etc.). Scaling laws act as the “control plane” that decides how much data and how large a model to train, while post-training turns the resulting base model into a useful product.
Top comments (0)