Understanding LLM Lifecycle - For Infra admins - Beginner

Shameer Sh — Tue, 18 Aug 2026 09:28:13 +0000

This LLM life cycle is an easy reading for an Infra admin comparing to Deployment life cycle of an OS.

1. Pretraining — building the golden image
This is where the base model learns general language understanding from massive datasets — expensive, done once by the model provider (Cohere, Meta, OpenAI). Think of this like building a golden Windows Server image: a huge, centralized effort to bake in a broad, general-purpose baseline that many teams will later build on top of. You don't do this yourself — you consume it, the same way you don't build your own base OS image from source, you pull the vendor-provided baseline.

2. Fine-Tuning — customizing the image for your environment
This is where you adapt that general-purpose base model to perform well on a specific task using your own labeled data. Directly analogous to customizing a golden image for a specific business unit — installing department-specific software, applying tailored GPOs, baking in your org's configuration standards. The deck's distinction between Vanilla (retrain most of the model's weights — slow, expensive) vs. T-Few (only touch a tiny ~0.01% slice of additional layers — fast, cheap) maps almost exactly to full reimage vs. hotfix/patch: Vanilla is like rebuilding the entire image from scratch every time you need a change; T-Few is like applying a targeted config/registry patch on top of the existing image without touching the base.

3. Prompt Engineering — runtime configuration, no rebuild needed
This is adjusting how you ask the model to behave — instructions, examples, formatting — without touching the model's underlying weights at all. This is like adjusting a config file, environment variable, or GPO setting at the client/session level rather than rebuilding or repatching the image. Fast, reversible, zero deployment risk — which is exactly why the course's own guidance ("start with a simple prompt" before reaching for RAG or fine-tuning) mirrors the sysadmin instinct of always trying the cheap, reversible fix before a heavier one.

4. Evaluation and Iteration — UAT and health checks before wider rollout
This is where you test the model's output — using metrics like accuracy (how many predictions were right) and loss (how wrong the wrong ones were, so it's more like error severity than a pass/fail count) — against a validation dataset, and iterate: adjust hyperparameters, retrain if needed. Think UAT/pilot testing before a full production rollout, plus ongoing benchmarking — similar to how you'd validate a patched image in a staging environment before pushing it fleet-wide, using defined KPIs rather than just "it seems to work."

5. Deployment and Ongoing Maintenance — go-live and lifecycle management
Deploy the model (or fine-tuned version) for real inference traffic, then monitor performance and collect feedback for future fine-tuning cycles. This is your production rollout + patch/maintenance cycle.

Dedicated AI Clusters (Hosting vs. Fine-tuning) — think resource pools/clusters in vCenter: a fine-tuning cluster needs more GPU "horsepower" (2 units) than a hosting cluster (1 unit), the same way a build/compile server needs more compute than a server just serving traffic.
One base model endpoint + multiple custom model endpoints sharing the same GPU cluster — this is exactly like multiple VMs sharing a hypervisor host's resources, or several App Pools sharing an IIS server — efficient because they share a common "base" and only the deltas differ.
GPU memory reload overhead when switching between unrelated models ** — this is your **vMotion / cold-migration overhead intuition: moving between things that share the same base is cheap; switching to something fundamentally different means a full reload, which costs time and resources.

The one-line summary for your notes: Pretraining = build the golden image (someone else does this) → Fine-tuning = customize the image for your org → Prompt Engineering = runtime config tweaks → Evaluation = UAT/benchmarking → Deployment = go-live + ongoing patch/monitor cycle. Same lifecycle discipline you already run for infrastructure, just applied to a model instead of a server fleet.

RAG Framework with the Infra Lens

Shameer Sh — Tue, 18 Aug 2026 08:56:27 +0000

This is for our Infra Admins who would be more interested in understanding RAG (Retrieval-Augmented Generation) with Infra lens.

RAG Framework — Retriever, Ranker, LLM

Think of this as a three-tier help-desk / ticketing search system:

Retriever — sources relevant information from a large corpus or database using retrieval techniques. This is your search index across your runbooks, KB articles, and past ticket resolutions — like querying your AD/VMware documentation repository for anything that might match the current issue. It casts a wide net, pulling anything plausibly relevant.
Ranker — evaluates and prioritizes what the retriever pulled back, ensuring the LLM gets the most pertinent, high-quality input. This is exactly like triaging search results by relevance before escalating — you don't hand a level-1 tech every KB article that mentions "DNS," you rank by how closely each one matches the actual symptom.
LLM — generates the final human-like response using the ranked, retrieved information plus the original query, aiming for a response that's factually accurate, coherent, and grounded. This is your senior engineer writing up the final resolution note, synthesizing the best-ranked reference material into a clear answer — not inventing one from memory.

RAG Pipeline — the three stages that make this run (Ingestion → Retrieval → Generation): Ingestion is your one-time (or scheduled) documentation import job — documents get chunked, embedded, and indexed into a database, much like you'd batch-import and index a KB into a searchable system. Retrieval is the live query time lookup — a user's question gets matched against that index to pull top-K results. Generation is the final answer synthesis — those top results get handed to the LLM to produce the response.

RAG Techniques — RAG Sequence vs. RAG Token

RAG Sequence: retrieve documents once for the whole query, then generate one cohesive response using all of them together. Like pulling every relevant runbook for an incident up front, then writing one complete resolution report referencing all of them together.
RAG Token: retrieve fresh documents for each part of the response as it's being generated, building the answer incrementally. Like looking something up mid-sentence while writing a report — checking a different reference for each paragraph as you go, rather than gathering everything first. More flexible, but more overhead per response.

Types of RAG — Naive, Advanced, Agentic (this is really a maturity ladder, and it maps almost exactly onto how monitoring/automation tooling matures in an infra team):

Naive RAG — embed the query, grab the top-K matches once, stuff them in the prompt, generate. Simple but brittle if the first search misses. This is like a basic keyword search against your KB with no re-checking — if the search terms don't match well, you get a bad answer with no safety net.
Advanced RAG — adds query rewriting, re-ranking, and better chunking before generation. This is like adding synonym matching, tagging, and a relevance-scoring layer to your ticketing search — you're improving the quality of what gets pulled before anyone acts on it.
Agentic RAG — the LLM itself plans its own retrieval: deciding what to search for, issuing multiple/iterative queries, calling tools, and judging whether it has enough context before answering. This is the leap from a static runbook lookup to a self-driving diagnostic agent — like a monitoring system that doesn't just alert once but iteratively checks multiple data sources (event logs, performance counters, AD replication status) on its own before concluding root cause.

One practical connection if you ever build a "chatbot over your AD/VMware runbooks" idea from earlier, you'd start with Naive RAG to prove it works, then move to Advanced RAG once you notice bad retrievals (mismatched KB articles), and only reach for Agentic RAG if the questions genuinely require multi-step investigation rather than a single lookup — same incremental-rollout instinct you'd apply to any new tooling in production.

DEV Community: Shameer Sh

Understanding LLM Lifecycle - For Infra admins - Beginner

RAG Framework with the Infra Lens