DEV Community: Ege Pakten

NVIDIA DGX Spark vs. Mac Studio M5 Ultra: Which One Should You Actually Buy to Run AI Locally?

Ege Pakten — Thu, 03 Sep 2026 14:19:40 +0000

Two machines just became real purchase options for anyone who wants to run AI models on their own hardware instead of renting time on someone else's cloud servers: NVIDIA's DGX Spark and Apple's new Mac Studio with the M5 Ultra chip. Both get marketed with big numbers. Neither marketing page tells you which one actually fits what you're trying to do. This article works through the real differences — in plain terms, but without skipping the technical details that actually decide which machine is right for you.

The Machines, Side by Side

NVIDIA DGX Spark: $4,699. 128GB of memory, shared between the CPU and GPU. Memory bandwidth of about 273GB/s. It has been shipping and available to buy since late 2025.

Mac Studio (M5 Ultra, 36-core CPU / 80-core GPU, 256GB): $10,799. 256GB of memory. Memory bandwidth of about 1.2TB/s (roughly 1,200GB/s). Ships September 22, 2026. A 512GB version of the same chip exists but won't be available until late October.

Put simply: the Mac has twice the memory, more than four times the memory bandwidth, and costs more than twice as much. But "more numbers" doesn't mean "better for you" — these two machines are built for different jobs, and the numbers only make sense once you understand what each one is actually for.

Two Different Kinds of Machines

The Mac Studio is a complete workstation. You sit at it. Your files, your applications, and the AI model all live on the same box, and you interact with it directly like any computer on your desk.

The DGX Spark is not meant to be sat at. It's designed to live somewhere else — a shelf, a closet, a server rack — and be reached over your network. A common setup is to keep using your regular laptop for everyday work, and connect to the Spark remotely (tools like Tailscale make this simple, creating a private network connection to it from anywhere) to send it AI requests the way you'd call an API. Your daily computer doesn't change at all; the Spark just becomes a private AI server you access when you need it.

This distinction matters more than the spec sheet does. If you want one machine that does everything, including running AI models, the Mac is the obvious fit. If you already have a laptop you're happy with and just want a dedicated AI engine sitting quietly in the background, the Spark is the better shape for that job.

Why Memory Bandwidth Is the Bottleneck That Actually Matters

To understand why the bandwidth gap (273GB/s vs. 1.2TB/s) matters so much, it helps to know what an AI model is actually doing on this hardware, and what "memory bandwidth" means here.

An AI language model is essentially a very large set of numbers (its "parameters" — think of them as everything the model learned during training) stored in memory. To generate a response, the machine has to repeatedly read those numbers out of memory and run calculations with them. Memory bandwidth is simply how fast the hardware can move that data from memory to the processor doing the math. A slower pipe means the processor spends more time waiting for data and less time actually computing — the same way a fast chef with a narrow doorway to the pantry still can't cook any faster than ingredients can be carried through it.

Running an AI model on your own machine happens in two distinct phases, and they behave very differently:

Prefill is when the machine reads and processes your entire prompt — everything you typed — before it starts responding. This step can be split across many parallel calculations at once, so it's mostly limited by raw processing power rather than memory bandwidth. Both machines handle this reasonably well.

Decode is when the machine generates the actual response, one word (technically, one "token" — a chunk of text, sometimes a whole word, sometimes a fragment) at a time. Each new token has to be produced by reading through the model's parameters again, in sequence, one step at a time. This step cannot be parallelized the same way prefill can, which means it's almost entirely limited by memory bandwidth — how fast that data can be pulled from memory, over and over, for every single word of the response.

This is exactly why the DGX Spark's low bandwidth (273GB/s) shows up as sluggish response generation: it can prepare for an answer quickly, but producing that answer word by word is bottlenecked by how much data it can move per second. The Mac Studio's 1.2TB/s bandwidth means it can sustain much faster token-by-token generation, which is the part of using an AI model you actually sit and wait for.

Both Machines Still Crush a Normal PC on Raw Computing Power

None of this should be read as "bandwidth is everything and raw compute doesn't matter" — it very much does, just not for the specific step described above. It's worth being clear about how much more computing power both of these machines have compared to an ordinary computer, because that gap is enormous and it shows up everywhere except that one narrow bottleneck.

A typical PC without a dedicated AI-capable GPU is doing this kind of math on a handful of general-purpose CPU cores, built for everyday tasks like running apps and browsing the web, not for crunching millions of parallel calculations at once. The DGX Spark's Blackwell chip, by contrast, is rated for up to 1 PFLOP (1,000 trillion calculations per second) of AI compute at the precision level typically used for running these models, backed by a 20-core processor and dedicated tensor cores built specifically for this kind of math. The Mac Studio's M5 Ultra pairs a 36-core CPU with an 80-core GPU and a 32-core Neural Engine, and Apple states it delivers up to 4.5 times the AI compute of the previous M3 Ultra generation. Both numbers are on a completely different scale from what a normal desktop or laptop can do.

That raw compute advantage is exactly why both machines can handle tasks a normal PC simply can't do at a usable speed at all: processing your prompt (the prefill step, which — as noted above — is parallel-friendly and genuinely compute-bound), running image or video generation, handling several AI requests from different people at once, or fine-tuning a model on your own data. In all of those cases, having far more cores and far higher raw throughput than a regular computer translates directly into faster, more capable performance — this is the everyday sense in which these machines are dramatically faster than a normal PC, and it's true for both of them.

The nuance is specifically about one narrow, sequential step: generating a chat response one token at a time from a model already sitting in memory. That step doesn't get to spread its work across all those extra cores, because each token depends on the one before it — so it's gated by how fast data can be pulled from memory, not by how many cores are sitting idle waiting for that data to arrive. That's the one case where a chip with less raw compute but far more memory bandwidth (like the Mac Studio here) can end up feeling faster than a chip with more raw compute but less bandwidth. Everywhere else, more cores and higher compute really do mean better performance — over a normal PC, and often over each other.

The Software Problem Nobody Mentions: CUDA vs. MLX

Raw hardware specs are only half the story. Software support is the other half, and it's where NVIDIA has a decades-long head start.

CUDA is NVIDIA's software platform for running heavy computations on its GPUs, and it has been the default foundation for machine learning for over a decade. The overwhelming majority of AI tools, libraries, and frameworks (PyTorch chief among them) were built and optimized for CUDA first. If you're using an NVIDIA machine like the DGX Spark, you're plugging into an enormous, mature ecosystem that "just works" with almost anything you download.

Apple Silicon, including the M5 Ultra, doesn't run CUDA at all. Instead, Apple has built its own framework called MLX, designed specifically to take advantage of Apple's unified memory architecture. MLX is genuinely good, and it handles a large share of everyday AI tasks (particularly running, or "inferencing," existing models) perfectly well. But it's a smaller, newer ecosystem — plenty of cutting-edge research code and tools are released for CUDA first, or only for CUDA, and never get an MLX version at all.

Practically, this means: if you want to run a well-established model for everyday use, the Mac's software situation is fine. If you want to experiment with the newest research code, fine-tune models yourself, or use tools that assume an NVIDIA GPU (which is most of them), the DGX Spark's CUDA compatibility gives it a real, practical advantage that the spec sheet alone doesn't show.

How a Machine This Small Can Run a "Huge" Model

Both of these machines are, by AI infrastructure standards, tiny. Yet both are being marketed as able to run models with hundreds of billions of parameters — sizes that used to require racks of datacenter GPUs. The reason this works is an architectural choice called Mixture of Experts (MoE), and it's worth understanding because it's the actual enabling technology behind both machines' big claims.

A traditional ("dense") AI model activates every single one of its parameters for every word it processes. If the model has 200 billion parameters, generating one word means running calculations across all 200 billion of them — regardless of how simple or complex that particular word is to predict.

A Mixture of Experts model doesn't work that way. Instead of one giant network, it's built from many smaller sub-networks (called "experts"), and a routing mechanism decides, word by word, which small subset of those experts actually needs to be used. A model might have 200 billion parameters in total, but only activate around 20 billion of them for any given word — the rest sit unused for that step. Real-world examples of this design include Mixtral and DeepSeek's models.

This is exactly why a single consumer-scale machine can handle a "200-billion-parameter" model at usable speed: you get the knowledge and capability that comes from having a large total model, but the actual compute cost per word is much closer to what a much smaller model would require. Without MoE, neither the Spark nor the Mac Studio would stand a chance of running models this large.

A Common Misunderstanding: Buying Two DGX Sparks Doesn't Make It Faster

One thing worth clearing up, because it trips people up: if you buy two DGX Spark units and connect them together (NVIDIA sells them with networking hardware specifically for this), you might assume you're doubling your speed, the way adding a second worker can speed up a job. That's not what happens.

Linking two Sparks together lets you pool their memory — 128GB plus 128GB gives you 256GB total, which means you can now fit and run bigger models that wouldn't fit on a single unit at all (up to roughly 405 billion parameters, instead of around 200 billion on one machine). That part is a genuine benefit.

What it does not do is make each response come back faster. The connection between the two units is far slower than the memory inside a single one — roughly ten times slower, in practical terms. Since generating each word of a response (the "decode" step described earlier) already depends on reading through the model's data as fast as possible, adding a second machine that data has to travel across a slower connection to reach actually adds waiting time to every step, rather than removing it. So a model spread across two linked Sparks will typically produce each word more slowly than a model that fits entirely on one Spark by itself.

The honest way to think about it: connecting two Sparks buys you room for a bigger model, or the ability to handle more requests at once — not a faster single response. If speed for a single conversation is what you care about most, adding a second Spark won't get you there; you'd need a machine with genuinely higher memory bandwidth, like the Mac Studio, or a different architecture entirely.

The Real Alternative: Renting a Cloud GPU

It's worth being honest about what these machines are actually competing with: cloud GPU rental. Instead of buying either machine, you can rent GPU time from a cloud provider by the hour, running your AI workloads on someone else's hardware and paying only while you're using it.

The problem is that this bill scales with usage — the longer a job runs and the more often you run it, the more it costs, and for anyone running AI workloads regularly (a team, a product with real users, a daily research workflow) that adds up fast and never stops. A machine sitting in your own building, whether it's a Spark tucked in a closet or a Mac Studio on a desk, is that same GPU capability without a running meter attached. You pay once, up front, and then it's yours.

Who Should Actually Buy Which

If you want one computer that is your daily workstation and also runs AI models directly, at the fastest response speed available in this category, and you're willing to pay more for the full package — buy the Mac Studio M5 Ultra.

If you already have a machine you use daily, you want a dedicated AI engine you can reach over your network like a private API, you care about compatibility with the widest possible range of AI software and research tools, and you want to spend less up front — the DGX Spark is the better fit.

Why This Matters for a Business

For a company evaluating either option, the decision usually comes down to two things: data control and predictable cost. Running a model on hardware you own means client data, internal documents, or proprietary information never has to leave your network to reach an outside AI provider — a meaningful advantage for legal, healthcare, financial, or any business handling sensitive data under compliance requirements. It also converts an open-ended, usage-based cloud bill into a fixed, one-time hardware cost, which is far easier to budget for a team running AI workloads continuously rather than occasionally.

A small business or solo developer testing AI features on a budget will likely get more mileage out of the DGX Spark's lower price and CUDA compatibility. A team that wants a single, fast, all-in-one machine for both regular work and heavier AI tasks — and can absorb the higher upfront cost — is better served by the Mac Studio.

Neither machine is a universal answer. They're built for different shapes of use, and now, for the first time, both are things you can actually put in a shopping cart rather than numbers on a slide.

Which one are you buying? Or do you already have one?

More can be read on our website:
https://www.kmcpsolutions.co.uk/news

When Should You Retrain Your ML Model? A New Study Says Most Teams Are Solving the Wrong Problem

Ege Pakten — Tue, 25 Aug 2026 15:14:15 +0000

Every production ML model starts degrading the moment it ships. Customer behavior shifts, fraud patterns evolve, market conditions change — the technical term is concept drift, and it's the reason a model that scored great in testing quietly gets worse in the real world, often without anyone noticing until revenue or accuracy numbers show it.

The obvious question teams ask is: when should we retrain? A recent study, "When to Retrain: An Empirical Study of Retraining Policies for Streaming ML Under Concept Drift, Budget, and Latency Constraints" by Sawan Dasari, ran nearly 4,000 experiments to actually answer that question — and the answer isn't the one most teams are optimizing for.

What the Study Actually Tested

The paper compares four approaches to keeping a deployed model current:

Doing nothing — the model is trained once and never touched again (the baseline).
Periodic retraining — retrain on a fixed schedule (every day, every week, every N samples), regardless of whether performance has actually dropped.
Error-threshold triggering — watch the model's live error rate, and retrain only once it crosses a defined threshold.
Drift-detection with ADWIN — use a statistical drift-detection algorithm (ADaptive WINdowing) that watches the incoming data stream and signals a retrain the moment it detects a meaningful shift in the underlying distribution.

These represent the real menu of options any team building a production ML system has to choose from, and each has an intuitive appeal: periodic is simple, error-threshold reacts to actual damage, drift-detection is the "smart" option that reacts to the cause rather than the symptom.

The Finding That Changes the Conversation

Here's the part worth sitting with: which retraining policy you pick barely matters if your model can already learn incrementally. When a model supports incremental learning — updating itself continuously from new samples as they arrive, rather than being retrained from scratch as a discrete event — the choice between periodic, error-threshold, and drift-detection produces only negligible performance differences.

But take incremental learning away, and the story flips completely. Without it, the choice of retraining policy causes accuracy swings of 15 to 55 percentage points. In other words: for models that only get updated in occasional, discrete retraining events (which describes the overwhelming majority of production ML systems today), when you pull the trigger is one of the most consequential engineering decisions in the entire system — far more consequential than most teams treat it.

A second finding cuts against the instinct to build the "smartest" possible system: under both abrupt and gradual drift, simple periodic retraining outperformed the more sophisticated reactive approaches (error-threshold and drift-detection). The fancier, more responsive-sounding strategy didn't win. A dumb, predictable schedule did.

A third finding is the kind of thing that only shows up when you actually measure a full production pipeline rather than a clean benchmark: there's a latency-budget interaction that can silently cut your effective retraining capacity in half. If retraining a model takes real wall-clock time and you're operating under a compute or cost budget, the two constraints compound in a way that isn't obvious until you model them together — you can end up with far less real retraining headroom than your budget alone would suggest.

Why This Matters Far Beyond the Lab

This isn't an academic curiosity — it's directly relevant to any company running a model against live, changing data, which today is most companies with any kind of ML in production.

Fraud and abuse detection. Attackers actively adapt to whatever your model currently catches, making this one of the fastest-drifting domains that exists. The finding that periodic retraining can outperform "smart" drift-triggered retraining is a genuinely useful, cost-saving insight here — it suggests fraud teams may get more reliability from a predictable retraining cadence than from an elaborate drift-detection pipeline they still have to build and maintain.

Demand forecasting and dynamic pricing. Retail, travel, and logistics companies retrain forecasting models against seasonal and macroeconomic shifts. The latency-budget interaction is a direct warning: if your retraining job takes six hours and your infrastructure budget was sized without accounting for that, you may be retraining far less often than your budget was supposed to allow — and no one will notice until forecast accuracy quietly slips.

Recommendation and personalization engines. User taste and trending content shift constantly. Since incremental learning is what makes retraining-policy choice mostly irrelevant, this is a strong argument for prioritizing an incremental-learning architecture (models that update continuously from a stream of user interactions) over building an increasingly elaborate drift-detection layer bolted onto a static model.

Customer churn and credit risk models. Behavior patterns shift with the economy, product changes, and competitor actions. The 15-55 point swing found here is the difference between a model that's still doing its job and one that's silently misclassifying a meaningful share of customers — with real financial consequences riding on which retraining policy was picked, if incremental learning isn't in place.

Support ticket routing and internal AI assistants. The language customers use, the topics they raise, and the products they ask about all shift over time. Teams running these systems on a "retrain occasionally when someone notices it's gotten worse" basis are, per this study, choosing close to the worst-performing strategy available.

The Practical Takeaway for Teams Building ML in Production

If you're deciding how to handle model staleness in a live system, this research points to a concrete priority order: first, find out whether your model architecture can realistically support incremental learning — if it can, invest there before anything else, because it makes the retraining-policy question nearly moot. If incremental learning genuinely isn't feasible for your setup, don't assume the most sophisticated drift-detection system is the safest choice; a disciplined periodic schedule is a strong, low-complexity default that this study shows holding up well under real drift conditions. And regardless of which policy you pick, model your retraining latency and compute budget together, not separately — treating them as independent numbers is exactly how teams end up with half the retraining capacity they thought they had.

Common Misconceptions

"The fanciest drift-detection algorithm is always the safest bet." Not according to this data — simple periodic retraining beat reactive, drift-triggered approaches under both abrupt and gradual drift.

"If I have a retraining budget, I know how often I can retrain." Not necessarily. Latency and budget interact, and that interaction can quietly halve your real capacity if you haven't modeled it explicitly.

"Retraining policy is the main lever for handling drift." For models with incremental learning, it barely matters. The bigger lever is whether the model can learn continuously in the first place.

Concept drift isn't a rare edge case — it's the default condition of any model touching real-world, real-time data. This study is a useful reminder that the engineering decision worth the most scrutiny isn't always the flashiest one; sometimes it's whether you've built the right foundation (incremental learning) before optimizing the policy sitting on top of it.

Sources:

When to Retrain: An Empirical Study of Retraining Policies for Streaming ML Under Concept Drift, Budget, and Latency Constraints (arXiv:2608.19488)

Training vs. RAG: What's Actually Different (and Why AI Keeps Telling You to Use RAG)

Ege Pakten — Sat, 22 Aug 2026 19:35:09 +0000

"Can we just train the model on this data together?" is one of the most common things people ask an AI assistant. And almost every time, the answer comes back: "Let's use RAG instead." That answer sounds like a dodge if you don't know what's actually happening underneath it. It isn't. Training and RAG solve genuinely different problems, and once you see the mechanism behind each one, the recommendation makes complete sense.

This article breaks down what training actually does to a model, what RAG actually does (spoiler: it doesn't touch the model at all), and why the two aren't interchangeable — plus when fine-tuning genuinely is the right call.

What Training Actually Does

A language model is, at its core, a huge set of numbers called weights — often billions of them. Training is the process of adjusting those weights so the model gets better at predicting the next piece of text.

There are two stages worth separating:

Pretraining is where a model learns language, reasoning patterns, and general world knowledge by processing enormous amounts of text and repeatedly adjusting its weights based on how wrong its predictions were. This is the expensive, months-long process that produces a base model like GPT or Claude's underlying model. It requires massive compute clusters and datasets measured in trillions of tokens.

Fine-tuning takes an already-pretrained model and continues that same weight-adjustment process on a smaller, more specific dataset. If you "train the model on your company's data," this is technically what you'd be doing: showing the model your data and further updating its weights so its behavior shifts toward what that data represents.

Here's the part that surprises most people: fine-tuning doesn't work well as a way to inject facts. Gradient descent (the math behind weight adjustment) is built to learn patterns and behaviors — tone, structure, task style, domain vocabulary — not to reliably memorize discrete, specific facts the way a database does. Feed a model 200 pages of your internal documentation through fine-tuning, and it will pick up your writing style and general vocabulary far more reliably than it will remember the exact number in table 14 on page 87. It can also just as easily forget parts of what it already knew — a problem called catastrophic forgetting — or overfit and start reciting fragments of your data out of context.

Fine-tuning also has no built-in way to say where an answer came from. Once the weights are adjusted, the model's response is fused with everything else it knows. It can't point back to "this came from your document" because, mechanically, nothing is being "looked up" anymore — it's just been folded into the same statistical blend as the rest of its training.

And practically: fine-tuning needs real infrastructure, curated example data (usually well-formed prompt/response pairs, not raw documents), engineering time, and repeated retraining every time you want to update the information. If your documents change next week, you're fine-tuning again.

What RAG Actually Does

RAG (Retrieval-Augmented Generation) does not touch the model's weights at all. It leaves the model completely untouched and instead changes what goes into the prompt at the moment you ask a question.

The mechanism: your documents get broken into chunks and converted into numerical representations called embeddings, which capture meaning rather than exact wording. These embeddings are stored in a vector database. When you ask a question, that question also gets converted into an embedding, and the system searches the database for the chunks whose meaning is closest to your question. Those retrieved chunks get inserted directly into the prompt, alongside your question, before the model ever generates an answer.

So the model isn't recalling your documents from memory — it's reading them fresh, every single time, because you handed them to it in the prompt. That's why RAG answers can cite sources: the system knows exactly which chunk of text the answer came from, because that chunk was passed in explicitly.

This is also why RAG is so much easier to keep current. Add a new document to the database, and it's immediately part of what can be retrieved — no retraining, no weight updates, no waiting. Delete an outdated document, and it stops showing up. The model's core reasoning ability never changes; only the material it's given to reason over changes.

The Actual Difference, Side by Side

	Fine-tuning	RAG
What changes	The model's weights	The prompt's content
Good for	Style, tone, output format, task behavior	Facts, up-to-date information, source citation
Data needed	Curated example pairs, often thousands	Your raw documents, as-is
Update cost	Retrain from scratch (or continue training)	Add/remove a document
Can cite sources	No	Yes
Compute cost	High, ongoing	Low, mostly at query time
Risk	Catastrophic forgetting, unreliable fact recall	Answers limited by what's retrieved

Why "Train the Model on My Data" Usually Means RAG

When someone asks to train a model on their data, what they almost always actually want is: "I want the model to answer questions accurately using my documents, and ideally tell me where the answer came from." That specific goal — accurate, current, traceable answers grounded in specific documents — is precisely the problem RAG was designed to solve. Fine-tuning was never designed to solve it; it was designed to change how a model behaves, not to serve as a substitute for a database.

This is also a data-volume issue in practice. Fine-tuning that reliably improves behavior typically needs thousands of well-constructed examples. Most people asking to "train on this" have a folder of PDFs or a wiki — exactly the raw material RAG consumes directly, with no curation step required.

When Fine-Tuning Is Actually the Right Call

Fine-tuning earns its place when the goal is behavioral, not factual:

You need the model to consistently output a specific format (a strict JSON schema, a particular report structure) across thousands of requests.
You need it to adopt a very specific tone or domain style that's hard to enforce through prompting alone.
You've already built a RAG system, it works, and now you want to bake its typical response pattern into the model to cut down prompt length and cost at scale.
You're teaching a narrow skill (like parsing a specific legal document format) where the pattern matters more than any individual fact.

In real production systems, it's common to combine both: RAG supplies the facts, fine-tuning shapes how the model uses them.

Common Misconceptions

"If I fine-tune on my documents, the model will know them forever." Not reliably. It will pick up style and general themes; it won't reliably recall specific facts, and it can't tell you where a fact came from.

"RAG is the cheap, inferior version of real training." It's not a downgrade — it's the correct tool for a different job. Most production systems that need a model to answer from specific, current, citable documents use RAG, including large-scale assistants with access to internal knowledge bases.

"I need huge amounts of data either way." For RAG, you need your documents, unmodified. For fine-tuning, you need curated behavioral examples, which is a completely different (and usually much smaller and harder-to-produce) kind of dataset.

What to Do in Practice

If the goal is "answer questions based on my documents, accurately, with the ability to show sources" — start with RAG. Set up an embedding model, a vector database, and a retrieval step in front of your existing model. Only consider fine-tuning afterward, and only if there's a specific behavioral or formatting problem that prompting and retrieval genuinely can't solve.

The short version: training changes what a model is. RAG changes what a model sees at the moment it answers. Most requests to "teach the model your data" are really requests for the second thing.

What Is AI Inference? The Difference Between Training a Model and Actually Using It

Ege Pakten — Sat, 22 Aug 2026 17:57:20 +0000

You type a prompt into ChatGPT. Two seconds later, a full answer appears. Nothing about the model changed during those two seconds — it didn't learn anything, it didn't get smarter, it didn't update itself based on your question. So what actually happened in that gap?

That's inference. It's the least talked-about part of AI, and it's also the part you interact with every single time you use an AI product. Training gets all the headlines — the huge datasets, the massive GPU clusters, the eye-watering costs. Inference is the quiet workhorse that runs millions of times a day, for every user, every request, forever.

In this post, you'll learn what inference actually is, how it's different from training, what happens under the hood when a model generates a response, and the misconceptions that trip up almost everyone when they first hear the term.

The Restaurant Kitchen Analogy

Imagine a chef going through culinary school. For years, they practice recipes, taste thousands of dishes, adjust seasoning, throw out what doesn't work, and slowly develop a set of recipes they trust. This process is slow, expensive, and happens in a controlled environment — a training kitchen, not a real restaurant. Nobody is being served yet.

That's training. It's the phase where a model looks at huge amounts of data, adjusts its internal "recipe" (its weights, the numbers that define how it behaves), and gradually gets better at a task.

Now the chef opens a restaurant. A customer orders the dish. The chef doesn't re-invent the recipe or run new experiments — they use the recipe they already perfected, and they cook it, fast, consistently, correctly, over and over, for every customer who walks in. No more learning happens during dinner service. The recipe book is closed and finalized.

That's inference. It's the model using what it already learned to produce an answer for you, right now, without changing anything about itself in the process.

How It Actually Works

Once training finishes, a model's weights are frozen — the recipe book is locked. Inference is simply feeding new input through that frozen model and reading out what comes on the other end. Engineers call this a forward pass: your input goes in one end, gets transformed step by step through the model's layers, and a result comes out the other end. There's no backward step, no correcting mistakes, no updating the recipe. Just: input in, output out.

For a chatbot, there's an extra wrinkle worth knowing. Large language models don't generate a whole response in one shot — they generate it one token at a time (a token is roughly a word or a word-fragment). It's like the chef plating a dish one element at a time: place the protein, step back, look at what's on the plate so far, decide what goes next, add the sauce, step back again, decide again, and keep going until the dish looks done. Each new token the model produces gets fed back in as part of the input for producing the next one. That's why longer answers take visibly longer to appear — the model is doing this "look at everything so far, decide the next piece" step again and again.

A Concrete Example

Say you ask a model: "Write a haiku about the ocean."

First, your sentence gets chopped into tokens — this is like a prep cook breaking ingredients down into usable pieces before cooking starts. Those tokens get converted into numbers the model can process, and pushed through the frozen network in a forward pass. The output isn't a word — it's a probability distribution over every possible next token, essentially the model saying "here's how likely each possible next word is, given everything so far." The system picks one (usually the most likely, sometimes with a bit of controlled randomness), appends it to the growing response, and repeats the whole process to pick the next token. This continues until the model produces an end-of-response signal or hits a length limit.

Every one of those steps costs compute time — this is where terms like latency (time to get an answer), throughput (how many requests a system can serve per second), and cost per token come from. Techniques you'll hear about in production AI — quantization (using smaller, less precise numbers to run faster, like using a simplified prep technique that's a little less exact but much quicker), batching (cooking several orders together instead of one at a time), and caching (keeping frequently-used prep work ready instead of redoing it from scratch) — all exist specifically to make inference cheaper and faster. None of them touch training at all.

Common Misconceptions

"Using a model teaches it more." Not by default. A production model doesn't update its weights based on your conversation. The recipe book stays closed. (A company might later use logged conversations to train a future version — but that's a separate training run, not something happening live while you chat.)

"A bigger model is always the better choice." Bigger models often produce better answers, but they're also slower and more expensive to run at inference time. In production, teams frequently choose a smaller or compressed model on purpose, because a slightly-less-brilliant answer in 200 milliseconds beats a slightly-better one in four seconds.

"Once training is done, the hard part is over." For any AI product with real users, inference — not training — is usually where the bulk of ongoing computing cost goes. Training happens occasionally. Inference happens every single time someone hits "send."

What to Try Next

If you want to actually feel the difference between model sizes at inference time, install Ollama or LM Studio and run a small open model locally (something like a 1–3 billion parameter model). Watch the tokens stream in one at a time, then try a larger model on the same machine and notice how much slower it feels. That speed difference, multiplied across millions of users, is the entire reason inference optimization is its own engineering discipline.

Training teaches a model what to know. Inference is everything that happens after — every time it actually gets used. Once you can see that difference, a lot of how AI products are actually built (and why they're priced the way they are) starts making a lot more sense.

If this helped you understand inference, drop a reaction.

Overfitting vs Underfitting: Why Your ML Model Acts Like a Student Who Either Memorizes or Doesn't Study at All

Ege Pakten — Fri, 21 Aug 2026 18:55:06 +0000

You trained a model. It scored 99% on your training data. You celebrate. Then it sees real-world data and falls flat on its face. What happened?

You just met overfitting — one of the two most common ways a machine learning model can go wrong. Its evil twin is underfitting. Together, they're responsible for more failed ML projects than bad data and wrong algorithms combined.

In this post, you'll learn what overfitting and underfitting actually are (through an analogy that sticks), how to spot each one, and — most importantly — how to fix them.

The Exam Analogy

Imagine two students preparing for a history exam.

Student A memorizes the textbook word for word. Every date, every footnote, every example question and its exact answer. On a practice test using those same questions, Student A scores 100%. But when the real exam asks a question phrased slightly differently — even about the same event — Student A freezes. They memorized the answers, not the concepts.

Student B glances at the chapter titles the night before. They get the general vibe: "something about a war, some treaties." On the practice test, they score 40%. On the real exam, also 40%. They didn't learn enough to answer anything well.

Now imagine Student C — they read the textbook, understood the key themes, practiced with different types of questions, and can apply what they learned to questions they've never seen before. Practice test: 88%. Real exam: 85%. Not perfect, but consistently solid.

In machine learning: Student A is overfitting. Student B is underfitting. Student C is balanced — and that's what we're aiming for.

What Is Overfitting?

Overfitting happens when your model learns the training data too well. It doesn't just learn the patterns — it memorizes the noise, the outliers, the random quirks that are specific to your training set but don't exist in the real world.

The telltale sign: high performance on training data, poor performance on test/unseen data.

If you plot an overfitting model's predictions, it looks like a line that zigzags wildly through every single data point — including the ones that are just random noise. It's too complex. It's trying too hard.

Why does it happen?

Too little training data. With few examples, the model has no choice but to memorize what's there.
Too complex a model. A neural network with millions of parameters trained on 500 data points will memorize those 500 points perfectly — and learn nothing generalizable.
Training for too long. The longer you train, the more the model starts fitting to noise instead of signal.
Too many features. If you feed the model 200 features for 1,000 samples, it finds spurious correlations that don't hold up in production.

What Is Underfitting?

Underfitting is the opposite problem. Your model is too simple to capture the actual patterns in the data. It hasn't learned enough.

The telltale sign: poor performance on both training AND test data.

An underfitting model draws a straight line through data that clearly curves. It misses the pattern entirely — not because the pattern isn't there, but because the model doesn't have the capacity to see it.

Why does it happen?

Model is too simple. Trying to fit a linear regression to data that has a non-linear relationship.
Not enough features. You're trying to predict house prices but only gave the model the number of bedrooms — ignoring square footage, location, age, condition.
Not trained long enough. You stopped training before the model had a chance to learn.
Too much noise in data. If the data is so noisy that the signal is buried, even a good model will underfit.

How to Tell Which One You Have

Here's the cheat sheet:

Symptom	Training Score	Test Score	Diagnosis
Both scores high and close	92%	89%	Balanced — you're in good shape
Training high, test low	99%	62%	Overfitting — model memorized
Both scores low	55%	52%	Underfitting — model didn't learn

The gap between training and test performance is your key diagnostic tool. A large gap screams overfitting. Uniformly poor scores scream underfitting.

How to Fix Overfitting

1. Get more training data. More diverse examples make it harder for the model to memorize and force it to learn actual patterns. This is the single most effective cure.

2. Reduce model complexity. Use fewer layers, fewer parameters, a simpler architecture. If a decision tree works, don't use a 50-layer neural network.

3. Apply regularization. Techniques like L1 (Lasso) and L2 (Ridge) regularization penalize the model for becoming too complex, keeping weights small and forcing generalization.

4. Use dropout. In neural networks, dropout randomly "turns off" some neurons during training. This prevents the network from relying too heavily on any single neuron — like forcing Student A to study without their favorite highlighter.

5. Apply early stopping. Monitor the model's performance on validation data during training. When validation performance starts getting worse (even though training performance keeps improving), stop. That inflection point is where memorization begins.

6. Clean and preprocess your data. Remove noise, handle outliers, fix inconsistencies. Cleaner data gives the model less garbage to memorize.

How to Fix Underfitting

1. Increase model complexity. Use a more powerful model — go from linear regression to polynomial, from a shallow network to a deeper one.

2. Better feature engineering. Create new features that capture the patterns the model is missing. Maybe "price per square foot" matters more than price and square footage separately.

3. Clean noise from data. Reduce noise so the actual signal becomes learnable.

4. Train longer. Give the model more epochs to learn. Sometimes it just needs more time.

5. Reduce regularization. If you've applied too much regularization, you might be preventing the model from learning even the real patterns. Dial it back.

The Bias-Variance Tradeoff

What you've just learned has a formal name: the bias-variance tradeoff.

High bias = underfitting. The model makes too many assumptions and misses the real pattern.
High variance = overfitting. The model is too sensitive to the specific training data.

You want the sweet spot: low enough bias to capture the pattern, low enough variance to generalize. In practice, you're always balancing these two — pushing one down often pushes the other up.

Think of it like adjusting the focus on a camera. Too blurry (underfitting) and you can't see anything. Too zoomed-in on one pixel (overfitting) and you lose the bigger picture. The sharp image is somewhere in between.

Common Misconceptions

"Higher accuracy is always better." Not if it's only on training data. A model with 99% training accuracy and 60% test accuracy is worse than one with 85% on both. Generalization is what matters.

"More complex models are always better." Complexity without enough data leads straight to overfitting. Start simple, increase complexity only when you have evidence it helps.

"Overfitting only happens with deep learning." Nope. A decision tree with no depth limit will happily overfit a small dataset. Overfitting is a universal ML problem.

What to Try Next

If you're learning ML, here's a quick experiment: take any dataset (Scikit-learn's built-in datasets work great), train a decision tree with no depth limit, and compare training vs test accuracy. Then set max_depth=3 and compare again. You'll see overfitting and the fix in action — in under 10 lines of code.

The best ML models aren't the ones that score highest on training data. They're the ones that perform consistently well on data they've never seen before. Learning to navigate the overfitting-underfitting spectrum is one of the most practical skills in machine learning — and now you know how.

*If this helped you understand the difference, drop a reaction.

The Machine Learning Lifecycle: 10 Steps From Problem to Production (And Why Most Projects Fail at Step 3)

Ege Pakten — Sat, 25 Apr 2026 09:15:35 +0000

Every ML tutorial jumps straight to model training. But in the real world, training is step 7 out of 10 — and the steps before it are where projects succeed or fail. This post walks through the full Machine Learning Lifecycle, from defining your problem to keeping your model healthy in production, with real examples and practical advice at every stage.

The Big Picture

Machine Learning is an iterative and structured process. It's not "throw data at an algorithm and hope for magic." It's a cycle — and most teams loop through it multiple times before they get something that works in production.

Here are the 10 stages:

1. Problem Definition → 2. Data Collection → 3. Data Cleaning & Preprocessing
→ 4. Exploratory Data Analysis (EDA) → 5. Feature Engineering & Selection
→ 6. Model Selection → 7. Model Training → 8. Model Evaluation & Tuning
→ 9. Model Deployment → 10. Monitoring & Maintenance → (back to 1)

Let's go through each one.

1. Problem Definition — "What are we actually solving?"

This is where most failed ML projects go wrong. Before touching any data or code, you need to answer:

What business problem am I solving? Not "I want to use AI," but "I want to reduce customer churn by 15%" or "I want to detect fraudulent transactions in real time."
Is ML even the right tool? Sometimes a simple rule-based system or a SQL query is better. ML is expensive overkill for problems that have clear, deterministic rules.
What does success look like? Define a measurable metric: accuracy, precision, recall, revenue impact, latency requirements.
What type of ML problem is this? Classification? Regression? Clustering? Recommendation? This dictates everything downstream.

The problem definition dictates the type of data you need. If you define the problem wrong, you'll collect the wrong data, build the wrong model, and ship something nobody wanted.

Example: A bank wants to "use AI." That's not a problem definition. "Predict which credit card transactions are fraudulent with less than 0.1% false positive rate and under 200ms latency" — that's a problem definition.

2. Data Collection — "Do we have enough?"

Once you know what you're solving, you need data. This step is about gathering enough high-quality, relevant data to train a model.

Key questions:

Where does the data come from? Internal databases, APIs, web scraping, third-party vendors, public datasets, user-generated content?
How much data do I need? Depends on complexity. A simple classifier might need 1,000 examples. A computer vision model might need 100,000+ labeled images. An LLM needs billions of tokens.
Is the data labeled? For supervised learning, you need labels (the "right answers"). Labeling is often the most expensive and time-consuming part.
Is the data representative? If you train a facial recognition system only on photos of one demographic, it will fail on others. Your data must represent the real-world distribution.

Common pitfalls:

Assuming you have "big data" when you actually have big noise
Not checking for sampling bias
Ignoring data privacy regulations (GDPR, KVKK, HIPAA)
Collecting too many features and not enough samples

3. Data Cleaning and Preprocessing — "Garbage in, garbage out"

This is where you spend 60-80% of your actual project time. Raw data is messy. Always.

What you're doing here:

Handling Missing Values

Some rows have blank fields. Do you fill them with the mean? The median? A prediction? Or drop them entirely?
The right answer depends on why the data is missing. "Random missing" and "systematically missing" require different approaches.

Removing Duplicates

Duplicate records distort your model's understanding of the distribution.

Fixing Inconsistent Data

"New York", "new york", "NY", "N.Y." are the same city but four different strings.
Date formats: "04/20/2026" vs "2026-04-20" vs "20 April 2026"
Units: meters vs feet, Celsius vs Fahrenheit

Handling Outliers

A salary dataset where most values are $40K-$120K but one entry says $99,999,999. Is it real or a typo? Outliers can destroy model performance or provide critical signal — you have to decide.

Data Type Conversions

Categorical variables need encoding (one-hot, label encoding)
Text needs tokenization
Images need resizing, normalization
Dates need feature extraction (day of week, month, holiday flag)

Normalization and Scaling

Features on different scales (age: 0-100, salary: 20,000-500,000) can bias models that use distance calculations. Standard scaling (z-score) or min-max scaling fixes this.

The motto: garbage in, garbage out. No model, no matter how sophisticated, can learn good patterns from bad data.

4. Exploratory Data Analysis (EDA) — "What does the data actually look like?"

Before building any model, you need to understand your data. EDA is about getting the big picture.

What you're looking for:

Distributions — Is your target variable balanced? If 99% of transactions are legitimate and 1% are fraud, you have a class imbalance problem.
Correlations — Which features are related to each other? Which features predict your target?
Patterns and trends — Seasonal effects? Time-based shifts? Geographic clusters?
Data quality issues you missed in step 3 — Sometimes problems only become visible in visualization.

Tools: histograms, scatter plots, correlation matrices, box plots, pair plots. Libraries: Pandas, Matplotlib, Seaborn, Plotly.

Example: You're building a house price predictor. EDA reveals that "number of bedrooms" and "square footage" are highly correlated (0.92). Including both might cause multicollinearity. You might drop one or combine them.

EDA often sends you back to step 2 or 3 — you realize you need more data, or your data has problems you didn't see before.

5. Feature Engineering and Selection — "Good features > fancy models"

This is often the difference between a mediocre model and a great one. Feature engineering is the art of creating new input variables that help the model learn patterns better.

Feature Engineering (Creating)

From dates: extract day_of_week, is_weekend, month, quarter, days_since_last_event
From text: word count, sentiment score, TF-IDF values, embeddings
From location: distance to nearest city, population density, latitude buckets
Combining features: price_per_sqft = price / square_footage, BMI = weight / height²
Domain knowledge: a doctor knows that "blood pressure × age" interaction matters; encode that

Feature Selection (Removing)

Not all features help. Some add noise. Too many features cause overfitting and slow training. Techniques:

Correlation analysis — drop features that are highly correlated with each other
Feature importance from tree-based models (Random Forest, XGBoost)
Recursive Feature Elimination (RFE) — iteratively remove the least important feature
L1 Regularization (Lasso) — automatically zeroes out unimportant features during training

Key insight: a simple model with great features almost always beats a complex model with bad features.

6. Model Selection — "Choose the right tool for the job"

Now you pick which algorithm(s) to try. This depends on:

Problem type: Classification → Logistic Regression, Random Forest, SVM, Neural Network. Regression → Linear Regression, XGBoost, Neural Network. Clustering → K-Means, DBSCAN. Sequence → RNN, LSTM, Transformer.
Data size: Small data → simpler models (logistic regression, SVM). Large data → deep learning can shine.
Interpretability needs: Healthcare and finance often need explainable models (decision trees, linear models). Recommendation engines can afford black boxes (deep learning).
Latency requirements: Real-time inference needs fast models. Batch processing can afford slower ones.

Best practice: Start simple. Try logistic regression or a decision tree first. If it gets 85% accuracy, you have a strong baseline. Then try more complex models and see if the improvement justifies the complexity.

You often try 3-5 different models and compare their performance.

7. Model Training — "The model learns about the data"

This is the step everyone thinks ML is about — but as you've seen, it's step 7 of 10.

Training means:

Feed data into the algorithm — the model sees examples and adjusts its internal parameters (weights) to minimize error
Split data into train/validation/test sets — typically 70/15/15 or 80/10/10. Never evaluate on data the model trained on.
Choose a loss function — the mathematical definition of "what is wrong." Cross-entropy for classification, MSE for regression, etc.
Set hyperparameters — learning rate, batch size, epochs, regularization strength. These are not learned by the model; you set them.
Iterate — training is rarely one-shot. You train, look at results, adjust, retrain.

Key concept: train/test split. If you evaluate your model on the same data it trained on, you get misleadingly high scores. It's like grading a student using the exact exam questions they practiced on.

8. Model Evaluation and Tuning — "How is your model doing?"

Training is done. Now: is the model actually good?

Evaluation Metrics

Different problems need different metrics:

Accuracy — % of correct predictions. Misleading with imbalanced data (99% accuracy on fraud detection means nothing if you just predict "not fraud" every time).
Precision — Of all things the model flagged as positive, how many were actually positive?
Recall — Of all actual positives, how many did the model catch?
F1 Score — Harmonic mean of precision and recall. Good when you need to balance both.
AUC-ROC — Area under the curve. Measures how well the model separates classes across all thresholds.
MSE / RMSE / MAE — For regression: how far off are predictions from actual values?

Hyperparameter Tuning

If results aren't good enough, adjust hyperparameters:

Grid Search — try every combination of predefined values
Random Search — randomly sample combinations (often faster than grid search)
Bayesian Optimization — smart search that learns from previous trials

Dealing with Problems

Overfitting (training score high, test score low) → more data, simpler model, regularization, dropout
Underfitting (both scores low) → more complex model, more features, longer training
Class imbalance → oversampling (SMOTE), undersampling, class weights, different metrics

This step often sends you back to steps 3, 4, 5, or 6. That's the iterative nature of ML.

9. Model Deployment — "Integrate model to the real world"

Your model works in a Jupyter notebook. Now it needs to work in production — handling real users, real data, and real scale.

Deployment means:

Packaging the model — save weights, serialize with ONNX, TorchScript, or pickle
Creating an API — wrap the model in a REST API (Flask, FastAPI) or gRPC endpoint
Infrastructure — where does it run? AWS SageMaker, Google Vertex AI, Azure ML, self-hosted Kubernetes, or edge devices?
Scaling — handle 10 requests/second? 10,000? Auto-scaling, load balancing, caching
CI/CD for ML — automated testing, model versioning, rollback capabilities

Common deployment patterns:

Real-time inference — API call, response in milliseconds (fraud detection, chatbot)
Batch inference — process large datasets periodically (weekly churn predictions, nightly recommendations)
Edge deployment — model runs on device (mobile app, IoT sensor, self-driving car)

Deployment is NOT the finish line. It's where the real work begins.

10. Monitoring and Maintenance — "Keep model healthy"

A deployed model is a living system. It degrades over time because the real world changes.

What to Monitor

Model performance — are accuracy/precision/recall staying stable?
Data drift — is incoming data different from training data? (seasonal changes, new user demographics, market shifts)
Concept drift — has the relationship between features and target changed? (what predicted churn in 2023 might not in 2026)
Latency — is inference speed within requirements?
Resource usage — CPU, memory, cost

When to Retrain

Performance drops below a threshold
Data distribution shifts significantly
Business requirements change
New data categories appear that the model has never seen

Best Practices

Set up automated alerts for performance degradation
Keep a champion/challenger system: new model version runs alongside the old one; switch only when the new one proves better
Log everything: predictions, input data, confidence scores. You'll need this for debugging.
Version your models like you version code. Know exactly which model version produced which prediction.

Monitoring is vital. A model that was 95% accurate at launch can silently drop to 70% if nobody's watching.

Summary — The Key Takeaways

ML is an iterative and structured process. It's a cycle, not a line. You will loop back to earlier steps repeatedly — that's normal, not failure.

Data quality and feature engineering are critical. Steps 3 and 5 have more impact on final model performance than the choice of algorithm at step 6. Good features beat fancy models.

Evaluation and tuning improve model performance. Don't ship the first model that trains. Rigorously evaluate, tune hyperparameters, and test on data the model has never seen.

Deployment isn't the end; monitoring is vital. The real world changes. Your model will degrade. Monitor, retrain, and iterate continuously.

The lifecycle is a loop. The best ML teams are the ones that spin through it fastest — not the ones with the fanciest models.

*If this helped you see the full picture of ML beyond "just training," drop a reaction.

Two Main Sources of ML Models: Pre-trained vs Custom — Which One Should You Use?

Ege Pakten — Mon, 20 Apr 2026 09:07:42 +0000

You want to build an AI-powered feature. Maybe a chatbot, an image classifier, or a recommendation engine. The first decision you'll face is: should I use someone else's model or train my own? This post breaks down both paths, when to pick which, and the real-world tradeoffs nobody tells you about upfront.

Why This Matters

Before you write a single line of code, this one decision determines three things: how fast you ship, how much you spend, and how well your product works. Pick wrong and you'll either waste months reinventing something that already exists, or ship a generic solution that doesn't actually solve your problem.

Source 1: Open Source Pre-trained Models

What Are They?

Pre-trained models are models that someone else (usually a big tech company or research lab) has already trained on massive datasets. They've spent the GPU hours, the engineering time, and the data collection effort. You just download and use.

Think of it like buying a car vs building one from scratch. The car (pre-trained model) already works — you just need to learn to drive it and maybe customize the seats.

Real Examples

Here are some of the most well-known pre-trained models and what they do:

Natural Language Processing (NLP):

BERT (Google) — understands text, great for search, Q&A, classification
GPT (OpenAI) — generates text, powers ChatGPT
LLaMA (Meta) — open-weight LLM, fine-tunable for custom use
T5 (Google) — text-to-text framework, versatile for many NLP tasks

Computer Vision:

ResNet — image classification, object detection
YOLO (You Only Look Once) — real-time object detection
CLIP (OpenAI) — connects text and images in the same space

Speech AI:

Whisper (OpenAI) — speech-to-text in 99 languages
Wav2Vec (Meta) — speech recognition with minimal labeled data

Healthcare & Science:

AlphaFold (DeepMind) — protein structure prediction
BioGPT (Microsoft) — biomedical text generation

Art & Creative:

Stable Diffusion — text-to-image generation
MusicGen (Meta) — text-to-music

Benefits

Saves development time. Training a model like BERT from scratch would take weeks on expensive GPUs. Downloading it takes 30 seconds.

Great for transfer learning. This is a key concept. The model has already learned general patterns (what a face looks like, how sentences are structured, etc.). You can then fine-tune it on your specific data to adapt it for your use case — often with very little data.

For example: BERT was trained on all of Wikipedia and BookCorpus. If you fine-tune it on 1,000 customer support tickets, it can classify support requests with 90%+ accuracy — because it already understands language, it just needs to learn your categories.

Community and ecosystem. Popular pre-trained models have huge communities, tutorials, and tooling. Platforms like Hugging Face host thousands of pre-trained models you can try in minutes.

When to Use Pre-trained Models

You're solving a common problem (text classification, image recognition, translation, Q&A)
You need to ship fast (prototype in days, not months)
You have limited data (transfer learning works with small datasets)
You have limited budget (no GPU cluster needed)
Your domain isn't extremely specialized

Source 2: Training Custom Models

What Are They?

You collect your own data and train a model from scratch — or from a very early starting point. You design the architecture, curate the dataset, define the training process, and iterate until it works.

When to Use Custom Models

When pre-trained models aren't specific enough. If you're doing something that generic models haven't seen — like detecting manufacturing defects on a very specific assembly line, or classifying rare medical conditions from proprietary scan data — a pre-trained model won't cut it.

For proprietary, domain-specific problems. Some industries have data that no public model has ever seen: financial fraud patterns unique to your bank, satellite imagery of your specific crop types, sonar data for underwater pipeline inspection. These need custom models.

Benefits

Tailored performance. A custom model trained on your data for your problem will almost always outperform a generic one — if you have enough data and expertise.

Control over training data and bias handling. You decide exactly what goes in. You can audit, clean, balance, and de-bias your dataset. With a pre-trained model, you inherit whatever biases were in their training data — and you often can't even see what that data was.

The Real Cost

Training custom models is expensive in multiple ways:

Data collection and labeling — You need thousands to millions of labeled examples. Getting quality labels is slow and costly (think: paying domain experts to annotate medical images).
Compute — Training from scratch needs powerful GPUs/TPUs, often for days or weeks. A single training run for a large model can cost $10,000–$100,000+.
Expertise — You need ML engineers who understand model architecture, loss functions, hyperparameter tuning, evaluation metrics, overfitting, and deployment.
Time — Months of iteration before you have something production-ready.

The Decision Framework

Here's how to actually choose. Ask yourself these questions in order:

Question 1: Has someone already solved this?

Search Hugging Face, TensorFlow Hub, PyTorch Hub. If a pre-trained model exists for your exact task — use it. Don't reinvent the wheel.

Question 2: Is my problem close to something that's been solved?

If yes, start with a pre-trained model and fine-tune it. This is the sweet spot for most companies. You get 80% of the benefit of custom training at 10% of the cost.

Fine-tuning means: take a pre-trained model, freeze most of its layers, and retrain just the last few layers on your specific data. The model keeps its general knowledge but adapts to your task.

Question 3: Is my problem truly unique?

If your data is proprietary, your domain is narrow, no pre-trained model comes close, and you have the budget and expertise — then train custom.

But even here, most teams start from a pre-trained base and do heavy fine-tuning rather than literally training from a random weight initialization.

The Spectrum

In practice, it's rarely a binary choice. Think of it as a spectrum:

Use as-is ← Fine-tune ← Heavy fine-tune ← Train from scratch
(cheapest)                                    (most expensive)

Most real-world ML projects land somewhere in the middle.

Key Terms You Should Know

Transfer Learning — Taking a model trained on one task and adapting it for a different (but related) task. The core technique that makes pre-trained models so powerful.

Fine-tuning — Retraining some or all layers of a pre-trained model on your specific dataset. Keeps general knowledge, adds specialized knowledge.

Feature Extraction — Using a pre-trained model as a fixed "feature extractor" — you feed data through it, take the intermediate representations, and train a simple classifier on top. Even cheaper than fine-tuning.

Hyperparameters — Settings you choose before training starts: learning rate, batch size, number of epochs, model architecture. Getting these right is critical and often requires experimentation.

Overfitting — When a model performs great on training data but poorly on new data. More common with small datasets and complex models. Regularization, dropout, and data augmentation help prevent it.

Data Augmentation — Artificially expanding your dataset by creating modified versions of existing data (rotating images, adding noise, paraphrasing text). Especially useful when you have limited training data.

Model Hub — Platforms that host thousands of pre-trained models ready to download: Hugging Face, TensorFlow Hub, PyTorch Hub, AWS SageMaker JumpStart.

Inference — Using a trained model to make predictions on new data. This is what happens in production. Different from training — inference is cheap and fast.

Common Mistakes

Mistake 1: Training from scratch when a pre-trained model exists. This is the most common waste of time and money in ML projects. Always search first.

Mistake 2: Using a pre-trained model without evaluation. Just because a model is on Hugging Face doesn't mean it works for your data. Always benchmark it on your own test set before shipping.

Mistake 3: Ignoring bias. Pre-trained models inherit biases from their training data. If you're deploying in a sensitive domain (hiring, lending, healthcare), you must audit for fairness.

Mistake 4: Underestimating data quality. A custom model is only as good as its training data. Garbage in, garbage out — no amount of GPU power fixes bad labels.

Mistake 5: Skipping the "boring" baseline. Before reaching for deep learning, try a simple approach (logistic regression, decision tree, keyword matching). Sometimes it's 90% as good at 1% of the complexity.

Wrapping Up

Pre-trained models are great for common tasks and faster deployment. They save you time, money, and let you benefit from massive datasets and research you couldn't replicate yourself.

Custom models offer flexibility for solving unique or proprietary problems. They give you full control over performance and bias, but at significantly higher cost and complexity.

Choosing the right source depends on your goal, your data, and your resources. Most teams start with pre-trained, fine-tune to their needs, and only train from scratch when they absolutely must.

The best ML engineers aren't the ones who build everything from zero — they're the ones who know when not to.

If this helped, drop a reaction. More AWS notes coming soon.

What is RAG? A Beginner's Guide to Retrieval-Augmented Generation (With a Full Pipeline Walkthrough)

Ege Pakten — Sat, 18 Apr 2026 09:21:50 +0000

If you've ever wondered how ChatGPT-style apps can suddenly "know" about your company's internal documents, product manuals, or legal files without being retrained, the answer is almost always RAG — Retrieval-Augmented Generation. In this post, we'll break down what RAG is, why it exists, and walk through the full pipeline step-by-step with a real example.

1. What is RAG?

Retrieval-Augmented Generation (RAG) is an AI framework that integrates an information retrieval component into the generation process of Large Language Models (LLMs) to improve factuality and relevance.

In plain English:

Instead of making the LLM remember everything, we let it look things up in a knowledge base right before answering.

The term RAG was coined in a 2020 research paper by Patrick Lewis et al. ("Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks") published on arXiv. The core insight: combine a parametric memory (the LLM's weights) with a non-parametric memory (a searchable document store) — and you get the best of both worlds.

2. Why RAG? The Motivation

Three big problems drove the invention of RAG:

LLM Limitations

LLMs are frozen snapshots. Once a model is trained, it only knows what was in its training data. It doesn't know:

What your company policies say
What happened after its training cutoff
What's in your private documents
What yesterday's sales numbers were

And even with what it does know, it can hallucinate confidently.

Cost of Retraining vs. Dynamic Retrieval

You could retrain or fine-tune the model every time your data changes. But:

Retraining a large model can cost tens of thousands to millions of dollars
It takes days or weeks
You have to do it again every time the data updates

Dynamic retrieval (looking things up at query time) is vastly cheaper and always up-to-date.

Need for Grounded, Up-to-Date Knowledge

For regulated industries (finance, healthcare, legal), you can't ship answers that come from "the model's memory." You need answers backed by sources you can cite and audit.

RAG addresses all three challenges by decoupling knowledge from the model.

3. The RAG Pipeline — Step-by-Step With a Real Example

This is the part most tutorials rush through. We're going to slow down.

Let's use a concrete example. Imagine you're building an internal developer assistant at a company called Acme Corp. Employees can ask it questions about the engineering handbook, API docs, and on-call runbooks.

A developer asks:

"How do I rotate the database credentials for the billing service?"

Here's exactly what happens behind the scenes.

Phase 1: Indexing (Done Once, Ahead of Time)

Before anyone can ask anything, we need to prepare the knowledge base.

Step 1a — Knowledge Corpus

First, we gather every document we want the assistant to know about:

The engineering handbook (Markdown files)
API documentation (HTML + Swagger specs)
Runbooks (Confluence pages)
Past incident post-mortems (Google Docs)
Security policies (PDFs)

Let's say this gives us 8,000 documents.

Step 1b — Document Chunking

An LLM can't efficiently search through a 50-page PDF. And you don't want to return a whole 50-page PDF to the user either — you want the one paragraph that actually answers their question.

So we chunk each document into smaller pieces. A common approach:

500 tokens per chunk (~300 words)
50 token overlap between chunks (so we don't split an idea across a boundary)

One chunk in our knowledge base might look like this:

[Chunk #4729 — Source: runbooks/billing-service.md]
"To rotate database credentials for the billing service:
1. Generate a new password in AWS Secrets Manager.
2. Update the 'billing-db' secret with the new value.
3. Trigger a rolling restart via: kubectl rollout restart deploy/billing.
4. Verify health endpoints return 200 OK.
5. Revoke the old credentials after 24h grace period."

After chunking, our 8,000 documents become maybe 120,000 chunks.

Step 1c — Vector Embeddings

For each chunk, we call an embedding model (like BERT, OpenAI's text-embedding-3-small, or Cohere's embedder). This turns each chunk into a vector — a list of ~1,536 numbers that represents the meaning of that chunk.

Chunk #4729 → [0.12, -0.08, 0.44, ..., 0.91]   (1,536 numbers)

Step 1d — Vector Database

We store all 120,000 of these vectors in a vector database — something like FAISS, Pinecone, Weaviate, Milvus, or Qdrant. The database indexes them so we can search across all of them in milliseconds.

Indexing is done. This usually runs as a background job, and you only re-run it when documents change.

Phase 2: Retrieval (Happens at Query Time)

Now a developer types:

"How do I rotate the database credentials for the billing service?"

Step 2a — User Query

The question comes in as plain text.

Step 2b — Query Embedding

We run the same embedding model on the question, producing a query vector:

Query → [0.15, -0.11, 0.48, ..., 0.87]

This is critical: you must embed the query with the same model you used to embed the chunks, otherwise the vectors live in different spaces and similarity becomes meaningless.

Step 2c — Similarity Search

Now we ask the vector database: "Which chunks have vectors closest to this query vector?"

Closeness is measured with a similarity metric, most commonly cosine similarity — it measures the angle between two vectors. The smaller the angle, the more similar the meaning.

Under the hood, the database uses Approximate Nearest Neighbors (ANN) tricks to search 120,000 vectors in ~5 milliseconds instead of comparing one by one.

Step 2d — Relevant Passages

The database returns the top-k most similar chunks (typically k=3 to k=10). For our query, we might get:

1. Chunk #4729 (score 0.94) — billing-service runbook, credential rotation
2. Chunk #3180 (score 0.89) — AWS Secrets Manager general guide
3. Chunk #5512 (score 0.85) — rolling restart playbook

These are the passages most likely to contain the answer.

Phase 3: Augmentation

Now we have relevant chunks, but we don't just show them to the user. We want the LLM to write a nice, synthesized answer using them.

Step 3a — Original Prompt

The user's raw question:

"How do I rotate the database credentials for the billing service?"

Step 3b — Augmented Prompt

We wrap it in a prompt template that injects the retrieved chunks as context:

You are Acme Corp's internal engineering assistant.
Answer the user's question using ONLY the context below.
If the answer isn't in the context, say you don't know.

---CONTEXT---
[Chunk #4729]: To rotate database credentials for the billing
service: 1. Generate a new password in AWS Secrets Manager...
[Chunk #3180]: AWS Secrets Manager allows you to store and
rotate database credentials...
[Chunk #5512]: Rolling restarts can be triggered via kubectl
rollout restart deploy/<service-name>...
---END CONTEXT---

USER QUESTION: How do I rotate the database credentials
for the billing service?

This is the "A" in RAG — Augmentation. We're augmenting the prompt with retrieved knowledge.

Phase 4: Generation

Step 4a — LLM Call

We send the augmented prompt to an LLM (GPT-3/4/5, Claude, LLaMA, T5, Mistral — your choice).

Because the model has the actual documentation right there in its context window, it doesn't need to remember anything — it just reads and synthesizes.

Step 4b — Final Response

The model produces a grounded answer:

"To rotate the database credentials for the billing service:

Generate a new password in AWS Secrets Manager.

Update the billing-db secret with the new value.

Trigger a rolling restart with kubectl rollout restart deploy/billing.

Verify that the health endpoints return 200 OK.

Revoke the old credentials after a 24-hour grace period.

Source: runbooks/billing-service.md"

Notice three things:

The answer is accurate — it comes from your actual docs.
The answer is current — if you update the runbook, the next query uses the new version. No retraining needed.
The answer can be cited — you know exactly which document it came from.

That's the whole RAG pipeline. Indexing → Retrieval → Augmentation → Generation.

4. The Retrieval Component in Detail

Three pieces make retrieval work:

Embedding Models

The model that turns text into vectors. Examples: BERT, text-embedding-3-small, Cohere Embed, Sentence-BERT. Choose one that's trained well for your language and domain.

Vector Stores

Databases optimized for vector similarity search. Popular options: FAISS (local, Facebook), Pinecone (managed), Weaviate, Milvus, Qdrant, and pgvector (Postgres extension).

Similarity Metrics

How we measure "closeness" between vectors. The go-to is cosine similarity, but Euclidean distance and dot product also show up. Cosine similarity is popular because it ignores vector length and focuses on direction — which is what semantic meaning lives in.

5. Augmentation & Generation in Detail

Prompt Templates

The structure that tells the LLM how to use the retrieved context. Good templates specify:

The assistant's role
What to do if context is missing
Output format (JSON, bullet points, prose)
Citation rules

Managing Model Context

The LLM only has so much context window. If retrieval returns 30 chunks but each chunk is 500 tokens, that's 15,000 tokens just for context. You have to:

Pick top-k carefully (more isn't always better)
Rerank retrieved chunks
Sometimes summarize chunks before injection

LLM Choices

Any generative LLM can work: GPT-3/4/5, T5, LLaMA, Claude, Mistral, Gemini. The RAG pipeline is mostly model-agnostic.

6. Applications and Benefits

RAG is behind a huge number of real-world AI products:

Knowledge-centric chatbots — customer support bots grounded in your docs
Document summarization & Q&A — ask questions about contracts, research papers, medical records
Enterprise search & knowledge management — "Glean for your company" style tools

Benefits:

No retraining required when data changes
Answers are traceable back to sources
Private data stays in your vector DB — never baked into model weights
Cheaper than fine-tuning for most use cases
Can mix multiple knowledge bases with one model

7. Challenges and Future Directions

RAG isn't magic. Here are the real tradeoffs.

Source Reliability & Bias

Garbage in, garbage out. If your knowledge base has outdated or biased content, your RAG system will confidently repeat it. Curation matters.

Latency & System Complexity

A RAG query is actually: embed → ANN search → rerank → build prompt → LLM call. That's a lot of moving parts. Each step adds latency, and each step can fail. Production RAG systems require serious observability.

Privacy & Security Safeguards

Your vector DB now contains sensitive content. Access control, encryption, and embedding leakage (yes, embeddings can leak information) all need attention.

Research Frontiers

Where RAG is heading:

Multi-hop retrieval — answering questions that need multiple retrieval rounds (e.g., "Find X, then look up Y for X")
Adapters — lightweight modules that specialize the LLM for using retrieved content better
Self-improving retrieval — the model learns which retrievals helped and which didn't

Wrapping Up — The TL;DR

What is RAG?
A framework that lets an LLM look things up in a knowledge base before answering, so its responses are grounded in real, current, specific information.

Why does it exist?
Because retraining is expensive, LLMs hallucinate, and most real-world apps need to answer from your data — not what the model memorized during training.

How does the pipeline work?

Indexing — Chunk documents → embed each chunk → store vectors in a vector DB.
Retrieval — Embed the user query → find nearest chunks with cosine similarity.
Augmentation — Inject retrieved chunks into a prompt template.
Generation — Send augmented prompt to an LLM → return grounded answer.

Where do you use it?
Anywhere you need an AI that can answer from your own content: internal docs, product support, legal research, medical Q&A, academic search, knowledge management, developer assistants, and more.

Once you understand the four-phase pipeline — Indexing, Retrieval, Augmentation, Generation — every RAG system you encounter becomes a variation on the same theme.

If this helped you finally "get" RAG, drop a reaction. More notes coming soon.

Embeddings Explained: The Secret Language AI Uses to Understand the World

Ege Pakten — Sat, 18 Apr 2026 09:20:32 +0000

If you've ever wondered how ChatGPT "knows" that king and queen are related, or how Spotify recommends songs you actually like, the answer is almost always the same: embeddings. This post breaks down what embeddings are, how they work, where they're used, and what you can actually do with them — no PhD required.

1. What Are Embeddings?

At their core, embeddings are just numbers — more specifically, a list of numbers (a vector) that represents something like a word, a sentence, an image, or even a user.

Computers don't understand the word "cat." They understand numbers. So we need a way to turn "cat" into numbers in a way that preserves its meaning. That's what an embedding does.

Simple example:

"cat"  → [0.21, -0.44, 0.89, 0.12, ..., 0.03]   (e.g., 768 numbers)
"dog"  → [0.19, -0.41, 0.85, 0.15, ..., 0.06]
"car"  → [-0.72, 0.31, -0.12, 0.88, ..., -0.44]

Notice how cat and dog have similar-looking numbers, while car looks very different. That's not an accident — it's the whole point. Similar meanings produce similar vectors.

The key idea: Embeddings are a way of placing concepts on a giant invisible map, where things that mean similar things end up close together, and things that mean different things end up far apart.

2. How Do Embeddings Work?

Embeddings don't appear out of nowhere. They're learned by a model during training. There are three core mechanisms worth understanding:

a) Self-Supervised Contrastive Learning

The model looks at massive amounts of raw data (text, images, etc.) and learns by playing a game: "pull similar things together, push dissimilar things apart."

For example, during training:

A sentence and a slightly rephrased version of it → should be close
A sentence about cats and a sentence about quantum physics → should be far apart

No human has to label anything. The model figures it out from the structure of the data itself. That's the "self-supervised" part.

b) Contextual Embeddings

Older embeddings gave every word a single fixed vector. That's a problem, because words can mean different things in different contexts:

"I deposited money at the bank." (financial institution)
"We had a picnic by the river bank." (side of a river)

Modern embeddings (like those from BERT or GPT) generate a different vector depending on the surrounding words. The model reads the whole sentence first, then decides what "bank" means here.

c) Dimensionality Reduction

Raw data (like a full image or a giant sparse word matrix) has way too many numbers. Embeddings compress this into a smaller, dense, meaningful representation — typically 256, 512, 768, or 1536 dimensions.

Think of it like writing a movie review: instead of describing every pixel in every frame, you capture the essence in a paragraph.

3. Embeddings Deep Dive

Let's go one layer deeper. Three properties make embeddings actually useful:

Mapping to a Vector Space

Every piece of data becomes a point in a multi-dimensional space. You can't visualize 768 dimensions, but you can imagine a 3D version:

        cat •
   dog •
         • kitten
                               • airplane
                                      • rocket

Cats, dogs, and kittens cluster together. Airplanes and rockets cluster together. The space itself has meaning baked into distance and direction.

Preserving Semantic Relationships

The famous example:

king - man + woman ≈ queen

You can literally do math on meanings. This works because "royalty," "gender," and other concepts become directions in the embedding space.

Efficient Processing

Once everything is a vector, you can do fast operations on millions or billions of items:

Compare two things? → compute cosine similarity (one quick math operation)
Find the nearest match? → use Approximate Nearest Neighbors (ANN)
Cluster similar items? → run k-means

This is why embeddings power huge real-world systems.

4. Types of Embeddings

Not all embeddings are created equal. Here are the major families:

Static Word Embeddings — Word2Vec, GloVe

These were the breakthrough that started it all. Each word gets exactly one vector, learned from how words co-occur in giant text corpora.

Pros: Fast, simple, very cheap to use.
Cons: Can't handle context ("bank" is always the same vector).

Contextual Embeddings — ELMo, BERT

These read the whole sentence and produce a vector for each word in context.

Pros: Much more accurate for real language understanding.
Cons: Heavier to compute, need a bigger model.

Sentence / Document Embeddings — Universal Sentence Encoder, Sentence-BERT

Instead of one vector per word, you get one vector for an entire sentence, paragraph, or document. Super useful for search, clustering, and classification.

Multimodal Embeddings — CLIP

These put text and images in the same vector space. A photo of a beach and the sentence "a sunny day at the ocean" end up close together. This is what powers most modern image search and text-to-image tools.

5. Key Use Cases — Where Embeddings Actually Shine

This is the "so what" section. Here's what you can build with embeddings.

Semantic Search

Forget keyword matching. With embeddings, a user can search for:

"How do I stop my laptop from overheating?"

…and you can return a document that says:

"Thermal management tips for portable computers"

No shared keywords, but the meaning is almost identical — and the vectors are close. This is the foundation of modern search, documentation bots, and RAG (Retrieval Augmented Generation).

Clustering and Recommendation

Group similar items automatically. Examples:

Netflix grouping movies you'd like based on what you've watched
Spotify building "Discover Weekly" playlists
Customer segmentation for marketing
Automatically grouping support tickets by topic

Anomaly Detection

If everything "normal" clusters in one region of the vector space, then anything far away from that cluster is probably weird. This is used for:

Credit card fraud detection
Network intrusion detection
Spotting defective products on factory lines
Finding unusual user behavior

Classification

Train a lightweight classifier on top of embeddings for things like spam detection, sentiment analysis, or intent recognition. You get great accuracy with almost no data.

6. Properties and Best Practices

If you're going to actually use embeddings, here are the things that matter in practice.

Normalize Your Vectors

Most similarity math works better when vectors are normalized to length 1. This means you're comparing direction, not magnitude — which is usually what you want semantically.

Pick the Right Dimensionality

Smaller (128–384): Faster, cheaper storage, less memory. Good for mobile or massive-scale systems.
Larger (768–1536+): More expressive, better accuracy, higher cost.

There's no free lunch. Start small, go bigger only if quality suffers.

Use Proper Indexing

If you have millions of vectors, you can't compare them one by one. Use a vector database or library:

FAISS (Facebook)
Pinecone
Weaviate
Milvus
Qdrant

These use tricks like ANN (Approximate Nearest Neighbors) to search billions of vectors in milliseconds.

Match the Embedding Model to Your Task

A general-purpose embedding model is fine to start. But for specialized domains (medical, legal, code), a fine-tuned or domain-specific model will often double your accuracy.

7. Challenges and Limitations

Embeddings are powerful, but they are not magic. Know the tradeoffs.

Memory and Compute

Storing a billion 1536-dimensional float vectors is not cheap. High-dimensional search can get expensive quickly. You'll eventually need to think about quantization, sharding, and cost.

Privacy and Data Leakage

Here's something that surprises most people: embeddings can leak information. Even though a vector looks like "just numbers," research has shown attackers can sometimes reconstruct or infer parts of the original text from an embedding ("embedding inversion attacks").

If you're embedding sensitive data (medical records, private messages, internal docs), treat the embeddings themselves as sensitive and protect them like you would the raw data.

Interpretability

A 1536-dimensional vector is a black box. You can't easily explain why two things are close. For regulated industries (finance, healthcare, EU AI Act compliance), this is a real concern.

Bias

Embeddings learn from data, and data contains human biases. If your training text associates certain jobs with certain genders, your embeddings will too — and any downstream system will inherit that bias.

8. Future Directions

Where is this all heading?

Hierarchical Embeddings

Instead of one flat vector, future systems will learn representations at multiple levels — word → sentence → paragraph → document — all connected, all meaningful.

Continual and Federated Learning

Today, most embedding models are trained once and frozen. The future is models that keep learning safely, updating over time without forgetting old knowledge — and learning across devices (federated learning) without centralizing private data.

Richer Multimodal Embeddings

Text + image is just the beginning. Expect models that unify text, image, audio, video, sensor data, and 3D scenes all in the same space. Search "the sound of rain on a metal roof" and get back audio clips and matching videos.

Wrapping Up — The TL;DR

Let's tie it all together.

What is an embedding?
A list of numbers that represents the meaning of something (a word, image, sentence, user, product) in a way a computer can work with.

Where are embeddings used?
Semantic search, RAG systems, recommendations, clustering, anomaly detection, fraud detection, classification, and multimodal search — basically anywhere you need a machine to understand "similarity" or "meaning."

What can you actually do with them?

Build a search engine that understands meaning, not just keywords
Power a chatbot with RAG using your own documents
Detect fraud, spam, or defects
Group customers, songs, movies, or articles automatically
Search images with text, or text with images
Add semantic understanding to almost any existing product

Embeddings are the quiet backbone of almost every modern AI system. You won't see them in the UI — but they're doing most of the real work behind the scenes. Once you understand embeddings, a huge amount of what seems "magical" about modern AI suddenly makes sense.

*If this helped you click on what embeddings really are, drop a reaction.

AI Agents Explained: 5 Types, Components, Frameworks, and Real-World Use Cases

Ege Pakten — Sun, 12 Apr 2026 09:55:27 +0000

AI agents are no longer just chatbots. They think, plan, use tools, and work together to solve complex problems autonomously. In this post, I break down the 5 types of AI agents, how they work, ReAct vs ReWOO frameworks, multi-agent systems, and guardrails — written so anyone can understand.

Introduction: AI Doesn't Just "Answer" Anymore

For most of us, our first experience with AI was simple: ask a question, get an answer, done. But there is a quiet revolution happening. AI agents are systems that don't just respond — they think, plan, use tools, and even talk to each other to solve complex tasks without human hand-holding.

This post is based on notes I took from IBM's "What Are AI Agents?" page. My goal is to write something clear enough that even someone who has never heard the term "AI agent" can walk away understanding the full picture.

What Is an AI Agent?

The simplest definition:

An AI agent is an AI tool that can autonomously perform complex tasks that would otherwise require human involvement.

The key word is "autonomously." A regular chatbot needs you to tell it what to do at every step. An AI agent takes a goal, plans how to reach it on its own, selects the right tools, and delivers the result.

Agentic vs Non-Agentic: What's the Difference?

Not every AI is an "agent." This distinction matters.

Non-Agentic AI:

No tools — can only say what it already knows
No memory — doesn't remember previous conversations
Limited reasoning — can't plan ahead
Needs constant human input to function
Example: A basic FAQ chatbot

Agentic AI:

Has access to tools — can search the web, call APIs, read files
Has memory — remembers past interactions and learns from them
Performs reasoning — creates step-by-step plans to reach goals
Works autonomously — minimal human intervention needed
Example: A hospital agent that handles insurance authorizations end to end

Think of it this way: non-agentic AI is a calculator — press a button, get a result. Agentic AI is an accountant — say "prepare my tax return" and it figures out the rest.

The 3 Core Components of Every AI Agent

No matter how simple or complex, every AI agent runs on three building blocks:

1. Goal (from the User)

Everything starts with a goal. The user tells the agent what needs to happen: "get insurance approval for this patient" or "find broken links on my website and report them." The agent takes this goal and performs task decomposition — breaking it into smaller, manageable subtasks.

2. Tools

Agents rarely have all the information they need on their own. So they reach out to external tools. These can be web search engines, APIs (weather, stock market, health records), databases, or even other AI agents (yes, an agent can use another agent as a tool).

3. Agentic Reasoning

This is the agent's "brain." It evaluates the information it perceives, uses its memory, and selects the best action to move toward the goal. It uses conditional logic, heuristics, and feedback loops to make decisions continuously.

The 5 Types of AI Agents

According to IBM, there are 5 distinct types, from simplest to most sophisticated:

1. Simple Reflex Agent

The most basic type. It works on "if X happens, do Y" rules. No memory, no planning — just pre-programmed reflexes.

Real-world analogy: A night light that turns on automatically when the room gets dark. Sense → react, nothing else.

Example: A temperature sensor that turns on the AC when it hits 30°C. It doesn't know why the temperature rose — it just follows the rule.

Limitation: If it encounters a situation it has no rule for, it's stuck.

2. Model-Based Reflex Agent

One step above simple reflex. The key difference: it has memory. It maintains a model of its environment and can operate in changing, partially observable settings.

Real-world analogy: A robot vacuum. It maps the room, remembers which areas it already cleaned, and navigates around furniture. It won't re-clean the same spot because it remembers it was already done.

Limitation: Still rule-based — smarter, but can't go beyond its predefined rules.

3. Goal-Based Agent

This is where things get serious. This agent doesn't just react — it has a goal and uses planning and reasoning to achieve it.

Real-world analogy: GPS navigation. When you say "take me to the airport," it doesn't just look at the current street. It examines the map, calculates traffic, evaluates alternative routes, and picks the best path.

Example: A customer service agent. Goal: "resolve the customer's issue." The agent understands the problem, searches a knowledge base, escalates if needed — always moving toward the goal.

4. Utility-Based Agent

Takes goal-based thinking one step further. This agent doesn't just achieve the goal — it aims to achieve it in the best possible way. It uses a utility function to evaluate different actions and pick the one that maximizes overall benefit.

Real-world analogy: An investment advisor. Doesn't just say "make money." It considers your risk tolerance, market conditions, time horizon, and recommends the strategy with the best balance of risk and reward.

Example: A logistics optimization agent. There are many ways to deliver a package, but this agent calculates the cheapest, fastest, and least risky route.

5. Learning Agent

The most advanced type. It has all the capabilities of the other types, plus the ability to learn. It improves with every experience, updates its knowledge base, and makes better decisions over time.

Real-world analogy: A medical resident. On day one, they're inexperienced, but with every patient interaction they learn. Years later, their diagnoses are far more accurate because of accumulated experience.

Example: An e-commerce recommendation engine. At first, it recommends the same products to everyone. Over time, it learns individual preferences and delivers personalized suggestions.

AI Agent Frameworks: ReAct vs ReWOO

How do agents actually "think"? Different architectural approaches have been developed. Two stand out:

ReAct (Reasoning + Acting)

ReAct follows a "Think → Act → Observe" loop. The agent pauses at each step, thinks, and plans its next move.

Flow:

Agent looks at the goal and thinks: "I need this information to solve this"
Uses a tool (e.g., web search)
Observes the tool's result
Thinks again: "Is this enough, or do I need more?"
Loop continues until the goal is met

Advantage: Highly flexible — can adapt its strategy at every step and handle unexpected results.

Disadvantage: The "thinking" at each step burns extra tokens and takes time. Higher cost and latency.

ReWOO (Reasoning Without Observation)

ReWOO follows a "plan first, execute later" approach. The agent does all its thinking upfront, creates a complete plan, and then executes the steps sequentially.

Flow:

Agent looks at the goal and generates a complete plan: "Step 1 is this, Step 2 is this, Step 3 is this"
Executes the plan in order — doesn't rethink at each step
Collects results

Advantage: Uses approximately 80% fewer tokens than ReAct. Much cheaper and faster.

Disadvantage: Struggles when it has limited context about its environment. Harder to handle unexpected situations since the plan was made upfront.

Which Should You Use?

Dynamic, unpredictable environments (customer service, chat, research) → ReAct
Well-defined, repetitive tasks (data processing, report generation, batch operations) → ReWOO

Multi-Agent Systems

A single agent is powerful, but some problems need a team. In multi-agent systems, multiple specialized agents work together.

Why isn't one agent enough?

Think of a human team: one person can't be the researcher, editor, and graphic designer efficiently. Same logic applies to AI agents.

Multi-agent structure example:

Researcher Agent: Gathers data and finds sources
Critic Agent: Audits data quality and fact-checks
Writer Agent: Synthesizes results and produces output

Each agent focuses on its specialty. According to IBM, multi-agent systems produce higher quality and more reliable outcomes than single agents.

Real-world example: A supply chain management system. One agent monitors inventory levels, another forecasts demand, a third optimizes logistics routes. They communicate with each other to make holistic decisions.

AI Agent Memory: Why It Matters

The real power of agents lies in their memory. Traditional AI processes each task independently — it doesn't remember yesterday's conversation. Agents use multiple types of memory:

Working Memory: Holds information about the active task. Like knowing which sources you've already checked during a research assignment.

Long-term Memory: Stores past experiences and learnings. When a similar task comes up again, the agent draws on previous experience.

Procedural Memory: Stores learned skills and automated behaviors. When an agent repeats a complex operation it has done before, it doesn't need to reason through every step from scratch.

AI Guardrails: The Safety Barriers

Agents are powerful, but powerful tools can be dangerous. Guardrails are boundaries an agent should never cross.

IBM defines guardrails like highway barriers: they don't slow the car down, but they keep it from going off the road.

What guardrails protect against:

Harmful content generation — preventing offensive, misleading, or dangerous outputs
Sensitive data exposure — preventing the agent from leaking personal or confidential information
Authority overreach — preventing the agent from making decisions outside its scope

With proper guardrails, agents can improve continuously while staying safe.

Final Thoughts

The thing that struck me most reading IBM's page was this: AI agents are no longer a technical concept sitting in research papers. They are real operational systems running in hospitals, supply chains, customer service centers, and more.

If you're new to this space, here's my suggested learning path:

Understand the 5 agent types — know which one fits which problem
Learn the ReAct vs ReWOO difference — framework choice directly affects cost and performance
Never skip guardrails — a powerful agent without controls is a dangerous agent
Think multi-agent — a single agent can do everything, but a team always does it better

I study one new AI topic per day from IBM's resources. This approach has been great for building deep understanding fast. I highly recommend the same: read, watch, but most importantly write. Writing forces you to truly understand the material — far more than passive consumption.

This post is based on IBM's "What Are AI Agents?" page. For more detail, I recommend checking the original source.

What Is AgentOps? A Beginner-Friendly Guide Using a Real Hospital Use Case

Ege Pakten — Sat, 11 Apr 2026 08:20:07 +0000

AI agents are no longer just chatbots; they are autonomous workers running real operations. But how do we know they are doing a good job? AgentOps is the answer. In this post, I break down the three layers of AgentOps using a hospital scenario anyone can follow.

Introduction: AI Agents Have Grown Up

A few years ago, when most people heard "AI", they pictured a simple chatbot: you ask a question, you get an answer, end of story. That world is gone.

Today's AI agents can think for themselves, talk to other software systems, read and write files, call APIs, and even hand off tasks to other agents. They are starting to behave less like a search box and more like a junior employee who shows up every day, takes assignments, and tries to get things done.

This is exciting. But it raises a serious question: how do we know these "employees" are doing a good job? What happens if an agent makes a mistake — especially in a setting like a hospital where mistakes can affect a patient's life?

This is exactly the gap that AgentOps fills. This post is built on notes I took while watching an IBM Technology video on the topic. My goal is to write something that even someone who has never heard the term "AI agent" can read and walk away understanding.

So, What Is AgentOps?

Here is the simplest way to define it:

AgentOps is the discipline of managing, improving, and monitoring AI agents in production.

If you have a software background, you may know "DevOps". DevOps is about continuously running, watching, and improving software systems. AgentOps is the same idea, but for AI agents. The difference is that the thing you are monitoring is not a static piece of code — it's a non-deterministic, decision-making "AI worker".

AgentOps is built on three layers:

Observability
Evaluation
Optimization

We will walk through each layer using one running example so the concepts don't feel abstract.

Our Running Example: Two AI Agents in a Hospital

Imagine a hospital. A doctor prescribes a new medication to a patient. Before the patient can pick it up, the insurance company needs to approve it. Traditionally this is a long, painful process involving phone calls, faxes, paperwork, and lots of human waiting.

Now imagine we automate it with two AI agents:

1. Clinical Documentation Agent

This agent connects to the hospital's Electronic Health Record (EHR) system. It pulls the doctor's notes, lab results, the patient's medical history, and any prior treatments. Then it bundles all the relevant information into a clean package that an insurance company would expect.

2. Payer Authorization Agent

This second agent takes that package, logs into the insurance company's portal, fills out the authorization form, submits it, and waits for approval. Once it gets a green light, it notifies the pharmacy and the doctor.

These two agents talk to each other (we call this A2A, short for agent-to-agent) and they each call out to external systems like the EHR, the insurance portal, and the pharmacy.

Sounds great. But how reliable is this system? How fast? How expensive? What if it makes a mistake one day? That is why we need AgentOps.

Layer 1: Observability

Observability answers the question: "What is my agent doing right now, and how long is it taking?" Instead of treating the agent like a mysterious black box, you turn it into a glass box.

In our hospital example, there are four key signals we want to watch.

End-to-End (E2E) Trace Duration

This is the total time from the moment the user makes a request to the moment they get an answer back. In our case, from the second the doctor says "get insurance approval for this patient" to the second that approval is confirmed.

If that takes 10 seconds, fantastic. If it takes 4 hours, something is wrong somewhere in the pipeline and you need to find it.

Agent-to-Agent (A2A) Handoff Latency

In our example, the Clinical Documentation Agent finishes its work and hands the task off to the Payer Authorization Agent. How long does that handoff take?

This matters more than people realize. Sometimes the time spent passing tasks between agents is longer than the time spent doing the actual work. A clean handoff protocol can save you minutes per request.

Tool Execution Latency

Agents rarely work alone. They use tools — calling the EHR system is a tool, opening the insurance portal is a tool, sending a message to the pharmacy is a tool.

How long does each tool take to respond? Maybe the insurance portal is slow and bottlenecking everything. Tool execution latency lets you spot exactly where the slowdown is, instead of blaming "the AI" in general.

Cost per Authorization

How much does a single insurance approval actually cost us? AI agents are not free. Every model call burns tokens, every tool call costs money. If a single approval costs $50 to run while a human staff member could do the same job for $10, the math doesn't work.

This is the metric that tells you whether your AI investment is actually paying off — or quietly bleeding the budget.

Layer 2: Evaluation

Observability tells you what is happening. Evaluation tells you whether what's happening is actually good. An agent that responds in two seconds but gives wrong answers is worse than useless — it's dangerous.

Task Completion Rate

Out of every 100 requests, how many does my agent finish successfully without a human having to step in? If that number is 95%, great. If it's 40%, your humans are still doing more than half the work and the automation barely exists.

Factual Accuracy

Are the things your agent says actually true? In healthcare this is not optional — it's life or death.

If the Clinical Documentation Agent records "no penicillin allergy" when the patient actually has one, the consequences can be catastrophic. Factual accuracy measures whether the agent's outputs match the real underlying data.

Guardrail Violations

Think of "guardrails" like the railings on a highway — boundaries the agent should never cross. For a hospital, those might be:

Never leak patient information to anyone unauthorized
Always comply with HIPAA and similar privacy laws
Never make decisions outside its authority

The guardrail violation rate measures how often the agent crosses one of those lines. The closer this number is to zero, the safer your system.

Clinical Appropriateness

This is a healthcare-specific check. Are the agent's decisions medically reasonable? For example, approving an adult dosage for a pediatric patient is technically a "decision" — but it's not clinically appropriate. This is usually scored using rules designed by actual clinicians.

First-pass Approval Rate

Of all the insurance authorizations the agent submits, what percentage get approved on the first try? This single metric secretly measures two things at once:

How well the agent prepares documentation
How well the agent understands the insurance company's rules

If your first-pass approval rate is 85%, the other 15% have to be redone, which means lost time and lost money.

Layer 3: Optimization

The first two layers tell you what is happening and whether it's good. The third layer answers: "How do we make it better?"

Prompt Token Efficiency

Language models are billed by "tokens". Every token is money. This metric asks: "How much output quality am I getting for each token I spend?"

If you're shoving a 5000-token mega-prompt into the model and only getting a one-line answer back, you are wasting money. If you can get the same quality with 500 tokens, you just made the system 10x cheaper. Multiplied over thousands of requests a day, this is a huge deal.

Flow Step Efficiency

How many steps does the agent take to complete a task? Sometimes agents loop, double-check things they already know, or ask the same question twice. For example, an agent might query the patient's name five times when it could have stored it once and reused it.

Cutting unnecessary steps makes the agent faster and cheaper at the same time.

Retrieval Precision (at K)

This one is a little technical, but it's important. Most agents don't know everything — they pull information from a knowledge base on demand. This is called RAG, short for Retrieval Augmented Generation. For example, an agent might pull the patient's previous lab reports to figure out their condition.

"Retrieval Precision at K" measures: out of the K documents the agent pulled, how many were actually relevant? If the agent grabs 10 documents but only 2 are useful, the other 8 are noise. Noise slows the agent down, confuses the model, and can lead to wrong decisions.

Handoff Success Rate

How often do handoffs between agents succeed cleanly? The Clinical Documentation Agent prepares a file, but does the Payer Authorization Agent actually receive it correctly? Or does it get a half-broken version? Failed handoffs are silent killers — the system looks like it's running but the wrong things are flowing through it.

Improvement Velocity

How fast is your agent actually getting better over time? This is a meta-metric. Are you continuously testing, measuring, tweaking, and re-deploying? Or is the agent today exactly as good as it was on day one? The real power of AgentOps is creating a tight feedback loop where the system improves itself week after week.

Back to the Hospital: What the Numbers Could Look Like

Imagine we apply all three layers in our hospital. After a few months of running this system with proper AgentOps, we might see results like:

Time to get an insurance approval reduced by 85% (from days to hours)
Cases requiring human intervention dropped by 50%
Cost per authorization down to $0.47

These are not just cute numbers on a dashboard. They translate into:

Patients getting their medications faster
Nurses and doctors spending less time on paperwork and more time with patients
The hospital running more efficiently as a business

That's the real promise of agents in healthcare — and AgentOps is the discipline that makes it possible.

Why You Cannot Skip AgentOps

Putting an AI agent into a hospital, a bank, or any other critical environment is like handing the car keys to a teenager. Can they drive? Maybe. Would you take your eyes off them? Absolutely not.

AgentOps is the "eyes" of your AI system. It watches, it grades, it improves. Running an AI agent without AgentOps is like:

Driving a car with no speedometer
Running a company with no financial reports
Treating a patient without a thermometer

It's technically possible, but it is asking for trouble.

Final Thoughts

The thing that hit me the hardest watching the IBM video was this: AI agents are no longer "experiments". They are becoming real operational systems. And real systems need real metrics.

To recap the three layers in plain English:

Observability: What is happening, how long is it taking, and how much does it cost?
Evaluation: Is what's happening correct, safe, and useful?
Optimization: How do we make it better, cheaper, and faster?

If you ever deploy your own AI agent into something that matters, do not skip these three layers. Building a flashy demo is easy. Building an agent that runs reliably every hour of every day is only possible with a solid AgentOps practice behind it.

I'm trying to learn one new AI concept per day by watching IBM Technology videos. If you're doing something similar, I highly recommend turning what you watch into written notes like this one. Writing forces you to actually understand the material — much more than just hitting play and nodding along.

This post is based on notes I took from an IBM Technology video on AgentOps. If you want to go deeper, I recommend watching the original.

AWS Basics But Needs to Be Known Before You Start Your Certification

Ege Pakten — Fri, 27 Mar 2026 16:30:23 +0000

You don't need to memorize 200 services to start your AWS journey. But you do need to understand the foundations. Here's everything I wish someone had explained to me before I started studying for my AWS certification.

1. What Even Is Cloud Computing?

Let's start from the very beginning. Cloud computing sounds fancy, but the core idea is simple: there's a physical server somewhere, and you're renting it over the internet.

Instead of buying your own hardware, setting up a server room, and hiring people to maintain it — you use someone else's infrastructure (like AWS) and pay only for what you use.

Here's the official NIST definition:

"A model for enabling ubiquitous, convenient, on-demand network access to a shared pool of **configurable computing resources* that can be rapidly provisioned and released with minimal management effort or service provider interaction."*

Let's break that down into human language:

On-demand → You get resources whenever you want them
Shared pool → Multiple customers share the same physical infrastructure
Configurable → You choose how much CPU, RAM, storage you need
Rapidly provisioned and released → Spin up a server in minutes, shut it down when you're done
Minimal management effort → No need to physically touch any hardware

2. What's Actually Behind the "Cloud"?

Behind the cloud, there are real, physical server racks sitting in massive data centers around the world. AWS has these data centers spread across the globe, and the infrastructure is organized in a clear hierarchy:

Region

A large geographic area (e.g., eu-west-1 for Ireland, us-east-1 for N. Virginia). You choose a region based on where your users are, compliance requirements, or pricing.

Availability Zone (AZ)

Each Region contains multiple AZs (e.g., eu-west-1a, eu-west-1b, eu-west-1c). An AZ is one or more data centers with independent power, networking, and connectivity. Multiple AZs exist for redundancy — if one goes down, the others keep running.

Edge Locations / Points of Presence (PoP)

These are smaller, lightweight locations spread even more widely than Regions. They're used primarily for caching content closer to end users.

3. Edge Networks and CDN — Speed Matters

Imagine your origin server is in the US, but a user in Southeast Asia wants to load your website. Without any optimization, every single request travels across the Pacific Ocean and back. That's slow.

This is where Edge Networks and CDN (Content Delivery Network) come in.

How it works:

Your original content lives on the Origin Server (e.g., in us-east-1)
AWS caches copies of your static content (images, CSS, JS, videos) at PoP / Edge Locations around the world
When a user in Singapore requests your site, they get the cached version from the nearest PoP — not from the US
Result: dramatically lower latency

CDN vs Edge Network — are they the same thing?

Not exactly. The Edge Network is the infrastructure — the distributed network of servers worldwide. CDN is the most well-known use case of that infrastructure. In practice, people use the terms interchangeably, and that's mostly fine.

AWS's CDN service is called CloudFront, and it leverages these Edge Locations.

4. Virtualization and the Hypervisor

Here's where it gets interesting. When you launch an EC2 instance, you're not getting a dedicated physical server. Here's what's actually happening:

Host Computer → The physical AWS server rack in a data center
Host OS → The operating system running on that physical machine
Hypervisor → Software that sits on top of the Host OS and splits the physical machine into multiple virtual machines
Your EC2 Instance → One of those virtual machines

So if a physical server has 64 CPUs and 256 GB RAM, the hypervisor might split it into four isolated instances of 16 CPUs / 64 GB RAM each. Different customers can be using the same physical hardware without ever knowing about each other — completely isolated.

AWS built their own hypervisor called Nitro, which operates at the hardware level for minimal performance overhead.

Key takeaway: EC2 instances are virtual slices of physical servers, managed by a hypervisor. Multiple instances from different customers can live on the same physical machine, fully isolated from each other.

5. The Shared Responsibility Model — Who's Responsible for What?

This is probably the most important concept for your certification exam. AWS and the customer share security responsibilities, but the line between them is very clear.

Security OF the Cloud → AWS's Job

AWS is responsible for protecting the infrastructure that runs all the services:

Physical security — data center access, cameras, biometric entry
Hardware — servers, storage, networking equipment
Software — compute, storage, database, networking services
Global infrastructure — Regions, Availability Zones, Edge Locations
Host OS & Hypervisor — you can't even access these

Security IN the Cloud → Your Job

You are responsible for everything you put in and configure on the cloud:

Customer data — whatever you upload, store, or process
Platform, applications & IAM — who has access to what, roles, permissions, secret keys
OS & firewall configuration — if you launched an EC2 with Ubuntu, patching it is on you
Encryption — client-side, server-side, in-transit, at-rest decisions
Network traffic protection — security groups, NACLs, VPN configurations

The Three Control Categories

Category	Who?	Examples
Inherited	AWS only	Physical & environmental security, host OS, physical servers
Shared	Both	Patch management, configuration management, awareness & training
Customer-Specific	Customer only	Guest OS, custom applications, data encryption strategies

Quick Quiz — Test Yourself

Question	Answer
Patching your EC2 instance's operating system?	You
Guest OS security patches?	You
Running the host OS and virtualization layer?	AWS
Managing IAM user access and secret keys?	You
Maintaining the server under your Lambda functions?	AWS
Physical security of data centers?	AWS
Encryption-at-rest strategy for your RDS database?	You

Simple rule of thumb: If you can configure it in the AWS Console or CLI → it's your responsibility. If you can't even touch it → it's AWS's responsibility. If both sides need to do their part → it's shared.

Think of it like renting an apartment: AWS gives you a secure building (locked doors, fire alarms, security guards). But whether you lock your own door, put your valuables in a safe, or leave your windows open — that's entirely on you.

6. AWS Support Plans — Developer vs Business

When you create an AWS account, you'll need to choose a support plan. Here's the practical difference between the two most common ones:

Feature	Developer (~$29/mo)	Business (~$100/mo or % of usage)
Who can open tickets	1 person (primary contact)	Unlimited team members
Response time (critical)	12 hours	1 hour
Support channels	Email only	Email + Phone + Chat
Trusted Advisor	Limited checks	Full access
Third-party software support	No	Yes

Which one should you pick?

Developer → Great for learning, experimenting, and building prototypes. Start here if you're just getting started.
Business → Essential when you're running production workloads with real users. The 1-hour critical response time alone is worth it when something goes down.

Good news: You can start with Developer and upgrade to Business anytime from the AWS Console (Support Center > Change Plan). Billing is prorated, so you only pay for what you use. You can also downgrade later if needed.

Wrapping Up

Before diving into specific services like S3, Lambda, or DynamoDB, make sure these foundational concepts are solid:

Cloud = renting virtual resources from physical infrastructure over the internet
Regions → AZs → Edge Locations form the backbone of AWS's global presence
CDN / Edge Networks cache content close to users for speed
Hypervisors split physical servers into isolated virtual machines (EC2 instances)
Shared Responsibility Model — know what's yours vs what's AWS's (this will be on the exam)
Support Plans can be upgraded/downgraded as your needs evolve

These aren't glamorous topics, but they're the bedrock everything else is built on. Nail these, and the rest of your certification journey will make a lot more sense.

Happy cloud learning! If you found this helpful, drop a reaction or follow for more AWS certification notes.