DEV Community: yash-nigam

LLM Concepts Deep Dive - Conceptual Mastery for Developers | Udemy

yash-nigam — Wed, 16 Sep 2026 12:00:03 +0000

LLM Concepts Deep Dive

Course: LLM Concepts Deep Dive: Conceptual Mastery for Developers | Udemy Business
Notes from: Monday, September 14, 2026
*https://www.udemy.com/course/llm-concepts-deep-dive/?couponCode=25BBPMXNVD35

What is a Model
- 1.1 What is a Language Model?
- 1.2 How Does a Language Model Learn?
- 1.3 What Does "Predict the Likelihood of Next Set of Words" Mean?
- 1.4 Language Model vs. Generative AI
- 1.5 Modern LLMs
Two Types of Language Modeling Tasks
- 2.1 Autoencoding
- 2.2 Autoregressive
LLM Training Methodology
- 3.1 Learn/Model Language
- 3.2 Pre-training
- 3.3 Weights
- 3.4 Pre-training Alone Is Not Enough
- 3.5 Fine-tuning
Tokens and Embeddings
- 4.1 How LLMs Process Tokens
- 4.2 Visualizing Tokenization
- 4.3 Why Isn't One Word Always One Token?
- 4.4 The Solution
- 4.5 Token Decoding
- 4.6 What Numbers Are Assigned to Tokens
- 4.7 How the Model Learns
Feature Matrix
- 5.1 Feature Matrix for Language
Embedding
- 6.1 Understanding Embeddings Using Dimensions
- 6.2 Embedding Math
Tokens and Their Values
- 7.1 Text Similarity
Transformer Architecture
Context Length in LLMs
- 9.1 LLM State
What is RAG
- 10.1 Vectors and Chunks
RAG Pipeline
Vector Databases
- 12.1 How It Works — Ingestion
- 12.2 How It Works — Indexing
- 12.3 How It Works — Querying
- 12.4 Some Vector DBs

What is a Model

A model is something that represents, simulates, or predicts something else.
It is a simplified version of a real thing that helps us understand it or predict what might happen.

Example:

🏠 Model of a house - Represents what a real house looks like.
🏙️ Model of a city - Represents the roads, buildings, and layout of a real city.
🌦️ Weather model - Uses weather data and patterns from the past to predict how the weather may be in the future.
🗣️ Language model - Learns patterns in language and predicts what words or tokens are likely to come next.

1.1 What is a Language Model?

A language model is a model that learns how language works, its patterns from large amounts of text, such as:

Which words commonly appear together.
How sentences are structured.
How people communicate.
How words relate to each other.

It then uses these learned patterns to predict and generate language.

1.2 How Does a Language Model Learn?

Give thousands of examples. such as:

The cat is sleeping.
The dog is running.
I am drinking water.

Over time, the following patterns are learnt:

Cat and dog are animals.
Sleeping and running are actions.
Words follow certain grammatical patterns.

A language model learns language patterns from text in a similar broad sense, although its internal learning process is based on neural networks and mathematical optimization.

Example of next-word prediction:

Prompt	Predicted next word
The sky is	blue
I am going to the	market
Please open the	door

1.3 What Does "Predict the Likelihood of Next Set of Words" Mean?

A language model looks at the words already written and calculates how likely different next words or tokens are.

Example:

Input: I am going to the

The model might assign probabilities like this:

Possible next word	Illustrative likelihood
market	35%
office	25%
park	15%
school	10%

The model selects or samples a token according to its generation process, then continues predicting the next token.

1.4 Language Model vs. Generative AI

This is an important distinction in your notes.

Language model - A model that learns patterns in language and predicts language.
Generative AI - AI that can create new content, such as text, images, music, and videos.
LLM (Large Language Model) - A language model trained on a large amount of data, with many learned parameters, to understand and generate language.

1.5 Modern LLMs

Are Text prediction machines
Recursive completion – generate text based on previously generated text hence can do this infinitely
Generates text based on parameters – temperatures
Uses training data for prediction
LLM cannot do anything other except generate text
However, tools can be used to perform external operations using the generated output

Two Types of Language Modeling Tasks

2.1 Autoencoding

A neural network architecture that learns to understand and represent data by encoding it into essential features and reconstructing the original input. In language modeling, it learns by predicting a missing word in a sentence using the surrounding context. Example: "The cat sat on the ___." → "mat".

2.2 Autoregressive

A language modeling approach that predicts the next word or token based on the previous words. It generates text one token at a time. Example: "The cat sat on the" → "mat". ChatGPT and other GPT-style LLMs use autoregressive language modeling.

Remember:

Autoencoding: Predict the missing word → Understand language.
Autoregressive: Predict the next word → Generate language.

Autoregression can also be used for stock price and weather prediction, where future values are predicted from past values.

LLM Training Methodology

3 main stages that enable an LLM to perform useful tasks.

3.1 Learn/Model Language

First learn/model language
To model language, the LLM is trained on very large amounts of text so it can learn patterns, relationships, syntax, context, and knowledge contained in that data.

3.2 Pre-training

Initial phase of LLM training - where the model learns from a very large and diverse dataset containing billions or trillions of tokens.
Develop broad language understanding, contextual patterns, reasoning capabilities, and knowledge represented in the training data.
For an autoregressive LLM, the model learns to predict the probability of the next token given the previous tokens across a huge number of text sequences.
Training data is divided into sequences/chunks of tokens; each sequence provides many next-token prediction examples.

The model learns by adjusting its internal parameters (weights) based on the difference between its predictions and the actual training tokens.

3.3 Weights

a. They are numerical values (parameters) inside the neural network.
b. During training, these parameters are continuously adjusted to capture statistical relationships between tokens and their contexts.
c. For example, the model learns that "cat sat on the" is statistically more likely to be followed by "mat" than "house" in certain contexts.

The goal is to optimize the model parameters so that the model performs well on predicting tokens across diverse language examples.

Training uses backpropagation and gradient descent to calculate how the weights should be adjusted, repeated across many training iterations.

Training data can include:

a. Wikipedia and other reference sources
b. Books and curated text datasets
c. Web/internet text and other licensed or publicly available data

3.4 Pre-training Alone Is Not Enough

A pretrained model primarily learns to predict/continue text; it is not automatically a helpful conversational assistant.
Model should respond appropriately to user requests and have useful conversations.
This requires additional training to improve instruction following, helpfulness, safety, and response behavior.
Pre-training is therefore more than simple phone autocomplete, but the underlying next-token prediction objective is similar.
Additional instruction tuning/post-training is used to teach the model how to respond to instructions and conversations.

3.5 Fine-tuning

Instruction Tuning

Also called instruction fine-tuning (IFT); it is a form of supervised fine-tuning rather than simply "transfer learning."

The model is trained to follow natural-language instructions and produce appropriate responses.

Use an instruction dataset:

a. Contains instruction/prompt → response examples.
b. The pretrained model already contains broad language patterns and learned representations, but needs examples of how to follow instructions and format useful responses.
c. These datasets can be manually created or curated, often containing high-quality question-and-answer or task-and-response examples.
d. Reinforcement Learning from Human Feedback (RLHF) can be an additional post-training stage used to align responses with human preferences; it is not the same thing as instruction tuning.
e. A reward model can score responses based on learned human preferences, and reinforcement learning can optimize the model toward higher-reward behavior.

Fine-tuning

Further training of an already pretrained/instruction-tuned model for a specific task, domain, behavior, or use case.
Focused dataset relevant to the target task—for example, customer-support conversations, legal text, or a specific classification task.
Additional training is performed on the existing model

Process:

a. Prepare examples representative of the desired task and behavior.
b. Provide input/output examples showing what response or behavior is desired.

Fine-tuning APIs offered by some model providers allow developers to upload training examples that teach the model "given this type of input, produce this type of output."

Tokens and Embeddings

Search engines primarily relied on keyword matching—pages containing more relevant occurrences of the search terms could rank higher.
However, we want to search based on the meaning and context of words, known as Semantic search, rather than exact word matching.
Example: "Spring framework" should understand that Spring refers to the Java software framework, not the spring season or a car spring.

4.1 How LLMs Process Tokens

LLMs ultimately process numbers, so text must first be converted into numerical representations.
Character codes such as ASCII/Unicode are not sufficient because they represent characters, not learned linguistic relationships or meaning.
LLMs learn language patterns and relationships from token sequences; semantic understanding emerges from the model's learned representations.
Therefore, text is broken into manageable units called tokens, which the model can process efficiently.
A token is a unit of text, not strictly a "unit of meaning"; meaning is represented through the model's learned embeddings and internal representations.

This is done using tokenization:

The tokenizer breaks text into tokens.
Each token is assigned a unique token ID from the model's vocabulary.
The model receives token IDs and processes them to predict/output token IDs.
One token is not necessarily one word; it can be a word, part of a word, punctuation, whitespace, or other text fragment.

4.2 Visualizing Tokenization

Tokenizer playground: GPT-4o, o200k_base
Prompt: i am doing tokenization
Token breakdown: i am doing token ization
Token count: 5 tokens
Token IDs: 72, 939, 5306, 6602, 2860

4.3 Why Isn't One Word Always One Token?

Token IDs are assigned to vocabulary entries so the model can efficiently process text as numbers.
Words are useful linguistic units, but they are not always the most efficient units for an LLM.
Giving every possible word its own token would require a very large vocabulary.
Related words such as learn, learning, learned can share subword pieces rather than requiring completely separate tokens.
Languages contain an enormous number of words, forms, names, and new words.
Subword tokenization allows the model to handle related and previously unseen words more efficiently.
Words can also have multiple meanings; context helps the model determine the intended meaning.

4.4 The Solution

Break text into commonly occurring subword/character sequences, rather than relying only on complete words.
Tokenizers generally use frequency/statistical patterns in training data to build an efficient vocabulary.
Punctuation such as a comma can be its own token or part of another token.
Frequently occurring text sequences can receive their own tokens, making processing more efficient.
Tokenization is not necessarily based on grammatical word boundaries.
Different models use different tokenizers and vocabularies, so the same text can produce different tokens and token counts.

Overall process: Text → Tokenizer → Token IDs → LLM → learned embeddings/internal representations → predicted Token IDs → Tokenizer → Text. Embeddings are the numerical vectors the model uses to represent tokens in a learned continuous space; they are different from token IDs.

4.5 Token Decoding

The tokenizer processes the text from left to right and splits it into tokens based on the tokenizer's vocabulary and tokenization rules.
It generally prefers token pieces that efficiently match the available vocabulary; it is not simply a greedy search for the longest English word.
Example: tokenization may be split into token + ization because those pieces exist in the tokenizer's vocabulary.

4.6 What Numbers Are Assigned to Tokens

Each token is assigned a token ID from the tokenizer's vocabulary. The ID itself does not store the meaning of the token.
The token ID is then converted into an embedding vector, which is a learned set of numerical values [,,,] helps the model represent relationships between tokens.

4.7 How the Model Learns

Example: Picture classification: how does AI determine whether a picture is a cat?
The model learns to identify cat vs. not-cat from many examples.
It adjusts its internal parameters based on more examples and learns which visual patterns are useful.
Features: whiskers, ears, eyes, fur, and body shape can provide signals that a picture may contain a cat.
Different features can have different importance; a feature such as whiskers may provide a stronger signal than something like background color.
The model learns patterns by representing features numerically and combining many signals rather than relying on a single feature.
The model examines features/patterns across many examples and learns which visual characteristics are associated with cats.
It also learns from incorrect predictions, adjusting its internal parameters when its prediction differs from the correct answer.
Over many examples, it learns that some attributes have a stronger statistical relationship with the target than others.
For example, whiskers, ears, eyes, and body shape can be useful signals for identifying a cat, while features such as day vs. night or background are generally less reliable.

Feature Matrix

Why does AI require matrix multiplication?

In AI, matrix operations are fundamental because models need to process and transform large numbers of values efficiently.
A feature can be represented as a numerical dimension in a vector; for example, whiskers can be one feature.
For a simple example: whiskers = 0 for a spoon and whiskers = 1 for a cat or dog.
Attributes/features can therefore be represented using numerical values for each object.
In real AI models, the model learns useful features automatically rather than us manually defining features such as whiskers.
During training, the model adjusts its parameters to capture patterns and correlations that help it perform its task.

5.1 Feature Matrix for Language

LLM create feature matrix for every token

An LLM represents each token using a vector of learned numerical features.
Columns represent learned dimensions/features, and rows represent tokens.
Each token therefore has a value for every learned dimension.
The value in each cell represents how strongly that token is represented along that particular learned dimension; these are embedding values, not simply weights.
For example, a token such as "hello" may have a strong value along a learned dimension associated with greetings or conversational context.
Because there can be hundreds or thousands of dimensions, each token is represented by an N-dimensional vector.

Embedding

[1, 1.04, -2.55, ...]

Away of representing things using numbers/vectors based on learned patterns and relationships.
This set of numerical values is called an embedding.
Every token is mapped to an embedding vector that captures learned patterns and relationships.
These embeddings are part of how the model represents and processes language; meaning is not stored in a single number or dimension.

6.1 Understanding Embeddings Using Dimensions

Imagine, for simplicity, that we have three dimensions:

Dimension 1: Pet ↔ Not pet
Dimension 2: Wild ↔ Not wild
Dimension 3: Mammal ↔ Not mammal
Each word/token receives a numerical value along each dimension.
For example, dog, cat, lion, and crocodile would have different values across these dimensions.
Dog and cat would tend to have more similar representations because they share several characteristics.
In real LLMs, there are many more dimensions—often hundreds or thousands—not just three manually defined dimensions.
Tokens with similar learned representations tend to be closer together in the embedding space.
Tokens with different representations tend to be farther apart.
Therefore, similar concepts or usage patterns can occupy nearby regions of the embedding space.

6.2 Embedding Math

Slide: Embedding math!

Embedding math!

v("king") − v("man") + v("woman") ≈ v("queen")

v("king") represents the embedding vector for king.
The relationship captured between king/man and woman/queen can approximately appear through vector arithmetic.
This demonstrates that embeddings can capture relationships and patterns between words/tokens.
Embeddings influence how an LLM processes language and are an important part of how it develops useful representations of meaning and context.

Tokens and Their Values

Each token has a number called a token ID (an identifier used by the model).
The token ID is used to look up the token's embedding vector.
An embedding is a set of numerical values representing how strongly a token is represented across the model's learned dimensions/features.

Q&A: tokenizer vocab splitting

Question 1
If our tokenizer's vocab contains the subwords {"uni", "vers", "ity", "un", "iverse"}, how would it most likely split the word "university"?

Answer
"uni", "vers", "ity"

7.1 Text Similarity

From tokens to text

Problem statement

Given a sequence of tokens x₁, x₂, …, xₜ, predict the most likely next token xₜ₊₁.

Two things need to happen:

The next token depends on the context of the preceding text, potentially including many earlier tokens.
Not every part of the previous text is equally relevant to predicting the next token.

How do you know what's important?

The model learns during training which parts of the preceding context are relevant to other parts.
During processing, mechanisms such as attention allow the model to give more importance to relevant tokens and less importance to less-relevant tokens.
In this way, the model learns from patterns in the training data which information is useful for predicting the next token.

Transformer Architecture

What transformers do

They transform text to an "intermediate" language (encoding)

Language is embeddings + positional embeddings

Get the best next embeddings

Transform that back to text (decoding)

A Transformer is a neural-network architecture designed to solve the language-modeling problem of understanding context and predicting the next token.
It uses attention to determine which other tokens are important to each token when building its contextual representation.
This idea was introduced in the landmark paper "Attention Is All You Need."
They transform tokens into context-aware numerical representations inside the network.
The input representation combines token embeddings + positional information.
Attention and other Transformer layers transform these representations to produce information useful for predicting the next token.
The model converts the final numerical output into probabilities over possible tokens, and a token is selected/generated from those probabilities.
Each token effectively "asks" other tokens how relevant they are to its current context through the attention mechanism.
This is especially important because many words have multiple meanings, and their meaning depends on the surrounding context.
The token's representation is updated through the Transformer layers based on its relationship with other tokens in the sentence/context.

Get the next best embedding

Each token "asks" every other token "how relevant are you to me?"

Builds a context-aware representation

Goes through a process to decide what to do with the relevant tokens

Goes through weight matrix to generate the final output tokens

There are lot of ambiguous words in english language and meaning depends on the context

Embedding is transformed on the fly or adjusted on the fly depending on the meaning of each token with rest pect to the other token in the sentence

"How relevant are you?"

Consider these sentences

The animal didn't cross the street because it was too tired

The animal didn't cross the street because it was too wide

In the first sentence, "it" is likely referring to the animal, because an animal can be tired.
In the second sentence, "it" is likely referring to the street, because a street can be wide.
Attention helps the Transformer use these contextual relationships to build different representations for the same token depending on the surrounding text.

Key idea: The embedding starts as a learned representation of a token, but Transformer layers transform it into a context-dependent representation based on the surrounding tokens.

Context Length in LLMs

LLM have context limit

LLMs have a context limit — the maximum number of tokens the model can consider in a single request.
The Transformer processes relationships between tokens using attention, and practical/model-design limits determine how many tokens can be handled at once.
The context typically includes both input tokens and the tokens generated as output, subject to the model's total context window.
Context length is therefore measured in tokens, not words or characters.

9.1 LLM State

LLMs are stateless and do not have memory

LLMs are generally stateless between separate API requests; the model itself does not automatically remember previous conversations.
Each new request is processed based on the information provided in that request and any external state/memory system.
The trained model primarily consists of learned parameters (weights); conversation history is not stored inside those weights.
Therefore, the model does not inherently retain your previous conversation after the request ends.
To make an LLM behave as though it remembers a conversation, the application can send relevant previous conversation/context with each request.
Conversation context can contain sensitive or private information, so applications need appropriate data-handling and privacy controls.
As context becomes very large, cost, latency, and sometimes the model's ability to use all the information effectively can become concerns.

What is RAG

Retrieval Augmented Generation

A common way to handle knowledge that is too large, private, or frequently changing to put entirely into the LLM context.
Dynamically retrieves relevant document chunks at query time.
Retrieved text should be relevant and useful; the LLM uses the retrieved context to generate the answer, but retrieval quality still matters.
Often used when the information changes frequently or comes from private documents.

Goal of RAG – retrieve relevant information that fits within the model's available context window and provide it to the LLM.

Example: "Hello bot, what is the return policy for this particular product of company x?"

The LLM may not have the company's latest or specific return policy in its model knowledge, so the application needs to retrieve the relevant information from the company's documents or knowledge base.
The basic process is: find the relevant part of the policy → insert it into the LLM's context → provide the user's question → ask the LLM to generate an answer based on that retrieved context.

10.1 Vectors and Chunks

The key challenge is: how do we find the specific portion of the documents that is relevant to the user's question?
First, documents are typically split into smaller chunks that can be individually retrieved.
When the user asks a question, the system searches for the chunks most relevant to that question.
The relevant chunks are then inserted into the LLM's context along with the user's query.
To find the appropriate chunks, we can use semantic similarity—representing the query and document chunks as embeddings and comparing their vectors.
The system generates an embedding for the user's query and compares it with the embeddings of the document chunks.
It retrieves the top-K most similar/relevant chunks and provides them to the LLM.
This overall approach is called Retrieval-Augmented Generation (RAG):
- Retrieval: find relevant information from the knowledge base.
- Augmentation: add that retrieved information to the user's query/context.
- Generation: the LLM uses the augmented context to generate the response.

RAG Pipeline

RAG Pipeline

User prompt → User prompt arrives — this contains what the user is asking LLM
LLM - The LLM or application creates a retrieval query, which may be the original prompt or a reformulated query to retrieve data from vector database.
Vector DB — The query is embedded and searched against the vector database. Top-k relevant chunks are retrieved.
Context assembly — Retrieved chunks are added to the original prompt/context.
Final generation — Feed the assembled context to the LLM to generate the response.

Vector Databases

What is a vector database?

A database optimized for storing and searching high-dimensional vectors, often alongside metadata or references to the original documents.
Can retrieve records using IDs/metadata, but its key RAG capability is vector similarity search.
Finds vectors that are nearest or most similar to a given query vector.

The problem

Finding the right text is not always easy.
Simple keyword/text matching may miss relevant content with different wording.
Semantic/vector search can find text based on meaning and contextual similarity.

The solution

Chunk the documents into smaller pieces.
Generate an embedding for each document chunk.
Generate an embedding for the user's query.
Find the chunks whose vectors are most similar/relevant to the query vector.
Add the retrieved chunks to the LLM's context.

Vector databases help in these.

A vector database query

"Which stored embedding vectors are closest to my query vector, and what documents do they point at?"

12.1 How It Works — Ingestion

Precompute embeddings for each document/chunk using a chosen embedding model.
Store the vector along with a unique ID and metadata/payload, such as document ID, source, or text.

12.2 How It Works — Indexing

The database builds a vector-search index, often using an Approximate Nearest Neighbor (ANN) algorithm.
ANN structures organize/search vectors efficiently so similar vectors can be found much faster than comparing the query with every stored vector.
Performance depends on the index and implementation; O(log N) is not guaranteed and should not be treated as a general property of vector databases.

12.3 How It Works — Querying

Generate an embedding for the incoming question/query.
The vector index quickly returns the top-k most similar vectors, usually approximately rather than by exhaustive comparison.
Retrieve their IDs/metadata and associated document chunks for use as context.

Code example: vector DB usage

from your_vector_db import Client
from your_embedding_model import embed

# 1. Initialize
db = Client(api_key="...")

# 2. Upsert documents
for doc_id, text in docs.items():
    vec = embed(text)
    db.upsert(id=doc_id, vector=vec, metadata={"text": text})

# 3. Query
query = "What is the return policy for electronic items?"
q_vec = embed(query)
results = db.search(vector=q_vec, top_k=5)

for hit in results:
    print(hit.id, hit.score, hit.metadata["text"])

12.4 Some Vector DBs

Pinecone
Qdrant
Weaviate
Milvus

Claude Certified Developer - Foundations certification Overview

yash-nigam — Sun, 13 Sep 2026 20:02:28 +0000

I recently completed the Claude Certified Developer - Foundations certification.

This certification is based on the official course from Anthropic: https://anthropic-partners.skilljar.com/path/claude-certified-developer-foundations

However, this course is only available to Anthropic partners. Below is my overview of the modules and structure of the Prep Course.

Claude Certified Developer - Foundations Prep Course

The course consists of the following main modules:

MSO Foundations
Learn the model fundamentals and technical foundations the rest of the Developer Foundations course builds on.
- 1.1 How LLMs Behave
- 1.2 Models & Reasoning
- 1.3 Prompting Modes
- 1.4 Technical Substrate
Production-Grade Prompting, Agents & Tool Use
Build your first production integration on Claude, with reliable prompts, tools, context management, and agent loops.
- 2.1 Prompting Craft
- 2.2 Extended Thinking
- 2.3 Tool-Use and Schema Design
- 2.4 Streaming Responses
- 2.5 Context Engineering
- 2.6 Agent Construction
- 2.7 Agent Memory
- 2.8 Cumulative Debug Task
- 2.9 Multimodal and Batch Ingestion
Claude Code, MCP & Integration
Learn to make a working Claude integration configurable, shareable, and safe to connect to real systems.
- 3.1 Module Orientation & Learning Objectives
- 3.2 The Explore → Plan → Code Loop
- 3.3 Permission Modes
- 3.4 Settings Configuration Hierarchy
- 3.5 Placing the Human Review Gate
- 3.6 Durable Project Context: CLAUDE.md
- 3.7 Rules Instruction Files
- 3.8 Hooks
- 3.9 Subagents
- 3.10 Packaging Workflows: Skills
- 3.11 Custom Commands & Plugin Packaging
- 3.12 Building an MCP Server
- 3.13 MCP Transport
- 3.14 Prompt Caching (Applied to MCP/Tool Definitions)
- 3.15 Retrieval-Augmented Generation (RAG) — Two Forms
- 3.16 MCP Configuration Scope
- 3.17 MCP Tool-Level Permissions
- 3.18 GitHub MCP Server — Worked Authentication Example
- 3.19 Enterprise Authentication Patterns
- 3.20 Regulated/Enterprise Deployment Requirements
- 3.21 Code Modernization as an Applied Case Study
Production Engineering, Evals & Security
Learn to take an agent that works in development and prove it holds up under real production traffic.
- 4.1 Module Introduction
- 4.2 Evals & Judges
- 4.3 Testing & Tracing
- 4.4 Failure Handling & Model Selection
- 4.5 Cost & Orchestration
- 4.6 Security
- 4.7 Cumulative Task
Accelerators & IP Contribution
Package a build that works into one that survives reuse, review, and deployment beyond the engagement that created it.
- 5.1 Module Introduction
- 5.2 Packaging for Reuse
- 5.3 Contributing Back
- 5.4 Requirements & Lifecycle
- 5.5 Systems Lifecycle for Claude Applications
- 5.6 Deployment & Versioning
- 5.7 Comparing Platforms
- 5.8 Trust Boundaries
- 5.9 Cumulative Task

Module 1: MSO Foundations

1.1 How LLMs Behave

1.1.1 Tokens
Everything Claude processes (prompt, history, tools, results, output) is measured in tokens, not words/characters, and both pricing and context budget are token-based. Output tokens cost more than input tokens.

1.1.2 Context Window
The max tokens allowed in a single request — system prompt, user prompt, documents, history, tool results, and output combined. Exceeding it returns a model_context_window_exceeded stop reason.

1.1.3 Sampling & Temperature
Claude samples the next token from a probability distribution rather than picking one fixed "best" token. Temperature (a request parameter) tunes this: lower = more repeatable, higher = more varied/creative.

1.1.4 Non-Determinism
Sampling means identical inputs can yield different, equally valid outputs, so exact-string-match tests are unreliable. Test for properties instead — e.g., "required field present" or "value within range."

1.1.5 Testing Model Output: Structural vs. Semantic Correctness

Check Type	Definition	Examples
Structural correctness	Deterministic, yes/no checks	Regex match, valid JSON, exact substring, value within tolerance
Semantic correctness	Meaning-based checks, can't be scripted deterministically	Summary captures key points, correct tone, accurate despite different phrasing

Semantic checks need an LLM-as-judge: a separate model call scores the output, often against a rubric/reference answer.

1.1.6 Evals
A testing framework for non-deterministic outputs: scores quality across many test cases and reports an aggregate pass rate (e.g., "87% passed"). Consists of test inputs, correctness criteria (structural or LLM-judge), and an aggregation method.

1.2 Models & Reasoning

1.2.1 Claude Model Family
Four tiers trading off cost, latency, capability, and quality:

Tier	Description
Sonnet	Balanced default for most production workloads
Haiku	Optimized for speed/cost within its capability range
Opus	For demanding work beyond Sonnet's envelope
Fable	Highest-capability tier, for the hardest reasoning/coding/agentic tasks

1.2.2 Model Selection Strategy
Start with Sonnet by default; move up a tier only when an eval shows it fails the quality bar, or down to Haiku only when an eval shows the quality drop is acceptable. Model choice should be eval-driven, not assumed.

1.2.3 Reasoning Modes
Reasoning mode (on/off, separate from model choice) lets the model spend extra tokens "thinking" before answering; current models use adaptive thinking tuned via an effort setting (the older budget_tokens control is deprecated, now returns a 400 error). Thinking is hidden by default and worth the cost only on hard, multi-step problems — not simple lookups.

1.3 Prompting Modes

1.3.1 Zero-shot / One-shot / Multi-shot (Few-shot) Prompting
Distinguished by how many worked examples are given in the prompt:

Mode	Examples Given	Best Used When
Zero-shot	None (instruction only)	Task is simple, output shape is obvious
One-shot	One input/output example	A single reference clarifies expected output
Multi-shot (Few-shot)	Several examples	Needs specific structure, casing, or edge-case handling

1.3.2 Cost Tradeoff of Examples
Each example consumes tokens on every call and eats into context budget, so examples aren't free — they trade quality/precision against cost.

1.3.3 Interaction with Model Choice
More capable models often succeed zero-shot where smaller ones need few-shot examples, so added examples can substitute for a cheaper model. Best practice: start with the simplest model and fewest examples meeting your eval bar, then add either only if needed.

1.4 Technical Substrate

1.4.1 SDK vs. Raw REST API
Claude is accessed via an HTTP REST API (JSON over your API key); the SDK just wraps auth, request construction, retries, and parsing. Both hit the same API and models.

1.4.2 Response Delivery Patterns

Pattern	Description
Synchronous	Send request, wait for full response, then act — simplest pattern
Streaming	Delivered in pieces via server-sent events as generated; output appears immediately, client reassembles the final message
Asynchronous (`AsyncAnthropic`)	Non-blocking async/await enables concurrency without blocking, while each call still returns in real time
Message Batches API	High-volume/offline: submit a batch, poll for completion; up to 24h latency for lower per-token cost

Module 2: Production-Grade Prompting, Agents & Tool Use

2.1 Prompting Craft

2.1.1 Diagnosing Prompt Failures Instead of Adding Words
When a prompt fails, fix it by identifying which structural technique is missing — not by rewording or adding instructions. Rewording doesn't fix boundary confusion or format drift; only the matching technique does.

2.1.2 The Four Failure → Fix Mapping

Symptom	Missing Technique	Why
Wrong output shape (prose instead of JSON/label)	Output constraint	Nothing specified the response's form/stopping point
Content/scope drift over turns	System prompt (or a more specific one)	Behavioral contract too vague to hold across turns
Right task, invented structure	Few-shot examples	Claude can't infer exact structure from description alone
Works on tested inputs, breaks on edge case	Constraint covering that variant	Prompt only validated against a narrow input set

2.1.3 System Prompts
Carry the persistent behavioral contract for the whole session — role, output format, rules that must not change between turns. Write once as a stable instruction layer.

2.1.4 XML Tags
Used to separate instructions from examples/data (e.g., <sample_input>, <ideal_output>) so Claude doesn't misread examples as part of the task itself.

2.1.5 Few-Shot Examples
Show the exact input→output pattern (structure, casing, format) rather than describing it — closes gaps written instructions leave open, especially for edge cases.

2.1.6 Output Constraints
Explicitly define the exact form of the response (e.g., "return only one label, no other text") — controls form independent of content, preventing parser-breaking variability.

2.1.7 When to Stack vs. Simplify vs. Diagnose
Stack all four techniques for complex tasks with defined output contracts and edge cases; simplify for simple tasks (e.g., plain summarization). If a prompt has grown longer over 3+ re-prompts without improving, stop and diagnose the missing technique instead of adding more text.

2.1.8 Worked Postmortem: Six-Pass Classification Prompt
A ticket-classifier prompt was revised 6 times, growing longer without fixing inconsistency — Pass 4 added description instead of a constraint, Pass 5's verbosity induced verbose output (latency regression, no accuracy gain). Pass 6 fixed it by replacing prose with an output constraint + 2 few-shot examples.

2.1.9 Structured Outputs: Moving Control from Prompt to API
Instead of asking for a shape via prompt text, you give the API a JSON schema and the model is constrained at generation time (constrained decoding) — only schema-matching tokens can be produced.

Mechanism	What It Does
JSON outputs	Set `output_config.format` to `type: json_schema` + your schema — constrains the final response text
Strict tool use	Set `strict: true` on a tool definition — constrains/validates arguments before your code runs

Tradeoffs: first request on a new schema is slower (grammar compiled, cached 24h); input tokens rise slightly (format-description system prompt injected); a guaranteed schema doesn't guarantee success — check stop_reason for refusal/max_tokens; incompatible with message prefilling.

2.2 Extended Thinking

2.2.1 What It Does
When enabled, Claude reasons step-by-step in a separate thinking block before the final answer. On newest models this content is hidden by default — request a summarized display to see it.

2.2.2 Adaptive Thinking / Effort Setting
Enabled via the thinking parameter (off by default); depth is tuned via an effort setting, not a fixed token budget. The older budget_tokens control is deprecated (400 error on newest models), and thinking tokens bill the same as output tokens.

2.2.3 When to Use Extended Thinking

Task	Decision
Multi-step reasoning (math, multi-hop logic, dependent action planning)	Enable, match effort to problem depth
Mechanical tasks (classification, format conversion, lookups)	Leave off — no benefit, wastes tokens
Agentic loops planning across tool calls	Enable, budget for the planning step

2.2.4 The Carry-Back Rule (Critical Constraint)
In tool-use loops with thinking enabled, every thinking block must be sent back unchanged next turn — each has a signature, and editing/dropping it (even redacted blocks) breaks the signature and the API rejects the request. Manage context bloat via context engineering, never by stripping thinking blocks.

2.3 Tool-Use and Schema Design

2.3.1 Tool-Use and Schema Design
Covered indirectly via structured outputs/strict tool use (see 2.1.9); explicit schema-design content beyond tool descriptions affecting routing appears under Agent Construction (see 2.6).

2.4 Streaming Responses

2.4.1 Why Streaming
Sends the response in pieces as generated (via server-sent events) instead of waiting for the full message, removing blank-screen wait on long responses. Your code must reassemble blocks itself and handle early termination.

2.4.2 Event Sequence and Handler Actions

Event	Meaning	Handler Action
`message_start`	New message beginning	Initialize empty content array
`content_block_start`	New block opening (text/tool_use/thinking)	Create slot at that index
`content_block_delta`	Incremental fragment of a block	Append to block; tool_use JSON isn't parseable until block closes
`content_block_stop`	Block complete	Finalize block (first point tool_use JSON is parseable)
`message_delta`	Top-level changes (stop_reason, usage)	Record stop_reason
`message_stop`	Stream complete	Assembled content is now the finished message

2.4.3 Never Act on a Partial Block
tool_use inputs arrive as fragmented JSON strings, invalid until content_block_stop. Parsing/running tools before block closure causes malformed JSON or missing arguments.

2.4.4 Commit to History Only After message_stop
Only add a streamed assistant turn to history once message_stop arrives and every block is fully assembled — a turn from an interrupted stream has an incomplete tool_use block and violates tool-use pairing rules on the next request.

2.4.5 Handling Interrupted Streams
Treat accumulated content as provisional until message_stop; on interruption, discard the partial turn and retry. Check stop_reason from message_delta before continuing a loop — tool_use means calls are ready to run.

2.4.6 Postmortem: "Read Loop Ended" ≠ "Message Complete"
A handler that commits history whenever its read loop exits (instead of gating on message_stop) can silently commit corrupted tool_use blocks — the validation error surfaces on the next request, not the one that caused it.

2.5 Context Engineering

2.5.1 Model Selection Recap
Four-tier family (Fable, Opus, Sonnet, Haiku) trading cost/latency/capability — start with Sonnet, move tiers only on eval results. Confirm current lineup/identifiers against docs at build time.

2.5.2 Context Window Is a Shared, Finite Budget
Covers system prompt, history, every persisting tool result, and output. An over-budget request is rejected pre-generation; hitting the ceiling mid-response returns partial output with model_context_window_exceeded — neither path auto-trims, so the app must manage it, and production sessions fill context faster than dev/test.

2.5.3 Four Strategies for Managing Context Budget

Strategy	What It Does	When to Use	What's Lost
Pruning	Rewind to an earlier message, drop everything after	After an unproductive/dead-end path	Everything after the rewind point
Compaction (`/compact`; server-side beta or manual summarization)	Summarizes history into a condensed version	Approaching context ceiling, want to retain knowledge	Any detail not captured in the summary
Clearing (`/clear`; new API session)	Starts fresh, empty context	Next task is unrelated	All session context (persist elsewhere, e.g. CLAUDE.md)
Subagent handoffs	Spawn isolated subagent with task-specific context; returns summary	Self-contained delegable subtasks	Visibility into subagent's intermediate reasoning

2.5.4 Prompt Caching & Token Counting
Prompt caching reuses processing of a stable prefix at a fraction of cost via cache_control (type: ephemeral, up to 4 breakpoints) — the highest-leverage cost reduction for multi-turn sessions. count_tokens returns a request's token count without running inference, for dev validation and production budget gating.

2.5.5 RAG: Three Failure Points
RAG lets an LLM retrieve external documents via chunking → embedding match → assembly → answer. Chunking must balance size (too small loses context, too large dilutes matches); embedding match can miss exact-term matches (pair with lexical search); sloppy assembly causes the model to ignore retrieved content entirely.

2.5.6 Indexed vs. Iterative RAG
Indexed (fetch-once) is inspectable/testable but needs an index to build/maintain/secure — good for a stable corpus. Iterative (search-across-rounds) avoids staleness but costs more tokens/time and is less inspectable — better for changing corpora or multi-step questions.

2.5.7 Compaction: Summarizer Prompt Design
A vague summarizer prompt ("summarize so far") loses task-critical detail; a well-specified one (preserve file paths, decisions, errors/resolutions) retains what matters. Under-specified summarizers are a leading cause of multi-session agent failures.

2.5.8 Subagent Handoffs for Long-Horizon Tasks
Instead of growing the context window, decompose the task and give each subagent only a scoped task, minimal context, relevant prior results, needed tools, and clear exit conditions. Keeps per-turn cost low at the expense of implementation overhead.

2.5.9 Postmortem: Context Budget Not Tested Against Production Data
A receipt-processing agent's 40k-token self-imposed cap worked fine against 800-token/call dev fixtures but was exhausted by turn 8 against real 3,200-token production tool outputs, crowding out system instructions. Fix: prune tool outputs after use and compact proactively — always measure actual production-scale token costs before shipping.

2.6 Agent Construction

2.6.1 Definition
An agent is a multi-step tool-use loop with managed context and a defined goal. Key upfront question: does the problem actually require an agent, given the added coordination overhead, context cost, and failure surface?

2.6.2 Workflow vs. Agent Decision

Choose a Workflow When...	Choose an Agent When...
Steps can be enumerated in code	Path can't be enumerated in advance
Error cost is high, step-level guardrails needed	Non-determinism acceptable, actions constrained by toolset
Standard observability tooling required	Inputs vary unpredictably
Inputs are well-constrained	Task requires creative tool sequencing

2.6.3 Three Wiring Paths

Path	Who Runs the Loop	You Own	Best For
Raw API loop	Your code	Everything: loop, execution, context mgmt, retries, exit conditions	Full control / learning / library constraints
Agent SDK	SDK, in your process	Tool execution + app; SDK gives loop structure, context mgmt, tool registration	Claude Code's scaffolding without rebuilding it
Claude Managed Agents (public beta)	Anthropic (server-side, via SSE)	App layer + agent definition as a versioned resource	Long-running tasks (mins–hours), avoid building sandbox/loop

2.6.4 Managed Agents Specifics
Anthropic runs the loop/sandbox/retries server-side with stateful stored sessions; not currently eligible for Zero Data Retention or HIPAA BAA (rules out PHI/ZDR workloads). Requires the managed-agents-2026-04-01 beta header; common path is prototyping on Agent SDK then re-expressing config as a Managed Agents resource for production.

2.6.5 Four Steps Common to Every Agent Loop
(1) Register tools with consistent schema, (2) set a system prompt scoped to the agent's task/toolset, (3) handle the tool-use loop — every tool_use block must get a tool_result and all blocks from one turn resolved together, (4) define explicit exit conditions rather than relying on Claude to volunteer completion.

2.6.6 Human-in-the-Loop (HITL) Insertion Points

Insertion Point	Trigger	Risk Addressed
Before destructive tool call	Write/delete/send operation	High — irreversible actions
After a planning step	Plan generated, about to execute	Medium — wrong plan even if execution is correct
On unexpected output	Error flag, empty result, out-of-bounds value	Variable — catches failures retries won't fix

2.6.7 Tool Orchestration: Over-Tooling vs. Under-Tooling
Too many overlapping tools causes erratic routing (the more common production problem, from registering tools "just in case"); too few causes hallucinated paths. Start with the minimum tool set and add only on confirmed capability gaps.

2.6.8 Regulated Data Constraints Determine Delivery Route

Constraint	Rules Out	Survives Review
Attorney-client privilege	Unaudited consumer Claude.ai calls	Direct API/SDK via firm's SSO-authenticated app, routed through firm-approved LLM gateway with full logging
HIPAA (PHI)	Any endpoint without a covering BAA	BAA-covered direct API, or Bedrock/Vertex on a HIPAA-eligible account (BAA excludes Console, Workbench, betas, consumer plans)
GDPR / data residency	Routes without pinned execution region; direct API (no EU residency)	Bedrock/Vertex with region pinned to the covered jurisdiction
FedRAMP / government	Any non-authorized endpoint, incl. dev/test on commercial endpoint	Claude for Government (C4G), Bedrock GovCloud, Vertex Assured Workloads (Claude Enterprise on AWS Marketplace is NOT FedRAMP authorized)
Internal data-residency policy	Any vendor outside the approved list	Delivery route on the approved vendor only

(SOC 2 governs system operation, not endpoint selection — covered in Module 4.)

2.7 Agent Memory

2.7.1 Memory Scope

Scope	Persists	Cost	Use When	Lost
In-context	Within a single session	Zero retrieval overhead, token cost grows with conversation	Short sessions fitting fully in context	Everything at session end
External storage	Across sessions/users/instances, in a DB	Retrieval latency + read/write engineering	Cross-session continuity needed	Nothing (cost is latency/complexity)
Summarized memory	Condensed version injected next session	Lower cost than full replay, drops detail	Long-running conversations exceeding budget	Anything summarizer didn't preserve
Stateless (none)	Nothing	Zero overhead	Self-contained, one-off jobs	All prior context

2.7.2 Design-Time Decision, Not Refactor-Time
Choosing memory scope during design avoids costly production refactors; a common failure pattern defaults to full in-context history, which works until token cost/latency climbs and hits the context ceiling.

2.7.3 Skills: Reusable, On-Demand Instruction Sets
A Skill is a reusable SKILL.md (frontmatter: name, description + instructions body) that Claude loads only when the description matches an incoming request — unlike in-context memory, instructions aren't resident every session.

2.7.4 Skills vs. CLAUDE.md vs. In-Context Instructions

Pattern	Loads	Context Cost	Best For
Skill	On-demand, when description matches	Low (only name+description loaded upfront)	Task-specific expertise not needed every session
CLAUDE.md	Every session unconditionally (CLI); controlled by `settingSources` in Agent SDK	Fixed overhead per session	Always-on project-wide standards
In-context instructions	Every turn in that session	Grows with session length, doesn't survive session end	Short, one-off sessions

2.7.5 Skills on the Messages API (Beta)
Requires two beta headers (code-execution-2025-08-25, skills-2025-10-02); Skills run inside the code execution container, not the app's own environment, affecting available tools/filesystem access.

2.7.6 Subagents and Skills
Subagents don't automatically inherit Skills or conversation history from the parent (clean context on delegation), but do inherit the parent's permission context. A subagent needing a Skill must have it explicitly listed in its own configuration.

2.7.7 Postmortem: In-Context Memory Filling by Session Four
An escalation-support agent worked fine in dev (long continuous sessions) but failed in production, where many shorter sessions accumulated state until injected history exceeded 40k tokens by session 4 — before any tool call. Fix: refactor to external storage, though this took longer under production pressure than doing it at design time.

2.8 Cumulative Debug Task

2.8.1 Cumulative Debug Task
Applied exercise combining prior concepts, not new theory. Observed bug-diagnosis mapping: vague tool descriptions → schema/routing failure; interrupted streams/stripped thinking blocks → carry-back rule violation; orphaned tool_result without matching tool_use → context/pairing violation; concatenated full session transcripts → memory scope failure at scale.

2.9 Multimodal and Batch Ingestion

2.9.1 Image Token Cost
Images process in 28×28-pixel patches: cost = ⌈width/28⌉ × ⌈height/28⌉ visual tokens (e.g., 1000×1000px ≈ 1,296 tokens). Each tier has a max native resolution (oversized images downscaled first) — confirm current limits against docs, and measure production image token cost before building a pipeline.

2.9.2 Three Ways to Send an Image/File

Method	Mechanism	Overhead	Best For
Inline base64	Encode bytes directly in the message	Full payload sent every request	One-off images unlikely to be reused
URL reference	Pass a public URL; Claude fetches it	No payload, but URL must stay stable/public/reachable	Already-hosted, stable public images
Files API (beta; not on Bedrock/Vertex)	Upload once → get `file_id` → reference later	One-time upload cost, near-zero overhead thereafter	Reused assets, large files, multi-turn conversations

2.9.3 Sending PDFs
Uses a document block type (vs. image) with the same source structure (base64/URL/file_id); no required name field, optional title/context fields, same token-cost and Files API reuse mechanics.

2.9.4 Prompting Technique Carryover to Multimodal
The same four prompting techniques apply to image/PDF analysis, but images introduce visual ambiguity (overlapping objects, depth, occlusion) that prompts should explicitly instruct how to handle.

2.9.5 Message Batches API
For high-volume async processing — up to 100,000 requests or 256MB per batch. Submit → get batch_id → poll → download results (arbitrary order, matched via custom_id). Lower per-token cost, latency up to 24h — suited to offline pipelines/evals/bulk jobs, not real-time interactions.

2.9.6 Postmortem: "Chunked Loop" Mistaken for Batching
Looping over a list calling the synchronous API per item (even in smaller chunks) is not batching — it's serialized calls still hitting the same rate limits at volume. True batching requires the actual Batch API.

2.9.7 Use-Case Fit

Scenario	API	Why
User uploads photo, expects immediate result	Synchronous	Real-time required
Nightly job classifying 5,000 records	Batches API	No latency constraint; cost savings matter
Eval run against 2,000 examples	Batches API	Offline, no real-time need
Chatbot reply generation	Synchronous	User is actively waiting

2.9.8 Two Failure Modes Combining Multimodal + Batch
(1) Misreading latency needs — using batch for a user-facing image flow fails because the user is waiting; (2) underestimating context cost — multiple large images/PDFs per request can blow past token limits at scale.

Module 3: Claude Code, MCP & Integration

3.1 Module Orientation & Learning Objectives

3.1.1 Module Orientation
Claude Code is a terminal-native dev partner running the same agent loop as the API, plus a permission layer, config system, and team-sharing features; MCP enables secure external-service integration. Core theme: configs that "work on your machine" often fail once shared, deployed, or run against production.

3.2 The Explore → Plan → Code Loop

3.2.1 The Three-Phase Loop
Claude Code works in three phases: Explore (reads files, traces logic, no edits), Plan (proposes a structured edit description, requires human approval), Code (writes/executes changes only after approval) — this sequence reduces wrong assumptions.

3.2.2 Plan Mode as the Hook Point
Plan mode holds the agent in the explore phase, blocking all edits/commands until released — a good default for unfamiliar or high-stakes codebases.

3.3 Permission Modes

3.3.1 Permission Modes

Mode	Auto-Approves	Still Gated	Limitations
Default	Reads only	All edits/commands	Safe but slow; baseline for new/unfamiliar projects
AcceptEdits	Reads, file edits, common filesystem commands (`mkdir`, `touch`, `rm`, `rmdir`, `mv`, `cp`, `sed`) in working dir	All other shell commands; writes outside working dir; protected paths	Good for trusted local work; not for running scripts
Plan	Reads only	All edits/commands until plan approved	Exploration on sensitive/unfamiliar code; not for tasks needing output
Auto	Everything, but a classifier reviews each action and blocks escalation/hostile intent	Production deploys, migrations, mass deletes, credential exfiltration, force-push to main	Research preview — not a safety guarantee; varies by plan/model
DontAsk	Only pre-approved allow-listed tools + read-only commands	Everything else auto-denied	Built for locked-down CI/scripts, not for reducing local friction
BypassPermissions	All tool calls, no prompts, no safety checks	Nothing (except catastrophic `rm -rf /` or `rm -rf ~`)	Only safe in isolated/disposable containers — never on a live dev workstation

3.4 Settings Configuration Hierarchy

3.4.1 Configuration Hierarchy

Level	Location	Scope	Use For
User	`~/.claude/settings.json`	Every project on the machine, not committed	Personal defaults (e.g., preferred mode)
Project	`.claude/settings.json` (committed)	Everyone who clones the repo	Team-wide conventions, allow/deny rules
Local project	`.claude/settings.local.json` (git-ignored)	Personal overrides for one project	Individual preferences not meant for the team
Enterprise	`managed-settings.json` (admin-set)	Cannot be overridden by users/projects	Org-wide security controls

3.4.2 Rule Precedence
A deny rule always wins over an allow rule regardless of mode; enterprise-level deny rules are the most durable — they can't be removed by any developer and apply even under bypass mode.

3.5 Placing the Human Review Gate

3.5.1 The Governing Question & Gate Placement
Ask "what's the worst outcome if this action runs unchecked?" — let low-stakes reversible actions through, gate hard-to-undo/sensitive-path actions via deny rules or prompts, and always require human review before merging changes to team-flagged sensitive code.

3.5.2 Postmortem: BypassPermissions Removed a Safety Prompt
Bypass mode skipped the confirmation prompt and protected-path guard that would have caught an overly broad file-pattern match, causing accidental deletion of production config files. Mitigation: set explicit deny rules on sensitive directories first; prefer classifier-gated modes (e.g., Auto) over full bypass.

3.6 Durable Project Context: CLAUDE.md

3.6.1 CLAUDE.md Basics & /init
CLAUDE.md at the project root loads into every session, prepended before any user message, so conventions/constraints/commands persist without restating; /init generates a starter file by scanning the codebase (validate before relying on it).

3.6.2 Size Dilution Failure Mode
As the file grows, each instruction becomes a smaller fraction of loaded context, reducing the chance any single rule is followed — keep CLAUDE.md to behavior-changing constraints and move everything else into on-demand Skills.

3.6.3 Postmortem: 847-Line CLAUDE.md
A real path restriction (line 347) was diluted among hundreds of unrelated lines (historical logs, archived notes) and the agent violated it. Fix: keep it as a working rule set, move path-specific rules to rules files, historical context to reference docs, non-negotiable constraints to hooks.

3.7 Rules Instruction Files

3.7.1 Rules Files: Path-Scoped Context
Live in .claude/rules/, scoped via a paths glob in YAML frontmatter, loading into context only when Claude works with matching files — avoiding CLAUDE.md-style dilution.

3.7.2 Scoping Comes from Frontmatter, Not Location
A rules file without a paths field loads unconditionally at launch (same priority as CLAUDE.md) regardless of subdirectory; pattern is broad/universal → CLAUDE.md, narrow/path-specific → rules files (e.g., .claude/rules/database.md with paths: ["src/db/**/*.sql"]).

3.8 Hooks

3.8.1 Hooks: Deterministic Lifecycle Control
Hooks intercept/control tool calls at fixed lifecycle points deterministically, unlike a CLAUDE.md instruction the model might follow inconsistently — defined in settings files, configured via /hooks.

3.8.2 Hook Events

Event	Timing	Can Block?	Use For
`PreToolUse`	Before tool call executes	Yes — exit code 2 blocks it, stderr shown to agent	Access control enforcement
`PostToolUse`	After tool call completes	No	Formatting, tests, audit logging
`UserPromptSubmit`	On prompt submission, before processing	—	Inject context, validate request
`Stop`	When model finishes responding	—	Notifications, cleanup, audit commits
`Notification`	On Claude Code notifications (permission requests, 60s idle)	—	Route to external channel/logging
`SessionStart`	Session start/resume	—	Initialize state, validate env vars
`SessionEnd`	Session end	—	Teardown, final audit writes

3.8.3 Hook vs. Convention
A PreToolUse hook enforces a constraint (e.g., blocking edits to a production config path) at every tool call, every session, regardless of permission mode — the difference between a guardrail (hook) and a convention (CLAUDE.md instruction).

3.9 Subagents

3.9.1 Subagents: Isolated Context
Specialized assistants that run tasks in an isolated context — no inheritance of main conversation history, accumulated files, or session state; return only their output.

3.9.2 Built-in vs. Custom Subagent Behavior
Built-in Explore/Plan subagents skip CLAUDE.md and git status (optimized for speed, so project rules don't apply); general-purpose loads both. Custom subagents don't auto-inherit skills — list needed skills explicitly in the subagent's frontmatter (.claude/agents).

3.9.3 Mechanism Comparison

Mechanism	Loads	When	Context Cost	Belongs
CLAUDE.md	Full file, prepended	Every session	Persistent, dilutes with size	Universal constraints/commands
Rules file	File contents	On matching file access (or session start if unscoped)	Path-scoped: only when triggered; unscoped: same as CLAUDE.md	Path-specific guidance
Hook	Runs a script	At configured lifecycle event	Minimal	Guardrails, automation, audit
Subagent	Task context only	When dispatched	Returns summary only	Exploration/investigation, parallelizable work

3.10 Packaging Workflows: Skills

3.10.1 Skills: Portable Markdown Procedures
A Skill is a portable SKILL.md in .claude/skills (frontmatter identifies/describes it, body holds steps); the same file can run in Claude Code, via Messages API, or via Agent SDK — only where it runs and what it can access changes.

3.10.2 Skill Runtimes

Runtime	How It Loads	Where Steps Run	Key Requirement
Claude Code	Filesystem discovery (description match or invoke by name)	Local terminal, under active permission mode/deny rules	Filesystem-based, governed by settings layer
Messages API	Sent with request, runs in code execution container	Anthropic's container, not local machine	Requires code-execution + skills beta headers; no local file/tool assumptions
Agent SDK	Loaded by agent; controlled by `settingSources` (TS) / `setting_sources` (Python)	The SDK's own process, once filesystem sources are enabled	Must explicitly set `settingSources` — no reliable default
Claude Managed Agents	Defined once as an API resource (model, prompt, tools, MCP, skills)	Anthropic-provisioned sandbox	Requires `managed-agents-2026-04-01` beta header; not ZDR/HIPAA-eligible; skills attached at agent-definition time

3.10.3 Three Portability Rules
(1) Write the skill's description as a precise matching criterion; (2) don't assume local filesystem/tools exist — document dependencies explicitly; (3) subagents don't inherit skills in any runtime — must be explicitly listed.

3.11 Custom Commands & Plugin Packaging

3.11.1 Custom Commands (Legacy vs. Skills)
Skills are now the recommended format for both explicit (/skill-name) and automatic invocation; the older .claude/commands/ format still works but is legacy. Use disable-model-invocation: true in frontmatter for commands that should only run when explicitly called.

3.11.2 Plugin Namespacing
A plugin name becomes the command prefix (e.g., /payments:run-tests), preventing collisions across plugins; renaming a plugin renames all its commands.

3.11.3 Plugins & Marketplaces
A plugin is a versioned bundle of skills, hooks, subagents, and MCP servers distributed via a marketplace. Anthropic's official marketplace is available by default; third-party ones are added via /plugin marketplace add <owner/repo>. Enterprise admins can deploy plugins org-wide via managed settings, gated by a managed marketplace allowlist, paired with extraKnownMarketplaces to auto-register for all users; managed-scope settings sit above user/project settings and cannot be overridden.

3.11.4 Packaging Decision Table

Layer	Reach For When
Skill	Task-specific procedure should stay out of context until needed
Custom command	Predictable, explicit high-frequency entry point wanted
Plugin	A working local setup needs to be shared/versioned across a team

3.11.5 Postmortem: Plugin Portability Failure
A plugin skill referenced an author's absolute local path and an undocumented env var — installed fine everywhere, but execution failed for every teammate. Fix: use $CLAUDE_PROJECT_DIR / ${CLAUDE_PLUGIN_ROOT} for paths, bundle dependent assets, document/validate env vars at install, and test on a clean machine; note a locally-relied-on deny rule/hook isn't auto-included in a plugin bundle.

3.12 Building an MCP Server

3.12.1 What MCP Is
MCP (Model Context Protocol) separates tool definitions from individual applications into a standalone server process exposing tools, resources, and prompts to any connecting client — build once, reuse everywhere.

3.12.2 Tools, Resources, Prompts
Tools are actions the model can call; Resources are read-only data fetched directly into context (not via a tool call) — direct (fixed address) or templated (parameterized), used when cheap predictable injection beats a tool call (client support varies); Prompts are pre-written, vetted instruction templates exposed by name for consistent wording across clients.

3.13 MCP Transport

3.13.1 Transport Options

Transport	Mechanism	Use For
stdio	Local subprocess via standard input/output	Local tools, personal scripts, dev servers on your own machine — not shareable/remote
HTTP	Standard network connection via URL	Recommended for any non-local server — shared team servers, hosted integrations
SSE	Legacy transport predating HTTP	No longer recommended for new servers; treat as legacy if encountered

3.13.2 Context Cost & Tool Discovery
Connected servers' tool definitions would occupy context if loaded upfront; Claude Code defers loading and searches to discover/load only relevant tools per task by default (an opt-in mode loads upfront if definitions fit within ~10% of the context window). Connect only needed servers to keep requests lean.

3.14 Prompt Caching (Applied to MCP/Tool Definitions)

3.14.1 How Prompt Caching Works
Caches processing of a stable request prefix so follow-up requests reuse it at a fraction of cost; the first request writes to cache, and an exact match is required — a single changed character before the cache point invalidates it. Best candidates: long system prompts, large tool-definition sets, frequently-queried reference docs.

3.14.2 Cache Configuration Details
Enabled via cache_control: {type: ephemeral} on the last block to cache (up to 4 breakpoints); request order is fixed (tools → system prompt → messages), so a breakpoint after tools caches definitions while messages stay dynamic. Default lifetime is 5 minutes from last read; opt-in 1-hour via ttl: "1h"; a minimum token threshold applies (1,024 for most current models).

3.15 Retrieval-Augmented Generation (RAG) — Two Forms

3.15.1 Classical RAG vs. Agentic Search
Classical RAG pre-chunks and embeds source material into a database, matched by similarity at query time (index built in advance); agentic search has no pre-built index — the model searches/fetches live sources on demand (e.g., MCP tool discovery, Projects surfacing relevant sections). Both retrieve a relevant slice and generate from it — the difference is timing.

3.15.2 Two Properties of Retrieval
(1) Scales flat — request cost stays roughly constant as source material grows, since only a relevant slice is retrieved per query; (2) quality depends on retrievability — if retrieval misses the needed document the model never sees it, so well-named, well-organized files matter.

3.16 MCP Configuration Scope

3.16.1 Scope Levels

Scope	Location	Applies To	Use For
Local	`~/.claude.json` (per-project entry)	Current project only, not shared	Project-specific, not-yet-shared config
User	Personal Claude settings	All your projects, still personal	Personal utilities used everywhere
Project	`.mcp.json` (committed to repo)	Everyone who clones the repo	Team-wide shared server access
Enterprise	Centrally managed config (admin-controlled)	Entire organization	Shared internal services, security tooling

3.16.2 Note on Project-Scoped stdio Servers
A project-scoped stdio server still runs from each teammate's own machine — each clone needs the runtime (e.g., Node for an npx-launched server) installed locally.

3.17 MCP Tool-Level Permissions

3.17.1 Per-Tool Permission Rules
MCP tools are identified as mcp__server__tool, so rules can target individual tools rather than the whole server — an allow rule on one tool lets it run without prompting while others on the same server still prompt; a deny rule on a write-capable tool blocks it while read-only tools remain available, and deny always overrides allow.

3.17.2 API MCP Connector: Scope vs. Governance
An mcp_toolset object with a per-tool enabled flag controls whether the model even sees a tool (context/scope control), distinct from a permission rule controlling whether an exposed tool may run (governance control) — often used together; verify exact syntax/beta headers against docs.

3.18 GitHub MCP Server — Worked Authentication Example

3.18.1 GitHub MCP Setup
Transport: HTTP (remote, hosted by GitHub), registered via server URL; scope: Project for whole-team access, local for individual use; auth: a Personal Access Token passed as a Bearer token, supplied via environment variable and referenced in config — never committed inline (a committed token enters repo history permanently).

3.18.2 OAuth Alternative
For services authenticating individual users via browser sign-in (e.g., Linear MCP) — client redirects to the provider's sign-in page, token issued/stored automatically after approval, no manual credential handling; right pattern when authorization is tied to user identity.

3.18.3 MCP Setup Reference

Context	Transport	Scope	Secrets Handling
Personal local tool	stdio	Local	Env vars only, never in config file
Shared team server	HTTP	Project (`.mcp.json`)	OAuth or env vars; never committed
Personal experiment	stdio/HTTP	Local	Env vars only
Org-wide deployment	HTTP	Enterprise	Admin-managed secrets, config locked

3.19 Enterprise Authentication Patterns

3.19.1 Auth Method by Service Type

Service Type	Auth Method	Secret Location
Remote, user identity (SaaS/cloud)	OAuth (server returns 401 → browser sign-in)	Token issued/stored by client
Remote, service identity (internal API)	API key via environment variable	Environment only, injected by CI/pipeline runner
Local, filesystem access	stdio, no network auth	Filesystem permission model + deny rules as governance

3.19.2 Secret Management: Three Practices
(1) Separation — config holds only a variable reference, never the value, since committed values persist in repo history; (2) storage — env var for local/short-lived secrets, a secret store for shared/audited ones (enables single-point rotation); (3) rotation — replace on a schedule and immediately after suspected exposure, scoped narrowly per credential.

3.20 Regulated/Enterprise Deployment Requirements

3.20.1 Beyond Authentication: Three Requirements
(1) Configuration lock — enterprise managed settings prevent developers from overriding auth setup; (2) audit logging — a PostToolUse hook logging every tool call/parameters, firing deterministically and unskippable by the model; (3) data residency — an HTTP endpoint pinned to a region plus a platform deployment enforcing regional processing.

3.20.2 Authentication/Integration Checklist

Service Type	Auth	Secrets	Logging	Config Lock
Remote, user identity	OAuth	Client-stored token	PostToolUse hook	Enterprise managed settings
Remote, service identity	API key (env var)	Environment only	PostToolUse hook	Enterprise managed settings
Local	Filesystem permissions	None needed	PostToolUse hook	Deny rules in managed settings

3.20.3 Postmortem: OAuth Staging-to-Production Failure
OAuth redirect URIs are registered per host, so a working staging connection doesn't guarantee production works — the production host's URI isn't automatically authorized, and regulated customers often require separate app registrations per environment. Fix: add the new host's redirect URI before cutover and include this in the deployment checklist.

3.21 Code Modernization as an Applied Case Study

3.21.1 Modernization Risk Profile
Legacy modernization concentrates high blast radius, unpredictable dependencies, and limited reversibility; the Explore/Plan/Code loop and Plan mode hold the agent in read-only review, hooks guard sensitive paths during high-risk phases, and CLAUDE.md carries target-pattern conventions so the agent doesn't drift back to legacy patterns.

3.21.2 Three Scoping Questions
Before high-risk agentic work, ask: (1) what's the blast radius — which systems depend on this code; (2) how are changes audited — is a PostToolUse hook logging every tool call sufficient; (3) who approves each phase before the next begins — Plan mode enforces the explore/execute boundary, but the approval process itself must be defined beforehand.

Module 4: Production Engineering, Evals & Security

4.1 Module Introduction

4.1.1 Core Theme: Production Reveals What Development Hides
Success criteria never written down, retriable errors never given a path, budgets never instrumented, action boundaries never enforced — these are decisions, not bugs, and must be made on paper before failures happen live.

4.1.2 The Design Document — Four Decisions

Decision	What It Defines
Success criteria	Concrete, checkable output definitions (e.g., "a two-sentence summary listing every action item and owner") — what the eval set is built from
Failure handling	Expected production errors, each marked retriable/terminal, and what the user sees on unrecoverable failure
Cost and latency budget	Per-request budget, monthly cost ceiling, latency target, minimum reliability floor — set before architecture
Trust boundary	Which inputs are untrusted, and the smallest set of actions/access the feature needs — turns least privilege into an enforceable hook, not a remembered setting

4.2 Evals & Judges

4.2.1 What an Eval Is
A fixed set of input cases + expected behavior + grading, run and averaged into a trackable score — turns "done" from a feeling into a number. Write the eval before building the feature, forcing success to be defined upfront.

4.2.2 Minimal Eval Pipeline
Load dataset → run each case → grade output → average scores. Change one variable (prompt, tool, or model) at a time between runs so you know what caused any score movement.

4.2.3 Three Grading Methods

Method	Best For	Catches	Blind Spot
Exact/string match	Output with exactly one correct form	Wrong answer, zero ambiguity, near-zero cost	Fails on any valid paraphrase or reordering
Code-graded check	Structured/code output	Invalid JSON, unparseable code, out-of-range values, missing fields	Confirms well-formedness only, not content quality
LLM-as-judge	Open-ended quality (faithfulness, instruction-following, tone)	What no code rule can express	Noisy, costly, produces a confident-looking but meaningless number until calibrated

4.2.4 Cost Dimension
Exact match/code checks run locally at near-zero cost (thousands per change); a judge is a second model call per case — grade format/structure with code on every commit, reserve judge calls for slower scheduled quality passes.

4.2.5 Building/Calibrating a Judge
Prompt the judge for strengths, weaknesses, and reasoning alongside the score, or it drifts to a "safe middle" (~6) regardless of quality. Calibrate by running it against a human-labeled set and measuring agreement — low agreement means an untrustworthy score, fixed by tightening the rubric.

4.2.6 Coverage Over Perfection
A larger, noisier automated eval set usually reveals more than a small hand-graded one; use Claude to generate additional edge cases from a small labeled starting set, then spot-check them.

4.2.7 Iteration Workflow
Set goal → write initial prompt → run eval → read per-case failures → apply one change → re-run. Per-case breakdown matters as much as the average (can hide fixes canceling breaks); categorize failure cause (formatting, retrieval, long-input) to target the next fix.

4.2.8 Postmortem: Field-Extraction Two-Date Failure
A feature passed ~12 manual checks (all single-date messages) and shipped with shape-only validation; a message with two dates extracted the wrong one, since validation confirmed shape, not correctness, and no holdout set covered that case. Fix: define graded cases including model-generated edge cases before shipping — the eval guards against regressions, it doesn't fix the underlying prompt.

4.3 Testing & Tracing

4.3.1 Four Test Levels

Level	Isolates	Cannot Catch
Unit	One function (parser, tool wrapper) alone	How components fit together
Functional	One Claude call returns expected shape for an input	Failures in the surrounding system
Integration	The handoff/seam between two components (e.g., retrieval → model)	Whole-flow behavior emerging only end-to-end
End-to-end	Full flow as a user would run it	Where the break is — slowest, hardest to localize

4.3.2 Integration Seam Failures
Most silent production failures live at the integration seam — each side passes its own tests while the handoff between them is broken (e.g., one returns a list of dicts, the other expects a plain string).

4.3.3 Tracing
Records each run step (prompt, tool calls, intermediate outputs, timing), turning "the case failed" into "step 4: parser raised KeyError on a field the model didn't return" — a 5-minute fix instead of a day of investigation.

4.3.4 Retrieval Routing (Cost-Aware)
A cheap classifier routes simple lookups to fetch-once retrieval and multi-part questions to iterative/agentic search, avoiding defaulting everything to the expensive or shallow path. Skip the router if all traffic is one shape.

4.3.5 Postmortem: Unit/Functional Passed, End-to-End Failed
retrieve() returned a list of dicts while build_prompt() expected a plain string, causing malformed context and the model to answer from memory instead of retrieved content — only an integration test exercising the real handoff would catch this.

4.4 Failure Handling & Model Selection

4.4.1 Retriable vs. Terminal

Category	HTTP Status Codes	Examples
Retriable	429, 529, 500, 502, 503, 504	Rate limit, overload, transient server fault, timeout
Terminal	400, 401, 403, 404	Bad request, auth failure, missing resource, permissions problem

Misclassifying as terminal fails loudly and gets fixed (safe default when unsure); misclassifying as retriable hammers the service and hides the real problem.

4.4.2 SDK-Level Retries
Anthropic client libraries auto-retry transient failures with progressive delays up to a configurable cap — avoid stacking a custom retry loop on top (multiplies attempts against a rate limit); either let the SDK own retries or turn them off and own the whole path.

4.4.3 retry-after Header
Present on 429/529 responses, gives the exact wait time — more precise than blind backoff. Read it first; fall back to exponential backoff only when absent.

4.4.4 Tool Errors & Refusals
Tool errors must return to Claude explicitly with is_error: true — a silent empty result is treated as valid data, producing a confident-but-wrong answer. A refusal (stop_reason: "refusal") is a 200 at the HTTP layer, so the retriable classifier won't catch it — treat it as terminal, raise and log, never retry.

4.4.5 Error-Handling Decision Table

Error	Retriable?	Strategy	Fallback
Rate limit (429)	Yes	Exponential backoff + jitter, honor `retry-after`, capped attempts	Clean error or cached/simpler result after cap
Overloaded (529)	Yes	Backoff (Anthropic-side load, not a rate-limit signal)	Fail over or graceful error if persists
Bad request (400)	No	No retry	Fix/reject input, surface to caller
Tool result error	Depends	Retry only if cause is transient	Return error flag to Claude, never silence
Refusal (200, `stop_reason: refusal`)	No	No retry	Raise to caller, log, never treat as valid output

4.4.6 Postmortem: Unhandled Rate Limit + Retry Storm
A loop with no error handling worked in low-volume dev; the first production 429 raised an unhandled exception, and the instinct to add immediate retries deepened the rate limit. Fix: honor retry-after first, fall back to capped exponential backoff with jitter, and fail fast on terminal statuses (400/401/403/404).

4.4.7 Model Selection in Production
Family order: Fable (hardest reasoning/coding/agentic) → Opus (above Sonnet's envelope) → Sonnet (balanced default) → Haiku (speed/cost-optimized). Start with Sonnet, move up only when an eval shows a missed quality bar, move down only when an eval shows an acceptable drop — defaulting to the most capable model "just in case" is the most common, expensive mistake. Route bulk traffic to a default model, override via a cheap signal (task type, length, difficulty) only where needed.

4.5 Cost & Orchestration

4.5.1 Observability: Three Metrics
Instrument token usage (input/output), latency, and error rate per call from the start — per-call logging turns "why is the bill high?" into "which step, on which request type, is responsible?"

4.5.2 Cost/Latency Levers: Model Selection & Streaming
Route simpler work to smaller/faster models, reserve capable ones for steps that need them. With streaming + tool use, accumulate content_block_delta events by index and never act on a tool_use block until the stream closes — a broken mid-stream response needs a full retry, not partial output passed downstream.

4.5.3 Prompt Caching Economics
Cache writes cost a premium (1.25x base for 5-min TTL, 2x for 1-hour); cache reads cost ~0.1x standard input — only pays off when reads outnumber writes. Two modes: automatic (single flag, system manages breakpoints) or explicit cache_control breakpoints.

4.5.4 Message Batches API
Async processing at lower per-request cost in exchange for non-immediate completion — right for non-urgent, high-volume work (overnight jobs, backfills), wrong for anything a user is waiting on; compounds with prompt caching when a batch reuses shared context.

4.5.5 Multi-Agent Orchestration (Orchestrator-Worker)
A lead agent decomposes a task, delegates to parallel subagents, then synthesizes results — genuinely helps independently-splittable tasks (e.g., research) but Anthropic's internal eval showed it costs ~15x a normal chat interaction, and is less effective on tightly coupled tasks like coding where steps can't be parallelized. Use a capable lead with cheaper subagents to reduce the multiplier; failure handling multiplies with agent count.

4.5.6 Reliability Floor
Define a base (retry budget, latency ceiling) first, then tune cost above it, never below — cost pressure is more visible daily than reliability pressure, so the eval's pinned baseline score is what makes the floor enforceable before shipping.

4.5.7 Postmortem: Fan-Out on a Tightly-Coupled Task
Parallel fan-out applied to a sequential, dependent-step task tripled the bill with barely improved quality, since subagents mostly waited on each other. Reverting to single-agent restored cost with equal quality.

4.6 Security

4.6.1 Prompt Injection: The Core Threat
The model processes its entire context as one undifferentiated token stream — no structural boundary separates trusted instructions from untrusted data, so hidden instructions in fetched content are read as commands.

4.6.2 Defense: Treat Content as Data
Treat all fetched/user-supplied content as data to examine, never instructions to follow — trusting your own users doesn't help, since the hostile instruction typically arrives via retrieved content. Delimiter-wrapping helps but is a soft boundary; the reliable boundary is what the agent is allowed to do.

4.6.3 Threat Scope & Jailbreak vs. Injection
Any content someone else can write is a vector (shared docs, DB records, emails, chained tool outputs), direct or hidden. Jailbreaks target the model's own safety constraints; injections hijack the application's instructions — both need the same layered defense.

4.6.4 Secure-by-Design Identity and Access
Agent identity carries only the narrowest permission set the task requires; secrets live in env vars or a secret manager, never committed config. Least privilege bounds the blast radius of a successful injection; committed secrets are permanent exposures since only rotation (impossible for a hardcoded value) fixes a leak.

4.6.5 Hook-Based Guardrails
A PreToolUse hook blocks a tool call before execution and logs every privileged action — enforced control vs. an unenforced prompt-level rule. Precedence when rules conflict: deny > ask > allow.

4.6.6 Scoping for Regulated Review
Three early questions: where is data processed (residency), how is access logged (per-action audit trail), and can configuration be centrally administered (prevents developers quietly widening permissions).

4.6.7 ZDR Note
Zero Data Retention eligibility varies by model and platform, not guaranteed even under an existing agreement — confirm against Anthropic's Trust Center (and Bedrock/Vertex/Foundry retention policies separately) at scoping time.

4.6.8 Layered Security Model
Model training + classifiers (reduce injection landing) → treating content as data (reduce acted-upon injections) → least privilege + locked config (bound blast radius) → hooks (enforce + record) → regulated-review scoping (make it auditable) — no single layer is sufficient alone.

4.6.9 OS-Level Sandboxing
Isolates at the process level regardless of hook/identity config: filesystem isolation restricts the agent to its working directory, network isolation restricts outbound connections to a named endpoint set. Holds even when a hook is missing or bypassed — typically the first thing enterprise security reviewers ask about.

4.6.10 Postmortem: Hidden Instruction Redirected a Write
An internal-only agent skipped validating fetched web content, assuming trusting the user meant trusting the request; a hidden instruction in a fetched page redirected the agent's write to an unintended path. Fix: treat fetched content as data plus a PreToolUse hook denying writes outside the permitted path.

4.6.11 Defense Checklist

Threat	Entry Point	Control	Logged
Prompt injection	Hidden instructions in fetched content	Treat as data + hook refusing untrusted-triggered actions	Source, attempted action, block
Jailbreak	Crafted user prompt	Input validation + model action constraints	Flagged prompt, refusal
Over-broad access	Identity scoped wider than needed	Least privilege, secrets manager, locked auth config	Every privileged action + identity
Sandbox escape	Steered agent reaching uncovered paths/endpoints	OS-level filesystem/network isolation	Every denied access attempt + trigger

4.7 Cumulative Task

4.7.1 Cumulative Task Structure
A single runnable application containing three planted defects, one per layer (eval/testing, failure-handling, security) — e.g., an agent that writes based on untrusted fetched content without a hook, a retry loop with no real backoff that retries terminal errors too, and no is_error handling on tool results.

Module 5: Accelerators & IP Contribution

5.1 Module Introduction

5.1.1 Core Theme
A build that works is not yet a build that survives reuse, review, or deployment — templates aren't configurable, contributions aren't verifiable by a stranger, models aren't pinned, platforms haven't cleared compliance, and trust boundaries haven't been mapped. Much of this work is driven by the customer's cloud/compliance posture, not technical preference.

5.2 Packaging for Reuse

5.2.1 Accelerator Definition
A solution packaged so future engagements start from a working foundation instead of a blank repo — separating engagement-specific code from a parameterized reusable core. Package while the build is fresh, before the reasoning behind hardcoded values is lost.

5.2.2 Three Asset Types

Asset Type	What It Bundles	Correct Packaging Requires
Agent Template	System prompt, tool schemas, loop structure	Pull domain values into configuration with documented defaults — new team sets values, doesn't edit the loop
MCP Server Package	Exposed tools, their inputs, controllable scope	Document each tool input; let the installing team set scope — installs without code edits
Eval Suite	Graded test set + judge rubric	Ship dataset and rubric together as the deployment gate (run against a pinned baseline before promoting a new model version)

5.2.3 Common Failure: Loose Scripts
Shipping an agent as loose scripts instead of a template looks reusable because it "runs," but customer-specific values stay buried across files, so the next team copies and diverges instead of configuring one asset.

5.2.4 Documentation & Audit Bundling
Documentation must cover environment assumptions, expected inputs, handled failure modes, and the defining eval, or the next team treats the asset as a black box. Bundle the audit log (data touched, identity acted under) too — a regulated reviewer asks for this at the first security review.

5.2.5 Packaging Checklist

Asset Type	Parameterize	Document	Bundle for Audit
Agent template	Prompts, paths, scopes, credentials by reference, thresholds	Environment assumptions, expected inputs, handled failures, defining eval	Data touched, identity acted under, action log
MCP Server	Scopes, credentials by reference, per-customer paths	Expected inputs per tool, scope boundaries, handled failures	Data touched, identity acted under, action log
Eval Suite	Thresholds, dataset paths	Rubric logic, what scores mean, pinned baseline	Data touched, identity acted under, action log

5.2.6 Postmortem: Hardcoded Values Labeled Reusable
A team hardcoded customer-specific values into a template to hit a deadline, then labeled it "reusable"; a second team couldn't configure it (no parameters, no documentation, no bundled eval) and had to rewrite it fully. A template that runs has not been packaged for reuse — these are different finishing states.

5.3 Contributing Back

5.3.1 Definition
Moving an asset from private reuse to shared infrastructure through a documented channel carrying version, install steps, and components as one unit — an asset already packaged for internal reuse is already close to what a maintainer needs.

5.3.2 Matching Contribution to Channel
The Claude Cookbook takes self-contained, single/multi-pattern reference implementations demonstrated end to end, not a full application; open-source MCP servers/tools each have their own repo and conventions. Misplacement — sending a full multi-component app to the Cookbook — is one of the most common reasons a contribution never gets reviewed.

5.3.3 Four Things That Make Verification Possible
(1) Does one thing, (2) an example shows it running, (3) a test proves it works, (4) a short statement names the assumptions — the bar is set by what needs checking, not code cleverness.

5.3.4 Rights and Attribution Come First
Licensing decides whether a contribution can be accepted at all; code carried in from a customer engagement may have constraints on where it can go. Confirming the right to contribute and attributing prior work is a gate that must be passed before technical review.

5.3.5 Contribution-Readiness Reference

Channel	What a Maintainer Checks	Licensing/Attribution	Example/Test Bar
Cookbook (focused example) or tool/server's own repo	Code does one thing, fully readable	Confirm right to contribute engagement code, prior work attributed	Runnable example + a test proving behavior, not just description

5.3.6 Postmortem: Unreviewed PR for Three Weeks
A PR sat unreviewed because it had no test, no example, and no stated environment assumptions — the maintainer couldn't verify it without reconstructing the developer's work. A contribution the reviewer can't verify sits at the back of the queue regardless of code correctness.

5.4 Requirements & Lifecycle

5.4.1 From Business Problem to Functional Requirements
A functional requirement states what the system must do with enough detail to check (e.g., "classify each ticket into one of four queues... never auto-send without human approval," not "help agents answer faster") — a specific goal becomes an eval line and a review criterion.

5.4.2 Deriving Infrastructure Requirements
Non-functional constraints derived by asking: latency (how fast, measured where?), scale (how many requests, at what peak?), residency (where must data be processed?), identity (who acts, under what credentials, what's auditable?) — these four most often decide the deployment platform.

5.4.3 Documenting Requirements
A short record of functional behaviors, infrastructure constraints, and the regulation each derives from lets a platform choice be defended as following from requirements rather than familiarity.

5.5 Systems Lifecycle for Claude Applications

5.5.1 Seven Phases
Requirements (capture functional/infrastructure needs) → Design (platform, model, trust boundaries) → Build (agent, tools, prompts) → Test (evals, unit/integration/e2e) → Deploy (pin version, gate on eval) → Operate (instrument cost/latency/errors, enforce guardrails) → Iterate (feed production findings back into requirements).

5.5.2 Gating Between Phases
A gate is the decision point to move between phases where a regulated engagement retains control (e.g., don't move design→build until the platform satisfies residency). Refusing to skip a gate is what keeps an application reviewable; a one-off experiment may collapse phases, a regulated deployment cannot.

5.6 Deployment & Versioning

5.6.1 Platform Choice Driven by Customer's Cloud

Platform	Identity/Data Model	When to Choose
First-party Claude API	Anthropic identity and terms	No binding cloud/residency constraint; wants newest capabilities first
Claude Platform on AWS	Anthropic identity/terms via customer's AWS account; inference outside the AWS boundary	On AWS but wants Anthropic model IDs/lifecycle parity with first-party API
Claude in Amazon Bedrock	Messages API at `/anthropic/v1/messages`; data stays inside customer's AWS boundary	On AWS, wants feature parity + compliance posture there
Claude on Amazon Bedrock (legacy)	AWS identity/billing; `InvokeModel`/`Converse` APIs, ARN-versioned IDs	Existing (unmigrated) Bedrock integration
Google Vertex AI	Google Cloud identity/IAM/billing; regional or global endpoints	On Google Cloud with compliance posture there
Third-party (e.g., Microsoft Foundry)	Wrapping product's identity/billing	Already runs the platform embedding Claude; residency depends on hosting form (Azure-hosted vs. Anthropic-hosted) per model

5.6.2 Identity and Residency Answered by Platform
Bedrock uses AWS identity and keeps data in the customer's AWS boundary; Vertex uses Google Cloud identity/boundary; both offer regional routing. Matching platform to the customer's existing compliance agreement avoids a residency review from scratch.

5.6.3 Pinning Versions
Every model ID points to a specific snapshot; aliases (e.g., "Opus," "Sonnet") evolve over time, so pin the full model ID, not the alias (e.g., claude-haiku-4-5-20251001 vs. the moving claude-haiku-4-5), and version prompts/assets alongside code with a rollback-ready prior version.

5.6.4 Promote via the Eval
Send a new version to a portion of traffic, compare against the pinned baseline, promote or roll back on the result — the eval is the deployment gate, not just a one-time test.

5.6.5 Deployment-Platform Versioning

Platform	Versioning
First-party API	Pin full model ID, keep prior snapshot
Claude Platform on AWS	Same ID format as Claude API; lifecycle follows Anthropic's schedule
Claude in Amazon Bedrock	Pin full model ID with `anthropic.` prefix; partner retirement dates differ
Claude on Amazon Bedrock (legacy)	Pin via ARN-versioned identifiers
Google Vertex AI	Pin full model ID before rollout; partner retirement dates differ
Third-party platform	Pin per the platform's own versioning controls

5.6.6 Postmortem: Moving Alias Broke Production
A deployment shipped against a moving alias ("opus"); it silently advanced to a new version, breaking downstream parsing with no pinned prior version to roll back to, forcing a hotfix instead. Lesson: pin the full model ID, retain the prior pinned version, gate promotions through the eval.

5.7 Comparing Platforms

5.7.1 Latency
Depends on platform location relative to the customer and feature-access timing (first-party API usually gets new capabilities first) — must be measured from the customer's actual region/payload. Within Bedrock, global vs. regional endpoints is the primary residency control and can affect cost.

5.7.2 Compliance Often Ends the Debate
A customer already certified on one cloud is unlikely to re-certify on another; residency/certifications/audit access differ by platform and are pass-or-fail for regulated customers. First-party API may lack EU residency (Bedrock/Vertex typically required); on Foundry, hosting is per-model. Raise compliance constraints at scoping, not at contract review.

5.7.3 Cost Beyond Per-Token Rate
Token rates are broadly aligned across platforms; total cost is driven by egress, platform fees, and integration effort — instrument cost per call per platform rather than comparing token price alone.

5.7.4 Cross-Platform Comparison Reference

Dimension	How It Differs	How to Measure	Winner
Latency	In-region platform shortens round trip; first-party API gets features first	From customer's actual region + payload	In-region cloud wins on latency; first-party wins on feature access
Compliance	Residency, certifications, audit controls vary by platform	Against customer's existing certification/residency requirements at scoping	The already-certified platform wins
Cost	Token price, egress, platform fees, integration effort all vary	Total cost per call including egress/integration	Lowest total cost for the actual workload

5.7.5 Postmortem: Familiar Platform Failed Residency Review
A team picked the platform they knew best for a regulated customer since migration was fast; it passed functional tests but failed the customer's residency requirement at security review, requiring a rebuild. Familiarity optimizes for build speed, not for whether the deployment is allowed to ship.

5.8 Trust Boundaries

5.8.1 Multi-Component Coordination
An app might chain an API request → Claude Code task → MCP server reaching a customer system; each connection creates a place where identity, secrets, and untrusted input can cross — map what each component does before connecting anything.

5.8.2 Least Privilege Applies to the Whole Application
Each component operates under its own identity, scoped to only what its task needs — the application is only as contained as its most privileged seam, so one overly-broad component becomes the weak point even if others are properly scoped.

5.8.3 Regulated Review Requirements
Requires justifying audit logging, data-residency decisions, and permission controls across the full application, not per component — confirm ZDR/HIPAA BAA eligibility for each individual component against the Trust Center before scoping.

5.8.4 Multi-Component Integration Map

Component	Contributes	Trust Boundary at Seam	Control
First-party API	Orchestrates workflow, holds entry point	Request entering the app from outside	Input validation + identity the call runs under
Claude Code task	Runs agentic work, may fetch external content	Content it fetched — untrusted downstream	Treat fetched content as data at the next seam
MCP server	Reaches a customer system to read/act	System access held on the app's behalf	Scope to least privilege + log the access

5.8.5 Postmortem: Untrusted Content Passed as Trusted Input
Three components each passed their own tests; content fetched by the Claude Code task was passed directly into the next call as trusted input with no boundary control at that seam — a hidden instruction there would have executed. A component being trusted in isolation says nothing about the seam leaving it.

5.9 Cumulative Task

5.9.1 Cumulative Task Structure
A single deployed accelerator with three planted defects: a hardcoded customer-specific value where a parameter belongs (packaging), a moving model alias instead of a pinned full ID (versioning), and fetched untrusted content passed directly into the next call with no boundary control (trust boundary). Task: identify all three, explain the runtime consequence, and write the corrected lines.

Everything I Learned From My Claude Code Training

yash-nigam — Mon, 31 Aug 2026 17:28:38 +0000

Everything I Learned From My Claude Code Training

I recently went through a hands-on training on Claude Code and the broader Claude ecosystem. Below are my complete notes, organized by topic, with screenshots from the session included at each relevant step.

The Claude Ecosystem
- AI-Code-Design-Cowork-Security-Chrome-Office
The Agentic Loop (Query Engine)
- Architecture
Models, Effort, and Slash Commands
- Models-Commands
Session Management
- Export-Fork-Resume-Compact
Project Configuration
- ClaudeFolder-Permissions-Init-ClaudeMd
Auto Mode vs. Plan Mode
- Auto-Plan
Commands vs. Skills vs. Agents
- Standup-Metaprompting-PR-Invocation
Code Review via Subagents
- Security-Performance-Coverage-Quality-Usage-Deploy-Setup-Execution-Tests-Stack-Creation-Example-Focus-OWASP-Memory-Output-Background
Multi-Agent Harness Concepts
- SecondPass-Comparison
Hooks
- Flow-PreToolUse-Types-Blocking-Slack-Sharing
MCP (Model Context Protocol)
- Concept-Rationale-Types
End-to-End Automation
- End-to-end automation of security review → issue creation → issue fix → PR creation → PR fix, all through agents.
Agent Teams
- Multiple agents (security, performance, test) talk to each other and take actions autonomously — filing issues, creating PRs, merging PRs, and closing issues, all automated.
Key Takeaways
- Loop-Config-Commands-Hooks-Sessions-Teams

1. The Claude Ecosystem

Claude Code is one tool within a larger ecosystem, not a standalone product:

Claude AI — the conversational assistant
Claude Code — the agentic coding CLI (the focus of this article), which has:
- An agentic loop that loops until the task is completed
- A tool system: git, bash, read, write files
- A permission pipeline, configurable per tool (Allow / Deny / Ask)
- A context manager: compaction and pruning of context within a session
- Session persistence: work can be paused and resumed later, with context and memory maintained
Claude Design — design tooling
Claude Co-work — collaboration tooling
Claude Security — security-focused tooling
Claude Chrome plugin — browser automation
Claude Office plugin — office document integration

In the backend, Claude Code talks to the LLM hosted by Anthropic via the /v1/messages API.

Diagram of the agentic loop: gather context, take action, verify results, repeat until done.

Tools like Cursor, Windsurf, and Perplexity act as integrators — giving access to multiple LLMs. Claude, by contrast, offers an entire vertically integrated ecosystem.

2. The Agentic Loop (Query Engine)

This is the core engine behind Claude Code. Given a request — e.g. "Create a unit test case for my function" — the loop works like this:

Gather context — goes to the file, reads the imported files also, reads the function in the file (which needs to be mocked).
Send context to the LLM — the LLM decides, based on the context, what unit test cases must be created after it has all the info, and tells Claude Code what actions to take (e.g. "create new test case").
Use tools — Claude Code creates or modifies the file using its tool system.
Verify — the agent checks whether the job is done or not (e.g. runs the test to confirm it passes).

Additional properties:

The user can modify the context in between the loop or after it.
The loop runs continuously until the task is not done.
Everything depends on the LLM being used — so it's worth knowing which LLM is best suited for the task.

Agentic Coding Tools Architecture

Three-layer architecture diagram showing coding agents (Claude Code, Copilot, Codex, Cursor, etc.) between the chat interface and various LLM hosting backends.

Claude Code's runtime architecture, from terminal command through the CLI parser and QueryEngine agentic loop to tool calls and API routing.

Claude Code always connects to the Anthropic base URL.
It can also connect to Ollama and open-source models such as Qwen or Llama 3.2 — enabling a fully local setup. This is currently the only fully local option available.

3. Models, Effort, and Slash Commands

Models, effort levels, and sessions are the main levers for controlling how Claude Code works.

Models

Four main models, chosen based on the type of task and usage:

Fable — for complex tasks: architecture, design, security, coding
Opus
Sonnet
Haiku — for simpler tasks

Qwen-local is the local-model equivalent of Haiku.
opusplan = Opus for planning + Sonnet for doing the task.

Guidance: try a smaller model at higher effort first — otherwise switch to a new/bigger model as required. Use Fable only if Opus cannot handle the task.

Terminal status line showing the active model, session name, context usage, token count, and running cost.

Screen clippings taken: 29-08-2026 02:21 and 29-08-2026 02:14

Slash commands for managing everything

/model — switch between different models
/effort — decides the effort to put in: high → xhigh → max
/statusline — shown in the Claude prompt, displays total tokens used; helps decide when to compact the session

4. Session Management

Before starting with Claude Code, it's worth understanding /sessions — session management.

We can resume from wherever we stopped using a session ID.
Start a new session: claude -n "session_name"
Resume a previous session: claude --resume, claude --continue
/status — check session status

Practical guidance

We can create 3 different sessions for 3 different issues, potentially working on 3 different models.
/export explore-current-project.md — exports the conversation to a markdown file, e.g.:

   ⎿  Conversation exported to: H:\Claude-Code-Projects-17-19-Aug\claude_showcase_17_aug_26\explore-current-project.md

/fork — forks the current session so you can go to a new forked session without modifying the original session.
If 80% of context is used, we can compact it.
/resume — switch between sessions.
/compact — summarizes the session data; some data could be removed in the process.
If old context is not found in the current session, tokens would need to be spent again to recreate it — and wrong information could get compacted.
There is a usage limit at the account level; every session uses a part of that limit.
We should create multiple sessions — e.g. 3 different sessions for 3 different bugs. Information for the first bug may not be needed for the second bug, and sessions help Claude read old information without wasting tokens to recreate it.

Claude Code's /status panel showing version, session details, login/account info, and connected MCP servers.

5. Project Configuration

README.md

The most important file for reading all the information about a project.

Before telling the tool what to do, plan first — create the architecture first.
Decide on a design template.

CLAUDE.md — Rules

Used to control the behaviour of Claude:

Which JavaScript library to use.
Preventing Claude Code from reading sensitive files so it does not use them.
How to give Claude permission to access your code — what it can read and what it cannot read.

`.claude` folder and settings

First, create a .claude folder.
Create settings.json — accessible to the whole team, pushed to the repo.
Create settings.local.json — will not be pushed to the repo, and will override settings.json.

Here you can decide which model to use:

{
  "model": "opusplan"
}

Permissions:

{
  "permissions": {
    "allow": [
      "Bash(npm run *)",
      "Bash(git status)",
      "Bash(git diff)",
      "Bash(git add *)",
      "Bash(git commit *)",
      "Read(**)",
      "Write(src/**)",
      "Write(tests/**)"
    ],
    "ask": [
      "Bash(git push:*)",
      "Bash(npm install *)",
      "Write(package.json)"
    ],
    "deny": [
      "Read(./.env*)",
      "Read(./secrets/**)",
      "Read(./**/credentials*)",
      "Bash(rm -rf:*)",
      "Bash(curl:*)",
      "Bash(wget:*)"
    ]
  }
}

Claude declining to read a blocked .env.example file per project settings and offering alternative ways to proceed.

`/init`

Initializes a CLAUDE.md file with codebase documentation after going through the project at the root level.

Add conditions for Claude to always follow — should not go above 150 to 200 lines max; this is very specific to the root of the project.
We can have a CLAUDE.md file for every sub-folder.
If there is no CLAUDE.md file, don't rely on Claude having project-specific context — use CLAUDE.md for every project.

Example: the actual CLAUDE.md used in this training repo

## Coding Standards
- Always use async/await, never raw Promises
- All functions must have JSDoc comments
- No console.log in production code — use the logger utility
- Every new function must have at least one unit test
- Pure validation functions use a single return statement with a composed expression
- Never use multiple early-return guard clauses in validator functions

## Repository conventions
- CommonJS (`require`/`module.exports`) throughout, not ESM.
- ESLint enforces: `eqeqeq` (always `===`), `curly`, `no-var`/`prefer-const`,
  `no-return-await`, `require-await` (no `async` functions without an `await`),
  unused-arg exception for `_`-prefixed names.
- `.claude/settings.json` denies reads of `.env*`, `secrets/**`, and
  `credentials*` — don't try to work around this to inspect real secrets.

## What Claude Must Never Do
- Never modify .env or .env.\* files
- Never push directly to main branch
- Never remove existing tests
- Never install packages without confirming with the developer

## PR and Git Standards
- Commit messages follow Conventional Commits: feat:, fix:, docs:, test:
- PR descriptions must include: what changed, why it changed, how to test

## Tech Stack
- Runtime: Node.js 20
- Framework: Express.js
- Database: PostgreSQL with Prisma ORM
- Testing: Jest
- Language: ECMAScript 2022 for all new code only

## Architecture
Request flow: `src/index.js` wires Express with `requestLogger` middleware
globally, mounts all auth endpoints under `/api/auth` from `src/api/routes.js`,
and registers `errorHandler` last as the global error middleware.

Layering:
- `src/api/routes.js` — route handlers only; validates input shape, calls into
  `authService`, maps thrown errors to HTTP status codes (e.g. `'Invalid
  credentials'` → 401). It does NOT talk to `tokenHelper.js` directly.
- `src/api/middleware.js` — cross-cutting concerns: `requestLogger`,
  `authenticate` (verifies Bearer token via `tokenHelper.verifyToken`, attaches
  `req.user`), `validateBody` (checks required fields present), `errorHandler`
  (catch-all, returns generic 500).
- `src/auth/authService.js` — business logic: `loginUser`, `refreshToken`,
  `revokeToken`, token generation. Owns the in-memory `refreshTokenStore` (a
  `Map`, standing in for a database — resets on every restart, not shared
  across processes).
- `src/auth/tokenHelper.js` — low-level JWT primitives: `verifyToken`
  (signature + expiry check, throws typed errors), `decodeToken` (no
  verification, for reading claims only), `extractBearerToken`, `getTokenTTL`.
  `authService.js` and `middleware.js` both depend on this; it depends on
  nothing else in the app.
- `src/utils/validators.js` — pure, stateless input validators (email,
  password, UUID, sanitization). No I/O.
- `src/utils/logger.js` — minimal structured JSON logger (stdout/stderr),
  level-gated by `LOG_LEVEL` env var. Not a real observability stack.

There's no real user database: `routes.js` builds a `mockUserRecord` inline
for login, and `authService.getUserById` returns a hardcoded stub — expect
these to be replaced with real persistence rather than extended in place.

JWT config (`JWT_SECRET`, `JWT_EXPIRES_IN`, `REFRESH_EXPIRES_IN`) is read from
env vars in both `authService.js` and `tokenHelper.js` independently, each
with its own fallback default — keep them in sync if changing defaults.

### Request flow (login example)
`POST /api/auth/login` → `validateBody(['email','password'])` → route handler
validates format via `validators.js` → `authService.loginUser()` checks bcrypt
hash, calls `generateAccessToken` (signed JWT) + `generateRefreshToken` (uuid
stored in `refreshTokenStore`) → returns `{ accessToken, refreshToken, user }`.
Protected routes (`/logout`, `/me`) go through `authenticate` middleware,
which calls `tokenHelper.verifyToken` and attaches the decoded payload to
`req.user`.

### Error convention
Route handlers translate known domain errors to HTTP status codes inline
(e.g. `'Invalid credentials'` → 401) and pass everything else to `next(err)`,
where `errorHandler` logs it and returns a generic 500. Preserve this pattern
when adding routes — don't leak internal error messages to clients from the
global handler.

### Token model
Two-token scheme: short-lived signed JWT access token (claims: `sub`, `email`,
`role`) + opaque UUID refresh token stored server-side in `refreshTokenStore`
with `{ userId, createdAt }`. Refresh tokens are revoked by deleting the map
entry (`revokeToken`, logout).

Example: Generating `validateCreditCard` and Its Test Case from a Prompt

Note: creating a function like this automatically creates a test case also (via the write-tests skill — covered in section 7).

Claude adding a JSDoc-documented validateCreditCard function (Luhn checksum, 13-19 digits) to validators.js.

Claude adding Jest test cases for validateCreditCard covering valid Visa and Mastercard test numbers.

6. Auto Mode vs. Plan Mode

Auto mode — Claude decides what to do automatically and does not ask for confirmation.
Plan mode — we can ask Claude to create a plan only, without executing.
Use btw to ask a question which runs separately from the main section.
Always switch to manual mode before making changes you want to review first.
In plan mode, Claude will ask you to go ahead with the changes first.

Claude's plan mode proposal for a new validateBirthDate validator, with scope and constraints confirmed before execution.

In auto mode, this behaviour changes — Claude proceeds independently.

7. Commands vs. Skills vs. Agents

/commands — executed manually, defined via a markdown file.
Skills — run automatically for us, based on what's asked. A command can be converted into a skill. Skills go into a skills folder.
- There are a lot of third-party skills available.
- It's better to create our own skills, which use fewer tokens compared to those available externally, which tend to use more.
Agents — specialized assistants that do a task in the background and come up with an answer. Run an agent separately in the background, or run it via a skill — a skill can also hand over a task to an agent.

Rule of thumb: never use one-liners to do your job — don't go back to the tool again and again.

Example: custom command — `/standup`

---
name: standup
description: Generate a daily standup update from git history and open files
disable-model-invocation: true
---
You are a senior developer preparing a daily standup update for your team.
This is a Node.js/Express authentication API project. You have been working
in this codebase today. Check git log --since="00:00" --oneline and
git diff HEAD to understand what actually changed.

Draft a standup update based only on what you find in the git history
and open files — do not invent or assume work that is not visible.

Format:
Yesterday: [completed items, specific function or file names]
Today:     [in-progress items based on uncommitted changes or open TODOs]
Blockers:  [failing tests, TODO/FIXME comments, incomplete functions]

Keep each section to 3 bullet points maximum.
Use specific names — file names, function names, issue numbers where visible.
Be factual and brief. No filler. Write it as if reading it aloud in 30 seconds.

Output of the /standup command summarizing yesterday's commits, today's work, and current blockers.

If Claude Code is used to create code, it should also be used to create the commit.

Metaprompting

Take help of Claude to create a prompt using the RCTFCF framework (Role, Context, Task, Constraint, Format):

"Act as a prompt expert and help me create a structured prompt using role, context, task, constraint, and output format which can be used to do a security review for a Node.js API written in the Express framework. I want to check top 10 OWASP issues."

Also relevant: chain of thought prompting, and the observation that LLMs understand prompting better in their own language (i.e. structured, explicit formats they're trained to parse).

Claude implementing validateDateOfBirth in validators.js following the ISO-date, no-future, 120-year-limit convention.

The /raise-pr command refusing to push directly to main and creating a feature branch instead.

Example: custom command — `/raise-pr`

---
name: raise-pr
description: Raise a GitHub pull request for the current branch targeting main
disable-model-invocation: true
---
You are a senior developer raising a pull request for a fix or feature branch.

Steps:
1. Run git branch --show-current to get the current branch name.
2. Run git log main..HEAD --oneline to list all commits on this branch.
3. Run git diff main...HEAD to inspect all changes introduced by this branch.
4. Check if the branch name or commits reference an issue number (e.g. fix/issue-204
   or "closes #204"). Extract it if present.

Using what you observe, generate a PR title and body:

Title format:
  type(scope): short summary under 72 characters  (same as the commit message)

Body format:
  ## What changed
  - Bullet points describing each logical change
  ## Why it changed
  - The problem or issue this PR resolves (reference issue number if found, e.g. closes #204)
  ## How to test
  - Step-by-step instructions to verify the fix or feature works

Then run:
  gh pr create --base main --title "<title>" --body "<body>"

If the branch has no upstream yet, push it first:
  git push -u origin <branch-name>

Output only the PR title and body. No explanation, no commentary, no preamble.
Be precise and factual. Every word should earn its place.

Running /raise-pr:

Permission prompt confirming the git push of the new feature branch to origin.

Confirmation that the feature branch was pushed and PR #1 was opened on GitHub.

We can see the PR is created:

GitHub pull requests tab showing the newly opened validateDateOfBirth PR.

Skills are auto-invoked

/commands are executed manually, but skills get applied automatically based on the prompt. Seeing a list of all skills:

The /skills menu listing all 9 available skills and their lock/enabled status.

Skills could also come from installed plugins.

8. Code Review via Subagents

A code review skill can delegate each dimension of the review to a dedicated subagent using the Task tool:

## 1. Security
Delegate this dimension to the `security-analyst` subagent using the Task tool.

## 2. Performance
Delegate this dimension to the `performance-analyst` subagent using the Task tool.

## 3. Test Coverage
Delegate this dimension to the `test-coverage-analyst` subagent using the Task tool.

## 4. Code Quality
Delegate this dimension to the `code-quality-analyst` subagent using the Task tool.

How to use a skill

Example prompt:

❯ look at the auth module and tell me if there are any problems

The code review skill should auto-invoke:

Claude auto-invoking the code-review skill and launching four parallel review agents against the auth module.

No explicit integration is needed — it's auto-invoked. No need to use /command explicitly.

A sample of the code review report:

Consolidated code review report table listing critical and high-severity findings across the auth module.

Detailed write-up of two critical findings: a syntax-breaking stray await and a bypassed bcrypt password check.

Example: skill — `deploy`

If I want to deploy, deploy.sh should be used — so we use a deploy skill:

---
name: deploy
description: >
  Deploy the application to staging. Use when the user says "deploy",
  "ship it", "push to staging", or "release". Runs pre-flight checks,
  builds, tags, and verifies the deployment. Always confirm with the
  user before executing.
disable-model-invocation: true
---
# Deploy Skill
Deployment logic lives in `scripts/deploy.sh`. That script is the
source of truth — do not re-implement its steps here.

## Before Running
Confirm with the user:
- "Ready to deploy to staging. This will tag and push. Proceed?"
- Do not proceed without explicit confirmation.

## Execution
Run: `bash scripts/deploy.sh`

Example: skill — `write-tests`

Similarly, we can use a write-tests skill to write a test case:

---
name: write-tests
description: >
  Use this skill whenever new testable logic is added or modified — functions,
  methods, classes, route handlers, or modules in src/. Triggers on: "add a
  function", "create a utility", "implement this", "write a handler", "add a
  route", or any task that produces new logic with inputs and outputs or
  side effects. Do NOT wait to be asked — write tests as part of completing
  the task. Skip this skill only for: config constants, type definitions,
  pure re-exports, or framework boilerplate with no logic.
---
# Write Tests Skill
You are a senior QA engineer writing exhaustive Jest test suites for a
Node.js application.

## Stack & Conventions
- **Framework**: Jest with Supertest for HTTP integration tests
- **Test location**: `tests/` mirroring `src/` structure, `.test.js` suffix
- **Logger**: Structured JSON — never assert on `console` output
- **Auth**: `JWT_SECRET` defaults to `'dev-secret-key'` in tests — do not
  assert real security guarantees against this value
- **Test data**: Define inline for simple cases; use `tests/fixtures/` for
  objects reused across 3+ tests

Initiating the write-tests skill:

The write-tests skill identifying and filling a missing leap-year test case for validateDateOfBirth.

Skills are auto-invoked. LCOV coverage output:

Test suite results showing 76 passing tests with 100% coverage on validators.js.

Creating a custom agent

Skills, agents, and commands differ as follows:

Agent — a specialized assistant that does a task in the background and comes up with an answer. Run it separately in the background, or run it via a skill — a skill can also hand a task over to an agent.
In an agent definition, we can specify exactly what tools the agent can use.

Example of agents:

The project's custom subagent definitions listed in the .claude/agents folder.

Example: custom agent — `security-analyst`

---
name: security-analyst
description: >
  Use this agent for security reviews: authentication bypass, injection vulnerabilities, OWASP Top 10, token handling, and input validation. Invoke with: ask the security-analyst to audit this file.
tools: Read, Bash(npm audit *), Bash(grep *), Write, Edit
model: opus
memory: project
---
# Security Analyst Agent
You are a senior application security engineer. You read code exclusively through
a security lens. You do not suggest feature improvements or code style changes.
You find vulnerabilities and you explain how to fix them.

## Your Focus Areas
- Authentication bypass and broken access control
- Injection vulnerabilities: SQL, command, path traversal
- Sensitive data exposure in logs, responses, or error messages
- Insecure token handling: weak secrets, missing expiry, improper storage
- Missing input validation and sanitisation
- Dependency vulnerabilities (flag for npm audit review)

## OWASP Categorisation
Tag every finding with its OWASP Top 10 category where applicable.

## Memory Protocol
Before starting, read `MEMORY.md` (an index of one-line links to note files) and
skim any linked notes whose description looks relevant to the file(s) you're
about to audit.

After completing your audit, write to memory only if you found something
non-obvious, recurring, or specific to this codebase that would save real time
on a future audit — not routine findings you'd already report. If so:
1. Create a new note file named for the pattern (e.g. `jwt-secret-hardcoded-in-config.md`)
   with `name`, `description`, and `metadata: {type: feedback}` frontmatter, following
   the structure used in existing notes.
2. Add a one-line link to it from `MEMORY.md`.

Do not rewrite or duplicate an existing note — update it in place if the same
pattern recurs with new detail.

## Output Format
For each finding:
- OWASP Category
- Severity: Critical / High / Medium / Low
- File and line number
- What an attacker could do with this vulnerability
- Exact fix with corrected code snippet

End with: total finding count by severity, and one recommended immediate action.

How does a skill hand over a task to an agent?

Claude launching the security-analyst subagent in the background to audit the authentication codebase.

The agent runs in the background

● Agent "Security audit of auth codebase" finished · 4m 42s

The security-analyst audit found 18 findings across authService.js, tokenHelper.js, routes.js, middleware.js, and validators.js. Two are critical and one of them also breaks the server outright.

Critical

Auth bypass — bcrypt.compare(...) at authService.js:32 is called without await, so the truthy Promise always passes the check. Every password is accepted for any login.
Hardcoded JWT fallback secret — 'dev-secret-key' in both authService.js:11 and tokenHelper.js:8, used silently if JWT_SECRET is unset. Anyone can forge admin tokens.

High

Login always builds an inline mockUserRecord with a constant id: 'user-001' and attacker-supplied email — every session collapses to the same identity, with a spoofable email claim.
No rate limiting on /login or /refresh — unlimited credential stuffing / refresh-token brute-forcing.
REFRESH_EXPIRES_IN is declared but never enforced — refresh tokens never expire.

Medium (6 findings) — notably: authService.js:80 is missing the async keyword on refreshToken, which is a SyntaxError that prevents the server from booting at all (confirmed via node --check) — meaning none of this is currently under test/CI coverage; an inverted expiry comparison in isTokenExpired that also deletes valid sessions; /logout can revoke anyone's refresh token (no ownership check); access tokens aren't invalidated on logout; PII (raw emails) logged on every auth attempt; unbounded in-memory token store (memory-exhaustion DoS).

Low (7 findings) — untrimmed email reaching downstream sinks, internal error messages leaked to clients via substring matching, missing JWT algorithm allowlist, missing security headers/CORS/body-size limits, loose validateBody treating null/0/false as "present," a low-severity uuid advisory (not reachable, since uuidv4() is called with no args), and a latent timing side-channel for user enumeration.

Good news: the newly added validateDateOfBirth (commit de91f6f) was checked directly and is clean — correctly rejects rollover dates, future dates, ages over 120, and malformed input. It's not currently wired into any route, so it's not a live attack surface either way.

Recommended immediate fix (one commit): add the missing await on bcrypt.compare (line 32) and the missing async on refreshToken (line 80) together — the second is required just to get the app to boot, and without it nothing else can even be tested. Rotate JWT_SECRET and make it mandatory at startup right after.

9. Multi-Agent Harness Concepts

When Claude Code itself uses multiple subagents to work on a task, we can say it's using a multi-agent harness. The main thread uses the info from all the agents to create a single report.

A second pass: performance, test coverage, and code quality

Prompt: "Create agent for performance review, test coverage, and code quality — use the code review skill."

Claude clarifying review scope, then running performance, test-coverage, and code-quality agents in parallel while fixing the critical bugs.

Full output — Code Review Report (Performance / Test Coverage / Code Quality)

Context: Re-run of the performance, test-coverage, and code-quality dimensions against src/auth/authService.js (security was already covered in the prior review). Findings are consistent with the earlier pass; a few additional specifics surfaced this time.

#	Dimension	Finding	Severity
1	Code Quality / Test Coverage	Missing `async` on `refreshToken` (line 80) + `await` at line 92 → `SyntaxError`, module can't load, entire app fails to boot	🔴 Critical
2	Code Quality / Test Coverage	`bcrypt.compare` not awaited (line 32) → auth bypass, confirmed independently by ESLint's `require-await` flagging `loginUser` as having no real await	🔴 Critical
3	Test Coverage	No test file exists for `authService.js` at all — every exported function has 0% coverage	🔴 Critical
4	Test Coverage	`isTokenExpired` inverted logic (line 68) — verified via failing Jest assertions against real signed JWTs	🔴 Critical
5	Code Quality	`isTokenExpired` duplicates `tokenHelper.js`'s JWT logic instead of delegating (violates CLAUDE.md's documented layering), and is applied to opaque UUID refresh tokens where it always short-circuits via `jwt.decode() === null`	🟠 High
6	Code Quality	`REFRESH_EXPIRES_IN` and `createdAt` are dead code — confirmed via ESLint `no-unused-vars` — refresh tokens never actually expire despite CLAUDE.md documenting a 7-day window	🟠 High
7	Performance	Unbounded `refreshTokenStore` Map — no TTL sweep, grows with cumulative logins not concurrent sessions	🟠 High
8	Test Coverage	`refreshToken()`, `revokeToken()`, `generateRefreshToken()`, `generateAccessToken()` have no direct unit tests or edge-case coverage	🟠 High
9	Performance	`bcryptjs` (pure-JS, not native bcrypt) runs password comparison on the main thread, not the libuv threadpool — inflates tail latency under concurrent login bursts	🟡 Medium
10	Code Quality	Swallowed error in `isTokenExpired`'s `catch (err)` (line 69-71) — caught but never logged, inconsistent with the rest of the file's logging pattern	🟡 Medium
11	Code Quality	`curly` ESLint rule violation on line 64 (`if (...) return true;` without braces)	🟡 Medium
12	Code Quality	`require-await` violation on `getUserById` (line 153) — declared async with no await inside	🟡 Medium
13	Test Coverage	`loginUser` with no `userRecord`, JWT claims shape, `getUserById` stub contract — untested but low-risk edge cases	🟢 Low
14	Code Quality	`revokeToken`'s "not found" branch logs nothing, inconsistent with other failure branches in the file	🟢 Low
15	Code Quality	Imprecise JSDoc return shapes (`@returns {object}` instead of a typed shape)	🟢 Low

New since the last pass

Tooling is broken at the project level: npm run lint currently fails outright — the repo ships .eslintrc.json (legacy config) but has ESLint 10.9.1 installed, which requires flat eslint.config.*. No flat config exists at the repo root. The code-quality agent worked around this by building an equivalent flat config in the scratchpad to get real lint signal (never touched the actual repo files), but as things stand today, npm run lint gives false confidence — it fails before it even reaches your code. This is worth fixing independent of authService.js.
bcryptjs vs native bcrypt (finding 9): not raised as a distinct finding in the first pass — flagged now because it compounds with the login-bypass fix. Once await is added back (finding 2), every login request will run a real CPU-bound bcrypt comparison on the main thread; under concurrent login bursts this inflates latency for unrelated concurrent requests. Not urgent, but worth knowing before assuming the fix is "free."

Summary Score: 2/10 (unchanged) — the file can't boot, has zero test coverage, and now additionally the project's lint tooling can't even verify it.

Recommended First Action: unchanged — fix the missing async on refreshToken (line 80) together with the missing await on bcrypt.compare (line 32) in one commit — nothing else here (tests, lint, the rest of the app) is reachable until the module can actually load. Separately, worth a quick fix to add eslint.config.js so npm run lint works again — that's a repo-wide gap, not specific to this file.

This is a hub and spoke model — using subagents via a skill. However, what we actually want in some cases is agents communicating with each other independently in the background to finish a task; the hub-and-spoke approach above cannot do that. Agents can instead be run using other agentic harnesses — loop, schedule, etc.

Difference between agent and agentic

Comparison diagram contrasting framework-driven "Agent" orchestration with model-native "Agentic" execution.

Goal, loop, schedule, workflow, batch — agentic primitives available which help create subtasks (e.g. ultraplan, autofix-pr).
/commands are manual.
Skills get applied automatically based on the prompt.
/agent runs in the background.

10. Hooks

HOOKS, PLUGINS, MCP e2e flow — from evaluating code to fixing code, full auto flow.

Lifecycle methods available in Claude: from the start to the end of a session, internal events fire inside the session, which are used as plug points for our own code.

Hook event flow

Whenever Claude reaches a lifecycle boundary — e.g. when a tool is about to run — it spawns our custom script as a subprocess and feeds it a JSON blob on stdin. The script can execute anything, then return:

an exit code (0 or 2), or
a JSON object on stdout containing a structured decision, reason, and context for Claude to read.

Hooks can be created using Python or JavaScript. They take input from Claude Code, display it on screen, and allow the tool to run.

Example: `PreToolUse` hook on Bash

{
  "hooks": {
    "PreToolUse": [
      {
        "matcher": "Bash",
        "hooks": [
          {
            "type": "command",
            "command": "node .claude/hooks/bash-guard.js"
          },
          {
            "type": "command",
            "command": "node .claude/hooks/pretooluse-demo.js"
          }
        ]
      }
    ]
  }
}

Since this hook sits directly in front of every Bash call, it's a good interception point for anything you want enforced before a shell command runs. Some concrete uses, especially relevant to a repo like this:

Guardrails / safety (this is what bash-guard.js already seems to do) — block dangerous patterns beyond what settings.json's deny-list covers, e.g. rm -rf variants, git push --force, piping to sh, etc., using regex instead of exact string matching.

All hook handler types

{
  "hooks": {
    "PreToolUse": [
      {
        "matcher": "Bash",
        "hooks": [
          { "type": "command", "command": "node .claude/hooks/pretooluse-demo.js" },
          { "type": "http", "url": "https://www.example.com/validate" },
          { "type": "prompt", "prompt": "Evaluate the steps before running the tool" },
          { "type": "agent", "prompt": "Run security-analyst to verify the issues" },
          { "type": "mcp_tool", "server": "some-mcp-server", "tool": "validate_edit" }
        ]
      }
    ]
  }
}

The bash-guard hook's log file recording every Bash command it allowed during the session.

Blocking dangerous commands

const DANGER_PATTERNS = [
    /rm\s+-rf\s+\//,                          // recursive force-delete from root (e.g. "rm -rf /")
    /sudo\s+rm/,                              // any sudo-elevated delete — bypasses normal permission checks
    /:\(\)\{\s*:\|:&\s*\}\s*;\s*:/,           // fork bomb — ":(){ :|:& };:" spawns processes until the system locks up
    /dd\s+if=.*of=\/dev\/sd/,                 // raw disk write via dd — can overwrite an entire drive
    /chmod\s+777\s+\//,                       // world-writable permissions on root — opens up the whole filesystem
];

Notification on Slack

{
  "Notification": [
    {
      "matcher": "permission_prompt",
      "hooks": [
        {
          "type": "command",
          "command": "node .claude/hooks/slack-notify.js"
        }
      ]
    }
  ]
}

Logs can be used to audit our work — when we did what, and how much time we spent.

Sharing configuration

What if I want to use all my settings in the .claude folder in another project? This can be done via GitHub. But how can we give it to a third party? Through plugins — see code.claude.com/docs/en/plugins for how to create one.

11. MCP (Model Context Protocol)

The concept of an MCP server

The GitHub REST API knows how to connect to GitHub on your behalf.

Diagram showing Git Bash and the GH CLI both reaching GitHub's SaaS platform through the GitHub REST API.

Either Claude directly connects via Bash, or via the gh CLI — both of which need to be authenticated. Actually, the gh CLI is itself using Bash commands under the hood. But can Claude Code directly connect to the GitHub REST API to be faster and more efficient? curl can be used for that:

Expanded diagram showing how Claude Code uses its Bash tool (via curl, git, or gh CLI) to reach the GitHub REST API.

Why MCP exists

Every ecosystem/SaaS (like GitHub) has a separate, specific, proprietary way of connecting to it — so if Claude has to connect to multiple SaaS platforms, it would need to know all of them, which deviates Claude from its core functionality.

Can GitHub give a way to connect directly to a REST API? So Anthropic created a new protocol:

Claude Code has an inbuilt MCP client when installed.
GitHub/any SaaS provider provides an MCP server — basically an endpoint.
Our job is only to connect to the MCP server; the MCP server handles the rest.

Three types of MCP server

stdio — runs locally on the machine
http — hosted at an HTTP location
SSE — Server-Sent Events, a streaming protocol

All AI tools support MCP servers. The MCP client is part of the ecosystem.

Claude Code runs with an agentic harness — the LLM decides what to do, and the LLM tells Claude Code which tools to use.

12. End-to-End Automation

End-to-end automation of security review → issue creation → issue fix → PR creation → PR fix, all through agents.

The /security-review command loading its skill to audit tokenHelper.js.

Security review finding a critical hardcoded fallback JWT signing secret vulnerability in tokenHelper.js.

`/create-issue`

Claude using the gh CLI to file a GitHub issue for the hardcoded JWT secret vulnerability.

GitHub issue #2 detailing the hardcoded fallback JWT secret bug, its impact, and expected fix.

Now fix the issue

Claude implementing the fix: removing the hardcoded JWT secret fallback and requiring the env var explicitly.

Commit the changes

The /commit command staging, reviewing, and pushing the JWT secret fix with a detailed commit message.

GitHub activity showing the fix commit automatically linked to close issue #2.

Pull request description summarizing what changed and why for the JWT secret fix.

13. Agent Teams

Beyond single subagent delegation, Claude Code supports Agent Teams — multiple agents working the same task from different angles in parallel, then synthesizing a combined result.

Example prompt

Create an agent team to review PR #1.
Spawn three reviewers:

Reviewer 1: focused exclusively on security implications

Reviewer 2: focused on performance and scalability

Reviewer 3: validating test coverage and edge cases

Have them each review independently, then synthesize findings into a single report.
Create a new branch with custom name including today's date.
Merge and close the PR with comment.

User prompt instructing Claude to spawn three specialized reviewer agents to review and merge PR #1.

The 3 agents are invoked in the background:

Claude launching three specialized reviewer agents in parallel to review PR #1.

Progress log showing the security and test-coverage reviewer agents finishing with real findings.

Claude synthesizing all three reviewer agents' findings before applying fixes and merging.

Claude merging the verified fix, filing two follow-up issues, and preparing to post the review synthesis as a PR comment.

Final summary of the agent-team review: verdicts from all three reviewers and the actions taken to merge PR #1.

14. Key Takeaways

The agentic loop (gather context → decide → act → verify) is the mental model for everything Claude Code does.
CLAUDE.md and .claude/settings.json are how you shape Claude's behavior and permissions for a specific project.
Commands are manual, Skills auto-trigger, and Agents run scoped, background work — often orchestrated together.
Hooks and MCP are the two extension points: hooks intercept lifecycle events for guardrails/automation, MCP standardizes how Claude talks to external tools.
Session management (/resume, /fork, /compact) is essential for working on multiple issues without polluting context or wasting tokens.
Agent Teams unlock genuinely parallel, multi-perspective work — like having three reviewers look at a PR simultaneously.

This was a genuinely comprehensive look at how far the agentic coding model has come — from a single loop reading and writing files, to fully orchestrated, permissioned, multi-agent development workflows.

AI & ML Foundations: Cleaning Data, Building Regression & Classification Models in Python.

yash-nigam — Mon, 31 Aug 2026 16:53:20 +0000

My 5-Day Journey into Machine Learning: From Data Cleaning to My First ML Models

Learning goal: This is not a collection of code snippets to memorize. It is a beginner-friendly guide to understanding why each step in a typical machine-learning workflow exists, what problem it solves, and how the pieces fit together.

A note about the source material

This guide was built primarily from the four Jupyter notebooks you provided:

11_Aug_mpg.ipynb
12-Aug-2.ipynb
12th-Aug-Telco-Customer-churn.ipynb
13-Aug.ipynb
123

Your PDF notes were also provided, but the PDF is image-based and its text could not be reliably extracted. So I have not invented or silently reconstructed material from the PDF. The explanations below are grounded mainly in the notebooks, with general ML knowledge added to explain the concepts behind the code.

1. Overview: What did I actually learn in these five days?

At first glance, machine learning can look like this:

Load dataset
    ↓
Clean data
    ↓
Create graphs
    ↓
Train model
    ↓
Check accuracy
    ↓
Done!

But that misses the most important part.

The real ML workflow is closer to:

Business / real-world question
        ↓
Define what we want to predict
        ↓
Understand the data
        ↓
Clean and prepare the data
        ↓
Explore the data visually
        ↓
Choose useful features
        ↓
Convert data into numbers
        ↓
Split into training and testing data
        ↓
Choose an appropriate ML algorithm
        ↓
Train the model
        ↓
Evaluate it on unseen data
        ↓
Compare models
        ↓
Improve preprocessing / model settings
        ↓
Use the model to make predictions

The important insight is:

Machine learning is not mainly about choosing an algorithm. It is about turning a real-world question and messy data into a reliable prediction process.

Your notebooks actually demonstrate most of this workflow.

2. What is Artificial Intelligence?

Artificial Intelligence (AI) is the broad idea of building systems that can perform tasks that normally require some form of human intelligence.

Examples:

recognizing an image
understanding language
recommending a movie
detecting fraud
predicting whether a customer will leave
predicting the price/value of something
generating text or images

A useful mental model is:

Artificial Intelligence
│
├── Machine Learning
│   │
│   ├── Supervised Learning
│   ├── Unsupervised Learning
│   └── Reinforcement Learning
│
├── Deep Learning
│
└── Generative AI

This is simplified because these areas overlap, but it is a useful beginner mental model.

3. What is Machine Learning?

Machine learning is a way of building systems where the computer learns patterns from data instead of us manually writing every rule.

Imagine we want to predict whether a telecom customer will churn.

A traditional rule-based approach might look like:

IF customer has a month-to-month contract
AND monthly charges are high
AND tenure is low
THEN churn = yes

The problem is that we have to invent those rules ourselves.

With machine learning, we give the algorithm historical examples:

Customer information → Actual churn result

Customer A → No
Customer B → Yes
Customer C → No
Customer D → Yes
...

The algorithm tries to learn the relationship between the input information and the known outcome.

Then we can give it a new customer:

New customer information
        ↓
      Model
        ↓
Predicted churn

That is the core idea of supervised machine learning.

4. The most important ML vocabulary

Before learning algorithms, understand these words.

4.1 Dataset

A dataset is a collection of examples.

In your notebooks, a dataset is loaded into a Pandas DataFrame.

For example:

df = pd.read_csv("Telco_Customer_Churn.csv")

Think of the DataFrame as a spreadsheet.

4.2 Row / observation / sample

Each row represents one example.

For the Telco dataset:

one row = one customer

For the Auto MPG dataset:

one row = one car

For the insurance dataset:

one row = one customer/policy record

These words are often used interchangeably:

row
observation
sample
instance

4.3 Feature

A feature is an input variable that the model can use to make a prediction.

For customer churn:

tenure
MonthlyCharges
Contract
InternetService
PaymentMethod
...

These are features.

A useful question to ask is:

"What information is available to the model before it makes its prediction?"

That information is generally your feature set.

4.4 Target

The target is what we are trying to predict.

Examples from your notebooks:

Telco

Target = Churn

The model predicts:

0 → No churn
1 → Churn

Insurance

Target = Customer Lifetime Value

The model predicts a numerical value.

4.5 X and y

You repeatedly used:

X = df.drop(["Churn"], axis=1)
Y = df["Churn"]

This is one of the most important patterns in supervised ML.

X = inputs/features
Y = target/output

For example:

X
├── tenure
├── MonthlyCharges
├── Contract
├── InternetService
└── ...

Y
└── Churn

Think:

X = what the model knows

Y = what we want the model to learn to predict

5. The three major types of Machine Learning

5.1 Supervised Learning

In supervised learning, we have examples where the correct answer is already known.

Input X → Known answer Y

The model learns the relationship between X and Y.

Your notebooks mainly use supervised learning.

There are two major supervised learning tasks.

Classification

The target is a category.

Examples:

Spam / Not Spam
Churn / No Churn
Fraud / Not Fraud
Disease / No Disease

Your Telco project is a classification problem.

Regression

The target is a numerical value.

Examples:

House price
Temperature
Sales
Customer Lifetime Value
Fuel efficiency

Your Customer Lifetime Value project is a regression problem.

6. Classification vs Regression

This distinction is extremely important.

Classification

Question:

"Which category does this example belong to?"

Example:

Customer → Churn or No Churn

Possible output:

0
1

Common classification algorithms you tried:

Logistic Regression
Support Vector Classifier (SVC)
Decision Tree Classifier

Regression

Question:

"What numerical value should we predict?"

Example:

Customer → Customer Lifetime Value = 6,543.21

Common regression algorithms you tried:

Linear Regression
KNN Regressor
SVR
Decision Tree Regressor
Bagging Regressor
AdaBoost Regressor
Random Forest Regressor

7. Unsupervised Learning

In unsupervised learning, we do not have a known target.

Instead, the algorithm tries to discover structure in the data.

For example:

Customer data
     ↓
Find groups
     ↓
Group 1: high-value customers
Group 2: price-sensitive customers
Group 3: new customers

Common examples:

K-Means clustering
Hierarchical clustering
PCA

You did not build an unsupervised-learning project in these notebooks, but it is important to know where it fits.

8. Reinforcement Learning

Reinforcement learning is different.

An agent interacts with an environment and receives rewards or penalties.

Agent
  ↓ action
Environment
  ↓
Reward / penalty
  ↓
Agent learns

Examples:

game-playing agents
robotics
certain recommendation/control systems

This was not part of your five-day practical work.

9. Different ways to approach an ML problem

A useful way to think about ML is not:

"Which algorithm should I use?"

Instead ask:

Step 1 — What is the question?

For example:

Can I predict whether a telecom customer will churn?

Step 2 — What type of target do I have?

Churn → category → Classification
Customer Lifetime Value → number → Regression

Step 3 — What data is available?

Ask:

What columns exist?
Which are numerical?
Which are categorical?
Are values missing?
Are there strange values?
Are some columns identifiers?
Is the target balanced?

Step 4 — Explore before modeling

Use statistics and graphs.

Step 5 — Prepare the data

Convert it into a form the algorithm can understand.

Step 6 — Establish a baseline

Train a simple model first.

Step 7 — Compare alternatives

Try other appropriate models.

Step 8 — Evaluate on unseen data

A model is useful only if it works beyond the data it memorized.

10. Google Colab: Why did we use it?

Google Colab gives you a cloud-based Jupyter Notebook environment.

Instead of installing everything locally, you can run Python code in the browser.

A notebook combines:

Explanation
+
Python code
+
Output
+
Graphs

This makes it very useful for learning ML.

Your notebooks use paths such as:

/content/auto-mpg.csv

That is typical of a Colab environment.

A normal learning workflow is:

Upload dataset
      ↓
Open notebook
      ↓
Run cells
      ↓
Inspect output
      ↓
Change code
      ↓
Run again

11. The Python libraries you used

11.1 NumPy

import numpy as np

NumPy provides numerical operations and arrays.

You used it for things such as:

np.nan
np.log1p()
np.expm1()

`np.nan`

Represents a missing numerical value.

`np.log1p(x)`

Calculates:

log(1 + x)

It is useful when a target is heavily skewed.

`np.expm1(x)`

Reverses log1p:

exp(x) - 1

You used this later to convert log predictions back to the original CLV scale.

12. Pandas

import pandas as pd

Pandas is one of the most important Python libraries for data work.

Think of a Pandas DataFrame as a spreadsheet that Python can manipulate.

You used it to:

load CSV files
inspect data
remove columns
convert data types
detect missing values
fill missing values
select X and Y
create encoded columns
save predictions

Example:

df = pd.read_csv("Telco_Customer_Churn.csv")

13. Matplotlib

import matplotlib.pyplot as plt

Matplotlib is a general-purpose plotting library.

It provides the foundation for many Python visualizations.

You used it for:

plt.figure()
plt.title()
plt.show()

14. Seaborn

import seaborn as sns

Seaborn makes statistical visualizations easier to create.

You used:

countplot
boxplot
histplot
pairplot
scatterplot
heatmap

A key lesson:

A graph is not decoration. It is a tool for asking questions about the data.

15. Scikit-learn

Scikit-learn is the main ML library used in your notebooks.

You used it for:

splitting data
preprocessing
encoding
scaling
classification
regression
evaluation

The basic pattern is:

model = SomeModel()
model.fit(X_train, Y_train)
predictions = model.predict(X_test)

This pattern appears again and again in ML.

16. The most important workflow: EDA

EDA means:

Exploratory Data Analysis

EDA is the process of investigating the dataset before building the final model.

Your notebooks use:

df.head()
df.shape
df.describe()
df.info()
df.sample()
df.isnull().sum()

plus graphs.

These are not random commands.

They answer different questions.

17. `df.head()` — What does the data look like?

You used:

df.head()

It displays the first few rows.

Why?

Because before doing anything else, you want to see:

column names
example values
obvious data problems
whether the data loaded correctly

Think of it as opening the box before using what is inside.

18. `df.shape` — How much data do I have?

You used:

df.shape

For Auto MPG:

(398, 9)

That means:

398 rows
9 columns

Why does this matter?

Because dataset size affects:

model choice
computation time
confidence in results
risk of overfitting

19. `df.info()` — What types of data do I have?

You used:

df.info()

This tells you:

column names
number of non-null values
data types
memory usage

This is extremely important.

For example, in Auto MPG:

horsepower → object

At first glance horsepower sounds numerical.

But Pandas sees it as text.

Why?

Because the dataset contains values such as:

A column containing:

cannot be treated as a clean numerical column.

So:

df.info() helps you detect data-type problems before modeling.

20. `df.describe()`

You used:

df.describe()

and sometimes:

df.describe(include="all")

This gives summary statistics.

For numerical data you get things such as:

count
mean
standard deviation
minimum
25th percentile
median
75th percentile
maximum

This helps you understand the scale and distribution of variables.

For example:

minimum
   ↓
25%
   ↓
median
   ↓
75%
   ↓
maximum

A large difference between the 75th percentile and maximum can be a clue that there may be extreme values.

21. `df.sample()`

You used:

df.sample(5)

or similar.

This shows random rows.

Why?

Because the first rows may not represent the whole dataset.

Random samples can expose:

unusual values
formatting issues
unexpected categories
data-entry problems

22. Missing data

You used:

df.isnull().sum()

This asks:

"How many missing values are there in each column?"

Missing data is important because many ML algorithms cannot directly work with missing values.

But there is a subtle lesson from your Auto MPG notebook.

You first checked:

df.isnull().sum()

while horsepower still contained "?".

Pandas did not count "?" as a missing value because "?" is a string, not a true NaN.

That means:

"?" ≠ NaN

So data cleaning sometimes requires identifying fake missing values first.

23. Auto MPG: cleaning horsepower

Your notebook did:

df["horsepower"] = df["horsepower"].replace("?", np.nan)

This converts the placeholder into a real missing value.

Then:

df["horsepower"] = pd.to_numeric(df["horsepower"])

Now Pandas can treat horsepower as numerical data.

This is a very important data-cleaning pattern:

messy text
   ↓
identify invalid placeholder
   ↓
convert to NaN
   ↓
convert column to numeric
   ↓
handle missing values

24. Why use the median?

You calculated:

median_horsepower = df["horsepower"].median()

and then filled missing values:

df["horsepower"] = df["horsepower"].replace(
    np.nan,
    median_horsepower
)

Why median?

Because the median is less affected by extreme values than the mean.

Example:

10, 11, 12, 13, 1000

The mean is pulled heavily upward by 1000.

The median is much more representative of the middle of the data.

A useful rule of thumb:

fairly symmetric data → mean may work well
skewed data / outliers → median is often safer

But remember:

There is no universal rule that "median is always correct."

The best treatment depends on the data and the problem.

25. Dropping an identifier

You considered dropping:

df.drop("car name", axis=1)

and actually dropped customerID in the Telco work.

Why?

An identifier is usually not a meaningful predictive feature.

For example:

customerID = 759832

does not inherently mean the customer is more or less likely to churn.

The ID exists to identify the row, not to describe the customer.

Important distinction:

A column can be useful for identifying a record without being useful for predicting the target.

26. But don't blindly drop columns

This is an important improvement to your original reasoning.

A column should not be dropped merely because it is an object/string.

For example:

Contract
InternetService
PaymentMethod

are categorical strings, but they can contain very useful predictive information.

So the correct question is:

"Does this column contain useful information that is available at prediction time?"

not:

"Is this column numeric?"

27. Understanding your graphs

You created many graphs. The important thing is to understand what question each graph answers.

28. Countplot

Example:

sns.countplot(x="Churn", data=df)

A countplot answers:

"How many observations are in each category?"

For churn:

No churn → number of customers
Churn    → number of customers

This immediately helps you understand class balance.

If the graph looks like:

No Churn ███████████████████
Churn    ███████

the classes are imbalanced.

That matters because a model could achieve high accuracy simply by favoring the majority class.

29. Countplot with `hue`

You used:

sns.countplot(
    x="Contract",
    hue="Churn",
    data=df
)

This answers:

"How does the target category vary across another category?"

For example:

Contract type
     ↓
Churn / No churn

If month-to-month customers show much more churn than two-year customers, that is an important pattern worth investigating.

But remember:

A graph showing association does not automatically prove causation.

30. Histogram

You used:

df.hist(figsize=(10,10))

and:

sns.histplot(df["MonthlyCharges"], kde=True)

A histogram shows the distribution of a numerical variable.

It helps answer:

Where are most values?
Is the data symmetric?
Is it skewed?
Are there multiple groups?
Are there extreme values?

Example:

frequency
   ^
   |       ███
   |     ███████
   |   █████████
   | ███████████
   +-----------------> value

31. Why distributions matter

Suppose a feature looks like:

Most values: 0–100
A few values: 10,000

That may indicate:

genuine outliers
a different population
data-entry errors
a highly skewed distribution

A model may react differently depending on the algorithm.

This is one reason EDA happens before modeling.

32. Boxplot

You used:

sns.boxplot(y=df["Income"])

A boxplot summarizes a distribution.

Conceptually:

       outlier
          •
          |
      ┌───────┐
      │       │
------│  box  │------
      │       │
      └───────┘
          |
       outlier

The box represents the middle portion of the data.

The line inside the box is the median.

Points outside the whiskers may be treated as potential outliers.

Important:

An outlier is not automatically an error.

A high-income customer may be perfectly legitimate.

33. Boxplot: target vs category

You used:

sns.boxplot(
    x="Coverage",
    y="Customer Lifetime Value",
    data=df
)

This is a powerful graph.

It asks:

"How does the distribution of Customer Lifetime Value differ between categories?"

For example:

Coverage A → CLV distribution
Coverage B → CLV distribution
Coverage C → CLV distribution

Compare:

median
spread
outliers
overlap

This can help you identify potentially useful relationships.

34. Scatterplot

You used:

sns.scatterplot(
    x="Income",
    y="Customer Lifetime Value",
    data=df
)

A scatterplot examines the relationship between two numerical variables.

Each dot represents an observation.

It helps answer:

"As X changes, does Y appear to change?"

Possible patterns:

Positive relationship:
  •
    •
      •
        •

Negative relationship:
        •
      •
    •
  •

No obvious relationship:
 •    •
    •
  •      •
     •

The pattern does not have to be a straight line.

35. Pairplot

You used:

sns.pairplot(...)

A pairplot shows many numerical relationships at once.

It is useful for:

spotting correlations
identifying clusters
seeing distributions
finding obvious relationships
detecting possible outliers

The downside is that it becomes difficult to read when there are many columns.

So:

Pairplot is excellent for small-to-medium exploratory datasets, but not something you blindly run on every large dataset.

36. Correlation heatmap

You used:

sns.heatmap(
    df[["tenure", "MonthlyCharges", "TotalCharges"]].corr(),
    annot=True
)

Correlation measures how two numerical variables move together in a linear relationship.

A correlation close to:

+1 → strong positive linear relationship
 0 → little/no linear relationship
-1 → strong negative linear relationship

For example:

tenure ↑
TotalCharges ↑

could result in positive correlation.

But correlation has an important limitation:

Correlation does not prove causation.

Also, zero correlation does not necessarily mean "no relationship" because the relationship might be non-linear.

37. A key lesson from visualization

The graphs answer different questions:

Graph	Main question
Countplot	How many observations are in each category?
Countplot + hue	How does one category vary with another?
Histogram	What does a numerical distribution look like?
Boxplot	What are the median, spread and potential outliers?
Boxplot + category	How does a numerical distribution differ by category?
Scatterplot	How do two numerical variables relate?
Pairplot	What relationships exist among several numerical variables?
Heatmap	How strongly are numerical variables linearly correlated?

The goal is not:

"Create lots of graphs."

The goal is:

"Use the right graph to answer the right question."

38. Encoding: why does ML need it?

Many ML algorithms work with numbers.

But real-world data contains:

Male
Female

Yes
No

Month-to-month
One year
Two year

Fiber optic
DSL
No

The model needs these values represented numerically.

This is called encoding.

39. Label Encoding

You used:

from sklearn.preprocessing import LabelEncoder

and encoded categorical columns.

For a binary column:

No  → 0
Yes → 1

This can be reasonable for a binary variable.

For example:

Churn:
No  → 0
Yes → 1

That makes intuitive sense.

40. Why LabelEncoder can be dangerous for multi-category variables

Suppose:

Contract:
Month-to-month
One year
Two year

Label encoding might produce:

Month-to-month → 0
One year       → 1
Two year       → 2

A numerical model may interpret that as:

Two year > One year > Month-to-month

That ordering may not be appropriate for the algorithm.

The numbers are just codes.

They do not automatically mean that category 2 is "twice" category 1.

For nominal categories, one-hot encoding is usually safer.

41. One-hot encoding

You used:

pd.get_dummies(df, drop_first=True)

This transforms categories into separate binary columns.

For:

InternetService:
DSL
Fiber optic
No

we might create columns such as:

InternetService_Fiber optic
InternetService_No

with the remaining category represented by both values being 0.

This avoids inventing a fake numerical ordering.

42. Why `drop_first=True`?

If a categorical variable has several categories, using all one-hot columns can introduce redundancy in some models.

Dropping one category creates a reference category.

For example:

DSL
Fiber optic
No

could become:

Fiber optic
No

Then:

Fiber optic = 0
No = 0

means the reference category:

DSL

For beginner understanding, the key idea is:

One-hot encoding converts categories into machine-readable indicator variables without pretending the categories have numerical order.

43. Train/test split

You repeatedly used:

train_test_split(
    X,
    Y,
    test_size=0.3
)

This is one of the most important ML concepts.

Suppose we have 1,000 examples.

We might use:

700 → training
300 → testing

The training set is used to learn the model.

The test set is held back to evaluate how the trained model performs on unseen data.

Think of it like an exam:

Training data = practice questions
Test data     = unseen exam questions

44. Why not train and test on the same data?

Because the model could simply memorize the training examples.

Suppose:

Training score = 99%
Test score     = 65%

That is a warning sign.

The model learned the training data extremely well but does not generalize well.

This is called:

Overfitting

45. Underfitting

The opposite can happen.

If the model is too simple:

Training score = 60%
Test score     = 58%

It may not have learned enough from the data.

This is:

Underfitting

A useful mental picture:

Underfitting
Model too simple
       ↓
misses important patterns

Good fit
Learns useful patterns
       ↓
works on unseen data

Overfitting
Model learns noise/details
       ↓
great training performance
poor unseen performance

46. Why `random_state` matters

In one Telco notebook you used:

train_test_split(X, Y, test_size=0.3)

In the improved notebook you used:

random_state=42

A random state makes the split reproducible.

Without it, the random split can change between runs.

That means your score may change.

With:

random_state=42

you can reproduce the same split.

The number 42 is not magical.

Any fixed integer can serve this purpose.

47. Why `stratify=Y` is useful for classification

In your improved Telco notebook you used:

stratify=Y

This helps preserve the class distribution between training and testing data.

For example, if the full dataset contains:

73% No Churn
27% Churn

stratification aims to keep roughly the same proportion in both splits.

This is particularly useful when classes are imbalanced.

48. Logistic Regression

You used:

from sklearn.linear_model import LogisticRegression

model_lr = LogisticRegression()
model_lr.fit(X_train, Y_train)

Despite its name, Logistic Regression is commonly used for classification.

Its goal is to estimate the probability of belonging to a class.

For binary classification:

features
   ↓
Logistic Regression
   ↓
probability
   ↓
class

Example:

Probability of churn = 0.82

The model may classify that customer as:

Churn

depending on its decision threshold.

49. Why Logistic Regression is a good beginner model

It is useful because:

relatively simple
fast
often strong as a baseline
easier to interpret than many complex models
naturally suited to binary classification

A good ML habit is:

Start with a simple baseline before reaching for complex models.

50. Support Vector Machine / SVC

You used:

from sklearn.svm import SVC

model_lr = SVC()

The variable name model_lr is misleading here.

It is actually an SVC model.

SVC means:

Support Vector Classifier

The basic idea is to find a decision boundary that separates classes.

In simple 2D data:

Class A: ● ● ●

--------- decision boundary ---------

Class B: ▲ ▲ ▲

SVM tries to find a boundary with a useful margin between classes.

51. Why scaling matters for SVM

SVM is sensitive to feature scale.

Suppose one feature ranges from:

0–1

and another ranges from:

0–100,000

The larger-scale feature can dominate distance-related calculations.

That is why scaling is often important for:

SVM
KNN
Logistic Regression in many situations
neural networks

52. Decision Tree

You used:

DecisionTreeClassifier()

and later:

DecisionTreeRegressor()

A decision tree makes predictions using a sequence of questions.

Imagine:

Is tenure < 12?
       /       \
     Yes       No
     /           \
Is charge > X?   ...

Eventually the tree reaches a leaf containing a prediction.

This is attractive because it is easy to visualize conceptually.

53. Why trees can overfit

A tree can keep splitting the data until it creates extremely specific rules.

For example:

If feature A > 10
and feature B < 4
and feature C = 7
and feature D > 92
...

Eventually the tree may memorize training examples.

That is why you later tried:

max_depth=6
min_samples_split=10
min_samples_leaf=5

These are ways of controlling tree complexity.

54. Random Forest

You used:

RandomForestRegressor(
    n_estimators=200,
    max_depth=8,
    min_samples_leaf=5,
    random_state=42
)

A random forest combines many decision trees.

Instead of relying on one tree:

Tree 1 ─┐
Tree 2 ─┤
Tree 3 ─┤
Tree 4 ─┤ → combined prediction
...     ┘

The idea is that many different trees can produce a more robust prediction than a single tree.

This is an example of an ensemble method.

55. Bagging

You used:

BaggingRegressor(
    n_estimators=100,
    max_samples=0.7,
    max_features=0.7
)

Bagging means combining models trained on different samples of the data.

The overall idea:

Dataset
  ↓
Different samples
  ↓
Multiple models
  ↓
Combine predictions

The goal is often to reduce variance and make predictions more stable.

Random Forest can be viewed as a specialized tree-based ensemble that adds additional randomness in how trees are built.

56. AdaBoost

You used:

AdaBoostRegressor()

AdaBoost means:

Adaptive Boosting

Instead of training many independent models and simply averaging them, boosting builds models sequentially.

Conceptually:

Model 1
  ↓
find mistakes
  ↓
Model 2 focuses more on difficult cases
  ↓
Model 3 focuses further
  ↓
combine models

The key idea:

Later models try to improve on the weaknesses of earlier models.

57. K-Nearest Neighbors (KNN)

You used:

KNeighborsRegressor()

KNN predicts using nearby examples.

Imagine a new customer.

The algorithm asks:

"Which existing customers are most similar to this customer?"

Then it uses those neighbors to estimate the output.

For regression, the prediction is commonly based on the neighbors' target values.

58. Why KNN needs scaling

KNN relies on distance.

Imagine:

Income:   0–100,000
Age:      18–80

If raw values are used, income can dominate the distance calculation.

Scaling puts features onto comparable scales.

That is why you experimented with:

StandardScaler()

59. StandardScaler

You used:

scaler = StandardScaler()

X_train_scaled = scaler.fit_transform(X_train_log)
X_test_scaled = scaler.transform(X_test_log)

This is an important pattern.

Notice:

scaler.fit_transform(X_train)

but:

scaler.transform(X_test)

Why?

The scaler learns the training data's statistics.

Then the exact same transformation is applied to the test data.

You should not independently fit the scaler on the test set.

Otherwise information from the test set leaks into preprocessing.

This idea is called avoiding:

Data leakage

60. Data leakage

Data leakage happens when information that should not be available to the model during training accidentally influences the model.

Example:

Train preprocessing
     ↓
uses information from test data
     ↓
test is no longer truly unseen

That can make evaluation look better than real-world performance.

The correct principle is:

Fit preprocessing steps using training data, then apply them to validation/test/new data.

A production ML pipeline should ideally bundle preprocessing and modeling together.

61. Linear Regression

You used:

LinearRegression()

for Customer Lifetime Value.

Linear regression tries to model a relationship like:

Y = b0 + b1X1 + b2X2 + ...

For one feature:

Y = intercept + slope × X

The goal is to find coefficients that produce predictions close to the observed values.

62. Regression score: what does `.score()` mean?

You used:

model.score(X_train, Y_train)
model.score(X_test, Y_test)

For the scikit-learn regression models you used, .score() generally returns R², the coefficient of determination.

Very roughly:

R² = 1

means the model explains the observed variation perfectly on that dataset.

R² = 0

means the model is no better than a simple baseline based on predicting the mean target.

R² can also be negative on unseen data.

Important:

R² is not the same thing as accuracy.

For regression, do not say:

"My regression model has 90% accuracy"

just because R² is 0.90.

Instead say:

"The model achieved an R² of 0.90."

63. Classification accuracy

You used:

accuracy_score(Y_test, y_test_pred)

Accuracy is:

correct predictions
-------------------
total predictions

For example:

90 correct
100 total

gives:

90% accuracy

Accuracy is easy to understand.

But it can be misleading when classes are highly imbalanced.

64. Confusion matrix

Your later notes/workflow refer to concepts such as:

TP
TN
FP
FN

A confusion matrix organizes classification predictions.

                    Actual
                 No       Yes
Predicted No     TN       FN
Predicted Yes    FP       TP

Meaning:

True Positive

Model predicted positive and it was positive.

True Negative

Model predicted negative and it was negative.

False Positive

Model predicted positive but it was negative.

False Negative

Model predicted negative but it was positive.

This is much more informative than accuracy alone when the cost of errors differs.

65. Precision and Recall

For classification:

Precision

Of everything the model predicted as positive:

How many were actually positive?

Precision = TP / (TP + FP)

Recall

Of everything that was actually positive:

How many did the model find?

Recall = TP / (TP + FN)

The right metric depends on the problem.

For churn prediction, for example, missing a customer who is about to churn may be more important than contacting an extra customer who would have stayed.

66. Your Telco Customer Churn project

This is probably the clearest classification example in your notebooks.

The business question is:

Can we predict whether a telecom customer will churn?

The target is:

Y = df["Churn"]

Features are:

X = df.drop("Churn", axis=1)

The workflow becomes:

Customer data
     ↓
Clean data
     ↓
Explore data
     ↓
Encode categories
     ↓
Split train/test
     ↓
Train classifier
     ↓
Predict churn
     ↓
Evaluate

67. Telco: removing customerID

You did:

df = df.drop(columns=["customerID"])

This is reasonable because the ID is an identifier rather than a customer characteristic.

You also commented that it could be kept separately if you later wanted to map predictions back to customers.

That is a good practical idea.

In production:

customerID
   ↓
keep separately for business reporting

but:

customerID
   X
do not necessarily use as a model feature

68. Telco: fixing TotalCharges

You used:

df["TotalCharges"] = pd.to_numeric(
    df["TotalCharges"],
    errors="coerce"
)

errors="coerce" means invalid values are converted to NaN.

Then you used:

fillna(0)

in one notebook and median imputation in another.

These are two different decisions.

The important lesson is:

Missing-value handling should be based on the meaning of the data, not simply on whichever replacement is easiest.

For example, if missing TotalCharges occurs because a customer has zero tenure and has just joined, then filling with zero may have a meaningful business interpretation.

If missingness represents an unknown measurement, median imputation may be more appropriate.

69. Telco: the first modeling version

The earlier Telco notebook used:

LabelEncoder()

for many categorical columns.

This is okay as a learning exercise for understanding encoding, but it is not ideal for every categorical variable.

A better version in your later notebook used:

pd.get_dummies(
    df,
    columns=multi_cols,
    drop_first=True
)

That is a meaningful improvement.

70. Telco: your improved preprocessing pipeline

The later notebook did something closer to:

Raw data
   ↓
Drop customerID
   ↓
Convert TotalCharges
   ↓
Encode binary columns
   ↓
One-hot encode multi-category columns
   ↓
Separate X and Y
   ↓
Scale X
   ↓
Train Logistic Regression
   ↓
Create Gradio interface

This is much closer to a real ML application.

71. Gradio: turning a model into an application

You used:

import gradio as gr

and built a UI.

This is an important step because it changes the project from:

Notebook experiment

to:

User input
    ↓
Preprocessing
    ↓
Model
    ↓
Prediction
    ↓
Human-readable result

For example:

Gender: Female
Tenure: 12 months
Contract: Month-to-month
Monthly charges: $70
...
          ↓
    Predict Churn
          ↓
High Churn Risk
Probability: ...

This demonstrates an important ML engineering concept:

A model is useful only when it can be integrated into a workflow where people or systems can use its predictions.

72. Very important: preprocessing must match training

Your Gradio code contains a good lesson.

During training you:

encode the categorical values
create dummy columns
scale the features
train the model

During prediction you must do the same thing:

raw user input
    ↓
same encoding
    ↓
same columns
    ↓
same scaling
    ↓
model

If the training data has:

feature_1
feature_2
feature_3
feature_4

but the application sends:

feature_1
feature_3
feature_4

the model cannot interpret the input correctly.

Your use of:

input_df.reindex(
    columns=feature_columns,
    fill_value=0
)

is intended to make the input columns match the training feature structure.

That is an important practical idea.

73. A stronger production pattern: Pipeline

A cleaner production approach is often:

Pipeline([
    ("preprocessor", ...),
    ("model", LogisticRegression())
])

Then preprocessing and prediction are tied together.

This reduces the risk of accidentally applying different transformations during training and inference.

Your notebook demonstrates the concept manually, which is useful for learning.

74. Your Auto Insurance / Customer Lifetime Value project

The 13-Aug.ipynb notebook moves into regression.

The target is:

Y = df["Customer Lifetime Value"]

This changes the question from:

Will this customer churn?

to:

What numerical Customer Lifetime Value should we predict?

Therefore:

Classification → Churn
Regression     → Customer Lifetime Value

75. Insurance dataset: initial exploration

You loaded:

df = pd.read_csv("AutoInsurance.csv")

Then inspected:

df.info()
df.shape
df.describe()

You also removed:

Customer
Effective To Date

from the modeling DataFrame.

Again, the purpose is to separate identifiers / dates that were not being used in the current modeling approach.

However, an important ML lesson is:

Dates are not automatically useless.

A date may contain useful information such as:

month
day of week
season
year
time since an event

A stronger project could engineer useful date features instead of simply dropping every date column.

76. Insurance EDA

You explored:

sns.countplot(x="State", data=df)
sns.boxplot(y=df["Income"])
sns.boxplot(
    x="Coverage",
    y="Customer Lifetime Value",
    data=df
)
sns.scatterplot(
    x="Income",
    y="Customer Lifetime Value",
    data=df
)
df["Income"].hist()

This is a good progression:

Categorical distribution
        ↓
Numerical distribution
        ↓
Numerical vs categorical
        ↓
Numerical vs numerical

You are gradually asking:

"What might explain the target?"

77. Automatically finding numerical and categorical columns

You used:

numerical_cols = df.select_dtypes(
    include=["int64", "float64"]
).columns

categorical_cols = df.select_dtypes(
    include=["object"]
).columns

This is useful because instead of manually listing every column, Python can identify columns based on data type.

Then you looped through them to create graphs.

This is the beginning of writing reusable data-analysis code.

78. Automated boxplots

You created boxplots for all numerical columns.

This is useful because you can quickly inspect:

distributions
median
spread
potential outliers

However, remember that a large collection of graphs can become overwhelming.

The goal should eventually move from:

"Let's plot everything."

to:

"Let's plot the variables that help answer our question."

That is a sign of growing from beginner EDA toward professional analysis.

79. Binning numerical variables for boxplots

You used:

pd.cut(
    df[col],
    bins=5
)

This divides a numerical variable into five intervals.

Then you compare CLV across those ranges.

Conceptually:

Income range 1 → CLV distribution
Income range 2 → CLV distribution
Income range 3 → CLV distribution
...

This is useful for visualization because a boxplot expects categories on one axis.

But the number and boundaries of bins can affect the story you see.

So binning is mainly a visualization technique here, not necessarily something you should automatically use as a model feature.

80. Comparing many regression algorithms

One of the biggest learning steps in your final notebook was trying multiple regressors:

Linear Regression
KNN Regressor
SVR
Decision Tree Regressor
Bagging Regressor
AdaBoost Regressor
Random Forest Regressor

This is valuable because it teaches a core ML lesson:

Different algorithms make different assumptions and capture different types of patterns.

There is no universally best algorithm.

81. Linear Regression vs tree-based models

Linear Regression assumes a relationship that can be represented through a linear combination of features.

Tree-based methods can represent more complex non-linear relationships.

For example:

Linear model:

Y
│       /
│     /
│   /
│ /
└──────── X

A tree-based model can approximate more irregular patterns:

Y
│    ┌────
│    │
│ ───┘
│
└──────── X

This is why trying multiple model families can be useful.

82. Why model comparison matters

Suppose you obtain:

Model             Train R²    Test R²
Linear Regression   0.70       0.68
KNN                 0.90       0.62
Decision Tree       0.99       0.55
Random Forest       0.91       0.78

You should not automatically choose the model with the highest training score.

The more important question is:

Which model generalizes well to unseen data?

Here Random Forest would be more interesting than the tree with 0.99 training R².

83. Log transformation of Customer Lifetime Value

You later used:

Y_log = np.log1p(Y)

Why?

Because some target variables are heavily skewed.

Imagine:

Most CLV values: relatively small
A few CLV values: extremely large

The model may have difficulty fitting such a distribution.

A log transformation compresses large values.

Conceptually:

Original:
1
10
100
1000
10000

Log scale:
small differences between large values

This can make the target distribution easier for some models to learn.

84. Reversing the log transformation

After predicting:

predictions_log = model.predict(X)

you used:

predictions_actual = np.expm1(predictions_log)

This is important.

If:

Y_log = log(1 + Y)

then:

Y = exp(Y_log) - 1

np.expm1() performs:

exp(x) - 1

So:

log target
   ↓
model
   ↓
log prediction
   ↓
expm1
   ↓
original target scale

85. An important issue in the notebook: split consistency

In your log-transform section, you did:

Y_log = np.log1p(Y)

X_train_log, X_test_log, Y_train_log, Y_test_log = train_test_split(
    X,
    Y_log,
    test_size=0.3,
    random_state=42
)

This is okay as a new split, but it means your earlier train/test split and later train/test split are not necessarily the same.

For a clean experiment, define the split once and reuse it consistently.

For example:

X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.3,
    random_state=42
)

y_train_log = np.log1p(y_train)
y_test_log = np.log1p(y_test)

This makes comparisons easier to reason about.

86. Another important issue: predicting on the full dataset

You later did:

predictions_log = model_rf.predict(X)

and saved predictions for the full dataset.

That can be useful for producing a business-facing prediction file.

But these are not unbiased test predictions.

Why?

Because X includes training observations the model already saw.

So:

Predictions on X_test
→ useful for evaluating unseen data

Predictions on X
→ useful for generating predictions for all records,
   but not for measuring generalization

This distinction is very important.

87. Hyperparameters

Later you changed models from:

DecisionTreeRegressor()

to:

DecisionTreeRegressor(
    max_depth=6,
    min_samples_split=10,
    min_samples_leaf=5
)

These settings are called hyperparameters.

They are chosen by us before training.

The model learns its internal parameters from the training data.

Parameters

Learned by the algorithm.

Hyperparameters

Set by us.

This distinction is fundamental.

88. Understanding your tree hyperparameters

`max_depth`

Controls how deep the tree can grow.

Smaller:

simpler tree
less overfitting

Larger:

more complex tree
greater overfitting risk

`min_samples_split`

Minimum number of samples required before a node can be split.

Higher values make the tree more conservative.

`min_samples_leaf`

Minimum number of samples allowed in a leaf.

Higher values prevent extremely tiny leaves.

89. Random Forest hyperparameters

You used:

RandomForestRegressor(
    n_estimators=200,
    max_depth=8,
    min_samples_leaf=5,
    random_state=42
)

`n_estimators=200`

Build 200 trees.

More trees can make the ensemble more stable, though computation increases.

`max_depth=8`

Limits tree complexity.

`min_samples_leaf=5`

Prevents leaves from becoming too specific.

`random_state=42`

Makes the experiment reproducible.

90. Bagging hyperparameters

You used:

BaggingRegressor(
    n_estimators=100,
    max_samples=0.7,
    max_features=0.7,
    random_state=42
)

This means, conceptually:

build many estimators
each estimator sees a sample of the training rows
each estimator can use a subset of features
combine their predictions

Again, the goal is to create a robust ensemble rather than rely on one model.

91. Scaling experiment in your notebook

You used:

StandardScaler()

and compared KNN/SVR.

This is a very useful experiment because it demonstrates:

Preprocessing requirements depend on the algorithm.

Tree-based algorithms generally do not need feature scaling in the same way that distance- or margin-based methods do.

KNN and SVM are much more sensitive to feature scale.

92. A subtle issue in your scaling experiment

You created:

X_train_scaled
X_test_scaled

but your KNN code first trained on:

X_train_log

before later looping over scaled KNN.

That is useful as experimentation, but for a clean article you should present the comparison more systematically:

KNN without scaling
        vs
KNN with scaling

and explicitly explain why the result changed.

For SVR, you correctly used the scaled data in the shown section.

93. The complete mental model

After these five days, the entire workflow can be remembered as:

             REAL-WORLD QUESTION
                     ↓
             Define the target
                     ↓
              Collect data
                     ↓
              Understand data
                     ↓
          ┌──────────┴──────────┐
          ↓                     ↓
      Numerical             Categorical
          ↓                     ↓
      distributions        category counts
          └──────────┬──────────┘
                     ↓
                    EDA
                     ↓
              Clean the data
                     ↓
          Handle missing values
                     ↓
             Encode categories
                     ↓
          Select useful features
                     ↓
              Split train/test
                     ↓
          Scale when appropriate
                     ↓
              Train baseline
                     ↓
          Evaluate on test data
                     ↓
             Try other models
                     ↓
         Tune hyperparameters
                     ↓
       Select based on validation
                     ↓
             Final evaluation
                     ↓
              Make predictions
                     ↓
       Integrate into an application

94. What each notebook taught you

Day 1 — Auto MPG

Main lessons:

load a dataset
inspect rows and columns
understand shape and data types
inspect summary statistics
identify categorical/text columns
detect messy numerical values
visualize distributions
use countplots
use histograms
use pairplots
understand missing values
convert "?" to NaN
convert text to numerical data
use median imputation

The biggest lesson:

Before modeling, understand and clean your data.

Day 2 — Telco Customer Churn

Main lessons:

identify a classification target
remove an identifier
handle TotalCharges
visualize class balance
compare categories against churn
use boxplots and histograms
use pairplots
inspect correlations
encode categorical data
split into training/testing data
train Logistic Regression
train SVC
train Decision Tree
compare train/test performance

The biggest lesson:

A classification model predicts categories, but the quality of the prediction depends heavily on preprocessing and evaluation.

Day 3 — Improving the Telco project

Main lessons:

one-hot encode multi-category variables
use random_state
use stratify
scale features
train Logistic Regression
build a prediction interface with Gradio
reproduce preprocessing at inference time
output a human-readable prediction

The biggest lesson:

An ML model is only one part of an ML application.

Day 4/5 — Customer Lifetime Value

Main lessons:

formulate a regression problem
inspect numerical/categorical relationships
automate EDA
encode categories
split train/test
compare multiple regressors
understand R²
try target transformation
use log transformation
reverse the transformation
experiment with scaling
tune hyperparameters
use ensemble models
generate prediction files

The biggest lesson:

Different models capture different patterns, and model selection should be based on performance on unseen data—not training performance alone.

95. What I would change before publishing the project as "production-quality"

Your notebooks are excellent learning material because they show experimentation. For a public article, however, it is worth separating:

Learning experiments

from:

Recommended ML workflow

Here are the most important improvements.

Improvement 1 — Use clear train/test terminology

Prefer:

X_train, X_test, y_train, y_test

rather than mixing Y and y.

Python convention generally uses lowercase y.

Improvement 2 — Make experiments reproducible

Use:

random_state=42

consistently when appropriate.

Improvement 3 — Use stratification for classification

For churn:

train_test_split(
    X,
    y,
    test_size=0.3,
    random_state=42,
    stratify=y
)

Improvement 4 — Avoid LabelEncoder for nominal multi-class input features

Use one-hot encoding for categories such as:

PaymentMethod
Contract
InternetService
VehicleClass

Label encoding is much more naturally suited to targets or truly ordinal categories.

Improvement 5 — Fit preprocessing only on training data

For example:

Training data
    ↓
fit scaler
    ↓
transform training

Test data
    ↓
transform using existing scaler

Do not independently fit preprocessing on the test data.

Improvement 6 — Use proper evaluation metrics

For classification, consider:

Accuracy
Precision
Recall
F1
Confusion Matrix
ROC-AUC
PR-AUC

For regression:

R²
MAE
MSE
RMSE

Which metrics matter depends on the business problem.

Improvement 7 — Don't compare only training scores

Always look at:

Training performance
+
Validation/test performance

A very high training score with much lower test performance can indicate overfitting.

Improvement 8 — Keep a validation strategy for model selection

A single test set should ideally be kept for final evaluation.

During experimentation, use cross-validation or a validation set to choose models/hyperparameters.

Conceptually:

Training data
   ↓
Cross-validation / validation
   ↓
Choose model
   ↓
Final test set
   ↓
Final unbiased-ish estimate

96. A very important distinction: EDA vs ML

EDA asks:

"What does my data look like?"

Machine learning asks:

"Can I learn a useful relationship that lets me make predictions on new data?"

These overlap, but they are not the same.

For example:

sns.scatterplot(...)

does not build a model.

It helps you understand the data before deciding how to model it.

97. Another important distinction: correlation vs prediction

A strong correlation can be useful.

But:

high correlation

does not automatically mean:

excellent predictive model

Prediction depends on:

multiple features
data quality
model type
noise
generalization
preprocessing
evaluation

So don't judge an ML problem from one graph.

98. Another important distinction: model vs algorithm

It is common to hear:

"I trained a model."

The algorithm is the learning procedure.

For example:

Random Forest = algorithm/model family

After training:

model_rf.fit(X_train, y_train)

you have a fitted model containing learned information.

A useful mental model:

Algorithm
   +
Training data
   ↓
Fitted model
   ↓
Predictions

99. What "learning" actually means

This is probably the most important concept to understand.

When you run:

model.fit(X_train, y_train)

the model is not magically understanding the world.

It is optimizing internal parameters so that its predictions match the training examples according to the algorithm's objective.

For example, linear regression learns coefficients.

A tree learns split rules.

A neural network learns weights.

So:

Training means finding model parameters that make the model perform well according to a defined objective.

100. What happens during prediction?

Once training is complete:

prediction = model.predict(X_test)

the model does not learn again from the test examples.

It applies what it learned during training.

Conceptually:

Training:

X_train + y_train
        ↓
      model.fit()
        ↓
   learned model

Prediction:

X_test
   ↓
learned model
   ↓
prediction

101. The biggest beginner misconception to avoid

Do not think:

"I used Random Forest, so I did machine learning."

The real ML skill is being able to explain:

What problem are you solving?
What is the target?
What features are available?
What does the data look like?
What problems did you find?
How did you clean them?
How did you encode the data?
Why did you choose the algorithm?
How did you evaluate it?
Does it generalize?
What would you improve?

If you can explain these, you are learning ML—not just memorizing scikit-learn commands.

102. A simple explanation of your five-day journey

If someone asks:

"What did you actually learn in five days?"

A strong answer is:

I learned that machine learning is not just about selecting an algorithm. I started by understanding datasets using Pandas, cleaning missing and incorrectly formatted values, and using visualization to understand distributions and relationships. Then I learned the difference between classification and regression, converted categorical data into numerical representations, split data into training and testing sets, trained several models using scikit-learn, and compared their performance on unseen data. Finally, I experimented with scaling, transformations, ensemble models, hyperparameters, and even built a simple Gradio interface around a churn model.

That is a much stronger story than:

"I learned Logistic Regression, SVM and Random Forest."

103. Suggested article structure for dev.to

For your final public article, I recommend this structure:

Title

1. Why I started learning ML
2. What I thought ML was vs what I learned
3. AI vs ML
4. Types of ML
5. My five-day learning roadmap
6. Day 1 — Understanding and cleaning data
7. EDA and why graphs matter
8. Day 2 — Classification with customer churn
9. Encoding categorical data
10. Train/test split
11. Logistic Regression, SVM and Decision Trees
12. How I evaluated classification models
13. Day 3 — Turning the model into an application
14. Day 4/5 — Regression and Customer Lifetime Value
15. Linear Regression vs KNN vs SVR vs Trees vs Ensembles
16. Log transformation
17. Scaling
18. Hyperparameters
19. What I got wrong / what I would improve
20. What I learned
21. What I plan to learn next

This will make the article feel like a journey, rather than a textbook.

104. Recommended article narrative

The most interesting story is:

"I started with almost no ML knowledge."

        ↓

"I learned that data preparation matters."

        ↓

"I discovered that graphs help me ask questions
before training models."

        ↓

"I learned classification."

        ↓

"I learned regression."

        ↓

"I tried different algorithms and realized
there is no single best model."

        ↓

"I learned that a high training score
doesn't necessarily mean a good model."

        ↓

"I turned one model into a small application."

        ↓

"I now understand the basic ML workflow
and know what I need to learn next."

That is a much more authentic beginner-to-ML story.

105. What to learn next

Based on what your notebooks already cover, I would not recommend immediately jumping into deep learning.

First strengthen these foundations:

1. Statistics

Learn:

mean
median
variance
standard deviation
distributions
probability
correlation
conditional probability

2. ML evaluation

Learn:

confusion matrix
precision
recall
F1
ROC-AUC
MAE
RMSE
R²
cross-validation

3. Feature engineering

Learn:

handling dates
categorical features
transformations
interactions
outliers
missing values

4. Model selection

Learn:

cross-validation
hyperparameter tuning
GridSearchCV
RandomizedSearchCV
pipelines

5. ML fundamentals

Learn:

bias vs variance
underfitting
overfitting
regularization
data leakage
feature importance

Then move toward:

Classical ML
     ↓
Feature engineering
     ↓
Model selection
     ↓
Cross-validation
     ↓
ML pipelines
     ↓
Deployment
     ↓
Deep Learning
     ↓
Generative AI / LLMs

106. The one-page cheat sheet

Problem type

Predict a category → Classification
Predict a number   → Regression
Find hidden groups → Clustering
Learn through reward → Reinforcement Learning

Data

Rows       → observations
Columns    → variables/features
X          → input features
y          → target

First inspection

df.head()
df.shape
df.info()
df.describe()
df.sample()
df.isnull().sum()

Visualization

Countplot   → categories
Histogram   → numerical distribution
Boxplot     → spread/outliers
Scatterplot → two numerical variables
Pairplot    → many numerical relationships
Heatmap     → correlation matrix

Preparation

Missing values
Data types
Categorical encoding
Feature selection
Scaling when needed

Training

model.fit(X_train, y_train)

Prediction

model.predict(X_test)

Evaluation

Classification:

Accuracy
Precision
Recall
F1
Confusion Matrix
ROC-AUC

Regression:

R²
MAE
MSE
RMSE

Core warning signs

Training score >> Test score
        ↓
Possible overfitting

Test information used during preprocessing
        ↓
Possible data leakage

Identifier used as feature
        ↓
Possible meaningless pattern

Categorical values converted to arbitrary numbers
        ↓
Possible false ordering

107. Final takeaway

The biggest lesson from these five days is not a particular Python library or algorithm.

It is the workflow:

Understand the problem → understand the data → clean the data → explore the data → prepare features → train → evaluate → improve → deploy.

The algorithms are tools inside that workflow.

You have already touched many of the important building blocks:

Pandas
NumPy
Matplotlib
Seaborn
Scikit-learn
data cleaning
EDA
classification
regression
encoding
train/test split
scaling
model evaluation
hyperparameters
ensemble models
target transformation
Gradio

The next step is not to memorize more algorithms.

The next step is to become comfortable answering:

Why am I doing this step?

If you can answer that question for every important line in your notebook, you are moving from "following an ML tutorial" to actually understanding machine learning.

Appendix A — Notebook-by-notebook step map

`11_Aug_mpg.ipynb`

Notebook step	Why it was done	ML concept
Import NumPy/Pandas/Matplotlib/Seaborn	Load tools	Python ML stack
`read_csv()`	Load data	Data ingestion
`nunique()`	Understand uniqueness	Data exploration
`head()`	Inspect examples	EDA
`shape`	Understand dataset size	EDA
`describe()`	Summary statistics	EDA
`info()`	Types/non-null values	Data quality
`countplot()`	Inspect categorical distribution	Visualization
`hist()`	Inspect numerical distributions	Visualization
`isnull()`	Detect actual missing values	Data cleaning
`pairplot()`	Explore relationships	EDA
`sample()`	Inspect random examples	Data quality
Identify `car name`	Recognize identifier-like field	Feature selection
Replace `?`	Convert fake missing values	Data cleaning
`to_numeric()`	Make horsepower numerical	Data preparation
Median imputation	Fill missing horsepower	Missing-value handling
Train/test split	Separate learning/evaluation data	Generalization
Logistic Regression	Start classification experiment	Supervised learning

Important caveat

The notebook switches from Auto MPG to a loan_prediction.csv dataset near the end. That appears to be a copied/reused modeling experiment rather than a continuation of the Auto MPG workflow. For a polished article, present these as separate experiments rather than one continuous Auto MPG pipeline.

`12th-Aug-Telco-Customer-churn.ipynb`

Step	Purpose
Load Telco CSV	Get customer data
`head()`	Inspect records
`shape`	Understand size
`describe()`	Statistical overview
Drop `customerID`	Remove identifier from features
`info()`	Check data types
`sample()`	Inspect random rows
Countplot of Churn	Check class distribution
Countplot of gender	Explore category distribution
Histograms	Explore numerical distributions
Pairplot	Explore relationships
Convert TotalCharges	Fix numeric type
Fill missing TotalCharges	Handle missing data
LabelEncoder	Convert categories to numbers
Train/test split	Evaluate on unseen data
Logistic Regression	Classification model
SVC	Alternative classifier
Decision Tree	Alternative classifier
Gradio	Turn model into interactive app

Important caveat

The notebook uses LabelEncoder on many categorical input columns. This is useful for learning encoding, but one-hot encoding is generally safer for nominal multi-category features.

`12-Aug-2.ipynb`

This notebook is a more developed version of the Telco project.

Important improvements include:

dropping customerID
converting TotalCharges
one-hot encoding multi-category variables
using random_state=42
using stratify=Y
scaling features
training Logistic Regression
building a Gradio prediction interface
preserving training feature columns
applying the same preprocessing to new input

This is the notebook that most clearly demonstrates the transition from:

ML experiment

to:

small ML application

`13-Aug.ipynb`

Main steps:

Load AutoInsurance
       ↓
Inspect dataset
       ↓
Drop Customer / date columns
       ↓
EDA
       ↓
Identify numerical/categorical columns
       ↓
Generate many visualizations
       ↓
Encode categorical variables
       ↓
Set CLV as target
       ↓
Train/test split
       ↓
Try multiple regressors
       ↓
Try log target
       ↓
Try scaling
       ↓
Tune tree/ensemble hyperparameters
       ↓
Generate predictions
       ↓
Save prediction CSVs

This notebook demonstrates the transition from classification to regression and from a single baseline model to model comparison and tuning.

Appendix B — A cleaner ML template to remember

# 1. Load
df = pd.read_csv("data.csv")

# 2. Understand
df.head()
df.shape
df.info()
df.describe()

# 3. Clean
# handle missing values
# fix data types
# remove inappropriate identifiers

# 4. Explore
# histograms
# boxplots
# countplots
# scatterplots
# correlation

# 5. Prepare
X = df.drop("target", axis=1)
y = df["target"]

# 6. Split
X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.2,
    random_state=42
)

# 7. Preprocess
# encoding / scaling

# 8. Train
model.fit(X_train, y_train)

# 9. Predict
predictions = model.predict(X_test)

# 10. Evaluate
# choose metrics appropriate to the problem

# 11. Compare / improve
# try another model
# tune hyperparameters
# use cross-validation

# 12. Final model
# train using the selected approach

# 13. Predict new data

Closing thought for the article

Five days ago, an ML notebook could easily look like a collection of unfamiliar commands.

Now there is a structure behind those commands.

head() is not just a command — it is a way to understand the data.

isnull() is not just a command — it is part of data quality.

A histogram is not just a graph — it tells you about a distribution.

train_test_split() is not just boilerplate — it protects your evaluation from simply measuring memorization.

fit() is not magic — it is the learning stage.

predict() is the model applying what it learned.

And model comparison is not about finding the fanciest algorithm — it is about finding an approach that generalizes well to data the model has never seen.

That, more than anything else, is what I took away from my first five days of machine learning.

AI & ML Foundations: Cleaning Data, Building Regression & Classification Models in Python

yash-nigam — Mon, 31 Aug 2026 16:49:07 +0000

My 5-Day Journey into Machine Learning: From Data Cleaning to My First ML Models

Learning goal: This is not a collection of code snippets to memorize. It is a beginner-friendly guide to understanding why each step in a typical machine-learning workflow exists, what problem it solves, and how the pieces fit together.

A note about the source material

This guide was built primarily from the four Jupyter notebooks you provided:

11_Aug_mpg.ipynb
12-Aug-2.ipynb
12th-Aug-Telco-Customer-churn.ipynb
13-Aug.ipynb
123

1. Overview: What did I actually learn in these five days?

At first glance, machine learning can look like this:

Load dataset
    ↓
Clean data
    ↓
Create graphs
    ↓
Train model
    ↓
Check accuracy
    ↓
Done!

But that misses the most important part.

The real ML workflow is closer to:

Business / real-world question
        ↓
Define what we want to predict
        ↓
Understand the data
        ↓
Clean and prepare the data
        ↓
Explore the data visually
        ↓
Choose useful features
        ↓
Convert data into numbers
        ↓
Split into training and testing data
        ↓
Choose an appropriate ML algorithm
        ↓
Train the model
        ↓
Evaluate it on unseen data
        ↓
Compare models
        ↓
Improve preprocessing / model settings
        ↓
Use the model to make predictions

The important insight is:

Machine learning is not mainly about choosing an algorithm. It is about turning a real-world question and messy data into a reliable prediction process.

Your notebooks actually demonstrate most of this workflow.

2. What is Artificial Intelligence?

Artificial Intelligence (AI) is the broad idea of building systems that can perform tasks that normally require some form of human intelligence.

Examples:

recognizing an image
understanding language
recommending a movie
detecting fraud
predicting whether a customer will leave
predicting the price/value of something
generating text or images

A useful mental model is:

Artificial Intelligence
│
├── Machine Learning
│   │
│   ├── Supervised Learning
│   ├── Unsupervised Learning
│   └── Reinforcement Learning
│
├── Deep Learning
│
└── Generative AI

This is simplified because these areas overlap, but it is a useful beginner mental model.

3. What is Machine Learning?

Machine learning is a way of building systems where the computer learns patterns from data instead of us manually writing every rule.

Imagine we want to predict whether a telecom customer will churn.

A traditional rule-based approach might look like:

IF customer has a month-to-month contract
AND monthly charges are high
AND tenure is low
THEN churn = yes

The problem is that we have to invent those rules ourselves.

With machine learning, we give the algorithm historical examples:

Customer information → Actual churn result

Customer A → No
Customer B → Yes
Customer C → No
Customer D → Yes
...

The algorithm tries to learn the relationship between the input information and the known outcome.

Then we can give it a new customer:

New customer information
        ↓
      Model
        ↓
Predicted churn

That is the core idea of supervised machine learning.

4. The most important ML vocabulary

Before learning algorithms, understand these words.

4.1 Dataset

A dataset is a collection of examples.

In your notebooks, a dataset is loaded into a Pandas DataFrame.

For example:

df = pd.read_csv("Telco_Customer_Churn.csv")

Think of the DataFrame as a spreadsheet.

4.2 Row / observation / sample

Each row represents one example.

For the Telco dataset:

one row = one customer

For the Auto MPG dataset:

one row = one car

For the insurance dataset:

one row = one customer/policy record

These words are often used interchangeably:

row
observation
sample
instance

4.3 Feature

A feature is an input variable that the model can use to make a prediction.

For customer churn:

tenure
MonthlyCharges
Contract
InternetService
PaymentMethod
...

These are features.

A useful question to ask is:

"What information is available to the model before it makes its prediction?"

That information is generally your feature set.

4.4 Target

The target is what we are trying to predict.

Examples from your notebooks:

Telco

Target = Churn

The model predicts:

0 → No churn
1 → Churn

Insurance

Target = Customer Lifetime Value

The model predicts a numerical value.

4.5 X and y

You repeatedly used:

X = df.drop(["Churn"], axis=1)
Y = df["Churn"]

This is one of the most important patterns in supervised ML.

X = inputs/features
Y = target/output

For example:

X
├── tenure
├── MonthlyCharges
├── Contract
├── InternetService
└── ...

Y
└── Churn

Think:

X = what the model knows

Y = what we want the model to learn to predict

5. The three major types of Machine Learning

5.1 Supervised Learning

In supervised learning, we have examples where the correct answer is already known.

Input X → Known answer Y

The model learns the relationship between X and Y.

Your notebooks mainly use supervised learning.

There are two major supervised learning tasks.

Classification

The target is a category.

Examples:

Spam / Not Spam
Churn / No Churn
Fraud / Not Fraud
Disease / No Disease

Your Telco project is a classification problem.

Regression

The target is a numerical value.

Examples:

House price
Temperature
Sales
Customer Lifetime Value
Fuel efficiency

Your Customer Lifetime Value project is a regression problem.

6. Classification vs Regression

This distinction is extremely important.

Classification

Question:

"Which category does this example belong to?"

Example:

Customer → Churn or No Churn

Possible output:

0
1

Common classification algorithms you tried:

Logistic Regression
Support Vector Classifier (SVC)
Decision Tree Classifier

Regression

Question:

"What numerical value should we predict?"

Example:

Customer → Customer Lifetime Value = 6,543.21

Common regression algorithms you tried:

Linear Regression
KNN Regressor
SVR
Decision Tree Regressor
Bagging Regressor
AdaBoost Regressor
Random Forest Regressor

7. Unsupervised Learning

In unsupervised learning, we do not have a known target.

Instead, the algorithm tries to discover structure in the data.

For example:

Customer data
     ↓
Find groups
     ↓
Group 1: high-value customers
Group 2: price-sensitive customers
Group 3: new customers

Common examples:

K-Means clustering
Hierarchical clustering
PCA

You did not build an unsupervised-learning project in these notebooks, but it is important to know where it fits.

8. Reinforcement Learning

Reinforcement learning is different.

An agent interacts with an environment and receives rewards or penalties.

Agent
  ↓ action
Environment
  ↓
Reward / penalty
  ↓
Agent learns

Examples:

game-playing agents
robotics
certain recommendation/control systems

This was not part of your five-day practical work.

9. Different ways to approach an ML problem

A useful way to think about ML is not:

"Which algorithm should I use?"

Instead ask:

Step 1 — What is the question?

For example:

Can I predict whether a telecom customer will churn?

Step 2 — What type of target do I have?

Churn → category → Classification
Customer Lifetime Value → number → Regression

Step 3 — What data is available?

Ask:

What columns exist?
Which are numerical?
Which are categorical?
Are values missing?
Are there strange values?
Are some columns identifiers?
Is the target balanced?

Step 4 — Explore before modeling

Use statistics and graphs.

Step 5 — Prepare the data

Convert it into a form the algorithm can understand.

Step 6 — Establish a baseline

Train a simple model first.

Step 7 — Compare alternatives

Try other appropriate models.

Step 8 — Evaluate on unseen data

A model is useful only if it works beyond the data it memorized.

10. Google Colab: Why did we use it?

Google Colab gives you a cloud-based Jupyter Notebook environment.

Instead of installing everything locally, you can run Python code in the browser.

A notebook combines:

Explanation
+
Python code
+
Output
+
Graphs

This makes it very useful for learning ML.

Your notebooks use paths such as:

/content/auto-mpg.csv

That is typical of a Colab environment.

A normal learning workflow is:

Upload dataset
      ↓
Open notebook
      ↓
Run cells
      ↓
Inspect output
      ↓
Change code
      ↓
Run again

11. The Python libraries you used

11.1 NumPy

import numpy as np

NumPy provides numerical operations and arrays.

You used it for things such as:

np.nan
np.log1p()
np.expm1()

`np.nan`

Represents a missing numerical value.

`np.log1p(x)`

Calculates:

log(1 + x)

It is useful when a target is heavily skewed.

`np.expm1(x)`

Reverses log1p:

exp(x) - 1

You used this later to convert log predictions back to the original CLV scale.

12. Pandas

import pandas as pd

Pandas is one of the most important Python libraries for data work.

Think of a Pandas DataFrame as a spreadsheet that Python can manipulate.

You used it to:

load CSV files
inspect data
remove columns
convert data types
detect missing values
fill missing values
select X and Y
create encoded columns
save predictions

Example:

df = pd.read_csv("Telco_Customer_Churn.csv")

13. Matplotlib

import matplotlib.pyplot as plt

Matplotlib is a general-purpose plotting library.

It provides the foundation for many Python visualizations.

You used it for:

plt.figure()
plt.title()
plt.show()

14. Seaborn

import seaborn as sns

Seaborn makes statistical visualizations easier to create.

You used:

countplot
boxplot
histplot
pairplot
scatterplot
heatmap

A key lesson:

A graph is not decoration. It is a tool for asking questions about the data.

15. Scikit-learn

Scikit-learn is the main ML library used in your notebooks.

You used it for:

splitting data
preprocessing
encoding
scaling
classification
regression
evaluation

The basic pattern is:

model = SomeModel()
model.fit(X_train, Y_train)
predictions = model.predict(X_test)

This pattern appears again and again in ML.

16. The most important workflow: EDA

EDA means:

Exploratory Data Analysis

EDA is the process of investigating the dataset before building the final model.

Your notebooks use:

df.head()
df.shape
df.describe()
df.info()
df.sample()
df.isnull().sum()

plus graphs.

These are not random commands.

They answer different questions.

17. `df.head()` — What does the data look like?

You used:

df.head()

It displays the first few rows.

Why?

Because before doing anything else, you want to see:

column names
example values
obvious data problems
whether the data loaded correctly

Think of it as opening the box before using what is inside.

18. `df.shape` — How much data do I have?

You used:

df.shape

For Auto MPG:

(398, 9)

That means:

398 rows
9 columns

Why does this matter?

Because dataset size affects:

model choice
computation time
confidence in results
risk of overfitting

19. `df.info()` — What types of data do I have?

You used:

df.info()

This tells you:

column names
number of non-null values
data types
memory usage

This is extremely important.

For example, in Auto MPG:

horsepower → object

At first glance horsepower sounds numerical.

But Pandas sees it as text.

Why?

Because the dataset contains values such as:

A column containing:

cannot be treated as a clean numerical column.

So:

df.info() helps you detect data-type problems before modeling.

20. `df.describe()`

You used:

df.describe()

and sometimes:

df.describe(include="all")

This gives summary statistics.

For numerical data you get things such as:

count
mean
standard deviation
minimum
25th percentile
median
75th percentile
maximum

This helps you understand the scale and distribution of variables.

For example:

minimum
   ↓
25%
   ↓
median
   ↓
75%
   ↓
maximum

A large difference between the 75th percentile and maximum can be a clue that there may be extreme values.

21. `df.sample()`

You used:

df.sample(5)

or similar.

This shows random rows.

Why?

Because the first rows may not represent the whole dataset.

Random samples can expose:

unusual values
formatting issues
unexpected categories
data-entry problems

22. Missing data

You used:

df.isnull().sum()

This asks:

"How many missing values are there in each column?"

Missing data is important because many ML algorithms cannot directly work with missing values.

But there is a subtle lesson from your Auto MPG notebook.

You first checked:

df.isnull().sum()

while horsepower still contained "?".

Pandas did not count "?" as a missing value because "?" is a string, not a true NaN.

That means:

"?" ≠ NaN

So data cleaning sometimes requires identifying fake missing values first.

23. Auto MPG: cleaning horsepower

Your notebook did:

df["horsepower"] = df["horsepower"].replace("?", np.nan)

This converts the placeholder into a real missing value.

Then:

df["horsepower"] = pd.to_numeric(df["horsepower"])

Now Pandas can treat horsepower as numerical data.

This is a very important data-cleaning pattern:

messy text
   ↓
identify invalid placeholder
   ↓
convert to NaN
   ↓
convert column to numeric
   ↓
handle missing values

24. Why use the median?

You calculated:

median_horsepower = df["horsepower"].median()

and then filled missing values:

df["horsepower"] = df["horsepower"].replace(
    np.nan,
    median_horsepower
)

Why median?

Because the median is less affected by extreme values than the mean.

Example:

10, 11, 12, 13, 1000

The mean is pulled heavily upward by 1000.

The median is much more representative of the middle of the data.

A useful rule of thumb:

fairly symmetric data → mean may work well
skewed data / outliers → median is often safer

But remember:

There is no universal rule that "median is always correct."

The best treatment depends on the data and the problem.

25. Dropping an identifier

You considered dropping:

df.drop("car name", axis=1)

and actually dropped customerID in the Telco work.

Why?

An identifier is usually not a meaningful predictive feature.

For example:

customerID = 759832

does not inherently mean the customer is more or less likely to churn.

The ID exists to identify the row, not to describe the customer.

Important distinction:

A column can be useful for identifying a record without being useful for predicting the target.

26. But don't blindly drop columns

This is an important improvement to your original reasoning.

A column should not be dropped merely because it is an object/string.

For example:

Contract
InternetService
PaymentMethod

are categorical strings, but they can contain very useful predictive information.

So the correct question is:

"Does this column contain useful information that is available at prediction time?"

not:

"Is this column numeric?"

27. Understanding your graphs

You created many graphs. The important thing is to understand what question each graph answers.

28. Countplot

Example:

sns.countplot(x="Churn", data=df)

A countplot answers:

"How many observations are in each category?"

For churn:

No churn → number of customers
Churn    → number of customers

This immediately helps you understand class balance.

If the graph looks like:

No Churn ███████████████████
Churn    ███████

the classes are imbalanced.

That matters because a model could achieve high accuracy simply by favoring the majority class.

29. Countplot with `hue`

You used:

sns.countplot(
    x="Contract",
    hue="Churn",
    data=df
)

This answers:

"How does the target category vary across another category?"

For example:

Contract type
     ↓
Churn / No churn

If month-to-month customers show much more churn than two-year customers, that is an important pattern worth investigating.

But remember:

A graph showing association does not automatically prove causation.

30. Histogram

You used:

df.hist(figsize=(10,10))

and:

sns.histplot(df["MonthlyCharges"], kde=True)

A histogram shows the distribution of a numerical variable.

It helps answer:

Where are most values?
Is the data symmetric?
Is it skewed?
Are there multiple groups?
Are there extreme values?

Example:

frequency
   ^
   |       ███
   |     ███████
   |   █████████
   | ███████████
   +-----------------> value

31. Why distributions matter

Suppose a feature looks like:

Most values: 0–100
A few values: 10,000

That may indicate:

genuine outliers
a different population
data-entry errors
a highly skewed distribution

A model may react differently depending on the algorithm.

This is one reason EDA happens before modeling.

32. Boxplot

You used:

sns.boxplot(y=df["Income"])

A boxplot summarizes a distribution.

Conceptually:

       outlier
          •
          |
      ┌───────┐
      │       │
------│  box  │------
      │       │
      └───────┘
          |
       outlier

The box represents the middle portion of the data.

The line inside the box is the median.

Points outside the whiskers may be treated as potential outliers.

Important:

An outlier is not automatically an error.

A high-income customer may be perfectly legitimate.

33. Boxplot: target vs category

You used:

sns.boxplot(
    x="Coverage",
    y="Customer Lifetime Value",
    data=df
)

This is a powerful graph.

It asks:

"How does the distribution of Customer Lifetime Value differ between categories?"

For example:

Coverage A → CLV distribution
Coverage B → CLV distribution
Coverage C → CLV distribution

Compare:

median
spread
outliers
overlap

This can help you identify potentially useful relationships.

34. Scatterplot

You used:

sns.scatterplot(
    x="Income",
    y="Customer Lifetime Value",
    data=df
)

A scatterplot examines the relationship between two numerical variables.

Each dot represents an observation.

It helps answer:

"As X changes, does Y appear to change?"

Possible patterns:

Positive relationship:
  •
    •
      •
        •

Negative relationship:
        •
      •
    •
  •

No obvious relationship:
 •    •
    •
  •      •
     •

The pattern does not have to be a straight line.

35. Pairplot

You used:

sns.pairplot(...)

A pairplot shows many numerical relationships at once.

It is useful for:

spotting correlations
identifying clusters
seeing distributions
finding obvious relationships
detecting possible outliers

The downside is that it becomes difficult to read when there are many columns.

So:

Pairplot is excellent for small-to-medium exploratory datasets, but not something you blindly run on every large dataset.

36. Correlation heatmap

You used:

sns.heatmap(
    df[["tenure", "MonthlyCharges", "TotalCharges"]].corr(),
    annot=True
)

Correlation measures how two numerical variables move together in a linear relationship.

A correlation close to:

+1 → strong positive linear relationship
 0 → little/no linear relationship
-1 → strong negative linear relationship

For example:

tenure ↑
TotalCharges ↑

could result in positive correlation.

But correlation has an important limitation:

Correlation does not prove causation.

Also, zero correlation does not necessarily mean "no relationship" because the relationship might be non-linear.

37. A key lesson from visualization

The graphs answer different questions:

Graph	Main question
Countplot	How many observations are in each category?
Countplot + hue	How does one category vary with another?
Histogram	What does a numerical distribution look like?
Boxplot	What are the median, spread and potential outliers?
Boxplot + category	How does a numerical distribution differ by category?
Scatterplot	How do two numerical variables relate?
Pairplot	What relationships exist among several numerical variables?
Heatmap	How strongly are numerical variables linearly correlated?

The goal is not:

"Create lots of graphs."

The goal is:

"Use the right graph to answer the right question."

38. Encoding: why does ML need it?

Many ML algorithms work with numbers.

But real-world data contains:

Male
Female

Yes
No

Month-to-month
One year
Two year

Fiber optic
DSL
No

The model needs these values represented numerically.

This is called encoding.

39. Label Encoding

You used:

from sklearn.preprocessing import LabelEncoder

and encoded categorical columns.

For a binary column:

No  → 0
Yes → 1

This can be reasonable for a binary variable.

For example:

Churn:
No  → 0
Yes → 1

That makes intuitive sense.

40. Why LabelEncoder can be dangerous for multi-category variables

Suppose:

Contract:
Month-to-month
One year
Two year

Label encoding might produce:

Month-to-month → 0
One year       → 1
Two year       → 2

A numerical model may interpret that as:

Two year > One year > Month-to-month

That ordering may not be appropriate for the algorithm.

The numbers are just codes.

They do not automatically mean that category 2 is "twice" category 1.

For nominal categories, one-hot encoding is usually safer.

41. One-hot encoding

You used:

pd.get_dummies(df, drop_first=True)

This transforms categories into separate binary columns.

For:

InternetService:
DSL
Fiber optic
No

we might create columns such as:

InternetService_Fiber optic
InternetService_No

with the remaining category represented by both values being 0.

This avoids inventing a fake numerical ordering.

42. Why `drop_first=True`?

If a categorical variable has several categories, using all one-hot columns can introduce redundancy in some models.

Dropping one category creates a reference category.

For example:

DSL
Fiber optic
No

could become:

Fiber optic
No

Then:

Fiber optic = 0
No = 0

means the reference category:

DSL

For beginner understanding, the key idea is:

One-hot encoding converts categories into machine-readable indicator variables without pretending the categories have numerical order.

43. Train/test split

You repeatedly used:

train_test_split(
    X,
    Y,
    test_size=0.3
)

This is one of the most important ML concepts.

Suppose we have 1,000 examples.

We might use:

700 → training
300 → testing

The training set is used to learn the model.

The test set is held back to evaluate how the trained model performs on unseen data.

Think of it like an exam:

Training data = practice questions
Test data     = unseen exam questions

44. Why not train and test on the same data?

Because the model could simply memorize the training examples.

Suppose:

Training score = 99%
Test score     = 65%

That is a warning sign.

The model learned the training data extremely well but does not generalize well.

This is called:

Overfitting

45. Underfitting

The opposite can happen.

If the model is too simple:

Training score = 60%
Test score     = 58%

It may not have learned enough from the data.

This is:

Underfitting

A useful mental picture:

Underfitting
Model too simple
       ↓
misses important patterns

Good fit
Learns useful patterns
       ↓
works on unseen data

Overfitting
Model learns noise/details
       ↓
great training performance
poor unseen performance

46. Why `random_state` matters

In one Telco notebook you used:

train_test_split(X, Y, test_size=0.3)

In the improved notebook you used:

random_state=42

A random state makes the split reproducible.

Without it, the random split can change between runs.

That means your score may change.

With:

random_state=42

you can reproduce the same split.

The number 42 is not magical.

Any fixed integer can serve this purpose.

47. Why `stratify=Y` is useful for classification

In your improved Telco notebook you used:

stratify=Y

This helps preserve the class distribution between training and testing data.

For example, if the full dataset contains:

73% No Churn
27% Churn

stratification aims to keep roughly the same proportion in both splits.

This is particularly useful when classes are imbalanced.

48. Logistic Regression

You used:

from sklearn.linear_model import LogisticRegression

model_lr = LogisticRegression()
model_lr.fit(X_train, Y_train)

Despite its name, Logistic Regression is commonly used for classification.

Its goal is to estimate the probability of belonging to a class.

For binary classification:

features
   ↓
Logistic Regression
   ↓
probability
   ↓
class

Example:

Probability of churn = 0.82

The model may classify that customer as:

Churn

depending on its decision threshold.

49. Why Logistic Regression is a good beginner model

It is useful because:

relatively simple
fast
often strong as a baseline
easier to interpret than many complex models
naturally suited to binary classification

A good ML habit is:

Start with a simple baseline before reaching for complex models.

50. Support Vector Machine / SVC

You used:

from sklearn.svm import SVC

model_lr = SVC()

The variable name model_lr is misleading here.

It is actually an SVC model.

SVC means:

Support Vector Classifier

The basic idea is to find a decision boundary that separates classes.

In simple 2D data:

Class A: ● ● ●

--------- decision boundary ---------

Class B: ▲ ▲ ▲

SVM tries to find a boundary with a useful margin between classes.

51. Why scaling matters for SVM

SVM is sensitive to feature scale.

Suppose one feature ranges from:

0–1

and another ranges from:

0–100,000

The larger-scale feature can dominate distance-related calculations.

That is why scaling is often important for:

SVM
KNN
Logistic Regression in many situations
neural networks

52. Decision Tree

You used:

DecisionTreeClassifier()

and later:

DecisionTreeRegressor()

A decision tree makes predictions using a sequence of questions.

Imagine:

Is tenure < 12?
       /       \
     Yes       No
     /           \
Is charge > X?   ...

Eventually the tree reaches a leaf containing a prediction.

This is attractive because it is easy to visualize conceptually.

53. Why trees can overfit

A tree can keep splitting the data until it creates extremely specific rules.

For example:

If feature A > 10
and feature B < 4
and feature C = 7
and feature D > 92
...

Eventually the tree may memorize training examples.

That is why you later tried:

max_depth=6
min_samples_split=10
min_samples_leaf=5

These are ways of controlling tree complexity.

54. Random Forest

You used:

RandomForestRegressor(
    n_estimators=200,
    max_depth=8,
    min_samples_leaf=5,
    random_state=42
)

A random forest combines many decision trees.

Instead of relying on one tree:

Tree 1 ─┐
Tree 2 ─┤
Tree 3 ─┤
Tree 4 ─┤ → combined prediction
...     ┘

The idea is that many different trees can produce a more robust prediction than a single tree.

This is an example of an ensemble method.

55. Bagging

You used:

BaggingRegressor(
    n_estimators=100,
    max_samples=0.7,
    max_features=0.7
)

Bagging means combining models trained on different samples of the data.

The overall idea:

Dataset
  ↓
Different samples
  ↓
Multiple models
  ↓
Combine predictions

The goal is often to reduce variance and make predictions more stable.

Random Forest can be viewed as a specialized tree-based ensemble that adds additional randomness in how trees are built.

56. AdaBoost

You used:

AdaBoostRegressor()

AdaBoost means:

Adaptive Boosting

Instead of training many independent models and simply averaging them, boosting builds models sequentially.

Conceptually:

Model 1
  ↓
find mistakes
  ↓
Model 2 focuses more on difficult cases
  ↓
Model 3 focuses further
  ↓
combine models

The key idea:

Later models try to improve on the weaknesses of earlier models.

57. K-Nearest Neighbors (KNN)

You used:

KNeighborsRegressor()

KNN predicts using nearby examples.

Imagine a new customer.

The algorithm asks:

"Which existing customers are most similar to this customer?"

Then it uses those neighbors to estimate the output.

For regression, the prediction is commonly based on the neighbors' target values.

58. Why KNN needs scaling

KNN relies on distance.

Imagine:

Income:   0–100,000
Age:      18–80

If raw values are used, income can dominate the distance calculation.

Scaling puts features onto comparable scales.

That is why you experimented with:

StandardScaler()

59. StandardScaler

You used:

scaler = StandardScaler()

X_train_scaled = scaler.fit_transform(X_train_log)
X_test_scaled = scaler.transform(X_test_log)

This is an important pattern.

Notice:

scaler.fit_transform(X_train)

but:

scaler.transform(X_test)

Why?

The scaler learns the training data's statistics.

Then the exact same transformation is applied to the test data.

You should not independently fit the scaler on the test set.

Otherwise information from the test set leaks into preprocessing.

This idea is called avoiding:

Data leakage

60. Data leakage

Data leakage happens when information that should not be available to the model during training accidentally influences the model.

Example:

Train preprocessing
     ↓
uses information from test data
     ↓
test is no longer truly unseen

That can make evaluation look better than real-world performance.

The correct principle is:

Fit preprocessing steps using training data, then apply them to validation/test/new data.

A production ML pipeline should ideally bundle preprocessing and modeling together.

61. Linear Regression

You used:

LinearRegression()

for Customer Lifetime Value.

Linear regression tries to model a relationship like:

Y = b0 + b1X1 + b2X2 + ...

For one feature:

Y = intercept + slope × X

The goal is to find coefficients that produce predictions close to the observed values.

62. Regression score: what does `.score()` mean?

You used:

model.score(X_train, Y_train)
model.score(X_test, Y_test)

For the scikit-learn regression models you used, .score() generally returns R², the coefficient of determination.

Very roughly:

R² = 1

means the model explains the observed variation perfectly on that dataset.

R² = 0

means the model is no better than a simple baseline based on predicting the mean target.

R² can also be negative on unseen data.

Important:

R² is not the same thing as accuracy.

For regression, do not say:

"My regression model has 90% accuracy"

just because R² is 0.90.

Instead say:

"The model achieved an R² of 0.90."

63. Classification accuracy

You used:

accuracy_score(Y_test, y_test_pred)

Accuracy is:

correct predictions
-------------------
total predictions

For example:

90 correct
100 total

gives:

90% accuracy

Accuracy is easy to understand.

But it can be misleading when classes are highly imbalanced.

64. Confusion matrix

Your later notes/workflow refer to concepts such as:

TP
TN
FP
FN

A confusion matrix organizes classification predictions.

                    Actual
                 No       Yes
Predicted No     TN       FN
Predicted Yes    FP       TP

Meaning:

True Positive

Model predicted positive and it was positive.

True Negative

Model predicted negative and it was negative.

False Positive

Model predicted positive but it was negative.

False Negative

Model predicted negative but it was positive.

This is much more informative than accuracy alone when the cost of errors differs.

65. Precision and Recall

For classification:

Precision

Of everything the model predicted as positive:

How many were actually positive?

Precision = TP / (TP + FP)

Recall

Of everything that was actually positive:

How many did the model find?

Recall = TP / (TP + FN)

The right metric depends on the problem.

For churn prediction, for example, missing a customer who is about to churn may be more important than contacting an extra customer who would have stayed.

66. Your Telco Customer Churn project

This is probably the clearest classification example in your notebooks.

The business question is:

Can we predict whether a telecom customer will churn?

The target is:

Y = df["Churn"]

Features are:

X = df.drop("Churn", axis=1)

The workflow becomes:

Customer data
     ↓
Clean data
     ↓
Explore data
     ↓
Encode categories
     ↓
Split train/test
     ↓
Train classifier
     ↓
Predict churn
     ↓
Evaluate

67. Telco: removing customerID

You did:

df = df.drop(columns=["customerID"])

This is reasonable because the ID is an identifier rather than a customer characteristic.

You also commented that it could be kept separately if you later wanted to map predictions back to customers.

That is a good practical idea.

In production:

customerID
   ↓
keep separately for business reporting

but:

customerID
   X
do not necessarily use as a model feature

68. Telco: fixing TotalCharges

You used:

df["TotalCharges"] = pd.to_numeric(
    df["TotalCharges"],
    errors="coerce"
)

errors="coerce" means invalid values are converted to NaN.

Then you used:

fillna(0)

in one notebook and median imputation in another.

These are two different decisions.

The important lesson is:

Missing-value handling should be based on the meaning of the data, not simply on whichever replacement is easiest.

For example, if missing TotalCharges occurs because a customer has zero tenure and has just joined, then filling with zero may have a meaningful business interpretation.

If missingness represents an unknown measurement, median imputation may be more appropriate.

69. Telco: the first modeling version

The earlier Telco notebook used:

LabelEncoder()

for many categorical columns.

This is okay as a learning exercise for understanding encoding, but it is not ideal for every categorical variable.

A better version in your later notebook used:

pd.get_dummies(
    df,
    columns=multi_cols,
    drop_first=True
)

That is a meaningful improvement.

70. Telco: your improved preprocessing pipeline

The later notebook did something closer to:

Raw data
   ↓
Drop customerID
   ↓
Convert TotalCharges
   ↓
Encode binary columns
   ↓
One-hot encode multi-category columns
   ↓
Separate X and Y
   ↓
Scale X
   ↓
Train Logistic Regression
   ↓
Create Gradio interface

This is much closer to a real ML application.

71. Gradio: turning a model into an application

You used:

import gradio as gr

and built a UI.

This is an important step because it changes the project from:

Notebook experiment

to:

User input
    ↓
Preprocessing
    ↓
Model
    ↓
Prediction
    ↓
Human-readable result

For example:

Gender: Female
Tenure: 12 months
Contract: Month-to-month
Monthly charges: $70
...
          ↓
    Predict Churn
          ↓
High Churn Risk
Probability: ...

This demonstrates an important ML engineering concept:

A model is useful only when it can be integrated into a workflow where people or systems can use its predictions.

72. Very important: preprocessing must match training

Your Gradio code contains a good lesson.

During training you:

encode the categorical values
create dummy columns
scale the features
train the model

During prediction you must do the same thing:

raw user input
    ↓
same encoding
    ↓
same columns
    ↓
same scaling
    ↓
model

If the training data has:

feature_1
feature_2
feature_3
feature_4

but the application sends:

feature_1
feature_3
feature_4

the model cannot interpret the input correctly.

Your use of:

input_df.reindex(
    columns=feature_columns,
    fill_value=0
)

is intended to make the input columns match the training feature structure.

That is an important practical idea.

73. A stronger production pattern: Pipeline

A cleaner production approach is often:

Pipeline([
    ("preprocessor", ...),
    ("model", LogisticRegression())
])

Then preprocessing and prediction are tied together.

This reduces the risk of accidentally applying different transformations during training and inference.

Your notebook demonstrates the concept manually, which is useful for learning.

74. Your Auto Insurance / Customer Lifetime Value project

The 13-Aug.ipynb notebook moves into regression.

The target is:

Y = df["Customer Lifetime Value"]

This changes the question from:

Will this customer churn?

to:

What numerical Customer Lifetime Value should we predict?

Therefore:

Classification → Churn
Regression     → Customer Lifetime Value

75. Insurance dataset: initial exploration

You loaded:

df = pd.read_csv("AutoInsurance.csv")

Then inspected:

df.info()
df.shape
df.describe()

You also removed:

Customer
Effective To Date

from the modeling DataFrame.

Again, the purpose is to separate identifiers / dates that were not being used in the current modeling approach.

However, an important ML lesson is:

Dates are not automatically useless.

A date may contain useful information such as:

month
day of week
season
year
time since an event

A stronger project could engineer useful date features instead of simply dropping every date column.

76. Insurance EDA

You explored:

sns.countplot(x="State", data=df)
sns.boxplot(y=df["Income"])
sns.boxplot(
    x="Coverage",
    y="Customer Lifetime Value",
    data=df
)
sns.scatterplot(
    x="Income",
    y="Customer Lifetime Value",
    data=df
)
df["Income"].hist()

This is a good progression:

Categorical distribution
        ↓
Numerical distribution
        ↓
Numerical vs categorical
        ↓
Numerical vs numerical

You are gradually asking:

"What might explain the target?"

77. Automatically finding numerical and categorical columns

You used:

numerical_cols = df.select_dtypes(
    include=["int64", "float64"]
).columns

categorical_cols = df.select_dtypes(
    include=["object"]
).columns

This is useful because instead of manually listing every column, Python can identify columns based on data type.

Then you looped through them to create graphs.

This is the beginning of writing reusable data-analysis code.

78. Automated boxplots

You created boxplots for all numerical columns.

This is useful because you can quickly inspect:

distributions
median
spread
potential outliers

However, remember that a large collection of graphs can become overwhelming.

The goal should eventually move from:

"Let's plot everything."

to:

"Let's plot the variables that help answer our question."

That is a sign of growing from beginner EDA toward professional analysis.

79. Binning numerical variables for boxplots

You used:

pd.cut(
    df[col],
    bins=5
)

This divides a numerical variable into five intervals.

Then you compare CLV across those ranges.

Conceptually:

Income range 1 → CLV distribution
Income range 2 → CLV distribution
Income range 3 → CLV distribution
...

This is useful for visualization because a boxplot expects categories on one axis.

But the number and boundaries of bins can affect the story you see.

So binning is mainly a visualization technique here, not necessarily something you should automatically use as a model feature.

80. Comparing many regression algorithms

One of the biggest learning steps in your final notebook was trying multiple regressors:

Linear Regression
KNN Regressor
SVR
Decision Tree Regressor
Bagging Regressor
AdaBoost Regressor
Random Forest Regressor

This is valuable because it teaches a core ML lesson:

Different algorithms make different assumptions and capture different types of patterns.

There is no universally best algorithm.

81. Linear Regression vs tree-based models

Linear Regression assumes a relationship that can be represented through a linear combination of features.

Tree-based methods can represent more complex non-linear relationships.

For example:

Linear model:

Y
│       /
│     /
│   /
│ /
└──────── X

A tree-based model can approximate more irregular patterns:

Y
│    ┌────
│    │
│ ───┘
│
└──────── X

This is why trying multiple model families can be useful.

82. Why model comparison matters

Suppose you obtain:

Model             Train R²    Test R²
Linear Regression   0.70       0.68
KNN                 0.90       0.62
Decision Tree       0.99       0.55
Random Forest       0.91       0.78

You should not automatically choose the model with the highest training score.

The more important question is:

Which model generalizes well to unseen data?

Here Random Forest would be more interesting than the tree with 0.99 training R².

83. Log transformation of Customer Lifetime Value

You later used:

Y_log = np.log1p(Y)

Why?

Because some target variables are heavily skewed.

Imagine:

Most CLV values: relatively small
A few CLV values: extremely large

The model may have difficulty fitting such a distribution.

A log transformation compresses large values.

Conceptually:

Original:
1
10
100
1000
10000

Log scale:
small differences between large values

This can make the target distribution easier for some models to learn.

84. Reversing the log transformation

After predicting:

predictions_log = model.predict(X)

you used:

predictions_actual = np.expm1(predictions_log)

This is important.

If:

Y_log = log(1 + Y)

then:

Y = exp(Y_log) - 1

np.expm1() performs:

exp(x) - 1

So:

log target
   ↓
model
   ↓
log prediction
   ↓
expm1
   ↓
original target scale

85. An important issue in the notebook: split consistency

In your log-transform section, you did:

Y_log = np.log1p(Y)

X_train_log, X_test_log, Y_train_log, Y_test_log = train_test_split(
    X,
    Y_log,
    test_size=0.3,
    random_state=42
)

This is okay as a new split, but it means your earlier train/test split and later train/test split are not necessarily the same.

For a clean experiment, define the split once and reuse it consistently.

For example:

X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.3,
    random_state=42
)

y_train_log = np.log1p(y_train)
y_test_log = np.log1p(y_test)

This makes comparisons easier to reason about.

86. Another important issue: predicting on the full dataset

You later did:

predictions_log = model_rf.predict(X)

and saved predictions for the full dataset.

That can be useful for producing a business-facing prediction file.

But these are not unbiased test predictions.

Why?

Because X includes training observations the model already saw.

So:

Predictions on X_test
→ useful for evaluating unseen data

Predictions on X
→ useful for generating predictions for all records,
   but not for measuring generalization

This distinction is very important.

87. Hyperparameters

Later you changed models from:

DecisionTreeRegressor()

to:

DecisionTreeRegressor(
    max_depth=6,
    min_samples_split=10,
    min_samples_leaf=5
)

These settings are called hyperparameters.

They are chosen by us before training.

The model learns its internal parameters from the training data.

Parameters

Learned by the algorithm.

Hyperparameters

Set by us.

This distinction is fundamental.

88. Understanding your tree hyperparameters

`max_depth`

Controls how deep the tree can grow.

Smaller:

simpler tree
less overfitting

Larger:

more complex tree
greater overfitting risk

`min_samples_split`

Minimum number of samples required before a node can be split.

Higher values make the tree more conservative.

`min_samples_leaf`

Minimum number of samples allowed in a leaf.

Higher values prevent extremely tiny leaves.

89. Random Forest hyperparameters

You used:

RandomForestRegressor(
    n_estimators=200,
    max_depth=8,
    min_samples_leaf=5,
    random_state=42
)

`n_estimators=200`

Build 200 trees.

More trees can make the ensemble more stable, though computation increases.

`max_depth=8`

Limits tree complexity.

`min_samples_leaf=5`

Prevents leaves from becoming too specific.

`random_state=42`

Makes the experiment reproducible.

90. Bagging hyperparameters

You used:

BaggingRegressor(
    n_estimators=100,
    max_samples=0.7,
    max_features=0.7,
    random_state=42
)

This means, conceptually:

build many estimators
each estimator sees a sample of the training rows
each estimator can use a subset of features
combine their predictions

Again, the goal is to create a robust ensemble rather than rely on one model.

91. Scaling experiment in your notebook

You used:

StandardScaler()

and compared KNN/SVR.

This is a very useful experiment because it demonstrates:

Preprocessing requirements depend on the algorithm.

Tree-based algorithms generally do not need feature scaling in the same way that distance- or margin-based methods do.

KNN and SVM are much more sensitive to feature scale.

92. A subtle issue in your scaling experiment

You created:

X_train_scaled
X_test_scaled

but your KNN code first trained on:

X_train_log

before later looping over scaled KNN.

That is useful as experimentation, but for a clean article you should present the comparison more systematically:

KNN without scaling
        vs
KNN with scaling

and explicitly explain why the result changed.

For SVR, you correctly used the scaled data in the shown section.

93. The complete mental model

After these five days, the entire workflow can be remembered as:

             REAL-WORLD QUESTION
                     ↓
             Define the target
                     ↓
              Collect data
                     ↓
              Understand data
                     ↓
          ┌──────────┴──────────┐
          ↓                     ↓
      Numerical             Categorical
          ↓                     ↓
      distributions        category counts
          └──────────┬──────────┘
                     ↓
                    EDA
                     ↓
              Clean the data
                     ↓
          Handle missing values
                     ↓
             Encode categories
                     ↓
          Select useful features
                     ↓
              Split train/test
                     ↓
          Scale when appropriate
                     ↓
              Train baseline
                     ↓
          Evaluate on test data
                     ↓
             Try other models
                     ↓
         Tune hyperparameters
                     ↓
       Select based on validation
                     ↓
             Final evaluation
                     ↓
              Make predictions
                     ↓
       Integrate into an application

94. What each notebook taught you

Day 1 — Auto MPG

Main lessons:

load a dataset
inspect rows and columns
understand shape and data types
inspect summary statistics
identify categorical/text columns
detect messy numerical values
visualize distributions
use countplots
use histograms
use pairplots
understand missing values
convert "?" to NaN
convert text to numerical data
use median imputation

The biggest lesson:

Before modeling, understand and clean your data.

Day 2 — Telco Customer Churn

Main lessons:

identify a classification target
remove an identifier
handle TotalCharges
visualize class balance
compare categories against churn
use boxplots and histograms
use pairplots
inspect correlations
encode categorical data
split into training/testing data
train Logistic Regression
train SVC
train Decision Tree
compare train/test performance

The biggest lesson:

A classification model predicts categories, but the quality of the prediction depends heavily on preprocessing and evaluation.

Day 3 — Improving the Telco project

Main lessons:

one-hot encode multi-category variables
use random_state
use stratify
scale features
train Logistic Regression
build a prediction interface with Gradio
reproduce preprocessing at inference time
output a human-readable prediction

The biggest lesson:

An ML model is only one part of an ML application.

Day 4/5 — Customer Lifetime Value

Main lessons:

formulate a regression problem
inspect numerical/categorical relationships
automate EDA
encode categories
split train/test
compare multiple regressors
understand R²
try target transformation
use log transformation
reverse the transformation
experiment with scaling
tune hyperparameters
use ensemble models
generate prediction files

The biggest lesson:

Different models capture different patterns, and model selection should be based on performance on unseen data—not training performance alone.

95. What I would change before publishing the project as "production-quality"

Your notebooks are excellent learning material because they show experimentation. For a public article, however, it is worth separating:

Learning experiments

from:

Recommended ML workflow

Here are the most important improvements.

Improvement 1 — Use clear train/test terminology

Prefer:

X_train, X_test, y_train, y_test

rather than mixing Y and y.

Python convention generally uses lowercase y.

Improvement 2 — Make experiments reproducible

Use:

random_state=42

consistently when appropriate.

Improvement 3 — Use stratification for classification

For churn:

train_test_split(
    X,
    y,
    test_size=0.3,
    random_state=42,
    stratify=y
)

Improvement 4 — Avoid LabelEncoder for nominal multi-class input features

Use one-hot encoding for categories such as:

PaymentMethod
Contract
InternetService
VehicleClass

Label encoding is much more naturally suited to targets or truly ordinal categories.

Improvement 5 — Fit preprocessing only on training data

For example:

Training data
    ↓
fit scaler
    ↓
transform training

Test data
    ↓
transform using existing scaler

Do not independently fit preprocessing on the test data.

Improvement 6 — Use proper evaluation metrics

For classification, consider:

Accuracy
Precision
Recall
F1
Confusion Matrix
ROC-AUC
PR-AUC

For regression:

R²
MAE
MSE
RMSE

Which metrics matter depends on the business problem.

Improvement 7 — Don't compare only training scores

Always look at:

Training performance
+
Validation/test performance

A very high training score with much lower test performance can indicate overfitting.

Improvement 8 — Keep a validation strategy for model selection

A single test set should ideally be kept for final evaluation.

During experimentation, use cross-validation or a validation set to choose models/hyperparameters.

Conceptually:

Training data
   ↓
Cross-validation / validation
   ↓
Choose model
   ↓
Final test set
   ↓
Final unbiased-ish estimate

96. A very important distinction: EDA vs ML

EDA asks:

"What does my data look like?"

Machine learning asks:

"Can I learn a useful relationship that lets me make predictions on new data?"

These overlap, but they are not the same.

For example:

sns.scatterplot(...)

does not build a model.

It helps you understand the data before deciding how to model it.

97. Another important distinction: correlation vs prediction

A strong correlation can be useful.

But:

high correlation

does not automatically mean:

excellent predictive model

Prediction depends on:

multiple features
data quality
model type
noise
generalization
preprocessing
evaluation

So don't judge an ML problem from one graph.

98. Another important distinction: model vs algorithm

It is common to hear:

"I trained a model."

The algorithm is the learning procedure.

For example:

Random Forest = algorithm/model family

After training:

model_rf.fit(X_train, y_train)

you have a fitted model containing learned information.

A useful mental model:

Algorithm
   +
Training data
   ↓
Fitted model
   ↓
Predictions

99. What "learning" actually means

This is probably the most important concept to understand.

When you run:

model.fit(X_train, y_train)

the model is not magically understanding the world.

It is optimizing internal parameters so that its predictions match the training examples according to the algorithm's objective.

For example, linear regression learns coefficients.

A tree learns split rules.

A neural network learns weights.

So:

Training means finding model parameters that make the model perform well according to a defined objective.

100. What happens during prediction?

Once training is complete:

prediction = model.predict(X_test)

the model does not learn again from the test examples.

It applies what it learned during training.

Conceptually:

Training:

X_train + y_train
        ↓
      model.fit()
        ↓
   learned model

Prediction:

X_test
   ↓
learned model
   ↓
prediction

101. The biggest beginner misconception to avoid

Do not think:

"I used Random Forest, so I did machine learning."

The real ML skill is being able to explain:

What problem are you solving?
What is the target?
What features are available?
What does the data look like?
What problems did you find?
How did you clean them?
How did you encode the data?
Why did you choose the algorithm?
How did you evaluate it?
Does it generalize?
What would you improve?

If you can explain these, you are learning ML—not just memorizing scikit-learn commands.

102. A simple explanation of your five-day journey

If someone asks:

"What did you actually learn in five days?"

A strong answer is:

I learned that machine learning is not just about selecting an algorithm. I started by understanding datasets using Pandas, cleaning missing and incorrectly formatted values, and using visualization to understand distributions and relationships. Then I learned the difference between classification and regression, converted categorical data into numerical representations, split data into training and testing sets, trained several models using scikit-learn, and compared their performance on unseen data. Finally, I experimented with scaling, transformations, ensemble models, hyperparameters, and even built a simple Gradio interface around a churn model.

That is a much stronger story than:

"I learned Logistic Regression, SVM and Random Forest."

103. Suggested article structure for dev.to

For your final public article, I recommend this structure:

Title

1. Why I started learning ML
2. What I thought ML was vs what I learned
3. AI vs ML
4. Types of ML
5. My five-day learning roadmap
6. Day 1 — Understanding and cleaning data
7. EDA and why graphs matter
8. Day 2 — Classification with customer churn
9. Encoding categorical data
10. Train/test split
11. Logistic Regression, SVM and Decision Trees
12. How I evaluated classification models
13. Day 3 — Turning the model into an application
14. Day 4/5 — Regression and Customer Lifetime Value
15. Linear Regression vs KNN vs SVR vs Trees vs Ensembles
16. Log transformation
17. Scaling
18. Hyperparameters
19. What I got wrong / what I would improve
20. What I learned
21. What I plan to learn next

This will make the article feel like a journey, rather than a textbook.

104. Recommended article narrative

The most interesting story is:

"I started with almost no ML knowledge."

        ↓

"I learned that data preparation matters."

        ↓

"I discovered that graphs help me ask questions
before training models."

        ↓

"I learned classification."

        ↓

"I learned regression."

        ↓

"I tried different algorithms and realized
there is no single best model."

        ↓

"I learned that a high training score
doesn't necessarily mean a good model."

        ↓

"I turned one model into a small application."

        ↓

"I now understand the basic ML workflow
and know what I need to learn next."

That is a much more authentic beginner-to-ML story.

105. What to learn next

Based on what your notebooks already cover, I would not recommend immediately jumping into deep learning.

First strengthen these foundations:

1. Statistics

Learn:

mean
median
variance
standard deviation
distributions
probability
correlation
conditional probability

2. ML evaluation

Learn:

confusion matrix
precision
recall
F1
ROC-AUC
MAE
RMSE
R²
cross-validation

3. Feature engineering

Learn:

handling dates
categorical features
transformations
interactions
outliers
missing values

4. Model selection

Learn:

cross-validation
hyperparameter tuning
GridSearchCV
RandomizedSearchCV
pipelines

5. ML fundamentals

Learn:

bias vs variance
underfitting
overfitting
regularization
data leakage
feature importance

Then move toward:

Classical ML
     ↓
Feature engineering
     ↓
Model selection
     ↓
Cross-validation
     ↓
ML pipelines
     ↓
Deployment
     ↓
Deep Learning
     ↓
Generative AI / LLMs

106. The one-page cheat sheet

Problem type

Predict a category → Classification
Predict a number   → Regression
Find hidden groups → Clustering
Learn through reward → Reinforcement Learning

Data

Rows       → observations
Columns    → variables/features
X          → input features
y          → target

First inspection

df.head()
df.shape
df.info()
df.describe()
df.sample()
df.isnull().sum()

Visualization

Countplot   → categories
Histogram   → numerical distribution
Boxplot     → spread/outliers
Scatterplot → two numerical variables
Pairplot    → many numerical relationships
Heatmap     → correlation matrix

Preparation

Missing values
Data types
Categorical encoding
Feature selection
Scaling when needed

Training

model.fit(X_train, y_train)

Prediction

model.predict(X_test)

Evaluation

Classification:

Accuracy
Precision
Recall
F1
Confusion Matrix
ROC-AUC

Regression:

R²
MAE
MSE
RMSE

Core warning signs

Training score >> Test score
        ↓
Possible overfitting

Test information used during preprocessing
        ↓
Possible data leakage

Identifier used as feature
        ↓
Possible meaningless pattern

Categorical values converted to arbitrary numbers
        ↓
Possible false ordering

107. Final takeaway

The biggest lesson from these five days is not a particular Python library or algorithm.

It is the workflow:

Understand the problem → understand the data → clean the data → explore the data → prepare features → train → evaluate → improve → deploy.

The algorithms are tools inside that workflow.

You have already touched many of the important building blocks:

Pandas
NumPy
Matplotlib
Seaborn
Scikit-learn
data cleaning
EDA
classification
regression
encoding
train/test split
scaling
model evaluation
hyperparameters
ensemble models
target transformation
Gradio

The next step is not to memorize more algorithms.

The next step is to become comfortable answering:

Why am I doing this step?

If you can answer that question for every important line in your notebook, you are moving from "following an ML tutorial" to actually understanding machine learning.

Appendix A — Notebook-by-notebook step map

`11_Aug_mpg.ipynb`

Notebook step	Why it was done	ML concept
Import NumPy/Pandas/Matplotlib/Seaborn	Load tools	Python ML stack
`read_csv()`	Load data	Data ingestion
`nunique()`	Understand uniqueness	Data exploration
`head()`	Inspect examples	EDA
`shape`	Understand dataset size	EDA
`describe()`	Summary statistics	EDA
`info()`	Types/non-null values	Data quality
`countplot()`	Inspect categorical distribution	Visualization
`hist()`	Inspect numerical distributions	Visualization
`isnull()`	Detect actual missing values	Data cleaning
`pairplot()`	Explore relationships	EDA
`sample()`	Inspect random examples	Data quality
Identify `car name`	Recognize identifier-like field	Feature selection
Replace `?`	Convert fake missing values	Data cleaning
`to_numeric()`	Make horsepower numerical	Data preparation
Median imputation	Fill missing horsepower	Missing-value handling
Train/test split	Separate learning/evaluation data	Generalization
Logistic Regression	Start classification experiment	Supervised learning

Important caveat

`12th-Aug-Telco-Customer-churn.ipynb`

Step	Purpose
Load Telco CSV	Get customer data
`head()`	Inspect records
`shape`	Understand size
`describe()`	Statistical overview
Drop `customerID`	Remove identifier from features
`info()`	Check data types
`sample()`	Inspect random rows
Countplot of Churn	Check class distribution
Countplot of gender	Explore category distribution
Histograms	Explore numerical distributions
Pairplot	Explore relationships
Convert TotalCharges	Fix numeric type
Fill missing TotalCharges	Handle missing data
LabelEncoder	Convert categories to numbers
Train/test split	Evaluate on unseen data
Logistic Regression	Classification model
SVC	Alternative classifier
Decision Tree	Alternative classifier
Gradio	Turn model into interactive app

Important caveat

The notebook uses LabelEncoder on many categorical input columns. This is useful for learning encoding, but one-hot encoding is generally safer for nominal multi-category features.

`12-Aug-2.ipynb`

This notebook is a more developed version of the Telco project.

Important improvements include:

dropping customerID
converting TotalCharges
one-hot encoding multi-category variables
using random_state=42
using stratify=Y
scaling features
training Logistic Regression
building a Gradio prediction interface
preserving training feature columns
applying the same preprocessing to new input

This is the notebook that most clearly demonstrates the transition from:

ML experiment

to:

small ML application

`13-Aug.ipynb`

Main steps:

Load AutoInsurance
       ↓
Inspect dataset
       ↓
Drop Customer / date columns
       ↓
EDA
       ↓
Identify numerical/categorical columns
       ↓
Generate many visualizations
       ↓
Encode categorical variables
       ↓
Set CLV as target
       ↓
Train/test split
       ↓
Try multiple regressors
       ↓
Try log target
       ↓
Try scaling
       ↓
Tune tree/ensemble hyperparameters
       ↓
Generate predictions
       ↓
Save prediction CSVs

This notebook demonstrates the transition from classification to regression and from a single baseline model to model comparison and tuning.

Appendix B — A cleaner ML template to remember

# 1. Load
df = pd.read_csv("data.csv")

# 2. Understand
df.head()
df.shape
df.info()
df.describe()

# 3. Clean
# handle missing values
# fix data types
# remove inappropriate identifiers

# 4. Explore
# histograms
# boxplots
# countplots
# scatterplots
# correlation

# 5. Prepare
X = df.drop("target", axis=1)
y = df["target"]

# 6. Split
X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.2,
    random_state=42
)

# 7. Preprocess
# encoding / scaling

# 8. Train
model.fit(X_train, y_train)

# 9. Predict
predictions = model.predict(X_test)

# 10. Evaluate
# choose metrics appropriate to the problem

# 11. Compare / improve
# try another model
# tune hyperparameters
# use cross-validation

# 12. Final model
# train using the selected approach

# 13. Predict new data

Closing thought for the article

Five days ago, an ML notebook could easily look like a collection of unfamiliar commands.

Now there is a structure behind those commands.

head() is not just a command — it is a way to understand the data.

isnull() is not just a command — it is part of data quality.

A histogram is not just a graph — it tells you about a distribution.

train_test_split() is not just boilerplate — it protects your evaluation from simply measuring memorization.

fit() is not magic — it is the learning stage.

predict() is the model applying what it learned.

And model comparison is not about finding the fanciest algorithm — it is about finding an approach that generalizes well to data the model has never seen.

That, more than anything else, is what I took away from my first five days of machine learning.

ML Foundations: A Complete Data Cleaning & ML Pipeline

yash-nigam — Tue, 25 Aug 2026 13:42:00 +0000

Fundamentals of Machine Learning

Using Google Colab and Python Libraries to Run ML Models

Find all the jupiter notebooks and datasets here: https://github.com/yash-nigam/AI-ML-Foundations

What is Artificial Intelligence?
What is Machine Learning?
- 2.1 Types of Machine Learning
- 2.2 Supervised Learning
- 2.3 Unsupervised Learning
- 2.4 Reinforcement Learning
How Python Helps Implement Machine Learning
- 3.1 The Libraries and What Each One Handles
Exploratory Data Analysis: Investigating the Dataset
Working with the Loan Prediction Dataset
- 5.1 Making the Dataset Ready for ML Models
- 5.2 What Should Be Cleaned Up
- 5.3 Cleanup Steps
  - 5.3.1 Remove Unique/Identifier Columns
  - 5.3.2 Handle Missing Values
  - 5.3.3 Standardize & Encode Categorical Values
  - 5.3.4 Outliers, Duplicates, Imbalance, Scaling, Split
  - 5.3.5 Replace Non-Standard Placeholder Values
Encoding
- 6.1 Why Does ML Need Encoding?
- 6.2 Label Encoding
Train/Test Split
- 7.1 Why Not Train and Test on the Same Data?
- 7.2 Underfitting
- 7.3 Why random_state Matters
Machine Learning Models
- 8.1 Confusion Matrix
Gradio: Turning a Model into an Application
What "Learning" Actually Means
- 10.1 What Happens During Prediction?
The Real ML Skill — Questions to Be Able to Answer

1. What is Artificial Intelligence?

Artificial Intelligence (AI) is the broad idea of building systems that can perform tasks that normally require some form of human intelligence, such as:

Recognizing an image
Generating text / writing an email
Generating an image
Recommending a movie
Detecting fraud
Predicting the price/value of something

A useful mental model for the hierarchy:

2. What is Machine Learning?

Machine learning is teaching a computer to find patterns in examples, instead of telling it exact rules to follow.

Example: The old way to predict if a customer will cancel their telecom subscription — you could try writing rules by hand:

IF contract = month-to-month
AND monthly charges = high
AND tenure = low
THEN churn = yes

The machine learning way: show the ML algorithm many past customers along with what actually happened to them:

Customer	Contract type	Monthly charges	Tenure	Did they churn?
A	Month-to-month	High	Low	Yes
B	Two year	Low	High	No
C	Month-to-month	Low	High	No
D	Month-to-month	High	Low	Yes

The algorithm studies this data and figures out the pattern on its own — no one told it "low tenure + high charges = risky."

Using it to predict for a new customer

Once trained, you can hand it a brand-new customer it has never seen:

New customer's info
        ↓
   Trained model
        ↓
  Predicted: churn or not

2.1 Types of Machine Learning

	Supervised Learning	Unsupervised Learning	Reinforcement Learning
Core idea	Learn from examples with known answers	Find hidden patterns with no answers given	Learn by trial and error, guided by rewards
Data needed	Inputs + labels	Inputs only, no labels	No fixed dataset — agent generates its own data
Task types	Classification, Regression	Clustering, dimensionality reduction, anomaly detection	Choosing actions to maximize long-term reward
Common algorithms	Logistic Regression, Decision Trees, Random Forest	K-Means, DBSCAN, PCA	Q-Learning, DQN, Policy Gradient
How it's evaluated	Accuracy, precision, recall, MAE, RMSE	No ground truth — silhouette score, human judgment	Total reward earned
Real-world examples	Churn prediction, spam filtering, price estimation	Customer segmentation, topic discovery	Game-playing agents, robotics, ad bidding

2.2 Supervised Learning

Supervised learning is the case where you already know the right answers for your training data.

The model makes a guess, checks it against the known answer, sees how wrong it was, and adjusts. Repeat this thousands of times and it gets good at guessing.

Input (X)  →  Known answer (Y)

X is everything you know about a customer. Y is what actually happened to them.

X = [tenure=2, contract=month-to-month, charges=95.5]   →   Y = churned
X = [tenure=48, contract=two-year, charges=45.2]        →   Y = stayed

The model's job is to learn the relationship between X and Y well enough that when a new X shows up with no Y attached, it can produce a sensible guess.

The key requirement: you need historical data where the outcome is already recorded. No answer key, no supervised learning.

The only question that splits supervised learning in two

Is the thing I'm predicting a category, or a number?

That's it. That single question decides whether you're doing classification or regression — and it changes your algorithms, your metrics, and how you evaluate success.

	Classification	Regression
Question it answers	Which category?	How much?
Type of answer	A fixed label (bucket)	A number on a scale
Examples	Spam or Not spam Churn or No churn Cat / Dog / Horse	House price: ₹87,45,000 Temperature: 31.4°C Fuel efficiency: 23.7 MPG
What the output means	A label, not a quantity — `1` isn't "more" than `0`	A real quantity — 6,000 truly is twice 3,000
Model output	A probability → converted to a label e.g. `0.91 → "Churn"`	A direct number e.g. `6,500`
Threshold involved?	Yes — usually 0.5, adjustable	No
Being "wrong"	Binary — right label or wrong label	A matter of degree — off by a little or a lot
Common metrics	Accuracy, Precision, Recall	MAE, RMSE, R²
Common algorithms	Logistic Regression, SVC, Decision Tree Classifier	Linear Regression, KNN Regressor, SVR, Random Forest Regressor
In your project	Telco churn (Yes/No)	Customer Lifetime Value (a number)

Quick test for your own problems — when you get a new dataset, look at the target column and ask:

Does averaging two values in this column produce something meaningful?

  Average of ₹5,000 and ₹7,000 = ₹6,000  ✓ meaningful  → regression
  Average of "spam" and "not spam"       ✗ meaningless → classification

2.3 Unsupervised Learning

In unsupervised learning, we do not have a known target. Instead, the algorithm tries to discover structure in the data.

Customer data
     ↓
Find groups
     ↓
Group 1: high-value customers
Group 2: price-sensitive customers
Group 3: new customers

Common examples:

K-Means clustering
Hierarchical clustering
PCA

2.4 Reinforcement Learning

Reinforcement learning is different. An agent interacts with an environment and receives rewards or penalties.

Agent
  ↓ action
Environment
  ↓
Reward / penalty
  ↓
Agent learns

Examples:

Game-playing agents
Robotics
Certain recommendation/control systems

This was not part of the five-day practical work.

3. How Python Helps Implement Machine Learning

Machine learning is mostly mathematics — matrix operations, optimization, statistics. In principle you could write all of it yourself, but in practice nobody does. Python has become the default language for ML because a small set of mature libraries already implement those pieces, tested and optimized, so you can focus on the problem rather than the arithmetic.

What makes Python work well here is that these libraries fit together as a pipeline, not as isolated tools. Each one covers one stage of the journey, and they hand data to each other in a common format — a Pandas DataFrame or a NumPy array — so nothing needs translating in between.

A typical project moves through the stack like this:

Load and clean the data        →  Pandas (with NumPy underneath)
Explore and visualize it       →  Matplotlib + Seaborn
Prepare features and split     →  Scikit-learn
Train and predict              →  Scikit-learn
Evaluate the results           →  Scikit-learn (+ Seaborn to plot them)

Notice that roughly the first half of that pipeline has nothing to do with modeling at all. In real projects, loading, cleaning, and understanding the data usually takes far more time than calling .fit(). The libraries reflect that reality: Pandas and Seaborn get used constantly, while the actual model training is often two lines.

The other thing Python gives you is a consistent interface. Once you learn scikit-learn's fit → predict pattern, it works the same way for logistic regression, random forests, and support vector machines alike. Swapping one algorithm for another is a one-line change, which makes it cheap to try several and compare.

3.1 The Libraries and What Each One Handles

Library	Import	Role in the Pipeline	Key Functions/Methods	Notes
NumPy	`import numpy as np`	The numerical foundation — fast array math that every other library is built on	`np.nan`, `np.log1p()`, `np.expm1()`	`np.nan` = missing value. `log1p(x)` = log(1+x), useful for skewed targets. `expm1(x)` reverses it — used to convert log predictions back to the original CLV scale. You rarely use NumPy directly; it works underneath Pandas and scikit-learn
Pandas	`import pandas as pd`	Load and clean the data — a spreadsheet Python can manipulate	`read_csv()`, column selection/removal, dtype conversion, missing-value detection/fill, X/Y selection, encoding, saving predictions	Where most of your time actually goes. Example: `df = pd.read_csv("Telco_Customer_Churn.csv")`
Matplotlib	`import matplotlib.pyplot as plt`	The plotting engine — controls figures, titles, axes, display	`plt.figure()`, `plt.title()`, `plt.show()`	Rarely used alone; it's the layer Seaborn sits on top of
Seaborn	`import seaborn as sns`	Explore and understand the data through statistical plots	`countplot`, `boxplot`, `histplot`, `pairplot`, `scatterplot`, `heatmap`	One line gives you a plot that would take many in raw Matplotlib. A graph is a tool for asking questions about the data, not decoration
Scikit-learn	`from sklearn... import ...`	Everything modeling-related: splitting, preprocessing, scaling, encoding, training, prediction, evaluation	`train_test_split()`, `StandardScaler()`, `model.fit(X_train, Y_train)`, `model.predict(X_test)`, metrics functions	The `fit → predict` pattern is identical across nearly every model, so trying a different algorithm is a one-line change

The short version: Pandas gets the data into shape, Seaborn helps you understand it, and scikit-learn learns from it — with NumPy doing the arithmetic underneath and Matplotlib drawing the pictures.

4. Exploratory Data Analysis: Investigating the Dataset

Command	What It Does & How to Use It
`df.head()`	Shows the first few rows of the dataset. Run this right after loading data to confirm it loaded correctly and get a quick look at the columns and values.
`df.shape`	Returns the number of rows and columns, e.g. `(398, 9)`. Use it to quickly gauge dataset size, which affects model choice and computation time.
`df.info()`	Lists each column's data type and non-null count. Use it to spot missing values and type problems — e.g. a numeric column showing as `object` usually means it contains hidden text like `"?"`.
`df.describe()` / `df.describe(include="all")`	Shows summary statistics (count, mean, min, max, etc.) for numeric columns, or all columns with `include="all"`. Use it to check the scale of each variable and spot possible outliers.
`df.sample(n)`	Shows `n` random rows instead of just the first few. Use it after `head()` to catch inconsistencies or unusual values that only appear later in the data.
`df.isnull().sum()`	Counts missing (`NaN`) values per column. Use it to identify which columns need cleaning — note it only catches true `NaN`s, not disguised placeholders like `"?"`, which need to be checked separately.

5. Working with the Loan Prediction Dataset

5.1 Making the Dataset Ready for ML Models

Raw data collected from forms, surveys, or real-world systems is almost never ready to feed straight into a machine learning model.

ML models cannot by themselves guess around these problems, and these gaps could reduce the accuracy of your results.

5.2 What Should Be Cleaned Up

NaN (missing values) — ML models cannot handle these by themselves; they'll throw errors or silently produce garbage results.
Unique-identifier columns — Unique IDs create noise and aren't helpful for learning genuine patterns in the data.
Text/categorical values — Numbers are expected by most ML models, not strings like "Yes"/"No".
Imbalanced or skewed data — If 70% of your data is one outcome, a lazy model can get 70% accuracy by always predicting that outcome, without actually learning anything useful.
Outliers and skewed numeric columns — A few extreme values can pull the mean far from what's "typical," making mean-based decisions misleading.

5.3 Cleanup Steps

5.3.1 Remove Unique/Identifier Columns

Columns like IDs (e.g. Loan_ID) don't carry predictive signal and should be dropped.

df = df.drop("Loan_ID", axis=1)

Removes the Loan_ID column. It's a unique identifier for each row (like a primary key) — it has no predictive value and would only confuse or overfit a model if left in.

5.3.2 Handle Missing Values

Either drop rows/columns with too many nulls, or fill them in:

Use median for numeric columns that are skewed (robust to outliers, e.g. income, loan amount).
Use mean for numeric columns that are roughly symmetric/normally distributed.
Use mode (most frequent value) for categorical columns, since averaging text categories isn't meaningful.

Using Median

median_loanamount = df["LoanAmount"].median()

Calculates the median of LoanAmount. Median is preferred over mean here because loan/income data is typically skewed by a few very high values, and median is more robust to outliers.

median_loanamount_term = df["Loan_Amount_Term"].median()

Same logic — computes the median loan term to use for filling its missing values later.

median_credit_history = df["Credit_History"].median()

Computes the median of Credit_History (a 0/1 flag). Even though it's binary, median is used instead of mean to avoid producing a non-integer/ambiguous fill value.

Use the calculated values to fill:

df["LoanAmount"] = df["LoanAmount"].replace(np.nan, median_loanamount)
df["Loan_Amount_Term"] = df["Loan_Amount_Term"].replace(np.nan, median_loanamount_term)
df["Credit_History"] = df["Credit_History"].replace(np.nan, median_credit_history)

Fills missing values with the calculated medians. This avoids dropping rows (losing data) while keeping the columns usable for modeling, since most ML algorithms can't handle NaN.

Using Mode

Calculates the mode (most frequent value) for gender, married, dependents, and self_employed. Mode is used instead of median/mean because these are categorical columns — you can't average "Male" and "Female".

mode_gender = df["Gender"].mode()[0]
mode_married = df["Married"].mode()[0]
mode_dependents = df["Dependents"].mode()[0]
mode_self_employed = df["Self_Employed"].mode()[0]

Replace the missing entries with the most common category, which is a reasonable, low-bias guess for a categorical field:

df["Gender"] = df["Gender"].replace(np.nan, mode_gender)
df["Married"] = df["Married"].replace(np.nan, mode_married)
df["Dependents"] = df["Dependents"].replace(np.nan, mode_dependents)
df["Self_Employed"] = df["Self_Employed"].replace(np.nan, mode_self_employed)

5.3.3 Standardize & Encode Categorical Values

Standardize categorical values — fix inconsistent labels (e.g. "male", "Male", "MALE" should all become one consistent value).
Encode categorical variables — convert text categories into numbers using techniques like Label Encoding or One-Hot Encoding, since most ML models only accept numeric input.

LabelEncoder()  # + loop over cat_cols

Converts all text/categorical columns (Loan_Status, Married, Gender, etc.) into numeric codes (e.g., "Yes"/"No" → 1/0). This step is required because scikit-learn models only accept numeric input, not strings.

5.3.4 Outliers, Duplicates, Imbalance, Scaling, Split

Check and handle outliers — identify extreme values (via boxplots, .describe(), or z-scores) and decide whether to cap, remove, or transform them.
Check for and remove duplicate rows — duplicate records can bias the model toward those repeated patterns.
Address class imbalance — if one outcome dominates the target column, apply techniques like upsampling, downsampling, or SMOTE so the model doesn't just learn to predict the majority class.
Feature scaling (when needed) — normalize or standardize numeric ranges for models sensitive to scale (like SVM, KNN, or Logistic Regression with regularization).
Split before further transformation — separate train/test sets early so cleaning decisions (like fill values from training data) don't leak information from the test set.

5.3.5 Replace Non-Standard Placeholder Values

df["horsepower"] = df["horsepower"].replace("?", np.nan)

This converts the placeholder "?" into a real missing value.

df["horsepower"] = pd.to_numeric(df["horsepower"])

This then converts the column into a proper numeric type so it can be used in calculations and models.

6. Encoding

6.1 Why Does ML Need Encoding?

Many ML algorithms work with numbers. But real-world data contains categories like:

Male
Female

Yes
No

Month-to-month
One year
Two year

Fiber optic
DSL
No

The model needs these values represented numerically. This is called encoding.

6.2 Label Encoding

from sklearn.preprocessing import LabelEncoder

For a binary column:

No  → 0
Yes → 1

This can be reasonable for a binary variable. For example:

Churn:
No  → 0
Yes → 1

That makes intuitive sense.

7. Train/Test Split

Repeatedly used:

train_test_split(X, Y, test_size=0.3)

This is one of the most important ML concepts.

Suppose we have 1,000 examples. We might use:

700 → training
300 → testing

The training set is used to learn the model. The test set is held back to evaluate how the trained model performs on unseen data.

Think of it like an exam:

Training data = practice questions
Test data     = unseen exam questions

7.1 Why Not Train and Test on the Same Data?

Because the model could simply memorize the training examples. Suppose:

Training score = 99%
Test score     = 65%

That is a warning sign. The model learned the training data extremely well but does not generalize well. This is called:

Overfitting

7.2 Underfitting

The opposite can happen. If the model is too simple:

Training score = 60%
Test score     = 58%

It may not have learned enough from the data. This is:

Underfitting

A useful mental picture:

Underfitting
Model too simple
       ↓
misses important patterns

Good fit
Learns useful patterns
       ↓
works on unseen data

Overfitting
Model learns noise/details
       ↓
great training performance
poor unseen performance

7.3 Why `random_state` Matters

In one Telco notebook:

train_test_split(X, Y, test_size=0.3)

In the improved notebook:

train_test_split(X, Y, test_size=0.3, random_state=42)

A random state makes the split reproducible. Without it, the random split can change between runs — meaning your score may change between runs too. With random_state=42, you can reproduce the same split every time. The number 42 is not magical; any fixed integer can serve this purpose.

8. Machine Learning Models

Model / Concept	What It Does & How to Use It
Logistic Regression	Despite the name, used for classification — estimates the probability of belonging to a class, then applies a threshold to assign a label. A simple, fast, interpretable baseline model — good starting point before trying complex models.
SVC (Support Vector Machine)	Finds a decision boundary that best separates classes, aiming for maximum margin between them. Sensitive to feature scale, so scale your data (e.g. with `StandardScaler`) before using it.
Decision Tree	Predicts by asking a sequence of yes/no questions, ending at a leaf with the prediction. Easy to visualize but prone to overfitting — control this with `max_depth`, `min_samples_split`, `min_samples_leaf`.
Random Forest	Combines many decision trees and averages their predictions for a more robust result. An ensemble method — generally more reliable than a single decision tree.
Bagging	Trains multiple models on different random samples of the data, then combines their predictions. Reduces variance and makes predictions more stable; Random Forest is a specialized version of this.
AdaBoost	Builds models sequentially, where each new model focuses on the mistakes of the previous one. A boosting method — improves weak learners step by step rather than averaging independent ones.
KNN (K-Nearest Neighbors)	Predicts by finding the most similar existing examples ("neighbors") and using their values. Relies on distance, so it needs feature scaling to avoid large-scale features dominating the result.
StandardScaler	Scales features to comparable ranges before training distance-sensitive models. Always `fit_transform` on training data, then only `transform` (not fit) on test data, to avoid data leakage.
Linear Regression	Models the relationship between features and a numeric target as a straight-line equation (`Y = b0 + b1X1 + ...`). Used for predicting continuous values, e.g. Customer Lifetime Value.
R² Score (`.score()`)	Measures how well the model explains variation in the target — 1 is perfect, 0 means no better than predicting the average. Report it as "R² of 0.90," not "90% accuracy" — they're not the same thing.
Accuracy Score	The proportion of correct predictions out of all predictions made, used for classification. Easy to understand, but misleading when classes are imbalanced (e.g. 95% "no churn" data).

8.1 Confusion Matrix

A confusion matrix organizes classification predictions.

                    Actual
                 No       Yes
Predicted No     TN       FN
Predicted Yes    FP       TP

Meaning:

True Positive — model predicted positive and it was positive.
True Negative — model predicted negative and it was negative.
False Positive — model predicted positive but it was negative.
False Negative — model predicted negative but it was positive.

This is much more informative than accuracy alone when the cost of errors differs.

9. Gradio: Turning a Model into an Application

import gradio as gr

Building a UI around your model is an important step because it changes the project from:

Notebook experiment

to:

User input
    ↓
Preprocessing
    ↓
Model
    ↓
Prediction
    ↓
Human-readable result

For example:

Gender: Female
Tenure: 12 months
Contract: Month-to-month
Monthly charges: $70
...
          ↓
    Predict Churn
          ↓
High Churn Risk
Probability: ...

This demonstrates an important ML engineering concept:

A model is useful only when it can be integrated into a workflow where people or systems can use its predictions.

10. What "Learning" Actually Means

When you run:

model.fit(X_train, y_train)

the model is not magically understanding the world. It is optimizing internal parameters so that its predictions match the training examples according to the algorithm's objective.

For example, linear regression learns coefficients. A tree learns split rules. A neural network learns weights.

So:

Training means finding model parameters that make the model perform well according to a defined objective.

10.1 What Happens During Prediction?

Once training is complete:

prediction = model.predict(X_test)

the model does not learn again from the test examples. It applies what it learned during training.

Conceptually:

Training:

X_train + y_train
        ↓
      model.fit()
        ↓
   learned model

Prediction:

X_test
   ↓
learned model
   ↓
prediction

11. The Real ML Skill — Questions to Be Able to Answer

What problem are you solving?
What is the target?
What features are available?
What does the data look like?
What problems did you find?
How did you clean them?
How did you encode the data?
Why did you choose the algorithm?
How did you evaluate it?
Does it generalize?
What would you improve?
Why was a specific algorithm chosen?

Create a free static website using Github Pages and Jekyll

yash-nigam — Fri, 03 Mar 2023 17:39:15 +0000

Why use GitHub Pages and Jekyll
Prerequisites
1: Dev Machine Setup
2: Use Chirpy Starter Template
3: Install dependencies
4: Modify config yaml and preview changes
5: Using GitHub Actions
6: Commit Changes to Repo
7: View the website
8: Personalize by Adding Avatar, favicon, sidebar link and create first post

Why use GitHub Pages and Jekyll ?

GitHub Pages allows us to host a static website(html, CSS, JavaScript) on GitHub repository for absolutely free which is ideal for personal websites.

A repository with the naming convention of {github-user-name}.github.io makes that repository serve a static website with subdomain as {github-user-name} and domain name as github.io

Jekyll (static site generator tool) is integrated with GitHub pages providing professional looking themes and many free templates for your website.

When using Jekyll we do not have to code any HTML, CSS or JavaScript. Instead we only edit the markdown files to achieve the desired results in the templates.

Prerequisites

Windows/Linux developer machine
GitHub Account

1: setup Ruby, Gem and Jekyll on developer machine

If dev machine is windows, use Windows subsystem for Linux - which installs a ubuntu environment which is available from PowerShell terminal.
- Open a PowerShell prompt as an Administrator and run: wsl --install
Install Ruby and other prerequisites:

yash@LearningPC:~/github$ sudo apt-get install ruby-full build-essential zlib1g-dev
[sudo] password for yash:
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
build-essential is already the newest version (12.9ubuntu3).
zlib1g-dev is already the newest version (1:1.2.11.dfsg-2ubuntu9.2).
The following additional packages will be installed:
  ri
The following NEW packages will be installed:
  ri ruby-full
0 upgraded, 2 newly installed, 0 to remove and 4 not upgraded.
Need to get 6788 B of archives.
After this operation, 38.9 kB of additional disk space will be used.
Do you want to continue? [Y/n] Y
Get:1 http://archive.ubuntu.com/ubuntu jammy/universe amd64 ri all 1:3.0~exp1 [4206 B]
Get:2 http://archive.ubuntu.com/ubuntu jammy/universe amd64 ruby-full all 1:3.0~exp1 [2582 B]
Fetched 6788 B in 1s (4970 B/s)
Selecting previously unselected package ri.
(Reading database ... 47201 files and directories currently installed.)
Preparing to unpack .../ri_1%3a3.0~exp1_all.deb ...
Unpacking ri (1:3.0~exp1) ...
Selecting previously unselected package ruby-full.
Preparing to unpack .../ruby-full_1%3a3.0~exp1_all.deb ...
Unpacking ruby-full (1:3.0~exp1) ...
Setting up ri (1:3.0~exp1) ...
Setting up ruby-full (1:3.0~exp1) ...

Add environment variables to your ~/.bashrc file

echo '# Install Ruby Gems to ~/gems' >> ~/.bashrc
echo 'export GEM_HOME="$HOME/gems"' >> ~/.bashrc
echo 'export PATH="$HOME/gems/bin:$PATH"' >> ~/.bashrc
source ~/.bashrc

Install Jekyll and Bundler

yash@LearningPC:~/github$ gem install jekyll bundler
Fetching jekyll-4.3.2.gem
Successfully installed jekyll-4.3.2
Parsing documentation for jekyll-4.3.2
Installing ri documentation for jekyll-4.3.2
Done installing documentation for jekyll after 3 seconds
Fetching bundler-2.4.7.gem
Successfully installed bundler-2.4.7
Parsing documentation for bundler-2.4.7
Installing ri documentation for bundler-2.4.7
Done installing documentation for bundler after 0 seconds
2 gems installed

2: Use chirpy-starter template to create new repo

Sign in to GitHub.com with your account, and go to chirpy-starter, click the button Use this template > Create a new repository

In the next window type the repository name to be the same as your GitHub user name appended with .github.io <GitHub User Name>.githubpages.io This makes the repo. act as a source of static website, and the repo. name becomes the domain name website.
Once this operation is complete all the contents of chirpy-starter will be present in our new site: https://github.com/yash-nigam/yash-nigam.github.io

3: Installing dependencies of chirpy-starter

This site requires some GEM's to be installed before it can be used, the list of can be found in the Gemfile: https://github.com/cotes2020/chirpy-starter/blob/main/Gemfile
On your local development machine, clone the new repo git clone https://github.com/yash-nigam/yash-nigam.github.io.git
go to the repo root directory and run bundle command

cd yash-nigam.github.io

yash@LearningPC:~/github/yash-nigam.github.io$ bundle
Fetching gem metadata from https://rubygems.org/...........
Resolving dependencies...
Using bundler 2.4.7
Using concurrent-ruby 1.2.2
Using rainbow 3.1.1
Using ffi 1.15.5
Using forwardable-extended 2.6.0
Using mercenary 0.4.0
Using parallel 1.22.1
Using rouge 4.1.0
Using eventmachine 1.2.7
Using unicode-display_width 2.4.2
Using racc 1.6.2
Using colorator 1.1.0
Using i18n 1.12.0
Using ethon 0.16.0
Using yell 2.2.2
Using pathutil 0.16.2
Using liquid 4.0.4
Using public_suffix 5.0.1
Using typhoeus 1.4.0
Using addressable 2.8.1
Using webrick 1.8.1
Using jekyll-paginate 1.1.0
Using http_parser.rb 0.8.0
Using rb-inotify 0.10.1
Using rb-fsevent 0.11.2
Using rexml 3.2.5
Using terminal-table 3.0.2
Using nokogiri 1.14.2 (x86_64-linux)
Using safe_yaml 1.0.5
Using google-protobuf 3.22.0 (x86_64-linux)
Using em-websocket 0.5.3
Using listen 3.8.0
Using sass-embedded 1.58.3 (x86_64-linux-gnu)
Using html-proofer 3.19.4
Using kramdown 2.4.0
Using jekyll-watch 2.2.1
Using jekyll-sass-converter 3.0.0
Using kramdown-parser-gfm 1.1.0
Using jekyll 4.3.2
Using jekyll-archives 2.2.1
Using jekyll-seo-tag 2.8.0
Using jekyll-redirect-from 0.16.0
Using jekyll-sitemap 1.4.0
Fetching jekyll-theme-chirpy 5.5.2
Installing jekyll-theme-chirpy 5.5.2
Bundle complete! 6 Gemfile dependencies, 44 gems now installed.
Use `bundle info [gemname]` to see where a bundled gem is installed.
1 installed gem you directly depend on is looking for funding.
  Run `bundle fund` for details

4: Modify _config.yml and Preview the changes locally

Open _config.yml and modify the title and url

title: Yash Nigam

url: 'https://yash-nigam.github.io'

Preview the site contents locally:

yash@LearningPC:~/github/yash-nigam.github.io$ bundle exec jekyll s
Configuration file: /home/yash/github/yash-nigam.github.io/_config.yml
            Source: /home/yash/github/yash-nigam.github.io
       Destination: /home/yash/github/yash-nigam.github.io/_site
 Incremental build: disabled. Enable with --incremental
      Generating...
                    done in 1.634 seconds.
 Auto-regeneration: enabled for '/home/yash/github/yash-nigam.github.io'
    Server address: http://127.0.0.1:4000/

The local service will be published at http://127.0.0.1:4000.

5: Configure to Deploy the Website using GitHub Actions

Browse to your repository on GitHub, Select Settings > Pages. Then in the Source section (under Build and deployment), select GitHub Actions from the dropdown menu.

6: Commit and push changes to repo

Check git status

yash@LearningPC:~/github/yash-nigam.github.io$ git status
On branch main
Your branch is up to date with 'origin/main'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        modified:   _config.yml

no changes added to commit (use "git add" and/or "git commit -a")

add, Commit and push the _config.yaml file to repo

yash@LearningPC:~/github/yash-nigam.github.io$ git add _config.yml
yash@LearningPC:~/github/yash-nigam.github.io$ git commit -m "updated"
[main c30df68] updated
 1 file changed, 2 insertions(+), 2 deletions(-)
yash@LearningPC:~/github/yash-nigam.github.io$ git push
Username for 'https://github.com': yash-nigam
Password for 'https://yash-nigam@github.com':
Enumerating objects: 5, done.
Counting objects: 100% (5/5), done.
Delta compression using up to 12 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 334 bytes | 334.00 KiB/s, done.
Total 3 (delta 2), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (2/2), completed with 2 local objects.
To https://github.com/yash-nigam/yash-nigam.github.io.git
   aae12d1..c30df68  main -> main

7: Check the Actions Workflow and view the final website

Once the changes are pushed to the repo, a build and deploy workflow should be automatically triggered.

View the final website

8: Add Avatar, favicon, a new post and an sidebar tab.

Add Avatar and favicon images

Create a directory named img under assets in the root of repo yash-nigam.github.io/assets and favicons under img.
Follow the steps to generate and add faviconshttps://chirpy.cotes.page/posts/customize-the-favicon/

yash@LearningPC:~/github/yash-nigam.github.io/assets$ ls -ltr img/
total 96
drwxr-xr-x 2 yash yash  4096 Mar  3 23:28 favicons
-rw-r--r-- 1 yash yash    90 Mar  4  2023 favicon.ico:Zone.Identifier
-rw-r--r-- 1 yash yash 27019 Mar  4  2023 yn_small.jpg
-rw-r--r-- 1 yash yash 53754 Mar  4  2023 sidebar_bg.jpg
yash@LearningPC:~/github/yash-nigam.github.io$ ls -ltr assets/img/favicons/
total 168
-rw-r--r-- 1 yash yash 15531 Mar  4  2023 android-chrome-192x192.png
-rw-r--r-- 1 yash yash  8930 Mar  4  2023 mstile-150x150.png
-rw-r--r-- 1 yash yash   923 Mar  4  2023 favicon-16x16.png
-rw-r--r-- 1 yash yash   608 Mar  4  2023 safari-pinned-tab.svg
-rw-r--r-- 1 yash yash 15086 Mar  4  2023 favicon.ico
-rw-r--r-- 1 yash yash 66336 Mar  4  2023 android-chrome-512x512.png
-rw-r--r-- 1 yash yash 13736 Mar  4  2023 apple-touch-icon.png
-rw-r--r-- 1 yash yash  1270 Mar  4  2023 favicon-32x32.png

Also put your personal photo to be used as avatar in the img folder.
In the _config.yml file make following changes

avatar: /assets/img/yn_small.jpg

github:
  username: yash-nigam

Under the _posts folder create a file as follows:

yash@LearningPC:~/github/yash-nigam.github.io$ ls -ltr _posts/
total 4
-rw-r--r-- 1 yash yash 446 Mar  4 00:14 2023-03-04-FirstPage.md
yash@LearningPC:~/github/yash-nigam.github.io$ cat _posts/2023-03-04-FirstPage.md
---
title: "First Post"
date: 2023-03-03 12:00:00 +0530
categories: [TOP_CATEGORIE, SUB_CATEGORIE]
tags: [TAG]    , TAG names should always be lowercase
---

Under the _tabs folder create another markdown file of your choice.

yash@LearningPC:~/github/yash-nigam.github.io$ ls -ltr _tabs/
total 20
-rw-r--r-- 1 yash yash  47 Mar  3 15:00 tags.md
-rw-r--r-- 1 yash yash  56 Mar  3 15:00 categories.md
-rw-r--r-- 1 yash yash  55 Mar  3 15:00 archives.md
-rw-r--r-- 1 yash yash 194 Mar  3 15:00 about.md
-rw-r--r-- 1 yash yash 284 Mar  4 01:25 devto.md
yash@LearningPC:~/github/yash-nigam.github.io$ cat _tabs/devto.md
---
layout: page
title: dev.to Blog Posts
icon: fas fa-flask
order: 1
---
-----------
> <a href="https://dev.to/yashnigam/create-a-free-static-website-using-github-pages-and-jekyll-41a9" target="_blank">create a personal website using Github Pages with the Jekyll Framework</a>

Verify the site locally and view the changes in browser

bundle exec jekyll s

Commit and push the changes

yash@LearningPC:~/github/yash-nigam.github.io$ git status
On branch main
Your branch is up to date with 'origin/main'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        modified:   _config.yml

Untracked files:
  (use "git add <file>..." to include in what will be committed)
        _posts/2023-03-04-FirstPage.md
        _tabs/devto.md
        assets/css/
        assets/img/

no changes added to commit (use "git add" and/or "git commit -a")

Ignore any Zone.Identifier files as they are autogenerated

yash@LearningPC:~/github/yash-nigam.github.io$ git add --all
warning: CRLF will be replaced by LF in _tabs/devto.md.
The file will have its original line endings in your working directory

yash@LearningPC:~/github/yash-nigam.github.io$ git commit -m "Added changes"
[main 6eded15] Added changes
 24 files changed, 64 insertions(+), 8 deletions(-)
 create mode 100644 _posts/2023-03-04-FirstPage.md
 create mode 100644 _tabs/devto.md
 create mode 100644 assets/css/style.scss
 create mode 100644 assets/img/favicons/android-chrome-192x192.png
 create mode 100644 assets/img/favicons/android-chrome-512x512.png
 create mode 100644 assets/img/favicons/apple-touch-icon.png
 create mode 100644 assets/img/favicons/favicon-16x16.png
 create mode 100644 assets/img/favicons/favicon-32x32.png
 create mode 100644 assets/img/favicons/favicon.ico
 create mode 100644 assets/img/favicons/mstile-150x150.png
 create mode 100644 assets/img/favicons/safari-pinned-tab.svg
 create mode 100644 assets/img/sidebar_bg.jpg
 create mode 100644 assets/img/yn_small.jpg

yash@LearningPC:~/github/yash-nigam.github.io$ git push
remote: Resolving deltas: 100% (3/3), completed with 3 local objects.
To https://github.com/yash-nigam/yash-nigam.github.io.git
   c30df68..6eded15  main -> main

Once changes are pushed, make sure GitHub build and deploy action is successful

https://github.com/yash-nigam/yash-nigam.github.io/actions

Added changes
Build and Deploy #2: Commit 6eded15 pushed by yash-nigam

Now in the actual deployed site we can see that favicon, avatar, side bar tab and a new blog post is created successfully.

DEV Community: yash-nigam

LLM Concepts Deep Dive - Conceptual Mastery for Developers | Udemy

LLM Concepts Deep Dive

Table of Contents

What is a Model

1.1 What is a Language Model?

1.2 How Does a Language Model Learn?

1.3 What Does "Predict the Likelihood of Next Set of Words" Mean?

1.4 Language Model vs. Generative AI

1.5 Modern LLMs

Two Types of Language Modeling Tasks

2.1 Autoencoding

2.2 Autoregressive

LLM Training Methodology

3.1 Learn/Model Language

3.2 Pre-training

3.3 Weights

3.4 Pre-training Alone Is Not Enough

3.5 Fine-tuning

Instruction Tuning

Fine-tuning

Tokens and Embeddings

4.1 How LLMs Process Tokens

4.2 Visualizing Tokenization

4.3 Why Isn't One Word Always One Token?

4.4 The Solution

4.5 Token Decoding

4.6 What Numbers Are Assigned to Tokens

4.7 How the Model Learns

Feature Matrix

5.1 Feature Matrix for Language

Embedding

6.1 Understanding Embeddings Using Dimensions

6.2 Embedding Math

Tokens and Their Values

7.1 Text Similarity

Transformer Architecture

Context Length in LLMs

9.1 LLM State

What is RAG

10.1 Vectors and Chunks

RAG Pipeline

Vector Databases

12.1 How It Works — Ingestion

12.2 How It Works — Indexing

12.3 How It Works — Querying

12.4 Some Vector DBs

Claude Certified Developer - Foundations certification Overview

Claude Certified Developer - Foundations Prep Course

Module 1: MSO Foundations

1.1 How LLMs Behave

1.2 Models & Reasoning

1.3 Prompting Modes

1.4 Technical Substrate

Module 2: Production-Grade Prompting, Agents & Tool Use

2.1 Prompting Craft

2.2 Extended Thinking

2.3 Tool-Use and Schema Design

2.4 Streaming Responses

2.5 Context Engineering

2.6 Agent Construction

2.7 Agent Memory

2.8 Cumulative Debug Task

2.9 Multimodal and Batch Ingestion

Module 3: Claude Code, MCP & Integration

3.1 Module Orientation & Learning Objectives

3.2 The Explore → Plan → Code Loop

3.3 Permission Modes

3.4 Settings Configuration Hierarchy

3.5 Placing the Human Review Gate

3.6 Durable Project Context: CLAUDE.md

3.7 Rules Instruction Files

3.8 Hooks

3.9 Subagents

3.10 Packaging Workflows: Skills

3.11 Custom Commands & Plugin Packaging

3.12 Building an MCP Server

3.13 MCP Transport

3.14 Prompt Caching (Applied to MCP/Tool Definitions)

3.15 Retrieval-Augmented Generation (RAG) — Two Forms

`.claude` folder and settings

`/init`

Example: Generating `validateCreditCard` and Its Test Case from a Prompt

Example: custom command — `/standup`

Example: custom command — `/raise-pr`

Example: skill — `deploy`

Example: skill — `write-tests`

Example: custom agent — `security-analyst`

Example: `PreToolUse` hook on Bash