DEV Community: yash-nigam

Everything I Learned From My Claude Code Training

yash-nigam — Mon, 31 Aug 2026 17:28:38 +0000

Everything I Learned From My Claude Code Training

I recently went through a hands-on training on Claude Code and the broader Claude ecosystem. Below are my complete notes, organized by topic, with screenshots from the session included at each relevant step.

The Claude Ecosystem
- AI-Code-Design-Cowork-Security-Chrome-Office
The Agentic Loop (Query Engine)
- Architecture
Models, Effort, and Slash Commands
- Models-Commands
Session Management
- Export-Fork-Resume-Compact
Project Configuration
- ClaudeFolder-Permissions-Init-ClaudeMd
Auto Mode vs. Plan Mode
- Auto-Plan
Commands vs. Skills vs. Agents
- Standup-Metaprompting-PR-Invocation
Code Review via Subagents
- Security-Performance-Coverage-Quality-Usage-Deploy-Setup-Execution-Tests-Stack-Creation-Example-Focus-OWASP-Memory-Output-Background
Multi-Agent Harness Concepts
- SecondPass-Comparison
Hooks
- Flow-PreToolUse-Types-Blocking-Slack-Sharing
MCP (Model Context Protocol)
- Concept-Rationale-Types
End-to-End Automation
- End-to-end automation of security review → issue creation → issue fix → PR creation → PR fix, all through agents.
Agent Teams
- Multiple agents (security, performance, test) talk to each other and take actions autonomously — filing issues, creating PRs, merging PRs, and closing issues, all automated.
Key Takeaways
- Loop-Config-Commands-Hooks-Sessions-Teams

1. The Claude Ecosystem

Claude Code is one tool within a larger ecosystem, not a standalone product:

Claude AI — the conversational assistant
Claude Code — the agentic coding CLI (the focus of this article), which has:
- An agentic loop that loops until the task is completed
- A tool system: git, bash, read, write files
- A permission pipeline, configurable per tool (Allow / Deny / Ask)
- A context manager: compaction and pruning of context within a session
- Session persistence: work can be paused and resumed later, with context and memory maintained
Claude Design — design tooling
Claude Co-work — collaboration tooling
Claude Security — security-focused tooling
Claude Chrome plugin — browser automation
Claude Office plugin — office document integration

In the backend, Claude Code talks to the LLM hosted by Anthropic via the /v1/messages API.

Diagram of the agentic loop: gather context, take action, verify results, repeat until done.

Tools like Cursor, Windsurf, and Perplexity act as integrators — giving access to multiple LLMs. Claude, by contrast, offers an entire vertically integrated ecosystem.

2. The Agentic Loop (Query Engine)

This is the core engine behind Claude Code. Given a request — e.g. "Create a unit test case for my function" — the loop works like this:

Gather context — goes to the file, reads the imported files also, reads the function in the file (which needs to be mocked).
Send context to the LLM — the LLM decides, based on the context, what unit test cases must be created after it has all the info, and tells Claude Code what actions to take (e.g. "create new test case").
Use tools — Claude Code creates or modifies the file using its tool system.
Verify — the agent checks whether the job is done or not (e.g. runs the test to confirm it passes).

Additional properties:

The user can modify the context in between the loop or after it.
The loop runs continuously until the task is not done.
Everything depends on the LLM being used — so it's worth knowing which LLM is best suited for the task.

Agentic Coding Tools Architecture

Three-layer architecture diagram showing coding agents (Claude Code, Copilot, Codex, Cursor, etc.) between the chat interface and various LLM hosting backends.

Claude Code's runtime architecture, from terminal command through the CLI parser and QueryEngine agentic loop to tool calls and API routing.

Claude Code always connects to the Anthropic base URL.
It can also connect to Ollama and open-source models such as Qwen or Llama 3.2 — enabling a fully local setup. This is currently the only fully local option available.

3. Models, Effort, and Slash Commands

Models, effort levels, and sessions are the main levers for controlling how Claude Code works.

Models

Four main models, chosen based on the type of task and usage:

Fable — for complex tasks: architecture, design, security, coding
Opus
Sonnet
Haiku — for simpler tasks

Qwen-local is the local-model equivalent of Haiku.
opusplan = Opus for planning + Sonnet for doing the task.

Guidance: try a smaller model at higher effort first — otherwise switch to a new/bigger model as required. Use Fable only if Opus cannot handle the task.

Terminal status line showing the active model, session name, context usage, token count, and running cost.

Screen clippings taken: 29-08-2026 02:21 and 29-08-2026 02:14

Slash commands for managing everything

/model — switch between different models
/effort — decides the effort to put in: high → xhigh → max
/statusline — shown in the Claude prompt, displays total tokens used; helps decide when to compact the session

4. Session Management

Before starting with Claude Code, it's worth understanding /sessions — session management.

We can resume from wherever we stopped using a session ID.
Start a new session: claude -n "session_name"
Resume a previous session: claude --resume, claude --continue
/status — check session status

Practical guidance

We can create 3 different sessions for 3 different issues, potentially working on 3 different models.
/export explore-current-project.md — exports the conversation to a markdown file, e.g.:

   ⎿  Conversation exported to: H:\Claude-Code-Projects-17-19-Aug\claude_showcase_17_aug_26\explore-current-project.md

/fork — forks the current session so you can go to a new forked session without modifying the original session.
If 80% of context is used, we can compact it.
/resume — switch between sessions.
/compact — summarizes the session data; some data could be removed in the process.
If old context is not found in the current session, tokens would need to be spent again to recreate it — and wrong information could get compacted.
There is a usage limit at the account level; every session uses a part of that limit.
We should create multiple sessions — e.g. 3 different sessions for 3 different bugs. Information for the first bug may not be needed for the second bug, and sessions help Claude read old information without wasting tokens to recreate it.

Claude Code's /status panel showing version, session details, login/account info, and connected MCP servers.

5. Project Configuration

README.md

The most important file for reading all the information about a project.

Before telling the tool what to do, plan first — create the architecture first.
Decide on a design template.

CLAUDE.md — Rules

Used to control the behaviour of Claude:

Which JavaScript library to use.
Preventing Claude Code from reading sensitive files so it does not use them.
How to give Claude permission to access your code — what it can read and what it cannot read.

`.claude` folder and settings

First, create a .claude folder.
Create settings.json — accessible to the whole team, pushed to the repo.
Create settings.local.json — will not be pushed to the repo, and will override settings.json.

Here you can decide which model to use:

{
  "model": "opusplan"
}

Permissions:

{
  "permissions": {
    "allow": [
      "Bash(npm run *)",
      "Bash(git status)",
      "Bash(git diff)",
      "Bash(git add *)",
      "Bash(git commit *)",
      "Read(**)",
      "Write(src/**)",
      "Write(tests/**)"
    ],
    "ask": [
      "Bash(git push:*)",
      "Bash(npm install *)",
      "Write(package.json)"
    ],
    "deny": [
      "Read(./.env*)",
      "Read(./secrets/**)",
      "Read(./**/credentials*)",
      "Bash(rm -rf:*)",
      "Bash(curl:*)",
      "Bash(wget:*)"
    ]
  }
}

Claude declining to read a blocked .env.example file per project settings and offering alternative ways to proceed.

`/init`

Initializes a CLAUDE.md file with codebase documentation after going through the project at the root level.

Add conditions for Claude to always follow — should not go above 150 to 200 lines max; this is very specific to the root of the project.
We can have a CLAUDE.md file for every sub-folder.
If there is no CLAUDE.md file, don't rely on Claude having project-specific context — use CLAUDE.md for every project.

Example: the actual CLAUDE.md used in this training repo

## Coding Standards
- Always use async/await, never raw Promises
- All functions must have JSDoc comments
- No console.log in production code — use the logger utility
- Every new function must have at least one unit test
- Pure validation functions use a single return statement with a composed expression
- Never use multiple early-return guard clauses in validator functions

## Repository conventions
- CommonJS (`require`/`module.exports`) throughout, not ESM.
- ESLint enforces: `eqeqeq` (always `===`), `curly`, `no-var`/`prefer-const`,
  `no-return-await`, `require-await` (no `async` functions without an `await`),
  unused-arg exception for `_`-prefixed names.
- `.claude/settings.json` denies reads of `.env*`, `secrets/**`, and
  `credentials*` — don't try to work around this to inspect real secrets.

## What Claude Must Never Do
- Never modify .env or .env.\* files
- Never push directly to main branch
- Never remove existing tests
- Never install packages without confirming with the developer

## PR and Git Standards
- Commit messages follow Conventional Commits: feat:, fix:, docs:, test:
- PR descriptions must include: what changed, why it changed, how to test

## Tech Stack
- Runtime: Node.js 20
- Framework: Express.js
- Database: PostgreSQL with Prisma ORM
- Testing: Jest
- Language: ECMAScript 2022 for all new code only

## Architecture
Request flow: `src/index.js` wires Express with `requestLogger` middleware
globally, mounts all auth endpoints under `/api/auth` from `src/api/routes.js`,
and registers `errorHandler` last as the global error middleware.

Layering:
- `src/api/routes.js` — route handlers only; validates input shape, calls into
  `authService`, maps thrown errors to HTTP status codes (e.g. `'Invalid
  credentials'` → 401). It does NOT talk to `tokenHelper.js` directly.
- `src/api/middleware.js` — cross-cutting concerns: `requestLogger`,
  `authenticate` (verifies Bearer token via `tokenHelper.verifyToken`, attaches
  `req.user`), `validateBody` (checks required fields present), `errorHandler`
  (catch-all, returns generic 500).
- `src/auth/authService.js` — business logic: `loginUser`, `refreshToken`,
  `revokeToken`, token generation. Owns the in-memory `refreshTokenStore` (a
  `Map`, standing in for a database — resets on every restart, not shared
  across processes).
- `src/auth/tokenHelper.js` — low-level JWT primitives: `verifyToken`
  (signature + expiry check, throws typed errors), `decodeToken` (no
  verification, for reading claims only), `extractBearerToken`, `getTokenTTL`.
  `authService.js` and `middleware.js` both depend on this; it depends on
  nothing else in the app.
- `src/utils/validators.js` — pure, stateless input validators (email,
  password, UUID, sanitization). No I/O.
- `src/utils/logger.js` — minimal structured JSON logger (stdout/stderr),
  level-gated by `LOG_LEVEL` env var. Not a real observability stack.

There's no real user database: `routes.js` builds a `mockUserRecord` inline
for login, and `authService.getUserById` returns a hardcoded stub — expect
these to be replaced with real persistence rather than extended in place.

JWT config (`JWT_SECRET`, `JWT_EXPIRES_IN`, `REFRESH_EXPIRES_IN`) is read from
env vars in both `authService.js` and `tokenHelper.js` independently, each
with its own fallback default — keep them in sync if changing defaults.

### Request flow (login example)
`POST /api/auth/login` → `validateBody(['email','password'])` → route handler
validates format via `validators.js` → `authService.loginUser()` checks bcrypt
hash, calls `generateAccessToken` (signed JWT) + `generateRefreshToken` (uuid
stored in `refreshTokenStore`) → returns `{ accessToken, refreshToken, user }`.
Protected routes (`/logout`, `/me`) go through `authenticate` middleware,
which calls `tokenHelper.verifyToken` and attaches the decoded payload to
`req.user`.

### Error convention
Route handlers translate known domain errors to HTTP status codes inline
(e.g. `'Invalid credentials'` → 401) and pass everything else to `next(err)`,
where `errorHandler` logs it and returns a generic 500. Preserve this pattern
when adding routes — don't leak internal error messages to clients from the
global handler.

### Token model
Two-token scheme: short-lived signed JWT access token (claims: `sub`, `email`,
`role`) + opaque UUID refresh token stored server-side in `refreshTokenStore`
with `{ userId, createdAt }`. Refresh tokens are revoked by deleting the map
entry (`revokeToken`, logout).

Example: Generating `validateCreditCard` and Its Test Case from a Prompt

Note: creating a function like this automatically creates a test case also (via the write-tests skill — covered in section 7).

Claude adding a JSDoc-documented validateCreditCard function (Luhn checksum, 13-19 digits) to validators.js.

Claude adding Jest test cases for validateCreditCard covering valid Visa and Mastercard test numbers.

6. Auto Mode vs. Plan Mode

Auto mode — Claude decides what to do automatically and does not ask for confirmation.
Plan mode — we can ask Claude to create a plan only, without executing.
Use btw to ask a question which runs separately from the main section.
Always switch to manual mode before making changes you want to review first.
In plan mode, Claude will ask you to go ahead with the changes first.

Claude's plan mode proposal for a new validateBirthDate validator, with scope and constraints confirmed before execution.

In auto mode, this behaviour changes — Claude proceeds independently.

7. Commands vs. Skills vs. Agents

/commands — executed manually, defined via a markdown file.
Skills — run automatically for us, based on what's asked. A command can be converted into a skill. Skills go into a skills folder.
- There are a lot of third-party skills available.
- It's better to create our own skills, which use fewer tokens compared to those available externally, which tend to use more.
Agents — specialized assistants that do a task in the background and come up with an answer. Run an agent separately in the background, or run it via a skill — a skill can also hand over a task to an agent.

Rule of thumb: never use one-liners to do your job — don't go back to the tool again and again.

Example: custom command — `/standup`

---
name: standup
description: Generate a daily standup update from git history and open files
disable-model-invocation: true
---
You are a senior developer preparing a daily standup update for your team.
This is a Node.js/Express authentication API project. You have been working
in this codebase today. Check git log --since="00:00" --oneline and
git diff HEAD to understand what actually changed.

Draft a standup update based only on what you find in the git history
and open files — do not invent or assume work that is not visible.

Format:
Yesterday: [completed items, specific function or file names]
Today:     [in-progress items based on uncommitted changes or open TODOs]
Blockers:  [failing tests, TODO/FIXME comments, incomplete functions]

Keep each section to 3 bullet points maximum.
Use specific names — file names, function names, issue numbers where visible.
Be factual and brief. No filler. Write it as if reading it aloud in 30 seconds.

Output of the /standup command summarizing yesterday's commits, today's work, and current blockers.

If Claude Code is used to create code, it should also be used to create the commit.

Metaprompting

Take help of Claude to create a prompt using the RCTFCF framework (Role, Context, Task, Constraint, Format):

"Act as a prompt expert and help me create a structured prompt using role, context, task, constraint, and output format which can be used to do a security review for a Node.js API written in the Express framework. I want to check top 10 OWASP issues."

Also relevant: chain of thought prompting, and the observation that LLMs understand prompting better in their own language (i.e. structured, explicit formats they're trained to parse).

Claude implementing validateDateOfBirth in validators.js following the ISO-date, no-future, 120-year-limit convention.

The /raise-pr command refusing to push directly to main and creating a feature branch instead.

Example: custom command — `/raise-pr`

---
name: raise-pr
description: Raise a GitHub pull request for the current branch targeting main
disable-model-invocation: true
---
You are a senior developer raising a pull request for a fix or feature branch.

Steps:
1. Run git branch --show-current to get the current branch name.
2. Run git log main..HEAD --oneline to list all commits on this branch.
3. Run git diff main...HEAD to inspect all changes introduced by this branch.
4. Check if the branch name or commits reference an issue number (e.g. fix/issue-204
   or "closes #204"). Extract it if present.

Using what you observe, generate a PR title and body:

Title format:
  type(scope): short summary under 72 characters  (same as the commit message)

Body format:
  ## What changed
  - Bullet points describing each logical change
  ## Why it changed
  - The problem or issue this PR resolves (reference issue number if found, e.g. closes #204)
  ## How to test
  - Step-by-step instructions to verify the fix or feature works

Then run:
  gh pr create --base main --title "<title>" --body "<body>"

If the branch has no upstream yet, push it first:
  git push -u origin <branch-name>

Output only the PR title and body. No explanation, no commentary, no preamble.
Be precise and factual. Every word should earn its place.

Running /raise-pr:

Permission prompt confirming the git push of the new feature branch to origin.

Confirmation that the feature branch was pushed and PR #1 was opened on GitHub.

We can see the PR is created:

GitHub pull requests tab showing the newly opened validateDateOfBirth PR.

Skills are auto-invoked

/commands are executed manually, but skills get applied automatically based on the prompt. Seeing a list of all skills:

The /skills menu listing all 9 available skills and their lock/enabled status.

Skills could also come from installed plugins.

8. Code Review via Subagents

A code review skill can delegate each dimension of the review to a dedicated subagent using the Task tool:

## 1. Security
Delegate this dimension to the `security-analyst` subagent using the Task tool.

## 2. Performance
Delegate this dimension to the `performance-analyst` subagent using the Task tool.

## 3. Test Coverage
Delegate this dimension to the `test-coverage-analyst` subagent using the Task tool.

## 4. Code Quality
Delegate this dimension to the `code-quality-analyst` subagent using the Task tool.

How to use a skill

Example prompt:

❯ look at the auth module and tell me if there are any problems

The code review skill should auto-invoke:

Claude auto-invoking the code-review skill and launching four parallel review agents against the auth module.

No explicit integration is needed — it's auto-invoked. No need to use /command explicitly.

A sample of the code review report:

Consolidated code review report table listing critical and high-severity findings across the auth module.

Detailed write-up of two critical findings: a syntax-breaking stray await and a bypassed bcrypt password check.

Example: skill — `deploy`

If I want to deploy, deploy.sh should be used — so we use a deploy skill:

---
name: deploy
description: >
  Deploy the application to staging. Use when the user says "deploy",
  "ship it", "push to staging", or "release". Runs pre-flight checks,
  builds, tags, and verifies the deployment. Always confirm with the
  user before executing.
disable-model-invocation: true
---
# Deploy Skill
Deployment logic lives in `scripts/deploy.sh`. That script is the
source of truth — do not re-implement its steps here.

## Before Running
Confirm with the user:
- "Ready to deploy to staging. This will tag and push. Proceed?"
- Do not proceed without explicit confirmation.

## Execution
Run: `bash scripts/deploy.sh`

Example: skill — `write-tests`

Similarly, we can use a write-tests skill to write a test case:

---
name: write-tests
description: >
  Use this skill whenever new testable logic is added or modified — functions,
  methods, classes, route handlers, or modules in src/. Triggers on: "add a
  function", "create a utility", "implement this", "write a handler", "add a
  route", or any task that produces new logic with inputs and outputs or
  side effects. Do NOT wait to be asked — write tests as part of completing
  the task. Skip this skill only for: config constants, type definitions,
  pure re-exports, or framework boilerplate with no logic.
---
# Write Tests Skill
You are a senior QA engineer writing exhaustive Jest test suites for a
Node.js application.

## Stack & Conventions
- **Framework**: Jest with Supertest for HTTP integration tests
- **Test location**: `tests/` mirroring `src/` structure, `.test.js` suffix
- **Logger**: Structured JSON — never assert on `console` output
- **Auth**: `JWT_SECRET` defaults to `'dev-secret-key'` in tests — do not
  assert real security guarantees against this value
- **Test data**: Define inline for simple cases; use `tests/fixtures/` for
  objects reused across 3+ tests

Initiating the write-tests skill:

The write-tests skill identifying and filling a missing leap-year test case for validateDateOfBirth.

Skills are auto-invoked. LCOV coverage output:

Test suite results showing 76 passing tests with 100% coverage on validators.js.

Creating a custom agent

Skills, agents, and commands differ as follows:

Agent — a specialized assistant that does a task in the background and comes up with an answer. Run it separately in the background, or run it via a skill — a skill can also hand a task over to an agent.
In an agent definition, we can specify exactly what tools the agent can use.

Example of agents:

The project's custom subagent definitions listed in the .claude/agents folder.

Example: custom agent — `security-analyst`

---
name: security-analyst
description: >
  Use this agent for security reviews: authentication bypass, injection vulnerabilities, OWASP Top 10, token handling, and input validation. Invoke with: ask the security-analyst to audit this file.
tools: Read, Bash(npm audit *), Bash(grep *), Write, Edit
model: opus
memory: project
---
# Security Analyst Agent
You are a senior application security engineer. You read code exclusively through
a security lens. You do not suggest feature improvements or code style changes.
You find vulnerabilities and you explain how to fix them.

## Your Focus Areas
- Authentication bypass and broken access control
- Injection vulnerabilities: SQL, command, path traversal
- Sensitive data exposure in logs, responses, or error messages
- Insecure token handling: weak secrets, missing expiry, improper storage
- Missing input validation and sanitisation
- Dependency vulnerabilities (flag for npm audit review)

## OWASP Categorisation
Tag every finding with its OWASP Top 10 category where applicable.

## Memory Protocol
Before starting, read `MEMORY.md` (an index of one-line links to note files) and
skim any linked notes whose description looks relevant to the file(s) you're
about to audit.

After completing your audit, write to memory only if you found something
non-obvious, recurring, or specific to this codebase that would save real time
on a future audit — not routine findings you'd already report. If so:
1. Create a new note file named for the pattern (e.g. `jwt-secret-hardcoded-in-config.md`)
   with `name`, `description`, and `metadata: {type: feedback}` frontmatter, following
   the structure used in existing notes.
2. Add a one-line link to it from `MEMORY.md`.

Do not rewrite or duplicate an existing note — update it in place if the same
pattern recurs with new detail.

## Output Format
For each finding:
- OWASP Category
- Severity: Critical / High / Medium / Low
- File and line number
- What an attacker could do with this vulnerability
- Exact fix with corrected code snippet

End with: total finding count by severity, and one recommended immediate action.

How does a skill hand over a task to an agent?

Claude launching the security-analyst subagent in the background to audit the authentication codebase.

The agent runs in the background

● Agent "Security audit of auth codebase" finished · 4m 42s

The security-analyst audit found 18 findings across authService.js, tokenHelper.js, routes.js, middleware.js, and validators.js. Two are critical and one of them also breaks the server outright.

Critical

Auth bypass — bcrypt.compare(...) at authService.js:32 is called without await, so the truthy Promise always passes the check. Every password is accepted for any login.
Hardcoded JWT fallback secret — 'dev-secret-key' in both authService.js:11 and tokenHelper.js:8, used silently if JWT_SECRET is unset. Anyone can forge admin tokens.

High

Login always builds an inline mockUserRecord with a constant id: 'user-001' and attacker-supplied email — every session collapses to the same identity, with a spoofable email claim.
No rate limiting on /login or /refresh — unlimited credential stuffing / refresh-token brute-forcing.
REFRESH_EXPIRES_IN is declared but never enforced — refresh tokens never expire.

Medium (6 findings) — notably: authService.js:80 is missing the async keyword on refreshToken, which is a SyntaxError that prevents the server from booting at all (confirmed via node --check) — meaning none of this is currently under test/CI coverage; an inverted expiry comparison in isTokenExpired that also deletes valid sessions; /logout can revoke anyone's refresh token (no ownership check); access tokens aren't invalidated on logout; PII (raw emails) logged on every auth attempt; unbounded in-memory token store (memory-exhaustion DoS).

Low (7 findings) — untrimmed email reaching downstream sinks, internal error messages leaked to clients via substring matching, missing JWT algorithm allowlist, missing security headers/CORS/body-size limits, loose validateBody treating null/0/false as "present," a low-severity uuid advisory (not reachable, since uuidv4() is called with no args), and a latent timing side-channel for user enumeration.

Good news: the newly added validateDateOfBirth (commit de91f6f) was checked directly and is clean — correctly rejects rollover dates, future dates, ages over 120, and malformed input. It's not currently wired into any route, so it's not a live attack surface either way.

Recommended immediate fix (one commit): add the missing await on bcrypt.compare (line 32) and the missing async on refreshToken (line 80) together — the second is required just to get the app to boot, and without it nothing else can even be tested. Rotate JWT_SECRET and make it mandatory at startup right after.

9. Multi-Agent Harness Concepts

When Claude Code itself uses multiple subagents to work on a task, we can say it's using a multi-agent harness. The main thread uses the info from all the agents to create a single report.

A second pass: performance, test coverage, and code quality

Prompt: "Create agent for performance review, test coverage, and code quality — use the code review skill."

Claude clarifying review scope, then running performance, test-coverage, and code-quality agents in parallel while fixing the critical bugs.

Full output — Code Review Report (Performance / Test Coverage / Code Quality)

Context: Re-run of the performance, test-coverage, and code-quality dimensions against src/auth/authService.js (security was already covered in the prior review). Findings are consistent with the earlier pass; a few additional specifics surfaced this time.

#	Dimension	Finding	Severity
1	Code Quality / Test Coverage	Missing `async` on `refreshToken` (line 80) + `await` at line 92 → `SyntaxError`, module can't load, entire app fails to boot	🔴 Critical
2	Code Quality / Test Coverage	`bcrypt.compare` not awaited (line 32) → auth bypass, confirmed independently by ESLint's `require-await` flagging `loginUser` as having no real await	🔴 Critical
3	Test Coverage	No test file exists for `authService.js` at all — every exported function has 0% coverage	🔴 Critical
4	Test Coverage	`isTokenExpired` inverted logic (line 68) — verified via failing Jest assertions against real signed JWTs	🔴 Critical
5	Code Quality	`isTokenExpired` duplicates `tokenHelper.js`'s JWT logic instead of delegating (violates CLAUDE.md's documented layering), and is applied to opaque UUID refresh tokens where it always short-circuits via `jwt.decode() === null`	🟠 High
6	Code Quality	`REFRESH_EXPIRES_IN` and `createdAt` are dead code — confirmed via ESLint `no-unused-vars` — refresh tokens never actually expire despite CLAUDE.md documenting a 7-day window	🟠 High
7	Performance	Unbounded `refreshTokenStore` Map — no TTL sweep, grows with cumulative logins not concurrent sessions	🟠 High
8	Test Coverage	`refreshToken()`, `revokeToken()`, `generateRefreshToken()`, `generateAccessToken()` have no direct unit tests or edge-case coverage	🟠 High
9	Performance	`bcryptjs` (pure-JS, not native bcrypt) runs password comparison on the main thread, not the libuv threadpool — inflates tail latency under concurrent login bursts	🟡 Medium
10	Code Quality	Swallowed error in `isTokenExpired`'s `catch (err)` (line 69-71) — caught but never logged, inconsistent with the rest of the file's logging pattern	🟡 Medium
11	Code Quality	`curly` ESLint rule violation on line 64 (`if (...) return true;` without braces)	🟡 Medium
12	Code Quality	`require-await` violation on `getUserById` (line 153) — declared async with no await inside	🟡 Medium
13	Test Coverage	`loginUser` with no `userRecord`, JWT claims shape, `getUserById` stub contract — untested but low-risk edge cases	🟢 Low
14	Code Quality	`revokeToken`'s "not found" branch logs nothing, inconsistent with other failure branches in the file	🟢 Low
15	Code Quality	Imprecise JSDoc return shapes (`@returns {object}` instead of a typed shape)	🟢 Low

New since the last pass

Tooling is broken at the project level: npm run lint currently fails outright — the repo ships .eslintrc.json (legacy config) but has ESLint 10.9.1 installed, which requires flat eslint.config.*. No flat config exists at the repo root. The code-quality agent worked around this by building an equivalent flat config in the scratchpad to get real lint signal (never touched the actual repo files), but as things stand today, npm run lint gives false confidence — it fails before it even reaches your code. This is worth fixing independent of authService.js.
bcryptjs vs native bcrypt (finding 9): not raised as a distinct finding in the first pass — flagged now because it compounds with the login-bypass fix. Once await is added back (finding 2), every login request will run a real CPU-bound bcrypt comparison on the main thread; under concurrent login bursts this inflates latency for unrelated concurrent requests. Not urgent, but worth knowing before assuming the fix is "free."

Summary Score: 2/10 (unchanged) — the file can't boot, has zero test coverage, and now additionally the project's lint tooling can't even verify it.

Recommended First Action: unchanged — fix the missing async on refreshToken (line 80) together with the missing await on bcrypt.compare (line 32) in one commit — nothing else here (tests, lint, the rest of the app) is reachable until the module can actually load. Separately, worth a quick fix to add eslint.config.js so npm run lint works again — that's a repo-wide gap, not specific to this file.

This is a hub and spoke model — using subagents via a skill. However, what we actually want in some cases is agents communicating with each other independently in the background to finish a task; the hub-and-spoke approach above cannot do that. Agents can instead be run using other agentic harnesses — loop, schedule, etc.

Difference between agent and agentic

Comparison diagram contrasting framework-driven "Agent" orchestration with model-native "Agentic" execution.

Goal, loop, schedule, workflow, batch — agentic primitives available which help create subtasks (e.g. ultraplan, autofix-pr).
/commands are manual.
Skills get applied automatically based on the prompt.
/agent runs in the background.

10. Hooks

HOOKS, PLUGINS, MCP e2e flow — from evaluating code to fixing code, full auto flow.

Lifecycle methods available in Claude: from the start to the end of a session, internal events fire inside the session, which are used as plug points for our own code.

Hook event flow

Whenever Claude reaches a lifecycle boundary — e.g. when a tool is about to run — it spawns our custom script as a subprocess and feeds it a JSON blob on stdin. The script can execute anything, then return:

an exit code (0 or 2), or
a JSON object on stdout containing a structured decision, reason, and context for Claude to read.

Hooks can be created using Python or JavaScript. They take input from Claude Code, display it on screen, and allow the tool to run.

Example: `PreToolUse` hook on Bash

{
  "hooks": {
    "PreToolUse": [
      {
        "matcher": "Bash",
        "hooks": [
          {
            "type": "command",
            "command": "node .claude/hooks/bash-guard.js"
          },
          {
            "type": "command",
            "command": "node .claude/hooks/pretooluse-demo.js"
          }
        ]
      }
    ]
  }
}

Since this hook sits directly in front of every Bash call, it's a good interception point for anything you want enforced before a shell command runs. Some concrete uses, especially relevant to a repo like this:

Guardrails / safety (this is what bash-guard.js already seems to do) — block dangerous patterns beyond what settings.json's deny-list covers, e.g. rm -rf variants, git push --force, piping to sh, etc., using regex instead of exact string matching.

All hook handler types

{
  "hooks": {
    "PreToolUse": [
      {
        "matcher": "Bash",
        "hooks": [
          { "type": "command", "command": "node .claude/hooks/pretooluse-demo.js" },
          { "type": "http", "url": "https://www.example.com/validate" },
          { "type": "prompt", "prompt": "Evaluate the steps before running the tool" },
          { "type": "agent", "prompt": "Run security-analyst to verify the issues" },
          { "type": "mcp_tool", "server": "some-mcp-server", "tool": "validate_edit" }
        ]
      }
    ]
  }
}

The bash-guard hook's log file recording every Bash command it allowed during the session.

Blocking dangerous commands

const DANGER_PATTERNS = [
    /rm\s+-rf\s+\//,                          // recursive force-delete from root (e.g. "rm -rf /")
    /sudo\s+rm/,                              // any sudo-elevated delete — bypasses normal permission checks
    /:\(\)\{\s*:\|:&\s*\}\s*;\s*:/,           // fork bomb — ":(){ :|:& };:" spawns processes until the system locks up
    /dd\s+if=.*of=\/dev\/sd/,                 // raw disk write via dd — can overwrite an entire drive
    /chmod\s+777\s+\//,                       // world-writable permissions on root — opens up the whole filesystem
];

Notification on Slack

{
  "Notification": [
    {
      "matcher": "permission_prompt",
      "hooks": [
        {
          "type": "command",
          "command": "node .claude/hooks/slack-notify.js"
        }
      ]
    }
  ]
}

Logs can be used to audit our work — when we did what, and how much time we spent.

Sharing configuration

What if I want to use all my settings in the .claude folder in another project? This can be done via GitHub. But how can we give it to a third party? Through plugins — see code.claude.com/docs/en/plugins for how to create one.

11. MCP (Model Context Protocol)

The concept of an MCP server

The GitHub REST API knows how to connect to GitHub on your behalf.

Diagram showing Git Bash and the GH CLI both reaching GitHub's SaaS platform through the GitHub REST API.

Either Claude directly connects via Bash, or via the gh CLI — both of which need to be authenticated. Actually, the gh CLI is itself using Bash commands under the hood. But can Claude Code directly connect to the GitHub REST API to be faster and more efficient? curl can be used for that:

Expanded diagram showing how Claude Code uses its Bash tool (via curl, git, or gh CLI) to reach the GitHub REST API.

Why MCP exists

Every ecosystem/SaaS (like GitHub) has a separate, specific, proprietary way of connecting to it — so if Claude has to connect to multiple SaaS platforms, it would need to know all of them, which deviates Claude from its core functionality.

Can GitHub give a way to connect directly to a REST API? So Anthropic created a new protocol:

Claude Code has an inbuilt MCP client when installed.
GitHub/any SaaS provider provides an MCP server — basically an endpoint.
Our job is only to connect to the MCP server; the MCP server handles the rest.

Three types of MCP server

stdio — runs locally on the machine
http — hosted at an HTTP location
SSE — Server-Sent Events, a streaming protocol

All AI tools support MCP servers. The MCP client is part of the ecosystem.

Claude Code runs with an agentic harness — the LLM decides what to do, and the LLM tells Claude Code which tools to use.

12. End-to-End Automation

End-to-end automation of security review → issue creation → issue fix → PR creation → PR fix, all through agents.

The /security-review command loading its skill to audit tokenHelper.js.

Security review finding a critical hardcoded fallback JWT signing secret vulnerability in tokenHelper.js.

`/create-issue`

Claude using the gh CLI to file a GitHub issue for the hardcoded JWT secret vulnerability.

GitHub issue #2 detailing the hardcoded fallback JWT secret bug, its impact, and expected fix.

Now fix the issue

Claude implementing the fix: removing the hardcoded JWT secret fallback and requiring the env var explicitly.

Commit the changes

The /commit command staging, reviewing, and pushing the JWT secret fix with a detailed commit message.

GitHub activity showing the fix commit automatically linked to close issue #2.

Pull request description summarizing what changed and why for the JWT secret fix.

13. Agent Teams

Beyond single subagent delegation, Claude Code supports Agent Teams — multiple agents working the same task from different angles in parallel, then synthesizing a combined result.

Example prompt

Create an agent team to review PR #1.
Spawn three reviewers:

Reviewer 1: focused exclusively on security implications

Reviewer 2: focused on performance and scalability

Reviewer 3: validating test coverage and edge cases

Have them each review independently, then synthesize findings into a single report.
Create a new branch with custom name including today's date.
Merge and close the PR with comment.

User prompt instructing Claude to spawn three specialized reviewer agents to review and merge PR #1.

The 3 agents are invoked in the background:

Claude launching three specialized reviewer agents in parallel to review PR #1.

Progress log showing the security and test-coverage reviewer agents finishing with real findings.

Claude synthesizing all three reviewer agents' findings before applying fixes and merging.

Claude merging the verified fix, filing two follow-up issues, and preparing to post the review synthesis as a PR comment.

Final summary of the agent-team review: verdicts from all three reviewers and the actions taken to merge PR #1.

14. Key Takeaways

The agentic loop (gather context → decide → act → verify) is the mental model for everything Claude Code does.
CLAUDE.md and .claude/settings.json are how you shape Claude's behavior and permissions for a specific project.
Commands are manual, Skills auto-trigger, and Agents run scoped, background work — often orchestrated together.
Hooks and MCP are the two extension points: hooks intercept lifecycle events for guardrails/automation, MCP standardizes how Claude talks to external tools.
Session management (/resume, /fork, /compact) is essential for working on multiple issues without polluting context or wasting tokens.
Agent Teams unlock genuinely parallel, multi-perspective work — like having three reviewers look at a PR simultaneously.

This was a genuinely comprehensive look at how far the agentic coding model has come — from a single loop reading and writing files, to fully orchestrated, permissioned, multi-agent development workflows.

AI & ML Foundations: Cleaning Data, Building Regression & Classification Models in Python.

yash-nigam — Mon, 31 Aug 2026 16:53:20 +0000

My 5-Day Journey into Machine Learning: From Data Cleaning to My First ML Models

Learning goal: This is not a collection of code snippets to memorize. It is a beginner-friendly guide to understanding why each step in a typical machine-learning workflow exists, what problem it solves, and how the pieces fit together.

A note about the source material

This guide was built primarily from the four Jupyter notebooks you provided:

11_Aug_mpg.ipynb
12-Aug-2.ipynb
12th-Aug-Telco-Customer-churn.ipynb
13-Aug.ipynb
123

Your PDF notes were also provided, but the PDF is image-based and its text could not be reliably extracted. So I have not invented or silently reconstructed material from the PDF. The explanations below are grounded mainly in the notebooks, with general ML knowledge added to explain the concepts behind the code.

1. Overview: What did I actually learn in these five days?

At first glance, machine learning can look like this:

Load dataset
    ↓
Clean data
    ↓
Create graphs
    ↓
Train model
    ↓
Check accuracy
    ↓
Done!

But that misses the most important part.

The real ML workflow is closer to:

Business / real-world question
        ↓
Define what we want to predict
        ↓
Understand the data
        ↓
Clean and prepare the data
        ↓
Explore the data visually
        ↓
Choose useful features
        ↓
Convert data into numbers
        ↓
Split into training and testing data
        ↓
Choose an appropriate ML algorithm
        ↓
Train the model
        ↓
Evaluate it on unseen data
        ↓
Compare models
        ↓
Improve preprocessing / model settings
        ↓
Use the model to make predictions

The important insight is:

Machine learning is not mainly about choosing an algorithm. It is about turning a real-world question and messy data into a reliable prediction process.

Your notebooks actually demonstrate most of this workflow.

2. What is Artificial Intelligence?

Artificial Intelligence (AI) is the broad idea of building systems that can perform tasks that normally require some form of human intelligence.

Examples:

recognizing an image
understanding language
recommending a movie
detecting fraud
predicting whether a customer will leave
predicting the price/value of something
generating text or images

A useful mental model is:

Artificial Intelligence
│
├── Machine Learning
│   │
│   ├── Supervised Learning
│   ├── Unsupervised Learning
│   └── Reinforcement Learning
│
├── Deep Learning
│
└── Generative AI

This is simplified because these areas overlap, but it is a useful beginner mental model.

3. What is Machine Learning?

Machine learning is a way of building systems where the computer learns patterns from data instead of us manually writing every rule.

Imagine we want to predict whether a telecom customer will churn.

A traditional rule-based approach might look like:

IF customer has a month-to-month contract
AND monthly charges are high
AND tenure is low
THEN churn = yes

The problem is that we have to invent those rules ourselves.

With machine learning, we give the algorithm historical examples:

Customer information → Actual churn result

Customer A → No
Customer B → Yes
Customer C → No
Customer D → Yes
...

The algorithm tries to learn the relationship between the input information and the known outcome.

Then we can give it a new customer:

New customer information
        ↓
      Model
        ↓
Predicted churn

That is the core idea of supervised machine learning.

4. The most important ML vocabulary

Before learning algorithms, understand these words.

4.1 Dataset

A dataset is a collection of examples.

In your notebooks, a dataset is loaded into a Pandas DataFrame.

For example:

df = pd.read_csv("Telco_Customer_Churn.csv")

Think of the DataFrame as a spreadsheet.

4.2 Row / observation / sample

Each row represents one example.

For the Telco dataset:

one row = one customer

For the Auto MPG dataset:

one row = one car

For the insurance dataset:

one row = one customer/policy record

These words are often used interchangeably:

row
observation
sample
instance

4.3 Feature

A feature is an input variable that the model can use to make a prediction.

For customer churn:

tenure
MonthlyCharges
Contract
InternetService
PaymentMethod
...

These are features.

A useful question to ask is:

"What information is available to the model before it makes its prediction?"

That information is generally your feature set.

4.4 Target

The target is what we are trying to predict.

Examples from your notebooks:

Telco

Target = Churn

The model predicts:

0 → No churn
1 → Churn

Insurance

Target = Customer Lifetime Value

The model predicts a numerical value.

4.5 X and y

You repeatedly used:

X = df.drop(["Churn"], axis=1)
Y = df["Churn"]

This is one of the most important patterns in supervised ML.

X = inputs/features
Y = target/output

For example:

X
├── tenure
├── MonthlyCharges
├── Contract
├── InternetService
└── ...

Y
└── Churn

Think:

X = what the model knows

Y = what we want the model to learn to predict

5. The three major types of Machine Learning

5.1 Supervised Learning

In supervised learning, we have examples where the correct answer is already known.

Input X → Known answer Y

The model learns the relationship between X and Y.

Your notebooks mainly use supervised learning.

There are two major supervised learning tasks.

Classification

The target is a category.

Examples:

Spam / Not Spam
Churn / No Churn
Fraud / Not Fraud
Disease / No Disease

Your Telco project is a classification problem.

Regression

The target is a numerical value.

Examples:

House price
Temperature
Sales
Customer Lifetime Value
Fuel efficiency

Your Customer Lifetime Value project is a regression problem.

6. Classification vs Regression

This distinction is extremely important.

Classification

Question:

"Which category does this example belong to?"

Example:

Customer → Churn or No Churn

Possible output:

0
1

Common classification algorithms you tried:

Logistic Regression
Support Vector Classifier (SVC)
Decision Tree Classifier

Regression

Question:

"What numerical value should we predict?"

Example:

Customer → Customer Lifetime Value = 6,543.21

Common regression algorithms you tried:

Linear Regression
KNN Regressor
SVR
Decision Tree Regressor
Bagging Regressor
AdaBoost Regressor
Random Forest Regressor

7. Unsupervised Learning

In unsupervised learning, we do not have a known target.

Instead, the algorithm tries to discover structure in the data.

For example:

Customer data
     ↓
Find groups
     ↓
Group 1: high-value customers
Group 2: price-sensitive customers
Group 3: new customers

Common examples:

K-Means clustering
Hierarchical clustering
PCA

You did not build an unsupervised-learning project in these notebooks, but it is important to know where it fits.

8. Reinforcement Learning

Reinforcement learning is different.

An agent interacts with an environment and receives rewards or penalties.

Agent
  ↓ action
Environment
  ↓
Reward / penalty
  ↓
Agent learns

Examples:

game-playing agents
robotics
certain recommendation/control systems

This was not part of your five-day practical work.

9. Different ways to approach an ML problem

A useful way to think about ML is not:

"Which algorithm should I use?"

Instead ask:

Step 1 — What is the question?

For example:

Can I predict whether a telecom customer will churn?

Step 2 — What type of target do I have?

Churn → category → Classification
Customer Lifetime Value → number → Regression

Step 3 — What data is available?

Ask:

What columns exist?
Which are numerical?
Which are categorical?
Are values missing?
Are there strange values?
Are some columns identifiers?
Is the target balanced?

Step 4 — Explore before modeling

Use statistics and graphs.

Step 5 — Prepare the data

Convert it into a form the algorithm can understand.

Step 6 — Establish a baseline

Train a simple model first.

Step 7 — Compare alternatives

Try other appropriate models.

Step 8 — Evaluate on unseen data

A model is useful only if it works beyond the data it memorized.

10. Google Colab: Why did we use it?

Google Colab gives you a cloud-based Jupyter Notebook environment.

Instead of installing everything locally, you can run Python code in the browser.

A notebook combines:

Explanation
+
Python code
+
Output
+
Graphs

This makes it very useful for learning ML.

Your notebooks use paths such as:

/content/auto-mpg.csv

That is typical of a Colab environment.

A normal learning workflow is:

Upload dataset
      ↓
Open notebook
      ↓
Run cells
      ↓
Inspect output
      ↓
Change code
      ↓
Run again

11. The Python libraries you used

11.1 NumPy

import numpy as np

NumPy provides numerical operations and arrays.

You used it for things such as:

np.nan
np.log1p()
np.expm1()

`np.nan`

Represents a missing numerical value.

`np.log1p(x)`

Calculates:

log(1 + x)

It is useful when a target is heavily skewed.

`np.expm1(x)`

Reverses log1p:

exp(x) - 1

You used this later to convert log predictions back to the original CLV scale.

12. Pandas

import pandas as pd

Pandas is one of the most important Python libraries for data work.

Think of a Pandas DataFrame as a spreadsheet that Python can manipulate.

You used it to:

load CSV files
inspect data
remove columns
convert data types
detect missing values
fill missing values
select X and Y
create encoded columns
save predictions

Example:

df = pd.read_csv("Telco_Customer_Churn.csv")

13. Matplotlib

import matplotlib.pyplot as plt

Matplotlib is a general-purpose plotting library.

It provides the foundation for many Python visualizations.

You used it for:

plt.figure()
plt.title()
plt.show()

14. Seaborn

import seaborn as sns

Seaborn makes statistical visualizations easier to create.

You used:

countplot
boxplot
histplot
pairplot
scatterplot
heatmap

A key lesson:

A graph is not decoration. It is a tool for asking questions about the data.

15. Scikit-learn

Scikit-learn is the main ML library used in your notebooks.

You used it for:

splitting data
preprocessing
encoding
scaling
classification
regression
evaluation

The basic pattern is:

model = SomeModel()
model.fit(X_train, Y_train)
predictions = model.predict(X_test)

This pattern appears again and again in ML.

16. The most important workflow: EDA

EDA means:

Exploratory Data Analysis

EDA is the process of investigating the dataset before building the final model.

Your notebooks use:

df.head()
df.shape
df.describe()
df.info()
df.sample()
df.isnull().sum()

plus graphs.

These are not random commands.

They answer different questions.

17. `df.head()` — What does the data look like?

You used:

df.head()

It displays the first few rows.

Why?

Because before doing anything else, you want to see:

column names
example values
obvious data problems
whether the data loaded correctly

Think of it as opening the box before using what is inside.

18. `df.shape` — How much data do I have?

You used:

df.shape

For Auto MPG:

(398, 9)

That means:

398 rows
9 columns

Why does this matter?

Because dataset size affects:

model choice
computation time
confidence in results
risk of overfitting

19. `df.info()` — What types of data do I have?

You used:

df.info()

This tells you:

column names
number of non-null values
data types
memory usage

This is extremely important.

For example, in Auto MPG:

horsepower → object

At first glance horsepower sounds numerical.

But Pandas sees it as text.

Why?

Because the dataset contains values such as:

A column containing:

cannot be treated as a clean numerical column.

So:

df.info() helps you detect data-type problems before modeling.

20. `df.describe()`

You used:

df.describe()

and sometimes:

df.describe(include="all")

This gives summary statistics.

For numerical data you get things such as:

count
mean
standard deviation
minimum
25th percentile
median
75th percentile
maximum

This helps you understand the scale and distribution of variables.

For example:

minimum
   ↓
25%
   ↓
median
   ↓
75%
   ↓
maximum

A large difference between the 75th percentile and maximum can be a clue that there may be extreme values.

21. `df.sample()`

You used:

df.sample(5)

or similar.

This shows random rows.

Why?

Because the first rows may not represent the whole dataset.

Random samples can expose:

unusual values
formatting issues
unexpected categories
data-entry problems

22. Missing data

You used:

df.isnull().sum()

This asks:

"How many missing values are there in each column?"

Missing data is important because many ML algorithms cannot directly work with missing values.

But there is a subtle lesson from your Auto MPG notebook.

You first checked:

df.isnull().sum()

while horsepower still contained "?".

Pandas did not count "?" as a missing value because "?" is a string, not a true NaN.

That means:

"?" ≠ NaN

So data cleaning sometimes requires identifying fake missing values first.

23. Auto MPG: cleaning horsepower

Your notebook did:

df["horsepower"] = df["horsepower"].replace("?", np.nan)

This converts the placeholder into a real missing value.

Then:

df["horsepower"] = pd.to_numeric(df["horsepower"])

Now Pandas can treat horsepower as numerical data.

This is a very important data-cleaning pattern:

messy text
   ↓
identify invalid placeholder
   ↓
convert to NaN
   ↓
convert column to numeric
   ↓
handle missing values

24. Why use the median?

You calculated:

median_horsepower = df["horsepower"].median()

and then filled missing values:

df["horsepower"] = df["horsepower"].replace(
    np.nan,
    median_horsepower
)

Why median?

Because the median is less affected by extreme values than the mean.

Example:

10, 11, 12, 13, 1000

The mean is pulled heavily upward by 1000.

The median is much more representative of the middle of the data.

A useful rule of thumb:

fairly symmetric data → mean may work well
skewed data / outliers → median is often safer

But remember:

There is no universal rule that "median is always correct."

The best treatment depends on the data and the problem.

25. Dropping an identifier

You considered dropping:

df.drop("car name", axis=1)

and actually dropped customerID in the Telco work.

Why?

An identifier is usually not a meaningful predictive feature.

For example:

customerID = 759832

does not inherently mean the customer is more or less likely to churn.

The ID exists to identify the row, not to describe the customer.

Important distinction:

A column can be useful for identifying a record without being useful for predicting the target.

26. But don't blindly drop columns

This is an important improvement to your original reasoning.

A column should not be dropped merely because it is an object/string.

For example:

Contract
InternetService
PaymentMethod

are categorical strings, but they can contain very useful predictive information.

So the correct question is:

"Does this column contain useful information that is available at prediction time?"

not:

"Is this column numeric?"

27. Understanding your graphs

You created many graphs. The important thing is to understand what question each graph answers.

28. Countplot

Example:

sns.countplot(x="Churn", data=df)

A countplot answers:

"How many observations are in each category?"

For churn:

No churn → number of customers
Churn    → number of customers

This immediately helps you understand class balance.

If the graph looks like:

No Churn ███████████████████
Churn    ███████

the classes are imbalanced.

That matters because a model could achieve high accuracy simply by favoring the majority class.

29. Countplot with `hue`

You used:

sns.countplot(
    x="Contract",
    hue="Churn",
    data=df
)

This answers:

"How does the target category vary across another category?"

For example:

Contract type
     ↓
Churn / No churn

If month-to-month customers show much more churn than two-year customers, that is an important pattern worth investigating.

But remember:

A graph showing association does not automatically prove causation.

30. Histogram

You used:

df.hist(figsize=(10,10))

and:

sns.histplot(df["MonthlyCharges"], kde=True)

A histogram shows the distribution of a numerical variable.

It helps answer:

Where are most values?
Is the data symmetric?
Is it skewed?
Are there multiple groups?
Are there extreme values?

Example:

frequency
   ^
   |       ███
   |     ███████
   |   █████████
   | ███████████
   +-----------------> value

31. Why distributions matter

Suppose a feature looks like:

Most values: 0–100
A few values: 10,000

That may indicate:

genuine outliers
a different population
data-entry errors
a highly skewed distribution

A model may react differently depending on the algorithm.

This is one reason EDA happens before modeling.

32. Boxplot

You used:

sns.boxplot(y=df["Income"])

A boxplot summarizes a distribution.

Conceptually:

       outlier
          •
          |
      ┌───────┐
      │       │
------│  box  │------
      │       │
      └───────┘
          |
       outlier

The box represents the middle portion of the data.

The line inside the box is the median.

Points outside the whiskers may be treated as potential outliers.

Important:

An outlier is not automatically an error.

A high-income customer may be perfectly legitimate.

33. Boxplot: target vs category

You used:

sns.boxplot(
    x="Coverage",
    y="Customer Lifetime Value",
    data=df
)

This is a powerful graph.

It asks:

"How does the distribution of Customer Lifetime Value differ between categories?"

For example:

Coverage A → CLV distribution
Coverage B → CLV distribution
Coverage C → CLV distribution

Compare:

median
spread
outliers
overlap

This can help you identify potentially useful relationships.

34. Scatterplot

You used:

sns.scatterplot(
    x="Income",
    y="Customer Lifetime Value",
    data=df
)

A scatterplot examines the relationship between two numerical variables.

Each dot represents an observation.

It helps answer:

"As X changes, does Y appear to change?"

Possible patterns:

Positive relationship:
  •
    •
      •
        •

Negative relationship:
        •
      •
    •
  •

No obvious relationship:
 •    •
    •
  •      •
     •

The pattern does not have to be a straight line.

35. Pairplot

You used:

sns.pairplot(...)

A pairplot shows many numerical relationships at once.

It is useful for:

spotting correlations
identifying clusters
seeing distributions
finding obvious relationships
detecting possible outliers

The downside is that it becomes difficult to read when there are many columns.

So:

Pairplot is excellent for small-to-medium exploratory datasets, but not something you blindly run on every large dataset.

36. Correlation heatmap

You used:

sns.heatmap(
    df[["tenure", "MonthlyCharges", "TotalCharges"]].corr(),
    annot=True
)

Correlation measures how two numerical variables move together in a linear relationship.

A correlation close to:

+1 → strong positive linear relationship
 0 → little/no linear relationship
-1 → strong negative linear relationship

For example:

tenure ↑
TotalCharges ↑

could result in positive correlation.

But correlation has an important limitation:

Correlation does not prove causation.

Also, zero correlation does not necessarily mean "no relationship" because the relationship might be non-linear.

37. A key lesson from visualization

The graphs answer different questions:

Graph	Main question
Countplot	How many observations are in each category?
Countplot + hue	How does one category vary with another?
Histogram	What does a numerical distribution look like?
Boxplot	What are the median, spread and potential outliers?
Boxplot + category	How does a numerical distribution differ by category?
Scatterplot	How do two numerical variables relate?
Pairplot	What relationships exist among several numerical variables?
Heatmap	How strongly are numerical variables linearly correlated?

The goal is not:

"Create lots of graphs."

The goal is:

"Use the right graph to answer the right question."

38. Encoding: why does ML need it?

Many ML algorithms work with numbers.

But real-world data contains:

Male
Female

Yes
No

Month-to-month
One year
Two year

Fiber optic
DSL
No

The model needs these values represented numerically.

This is called encoding.

39. Label Encoding

You used:

from sklearn.preprocessing import LabelEncoder

and encoded categorical columns.

For a binary column:

No  → 0
Yes → 1

This can be reasonable for a binary variable.

For example:

Churn:
No  → 0
Yes → 1

That makes intuitive sense.

40. Why LabelEncoder can be dangerous for multi-category variables

Suppose:

Contract:
Month-to-month
One year
Two year

Label encoding might produce:

Month-to-month → 0
One year       → 1
Two year       → 2

A numerical model may interpret that as:

Two year > One year > Month-to-month

That ordering may not be appropriate for the algorithm.

The numbers are just codes.

They do not automatically mean that category 2 is "twice" category 1.

For nominal categories, one-hot encoding is usually safer.

41. One-hot encoding

You used:

pd.get_dummies(df, drop_first=True)

This transforms categories into separate binary columns.

For:

InternetService:
DSL
Fiber optic
No

we might create columns such as:

InternetService_Fiber optic
InternetService_No

with the remaining category represented by both values being 0.

This avoids inventing a fake numerical ordering.

42. Why `drop_first=True`?

If a categorical variable has several categories, using all one-hot columns can introduce redundancy in some models.

Dropping one category creates a reference category.

For example:

DSL
Fiber optic
No

could become:

Fiber optic
No

Then:

Fiber optic = 0
No = 0

means the reference category:

DSL

For beginner understanding, the key idea is:

One-hot encoding converts categories into machine-readable indicator variables without pretending the categories have numerical order.

43. Train/test split

You repeatedly used:

train_test_split(
    X,
    Y,
    test_size=0.3
)

This is one of the most important ML concepts.

Suppose we have 1,000 examples.

We might use:

700 → training
300 → testing

The training set is used to learn the model.

The test set is held back to evaluate how the trained model performs on unseen data.

Think of it like an exam:

Training data = practice questions
Test data     = unseen exam questions

44. Why not train and test on the same data?

Because the model could simply memorize the training examples.

Suppose:

Training score = 99%
Test score     = 65%

That is a warning sign.

The model learned the training data extremely well but does not generalize well.

This is called:

Overfitting

45. Underfitting

The opposite can happen.

If the model is too simple:

Training score = 60%
Test score     = 58%

It may not have learned enough from the data.

This is:

Underfitting

A useful mental picture:

Underfitting
Model too simple
       ↓
misses important patterns

Good fit
Learns useful patterns
       ↓
works on unseen data

Overfitting
Model learns noise/details
       ↓
great training performance
poor unseen performance

46. Why `random_state` matters

In one Telco notebook you used:

train_test_split(X, Y, test_size=0.3)

In the improved notebook you used:

random_state=42

A random state makes the split reproducible.

Without it, the random split can change between runs.

That means your score may change.

With:

random_state=42

you can reproduce the same split.

The number 42 is not magical.

Any fixed integer can serve this purpose.

47. Why `stratify=Y` is useful for classification

In your improved Telco notebook you used:

stratify=Y

This helps preserve the class distribution between training and testing data.

For example, if the full dataset contains:

73% No Churn
27% Churn

stratification aims to keep roughly the same proportion in both splits.

This is particularly useful when classes are imbalanced.

48. Logistic Regression

You used:

from sklearn.linear_model import LogisticRegression

model_lr = LogisticRegression()
model_lr.fit(X_train, Y_train)

Despite its name, Logistic Regression is commonly used for classification.

Its goal is to estimate the probability of belonging to a class.

For binary classification:

features
   ↓
Logistic Regression
   ↓
probability
   ↓
class

Example:

Probability of churn = 0.82

The model may classify that customer as:

Churn

depending on its decision threshold.

49. Why Logistic Regression is a good beginner model

It is useful because:

relatively simple
fast
often strong as a baseline
easier to interpret than many complex models
naturally suited to binary classification

A good ML habit is:

Start with a simple baseline before reaching for complex models.

50. Support Vector Machine / SVC

You used:

from sklearn.svm import SVC

model_lr = SVC()

The variable name model_lr is misleading here.

It is actually an SVC model.

SVC means:

Support Vector Classifier

The basic idea is to find a decision boundary that separates classes.

In simple 2D data:

Class A: ● ● ●

--------- decision boundary ---------

Class B: ▲ ▲ ▲

SVM tries to find a boundary with a useful margin between classes.

51. Why scaling matters for SVM

SVM is sensitive to feature scale.

Suppose one feature ranges from:

0–1

and another ranges from:

0–100,000

The larger-scale feature can dominate distance-related calculations.

That is why scaling is often important for:

SVM
KNN
Logistic Regression in many situations
neural networks

52. Decision Tree

You used:

DecisionTreeClassifier()

and later:

DecisionTreeRegressor()

A decision tree makes predictions using a sequence of questions.

Imagine:

Is tenure < 12?
       /       \
     Yes       No
     /           \
Is charge > X?   ...

Eventually the tree reaches a leaf containing a prediction.

This is attractive because it is easy to visualize conceptually.

53. Why trees can overfit

A tree can keep splitting the data until it creates extremely specific rules.

For example:

If feature A > 10
and feature B < 4
and feature C = 7
and feature D > 92
...

Eventually the tree may memorize training examples.

That is why you later tried:

max_depth=6
min_samples_split=10
min_samples_leaf=5

These are ways of controlling tree complexity.

54. Random Forest

You used:

RandomForestRegressor(
    n_estimators=200,
    max_depth=8,
    min_samples_leaf=5,
    random_state=42
)

A random forest combines many decision trees.

Instead of relying on one tree:

Tree 1 ─┐
Tree 2 ─┤
Tree 3 ─┤
Tree 4 ─┤ → combined prediction
...     ┘

The idea is that many different trees can produce a more robust prediction than a single tree.

This is an example of an ensemble method.

55. Bagging

You used:

BaggingRegressor(
    n_estimators=100,
    max_samples=0.7,
    max_features=0.7
)

Bagging means combining models trained on different samples of the data.

The overall idea:

Dataset
  ↓
Different samples
  ↓
Multiple models
  ↓
Combine predictions

The goal is often to reduce variance and make predictions more stable.

Random Forest can be viewed as a specialized tree-based ensemble that adds additional randomness in how trees are built.

56. AdaBoost

You used:

AdaBoostRegressor()

AdaBoost means:

Adaptive Boosting

Instead of training many independent models and simply averaging them, boosting builds models sequentially.

Conceptually:

Model 1
  ↓
find mistakes
  ↓
Model 2 focuses more on difficult cases
  ↓
Model 3 focuses further
  ↓
combine models

The key idea:

Later models try to improve on the weaknesses of earlier models.

57. K-Nearest Neighbors (KNN)

You used:

KNeighborsRegressor()

KNN predicts using nearby examples.

Imagine a new customer.

The algorithm asks:

"Which existing customers are most similar to this customer?"

Then it uses those neighbors to estimate the output.

For regression, the prediction is commonly based on the neighbors' target values.

58. Why KNN needs scaling

KNN relies on distance.

Imagine:

Income:   0–100,000
Age:      18–80

If raw values are used, income can dominate the distance calculation.

Scaling puts features onto comparable scales.

That is why you experimented with:

StandardScaler()

59. StandardScaler

You used:

scaler = StandardScaler()

X_train_scaled = scaler.fit_transform(X_train_log)
X_test_scaled = scaler.transform(X_test_log)

This is an important pattern.

Notice:

scaler.fit_transform(X_train)

but:

scaler.transform(X_test)

Why?

The scaler learns the training data's statistics.

Then the exact same transformation is applied to the test data.

You should not independently fit the scaler on the test set.

Otherwise information from the test set leaks into preprocessing.

This idea is called avoiding:

Data leakage

60. Data leakage

Data leakage happens when information that should not be available to the model during training accidentally influences the model.

Example:

Train preprocessing
     ↓
uses information from test data
     ↓
test is no longer truly unseen

That can make evaluation look better than real-world performance.

The correct principle is:

Fit preprocessing steps using training data, then apply them to validation/test/new data.

A production ML pipeline should ideally bundle preprocessing and modeling together.

61. Linear Regression

You used:

LinearRegression()

for Customer Lifetime Value.

Linear regression tries to model a relationship like:

Y = b0 + b1X1 + b2X2 + ...

For one feature:

Y = intercept + slope × X

The goal is to find coefficients that produce predictions close to the observed values.

62. Regression score: what does `.score()` mean?

You used:

model.score(X_train, Y_train)
model.score(X_test, Y_test)

For the scikit-learn regression models you used, .score() generally returns R², the coefficient of determination.

Very roughly:

R² = 1

means the model explains the observed variation perfectly on that dataset.

R² = 0

means the model is no better than a simple baseline based on predicting the mean target.

R² can also be negative on unseen data.

Important:

R² is not the same thing as accuracy.

For regression, do not say:

"My regression model has 90% accuracy"

just because R² is 0.90.

Instead say:

"The model achieved an R² of 0.90."

63. Classification accuracy

You used:

accuracy_score(Y_test, y_test_pred)

Accuracy is:

correct predictions
-------------------
total predictions

For example:

90 correct
100 total

gives:

90% accuracy

Accuracy is easy to understand.

But it can be misleading when classes are highly imbalanced.

64. Confusion matrix

Your later notes/workflow refer to concepts such as:

TP
TN
FP
FN

A confusion matrix organizes classification predictions.

                    Actual
                 No       Yes
Predicted No     TN       FN
Predicted Yes    FP       TP

Meaning:

True Positive

Model predicted positive and it was positive.

True Negative

Model predicted negative and it was negative.

False Positive

Model predicted positive but it was negative.

False Negative

Model predicted negative but it was positive.

This is much more informative than accuracy alone when the cost of errors differs.

65. Precision and Recall

For classification:

Precision

Of everything the model predicted as positive:

How many were actually positive?

Precision = TP / (TP + FP)

Recall

Of everything that was actually positive:

How many did the model find?

Recall = TP / (TP + FN)

The right metric depends on the problem.

For churn prediction, for example, missing a customer who is about to churn may be more important than contacting an extra customer who would have stayed.

66. Your Telco Customer Churn project

This is probably the clearest classification example in your notebooks.

The business question is:

Can we predict whether a telecom customer will churn?

The target is:

Y = df["Churn"]

Features are:

X = df.drop("Churn", axis=1)

The workflow becomes:

Customer data
     ↓
Clean data
     ↓
Explore data
     ↓
Encode categories
     ↓
Split train/test
     ↓
Train classifier
     ↓
Predict churn
     ↓
Evaluate

67. Telco: removing customerID

You did:

df = df.drop(columns=["customerID"])

This is reasonable because the ID is an identifier rather than a customer characteristic.

You also commented that it could be kept separately if you later wanted to map predictions back to customers.

That is a good practical idea.

In production:

customerID
   ↓
keep separately for business reporting

but:

customerID
   X
do not necessarily use as a model feature

68. Telco: fixing TotalCharges

You used:

df["TotalCharges"] = pd.to_numeric(
    df["TotalCharges"],
    errors="coerce"
)

errors="coerce" means invalid values are converted to NaN.

Then you used:

fillna(0)

in one notebook and median imputation in another.

These are two different decisions.

The important lesson is:

Missing-value handling should be based on the meaning of the data, not simply on whichever replacement is easiest.

For example, if missing TotalCharges occurs because a customer has zero tenure and has just joined, then filling with zero may have a meaningful business interpretation.

If missingness represents an unknown measurement, median imputation may be more appropriate.

69. Telco: the first modeling version

The earlier Telco notebook used:

LabelEncoder()

for many categorical columns.

This is okay as a learning exercise for understanding encoding, but it is not ideal for every categorical variable.

A better version in your later notebook used:

pd.get_dummies(
    df,
    columns=multi_cols,
    drop_first=True
)

That is a meaningful improvement.

70. Telco: your improved preprocessing pipeline

The later notebook did something closer to:

Raw data
   ↓
Drop customerID
   ↓
Convert TotalCharges
   ↓
Encode binary columns
   ↓
One-hot encode multi-category columns
   ↓
Separate X and Y
   ↓
Scale X
   ↓
Train Logistic Regression
   ↓
Create Gradio interface

This is much closer to a real ML application.

71. Gradio: turning a model into an application

You used:

import gradio as gr

and built a UI.

This is an important step because it changes the project from:

Notebook experiment

to:

User input
    ↓
Preprocessing
    ↓
Model
    ↓
Prediction
    ↓
Human-readable result

For example:

Gender: Female
Tenure: 12 months
Contract: Month-to-month
Monthly charges: $70
...
          ↓
    Predict Churn
          ↓
High Churn Risk
Probability: ...

This demonstrates an important ML engineering concept:

A model is useful only when it can be integrated into a workflow where people or systems can use its predictions.

72. Very important: preprocessing must match training

Your Gradio code contains a good lesson.

During training you:

encode the categorical values
create dummy columns
scale the features
train the model

During prediction you must do the same thing:

raw user input
    ↓
same encoding
    ↓
same columns
    ↓
same scaling
    ↓
model

If the training data has:

feature_1
feature_2
feature_3
feature_4

but the application sends:

feature_1
feature_3
feature_4

the model cannot interpret the input correctly.

Your use of:

input_df.reindex(
    columns=feature_columns,
    fill_value=0
)

is intended to make the input columns match the training feature structure.

That is an important practical idea.

73. A stronger production pattern: Pipeline

A cleaner production approach is often:

Pipeline([
    ("preprocessor", ...),
    ("model", LogisticRegression())
])

Then preprocessing and prediction are tied together.

This reduces the risk of accidentally applying different transformations during training and inference.

Your notebook demonstrates the concept manually, which is useful for learning.

74. Your Auto Insurance / Customer Lifetime Value project

The 13-Aug.ipynb notebook moves into regression.

The target is:

Y = df["Customer Lifetime Value"]

This changes the question from:

Will this customer churn?

to:

What numerical Customer Lifetime Value should we predict?

Therefore:

Classification → Churn
Regression     → Customer Lifetime Value

75. Insurance dataset: initial exploration

You loaded:

df = pd.read_csv("AutoInsurance.csv")

Then inspected:

df.info()
df.shape
df.describe()

You also removed:

Customer
Effective To Date

from the modeling DataFrame.

Again, the purpose is to separate identifiers / dates that were not being used in the current modeling approach.

However, an important ML lesson is:

Dates are not automatically useless.

A date may contain useful information such as:

month
day of week
season
year
time since an event

A stronger project could engineer useful date features instead of simply dropping every date column.

76. Insurance EDA

You explored:

sns.countplot(x="State", data=df)
sns.boxplot(y=df["Income"])
sns.boxplot(
    x="Coverage",
    y="Customer Lifetime Value",
    data=df
)
sns.scatterplot(
    x="Income",
    y="Customer Lifetime Value",
    data=df
)
df["Income"].hist()

This is a good progression:

Categorical distribution
        ↓
Numerical distribution
        ↓
Numerical vs categorical
        ↓
Numerical vs numerical

You are gradually asking:

"What might explain the target?"

77. Automatically finding numerical and categorical columns

You used:

numerical_cols = df.select_dtypes(
    include=["int64", "float64"]
).columns

categorical_cols = df.select_dtypes(
    include=["object"]
).columns

This is useful because instead of manually listing every column, Python can identify columns based on data type.

Then you looped through them to create graphs.

This is the beginning of writing reusable data-analysis code.

78. Automated boxplots

You created boxplots for all numerical columns.

This is useful because you can quickly inspect:

distributions
median
spread
potential outliers

However, remember that a large collection of graphs can become overwhelming.

The goal should eventually move from:

"Let's plot everything."

to:

"Let's plot the variables that help answer our question."

That is a sign of growing from beginner EDA toward professional analysis.

79. Binning numerical variables for boxplots

You used:

pd.cut(
    df[col],
    bins=5
)

This divides a numerical variable into five intervals.

Then you compare CLV across those ranges.

Conceptually:

Income range 1 → CLV distribution
Income range 2 → CLV distribution
Income range 3 → CLV distribution
...

This is useful for visualization because a boxplot expects categories on one axis.

But the number and boundaries of bins can affect the story you see.

So binning is mainly a visualization technique here, not necessarily something you should automatically use as a model feature.

80. Comparing many regression algorithms

One of the biggest learning steps in your final notebook was trying multiple regressors:

Linear Regression
KNN Regressor
SVR
Decision Tree Regressor
Bagging Regressor
AdaBoost Regressor
Random Forest Regressor

This is valuable because it teaches a core ML lesson:

Different algorithms make different assumptions and capture different types of patterns.

There is no universally best algorithm.

81. Linear Regression vs tree-based models

Linear Regression assumes a relationship that can be represented through a linear combination of features.

Tree-based methods can represent more complex non-linear relationships.

For example:

Linear model:

Y
│       /
│     /
│   /
│ /
└──────── X

A tree-based model can approximate more irregular patterns:

Y
│    ┌────
│    │
│ ───┘
│
└──────── X

This is why trying multiple model families can be useful.

82. Why model comparison matters

Suppose you obtain:

Model             Train R²    Test R²
Linear Regression   0.70       0.68
KNN                 0.90       0.62
Decision Tree       0.99       0.55
Random Forest       0.91       0.78

You should not automatically choose the model with the highest training score.

The more important question is:

Which model generalizes well to unseen data?

Here Random Forest would be more interesting than the tree with 0.99 training R².

83. Log transformation of Customer Lifetime Value

You later used:

Y_log = np.log1p(Y)

Why?

Because some target variables are heavily skewed.

Imagine:

Most CLV values: relatively small
A few CLV values: extremely large

The model may have difficulty fitting such a distribution.

A log transformation compresses large values.

Conceptually:

Original:
1
10
100
1000
10000

Log scale:
small differences between large values

This can make the target distribution easier for some models to learn.

84. Reversing the log transformation

After predicting:

predictions_log = model.predict(X)

you used:

predictions_actual = np.expm1(predictions_log)

This is important.

If:

Y_log = log(1 + Y)

then:

Y = exp(Y_log) - 1

np.expm1() performs:

exp(x) - 1

So:

log target
   ↓
model
   ↓
log prediction
   ↓
expm1
   ↓
original target scale

85. An important issue in the notebook: split consistency

In your log-transform section, you did:

Y_log = np.log1p(Y)

X_train_log, X_test_log, Y_train_log, Y_test_log = train_test_split(
    X,
    Y_log,
    test_size=0.3,
    random_state=42
)

This is okay as a new split, but it means your earlier train/test split and later train/test split are not necessarily the same.

For a clean experiment, define the split once and reuse it consistently.

For example:

X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.3,
    random_state=42
)

y_train_log = np.log1p(y_train)
y_test_log = np.log1p(y_test)

This makes comparisons easier to reason about.

86. Another important issue: predicting on the full dataset

You later did:

predictions_log = model_rf.predict(X)

and saved predictions for the full dataset.

That can be useful for producing a business-facing prediction file.

But these are not unbiased test predictions.

Why?

Because X includes training observations the model already saw.

So:

Predictions on X_test
→ useful for evaluating unseen data

Predictions on X
→ useful for generating predictions for all records,
   but not for measuring generalization

This distinction is very important.

87. Hyperparameters

Later you changed models from:

DecisionTreeRegressor()

to:

DecisionTreeRegressor(
    max_depth=6,
    min_samples_split=10,
    min_samples_leaf=5
)

These settings are called hyperparameters.

They are chosen by us before training.

The model learns its internal parameters from the training data.

Parameters

Learned by the algorithm.

Hyperparameters

Set by us.

This distinction is fundamental.

88. Understanding your tree hyperparameters

`max_depth`

Controls how deep the tree can grow.

Smaller:

simpler tree
less overfitting

Larger:

more complex tree
greater overfitting risk

`min_samples_split`

Minimum number of samples required before a node can be split.

Higher values make the tree more conservative.

`min_samples_leaf`

Minimum number of samples allowed in a leaf.

Higher values prevent extremely tiny leaves.

89. Random Forest hyperparameters

You used:

RandomForestRegressor(
    n_estimators=200,
    max_depth=8,
    min_samples_leaf=5,
    random_state=42
)

`n_estimators=200`

Build 200 trees.

More trees can make the ensemble more stable, though computation increases.

`max_depth=8`

Limits tree complexity.

`min_samples_leaf=5`

Prevents leaves from becoming too specific.

`random_state=42`

Makes the experiment reproducible.

90. Bagging hyperparameters

You used:

BaggingRegressor(
    n_estimators=100,
    max_samples=0.7,
    max_features=0.7,
    random_state=42
)

This means, conceptually:

build many estimators
each estimator sees a sample of the training rows
each estimator can use a subset of features
combine their predictions

Again, the goal is to create a robust ensemble rather than rely on one model.

91. Scaling experiment in your notebook

You used:

StandardScaler()

and compared KNN/SVR.

This is a very useful experiment because it demonstrates:

Preprocessing requirements depend on the algorithm.

Tree-based algorithms generally do not need feature scaling in the same way that distance- or margin-based methods do.

KNN and SVM are much more sensitive to feature scale.

92. A subtle issue in your scaling experiment

You created:

X_train_scaled
X_test_scaled

but your KNN code first trained on:

X_train_log

before later looping over scaled KNN.

That is useful as experimentation, but for a clean article you should present the comparison more systematically:

KNN without scaling
        vs
KNN with scaling

and explicitly explain why the result changed.

For SVR, you correctly used the scaled data in the shown section.

93. The complete mental model

After these five days, the entire workflow can be remembered as:

             REAL-WORLD QUESTION
                     ↓
             Define the target
                     ↓
              Collect data
                     ↓
              Understand data
                     ↓
          ┌──────────┴──────────┐
          ↓                     ↓
      Numerical             Categorical
          ↓                     ↓
      distributions        category counts
          └──────────┬──────────┘
                     ↓
                    EDA
                     ↓
              Clean the data
                     ↓
          Handle missing values
                     ↓
             Encode categories
                     ↓
          Select useful features
                     ↓
              Split train/test
                     ↓
          Scale when appropriate
                     ↓
              Train baseline
                     ↓
          Evaluate on test data
                     ↓
             Try other models
                     ↓
         Tune hyperparameters
                     ↓
       Select based on validation
                     ↓
             Final evaluation
                     ↓
              Make predictions
                     ↓
       Integrate into an application

94. What each notebook taught you

Day 1 — Auto MPG

Main lessons:

load a dataset
inspect rows and columns
understand shape and data types
inspect summary statistics
identify categorical/text columns
detect messy numerical values
visualize distributions
use countplots
use histograms
use pairplots
understand missing values
convert "?" to NaN
convert text to numerical data
use median imputation

The biggest lesson:

Before modeling, understand and clean your data.

Day 2 — Telco Customer Churn

Main lessons:

identify a classification target
remove an identifier
handle TotalCharges
visualize class balance
compare categories against churn
use boxplots and histograms
use pairplots
inspect correlations
encode categorical data
split into training/testing data
train Logistic Regression
train SVC
train Decision Tree
compare train/test performance

The biggest lesson:

A classification model predicts categories, but the quality of the prediction depends heavily on preprocessing and evaluation.

Day 3 — Improving the Telco project

Main lessons:

one-hot encode multi-category variables
use random_state
use stratify
scale features
train Logistic Regression
build a prediction interface with Gradio
reproduce preprocessing at inference time
output a human-readable prediction

The biggest lesson:

An ML model is only one part of an ML application.

Day 4/5 — Customer Lifetime Value

Main lessons:

formulate a regression problem
inspect numerical/categorical relationships
automate EDA
encode categories
split train/test
compare multiple regressors
understand R²
try target transformation
use log transformation
reverse the transformation
experiment with scaling
tune hyperparameters
use ensemble models
generate prediction files

The biggest lesson:

Different models capture different patterns, and model selection should be based on performance on unseen data—not training performance alone.

95. What I would change before publishing the project as "production-quality"

Your notebooks are excellent learning material because they show experimentation. For a public article, however, it is worth separating:

Learning experiments

from:

Recommended ML workflow

Here are the most important improvements.

Improvement 1 — Use clear train/test terminology

Prefer:

X_train, X_test, y_train, y_test

rather than mixing Y and y.

Python convention generally uses lowercase y.

Improvement 2 — Make experiments reproducible

Use:

random_state=42

consistently when appropriate.

Improvement 3 — Use stratification for classification

For churn:

train_test_split(
    X,
    y,
    test_size=0.3,
    random_state=42,
    stratify=y
)

Improvement 4 — Avoid LabelEncoder for nominal multi-class input features

Use one-hot encoding for categories such as:

PaymentMethod
Contract
InternetService
VehicleClass

Label encoding is much more naturally suited to targets or truly ordinal categories.

Improvement 5 — Fit preprocessing only on training data

For example:

Training data
    ↓
fit scaler
    ↓
transform training

Test data
    ↓
transform using existing scaler

Do not independently fit preprocessing on the test data.

Improvement 6 — Use proper evaluation metrics

For classification, consider:

Accuracy
Precision
Recall
F1
Confusion Matrix
ROC-AUC
PR-AUC

For regression:

R²
MAE
MSE
RMSE

Which metrics matter depends on the business problem.

Improvement 7 — Don't compare only training scores

Always look at:

Training performance
+
Validation/test performance

A very high training score with much lower test performance can indicate overfitting.

Improvement 8 — Keep a validation strategy for model selection

A single test set should ideally be kept for final evaluation.

During experimentation, use cross-validation or a validation set to choose models/hyperparameters.

Conceptually:

Training data
   ↓
Cross-validation / validation
   ↓
Choose model
   ↓
Final test set
   ↓
Final unbiased-ish estimate

96. A very important distinction: EDA vs ML

EDA asks:

"What does my data look like?"

Machine learning asks:

"Can I learn a useful relationship that lets me make predictions on new data?"

These overlap, but they are not the same.

For example:

sns.scatterplot(...)

does not build a model.

It helps you understand the data before deciding how to model it.

97. Another important distinction: correlation vs prediction

A strong correlation can be useful.

But:

high correlation

does not automatically mean:

excellent predictive model

Prediction depends on:

multiple features
data quality
model type
noise
generalization
preprocessing
evaluation

So don't judge an ML problem from one graph.

98. Another important distinction: model vs algorithm

It is common to hear:

"I trained a model."

The algorithm is the learning procedure.

For example:

Random Forest = algorithm/model family

After training:

model_rf.fit(X_train, y_train)

you have a fitted model containing learned information.

A useful mental model:

Algorithm
   +
Training data
   ↓
Fitted model
   ↓
Predictions

99. What "learning" actually means

This is probably the most important concept to understand.

When you run:

model.fit(X_train, y_train)

the model is not magically understanding the world.

It is optimizing internal parameters so that its predictions match the training examples according to the algorithm's objective.

For example, linear regression learns coefficients.

A tree learns split rules.

A neural network learns weights.

So:

Training means finding model parameters that make the model perform well according to a defined objective.

100. What happens during prediction?

Once training is complete:

prediction = model.predict(X_test)

the model does not learn again from the test examples.

It applies what it learned during training.

Conceptually:

Training:

X_train + y_train
        ↓
      model.fit()
        ↓
   learned model

Prediction:

X_test
   ↓
learned model
   ↓
prediction

101. The biggest beginner misconception to avoid

Do not think:

"I used Random Forest, so I did machine learning."

The real ML skill is being able to explain:

What problem are you solving?
What is the target?
What features are available?
What does the data look like?
What problems did you find?
How did you clean them?
How did you encode the data?
Why did you choose the algorithm?
How did you evaluate it?
Does it generalize?
What would you improve?

If you can explain these, you are learning ML—not just memorizing scikit-learn commands.

102. A simple explanation of your five-day journey

If someone asks:

"What did you actually learn in five days?"

A strong answer is:

I learned that machine learning is not just about selecting an algorithm. I started by understanding datasets using Pandas, cleaning missing and incorrectly formatted values, and using visualization to understand distributions and relationships. Then I learned the difference between classification and regression, converted categorical data into numerical representations, split data into training and testing sets, trained several models using scikit-learn, and compared their performance on unseen data. Finally, I experimented with scaling, transformations, ensemble models, hyperparameters, and even built a simple Gradio interface around a churn model.

That is a much stronger story than:

"I learned Logistic Regression, SVM and Random Forest."

103. Suggested article structure for dev.to

For your final public article, I recommend this structure:

Title

1. Why I started learning ML
2. What I thought ML was vs what I learned
3. AI vs ML
4. Types of ML
5. My five-day learning roadmap
6. Day 1 — Understanding and cleaning data
7. EDA and why graphs matter
8. Day 2 — Classification with customer churn
9. Encoding categorical data
10. Train/test split
11. Logistic Regression, SVM and Decision Trees
12. How I evaluated classification models
13. Day 3 — Turning the model into an application
14. Day 4/5 — Regression and Customer Lifetime Value
15. Linear Regression vs KNN vs SVR vs Trees vs Ensembles
16. Log transformation
17. Scaling
18. Hyperparameters
19. What I got wrong / what I would improve
20. What I learned
21. What I plan to learn next

This will make the article feel like a journey, rather than a textbook.

104. Recommended article narrative

The most interesting story is:

"I started with almost no ML knowledge."

        ↓

"I learned that data preparation matters."

        ↓

"I discovered that graphs help me ask questions
before training models."

        ↓

"I learned classification."

        ↓

"I learned regression."

        ↓

"I tried different algorithms and realized
there is no single best model."

        ↓

"I learned that a high training score
doesn't necessarily mean a good model."

        ↓

"I turned one model into a small application."

        ↓

"I now understand the basic ML workflow
and know what I need to learn next."

That is a much more authentic beginner-to-ML story.

105. What to learn next

Based on what your notebooks already cover, I would not recommend immediately jumping into deep learning.

First strengthen these foundations:

1. Statistics

Learn:

mean
median
variance
standard deviation
distributions
probability
correlation
conditional probability

2. ML evaluation

Learn:

confusion matrix
precision
recall
F1
ROC-AUC
MAE
RMSE
R²
cross-validation

3. Feature engineering

Learn:

handling dates
categorical features
transformations
interactions
outliers
missing values

4. Model selection

Learn:

cross-validation
hyperparameter tuning
GridSearchCV
RandomizedSearchCV
pipelines

5. ML fundamentals

Learn:

bias vs variance
underfitting
overfitting
regularization
data leakage
feature importance

Then move toward:

Classical ML
     ↓
Feature engineering
     ↓
Model selection
     ↓
Cross-validation
     ↓
ML pipelines
     ↓
Deployment
     ↓
Deep Learning
     ↓
Generative AI / LLMs

106. The one-page cheat sheet

Problem type

Predict a category → Classification
Predict a number   → Regression
Find hidden groups → Clustering
Learn through reward → Reinforcement Learning

Data

Rows       → observations
Columns    → variables/features
X          → input features
y          → target

First inspection

df.head()
df.shape
df.info()
df.describe()
df.sample()
df.isnull().sum()

Visualization

Countplot   → categories
Histogram   → numerical distribution
Boxplot     → spread/outliers
Scatterplot → two numerical variables
Pairplot    → many numerical relationships
Heatmap     → correlation matrix

Preparation

Missing values
Data types
Categorical encoding
Feature selection
Scaling when needed

Training

model.fit(X_train, y_train)

Prediction

model.predict(X_test)

Evaluation

Classification:

Accuracy
Precision
Recall
F1
Confusion Matrix
ROC-AUC

Regression:

R²
MAE
MSE
RMSE

Core warning signs

Training score >> Test score
        ↓
Possible overfitting

Test information used during preprocessing
        ↓
Possible data leakage

Identifier used as feature
        ↓
Possible meaningless pattern

Categorical values converted to arbitrary numbers
        ↓
Possible false ordering

107. Final takeaway

The biggest lesson from these five days is not a particular Python library or algorithm.

It is the workflow:

Understand the problem → understand the data → clean the data → explore the data → prepare features → train → evaluate → improve → deploy.

The algorithms are tools inside that workflow.

You have already touched many of the important building blocks:

Pandas
NumPy
Matplotlib
Seaborn
Scikit-learn
data cleaning
EDA
classification
regression
encoding
train/test split
scaling
model evaluation
hyperparameters
ensemble models
target transformation
Gradio

The next step is not to memorize more algorithms.

The next step is to become comfortable answering:

Why am I doing this step?

If you can answer that question for every important line in your notebook, you are moving from "following an ML tutorial" to actually understanding machine learning.

Appendix A — Notebook-by-notebook step map

`11_Aug_mpg.ipynb`

Notebook step	Why it was done	ML concept
Import NumPy/Pandas/Matplotlib/Seaborn	Load tools	Python ML stack
`read_csv()`	Load data	Data ingestion
`nunique()`	Understand uniqueness	Data exploration
`head()`	Inspect examples	EDA
`shape`	Understand dataset size	EDA
`describe()`	Summary statistics	EDA
`info()`	Types/non-null values	Data quality
`countplot()`	Inspect categorical distribution	Visualization
`hist()`	Inspect numerical distributions	Visualization
`isnull()`	Detect actual missing values	Data cleaning
`pairplot()`	Explore relationships	EDA
`sample()`	Inspect random examples	Data quality
Identify `car name`	Recognize identifier-like field	Feature selection
Replace `?`	Convert fake missing values	Data cleaning
`to_numeric()`	Make horsepower numerical	Data preparation
Median imputation	Fill missing horsepower	Missing-value handling
Train/test split	Separate learning/evaluation data	Generalization
Logistic Regression	Start classification experiment	Supervised learning

Important caveat

The notebook switches from Auto MPG to a loan_prediction.csv dataset near the end. That appears to be a copied/reused modeling experiment rather than a continuation of the Auto MPG workflow. For a polished article, present these as separate experiments rather than one continuous Auto MPG pipeline.

`12th-Aug-Telco-Customer-churn.ipynb`

Step	Purpose
Load Telco CSV	Get customer data
`head()`	Inspect records
`shape`	Understand size
`describe()`	Statistical overview
Drop `customerID`	Remove identifier from features
`info()`	Check data types
`sample()`	Inspect random rows
Countplot of Churn	Check class distribution
Countplot of gender	Explore category distribution
Histograms	Explore numerical distributions
Pairplot	Explore relationships
Convert TotalCharges	Fix numeric type
Fill missing TotalCharges	Handle missing data
LabelEncoder	Convert categories to numbers
Train/test split	Evaluate on unseen data
Logistic Regression	Classification model
SVC	Alternative classifier
Decision Tree	Alternative classifier
Gradio	Turn model into interactive app

Important caveat

The notebook uses LabelEncoder on many categorical input columns. This is useful for learning encoding, but one-hot encoding is generally safer for nominal multi-category features.

`12-Aug-2.ipynb`

This notebook is a more developed version of the Telco project.

Important improvements include:

dropping customerID
converting TotalCharges
one-hot encoding multi-category variables
using random_state=42
using stratify=Y
scaling features
training Logistic Regression
building a Gradio prediction interface
preserving training feature columns
applying the same preprocessing to new input

This is the notebook that most clearly demonstrates the transition from:

ML experiment

to:

small ML application

`13-Aug.ipynb`

Main steps:

Load AutoInsurance
       ↓
Inspect dataset
       ↓
Drop Customer / date columns
       ↓
EDA
       ↓
Identify numerical/categorical columns
       ↓
Generate many visualizations
       ↓
Encode categorical variables
       ↓
Set CLV as target
       ↓
Train/test split
       ↓
Try multiple regressors
       ↓
Try log target
       ↓
Try scaling
       ↓
Tune tree/ensemble hyperparameters
       ↓
Generate predictions
       ↓
Save prediction CSVs

This notebook demonstrates the transition from classification to regression and from a single baseline model to model comparison and tuning.

Appendix B — A cleaner ML template to remember

# 1. Load
df = pd.read_csv("data.csv")

# 2. Understand
df.head()
df.shape
df.info()
df.describe()

# 3. Clean
# handle missing values
# fix data types
# remove inappropriate identifiers

# 4. Explore
# histograms
# boxplots
# countplots
# scatterplots
# correlation

# 5. Prepare
X = df.drop("target", axis=1)
y = df["target"]

# 6. Split
X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.2,
    random_state=42
)

# 7. Preprocess
# encoding / scaling

# 8. Train
model.fit(X_train, y_train)

# 9. Predict
predictions = model.predict(X_test)

# 10. Evaluate
# choose metrics appropriate to the problem

# 11. Compare / improve
# try another model
# tune hyperparameters
# use cross-validation

# 12. Final model
# train using the selected approach

# 13. Predict new data

Closing thought for the article

Five days ago, an ML notebook could easily look like a collection of unfamiliar commands.

Now there is a structure behind those commands.

head() is not just a command — it is a way to understand the data.

isnull() is not just a command — it is part of data quality.

A histogram is not just a graph — it tells you about a distribution.

train_test_split() is not just boilerplate — it protects your evaluation from simply measuring memorization.

fit() is not magic — it is the learning stage.

predict() is the model applying what it learned.

And model comparison is not about finding the fanciest algorithm — it is about finding an approach that generalizes well to data the model has never seen.

That, more than anything else, is what I took away from my first five days of machine learning.

AI & ML Foundations: Cleaning Data, Building Regression & Classification Models in Python

yash-nigam — Mon, 31 Aug 2026 16:49:07 +0000

My 5-Day Journey into Machine Learning: From Data Cleaning to My First ML Models

Learning goal: This is not a collection of code snippets to memorize. It is a beginner-friendly guide to understanding why each step in a typical machine-learning workflow exists, what problem it solves, and how the pieces fit together.

A note about the source material

This guide was built primarily from the four Jupyter notebooks you provided:

11_Aug_mpg.ipynb
12-Aug-2.ipynb
12th-Aug-Telco-Customer-churn.ipynb
13-Aug.ipynb
123

1. Overview: What did I actually learn in these five days?

At first glance, machine learning can look like this:

Load dataset
    ↓
Clean data
    ↓
Create graphs
    ↓
Train model
    ↓
Check accuracy
    ↓
Done!

But that misses the most important part.

The real ML workflow is closer to:

Business / real-world question
        ↓
Define what we want to predict
        ↓
Understand the data
        ↓
Clean and prepare the data
        ↓
Explore the data visually
        ↓
Choose useful features
        ↓
Convert data into numbers
        ↓
Split into training and testing data
        ↓
Choose an appropriate ML algorithm
        ↓
Train the model
        ↓
Evaluate it on unseen data
        ↓
Compare models
        ↓
Improve preprocessing / model settings
        ↓
Use the model to make predictions

The important insight is:

Machine learning is not mainly about choosing an algorithm. It is about turning a real-world question and messy data into a reliable prediction process.

Your notebooks actually demonstrate most of this workflow.

2. What is Artificial Intelligence?

Artificial Intelligence (AI) is the broad idea of building systems that can perform tasks that normally require some form of human intelligence.

Examples:

recognizing an image
understanding language
recommending a movie
detecting fraud
predicting whether a customer will leave
predicting the price/value of something
generating text or images

A useful mental model is:

Artificial Intelligence
│
├── Machine Learning
│   │
│   ├── Supervised Learning
│   ├── Unsupervised Learning
│   └── Reinforcement Learning
│
├── Deep Learning
│
└── Generative AI

This is simplified because these areas overlap, but it is a useful beginner mental model.

3. What is Machine Learning?

Machine learning is a way of building systems where the computer learns patterns from data instead of us manually writing every rule.

Imagine we want to predict whether a telecom customer will churn.

A traditional rule-based approach might look like:

IF customer has a month-to-month contract
AND monthly charges are high
AND tenure is low
THEN churn = yes

The problem is that we have to invent those rules ourselves.

With machine learning, we give the algorithm historical examples:

Customer information → Actual churn result

Customer A → No
Customer B → Yes
Customer C → No
Customer D → Yes
...

The algorithm tries to learn the relationship between the input information and the known outcome.

Then we can give it a new customer:

New customer information
        ↓
      Model
        ↓
Predicted churn

That is the core idea of supervised machine learning.

4. The most important ML vocabulary

Before learning algorithms, understand these words.

4.1 Dataset

A dataset is a collection of examples.

In your notebooks, a dataset is loaded into a Pandas DataFrame.

For example:

df = pd.read_csv("Telco_Customer_Churn.csv")

Think of the DataFrame as a spreadsheet.

4.2 Row / observation / sample

Each row represents one example.

For the Telco dataset:

one row = one customer

For the Auto MPG dataset:

one row = one car

For the insurance dataset:

one row = one customer/policy record

These words are often used interchangeably:

row
observation
sample
instance

4.3 Feature

A feature is an input variable that the model can use to make a prediction.

For customer churn:

tenure
MonthlyCharges
Contract
InternetService
PaymentMethod
...

These are features.

A useful question to ask is:

"What information is available to the model before it makes its prediction?"

That information is generally your feature set.

4.4 Target

The target is what we are trying to predict.

Examples from your notebooks:

Telco

Target = Churn

The model predicts:

0 → No churn
1 → Churn

Insurance

Target = Customer Lifetime Value

The model predicts a numerical value.

4.5 X and y

You repeatedly used:

X = df.drop(["Churn"], axis=1)
Y = df["Churn"]

This is one of the most important patterns in supervised ML.

X = inputs/features
Y = target/output

For example:

X
├── tenure
├── MonthlyCharges
├── Contract
├── InternetService
└── ...

Y
└── Churn

Think:

X = what the model knows

Y = what we want the model to learn to predict

5. The three major types of Machine Learning

5.1 Supervised Learning

In supervised learning, we have examples where the correct answer is already known.

Input X → Known answer Y

The model learns the relationship between X and Y.

Your notebooks mainly use supervised learning.

There are two major supervised learning tasks.

Classification

The target is a category.

Examples:

Spam / Not Spam
Churn / No Churn
Fraud / Not Fraud
Disease / No Disease

Your Telco project is a classification problem.

Regression

The target is a numerical value.

Examples:

House price
Temperature
Sales
Customer Lifetime Value
Fuel efficiency

Your Customer Lifetime Value project is a regression problem.

6. Classification vs Regression

This distinction is extremely important.

Classification

Question:

"Which category does this example belong to?"

Example:

Customer → Churn or No Churn

Possible output:

0
1

Common classification algorithms you tried:

Logistic Regression
Support Vector Classifier (SVC)
Decision Tree Classifier

Regression

Question:

"What numerical value should we predict?"

Example:

Customer → Customer Lifetime Value = 6,543.21

Common regression algorithms you tried:

Linear Regression
KNN Regressor
SVR
Decision Tree Regressor
Bagging Regressor
AdaBoost Regressor
Random Forest Regressor

7. Unsupervised Learning

In unsupervised learning, we do not have a known target.

Instead, the algorithm tries to discover structure in the data.

For example:

Customer data
     ↓
Find groups
     ↓
Group 1: high-value customers
Group 2: price-sensitive customers
Group 3: new customers

Common examples:

K-Means clustering
Hierarchical clustering
PCA

You did not build an unsupervised-learning project in these notebooks, but it is important to know where it fits.

8. Reinforcement Learning

Reinforcement learning is different.

An agent interacts with an environment and receives rewards or penalties.

Agent
  ↓ action
Environment
  ↓
Reward / penalty
  ↓
Agent learns

Examples:

game-playing agents
robotics
certain recommendation/control systems

This was not part of your five-day practical work.

9. Different ways to approach an ML problem

A useful way to think about ML is not:

"Which algorithm should I use?"

Instead ask:

Step 1 — What is the question?

For example:

Can I predict whether a telecom customer will churn?

Step 2 — What type of target do I have?

Churn → category → Classification
Customer Lifetime Value → number → Regression

Step 3 — What data is available?

Ask:

What columns exist?
Which are numerical?
Which are categorical?
Are values missing?
Are there strange values?
Are some columns identifiers?
Is the target balanced?

Step 4 — Explore before modeling

Use statistics and graphs.

Step 5 — Prepare the data

Convert it into a form the algorithm can understand.

Step 6 — Establish a baseline

Train a simple model first.

Step 7 — Compare alternatives

Try other appropriate models.

Step 8 — Evaluate on unseen data

A model is useful only if it works beyond the data it memorized.

10. Google Colab: Why did we use it?

Google Colab gives you a cloud-based Jupyter Notebook environment.

Instead of installing everything locally, you can run Python code in the browser.

A notebook combines:

Explanation
+
Python code
+
Output
+
Graphs

This makes it very useful for learning ML.

Your notebooks use paths such as:

/content/auto-mpg.csv

That is typical of a Colab environment.

A normal learning workflow is:

Upload dataset
      ↓
Open notebook
      ↓
Run cells
      ↓
Inspect output
      ↓
Change code
      ↓
Run again

11. The Python libraries you used

11.1 NumPy

import numpy as np

NumPy provides numerical operations and arrays.

You used it for things such as:

np.nan
np.log1p()
np.expm1()

`np.nan`

Represents a missing numerical value.

`np.log1p(x)`

Calculates:

log(1 + x)

It is useful when a target is heavily skewed.

`np.expm1(x)`

Reverses log1p:

exp(x) - 1

You used this later to convert log predictions back to the original CLV scale.

12. Pandas

import pandas as pd

Pandas is one of the most important Python libraries for data work.

Think of a Pandas DataFrame as a spreadsheet that Python can manipulate.

You used it to:

load CSV files
inspect data
remove columns
convert data types
detect missing values
fill missing values
select X and Y
create encoded columns
save predictions

Example:

df = pd.read_csv("Telco_Customer_Churn.csv")

13. Matplotlib

import matplotlib.pyplot as plt

Matplotlib is a general-purpose plotting library.

It provides the foundation for many Python visualizations.

You used it for:

plt.figure()
plt.title()
plt.show()

14. Seaborn

import seaborn as sns

Seaborn makes statistical visualizations easier to create.

You used:

countplot
boxplot
histplot
pairplot
scatterplot
heatmap

A key lesson:

A graph is not decoration. It is a tool for asking questions about the data.

15. Scikit-learn

Scikit-learn is the main ML library used in your notebooks.

You used it for:

splitting data
preprocessing
encoding
scaling
classification
regression
evaluation

The basic pattern is:

model = SomeModel()
model.fit(X_train, Y_train)
predictions = model.predict(X_test)

This pattern appears again and again in ML.

16. The most important workflow: EDA

EDA means:

Exploratory Data Analysis

EDA is the process of investigating the dataset before building the final model.

Your notebooks use:

df.head()
df.shape
df.describe()
df.info()
df.sample()
df.isnull().sum()

plus graphs.

These are not random commands.

They answer different questions.

17. `df.head()` — What does the data look like?

You used:

df.head()

It displays the first few rows.

Why?

Because before doing anything else, you want to see:

column names
example values
obvious data problems
whether the data loaded correctly

Think of it as opening the box before using what is inside.

18. `df.shape` — How much data do I have?

You used:

df.shape

For Auto MPG:

(398, 9)

That means:

398 rows
9 columns

Why does this matter?

Because dataset size affects:

model choice
computation time
confidence in results
risk of overfitting

19. `df.info()` — What types of data do I have?

You used:

df.info()

This tells you:

column names
number of non-null values
data types
memory usage

This is extremely important.

For example, in Auto MPG:

horsepower → object

At first glance horsepower sounds numerical.

But Pandas sees it as text.

Why?

Because the dataset contains values such as:

A column containing:

cannot be treated as a clean numerical column.

So:

df.info() helps you detect data-type problems before modeling.

20. `df.describe()`

You used:

df.describe()

and sometimes:

df.describe(include="all")

This gives summary statistics.

For numerical data you get things such as:

count
mean
standard deviation
minimum
25th percentile
median
75th percentile
maximum

This helps you understand the scale and distribution of variables.

For example:

minimum
   ↓
25%
   ↓
median
   ↓
75%
   ↓
maximum

A large difference between the 75th percentile and maximum can be a clue that there may be extreme values.

21. `df.sample()`

You used:

df.sample(5)

or similar.

This shows random rows.

Why?

Because the first rows may not represent the whole dataset.

Random samples can expose:

unusual values
formatting issues
unexpected categories
data-entry problems

22. Missing data

You used:

df.isnull().sum()

This asks:

"How many missing values are there in each column?"

Missing data is important because many ML algorithms cannot directly work with missing values.

But there is a subtle lesson from your Auto MPG notebook.

You first checked:

df.isnull().sum()

while horsepower still contained "?".

Pandas did not count "?" as a missing value because "?" is a string, not a true NaN.

That means:

"?" ≠ NaN

So data cleaning sometimes requires identifying fake missing values first.

23. Auto MPG: cleaning horsepower

Your notebook did:

df["horsepower"] = df["horsepower"].replace("?", np.nan)

This converts the placeholder into a real missing value.

Then:

df["horsepower"] = pd.to_numeric(df["horsepower"])

Now Pandas can treat horsepower as numerical data.

This is a very important data-cleaning pattern:

messy text
   ↓
identify invalid placeholder
   ↓
convert to NaN
   ↓
convert column to numeric
   ↓
handle missing values

24. Why use the median?

You calculated:

median_horsepower = df["horsepower"].median()

and then filled missing values:

df["horsepower"] = df["horsepower"].replace(
    np.nan,
    median_horsepower
)

Why median?

Because the median is less affected by extreme values than the mean.

Example:

10, 11, 12, 13, 1000

The mean is pulled heavily upward by 1000.

The median is much more representative of the middle of the data.

A useful rule of thumb:

fairly symmetric data → mean may work well
skewed data / outliers → median is often safer

But remember:

There is no universal rule that "median is always correct."

The best treatment depends on the data and the problem.

25. Dropping an identifier

You considered dropping:

df.drop("car name", axis=1)

and actually dropped customerID in the Telco work.

Why?

An identifier is usually not a meaningful predictive feature.

For example:

customerID = 759832

does not inherently mean the customer is more or less likely to churn.

The ID exists to identify the row, not to describe the customer.

Important distinction:

A column can be useful for identifying a record without being useful for predicting the target.

26. But don't blindly drop columns

This is an important improvement to your original reasoning.

A column should not be dropped merely because it is an object/string.

For example:

Contract
InternetService
PaymentMethod

are categorical strings, but they can contain very useful predictive information.

So the correct question is:

"Does this column contain useful information that is available at prediction time?"

not:

"Is this column numeric?"

27. Understanding your graphs

You created many graphs. The important thing is to understand what question each graph answers.

28. Countplot

Example:

sns.countplot(x="Churn", data=df)

A countplot answers:

"How many observations are in each category?"

For churn:

No churn → number of customers
Churn    → number of customers

This immediately helps you understand class balance.

If the graph looks like:

No Churn ███████████████████
Churn    ███████

the classes are imbalanced.

That matters because a model could achieve high accuracy simply by favoring the majority class.

29. Countplot with `hue`

You used:

sns.countplot(
    x="Contract",
    hue="Churn",
    data=df
)

This answers:

"How does the target category vary across another category?"

For example:

Contract type
     ↓
Churn / No churn

If month-to-month customers show much more churn than two-year customers, that is an important pattern worth investigating.

But remember:

A graph showing association does not automatically prove causation.

30. Histogram

You used:

df.hist(figsize=(10,10))

and:

sns.histplot(df["MonthlyCharges"], kde=True)

A histogram shows the distribution of a numerical variable.

It helps answer:

Where are most values?
Is the data symmetric?
Is it skewed?
Are there multiple groups?
Are there extreme values?

Example:

frequency
   ^
   |       ███
   |     ███████
   |   █████████
   | ███████████
   +-----------------> value

31. Why distributions matter

Suppose a feature looks like:

Most values: 0–100
A few values: 10,000

That may indicate:

genuine outliers
a different population
data-entry errors
a highly skewed distribution

A model may react differently depending on the algorithm.

This is one reason EDA happens before modeling.

32. Boxplot

You used:

sns.boxplot(y=df["Income"])

A boxplot summarizes a distribution.

Conceptually:

       outlier
          •
          |
      ┌───────┐
      │       │
------│  box  │------
      │       │
      └───────┘
          |
       outlier

The box represents the middle portion of the data.

The line inside the box is the median.

Points outside the whiskers may be treated as potential outliers.

Important:

An outlier is not automatically an error.

A high-income customer may be perfectly legitimate.

33. Boxplot: target vs category

You used:

sns.boxplot(
    x="Coverage",
    y="Customer Lifetime Value",
    data=df
)

This is a powerful graph.

It asks:

"How does the distribution of Customer Lifetime Value differ between categories?"

For example:

Coverage A → CLV distribution
Coverage B → CLV distribution
Coverage C → CLV distribution

Compare:

median
spread
outliers
overlap

This can help you identify potentially useful relationships.

34. Scatterplot

You used:

sns.scatterplot(
    x="Income",
    y="Customer Lifetime Value",
    data=df
)

A scatterplot examines the relationship between two numerical variables.

Each dot represents an observation.

It helps answer:

"As X changes, does Y appear to change?"

Possible patterns:

Positive relationship:
  •
    •
      •
        •

Negative relationship:
        •
      •
    •
  •

No obvious relationship:
 •    •
    •
  •      •
     •

The pattern does not have to be a straight line.

35. Pairplot

You used:

sns.pairplot(...)

A pairplot shows many numerical relationships at once.

It is useful for:

spotting correlations
identifying clusters
seeing distributions
finding obvious relationships
detecting possible outliers

The downside is that it becomes difficult to read when there are many columns.

So:

Pairplot is excellent for small-to-medium exploratory datasets, but not something you blindly run on every large dataset.

36. Correlation heatmap

You used:

sns.heatmap(
    df[["tenure", "MonthlyCharges", "TotalCharges"]].corr(),
    annot=True
)

Correlation measures how two numerical variables move together in a linear relationship.

A correlation close to:

+1 → strong positive linear relationship
 0 → little/no linear relationship
-1 → strong negative linear relationship

For example:

tenure ↑
TotalCharges ↑

could result in positive correlation.

But correlation has an important limitation:

Correlation does not prove causation.

Also, zero correlation does not necessarily mean "no relationship" because the relationship might be non-linear.

37. A key lesson from visualization

The graphs answer different questions:

Graph	Main question
Countplot	How many observations are in each category?
Countplot + hue	How does one category vary with another?
Histogram	What does a numerical distribution look like?
Boxplot	What are the median, spread and potential outliers?
Boxplot + category	How does a numerical distribution differ by category?
Scatterplot	How do two numerical variables relate?
Pairplot	What relationships exist among several numerical variables?
Heatmap	How strongly are numerical variables linearly correlated?

The goal is not:

"Create lots of graphs."

The goal is:

"Use the right graph to answer the right question."

38. Encoding: why does ML need it?

Many ML algorithms work with numbers.

But real-world data contains:

Male
Female

Yes
No

Month-to-month
One year
Two year

Fiber optic
DSL
No

The model needs these values represented numerically.

This is called encoding.

39. Label Encoding

You used:

from sklearn.preprocessing import LabelEncoder

and encoded categorical columns.

For a binary column:

No  → 0
Yes → 1

This can be reasonable for a binary variable.

For example:

Churn:
No  → 0
Yes → 1

That makes intuitive sense.

40. Why LabelEncoder can be dangerous for multi-category variables

Suppose:

Contract:
Month-to-month
One year
Two year

Label encoding might produce:

Month-to-month → 0
One year       → 1
Two year       → 2

A numerical model may interpret that as:

Two year > One year > Month-to-month

That ordering may not be appropriate for the algorithm.

The numbers are just codes.

They do not automatically mean that category 2 is "twice" category 1.

For nominal categories, one-hot encoding is usually safer.

41. One-hot encoding

You used:

pd.get_dummies(df, drop_first=True)

This transforms categories into separate binary columns.

For:

InternetService:
DSL
Fiber optic
No

we might create columns such as:

InternetService_Fiber optic
InternetService_No

with the remaining category represented by both values being 0.

This avoids inventing a fake numerical ordering.

42. Why `drop_first=True`?

If a categorical variable has several categories, using all one-hot columns can introduce redundancy in some models.

Dropping one category creates a reference category.

For example:

DSL
Fiber optic
No

could become:

Fiber optic
No

Then:

Fiber optic = 0
No = 0

means the reference category:

DSL

For beginner understanding, the key idea is:

One-hot encoding converts categories into machine-readable indicator variables without pretending the categories have numerical order.

43. Train/test split

You repeatedly used:

train_test_split(
    X,
    Y,
    test_size=0.3
)

This is one of the most important ML concepts.

Suppose we have 1,000 examples.

We might use:

700 → training
300 → testing

The training set is used to learn the model.

The test set is held back to evaluate how the trained model performs on unseen data.

Think of it like an exam:

Training data = practice questions
Test data     = unseen exam questions

44. Why not train and test on the same data?

Because the model could simply memorize the training examples.

Suppose:

Training score = 99%
Test score     = 65%

That is a warning sign.

The model learned the training data extremely well but does not generalize well.

This is called:

Overfitting

45. Underfitting

The opposite can happen.

If the model is too simple:

Training score = 60%
Test score     = 58%

It may not have learned enough from the data.

This is:

Underfitting

A useful mental picture:

Underfitting
Model too simple
       ↓
misses important patterns

Good fit
Learns useful patterns
       ↓
works on unseen data

Overfitting
Model learns noise/details
       ↓
great training performance
poor unseen performance

46. Why `random_state` matters

In one Telco notebook you used:

train_test_split(X, Y, test_size=0.3)

In the improved notebook you used:

random_state=42

A random state makes the split reproducible.

Without it, the random split can change between runs.

That means your score may change.

With:

random_state=42

you can reproduce the same split.

The number 42 is not magical.

Any fixed integer can serve this purpose.

47. Why `stratify=Y` is useful for classification

In your improved Telco notebook you used:

stratify=Y

This helps preserve the class distribution between training and testing data.

For example, if the full dataset contains:

73% No Churn
27% Churn

stratification aims to keep roughly the same proportion in both splits.

This is particularly useful when classes are imbalanced.

48. Logistic Regression

You used:

from sklearn.linear_model import LogisticRegression

model_lr = LogisticRegression()
model_lr.fit(X_train, Y_train)

Despite its name, Logistic Regression is commonly used for classification.

Its goal is to estimate the probability of belonging to a class.

For binary classification:

features
   ↓
Logistic Regression
   ↓
probability
   ↓
class

Example:

Probability of churn = 0.82

The model may classify that customer as:

Churn

depending on its decision threshold.

49. Why Logistic Regression is a good beginner model

It is useful because:

relatively simple
fast
often strong as a baseline
easier to interpret than many complex models
naturally suited to binary classification

A good ML habit is:

Start with a simple baseline before reaching for complex models.

50. Support Vector Machine / SVC

You used:

from sklearn.svm import SVC

model_lr = SVC()

The variable name model_lr is misleading here.

It is actually an SVC model.

SVC means:

Support Vector Classifier

The basic idea is to find a decision boundary that separates classes.

In simple 2D data:

Class A: ● ● ●

--------- decision boundary ---------

Class B: ▲ ▲ ▲

SVM tries to find a boundary with a useful margin between classes.

51. Why scaling matters for SVM

SVM is sensitive to feature scale.

Suppose one feature ranges from:

0–1

and another ranges from:

0–100,000

The larger-scale feature can dominate distance-related calculations.

That is why scaling is often important for:

SVM
KNN
Logistic Regression in many situations
neural networks

52. Decision Tree

You used:

DecisionTreeClassifier()

and later:

DecisionTreeRegressor()

A decision tree makes predictions using a sequence of questions.

Imagine:

Is tenure < 12?
       /       \
     Yes       No
     /           \
Is charge > X?   ...

Eventually the tree reaches a leaf containing a prediction.

This is attractive because it is easy to visualize conceptually.

53. Why trees can overfit

A tree can keep splitting the data until it creates extremely specific rules.

For example:

If feature A > 10
and feature B < 4
and feature C = 7
and feature D > 92
...

Eventually the tree may memorize training examples.

That is why you later tried:

max_depth=6
min_samples_split=10
min_samples_leaf=5

These are ways of controlling tree complexity.

54. Random Forest

You used:

RandomForestRegressor(
    n_estimators=200,
    max_depth=8,
    min_samples_leaf=5,
    random_state=42
)

A random forest combines many decision trees.

Instead of relying on one tree:

Tree 1 ─┐
Tree 2 ─┤
Tree 3 ─┤
Tree 4 ─┤ → combined prediction
...     ┘

The idea is that many different trees can produce a more robust prediction than a single tree.

This is an example of an ensemble method.

55. Bagging

You used:

BaggingRegressor(
    n_estimators=100,
    max_samples=0.7,
    max_features=0.7
)

Bagging means combining models trained on different samples of the data.

The overall idea:

Dataset
  ↓
Different samples
  ↓
Multiple models
  ↓
Combine predictions

The goal is often to reduce variance and make predictions more stable.

Random Forest can be viewed as a specialized tree-based ensemble that adds additional randomness in how trees are built.

56. AdaBoost

You used:

AdaBoostRegressor()

AdaBoost means:

Adaptive Boosting

Instead of training many independent models and simply averaging them, boosting builds models sequentially.

Conceptually:

Model 1
  ↓
find mistakes
  ↓
Model 2 focuses more on difficult cases
  ↓
Model 3 focuses further
  ↓
combine models

The key idea:

Later models try to improve on the weaknesses of earlier models.

57. K-Nearest Neighbors (KNN)

You used:

KNeighborsRegressor()

KNN predicts using nearby examples.

Imagine a new customer.

The algorithm asks:

"Which existing customers are most similar to this customer?"

Then it uses those neighbors to estimate the output.

For regression, the prediction is commonly based on the neighbors' target values.

58. Why KNN needs scaling

KNN relies on distance.

Imagine:

Income:   0–100,000
Age:      18–80

If raw values are used, income can dominate the distance calculation.

Scaling puts features onto comparable scales.

That is why you experimented with:

StandardScaler()

59. StandardScaler

You used:

scaler = StandardScaler()

X_train_scaled = scaler.fit_transform(X_train_log)
X_test_scaled = scaler.transform(X_test_log)

This is an important pattern.

Notice:

scaler.fit_transform(X_train)

but:

scaler.transform(X_test)

Why?

The scaler learns the training data's statistics.

Then the exact same transformation is applied to the test data.

You should not independently fit the scaler on the test set.

Otherwise information from the test set leaks into preprocessing.

This idea is called avoiding:

Data leakage

60. Data leakage

Data leakage happens when information that should not be available to the model during training accidentally influences the model.

Example:

Train preprocessing
     ↓
uses information from test data
     ↓
test is no longer truly unseen

That can make evaluation look better than real-world performance.

The correct principle is:

Fit preprocessing steps using training data, then apply them to validation/test/new data.

A production ML pipeline should ideally bundle preprocessing and modeling together.

61. Linear Regression

You used:

LinearRegression()

for Customer Lifetime Value.

Linear regression tries to model a relationship like:

Y = b0 + b1X1 + b2X2 + ...

For one feature:

Y = intercept + slope × X

The goal is to find coefficients that produce predictions close to the observed values.

62. Regression score: what does `.score()` mean?

You used:

model.score(X_train, Y_train)
model.score(X_test, Y_test)

For the scikit-learn regression models you used, .score() generally returns R², the coefficient of determination.

Very roughly:

R² = 1

means the model explains the observed variation perfectly on that dataset.

R² = 0

means the model is no better than a simple baseline based on predicting the mean target.

R² can also be negative on unseen data.

Important:

R² is not the same thing as accuracy.

For regression, do not say:

"My regression model has 90% accuracy"

just because R² is 0.90.

Instead say:

"The model achieved an R² of 0.90."

63. Classification accuracy

You used:

accuracy_score(Y_test, y_test_pred)

Accuracy is:

correct predictions
-------------------
total predictions

For example:

90 correct
100 total

gives:

90% accuracy

Accuracy is easy to understand.

But it can be misleading when classes are highly imbalanced.

64. Confusion matrix

Your later notes/workflow refer to concepts such as:

TP
TN
FP
FN

A confusion matrix organizes classification predictions.

                    Actual
                 No       Yes
Predicted No     TN       FN
Predicted Yes    FP       TP

Meaning:

True Positive

Model predicted positive and it was positive.

True Negative

Model predicted negative and it was negative.

False Positive

Model predicted positive but it was negative.

False Negative

Model predicted negative but it was positive.

This is much more informative than accuracy alone when the cost of errors differs.

65. Precision and Recall

For classification:

Precision

Of everything the model predicted as positive:

How many were actually positive?

Precision = TP / (TP + FP)

Recall

Of everything that was actually positive:

How many did the model find?

Recall = TP / (TP + FN)

The right metric depends on the problem.

For churn prediction, for example, missing a customer who is about to churn may be more important than contacting an extra customer who would have stayed.

66. Your Telco Customer Churn project

This is probably the clearest classification example in your notebooks.

The business question is:

Can we predict whether a telecom customer will churn?

The target is:

Y = df["Churn"]

Features are:

X = df.drop("Churn", axis=1)

The workflow becomes:

Customer data
     ↓
Clean data
     ↓
Explore data
     ↓
Encode categories
     ↓
Split train/test
     ↓
Train classifier
     ↓
Predict churn
     ↓
Evaluate

67. Telco: removing customerID

You did:

df = df.drop(columns=["customerID"])

This is reasonable because the ID is an identifier rather than a customer characteristic.

You also commented that it could be kept separately if you later wanted to map predictions back to customers.

That is a good practical idea.

In production:

customerID
   ↓
keep separately for business reporting

but:

customerID
   X
do not necessarily use as a model feature

68. Telco: fixing TotalCharges

You used:

df["TotalCharges"] = pd.to_numeric(
    df["TotalCharges"],
    errors="coerce"
)

errors="coerce" means invalid values are converted to NaN.

Then you used:

fillna(0)

in one notebook and median imputation in another.

These are two different decisions.

The important lesson is:

Missing-value handling should be based on the meaning of the data, not simply on whichever replacement is easiest.

For example, if missing TotalCharges occurs because a customer has zero tenure and has just joined, then filling with zero may have a meaningful business interpretation.

If missingness represents an unknown measurement, median imputation may be more appropriate.

69. Telco: the first modeling version

The earlier Telco notebook used:

LabelEncoder()

for many categorical columns.

This is okay as a learning exercise for understanding encoding, but it is not ideal for every categorical variable.

A better version in your later notebook used:

pd.get_dummies(
    df,
    columns=multi_cols,
    drop_first=True
)

That is a meaningful improvement.

70. Telco: your improved preprocessing pipeline

The later notebook did something closer to:

Raw data
   ↓
Drop customerID
   ↓
Convert TotalCharges
   ↓
Encode binary columns
   ↓
One-hot encode multi-category columns
   ↓
Separate X and Y
   ↓
Scale X
   ↓
Train Logistic Regression
   ↓
Create Gradio interface

This is much closer to a real ML application.

71. Gradio: turning a model into an application

You used:

import gradio as gr

and built a UI.

This is an important step because it changes the project from:

Notebook experiment

to:

User input
    ↓
Preprocessing
    ↓
Model
    ↓
Prediction
    ↓
Human-readable result

For example:

Gender: Female
Tenure: 12 months
Contract: Month-to-month
Monthly charges: $70
...
          ↓
    Predict Churn
          ↓
High Churn Risk
Probability: ...

This demonstrates an important ML engineering concept:

A model is useful only when it can be integrated into a workflow where people or systems can use its predictions.

72. Very important: preprocessing must match training

Your Gradio code contains a good lesson.

During training you:

encode the categorical values
create dummy columns
scale the features
train the model

During prediction you must do the same thing:

raw user input
    ↓
same encoding
    ↓
same columns
    ↓
same scaling
    ↓
model

If the training data has:

feature_1
feature_2
feature_3
feature_4

but the application sends:

feature_1
feature_3
feature_4

the model cannot interpret the input correctly.

Your use of:

input_df.reindex(
    columns=feature_columns,
    fill_value=0
)

is intended to make the input columns match the training feature structure.

That is an important practical idea.

73. A stronger production pattern: Pipeline

A cleaner production approach is often:

Pipeline([
    ("preprocessor", ...),
    ("model", LogisticRegression())
])

Then preprocessing and prediction are tied together.

This reduces the risk of accidentally applying different transformations during training and inference.

Your notebook demonstrates the concept manually, which is useful for learning.

74. Your Auto Insurance / Customer Lifetime Value project

The 13-Aug.ipynb notebook moves into regression.

The target is:

Y = df["Customer Lifetime Value"]

This changes the question from:

Will this customer churn?

to:

What numerical Customer Lifetime Value should we predict?

Therefore:

Classification → Churn
Regression     → Customer Lifetime Value

75. Insurance dataset: initial exploration

You loaded:

df = pd.read_csv("AutoInsurance.csv")

Then inspected:

df.info()
df.shape
df.describe()

You also removed:

Customer
Effective To Date

from the modeling DataFrame.

Again, the purpose is to separate identifiers / dates that were not being used in the current modeling approach.

However, an important ML lesson is:

Dates are not automatically useless.

A date may contain useful information such as:

month
day of week
season
year
time since an event

A stronger project could engineer useful date features instead of simply dropping every date column.

76. Insurance EDA

You explored:

sns.countplot(x="State", data=df)
sns.boxplot(y=df["Income"])
sns.boxplot(
    x="Coverage",
    y="Customer Lifetime Value",
    data=df
)
sns.scatterplot(
    x="Income",
    y="Customer Lifetime Value",
    data=df
)
df["Income"].hist()

This is a good progression:

Categorical distribution
        ↓
Numerical distribution
        ↓
Numerical vs categorical
        ↓
Numerical vs numerical

You are gradually asking:

"What might explain the target?"

77. Automatically finding numerical and categorical columns

You used:

numerical_cols = df.select_dtypes(
    include=["int64", "float64"]
).columns

categorical_cols = df.select_dtypes(
    include=["object"]
).columns

This is useful because instead of manually listing every column, Python can identify columns based on data type.

Then you looped through them to create graphs.

This is the beginning of writing reusable data-analysis code.

78. Automated boxplots

You created boxplots for all numerical columns.

This is useful because you can quickly inspect:

distributions
median
spread
potential outliers

However, remember that a large collection of graphs can become overwhelming.

The goal should eventually move from:

"Let's plot everything."

to:

"Let's plot the variables that help answer our question."

That is a sign of growing from beginner EDA toward professional analysis.

79. Binning numerical variables for boxplots

You used:

pd.cut(
    df[col],
    bins=5
)

This divides a numerical variable into five intervals.

Then you compare CLV across those ranges.

Conceptually:

Income range 1 → CLV distribution
Income range 2 → CLV distribution
Income range 3 → CLV distribution
...

This is useful for visualization because a boxplot expects categories on one axis.

But the number and boundaries of bins can affect the story you see.

So binning is mainly a visualization technique here, not necessarily something you should automatically use as a model feature.

80. Comparing many regression algorithms

One of the biggest learning steps in your final notebook was trying multiple regressors:

Linear Regression
KNN Regressor
SVR
Decision Tree Regressor
Bagging Regressor
AdaBoost Regressor
Random Forest Regressor

This is valuable because it teaches a core ML lesson:

Different algorithms make different assumptions and capture different types of patterns.

There is no universally best algorithm.

81. Linear Regression vs tree-based models

Linear Regression assumes a relationship that can be represented through a linear combination of features.

Tree-based methods can represent more complex non-linear relationships.

For example:

Linear model:

Y
│       /
│     /
│   /
│ /
└──────── X

A tree-based model can approximate more irregular patterns:

Y
│    ┌────
│    │
│ ───┘
│
└──────── X

This is why trying multiple model families can be useful.

82. Why model comparison matters

Suppose you obtain:

Model             Train R²    Test R²
Linear Regression   0.70       0.68
KNN                 0.90       0.62
Decision Tree       0.99       0.55
Random Forest       0.91       0.78

You should not automatically choose the model with the highest training score.

The more important question is:

Which model generalizes well to unseen data?

Here Random Forest would be more interesting than the tree with 0.99 training R².

83. Log transformation of Customer Lifetime Value

You later used:

Y_log = np.log1p(Y)

Why?

Because some target variables are heavily skewed.

Imagine:

Most CLV values: relatively small
A few CLV values: extremely large

The model may have difficulty fitting such a distribution.

A log transformation compresses large values.

Conceptually:

Original:
1
10
100
1000
10000

Log scale:
small differences between large values

This can make the target distribution easier for some models to learn.

84. Reversing the log transformation

After predicting:

predictions_log = model.predict(X)

you used:

predictions_actual = np.expm1(predictions_log)

This is important.

If:

Y_log = log(1 + Y)

then:

Y = exp(Y_log) - 1

np.expm1() performs:

exp(x) - 1

So:

log target
   ↓
model
   ↓
log prediction
   ↓
expm1
   ↓
original target scale

85. An important issue in the notebook: split consistency

In your log-transform section, you did:

Y_log = np.log1p(Y)

X_train_log, X_test_log, Y_train_log, Y_test_log = train_test_split(
    X,
    Y_log,
    test_size=0.3,
    random_state=42
)

This is okay as a new split, but it means your earlier train/test split and later train/test split are not necessarily the same.

For a clean experiment, define the split once and reuse it consistently.

For example:

X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.3,
    random_state=42
)

y_train_log = np.log1p(y_train)
y_test_log = np.log1p(y_test)

This makes comparisons easier to reason about.

86. Another important issue: predicting on the full dataset

You later did:

predictions_log = model_rf.predict(X)

and saved predictions for the full dataset.

That can be useful for producing a business-facing prediction file.

But these are not unbiased test predictions.

Why?

Because X includes training observations the model already saw.

So:

Predictions on X_test
→ useful for evaluating unseen data

Predictions on X
→ useful for generating predictions for all records,
   but not for measuring generalization

This distinction is very important.

87. Hyperparameters

Later you changed models from:

DecisionTreeRegressor()

to:

DecisionTreeRegressor(
    max_depth=6,
    min_samples_split=10,
    min_samples_leaf=5
)

These settings are called hyperparameters.

They are chosen by us before training.

The model learns its internal parameters from the training data.

Parameters

Learned by the algorithm.

Hyperparameters

Set by us.

This distinction is fundamental.

88. Understanding your tree hyperparameters

`max_depth`

Controls how deep the tree can grow.

Smaller:

simpler tree
less overfitting

Larger:

more complex tree
greater overfitting risk

`min_samples_split`

Minimum number of samples required before a node can be split.

Higher values make the tree more conservative.

`min_samples_leaf`

Minimum number of samples allowed in a leaf.

Higher values prevent extremely tiny leaves.

89. Random Forest hyperparameters

You used:

RandomForestRegressor(
    n_estimators=200,
    max_depth=8,
    min_samples_leaf=5,
    random_state=42
)

`n_estimators=200`

Build 200 trees.

More trees can make the ensemble more stable, though computation increases.

`max_depth=8`

Limits tree complexity.

`min_samples_leaf=5`

Prevents leaves from becoming too specific.

`random_state=42`

Makes the experiment reproducible.

90. Bagging hyperparameters

You used:

BaggingRegressor(
    n_estimators=100,
    max_samples=0.7,
    max_features=0.7,
    random_state=42
)

This means, conceptually:

build many estimators
each estimator sees a sample of the training rows
each estimator can use a subset of features
combine their predictions

Again, the goal is to create a robust ensemble rather than rely on one model.

91. Scaling experiment in your notebook

You used:

StandardScaler()

and compared KNN/SVR.

This is a very useful experiment because it demonstrates:

Preprocessing requirements depend on the algorithm.

Tree-based algorithms generally do not need feature scaling in the same way that distance- or margin-based methods do.

KNN and SVM are much more sensitive to feature scale.

92. A subtle issue in your scaling experiment

You created:

X_train_scaled
X_test_scaled

but your KNN code first trained on:

X_train_log

before later looping over scaled KNN.

That is useful as experimentation, but for a clean article you should present the comparison more systematically:

KNN without scaling
        vs
KNN with scaling

and explicitly explain why the result changed.

For SVR, you correctly used the scaled data in the shown section.

93. The complete mental model

After these five days, the entire workflow can be remembered as:

             REAL-WORLD QUESTION
                     ↓
             Define the target
                     ↓
              Collect data
                     ↓
              Understand data
                     ↓
          ┌──────────┴──────────┐
          ↓                     ↓
      Numerical             Categorical
          ↓                     ↓
      distributions        category counts
          └──────────┬──────────┘
                     ↓
                    EDA
                     ↓
              Clean the data
                     ↓
          Handle missing values
                     ↓
             Encode categories
                     ↓
          Select useful features
                     ↓
              Split train/test
                     ↓
          Scale when appropriate
                     ↓
              Train baseline
                     ↓
          Evaluate on test data
                     ↓
             Try other models
                     ↓
         Tune hyperparameters
                     ↓
       Select based on validation
                     ↓
             Final evaluation
                     ↓
              Make predictions
                     ↓
       Integrate into an application

94. What each notebook taught you

Day 1 — Auto MPG

Main lessons:

load a dataset
inspect rows and columns
understand shape and data types
inspect summary statistics
identify categorical/text columns
detect messy numerical values
visualize distributions
use countplots
use histograms
use pairplots
understand missing values
convert "?" to NaN
convert text to numerical data
use median imputation

The biggest lesson:

Before modeling, understand and clean your data.

Day 2 — Telco Customer Churn

Main lessons:

identify a classification target
remove an identifier
handle TotalCharges
visualize class balance
compare categories against churn
use boxplots and histograms
use pairplots
inspect correlations
encode categorical data
split into training/testing data
train Logistic Regression
train SVC
train Decision Tree
compare train/test performance

The biggest lesson:

A classification model predicts categories, but the quality of the prediction depends heavily on preprocessing and evaluation.

Day 3 — Improving the Telco project

Main lessons:

one-hot encode multi-category variables
use random_state
use stratify
scale features
train Logistic Regression
build a prediction interface with Gradio
reproduce preprocessing at inference time
output a human-readable prediction

The biggest lesson:

An ML model is only one part of an ML application.

Day 4/5 — Customer Lifetime Value

Main lessons:

formulate a regression problem
inspect numerical/categorical relationships
automate EDA
encode categories
split train/test
compare multiple regressors
understand R²
try target transformation
use log transformation
reverse the transformation
experiment with scaling
tune hyperparameters
use ensemble models
generate prediction files

The biggest lesson:

Different models capture different patterns, and model selection should be based on performance on unseen data—not training performance alone.

95. What I would change before publishing the project as "production-quality"

Your notebooks are excellent learning material because they show experimentation. For a public article, however, it is worth separating:

Learning experiments

from:

Recommended ML workflow

Here are the most important improvements.

Improvement 1 — Use clear train/test terminology

Prefer:

X_train, X_test, y_train, y_test

rather than mixing Y and y.

Python convention generally uses lowercase y.

Improvement 2 — Make experiments reproducible

Use:

random_state=42

consistently when appropriate.

Improvement 3 — Use stratification for classification

For churn:

train_test_split(
    X,
    y,
    test_size=0.3,
    random_state=42,
    stratify=y
)

Improvement 4 — Avoid LabelEncoder for nominal multi-class input features

Use one-hot encoding for categories such as:

PaymentMethod
Contract
InternetService
VehicleClass

Label encoding is much more naturally suited to targets or truly ordinal categories.

Improvement 5 — Fit preprocessing only on training data

For example:

Training data
    ↓
fit scaler
    ↓
transform training

Test data
    ↓
transform using existing scaler

Do not independently fit preprocessing on the test data.

Improvement 6 — Use proper evaluation metrics

For classification, consider:

Accuracy
Precision
Recall
F1
Confusion Matrix
ROC-AUC
PR-AUC

For regression:

R²
MAE
MSE
RMSE

Which metrics matter depends on the business problem.

Improvement 7 — Don't compare only training scores

Always look at:

Training performance
+
Validation/test performance

A very high training score with much lower test performance can indicate overfitting.

Improvement 8 — Keep a validation strategy for model selection

A single test set should ideally be kept for final evaluation.

During experimentation, use cross-validation or a validation set to choose models/hyperparameters.

Conceptually:

Training data
   ↓
Cross-validation / validation
   ↓
Choose model
   ↓
Final test set
   ↓
Final unbiased-ish estimate

96. A very important distinction: EDA vs ML

EDA asks:

"What does my data look like?"

Machine learning asks:

"Can I learn a useful relationship that lets me make predictions on new data?"

These overlap, but they are not the same.

For example:

sns.scatterplot(...)

does not build a model.

It helps you understand the data before deciding how to model it.

97. Another important distinction: correlation vs prediction

A strong correlation can be useful.

But:

high correlation

does not automatically mean:

excellent predictive model

Prediction depends on:

multiple features
data quality
model type
noise
generalization
preprocessing
evaluation

So don't judge an ML problem from one graph.

98. Another important distinction: model vs algorithm

It is common to hear:

"I trained a model."

The algorithm is the learning procedure.

For example:

Random Forest = algorithm/model family

After training:

model_rf.fit(X_train, y_train)

you have a fitted model containing learned information.

A useful mental model:

Algorithm
   +
Training data
   ↓
Fitted model
   ↓
Predictions

99. What "learning" actually means

This is probably the most important concept to understand.

When you run:

model.fit(X_train, y_train)

the model is not magically understanding the world.

It is optimizing internal parameters so that its predictions match the training examples according to the algorithm's objective.

For example, linear regression learns coefficients.

A tree learns split rules.

A neural network learns weights.

So:

Training means finding model parameters that make the model perform well according to a defined objective.

100. What happens during prediction?

Once training is complete:

prediction = model.predict(X_test)

the model does not learn again from the test examples.

It applies what it learned during training.

Conceptually:

Training:

X_train + y_train
        ↓
      model.fit()
        ↓
   learned model

Prediction:

X_test
   ↓
learned model
   ↓
prediction

101. The biggest beginner misconception to avoid

Do not think:

"I used Random Forest, so I did machine learning."

The real ML skill is being able to explain:

What problem are you solving?
What is the target?
What features are available?
What does the data look like?
What problems did you find?
How did you clean them?
How did you encode the data?
Why did you choose the algorithm?
How did you evaluate it?
Does it generalize?
What would you improve?

If you can explain these, you are learning ML—not just memorizing scikit-learn commands.

102. A simple explanation of your five-day journey

If someone asks:

"What did you actually learn in five days?"

A strong answer is:

I learned that machine learning is not just about selecting an algorithm. I started by understanding datasets using Pandas, cleaning missing and incorrectly formatted values, and using visualization to understand distributions and relationships. Then I learned the difference between classification and regression, converted categorical data into numerical representations, split data into training and testing sets, trained several models using scikit-learn, and compared their performance on unseen data. Finally, I experimented with scaling, transformations, ensemble models, hyperparameters, and even built a simple Gradio interface around a churn model.

That is a much stronger story than:

"I learned Logistic Regression, SVM and Random Forest."

103. Suggested article structure for dev.to

For your final public article, I recommend this structure:

Title

1. Why I started learning ML
2. What I thought ML was vs what I learned
3. AI vs ML
4. Types of ML
5. My five-day learning roadmap
6. Day 1 — Understanding and cleaning data
7. EDA and why graphs matter
8. Day 2 — Classification with customer churn
9. Encoding categorical data
10. Train/test split
11. Logistic Regression, SVM and Decision Trees
12. How I evaluated classification models
13. Day 3 — Turning the model into an application
14. Day 4/5 — Regression and Customer Lifetime Value
15. Linear Regression vs KNN vs SVR vs Trees vs Ensembles
16. Log transformation
17. Scaling
18. Hyperparameters
19. What I got wrong / what I would improve
20. What I learned
21. What I plan to learn next

This will make the article feel like a journey, rather than a textbook.

104. Recommended article narrative

The most interesting story is:

"I started with almost no ML knowledge."

        ↓

"I learned that data preparation matters."

        ↓

"I discovered that graphs help me ask questions
before training models."

        ↓

"I learned classification."

        ↓

"I learned regression."

        ↓

"I tried different algorithms and realized
there is no single best model."

        ↓

"I learned that a high training score
doesn't necessarily mean a good model."

        ↓

"I turned one model into a small application."

        ↓

"I now understand the basic ML workflow
and know what I need to learn next."

That is a much more authentic beginner-to-ML story.

105. What to learn next

Based on what your notebooks already cover, I would not recommend immediately jumping into deep learning.

First strengthen these foundations:

1. Statistics

Learn:

mean
median
variance
standard deviation
distributions
probability
correlation
conditional probability

2. ML evaluation

Learn:

confusion matrix
precision
recall
F1
ROC-AUC
MAE
RMSE
R²
cross-validation

3. Feature engineering

Learn:

handling dates
categorical features
transformations
interactions
outliers
missing values

4. Model selection

Learn:

cross-validation
hyperparameter tuning
GridSearchCV
RandomizedSearchCV
pipelines

5. ML fundamentals

Learn:

bias vs variance
underfitting
overfitting
regularization
data leakage
feature importance

Then move toward:

Classical ML
     ↓
Feature engineering
     ↓
Model selection
     ↓
Cross-validation
     ↓
ML pipelines
     ↓
Deployment
     ↓
Deep Learning
     ↓
Generative AI / LLMs

106. The one-page cheat sheet

Problem type

Predict a category → Classification
Predict a number   → Regression
Find hidden groups → Clustering
Learn through reward → Reinforcement Learning

Data

Rows       → observations
Columns    → variables/features
X          → input features
y          → target

First inspection

df.head()
df.shape
df.info()
df.describe()
df.sample()
df.isnull().sum()

Visualization

Countplot   → categories
Histogram   → numerical distribution
Boxplot     → spread/outliers
Scatterplot → two numerical variables
Pairplot    → many numerical relationships
Heatmap     → correlation matrix

Preparation

Missing values
Data types
Categorical encoding
Feature selection
Scaling when needed

Training

model.fit(X_train, y_train)

Prediction

model.predict(X_test)

Evaluation

Classification:

Accuracy
Precision
Recall
F1
Confusion Matrix
ROC-AUC

Regression:

R²
MAE
MSE
RMSE

Core warning signs

Training score >> Test score
        ↓
Possible overfitting

Test information used during preprocessing
        ↓
Possible data leakage

Identifier used as feature
        ↓
Possible meaningless pattern

Categorical values converted to arbitrary numbers
        ↓
Possible false ordering

107. Final takeaway

The biggest lesson from these five days is not a particular Python library or algorithm.

It is the workflow:

Understand the problem → understand the data → clean the data → explore the data → prepare features → train → evaluate → improve → deploy.

The algorithms are tools inside that workflow.

You have already touched many of the important building blocks:

Pandas
NumPy
Matplotlib
Seaborn
Scikit-learn
data cleaning
EDA
classification
regression
encoding
train/test split
scaling
model evaluation
hyperparameters
ensemble models
target transformation
Gradio

The next step is not to memorize more algorithms.

The next step is to become comfortable answering:

Why am I doing this step?

If you can answer that question for every important line in your notebook, you are moving from "following an ML tutorial" to actually understanding machine learning.

Appendix A — Notebook-by-notebook step map

`11_Aug_mpg.ipynb`

Notebook step	Why it was done	ML concept
Import NumPy/Pandas/Matplotlib/Seaborn	Load tools	Python ML stack
`read_csv()`	Load data	Data ingestion
`nunique()`	Understand uniqueness	Data exploration
`head()`	Inspect examples	EDA
`shape`	Understand dataset size	EDA
`describe()`	Summary statistics	EDA
`info()`	Types/non-null values	Data quality
`countplot()`	Inspect categorical distribution	Visualization
`hist()`	Inspect numerical distributions	Visualization
`isnull()`	Detect actual missing values	Data cleaning
`pairplot()`	Explore relationships	EDA
`sample()`	Inspect random examples	Data quality
Identify `car name`	Recognize identifier-like field	Feature selection
Replace `?`	Convert fake missing values	Data cleaning
`to_numeric()`	Make horsepower numerical	Data preparation
Median imputation	Fill missing horsepower	Missing-value handling
Train/test split	Separate learning/evaluation data	Generalization
Logistic Regression	Start classification experiment	Supervised learning

Important caveat

`12th-Aug-Telco-Customer-churn.ipynb`

Step	Purpose
Load Telco CSV	Get customer data
`head()`	Inspect records
`shape`	Understand size
`describe()`	Statistical overview
Drop `customerID`	Remove identifier from features
`info()`	Check data types
`sample()`	Inspect random rows
Countplot of Churn	Check class distribution
Countplot of gender	Explore category distribution
Histograms	Explore numerical distributions
Pairplot	Explore relationships
Convert TotalCharges	Fix numeric type
Fill missing TotalCharges	Handle missing data
LabelEncoder	Convert categories to numbers
Train/test split	Evaluate on unseen data
Logistic Regression	Classification model
SVC	Alternative classifier
Decision Tree	Alternative classifier
Gradio	Turn model into interactive app

Important caveat

The notebook uses LabelEncoder on many categorical input columns. This is useful for learning encoding, but one-hot encoding is generally safer for nominal multi-category features.

`12-Aug-2.ipynb`

This notebook is a more developed version of the Telco project.

Important improvements include:

dropping customerID
converting TotalCharges
one-hot encoding multi-category variables
using random_state=42
using stratify=Y
scaling features
training Logistic Regression
building a Gradio prediction interface
preserving training feature columns
applying the same preprocessing to new input

This is the notebook that most clearly demonstrates the transition from:

ML experiment

to:

small ML application

`13-Aug.ipynb`

Main steps:

Load AutoInsurance
       ↓
Inspect dataset
       ↓
Drop Customer / date columns
       ↓
EDA
       ↓
Identify numerical/categorical columns
       ↓
Generate many visualizations
       ↓
Encode categorical variables
       ↓
Set CLV as target
       ↓
Train/test split
       ↓
Try multiple regressors
       ↓
Try log target
       ↓
Try scaling
       ↓
Tune tree/ensemble hyperparameters
       ↓
Generate predictions
       ↓
Save prediction CSVs

This notebook demonstrates the transition from classification to regression and from a single baseline model to model comparison and tuning.

Appendix B — A cleaner ML template to remember

# 1. Load
df = pd.read_csv("data.csv")

# 2. Understand
df.head()
df.shape
df.info()
df.describe()

# 3. Clean
# handle missing values
# fix data types
# remove inappropriate identifiers

# 4. Explore
# histograms
# boxplots
# countplots
# scatterplots
# correlation

# 5. Prepare
X = df.drop("target", axis=1)
y = df["target"]

# 6. Split
X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.2,
    random_state=42
)

# 7. Preprocess
# encoding / scaling

# 8. Train
model.fit(X_train, y_train)

# 9. Predict
predictions = model.predict(X_test)

# 10. Evaluate
# choose metrics appropriate to the problem

# 11. Compare / improve
# try another model
# tune hyperparameters
# use cross-validation

# 12. Final model
# train using the selected approach

# 13. Predict new data

Closing thought for the article

Five days ago, an ML notebook could easily look like a collection of unfamiliar commands.

Now there is a structure behind those commands.

head() is not just a command — it is a way to understand the data.

isnull() is not just a command — it is part of data quality.

A histogram is not just a graph — it tells you about a distribution.

train_test_split() is not just boilerplate — it protects your evaluation from simply measuring memorization.

fit() is not magic — it is the learning stage.

predict() is the model applying what it learned.

And model comparison is not about finding the fanciest algorithm — it is about finding an approach that generalizes well to data the model has never seen.

That, more than anything else, is what I took away from my first five days of machine learning.

ML Foundations: A Complete Data Cleaning & ML Pipeline

yash-nigam — Tue, 25 Aug 2026 13:42:00 +0000

Fundamentals of Machine Learning

Using Google Colab and Python Libraries to Run ML Models

Find all the jupiter notebooks and datasets here: https://github.com/yash-nigam/AI-ML-Foundations

What is Artificial Intelligence?
What is Machine Learning?
- 2.1 Types of Machine Learning
- 2.2 Supervised Learning
- 2.3 Unsupervised Learning
- 2.4 Reinforcement Learning
How Python Helps Implement Machine Learning
- 3.1 The Libraries and What Each One Handles
Exploratory Data Analysis: Investigating the Dataset
Working with the Loan Prediction Dataset
- 5.1 Making the Dataset Ready for ML Models
- 5.2 What Should Be Cleaned Up
- 5.3 Cleanup Steps
  - 5.3.1 Remove Unique/Identifier Columns
  - 5.3.2 Handle Missing Values
  - 5.3.3 Standardize & Encode Categorical Values
  - 5.3.4 Outliers, Duplicates, Imbalance, Scaling, Split
  - 5.3.5 Replace Non-Standard Placeholder Values
Encoding
- 6.1 Why Does ML Need Encoding?
- 6.2 Label Encoding
Train/Test Split
- 7.1 Why Not Train and Test on the Same Data?
- 7.2 Underfitting
- 7.3 Why random_state Matters
Machine Learning Models
- 8.1 Confusion Matrix
Gradio: Turning a Model into an Application
What "Learning" Actually Means
- 10.1 What Happens During Prediction?
The Real ML Skill — Questions to Be Able to Answer

1. What is Artificial Intelligence?

Artificial Intelligence (AI) is the broad idea of building systems that can perform tasks that normally require some form of human intelligence, such as:

Recognizing an image
Generating text / writing an email
Generating an image
Recommending a movie
Detecting fraud
Predicting the price/value of something

A useful mental model for the hierarchy:

2. What is Machine Learning?

Machine learning is teaching a computer to find patterns in examples, instead of telling it exact rules to follow.

Example: The old way to predict if a customer will cancel their telecom subscription — you could try writing rules by hand:

IF contract = month-to-month
AND monthly charges = high
AND tenure = low
THEN churn = yes

The machine learning way: show the ML algorithm many past customers along with what actually happened to them:

Customer	Contract type	Monthly charges	Tenure	Did they churn?
A	Month-to-month	High	Low	Yes
B	Two year	Low	High	No
C	Month-to-month	Low	High	No
D	Month-to-month	High	Low	Yes

The algorithm studies this data and figures out the pattern on its own — no one told it "low tenure + high charges = risky."

Using it to predict for a new customer

Once trained, you can hand it a brand-new customer it has never seen:

New customer's info
        ↓
   Trained model
        ↓
  Predicted: churn or not

2.1 Types of Machine Learning

	Supervised Learning	Unsupervised Learning	Reinforcement Learning
Core idea	Learn from examples with known answers	Find hidden patterns with no answers given	Learn by trial and error, guided by rewards
Data needed	Inputs + labels	Inputs only, no labels	No fixed dataset — agent generates its own data
Task types	Classification, Regression	Clustering, dimensionality reduction, anomaly detection	Choosing actions to maximize long-term reward
Common algorithms	Logistic Regression, Decision Trees, Random Forest	K-Means, DBSCAN, PCA	Q-Learning, DQN, Policy Gradient
How it's evaluated	Accuracy, precision, recall, MAE, RMSE	No ground truth — silhouette score, human judgment	Total reward earned
Real-world examples	Churn prediction, spam filtering, price estimation	Customer segmentation, topic discovery	Game-playing agents, robotics, ad bidding

2.2 Supervised Learning

Supervised learning is the case where you already know the right answers for your training data.

The model makes a guess, checks it against the known answer, sees how wrong it was, and adjusts. Repeat this thousands of times and it gets good at guessing.

Input (X)  →  Known answer (Y)

X is everything you know about a customer. Y is what actually happened to them.

X = [tenure=2, contract=month-to-month, charges=95.5]   →   Y = churned
X = [tenure=48, contract=two-year, charges=45.2]        →   Y = stayed

The model's job is to learn the relationship between X and Y well enough that when a new X shows up with no Y attached, it can produce a sensible guess.

The key requirement: you need historical data where the outcome is already recorded. No answer key, no supervised learning.

The only question that splits supervised learning in two

Is the thing I'm predicting a category, or a number?

That's it. That single question decides whether you're doing classification or regression — and it changes your algorithms, your metrics, and how you evaluate success.

	Classification	Regression
Question it answers	Which category?	How much?
Type of answer	A fixed label (bucket)	A number on a scale
Examples	Spam or Not spam Churn or No churn Cat / Dog / Horse	House price: ₹87,45,000 Temperature: 31.4°C Fuel efficiency: 23.7 MPG
What the output means	A label, not a quantity — `1` isn't "more" than `0`	A real quantity — 6,000 truly is twice 3,000
Model output	A probability → converted to a label e.g. `0.91 → "Churn"`	A direct number e.g. `6,500`
Threshold involved?	Yes — usually 0.5, adjustable	No
Being "wrong"	Binary — right label or wrong label	A matter of degree — off by a little or a lot
Common metrics	Accuracy, Precision, Recall	MAE, RMSE, R²
Common algorithms	Logistic Regression, SVC, Decision Tree Classifier	Linear Regression, KNN Regressor, SVR, Random Forest Regressor
In your project	Telco churn (Yes/No)	Customer Lifetime Value (a number)

Quick test for your own problems — when you get a new dataset, look at the target column and ask:

Does averaging two values in this column produce something meaningful?

  Average of ₹5,000 and ₹7,000 = ₹6,000  ✓ meaningful  → regression
  Average of "spam" and "not spam"       ✗ meaningless → classification

2.3 Unsupervised Learning

In unsupervised learning, we do not have a known target. Instead, the algorithm tries to discover structure in the data.

Customer data
     ↓
Find groups
     ↓
Group 1: high-value customers
Group 2: price-sensitive customers
Group 3: new customers

Common examples:

K-Means clustering
Hierarchical clustering
PCA

2.4 Reinforcement Learning

Reinforcement learning is different. An agent interacts with an environment and receives rewards or penalties.

Agent
  ↓ action
Environment
  ↓
Reward / penalty
  ↓
Agent learns

Examples:

Game-playing agents
Robotics
Certain recommendation/control systems

This was not part of the five-day practical work.

3. How Python Helps Implement Machine Learning

Machine learning is mostly mathematics — matrix operations, optimization, statistics. In principle you could write all of it yourself, but in practice nobody does. Python has become the default language for ML because a small set of mature libraries already implement those pieces, tested and optimized, so you can focus on the problem rather than the arithmetic.

What makes Python work well here is that these libraries fit together as a pipeline, not as isolated tools. Each one covers one stage of the journey, and they hand data to each other in a common format — a Pandas DataFrame or a NumPy array — so nothing needs translating in between.

A typical project moves through the stack like this:

Load and clean the data        →  Pandas (with NumPy underneath)
Explore and visualize it       →  Matplotlib + Seaborn
Prepare features and split     →  Scikit-learn
Train and predict              →  Scikit-learn
Evaluate the results           →  Scikit-learn (+ Seaborn to plot them)

Notice that roughly the first half of that pipeline has nothing to do with modeling at all. In real projects, loading, cleaning, and understanding the data usually takes far more time than calling .fit(). The libraries reflect that reality: Pandas and Seaborn get used constantly, while the actual model training is often two lines.

The other thing Python gives you is a consistent interface. Once you learn scikit-learn's fit → predict pattern, it works the same way for logistic regression, random forests, and support vector machines alike. Swapping one algorithm for another is a one-line change, which makes it cheap to try several and compare.

3.1 The Libraries and What Each One Handles

Library	Import	Role in the Pipeline	Key Functions/Methods	Notes
NumPy	`import numpy as np`	The numerical foundation — fast array math that every other library is built on	`np.nan`, `np.log1p()`, `np.expm1()`	`np.nan` = missing value. `log1p(x)` = log(1+x), useful for skewed targets. `expm1(x)` reverses it — used to convert log predictions back to the original CLV scale. You rarely use NumPy directly; it works underneath Pandas and scikit-learn
Pandas	`import pandas as pd`	Load and clean the data — a spreadsheet Python can manipulate	`read_csv()`, column selection/removal, dtype conversion, missing-value detection/fill, X/Y selection, encoding, saving predictions	Where most of your time actually goes. Example: `df = pd.read_csv("Telco_Customer_Churn.csv")`
Matplotlib	`import matplotlib.pyplot as plt`	The plotting engine — controls figures, titles, axes, display	`plt.figure()`, `plt.title()`, `plt.show()`	Rarely used alone; it's the layer Seaborn sits on top of
Seaborn	`import seaborn as sns`	Explore and understand the data through statistical plots	`countplot`, `boxplot`, `histplot`, `pairplot`, `scatterplot`, `heatmap`	One line gives you a plot that would take many in raw Matplotlib. A graph is a tool for asking questions about the data, not decoration
Scikit-learn	`from sklearn... import ...`	Everything modeling-related: splitting, preprocessing, scaling, encoding, training, prediction, evaluation	`train_test_split()`, `StandardScaler()`, `model.fit(X_train, Y_train)`, `model.predict(X_test)`, metrics functions	The `fit → predict` pattern is identical across nearly every model, so trying a different algorithm is a one-line change

The short version: Pandas gets the data into shape, Seaborn helps you understand it, and scikit-learn learns from it — with NumPy doing the arithmetic underneath and Matplotlib drawing the pictures.

4. Exploratory Data Analysis: Investigating the Dataset

Command	What It Does & How to Use It
`df.head()`	Shows the first few rows of the dataset. Run this right after loading data to confirm it loaded correctly and get a quick look at the columns and values.
`df.shape`	Returns the number of rows and columns, e.g. `(398, 9)`. Use it to quickly gauge dataset size, which affects model choice and computation time.
`df.info()`	Lists each column's data type and non-null count. Use it to spot missing values and type problems — e.g. a numeric column showing as `object` usually means it contains hidden text like `"?"`.
`df.describe()` / `df.describe(include="all")`	Shows summary statistics (count, mean, min, max, etc.) for numeric columns, or all columns with `include="all"`. Use it to check the scale of each variable and spot possible outliers.
`df.sample(n)`	Shows `n` random rows instead of just the first few. Use it after `head()` to catch inconsistencies or unusual values that only appear later in the data.
`df.isnull().sum()`	Counts missing (`NaN`) values per column. Use it to identify which columns need cleaning — note it only catches true `NaN`s, not disguised placeholders like `"?"`, which need to be checked separately.

5. Working with the Loan Prediction Dataset

5.1 Making the Dataset Ready for ML Models

Raw data collected from forms, surveys, or real-world systems is almost never ready to feed straight into a machine learning model.

ML models cannot by themselves guess around these problems, and these gaps could reduce the accuracy of your results.

5.2 What Should Be Cleaned Up

NaN (missing values) — ML models cannot handle these by themselves; they'll throw errors or silently produce garbage results.
Unique-identifier columns — Unique IDs create noise and aren't helpful for learning genuine patterns in the data.
Text/categorical values — Numbers are expected by most ML models, not strings like "Yes"/"No".
Imbalanced or skewed data — If 70% of your data is one outcome, a lazy model can get 70% accuracy by always predicting that outcome, without actually learning anything useful.
Outliers and skewed numeric columns — A few extreme values can pull the mean far from what's "typical," making mean-based decisions misleading.

5.3 Cleanup Steps

5.3.1 Remove Unique/Identifier Columns

Columns like IDs (e.g. Loan_ID) don't carry predictive signal and should be dropped.

df = df.drop("Loan_ID", axis=1)

Removes the Loan_ID column. It's a unique identifier for each row (like a primary key) — it has no predictive value and would only confuse or overfit a model if left in.

5.3.2 Handle Missing Values

Either drop rows/columns with too many nulls, or fill them in:

Use median for numeric columns that are skewed (robust to outliers, e.g. income, loan amount).
Use mean for numeric columns that are roughly symmetric/normally distributed.
Use mode (most frequent value) for categorical columns, since averaging text categories isn't meaningful.

Using Median

median_loanamount = df["LoanAmount"].median()

Calculates the median of LoanAmount. Median is preferred over mean here because loan/income data is typically skewed by a few very high values, and median is more robust to outliers.

median_loanamount_term = df["Loan_Amount_Term"].median()

Same logic — computes the median loan term to use for filling its missing values later.

median_credit_history = df["Credit_History"].median()

Computes the median of Credit_History (a 0/1 flag). Even though it's binary, median is used instead of mean to avoid producing a non-integer/ambiguous fill value.

Use the calculated values to fill:

df["LoanAmount"] = df["LoanAmount"].replace(np.nan, median_loanamount)
df["Loan_Amount_Term"] = df["Loan_Amount_Term"].replace(np.nan, median_loanamount_term)
df["Credit_History"] = df["Credit_History"].replace(np.nan, median_credit_history)

Fills missing values with the calculated medians. This avoids dropping rows (losing data) while keeping the columns usable for modeling, since most ML algorithms can't handle NaN.

Using Mode

Calculates the mode (most frequent value) for gender, married, dependents, and self_employed. Mode is used instead of median/mean because these are categorical columns — you can't average "Male" and "Female".

mode_gender = df["Gender"].mode()[0]
mode_married = df["Married"].mode()[0]
mode_dependents = df["Dependents"].mode()[0]
mode_self_employed = df["Self_Employed"].mode()[0]

Replace the missing entries with the most common category, which is a reasonable, low-bias guess for a categorical field:

df["Gender"] = df["Gender"].replace(np.nan, mode_gender)
df["Married"] = df["Married"].replace(np.nan, mode_married)
df["Dependents"] = df["Dependents"].replace(np.nan, mode_dependents)
df["Self_Employed"] = df["Self_Employed"].replace(np.nan, mode_self_employed)

5.3.3 Standardize & Encode Categorical Values

Standardize categorical values — fix inconsistent labels (e.g. "male", "Male", "MALE" should all become one consistent value).
Encode categorical variables — convert text categories into numbers using techniques like Label Encoding or One-Hot Encoding, since most ML models only accept numeric input.

LabelEncoder()  # + loop over cat_cols

Converts all text/categorical columns (Loan_Status, Married, Gender, etc.) into numeric codes (e.g., "Yes"/"No" → 1/0). This step is required because scikit-learn models only accept numeric input, not strings.

5.3.4 Outliers, Duplicates, Imbalance, Scaling, Split

Check and handle outliers — identify extreme values (via boxplots, .describe(), or z-scores) and decide whether to cap, remove, or transform them.
Check for and remove duplicate rows — duplicate records can bias the model toward those repeated patterns.
Address class imbalance — if one outcome dominates the target column, apply techniques like upsampling, downsampling, or SMOTE so the model doesn't just learn to predict the majority class.
Feature scaling (when needed) — normalize or standardize numeric ranges for models sensitive to scale (like SVM, KNN, or Logistic Regression with regularization).
Split before further transformation — separate train/test sets early so cleaning decisions (like fill values from training data) don't leak information from the test set.

5.3.5 Replace Non-Standard Placeholder Values

df["horsepower"] = df["horsepower"].replace("?", np.nan)

This converts the placeholder "?" into a real missing value.

df["horsepower"] = pd.to_numeric(df["horsepower"])

This then converts the column into a proper numeric type so it can be used in calculations and models.

6. Encoding

6.1 Why Does ML Need Encoding?

Many ML algorithms work with numbers. But real-world data contains categories like:

Male
Female

Yes
No

Month-to-month
One year
Two year

Fiber optic
DSL
No

The model needs these values represented numerically. This is called encoding.

6.2 Label Encoding

from sklearn.preprocessing import LabelEncoder

For a binary column:

No  → 0
Yes → 1

This can be reasonable for a binary variable. For example:

Churn:
No  → 0
Yes → 1

That makes intuitive sense.

7. Train/Test Split

Repeatedly used:

train_test_split(X, Y, test_size=0.3)

This is one of the most important ML concepts.

Suppose we have 1,000 examples. We might use:

700 → training
300 → testing

The training set is used to learn the model. The test set is held back to evaluate how the trained model performs on unseen data.

Think of it like an exam:

Training data = practice questions
Test data     = unseen exam questions

7.1 Why Not Train and Test on the Same Data?

Because the model could simply memorize the training examples. Suppose:

Training score = 99%
Test score     = 65%

That is a warning sign. The model learned the training data extremely well but does not generalize well. This is called:

Overfitting

7.2 Underfitting

The opposite can happen. If the model is too simple:

Training score = 60%
Test score     = 58%

It may not have learned enough from the data. This is:

Underfitting

A useful mental picture:

Underfitting
Model too simple
       ↓
misses important patterns

Good fit
Learns useful patterns
       ↓
works on unseen data

Overfitting
Model learns noise/details
       ↓
great training performance
poor unseen performance

7.3 Why `random_state` Matters

In one Telco notebook:

train_test_split(X, Y, test_size=0.3)

In the improved notebook:

train_test_split(X, Y, test_size=0.3, random_state=42)

A random state makes the split reproducible. Without it, the random split can change between runs — meaning your score may change between runs too. With random_state=42, you can reproduce the same split every time. The number 42 is not magical; any fixed integer can serve this purpose.

8. Machine Learning Models

Model / Concept	What It Does & How to Use It
Logistic Regression	Despite the name, used for classification — estimates the probability of belonging to a class, then applies a threshold to assign a label. A simple, fast, interpretable baseline model — good starting point before trying complex models.
SVC (Support Vector Machine)	Finds a decision boundary that best separates classes, aiming for maximum margin between them. Sensitive to feature scale, so scale your data (e.g. with `StandardScaler`) before using it.
Decision Tree	Predicts by asking a sequence of yes/no questions, ending at a leaf with the prediction. Easy to visualize but prone to overfitting — control this with `max_depth`, `min_samples_split`, `min_samples_leaf`.
Random Forest	Combines many decision trees and averages their predictions for a more robust result. An ensemble method — generally more reliable than a single decision tree.
Bagging	Trains multiple models on different random samples of the data, then combines their predictions. Reduces variance and makes predictions more stable; Random Forest is a specialized version of this.
AdaBoost	Builds models sequentially, where each new model focuses on the mistakes of the previous one. A boosting method — improves weak learners step by step rather than averaging independent ones.
KNN (K-Nearest Neighbors)	Predicts by finding the most similar existing examples ("neighbors") and using their values. Relies on distance, so it needs feature scaling to avoid large-scale features dominating the result.
StandardScaler	Scales features to comparable ranges before training distance-sensitive models. Always `fit_transform` on training data, then only `transform` (not fit) on test data, to avoid data leakage.
Linear Regression	Models the relationship between features and a numeric target as a straight-line equation (`Y = b0 + b1X1 + ...`). Used for predicting continuous values, e.g. Customer Lifetime Value.
R² Score (`.score()`)	Measures how well the model explains variation in the target — 1 is perfect, 0 means no better than predicting the average. Report it as "R² of 0.90," not "90% accuracy" — they're not the same thing.
Accuracy Score	The proportion of correct predictions out of all predictions made, used for classification. Easy to understand, but misleading when classes are imbalanced (e.g. 95% "no churn" data).

8.1 Confusion Matrix

A confusion matrix organizes classification predictions.

                    Actual
                 No       Yes
Predicted No     TN       FN
Predicted Yes    FP       TP

Meaning:

True Positive — model predicted positive and it was positive.
True Negative — model predicted negative and it was negative.
False Positive — model predicted positive but it was negative.
False Negative — model predicted negative but it was positive.

This is much more informative than accuracy alone when the cost of errors differs.

9. Gradio: Turning a Model into an Application

import gradio as gr

Building a UI around your model is an important step because it changes the project from:

Notebook experiment

to:

User input
    ↓
Preprocessing
    ↓
Model
    ↓
Prediction
    ↓
Human-readable result

For example:

Gender: Female
Tenure: 12 months
Contract: Month-to-month
Monthly charges: $70
...
          ↓
    Predict Churn
          ↓
High Churn Risk
Probability: ...

This demonstrates an important ML engineering concept:

A model is useful only when it can be integrated into a workflow where people or systems can use its predictions.

10. What "Learning" Actually Means

When you run:

model.fit(X_train, y_train)

the model is not magically understanding the world. It is optimizing internal parameters so that its predictions match the training examples according to the algorithm's objective.

For example, linear regression learns coefficients. A tree learns split rules. A neural network learns weights.

So:

Training means finding model parameters that make the model perform well according to a defined objective.

10.1 What Happens During Prediction?

Once training is complete:

prediction = model.predict(X_test)

the model does not learn again from the test examples. It applies what it learned during training.

Conceptually:

Training:

X_train + y_train
        ↓
      model.fit()
        ↓
   learned model

Prediction:

X_test
   ↓
learned model
   ↓
prediction

11. The Real ML Skill — Questions to Be Able to Answer

What problem are you solving?
What is the target?
What features are available?
What does the data look like?
What problems did you find?
How did you clean them?
How did you encode the data?
Why did you choose the algorithm?
How did you evaluate it?
Does it generalize?
What would you improve?
Why was a specific algorithm chosen?

Create a free static website using Github Pages and Jekyll

yash-nigam — Fri, 03 Mar 2023 17:39:15 +0000

Why use GitHub Pages and Jekyll
Prerequisites
1: Dev Machine Setup
2: Use Chirpy Starter Template
3: Install dependencies
4: Modify config yaml and preview changes
5: Using GitHub Actions
6: Commit Changes to Repo
7: View the website
8: Personalize by Adding Avatar, favicon, sidebar link and create first post

Why use GitHub Pages and Jekyll ?

GitHub Pages allows us to host a static website(html, CSS, JavaScript) on GitHub repository for absolutely free which is ideal for personal websites.

A repository with the naming convention of {github-user-name}.github.io makes that repository serve a static website with subdomain as {github-user-name} and domain name as github.io

Jekyll (static site generator tool) is integrated with GitHub pages providing professional looking themes and many free templates for your website.

When using Jekyll we do not have to code any HTML, CSS or JavaScript. Instead we only edit the markdown files to achieve the desired results in the templates.

Prerequisites

Windows/Linux developer machine
GitHub Account

1: setup Ruby, Gem and Jekyll on developer machine

If dev machine is windows, use Windows subsystem for Linux - which installs a ubuntu environment which is available from PowerShell terminal.
- Open a PowerShell prompt as an Administrator and run: wsl --install
Install Ruby and other prerequisites:

yash@LearningPC:~/github$ sudo apt-get install ruby-full build-essential zlib1g-dev
[sudo] password for yash:
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
build-essential is already the newest version (12.9ubuntu3).
zlib1g-dev is already the newest version (1:1.2.11.dfsg-2ubuntu9.2).
The following additional packages will be installed:
  ri
The following NEW packages will be installed:
  ri ruby-full
0 upgraded, 2 newly installed, 0 to remove and 4 not upgraded.
Need to get 6788 B of archives.
After this operation, 38.9 kB of additional disk space will be used.
Do you want to continue? [Y/n] Y
Get:1 http://archive.ubuntu.com/ubuntu jammy/universe amd64 ri all 1:3.0~exp1 [4206 B]
Get:2 http://archive.ubuntu.com/ubuntu jammy/universe amd64 ruby-full all 1:3.0~exp1 [2582 B]
Fetched 6788 B in 1s (4970 B/s)
Selecting previously unselected package ri.
(Reading database ... 47201 files and directories currently installed.)
Preparing to unpack .../ri_1%3a3.0~exp1_all.deb ...
Unpacking ri (1:3.0~exp1) ...
Selecting previously unselected package ruby-full.
Preparing to unpack .../ruby-full_1%3a3.0~exp1_all.deb ...
Unpacking ruby-full (1:3.0~exp1) ...
Setting up ri (1:3.0~exp1) ...
Setting up ruby-full (1:3.0~exp1) ...

Add environment variables to your ~/.bashrc file

echo '# Install Ruby Gems to ~/gems' >> ~/.bashrc
echo 'export GEM_HOME="$HOME/gems"' >> ~/.bashrc
echo 'export PATH="$HOME/gems/bin:$PATH"' >> ~/.bashrc
source ~/.bashrc

Install Jekyll and Bundler

yash@LearningPC:~/github$ gem install jekyll bundler
Fetching jekyll-4.3.2.gem
Successfully installed jekyll-4.3.2
Parsing documentation for jekyll-4.3.2
Installing ri documentation for jekyll-4.3.2
Done installing documentation for jekyll after 3 seconds
Fetching bundler-2.4.7.gem
Successfully installed bundler-2.4.7
Parsing documentation for bundler-2.4.7
Installing ri documentation for bundler-2.4.7
Done installing documentation for bundler after 0 seconds
2 gems installed

2: Use chirpy-starter template to create new repo

Sign in to GitHub.com with your account, and go to chirpy-starter, click the button Use this template > Create a new repository

In the next window type the repository name to be the same as your GitHub user name appended with .github.io <GitHub User Name>.githubpages.io This makes the repo. act as a source of static website, and the repo. name becomes the domain name website.
Once this operation is complete all the contents of chirpy-starter will be present in our new site: https://github.com/yash-nigam/yash-nigam.github.io

3: Installing dependencies of chirpy-starter

This site requires some GEM's to be installed before it can be used, the list of can be found in the Gemfile: https://github.com/cotes2020/chirpy-starter/blob/main/Gemfile
On your local development machine, clone the new repo git clone https://github.com/yash-nigam/yash-nigam.github.io.git
go to the repo root directory and run bundle command

cd yash-nigam.github.io

yash@LearningPC:~/github/yash-nigam.github.io$ bundle
Fetching gem metadata from https://rubygems.org/...........
Resolving dependencies...
Using bundler 2.4.7
Using concurrent-ruby 1.2.2
Using rainbow 3.1.1
Using ffi 1.15.5
Using forwardable-extended 2.6.0
Using mercenary 0.4.0
Using parallel 1.22.1
Using rouge 4.1.0
Using eventmachine 1.2.7
Using unicode-display_width 2.4.2
Using racc 1.6.2
Using colorator 1.1.0
Using i18n 1.12.0
Using ethon 0.16.0
Using yell 2.2.2
Using pathutil 0.16.2
Using liquid 4.0.4
Using public_suffix 5.0.1
Using typhoeus 1.4.0
Using addressable 2.8.1
Using webrick 1.8.1
Using jekyll-paginate 1.1.0
Using http_parser.rb 0.8.0
Using rb-inotify 0.10.1
Using rb-fsevent 0.11.2
Using rexml 3.2.5
Using terminal-table 3.0.2
Using nokogiri 1.14.2 (x86_64-linux)
Using safe_yaml 1.0.5
Using google-protobuf 3.22.0 (x86_64-linux)
Using em-websocket 0.5.3
Using listen 3.8.0
Using sass-embedded 1.58.3 (x86_64-linux-gnu)
Using html-proofer 3.19.4
Using kramdown 2.4.0
Using jekyll-watch 2.2.1
Using jekyll-sass-converter 3.0.0
Using kramdown-parser-gfm 1.1.0
Using jekyll 4.3.2
Using jekyll-archives 2.2.1
Using jekyll-seo-tag 2.8.0
Using jekyll-redirect-from 0.16.0
Using jekyll-sitemap 1.4.0
Fetching jekyll-theme-chirpy 5.5.2
Installing jekyll-theme-chirpy 5.5.2
Bundle complete! 6 Gemfile dependencies, 44 gems now installed.
Use `bundle info [gemname]` to see where a bundled gem is installed.
1 installed gem you directly depend on is looking for funding.
  Run `bundle fund` for details

4: Modify _config.yml and Preview the changes locally

Open _config.yml and modify the title and url

title: Yash Nigam

url: 'https://yash-nigam.github.io'

Preview the site contents locally:

yash@LearningPC:~/github/yash-nigam.github.io$ bundle exec jekyll s
Configuration file: /home/yash/github/yash-nigam.github.io/_config.yml
            Source: /home/yash/github/yash-nigam.github.io
       Destination: /home/yash/github/yash-nigam.github.io/_site
 Incremental build: disabled. Enable with --incremental
      Generating...
                    done in 1.634 seconds.
 Auto-regeneration: enabled for '/home/yash/github/yash-nigam.github.io'
    Server address: http://127.0.0.1:4000/

The local service will be published at http://127.0.0.1:4000.

5: Configure to Deploy the Website using GitHub Actions

Browse to your repository on GitHub, Select Settings > Pages. Then in the Source section (under Build and deployment), select GitHub Actions from the dropdown menu.

6: Commit and push changes to repo

Check git status

yash@LearningPC:~/github/yash-nigam.github.io$ git status
On branch main
Your branch is up to date with 'origin/main'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        modified:   _config.yml

no changes added to commit (use "git add" and/or "git commit -a")

add, Commit and push the _config.yaml file to repo

yash@LearningPC:~/github/yash-nigam.github.io$ git add _config.yml
yash@LearningPC:~/github/yash-nigam.github.io$ git commit -m "updated"
[main c30df68] updated
 1 file changed, 2 insertions(+), 2 deletions(-)
yash@LearningPC:~/github/yash-nigam.github.io$ git push
Username for 'https://github.com': yash-nigam
Password for 'https://yash-nigam@github.com':
Enumerating objects: 5, done.
Counting objects: 100% (5/5), done.
Delta compression using up to 12 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 334 bytes | 334.00 KiB/s, done.
Total 3 (delta 2), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (2/2), completed with 2 local objects.
To https://github.com/yash-nigam/yash-nigam.github.io.git
   aae12d1..c30df68  main -> main

7: Check the Actions Workflow and view the final website

Once the changes are pushed to the repo, a build and deploy workflow should be automatically triggered.

View the final website

8: Add Avatar, favicon, a new post and an sidebar tab.

Add Avatar and favicon images

Create a directory named img under assets in the root of repo yash-nigam.github.io/assets and favicons under img.
Follow the steps to generate and add faviconshttps://chirpy.cotes.page/posts/customize-the-favicon/

yash@LearningPC:~/github/yash-nigam.github.io/assets$ ls -ltr img/
total 96
drwxr-xr-x 2 yash yash  4096 Mar  3 23:28 favicons
-rw-r--r-- 1 yash yash    90 Mar  4  2023 favicon.ico:Zone.Identifier
-rw-r--r-- 1 yash yash 27019 Mar  4  2023 yn_small.jpg
-rw-r--r-- 1 yash yash 53754 Mar  4  2023 sidebar_bg.jpg
yash@LearningPC:~/github/yash-nigam.github.io$ ls -ltr assets/img/favicons/
total 168
-rw-r--r-- 1 yash yash 15531 Mar  4  2023 android-chrome-192x192.png
-rw-r--r-- 1 yash yash  8930 Mar  4  2023 mstile-150x150.png
-rw-r--r-- 1 yash yash   923 Mar  4  2023 favicon-16x16.png
-rw-r--r-- 1 yash yash   608 Mar  4  2023 safari-pinned-tab.svg
-rw-r--r-- 1 yash yash 15086 Mar  4  2023 favicon.ico
-rw-r--r-- 1 yash yash 66336 Mar  4  2023 android-chrome-512x512.png
-rw-r--r-- 1 yash yash 13736 Mar  4  2023 apple-touch-icon.png
-rw-r--r-- 1 yash yash  1270 Mar  4  2023 favicon-32x32.png

Also put your personal photo to be used as avatar in the img folder.
In the _config.yml file make following changes

avatar: /assets/img/yn_small.jpg

github:
  username: yash-nigam

Under the _posts folder create a file as follows:

yash@LearningPC:~/github/yash-nigam.github.io$ ls -ltr _posts/
total 4
-rw-r--r-- 1 yash yash 446 Mar  4 00:14 2023-03-04-FirstPage.md
yash@LearningPC:~/github/yash-nigam.github.io$ cat _posts/2023-03-04-FirstPage.md
---
title: "First Post"
date: 2023-03-03 12:00:00 +0530
categories: [TOP_CATEGORIE, SUB_CATEGORIE]
tags: [TAG]    , TAG names should always be lowercase
---

Under the _tabs folder create another markdown file of your choice.

yash@LearningPC:~/github/yash-nigam.github.io$ ls -ltr _tabs/
total 20
-rw-r--r-- 1 yash yash  47 Mar  3 15:00 tags.md
-rw-r--r-- 1 yash yash  56 Mar  3 15:00 categories.md
-rw-r--r-- 1 yash yash  55 Mar  3 15:00 archives.md
-rw-r--r-- 1 yash yash 194 Mar  3 15:00 about.md
-rw-r--r-- 1 yash yash 284 Mar  4 01:25 devto.md
yash@LearningPC:~/github/yash-nigam.github.io$ cat _tabs/devto.md
---
layout: page
title: dev.to Blog Posts
icon: fas fa-flask
order: 1
---
-----------
> <a href="https://dev.to/yashnigam/create-a-free-static-website-using-github-pages-and-jekyll-41a9" target="_blank">create a personal website using Github Pages with the Jekyll Framework</a>

Verify the site locally and view the changes in browser

bundle exec jekyll s

Commit and push the changes

yash@LearningPC:~/github/yash-nigam.github.io$ git status
On branch main
Your branch is up to date with 'origin/main'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        modified:   _config.yml

Untracked files:
  (use "git add <file>..." to include in what will be committed)
        _posts/2023-03-04-FirstPage.md
        _tabs/devto.md
        assets/css/
        assets/img/

no changes added to commit (use "git add" and/or "git commit -a")

Ignore any Zone.Identifier files as they are autogenerated

yash@LearningPC:~/github/yash-nigam.github.io$ git add --all
warning: CRLF will be replaced by LF in _tabs/devto.md.
The file will have its original line endings in your working directory

yash@LearningPC:~/github/yash-nigam.github.io$ git commit -m "Added changes"
[main 6eded15] Added changes
 24 files changed, 64 insertions(+), 8 deletions(-)
 create mode 100644 _posts/2023-03-04-FirstPage.md
 create mode 100644 _tabs/devto.md
 create mode 100644 assets/css/style.scss
 create mode 100644 assets/img/favicons/android-chrome-192x192.png
 create mode 100644 assets/img/favicons/android-chrome-512x512.png
 create mode 100644 assets/img/favicons/apple-touch-icon.png
 create mode 100644 assets/img/favicons/favicon-16x16.png
 create mode 100644 assets/img/favicons/favicon-32x32.png
 create mode 100644 assets/img/favicons/favicon.ico
 create mode 100644 assets/img/favicons/mstile-150x150.png
 create mode 100644 assets/img/favicons/safari-pinned-tab.svg
 create mode 100644 assets/img/sidebar_bg.jpg
 create mode 100644 assets/img/yn_small.jpg

yash@LearningPC:~/github/yash-nigam.github.io$ git push
remote: Resolving deltas: 100% (3/3), completed with 3 local objects.
To https://github.com/yash-nigam/yash-nigam.github.io.git
   c30df68..6eded15  main -> main

Once changes are pushed, make sure GitHub build and deploy action is successful

https://github.com/yash-nigam/yash-nigam.github.io/actions

Added changes
Build and Deploy #2: Commit 6eded15 pushed by yash-nigam

Now in the actual deployed site we can see that favicon, avatar, side bar tab and a new blog post is created successfully.

DEV Community: yash-nigam

Everything I Learned From My Claude Code Training

Everything I Learned From My Claude Code Training

Table of Contents

1. The Claude Ecosystem

2. The Agentic Loop (Query Engine)

Agentic Coding Tools Architecture

3. Models, Effort, and Slash Commands

Models

Slash commands for managing everything

4. Session Management

Practical guidance

5. Project Configuration

README.md

CLAUDE.md — Rules

.claude folder and settings

/init

Example: the actual CLAUDE.md used in this training repo

Example: Generating validateCreditCard and Its Test Case from a Prompt

6. Auto Mode vs. Plan Mode

7. Commands vs. Skills vs. Agents

Example: custom command — /standup

Metaprompting

Example: custom command — /raise-pr

Skills are auto-invoked

8. Code Review via Subagents

How to use a skill

Example: skill — deploy

Example: skill — write-tests

Creating a custom agent

Example: custom agent — security-analyst

The agent runs in the background

9. Multi-Agent Harness Concepts

A second pass: performance, test coverage, and code quality

Difference between agent and agentic

10. Hooks

Hook event flow

Example: PreToolUse hook on Bash

All hook handler types

Blocking dangerous commands

Notification on Slack

Sharing configuration

11. MCP (Model Context Protocol)

The concept of an MCP server

Why MCP exists

Three types of MCP server

12. End-to-End Automation

/create-issue

Now fix the issue

Commit the changes

13. Agent Teams

Example prompt

14. Key Takeaways

AI & ML Foundations: Cleaning Data, Building Regression & Classification Models in Python.

My 5-Day Journey into Machine Learning: From Data Cleaning to My First ML Models

A note about the source material

1. Overview: What did I actually learn in these five days?

2. What is Artificial Intelligence?

3. What is Machine Learning?

4. The most important ML vocabulary

4.1 Dataset

4.2 Row / observation / sample

4.3 Feature

4.4 Target

Telco

Insurance

4.5 X and y

5. The three major types of Machine Learning

5.1 Supervised Learning

Classification

Regression

6. Classification vs Regression

Classification

Regression

7. Unsupervised Learning

8. Reinforcement Learning

9. Different ways to approach an ML problem

Step 1 — What is the question?

Step 2 — What type of target do I have?

Step 3 — What data is available?

`.claude` folder and settings

`/init`

Example: Generating `validateCreditCard` and Its Test Case from a Prompt

Example: custom command — `/standup`

Example: custom command — `/raise-pr`

Example: skill — `deploy`

Example: skill — `write-tests`

Example: custom agent — `security-analyst`

Example: `PreToolUse` hook on Bash

`/create-issue`

`np.nan`

`np.log1p(x)`

`np.expm1(x)`

17. `df.head()` — What does the data look like?

18. `df.shape` — How much data do I have?

19. `df.info()` — What types of data do I have?

20. `df.describe()`

21. `df.sample()`

29. Countplot with `hue`

42. Why `drop_first=True`?

46. Why `random_state` matters

47. Why `stratify=Y` is useful for classification

62. Regression score: what does `.score()` mean?