DEV Community: Biki Kalita

The Auto-DJ Case Files: Building a Self-Learning Music Brain From Your YouTube Habit

Biki Kalita — Sun, 26 Jul 2026 13:22:39 +0000

A 3–5 month field guide for turning a browsing habit into an AI orchestra

Prologue: The Case That Started With a Skip Button

Every great investigation starts with something small and irritating. Yours starts on a Tuesday night. You're three hours deep into a YouTube rabbit hole — lo-fi beats, a Vietnamese city-pop deep cut, a live session you didn't mean to watch twice — and you think, someone should be paying attention to this.

That someone is about to be you. Not as a listener. As a detective.

Here's the case file: somewhere in your YouTube history is a pattern. A shape. You skip certain songs after four seconds. You replay others three times in a row without noticing. You always end up on synthwave at 11pm and never before 6pm. Right now, that pattern is invisible — smeared across a history page that YouTube barely lets you search, let alone reason about. Your job over the next few months is to build the instrument that makes it visible: a system that watches what you actually do, figures out what it means, and hands you back playlists that feel like they were made by someone who knows you. Because they were. That someone is a machine you built, trained on a dataset of one: you.

This won't be a weekend project. It's closer to building a small orchestra — a browser extension as your field agent, a backend as your evidence locker, a couple of ML models as your forensic lab, and a frontend as the stage where the final performance happens. Every piece depends on every other piece. That's what makes it hard, and it's also what makes it worth doing slowly, in public, with your own hands on every layer.

Let's start where every good documentary starts: by following one song from the crime scene to the credits.

Chapter 1: The Case of the Repeating Refrain

Following one song's journey

It's 9:47pm. You click a YouTube video: "Kiss of Life – Midas Touch (Live Performance)." You watch 41 seconds, skip to the chorus, watch another 90 seconds, then close the tab. Here's what needs to happen for that unremarkable moment to eventually shape a playlist called "Monday Chill":

The Witness (Browser Extension). A content script sitting quietly inside the YouTube tab notices the video element load. It reads the video ID, title, channel name, and duration from the page. It watches the <video> element's timeupdate, pause, seeked, and ended events like a stenographer taking notes: play at 0:00, seek to 1:32, pause at 3:04, total watch time 41s + 90s = 131s out of 210s (62%).
The Tip-Off (Event API). Every so often — batched, not on every tick, because nobody wants to be that extension hammering a server — the extension bundles these observations into a small JSON payload and POSTs it to your backend: {video_id, title, channel, watch_segments, timestamp, session_id}.
The Evidence Locker (Database). Your FastAPI backend receives the tip-off, validates it, and writes a raw event row into PostgreSQL. Nothing clever happens yet — this is just chain-of-custody. You never want to lose the raw signal, because every clever thing you do later will be derived from it, and derived things need to be recomputable.
The Lab Work (ML Pipelines, offline). On a schedule — say, once a night — a pipeline wakes up, looks for new raw events, and does the actual detective work:
- Is this even a music video? (classification)
- What song is this, really? (metadata resolution via title parsing + external lookups)
- What does it sound/feel like? (audio embedding + mood tagging)
- How does this fit with everything else you've watched? (behavioral signal: implicit rating from watch %, skip pattern, replay count, time of day)
The Case Board (Vector store + relational tables). The song's audio embedding gets written to a vector column (via pgvector), its metadata to a normal relational table, and its behavioral score gets updated incrementally — like a detective updating a suspect's file every time new evidence comes in.
The Reveal (Playlist Generator). Periodically, a clustering job looks at the whole case board — all your songs, their embeddings, their mood tags, their time-of-day patterns — and groups them. A cluster of moody, low-energy, evening-heavy tracks with "chill," "lofi," or "acoustic" tags gets surfaced and, because most of its listens happened on Mondays, gets named "Monday Chill."
The Broadcast (Frontend + Player). Your React app queries the backend, gets back a list of playlists with their tracks, and renders them. When you hit play, it doesn't awkwardly embed a YouTube iframe — it either streams audio you've legitimately extracted for personal use, or falls back to a controlled YouTube player, and it logs that playback too, feeding the loop again.

That's the whole organism, end to end. Six weeks from now this will feel obvious. Right now, notice the shape: observe → transmit → store → understand → organize → present, with a feedback loop closing the circle back to step 1. Every phase of this guide builds one of these organs.

The architecture, explained like you're new here

[Browser: YouTube tab]
        |
   content script  (watches video element)
        |
   background script (batches + sends events)
        |
        v
   Event Ingestion API  (FastAPI, /events endpoint)
        |
        v
   Raw Events Table  (PostgreSQL)
        |
   +----+----------------------------+
   |                                 |
   v                                 v
Metadata & Classification      Audio & Behavior Pipelines
(is it music? what song?)      (embeddings, mood, implicit score)
   |                                 |
   +----------------+----------------+
                    v
        Enriched Track Store
   (PostgreSQL relational + pgvector embeddings)
                    |
                    v
          Clustering / Playlist Engine
        (HDBSCAN, labeling, scheduling)
                    |
                    v
             Playlist API (FastAPI)
                    |
                    v
           React Frontend + Player
     (renders playlists, plays audio, logs playback)
                    |
                    +--> feeds back into Raw Events

Plain English: a browser extension is the only part of this system that can see raw YouTube behavior, so it's your sensor. Everything after that is you progressively distilling raw noise into structured meaning — first "what is this," then "what does it sound like," then "what does it mean that you watched it," then "what group of songs does this belong to." A pipeline is just a name for "a sequence of these distillation steps that runs on a schedule instead of live," which matters because audio analysis and ML inference are too slow to run inline while you're browsing.

Why these particular tools (and not the flashier alternatives)

FastAPI, not Django or raw Flask. You'll be exposing a mix of simple CRUD endpoints and ML inference endpoints. FastAPI's automatic request validation via Pydantic means fewer "why did my extension send a malformed payload and silently corrupt my data" nights, and its async support matters once you're streaming audio or calling external APIs (metadata lookups) without blocking.
PostgreSQL + pgvector, not a separate vector database. You could stand up Pinecone or Weaviate. For a personal-scale project (thousands, maybe tens of thousands of tracks — not billions), that's solving a problem you don't have yet, at the cost of one more service to run, secure, and back up. pgvector lets you keep relational metadata (title, channel, mood tags) and embeddings in the same database, so a query like "find songs similar to this one, but only ones tagged 'study' and watched after 9pm" is one SQL query with a vector distance operator, not a join across two systems.
A CNN (or a pretrained audio model) for audio classification, not a giant end-to-end transformer. Audio classification at the "is this speech, music, or noise" and "what does this sound like" level has been solved well by relatively small convolutional architectures operating on spectrograms (think YAMNet, VGGish). You don't need GPT-scale compute to tell that a track is high-energy electronic versus a mellow acoustic set. This is a deliberate choice to keep your compute budget human-sized — you'll likely run this on a laptop or a modest cloud instance, not a GPU cluster.
Transformers for tagging, not for audio itself. Where transformers genuinely earn their complexity is text — parsing messy YouTube titles ("Artist - Song (Official Music Video) [4K]"), zero-shot classifying mood from lyrics or descriptions, and understanding channel/description context. Hugging Face's zero-shot classification pipelines let you throw a candidate label set ("chill," "energetic," "sad," "study," "workout") at a piece of text and get calibrated-ish scores back without training anything — perfect for a first pass, with fine-tuning as a stretch goal.
A browser extension, not scraping Google Takeout alone. Takeout gives you history, but it's after the fact, coarse, and doesn't include watch percentage, skips, or replays — exactly the implicit-feedback signal your recommendation logic needs. The extension is how you get behavioral data, not just occurrence data. Takeout is still useful — as bootstrap data and backfill, which Chapter 4 covers.

Notice the pattern in every choice above: pick the tool that matches the actual scale and shape of your problem, not the tool with the best conference talk. You're building a personal system for one very well-instrumented user (you), not a production service for millions. Keep saying that sentence to yourself whenever you're tempted to add Kubernetes.

The living organism

Here's the mental model to carry through the rest of this guide: this system has a nervous system (the extension + event API, always sensing), a subconscious (the nightly pipelines, working while you sleep, turning raw sensation into structured understanding), and a conscious voice (the frontend, presenting a curated, explainable result). If you break the nervous system, nothing gets sensed. If you break the subconscious, sensations pile up unprocessed. If you break the conscious voice, all that understanding has nowhere to be expressed. You'll spend the next several months building each of these organs, and — this is the important part — you'll build them so that each one can be tested and demoed on its own, before they're wired together. That's the difference between a project that has a working prototype at every milestone and a project that only "works" the week before a deadline, if you're lucky.

Chapter 2: The Learning Roadmap — Five Months, Six Phases

You know basic Python and JavaScript. You don't yet know how these pieces talk to each other, or how to think about audio and behavior as data. So the roadmap is designed to stack skills like sediment layers — each phase leaves you with a thin, ugly, but genuinely working slice of the system, and each phase's slice becomes the foundation the next phase builds on. You will never be staring at a blank canvas with six unconnected concepts in your head. You will always be extending something that already runs.

Treat each phase as 2–4 weeks depending on how much time you have per week. If you have 10+ hours/week, aim for the shorter end; if you're doing this around a full-time job, take the longer end. Five months is the "comfortable, learn deeply" pace. Three months is achievable if you already move fast and cut a few of the "stretch" items.

Phase 0: Recon (3–5 days, before Week 1 really starts)

Goal: Set up the skeleton of every future piece, even if each piece does almost nothing.

Create the repo structure: extension/, backend/, ml/, frontend/, infra/.
Get a FastAPI "hello world" running locally with one route, GET /ping.
Get a bare Chrome extension loaded in developer mode that does nothing but log "extension loaded" to the console on any youtube.com page.
Spin up a local PostgreSQL instance (Docker is your friend here — docker run postgres).

Learning pit stop:

15 min: FastAPI's official "First Steps" tutorial — just enough to run a server.
20 min: Chrome's Extension "Getting Started" tutorial — focus on the manifest and content scripts, skip the rest for now.

You know you're ready when: you can open a YouTube tab, see "extension loaded" in the console, and hit http://localhost:8000/ping in your browser and get {"status": "ok"}. That's it. Two independent heartbeats.

Phase 1: The Witness — Event Tracking (Weeks 1–3)

Goal: A Chrome extension that logs real YouTube watch events, and a backend endpoint that stores them.

Skills gained: DOM event listeners, Chrome extension messaging (content script <-> background script), REST API design basics, relational schema design for event logs.

What you'll build:

A content script that attaches listeners to the YouTube <video> element (play, pause, seeked, ended, timeupdate sampled every few seconds — not every frame, that's overkill and will flood your database).
A background script that batches events (e.g., flush every 30 seconds or every 10 events) and sends them via fetch to your backend.
A POST /events FastAPI endpoint with a Pydantic model validating the payload, writing rows into a raw_events table.
A minimal schema: raw_events(id, video_id, title, channel, event_type, position_seconds, video_duration_seconds, session_id, created_at).

Architecture decision point: Do you send every micro-event (every timeupdate tick) to the server, or do you compute a per-video summary (total watch time, max position reached, number of seeks) in the extension and send one summary per video? Sending raw ticks gives you more analytical flexibility later (you could reconstruct exact skip behavior) at the cost of way more data and server load. Summarizing in-extension is simpler and cheaper but throws away granularity. Recommendation for a learning project: send raw ticks for now — you're optimizing for "what can I learn from this," not production efficiency, and you can always summarize later. You can throttle it (sample every 5 seconds) to keep volume sane.

Learning pit stops:

20 min video: search "Chrome extension message passing background content script" — get comfortable with chrome.runtime.sendMessage.
Skim MDN's HTMLMediaElement events page — you need timeupdate, pause, seeked, ended, play.
10 min: Pydantic's "Models" quickstart — you'll lean on this for every endpoint from here forward.

Skeleton to work from (not a finished file):

// content_script.js -- the witness's notebook
function attachListeners(videoEl, videoId) {
  let lastPosition = 0;

  videoEl.addEventListener('timeupdate', () => {
    // TODO: throttle this -- don't fire on every tick
    // TODO: send {event_type: 'progress', position: videoEl.currentTime}
  });

  videoEl.addEventListener('seeked', () => {
    // TODO: compare videoEl.currentTime to lastPosition
    // to detect skip-forward vs rewind
  });

  videoEl.addEventListener('pause', () => {
    // TODO: send pause event with position
  });
}

// TODO: figure out how to get videoId from the page URL
// TODO: figure out how to detect when YouTube's SPA navigation
//       changes video without a full page reload (hint: MutationObserver
//       on the title element, or listen for yt-navigate-finish)

# backend/main.py -- the evidence intake desk
from fastapi import FastAPI
from pydantic import BaseModel

app = FastAPI()

class RawEvent(BaseModel):
    video_id: str
    title: str
    channel: str | None = None
    event_type: str  # 'play' | 'pause' | 'seeked' | 'ended' | 'progress'
    position_seconds: float
    video_duration_seconds: float | None = None
    session_id: str

@app.post("/events")
def ingest_event(event: RawEvent):
    # TODO: open a DB connection (consider SQLAlchemy or raw psycopg)
    # TODO: INSERT into raw_events
    # TODO: return a lightweight ack, don't do heavy work here --
    #       this endpoint must stay fast, it's on the hot path
    ...

Milestone / "you know you're ready when": you watch three different YouTube videos for real, then query your database and see a believable, timestamped trail of what you actually did -- including at least one skip you remember making. Take a screenshot. This is the first time the system has seen you.

Phase 2: The Classifier -- Is This Even Music? (Weeks 4-6)

Goal: Given a raw event's title/channel/duration, decide whether it's a music video at all (versus a podcast, tutorial, vlog) -- and if it is, extract a clean artist/song guess.

Skills gained: Text classification basics, regex-as-first-resort vs ML-as-fallback thinking, working with the YouTube Data API, zero-shot classification with Hugging Face.

What you'll build:

A nightly (or on-demand) script that reads unprocessed rows from raw_events, groups by video_id, and for each unique video, runs a classification step.
First pass -- heuristics, not ML. Channel name contains "Official," "VEVO," "Music"? Title matches patterns like Artist - Title (Official ...)? These heuristics alone will correctly classify a surprising majority of real YouTube music content. Write these first. Resist the urge to reach for a model before you've squeezed out what regex and metadata can do -- this is a core "how to think like an engineer" lesson: cheap and boring beats fancy and unnecessary, every time it's available.
Second pass -- zero-shot text classification for the ambiguous remainder, using Hugging Face's zero-shot-classification pipeline with candidate labels like ["music video", "podcast", "tutorial", "vlog", "gaming"].
A tracks table that stores the resolved artist/title guess, separate from the raw event log -- this is your first "enriched" table, the first sign of the subconscious turning sensation into understanding.

Architecture decision point: Regex-first-then-ML, or ML-first-then-regex-cleanup? Going regex-first means your model only has to handle genuinely ambiguous cases, which is both faster (you skip inference on 70%+ of videos) and more accurate (you're not asking a general-purpose model to outperform a rule you could write in one line for "channel name ends in VEVO"). This is a recurring theme: use ML for the residual, not the whole problem.

Learning pit stop:

15 min: Hugging Face's zero-shot classification pipeline docs -- run their example locally before writing your own.
Skim: the YouTube Data API "Videos: list" reference -- you'll want snippet.description and snippet.tags for extra signal, and eventually contentDetails.duration.

Skeleton:

# ml/classify.py -- the detective's first read of the case file
MUSIC_CHANNEL_HINTS = ["vevo", "official", "records", "music"]

def heuristic_is_music(title: str, channel: str) -> bool | None:
    # Return True/False if confident, None if unsure (defer to ML)
    channel_lower = channel.lower()
    if any(hint in channel_lower for hint in MUSIC_CHANNEL_HINTS):
        return True
    # TODO: add a regex for "Artist - Title" pattern
    # TODO: add a denylist for obvious non-music (podcast, vlog, tutorial keywords)
    return None

def ml_classify(title: str, description: str) -> str:
    # TODO: load huggingface zero-shot pipeline once, reuse across calls
    #       (loading it per-call will make this painfully slow)
    # candidate_labels = ["music video", "podcast", "tutorial", "vlog", "other"]
    ...

def extract_artist_title(title: str) -> tuple[str | None, str | None]:
    # TODO: try a regex like r"^(.*?)\s*-\s*(.*?)(\(|\[|$)"
    # TODO: fall back to None, None if it doesn't match -- don't force a bad guess
    ...

Milestone: run the classifier over a week's worth of real watch history. Print a summary: "Classified 42 videos: 31 music, 11 other." Manually eyeball 10 of them -- if your accuracy feels roughly 80%+, move on; ML classifiers don't need to be perfect, because Phase 5's clustering step will naturally down-weight noise. Perfectionism here is a trap -- resist polishing this past "good enough to build on."

Phase 3: The Forensic Lab -- Audio & Metadata Enrichment (Weeks 7-10)

Goal: For every confirmed music video, extract audio and metadata rich enough to power mood tagging and similarity search.

Skills gained: Working with yt-dlp, basic digital signal processing (spectrograms, mel scales), running a pretrained audio embedding model, external metadata APIs (MusicBrainz or similar), and your first taste of "architecture decision with a real trade-off."

What you'll build:

A pipeline step that, given a video_id, downloads (or streams) audio using yt-dlp, for local processing only -- not redistribution. This is the moment to have your privacy/ethics conversation with yourself; more in Chapter 4.
A function that converts raw audio into a mel spectrogram -- the standard "image-like" representation that lets you use image-flavored models (CNNs) on sound.
Architecture decision point, spelled out for you: you could (a) use a pretrained audio embedding model like YAMNet or OpenL3 to get a general-purpose embedding vector per track with zero training, or (b) train a small CNN yourself on your own labeled data (mood-tagged tracks) to get embeddings tuned to your taste categories. (a) is faster, works immediately, and is what you should ship first. (b) is the better learning experience and gives you a genuinely custom system -- a great Phase 3.5 stretch goal once (a) is working end to end. Don't skip straight to (b); you'll spend three weeks on a training loop for a payoff you could've had in three days.
Store the resulting embedding vector in a pgvector column on your tracks table.
A metadata enrichment step: look up the resolved artist/title against MusicBrainz's API (free, no key needed for light use) to pull canonical artist name, genre tags if available, and release year.

Learning pit stops:

20 min video: search "mel spectrogram explained simply" -- you need the intuition (frequency content over time, perceptually weighted), not the full DSP math.
Read the yt-dlp README's "Usage and Options" section, specifically audio extraction flags (-x --audio-format).
Skim: YAMNet's model page on TensorFlow Hub -- its usage example is short and you can adapt it directly.

Skeleton:

# ml/audio_features.py -- the forensic lab bench
import yt_dlp
import librosa
import numpy as np

def download_audio(video_id: str, out_dir: str) -> str:
    # TODO: configure yt_dlp options for audio-only extraction
    # ydl_opts = {'format': 'bestaudio/best', 'outtmpl': f'{out_dir}/%(id)s.%(ext)s', ...}
    # NOTE: this is for local, personal analysis only -- see Chapter 4
    ...

def compute_mel_spectrogram(audio_path: str) -> np.ndarray:
    # TODO: y, sr = librosa.load(audio_path)
    # TODO: librosa.feature.melspectrogram(y=y, sr=sr, n_mels=128)
    # TODO: convert to log scale (librosa.power_to_db) -- raw mel values
    #       span orders of magnitude, log scale is what models expect
    ...

def get_embedding(audio_path: str) -> np.ndarray:
    # TODO: load a pretrained model (YAMNet via tensorflow_hub, or
    #       openl3.get_audio_embedding) and run inference
    # TODO: pool frame-level embeddings into a single fixed-length vector
    #       (mean pooling is a fine first choice)
    ...

Architecture decision point (data storage): store raw audio permanently, or delete it after extracting the embedding? Storing it burns disk and raises copyright/privacy questions for no real benefit -- you only need the embedding, not the waveform, for everything downstream. Recommendation: extract-then-delete, keep only the derived vector. Write this down as a design principle now; it'll save you a difficult conversation with yourself in Month 4 when your disk is full of MP3s you don't need.

Milestone: pick five songs you know well, run them through the pipeline, and eyeball their embeddings' pairwise cosine similarity. Two songs by the same artist in the same genre should be closer to each other than to a wildly different genre. If your embeddings pass this smell test, you have a forensic lab that actually works. Screenshot the similarity matrix -- this is the moment the system starts to have taste.

Phase 4: The Behavioral Profiler -- Learning What You Actually Like (Weeks 11-14)

Goal: Turn raw watch behavior into an implicit "how much did you like this" score, and build your first learned model of taste.

Skills gained: Implicit feedback theory, feature engineering from event logs, neural collaborative filtering, thinking in terms of "signal vs noise" in behavioral data.

What you'll build:

A feature engineering step that, per track, computes: percent watched, replay count within a session, skip-within-first-10-seconds flag, time-of-day distribution, day-of-week distribution.
An implicit score -- not a 1-5 star rating (you never gave one), but a derived number. A simple, honest starting formula:

implicit_score = w1 * completion_ratio
                + w2 * log(1 + replay_count)
                - w3 * early_skip_penalty

where completion_ratio = watched_seconds / video_duration, capped at 1.0, early_skip_penalty is 1 if you skipped within the first 10 seconds and never returned, else 0, and w1, w2, w3 are weights you pick by hand at first (try w1=1.0, w2=0.5, w3=0.8) and refine once you have enough data to eyeball whether the resulting rankings feel right. This is implicit feedback in a nutshell: you're inferring preference from behavior, not asking for it directly, and every implicit signal is a proxy, not ground truth -- a song you love but skip because you're not in the mood right now looks identical, in raw data, to a song you dislike. Time-of-day and mood context (Phase 5) is what starts to disambiguate the two.

The math of implicit feedback, briefly: classic explicit recommender systems (star ratings) treat missing data as "unknown." Implicit feedback systems (like this one) treat everything as a signal -- even the absence of a play is information (you had the chance to watch it and didn't). The seminal framing here, worth reading about, is "confidence-weighted preference": you're not just modeling whether you like something, you're modeling how confident the system should be in that preference, where confidence scales with how many times you've interacted with a track.
A first neural collaborative filtering sketch, once you have a few hundred (track, implicit_score, context) rows. Neural CF replaces the classic matrix-factorization approach (learn a vector per user, a vector per item, dot-product them to predict preference) with small embedding layers you can extend to include context features (time of day, mood tag) alongside the learned latent vectors.

Learning pit stops:

25 min: search "implicit feedback recommender systems explained" -- get comfortable with the idea that "no interaction" is still data.
Skim: Keras's Embedding layer docs -- this is the core building block of neural CF.
Optional deeper dive (30 min): the original "Collaborative Filtering for Implicit Feedback Datasets" paper's abstract and intro -- you don't need the full math, just the framing.

Skeleton (Keras):

# ml/behavior_model.py -- teaching the system your taste, one skip at a time
import tensorflow as tf
from tensorflow.keras import layers, Model

def build_neural_cf(n_tracks: int, embedding_dim: int = 32) -> Model:
    track_input = layers.Input(shape=(1,), name="track_id")
    context_input = layers.Input(shape=(4,), name="context_features")  # e.g. hour_of_day, day_of_week, recent_genre_onehot...

    track_embedding = layers.Embedding(n_tracks, embedding_dim)(track_input)
    track_vec = layers.Flatten()(track_embedding)

    # TODO: concatenate track_vec with context_input
    # TODO: pass through 1-2 Dense layers with ReLU
    # TODO: final Dense(1, activation='sigmoid') to predict a
    #       normalized implicit_score in [0, 1]

    ...
    return Model(inputs=[track_input, context_input], outputs=..., name="neural_cf")

def compute_implicit_score(row) -> float:
    completion_ratio = min(row["watched_seconds"] / row["duration_seconds"], 1.0)
    replay_bonus = 0.5 * (row["replay_count"] ** 0.5)  # diminishing returns
    early_skip_penalty = 0.8 if row["early_skip"] else 0.0
    return max(0.0, completion_ratio + replay_bonus - early_skip_penalty)

Architecture decision point: do you train one global model, or treat this as pure feature engineering with no trained model at all (just weighted heuristics)? With a personal dataset of maybe a few thousand tracks and a few weeks of history, you may not have enough data for a neural model to meaningfully outperform a well-tuned heuristic score. Recommendation: ship the heuristic first (it's basically free), and treat the neural CF model as a learning exercise you compare against the heuristic -- a great "does my fancy model actually beat my simple baseline?" experiment, which is one of the most important habits in applied ML.

Milestone: for a handful of tracks you have strong feelings about, check whether your implicit score ranks them the way you'd expect -- your most-replayed song should sit near the top, something you skipped every time near the bottom. If the ranking basically matches your gut, you've built a taste profile that didn't exist an hour ago in any explicit form.

Phase 5: The Case Board -- Clustering & Playlist Generation (Weeks 15-18)

Goal: Group tracks into meaningful clusters and turn each cluster into a named, explainable playlist.

Skills gained: Unsupervised clustering (HDBSCAN, K-Means), cluster labeling strategies, combining embeddings from different modalities (audio + behavior + text mood tags), scheduling recurring jobs.

What you'll build:

A combined feature vector per track: concatenate (or weighted-sum) the audio embedding, a mood-tag one-hot/multi-hot vector (from zero-shot classification against labels like "chill," "energetic," "sad," "focus," "romantic," "aggressive"), and behavioral context features (average time-of-day, day-of-week distribution).
A clustering pass using HDBSCAN over that combined vector space. Why HDBSCAN over plain K-Means: K-Means forces every point into a cluster and requires you to pre-specify the number of clusters -- a bad fit here, because you genuinely don't know how many "moods" exist in your listening history, and forcing an odd one-off track into a cluster it doesn't belong in produces bad playlists. HDBSCAN discovers the number of clusters itself and explicitly labels outliers as noise (cluster = -1) rather than cramming them somewhere.
A cluster-labeling step: for each cluster, look at the most common mood tags among its members, the most common time-of-day, and generate a human name via simple rules ("if dominant mood is 'chill' and dominant time is evening -> name candidates: 'Evening Chill', 'Wind Down'"). This is a fun place to eventually swap in an LLM call ("given these five representative songs and these mood tags, suggest a playlist name") as a stretch goal.
Specialized playlist types layered on top of the general clustering:
- Most-played: pure sort by implicit_score / play_count, no clustering needed.
- Mood playlists: direct output of the HDBSCAN clusters.
- Study / focus: filter tracks where the zero-shot classifier's "instrumental" or "low-vocal-density" signal is high and mood tag includes "calm" or "focus" -- this is a good spot to add a simple vocal-presence heuristic (spectral features like zero-crossing rate can proxy for vocal presence without a dedicated model).
- Travel: filter by tracks played during sessions your extension can tag as "away from home" if you choose to add coarse geolocation (optional, privacy-sensitive -- see Chapter 4) -- or more simply, tracks played during longer, uninterrupted sessions (a proxy for commutes or trips).
- Language: filter using a language-detection pass over the (translated or original) lyrics/title text, using a lightweight library like langdetect or a Hugging Face language-ID model.

Learning pit stops:

20 min: read HDBSCAN's own documentation "How HDBSCAN Works" page -- it's written for exactly this level of newcomer and has excellent visuals.
Skim: scikit-learn's clustering comparison page to see K-Means vs DBSCAN vs HDBSCAN side by side on toy datasets -- seeing the failure modes visually will save you hours of confusion later.

Skeleton:

# ml/cluster.py -- the case board, where scattered evidence becomes a story
import hdbscan
import numpy as np

def build_combined_vector(audio_emb, mood_vec, behavior_features) -> np.ndarray:
    # TODO: consider normalizing each component before concatenating --
    #       otherwise a component with larger raw magnitude (e.g. audio_emb)
    #       will dominate the distance metric HDBSCAN uses
    ...

def cluster_tracks(vectors: np.ndarray, min_cluster_size: int = 5):
    clusterer = hdbscan.HDBSCAN(min_cluster_size=min_cluster_size, metric="euclidean")
    labels = clusterer.fit_predict(vectors)
    # labels == -1 means "noise" -- HDBSCAN is telling you this track
    # doesn't fit cleanly anywhere yet. That's useful information, not a bug.
    return labels

def name_cluster(track_rows_in_cluster) -> str:
    # TODO: find most common mood tag, most common time-of-day bucket
    # TODO: map (mood, time_bucket) -> candidate name via a lookup table
    # TODO: fall back to "Mix #<id>" if nothing matches cleanly
    ...

Architecture decision point: re-cluster from scratch every time, or incrementally assign new tracks to existing clusters? Full re-clustering is simpler to reason about and is what you should build first -- run it nightly over your whole track history. Incremental assignment (using each cluster's centroid or a trained classifier) is faster and preserves playlist identity over time (your "Monday Chill" playlist doesn't get renamed every week), and is a great optimization once the naive approach is working and you notice it's slow or that playlist churn is annoying you.

Milestone: run the full pipeline end to end and get back a JSON structure of named playlists with real tracks in them. Take a screenshot of the raw output -- you now have an AI DJ's set list, even before there's a frontend to look at it.

Phase 6: The Broadcast -- Web Player & Full Integration (Weeks 19-22)

Goal: A React frontend that displays your playlists and actually plays music, wired end-to-end to everything you've built.

Skills gained: React component architecture, working with an audio player (HTML5 <audio> or the YouTube IFrame API), API integration patterns, and the specific discipline of wiring together five already-working pieces without breaking any of them.

What you'll build:

A GET /playlists endpoint that returns your generated playlists with track metadata.
A React app with a playlist grid, a track list per playlist, and a persistent bottom player bar (the "Spotify-like" feel).
Playback, decision point spelled out: you have two real options. (a) Embed the YouTube IFrame Player API and drive it programmatically -- simplest, fully compliant with YouTube's terms, but visually and functionally you're still "a YouTube player with a nicer wrapper," and you can't easily do gapless playback or true background audio. (b) Serve audio you've extracted yourself, from your own backend, for genuinely personal, non-redistributed use -- gives you a true from-scratch audio player experience (waveforms, gapless transitions, real <audio> element control) but carries real copyright and terms-of-service responsibilities you must take seriously and keep strictly personal/local, never exposed publicly. Recommendation: build the IFrame API version first, since it's unambiguously fine to build and ship as a personal tool, and treat the self-hosted audio player as an explicitly-labeled "local-only, for-my-eyes-only experiment" if you choose to explore it, never something you deploy publicly.
A "similar tracks" widget: given the currently playing track's embedding, query pgvector for nearest neighbors (ORDER BY embedding <-> :current_embedding LIMIT 5) and render them as suggestions.
Wire the frontend's play events back into the extension's event pipeline (Chapter 1's feedback loop closes here) -- plays that originate in your own app should count as behavioral signal too.

Learning pit stops:

20 min: the YouTube IFrame Player API "Getting Started" guide -- focus on loadVideoById, playVideo, and the onStateChange event.
Skim: any short React hooks refresher if it's been a while (useState, useEffect, useContext for a simple global "currently playing" state).

Skeleton:

// frontend/src/components/Player.jsx -- the stage where the case gets solved
import { useState, useEffect, useRef } from "react";

export default function Player({ currentTrack }) {
  const playerRef = useRef(null);

  useEffect(() => {
    if (!currentTrack) return;
    // TODO: if using YouTube IFrame API:
    //   playerRef.current.loadVideoById(currentTrack.video_id);
    // TODO: log a 'play' event back to your /events endpoint here,
    //       reusing the same schema Phase 1 established
  }, [currentTrack]);

  return (
    <div className="player-bar">
      {/* TODO: track title, artist, progress bar */}
      {/* TODO: mount the YouTube IFrame player (hidden or small) here */}
    </div>
  );
}

# backend/main.py (addition) -- serving the case board to the courtroom
@app.get("/playlists")
def get_playlists():
    # TODO: query your clusters/playlists table
    # TODO: for each playlist, include a handful of representative tracks
    #       and the playlist's dominant mood tags for display
    ...

@app.get("/tracks/{track_id}/similar")
def similar_tracks(track_id: str, limit: int = 5):
    # TODO: SELECT * FROM tracks ORDER BY embedding <-> (
    #         SELECT embedding FROM tracks WHERE id = :track_id
    #       ) LIMIT :limit
    ...

Milestone -- the big one: open your app, see real named playlists built from your real YouTube behavior, click play, and hear music. Take a screenshot of "Monday Chill" (or whatever your system named it) with real tracks in it. This is the moment described in the opening story finally closes the loop -- you are, quite literally, looking at the output of a detective story you wrote about yourself.

Chapter 3: The Brain of the System — Meet Your Neural Concierge

Phases give you a build order. This chapter zooms into the thinking tools you'll reach for repeatedly — the habits of mind that separate "I copied a tutorial" from "I understand what I built."

How to search, read docs, and debug like you've done this before

You haven't done this before, and that's fine — nobody starts out knowing how to google effectively for a niche integration. Here's the actual process, made explicit:

Name the smallest unit of your confusion. Not "how do I do audio classification" — that's a whole field. Instead: "how do I get a fixed-length vector out of a variable-length audio file using a pretrained model." That sentence is close to a good search query.
Search for the library name plus the verb, not the concept plus the verb. "yt-dlp extract audio python" beats "how to download youtube audio." Library-specific queries land you on docs and GitHub issues where someone already hit your exact wall.
Read the signature before the prose. When you land on a function's docs, look at its parameters and return type first. Half the time that alone answers your question faster than the paragraph above it.
When you're stuck for more than 20-30 minutes, write down what you expected to happen and what actually happened, in one sentence each. This is the single highest-leverage debugging habit there is — most of the time, writing that sentence reveals the bug before you've even searched anything, because it forces you to notice the gap between your mental model and reality.
Reproduce in isolation. If your pipeline breaks on the audio-embedding step, don't debug it inside the full nightly job — copy the failing call into a throwaway script or a notebook cell with one hardcoded input. You want the shortest possible loop between "change something" and "see the result."

You'll use this loop dozens of times across the next five months. It's worth more than any specific fact in this guide.

Trade-off thinking, one more time with feeling

Every phase above included at least one "architecture decision point," and that's not decoration — it's the actual skill this project is teaching you. Real systems are built from a sequence of decisions where there's no objectively correct answer, only a right answer for your constraints. The questions to ask yourself every time are the same four:

What does each option cost me right now (time, complexity, new concepts to learn)?
What does each option cost me later (technical debt, rework, scaling pain)?
Do I actually have the problem this option solves, or am I solving a problem I imagine having?
Can I try the cheap option first and upgrade later, or is this a decision that's expensive to reverse?

Notice that almost every recommendation in this guide follows the same shape: start with the boring, cheap, well-understood option; treat the fancy option as a deliberate, isolated upgrade once you've felt the limits of the simple one. That's not a lack of ambition — it's how you make sure the ambition is spent on the parts of the project that are genuinely novel (your specific taste model, your specific clustering logic) instead of getting burned on generic infrastructure choices that a thousand tutorials have already solved for you.

NLP for mood: zero-shot first, fine-tuning as a deliberate upgrade

You'll use Hugging Face's zero-shot-classification pipeline in two places: initial content classification (Phase 2) and mood tagging (Phase 5). It's worth understanding why zero-shot works at all — the underlying model (typically an NLI, natural-language-inference, model like BART-MNLI) was trained to judge whether one sentence entails another. Zero-shot classification is a clever repurposing: it turns "does this song feel energetic?" into "does the premise 'this text describes energetic music' entail the hypothesis built from your candidate label," and ranks labels by entailment confidence. You get calibrated-ish multi-label scores over any label set you dream up, with zero training.

The catch: zero-shot models are working from text — titles, descriptions, maybe lyrics if you fetch them — not from the audio itself. A song titled "Sunny Day Vibes" will zero-shot-classify as upbeat even if the actual track is a somber ballad using the title ironically. This is exactly why Phase 5's combined vector blends the zero-shot mood tags with the audio embedding — text tells you what the creator labeled the song as, audio tells you what it actually sounds like, and behavior tells you how you actually responded to it. None of the three signals is trustworthy alone; together they triangulate something closer to the truth.

When to fine-tune instead of relying on zero-shot: once you've hand-corrected mood tags on maybe 200-300 tracks (a natural byproduct of using your own app and noticing "this is tagged wrong"), you have enough labeled data to fine-tune a small classifier — even a lightweight one trained on top of frozen text embeddings — specifically on your vocabulary of moods and your taste in how songs get labeled. This is a genuinely great Month 4-5 stretch goal: it's the moment your mood tagger stops being "a generic pretrained model" and starts being "a model that knows your ears."

The math of implicit feedback, one level deeper

Section from Phase 4 gave you a working formula. Here's the conceptual core worth sitting with: in explicit feedback (star ratings), the model's job is to predict a rating, and the loss function punishes wrong ratings directly. In implicit feedback, there's no rating to predict — only binary or continuous signals of engagement. The standard reframe (going back to Hu, Koren & Volinsky's foundational work) is to treat every (user, item) pair as having a binary preference (did they like it, roughly, yes/no) plus a confidence in that preference, where confidence grows with how much engagement evidence you've observed. A song you've watched once for 10 seconds contributes a weak, low-confidence data point. A song you've replayed eleven times contributes a strong, high-confidence one. Your loss function (if you get to training a real model) should weight errors on high-confidence pairs more heavily than errors on low-confidence ones — getting a well-established favorite's ranking wrong is a bigger problem than getting an ambiguous one-play track wrong.

Practically, for a personal-scale project, you don't need to implement the full weighted matrix factorization machinery to benefit from this idea — just make sure your implicit_score formula and your neural CF's context features both account for "how many times have I actually interacted with this" as a first-class signal, not an afterthought.

Neural collaborative filtering, sketched a little further

The Phase 4 skeleton gave you the shape: an embedding layer per track, concatenated with hand-engineered context features, passed through dense layers to a final prediction. A few things worth understanding about why this works before you write more of it:

The embedding layer is learning a compressed "taste coordinate" for each track — not from any explicit attribute you gave it, but purely from patterns in which tracks tend to get high implicit scores together, in similar contexts. Two tracks can end up with similar embeddings even if their audio sounds nothing alike, if you consistently listen to them in the same moods/times.
This is a genuinely different signal from the audio embedding. Audio embeddings capture "sounds like." CF embeddings capture "gets consumed like." A great expansion once both exist: concatenate them (as Phase 5 does) so your clusters reflect both what things sound like and how you actually use them.
With a small personal dataset, expect this model to overfit fast. That's not a failure — it's expected, and it's a good opportunity to learn regularization basics (dropout between dense layers, L2 weight decay on the embedding) and to practice train/validation splitting even when your dataset is uncomfortably small. Keep a held-out slice of your most recent week's data and check whether the model's predicted scores actually track your real subsequent behavior — that's your only honest signal that any of this is working.

Chapter 4: The Data Story — Where Evidence Lives

Every detective needs an evidence locker with a sane filing system. Here's yours.

The schema, roughly

raw_events — the unfiltered sensor feed. Every play, pause, seeked, progress, ended tick from the extension, tied to a video_id and session_id. This table only grows; you never edit it, only append. Treat it the way you'd treat a security camera's footage — the raw record, kept intact, that every later interpretation can be checked against.
tracks — one row per resolved song: video_id, artist, title, channel, is_music (bool), duration_seconds, audio_embedding (a pgvector column), mood_tags (array or jsonb), language, musicbrainz_id (nullable), implicit_score (recomputed periodically), created_at, updated_at.
sessions — optional but genuinely useful: groups of events with a shared session_id, with derived fields like session_start, session_length, dominant_time_bucket (morning/afternoon/evening/night). This is where "travel" and "study" playlist heuristics get their signal.
playlists and playlist_tracks — the output of the clustering step: a playlist has a name, a generation timestamp, dominant mood tags, and a join table linking it to track rows, ordered.

The key design idea: raw data is immutable, derived data is regenerable. If your classification logic improves in Month 4, you should be able to delete and recompute the tracks table's is_music and mood_tags columns from raw_events without losing anything real. This is the same principle behind "extract-then-delete" for audio from Phase 3 — keep the cheap-to-regenerate stuff disposable, keep the expensive-to-recreate stuff (your actual watch history) sacred.

Bootstrapping with Google Takeout

Your extension only sees behavior from the moment you install it forward. If you want historical depth on day one, Google Takeout's YouTube history export gives you a JSON/HTML dump of your watch history going back as far as YouTube has recorded it. It's coarser than extension data — no watch percentage, no skip detection, just "you watched this, at this timestamp" — but it's real, and it's enough to seed your tracks table and get Phase 2's classifier something to chew on in week one, instead of waiting three weeks for organic data to accumulate.

How to use it: request a Takeout export (JSON format), write a one-off import script that reads the export and inserts synthetic raw_events rows with event_type = 'legacy_watch' and no position data, then let Phase 2's classifier run over them exactly as it would over live data. Treat legacy rows as lower-confidence in your implicit-score formula — you don't know how much of the video was actually watched, so don't let a legacy "watched" event count as strongly as a fully-instrumented one.

Simulating data for development

You don't want to wait around watching YouTube for hours just to test whether your clustering code handles 500 tracks reasonably. Write a small synthetic data generator early — even a rough one — that creates plausible-looking raw_events rows: pick from a pool of real song titles (you can hand-curate 100-200 from your own taste, spanning a few genres/moods), assign random-but-plausible watch percentages and timestamps clustered around a few "mood personas" (a synthetic "you" who watches chill music at night and energetic music in the morning). This does two things: it lets you develop and test the ML/clustering code without waiting on real data, and it gives you a known-answer test — since you control the synthetic personas, you can check whether your clustering actually recovers something close to them, which is a much stronger validation than "the code ran without crashing."

Privacy and local-first design

This system is, by design, an intimate profile of your habits — what you listen to, when, how obsessively. A few commitments worth making explicit before you write a line of storage code:

Keep it local by default. Run Postgres on your own machine or a private instance you control, not a shared cloud service, unless you've deliberately decided to and secured it (auth, no public ports, encrypted connections).
Never redistribute extracted audio. Phase 3's "extract-then-delete" isn't just an engineering nicety — it's the difference between "a personal analysis tool" and "a tool with a copyright problem." Keep audio files ephemeral, process-and-discard, never served to anyone but the pipeline that needs them for a few seconds.
Be deliberate about optional signals like geolocation. The "Travel" playlist idea in Phase 5 floated coarse geolocation as one possible signal. If you add it, make it opt-in, store it at low resolution (city-level, not GPS-precise), and be honest with yourself about whether the marginal playlist quality is worth the marginal privacy cost. The session-length proxy (no geolocation needed) gets you most of the value for none of the risk — a good default.
You are both the engineer and the subject. That's unusual and worth sitting with: normally, privacy-by-design is about protecting other people's data from your system. Here, you're the one whose intimate behavioral data is on the line, which makes it a genuinely good, low-stakes place to practice privacy-conscious engineering habits before you ever build something that touches anyone else's data.

Chapter 5: The Galaxy of Songs — Playlist Generation Magic

Embeddings, explained like you're teaching a friend at a bar

Imagine every song as a point of light in a vast dark space. Songs that sound alike, or that you tend to listen to in similar moods, drift close together — not because anyone placed them there by hand, but because the math that produced their coordinates was trained (or, for pretrained models, trained by someone else on a huge amount of audio) to put "things that behave similarly" near each other and "things that behave differently" far apart. That's an embedding: a long list of numbers (say, 128 or 512 of them) that represents a song not as an ID, but as a position in a meaningful space. Two songs' embeddings being close together, measured by something like cosine similarity, is a proxy for "these songs are alike in whatever sense the model was trained to notice."

You have two galaxies in this project: the audio galaxy (positions determined by what things sound like) and the behavioral galaxy (positions determined by how you actually consume them, learned via the collaborative-filtering embeddings from Chapter 3). A track's true "meaning" in your system lives at the intersection — which is exactly why Phase 5 concatenates both into one combined vector before clustering. A cluster in that combined space isn't just "songs that sound similar," and it isn't just "songs you listen to in similar contexts" — it's songs that are similar on both axes simultaneously, which is a much stronger, more personally meaningful notion of "these belong on the same playlist" than either signal alone could produce.

Clustering and automatic labeling, tied together

HDBSCAN doesn't know what a "mood" is. It only knows distances. The intelligence in the system comes from what you fed it (the combined embedding) and how you interpret its output (the labeling heuristic). This division of labor is worth internalizing as a general ML pattern: unsupervised algorithms are excellent at finding structure, and terrible at explaining what the structure means — that translation step is where your domain knowledge (you know what "chill" sounds like to you) has to do the work, usually through a simple, inspectable rule rather than another opaque model. Keep that labeling step simple and debuggable for as long as possible; it's the layer you'll be staring at and tweaking the most, because it's the layer that decides whether a playlist gets a name you'd actually want to click.

Filtering for activity-based playlists

"Study" and "Travel" playlists aren't clusters in the audio/behavior space so much as they're filters layered on top of it — a study playlist is really "the calm, low-vocal-density cluster, further filtered to sessions where you historically kept the same track playing for a long uninterrupted stretch (a proxy for focus, since skip-heavy sessions suggest browsing, not working)." Treat these as compositions of your existing signals rather than new things to model from scratch — it's both less work and more explainable, which matters a lot when you're debugging why a jazz track ended up in "Study" and a moment's thought about "oh, right, low vocal density plus long uninterrupted plays" tells you exactly why.

A future reinforcement learning layer

Once the system exists and you're using it daily, a natural next question emerges: instead of only observing your behavior passively, could the system learn from how you react to its own suggestions? This is where a lightweight RL framing becomes interesting — treat playlist generation as a policy that chooses which tracks to include, treat your subsequent behavior (did you skip the second track in "Monday Chill," did you let the whole playlist play through) as a reward signal, and use something like a contextual bandit (a simpler, more tractable cousin of full RL, well suited to problems where you're repeatedly choosing from a set of options and observing an immediate reward) to nudge future playlist composition toward what you actually engage with. This is explicitly a later idea — Chapter 9 revisits it as a genuine "beyond the guide" project, because it depends on having a stable, working system generating playlists you actually use regularly first. RL on top of a system that doesn't have real usage data yet is RL on noise.

Chapter 6: The Web Dev Spine — Extension, Backend, Frontend, Tied Together

The ML chapters get the glamour, but this is the skeleton everything else hangs on. A quick tour of the parts you haven't already built phase-by-phase, plus the connective tissue between them.

The extension, structurally

A Chrome extension (Manifest V3) has three moving pieces you need to hold in your head as separate execution contexts that can't directly touch each other's variables — they only talk via message passing:

manifest.json — declares permissions (activeTab, host permissions for *.youtube.com), which scripts run where, and whether you have a background service worker.
The content script — runs inside the YouTube page's context, so it can read the DOM and attach listeners to the <video> element, but it's sandboxed from the extension's other state and can't make cross-origin requests as freely.
The background service worker — has broader permissions (can make network requests to your backend without CORS headaches, can persist state across tab navigations) but can't touch the page DOM directly.

The message-passing pattern you'll lean on constantly: content script calls chrome.runtime.sendMessage({type: 'EVENT', payload: {...}}), background script listens via chrome.runtime.onMessage.addListener, batches, and does the actual fetch to your API. Get comfortable with this pattern early — Phase 1 depends on it, and every later extension feature (detecting SPA navigation between videos, badge icon updates showing tracking status) reuses it.

REST API design that won't fight you later

A few conventions worth committing to from Phase 1 onward, because retrofitting them later is annoying:

Version your API from day one (/api/v1/events), even though you're the only consumer. Future-you, three months in, adding a mobile client or a second extension version, will thank present-you.
Keep ingestion endpoints (POST /events) fast and dumb. They should do validation and a write, nothing else. Any classification, embedding, or clustering work belongs in the offline pipelines, not inline in the request path — mixing them means a slow ML call can make your extension feel laggy or drop events under load.
Design read endpoints (GET /playlists, GET /tracks/{id}/similar) around what the frontend actually needs to render, not around your database schema. It's fine, even good, for an endpoint to join and reshape data server-side so the frontend doesn't have to stitch together three separate calls just to draw one screen.

Streaming audio, and the ethical note this section owes you

If you go the self-hosted-audio route from Phase 6's decision point, you're technically capable of building a /stream/{track_id} endpoint that reads an extracted audio file and streams it with proper Range header support so the browser's <audio> element can seek. Build it if you want the learning experience of implementing HTTP range requests (a genuinely useful, transferable skill — it's the same mechanism video platforms use). But hold the line firmly: this stays a local, single-user, never-publicly-deployed feature. The moment "personal tool running on my own machine for my own previously-watched content" turns into "a service anyone else can reach," you've crossed from a personal fair-use-adjacent experiment into redistribution, which is a different, real legal category. If you ever want to share this project publicly (a portfolio demo, a hackathon submission), disable or strip the self-hosted audio path and demo with the YouTube IFrame API version instead — it's not a downgrade, it's the responsible default.

Frontend architecture, briefly

Keep state simple: a top-level currentTrack and currentPlaylist in React context (or a lightweight state library if you want the practice), a PlaylistGrid component that fetches from /playlists on mount, a Player component that owns the actual playback element and logs playback events. The "similar tracks" widget is a nice small component to practice component composition on: it takes a trackId prop, fetches /tracks/{id}/similar on prop change, and renders a small horizontal list — self-contained enough to build and test in isolation before wiring it into the main player view.

Tying the vector database to the frontend

The satisfying part of pgvector is that "find similar tracks" is one query, not a separate service call to a vector database plus a join back to your relational metadata:

SELECT id, artist, title, embedding <-> (
    SELECT embedding FROM tracks WHERE id = :current_track_id
) AS distance
FROM tracks
WHERE id != :current_track_id
ORDER BY distance
LIMIT 5;

That single query is doing something that would otherwise require standing up and syncing two separate systems. It's a small thing, but it's the kind of small thing that, once you've felt the alternative, you start actively choosing simpler architectures for the rest of your career.

Chapter 7: The Detective's Logbook — Milestones Across the Whole Case

You've hit a concrete milestone at the end of every phase in Chapter 2, but it's worth stepping back and seeing the whole arc as one continuous story, because that's genuinely what it is:

Phase 0: two independent heartbeats — an extension that logs, a server that responds.
Phase 1: the system sees you for the first time. A real, timestamped trail of your own behavior, sitting in a database, that didn't exist yesterday.
Phase 2: the system starts telling music apart from everything else — its first act of judgment.
Phase 3: the system develops senses beyond text — it can now "hear," in the limited but real sense of turning sound into a comparable, structured signal.
Phase 4: the system develops a memory of your taste that isn't just "what you told it" but "what you actually did," which is a fundamentally more honest signal than any explicit rating system.
Phase 5: scattered evidence becomes a story — songs organize themselves into groups you didn't manually create, with names that feel earned rather than assigned.
Phase 6: the system finds its voice, and hands the story back to you in a form you can actually use, day to day, the way you'd use Spotify — except every playlist in it is an artifact of your own engineering.

Each of those is worth a screenshot, a commit message that actually describes what changed and why, and — genuinely — a moment of letting yourself feel like this is a big deal. It is. Most people who use recommendation systems every day have no idea how the black box works. You're about to have built one, from the sensor all the way to the speaker.

Chapter 8: Beyond the Guide — Where the Case Reopens

Five months from now, you'll have a working, personal AI DJ. Here's where the story can keep going, roughly in order of "natural next step" to "ambitious departure":

Deploy it somewhere always-on, like a Raspberry Pi or a small home server, so the nightly pipeline runs whether or not your laptop is open. This is a good, contained systems-administration project on its own — cron jobs, systemd services, basic monitoring so you notice if the pipeline silently stops working.
Add voice control — "play something chill" parsed via a small intent-classification step (zero-shot classification again, this time over your playlist names and mood tags) wired to a wake-word library or a simple push-to-talk button in the frontend.
Integrate with the real Spotify API as a second data/playback source, letting your behavioral model learn from both platforms at once, and letting your generated playlists optionally get pushed into actual Spotify playlists you can use anywhere, on any device.
Build the reinforcement learning layer sketched in Chapter 5 — once you have real usage data (skips, replays, completions) on the system's own suggestions, not just your organic YouTube behavior, you have the reward signal that makes a bandit-style approach genuinely meaningful instead of speculative.
Turn it into a SaaS, if you want the entirely different (and genuinely valuable) experience of learning multi-tenant architecture, per-user data isolation, OAuth flows for other people's YouTube accounts, and all the "now it's not just your data anymore" privacy engineering that Chapter 4 flagged as a different category of problem. Don't take this step lightly or quickly — it's a good six-month project in its own right, layered on top of the five months you'll have already spent.

None of these are required to call the project "done." The five-month build is a complete story with a real ending. Everything in this chapter is a sequel hook, not an unfinished obligation.

Chapter 9: The Final Note

You started this case with an irritating, wonderful little observation: somewhere in your YouTube history was a pattern nobody, not even you, could see clearly. Over the next five months, you built the instrument to see it — a browser extension that learned to watch the way you watch, a pipeline that learned to hear the way songs sound, a model that learned your taste from what you actually did instead of what you said, and a clustering engine that turned all of that into playlists with names that feel like they know you, because in the most literal sense, they do.

Along the way you picked up a working fluency in things that sound intimidating from the outside and turn out to be learnable in exactly the incremental way this guide walked you through: DOM events and extension messaging, REST API design, mel spectrograms and audio embeddings, zero-shot NLP classification, the actual math behind implicit feedback, neural collaborative filtering, density-based clustering, vector similarity search, and a real frontend audio player. Each of those, on its own, is a legitimate skill line on a resume. Together, wired into one living system, they're something rarer: a complete, working thing you built end to end, that solves a problem only you have, in a way that only you could have tuned it.

You started with a YouTube habit. You're ending with a machine learning orchestra — one that watches, listens, remembers, and plays, quietly, in the background of your life, getting a little more you every single day it runs.

Case closed. Time to hit play.

ChronoWeave: The Documentary

Biki Kalita — Sun, 26 Jul 2026 06:53:09 +0000

From Framework User to Framework Understander — A 4-Month Journey Through Gradient Descent, Transformers, and the Physics of History

"Every timeline you've ever seen is a lie of omission. It shows you WHEN things happened. It never shows you WHY. ChronoWeave is our attempt to fix that — and along the way, you're going to learn what actually happens inside a neural network when you call .backward()."

How to Use This Guide

This is not a tutorial you skim. It's a companion for a 3–5 month build. Read one section, do the exercise, hit the wall, climb over it, then come back. Every chapter follows the same rhythm:

The Concept — the idea, explained with an analogy before a single line of code
The Code Challenge — a skeleton you fill in, never a finished solution
The Aha Moment — what you should see when it clicks
The Extension — how this piece bolts onto the larger machine
Resources — exact docs, not "just Google it"

And threaded throughout:

🧭 The Mentor Says — the advice I wish someone had given me
🕳️ The Rabbit Hole — optional, for when curiosity gets the better of you
😤 The Struggle — the wall you will hit, named in advance
✅ Checkpoint — stop and verify before you go further
💡 The Innovation — why this piece of the project is worth talking about in an interview

Let's begin.

CHAPTER 0: The Grand Vision — Seeing the Complete Picture

The Problem: Why Historical Timelines Are Broken

Open any history textbook's timeline. You'll see a horizontal line, some dots, some dates. 1914. 1917. 1929. 1939. It tells you when. It is silent on why.

The assassination of Archduke Franz Ferdinand didn't cause World War I in a vacuum — it was the spark that landed on a decade of alliance treaties, arms races, and colonial tension that had already turned the continent into dry kindling. A flat timeline shows you the spark. It hides the kindling.

What historians actually think in is a causal graph: event A enabled event B, which, combined with condition C, triggered event D. That graph is tangled, non-linear, and multi-dimensional — which is exactly why nobody draws it by hand. It's too much cognitive load.

ChronoWeave's bet: if a machine can extract the causal graph from raw text, and then lay it out so that causally-connected events cluster together and pull each other into readable arrangement, you get something a static timeline can never give you — a map you can explore, drag, and reorganize, where the layout itself is meaningful. Events that influence each other stay near each other. That's not a UI nicety. That's the entire value proposition.

The Solution: ChronoWeave's Architecture

Here's the complete system, before we write a single line of code:

┌─────────────────────────────────────────────────────────────────────┐
│                         THE USER'S BROWSER                           │
│  ┌──────────────┐        ┌────────────────────────────────────┐    │
│  │  Text Input   │──────▶│   React + D3.js Canvas               │    │
│  │  "Paste       │       │   (renders nodes = events,           │    │
│  │   history      │       │    edges = causal links)             │    │
│  │   here"        │       │                                       │    │
│  └──────────────┘        │   User drags a node ───────┐         │    │
│                            └─────────────────────────────┼─────────┘    │
└──────────────────────────────────────────────────────────┼──────────────┘
                    │ POST /extract                         │ WS: node_moved
                    ▼                                        ▼
┌─────────────────────────────────────────────────────────────────────┐
│                         FASTAPI BACKEND                              │
│                                                                        │
│  ┌────────────────────┐     ┌───────────────────────────────────┐  │
│  │  EXTRACTION ENGINE   │     │   SPATIAL INTELLIGENCE ENGINE       │  │
│  │  (Chapter 2)          │     │   (Chapter 3 — the heart of it)     │  │
│  │                        │     │                                      │  │
│  │  1. Split into         │     │  1. Build a graph: nodes=events,    │  │
│  │     sentences          │     │     edges=causal relations          │  │
│  │  2. Run NER            │────▶│  2. Init x,y for every node as a    │  │
│  │     (BERT) to find     │     │     TRAINABLE TENSOR                │  │
│  │     events, dates,     │     │     (requires_grad=True)            │  │
│  │     people, places     │     │  3. Define "Map Clarity Loss":      │  │
│  │  3. Run Relation       │     │     - causally linked nodes should  │  │
│  │     Extraction to      │     │       be CLOSE                     │  │
│  │     find "A causes B"  │     │     - unrelated nodes should be     │  │
│  │  4. Output: list of    │     │       FAR (repulsion)               │  │
│  │     (event, date,      │     │     - nothing should overlap        │  │
│  │      relation, event)  │     │  4. Run backprop on the             │  │
│  │     triples            │     │     COORDINATES (not the model's    │  │
│  │                        │     │     weights!) using a hand-written  │  │
│  │                        │     │     Adam optimizer                  │  │
│  │                        │     │  5. When user drags a node, PIN     │  │
│  │                        │     │     that node's coords, re-run      │  │
│  │                        │     │     optimization on everyone else   │  │
│  └────────────────────────┘     └───────────────────────────────────┘  │
│              │                                    │                      │
│              ▼                                    ▼                      │
│  ┌────────────────────────────────────────────────────────────────┐   │
│  │           PostgreSQL + pgvector (stores triples + embeddings)    │   │
│  └────────────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────────┘

Data Flow: What Happens When a User Clicks "Generate"

Walk through it end to end, because this sequence is the skeleton every later chapter hangs off of:

User pastes text (say, three paragraphs on the causes of WWI) and clicks Generate.
Frontend POSTs the raw text to /api/extract.
Backend NER model (a fine-tuned BERT) tags every token: is it part of an EVENT, a DATE, a PERSON, a PLACE? Output: a list of entity spans.
Backend Relation Extraction model looks at pairs of entities in the same or nearby sentences and predicts a relation label: CAUSES, ENABLED_BY, PRECEDES, PART_OF, or NONE.
The result is a set of triples: (Assassination of Franz Ferdinand, CAUSES, Austria-Hungary ultimatum to Serbia). These get written to Postgres, and each event also gets a text embedding stored via pgvector (this lets us later cluster thematically similar events even without an explicit extracted relation).
The triples define a graph: events are nodes, relations are edges.
The Spatial Intelligence Engine initializes every node at a random (x, y) position — as a PyTorch tensor with requires_grad=True. This is the part that makes ChronoWeave unusual: the coordinates themselves are the parameters being learned, not neural network weights.
We define a custom loss function — the Map Clarity Loss — that penalizes bad layouts: causally-linked nodes that are far apart, unrelated nodes that overlap, edges that cross each other unnecessarily.
We run gradient descent (a hand-written Adam optimizer) for a few hundred steps. Each step: compute the loss, call .backward(), get gradients with respect to the x,y coordinates, nudge the coordinates a little in the direction that reduces loss. Repeat.
After convergence, the backend streams the final (node_id, x, y) positions back to the frontend over the initial REST response (or WebSocket, for the live re-layout case).
React + D3.js renders nodes at those coordinates, with edges drawn between causally-linked events, styled by relation type.
The magic moment: the user drags a node to a new spot because they disagree with the layout, or just want to explore. That node's position gets pinned (requires_grad=False for it, fixed at the drag target). The backend re-runs the optimizer on every other node, live, streaming intermediate frames over a WebSocket, so the user watches the rest of the graph flow and resettle around their edit — like iron filings reorganizing around a magnet you just moved.

That last step is the entire reason this project exists. A physics engine (force-directed graph layout, e.g. D3's built-in forceSimulation) can also do node repulsion and edge attraction — but it can't easily express arbitrary, learned, semantically-weighted objectives (e.g., "cluster by decade AND by causal chain AND penalize edge crossings, with weights that adapt based on graph density"). Gradient descent on a custom loss can. That's the pitch. We'll come back to defending it rigorously in Chapter 3.

Technology Stack: Why Each Piece Was Chosen

Layer	Choice	Why
ML core	PyTorch (from raw tensors, then `nn.Module` later)	You need autograd exposed at the tensor level to do coordinate optimization; PyTorch's dynamic graph makes this natural
NER / Relation Extraction	Fine-tuned BERT-base, later maybe REBEL for joint extraction	BERT is small enough to fine-tune on a single GPU and well documented; REBEL is a strong upgrade path for joint entity+relation extraction
Backend	FastAPI + WebSockets	Async-native, typed, and WebSocket support is first-class for the live re-layout streaming
Database	PostgreSQL + pgvector	Relational structure for triples, vector column for semantic similarity search on event embeddings
Frontend	React + D3.js	D3 gives you full control over force/position rendering; React manages component state and drag events
Deployment	Docker Compose → cloud (Render/Fly.io/AWS)	Reproducibility first, then scale

The Journey Ahead

Four phases, each roughly a month, building strictly bottom-up:

Phase 1 (Weeks 1–4): You build the math — autograd, manual backprop, hand-rolled optimizers, a toy Transformer. No frameworks doing the work for you yet.
Phase 2 (Weeks 5–8): You build the reader — BERT-based NER and relation extraction, a real data pipeline, a real database.
Phase 3 (Weeks 9–12): You build the cartographer — the spatial intelligence engine. This is the conceptual core of the whole project and gets the most depth in this guide.
Phase 4 (Weeks 13–16): You build the body — the full-stack app, the live drag-and-reoptimize loop, and you ship it.

🧭 The Mentor Says: Don't let the ambition of the final product distract you from the ugliness of the early steps. In week 2 you'll be manually computing gradients for a two-layer network with a pencil-and-paper feel to it, and it will seem impossibly far from "beautiful interactive graph app." That gap is supposed to be there. Every person who deeply understands PyTorch went through exactly this tunnel. You don't skip it by using nn.Module early — you skip understanding by doing that.

CHAPTER 1: The Foundation — Building Your Neural Engine

(Weeks 1–4 · Prerequisite: comfort with Python, basic linear algebra — matrix multiply, dot products — and derivatives from a first calculus course)

Week 1: Environment and the Shape of a Tensor

The Concept

A tensor is just a multi-dimensional array with two superpowers bolted on: it knows how to run on a GPU, and it can remember the sequence of operations that produced it, so it can later compute how a tiny nudge to its inputs would change its output. That second superpower is autograd, and it's the single idea Phase 1 exists to demystify.

Think of a tensor with requires_grad=True as a spreadsheet cell that doesn't just hold a number — it holds a number and a formula referencing other cells. Change an input cell, and every downstream cell knows exactly how much it would change, without you re-deriving the formula by hand. That "how much would it change" is the gradient.

The Code Challenge

Set up your environment first:

poetry init
poetry add torch numpy matplotlib jupyter
poetry shell

Then in a notebook, don't build anything yet — just observe autograd:

import torch

# TODO: create a tensor x = 3.0 with requires_grad=True
x = ...

# TODO: create y = x**2 + 2*x + 1
y = ...

# TODO: call y.backward() and print x.grad
# Predict on paper what dy/dx should be at x=3 BEFORE you run this
...

The Aha Moment

x.grad should equal 2*x + 2 = 8.0 — exactly the calculus-class derivative, computed automatically. The click isn't "wow it did calculus," it's realizing PyTorch didn't look up a symbolic formula for x**2 + 2*x + 1 — it recorded the sequence of primitive operations (pow, mul, add) as you executed them, and walked that sequence backward, applying the chain rule at each step. This recorded sequence is called the computation graph, and it's rebuilt fresh every time you run forward — which is why PyTorch is called "define-by-run."

The Extension

Every gradient ChronoWeave ever computes — whether it's a loss with respect to a BERT weight, or a loss with respect to a node's (x, y) position on the canvas — goes through this exact mechanism. There is no different code path for "coordinates" versus "weights." This is the whole trick behind Phase 3, twelve weeks from now.

Resources

PyTorch autograd tutorial: https://docs.pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html

🕳️ The Rabbit Hole: Read about torch.autograd.grad() vs .backward() — the former lets you compute gradients without accumulating them into .grad, important later when you want gradients with respect to coordinates specifically, while leaving model weights untouched.

✅ Checkpoint 1.1: Explain, without looking anything up, why calling .backward() twice on the same graph (without retain_graph=True) throws an error. (Answer: the graph is freed after the first backward pass to save memory.)

Week 2: Manual Forward and Backward — No `nn.Module`

The Concept

nn.Module is a convenience wrapper. Underneath, a "layer" is nothing more than: take an input, multiply by a weight matrix, add a bias, pass through a nonlinearity. A "network" is several of those chained. "Training" is: compute a loss, get its gradient with respect to every weight, nudge each weight opposite to its gradient. Building this by hand once is the single highest-leverage exercise in this project.

The Code Challenge

Fit y = sin(x) on x in [-π, π] with manual parameters:

import torch
torch.manual_seed(0)

# Architecture: 1 -> 32 -> 1, tanh activation
W1 = ...  # TODO: (1,32), requires_grad=True, small random init
b1 = ...  # TODO: (32,)
W2 = ...  # TODO: (32,1)
b2 = ...  # TODO: (1,)

def forward(x):
    # TODO: z1 = x @ W1 + b1 ; a1 = tanh(z1) ; z2 = a1 @ W2 + b2
    ...
    return z2

def mse_loss(pred, target):
    # TODO: mean squared error, raw tensor ops
    ...

lr = 0.05
for step in range(2000):
    x = torch.linspace(-3.14, 3.14, 200).unsqueeze(1)
    y_true = torch.sin(x)
    pred = forward(x)
    loss = mse_loss(pred, y_true)
    # TODO: zero old grads, loss.backward(), manually update all 4 params
    #       (wrap update in `with torch.no_grad():`)
    ...
    if step % 200 == 0:
        print(step, loss.item())

The Aha Moment

Loss will not decrease if you forget to zero gradients — PyTorch accumulates them into .grad by default. Once burned by this, you'll never forget zero_grad() again — you'll understand why, not just cargo-cult it. Then plot forward(x) against sin(x): a sine wave emerges from 4 matrices you initialized as noise.

The Extension

This loop — forward, loss, zero grad, backward, manual update — is structurally identical to the loop you write in Phase 3 for optimizing node coordinates. Only what the "parameters" represent changes.

😤 The Struggle: You'll likely hit RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn, usually from overwriting a leaf tensor without torch.no_grad(). Print .requires_grad / .is_leaf on every parameter right before the failing line.

Resources

PyTorch neural network basics: https://docs.pytorch.org/tutorials/beginner/blitz/neural_networks_tutorial.html (concepts only this week)
3Blue1Brown, "Backpropagation calculus"

Week 3: Hand-Rolled SGD, Momentum, RMSprop, and Adam

The Concept

An optimizer answers: given a gradient, how exactly should I change the parameter? Plain SGD (subtract lr*grad) works but is slow and oscillates on ravine-shaped losses.

Momentum — running average of past gradients, damping oscillation.
RMSprop — running average of squared gradients per parameter; divides the step by its square root, shrinking effective learning rate for consistently-large-gradient parameters.
Adam — momentum + RMSprop, plus bias-correction for early steps.

The Code Challenge

class ManualSGD:
    def __init__(self, params, lr=0.01, momentum=0.0):
        self.params = list(params)
        self.lr = lr
        self.momentum = momentum
        self.velocities = ...  # TODO: zeros_like buffer per param

    def step(self):
        with torch.no_grad():
            for p, v in zip(self.params, self.velocities):
                if p.grad is None: continue
                # TODO: v = momentum*v - lr*p.grad ; p += v
                ...

    def zero_grad(self):
        for p in self.params:
            if p.grad is not None: p.grad.zero_()


class ManualAdam:
    def __init__(self, params, lr=0.001, betas=(0.9, 0.999), eps=1e-8):
        self.params = list(params)
        self.lr, self.b1, self.b2, self.eps, self.t = lr, *betas, eps, 0
        self.m = ...  # TODO: zeros_like per param (first moment)
        self.v = ...  # TODO: zeros_like per param (second moment)

    def step(self):
        self.t += 1
        with torch.no_grad():
            for i, p in enumerate(self.params):
                if p.grad is None: continue
                g = p.grad
                # TODO: m[i] = b1*m[i] + (1-b1)*g
                # TODO: v[i] = b2*v[i] + (1-b2)*g**2
                # TODO: m_hat = m[i]/(1-b1**t) ; v_hat = v[i]/(1-b2**t)
                # TODO: p -= lr * m_hat / (sqrt(v_hat) + eps)
                ...

    def zero_grad(self):
        for p in self.params:
            if p.grad is not None: p.grad.zero_()

Re-run the Week 2 sine-fit with ManualAdam, compare convergence speed to vanilla SGD.

The Aha Moment

Adam converges in hundreds of steps where SGD needed thousands, and is far less sensitive to learning rate choice. This robustness is why you'll reach for it again in Phase 3, where the loss surface (Map Clarity Loss over 2D positions) is even messier.

The Extension

Your Phase 3 CoordinateAdam class will be a near copy of this one — same math, different thing being optimized.

💡 The Innovation (early preview): "I wrote my own Adam optimizer for coordinate updates because I needed to freeze individual coordinates mid-optimization when a user grabs a node — the standard torch.optim API doesn't expose that cleanly." Keep this for Chapter 3.

Resources

Adam paper (Kingma & Ba, 2014): https://arxiv.org/abs/1412.6980
Sebastian Ruder, "An overview of gradient descent optimization algorithms": https://www.ruder.io/optimizing-gradient-descent/

✅ Checkpoint 1.2: Plot loss curves for SGD, SGD+momentum, RMSprop, Adam — same problem, same init, same step count. If Adam isn't fastest, check bias-correction first.

Week 4: A Transformer From Scratch (Tiny, But Real)

The Concept

A Transformer's core operation: for every token, compute a weighted average of every other token's representation, where weights ("attention") are learned and reflect relevance. Everything else — multi-head, positional encoding, layer norm, feedforward — is scaffolding.

Analogy: a room full of people (tokens) forming an opinion on a topic — you weight each person's input by relevance. Query = what you're looking for; Key = what each token offers; Value = what it contributes if attended to.

The Code Challenge

import torch, math
import torch.nn.functional as F

def scaled_dot_product_attention(Q, K, V, mask=None):
    d_k = Q.shape[-1]
    scores = ...  # TODO: Q @ K.T / sqrt(d_k)
    if mask is not None:
        scores = scores.masked_fill(mask == 0, float('-inf'))
    weights = ...  # TODO: softmax(scores, dim=-1)
    output = ...   # TODO: weights @ V
    return output, weights
# Sanity check: verify weights.sum(dim=-1) is all 1.0

class TinyMultiHeadAttention(torch.nn.Module):
    def __init__(self, d_model, n_heads):
        super().__init__()
        assert d_model % n_heads == 0
        self.d_k, self.n_heads = d_model // n_heads, n_heads
        ...  # TODO: four Linear layers W_q, W_k, W_v, W_o

    def forward(self, x, mask=None):
        ...  # TODO: project, reshape to heads, attend per head, concat, W_o

class TinyEncoderLayer(torch.nn.Module):
    def __init__(self, d_model, n_heads, d_ff):
        super().__init__()
        self.attn = TinyMultiHeadAttention(d_model, n_heads)
        self.norm1 = torch.nn.LayerNorm(d_model)
        self.ff = torch.nn.Sequential(
            torch.nn.Linear(d_model, d_ff), torch.nn.ReLU(),
            torch.nn.Linear(d_ff, d_model))
        self.norm2 = torch.nn.LayerNorm(d_model)

    def forward(self, x, mask=None):
        ...  # TODO: x = norm1(x + attn(x)) ; x = norm2(x + ff(x))

Train on next-character prediction over a small corpus (a few historical-text paragraphs fits the theme).

The Aha Moment

Visualize attention weights for one sentence as a heatmap — rows should light up on semantically relevant tokens (e.g. a pronoun attending to the noun it refers to). Attention stops being a diagram and becomes something you watch your model do.

The Extension

In Chapter 2 you'll swap this toy encoder for a pretrained BERT, understanding exactly what happens inside every layer because you built the smallest version yourself.

Resources

"The Annotated Transformer" (Harvard NLP): https://nlp.seas.harvard.edu/annotated-transformer/
"Attention Is All You Need": https://arxiv.org/abs/1706.03762
Jay Alammar, "The Illustrated Transformer": https://jalammar.github.io/illustrated-transformer/

🕳️ The Rabbit Hole: Attention has no inherent sense of token order. Read the sinusoidal positional encoding and ask why sine/cosine rather than a learned embedding.

✅ Checkpoint 1.3 — End of Phase 1: From memory, sketch a Transformer encoder layer's forward pass, and explain what breaks if you remove residual connections, layer norm, or the sqrt(d_k) scaling.

Phase 1 Milestone

By end of Week 4, from raw tensor operations, you have: (1) intuitive autograd understanding, (2) a hand-built 2-layer network with manual gradient updates, (3) three hand-built optimizers, (4) a working tiny Transformer with visualized attention. Your Week 3 optimizers become the Phase 3 coordinate optimizer almost verbatim; your Week 4 Transformer understanding is what lets you debug BERT rather than treat it as a black box.

CHAPTER 2: The Understanding — Teaching Your Engine to Read History

(Weeks 5–8 · Prerequisite: Phase 1 complete, comfort reading model architecture diagrams)

Week 5: Fine-Tuning BERT for Named Entity Recognition

The Concept

Your tiny Transformer from Week 4 learned character prediction from scratch on almost no data. BERT is the same architecture, scaled up (12 layers, 12 heads, 768-dim), pretrained on billions of words. Fine-tuning means: keep the pretrained weights as a strong starting point, attach a small task-specific head on top (here, a linear layer mapping each token's final hidden state to a label like B-EVENT, I-EVENT, B-DATE, O), and train the whole thing (or just the head) on your labeled data for a few epochs.

The reason this works with so little labeled data compared to training from scratch: BERT already learned general-purpose language structure from pretraining (grammar, word relationships, some world knowledge). Fine-tuning just teaches it to route that existing knowledge toward your specific labeling scheme.

The Code Challenge

from transformers import BertTokenizerFast, BertForTokenClassification
import torch

# TODO: load "bert-base-cased" tokenizer and BertForTokenClassification
#       with num_labels = len(your_label_list)
#       e.g. label_list = ["O","B-EVENT","I-EVENT","B-DATE","I-DATE","B-PERSON","I-PERSON"]
tokenizer = ...
model = ...

def tokenize_and_align_labels(text, word_labels):
    """
    text: list of words, e.g. ["The", "assassination", "of", "Franz", "Ferdinand"]
    word_labels: list of label ids, one per word
    Returns tokenized input + labels aligned to WordPiece subtokens
    (subtokens after the first get label -100 so they're ignored in the loss)
    """
    # TODO: tokenizer(text, is_split_into_words=True, ...) then use
    # .word_ids() to map each subtoken back to its source word
    ...

# TODO: set up a small labeled dataset (start with ~50-100 hand-labeled
# sentences from historical text — yes, you label it yourself first)
# TODO: standard fine-tuning loop: forward, CrossEntropyLoss (ignore_index=-100),
# backward, optimizer.step() -- use torch.optim.AdamW this time, not your manual one,
# since you've already proven you understand what it's doing

The Aha Moment

Run inference on a sentence the model never saw during fine-tuning and watch it correctly tag "the outbreak of war" as an EVENT span and "1914" as a DATE span, purely from ~100 examples. That's the pretraining transfer working — this would be essentially impossible to get right training from random initialization on 100 sentences.

The Extension

This NER output — spans tagged as EVENT, DATE, PERSON, PLACE — is exactly what Week 6's relation extraction will pair up into causal triples.

Resources

Hugging Face token classification guide: https://huggingface.co/docs/transformers/tasks/token_classification
BERT paper (Devlin et al., 2018): https://arxiv.org/abs/1810.04805

😤 The Struggle: Label alignment between words and WordPiece subtokens is the single most common source of silent bugs in NER fine-tuning — a misaligned label doesn't crash, it just quietly trains the model on garbage. Always spot-check tokenizer.convert_ids_to_tokens() against your aligned label array for a handful of examples before you trust any training run.

Week 6: Relation Extraction — Finding "A Causes B"

The Concept

NER tells you what the entities are. Relation extraction tells you how they relate. The simplest approach: for every pair of entities that co-occur in the same sentence (or a short window of sentences), feed their contextualized representations (plus the sentence itself) into a classifier that predicts one of a fixed set of relation labels: CAUSES, ENABLED_BY, PRECEDES, PART_OF, or NONE.

A more powerful approach — REBEL — does entity and relation extraction jointly, generating the full set of triples as a structured text sequence in one pass, rather than requiring you to first extract entities, then classify every pair. It's a strong upgrade path once your simpler pairwise classifier is working and you understand why the joint approach is more efficient (it doesn't blow up combinatorially with the number of entities in a sentence).

The Code Challenge

Start with the pairwise classifier — it's the better learning exercise, even though REBEL is more production-capable:

# For each entity pair (e1, e2) in the same sentence:
#   1. Mark their spans in the input with special tokens, e.g.
#      "[E1] The assassination [/E1] of Franz Ferdinand triggered
#       [E2] Austria-Hungary's ultimatum [/E2] to Serbia."
#   2. Run through BERT
#   3. TODO: take the hidden states at the [E1] and [E2] marker positions,
#      concatenate them, pass through a small classifier head
#      -> softmax over {CAUSES, ENABLED_BY, PRECEDES, PART_OF, NONE}

The Aha Moment

The output for that Franz Ferdinand sentence should be CAUSES with high confidence. Then feed it a sentence with two entities that are merely mentioned near each other with no causal link ("The war began in 1914. Assassinations were common in the region.") and watch it correctly predict NONE. That contrast — the model discriminating relevance, not just proximity — is the whole point of relation extraction over naive "entities near each other are related" heuristics.

The Extension

Every (entity_1, relation, entity_2) triple this produces becomes an edge in the causal graph that Phase 3's spatial engine will lay out.

Resources

REBEL paper (Huguet Cabot & Navigli, 2021): https://aclanthology.org/2021.findings-emnlp.204/
REBEL on Hugging Face: https://huggingface.co/Babelscape/rebel-large

🕳️ The Rabbit Hole: Read about "distant supervision" for relation extraction — a technique for auto-generating noisy training labels by aligning a knowledge base (like Wikidata) against text, instead of hand-labeling everything. Useful if your ~100 hand-labeled examples aren't enough once you scale up.

Week 7: The Data Pipeline — Text → Tokens → Entities → Relations → Storage

The Concept

A pipeline is only as trustworthy as its weakest stage boundary. This week isn't about new ML — it's about wiring Weeks 5 and 6 together into something that reliably takes raw pasted text and emits a clean, deduplicated, database-ready set of triples, with sane error handling for the inevitable garbage input (empty text, non-English text, text with no extractable events).

The Code Challenge

class ExtractionPipeline:
    def __init__(self, ner_model, relation_model, tokenizer):
        ...

    def extract(self, raw_text: str) -> list[dict]:
        """
        Returns: [{"subject": "...", "relation": "CAUSES", "object": "...",
                   "subject_date": "1914-06-28" or None, ...}, ...]
        """
        # TODO Step 1: sentence-split raw_text (spaCy or nltk sentence tokenizer)
        # TODO Step 2: run NER on each sentence -> entity spans
        # TODO Step 3: for entity pairs within a sentence AND across
        #      adjacent sentences (causal claims often span 2 sentences),
        #      run the relation classifier
        # TODO Step 4: filter relation predictions below a confidence
        #      threshold (start at 0.6, tune empirically)
        # TODO Step 5: deduplicate triples (same subject/object/relation
        #      appearing from overlapping sentence windows)
        # TODO Step 6: return structured triples
        ...

Set up PostgreSQL with pgvector:

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE events (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    text TEXT NOT NULL,
    date_text TEXT,
    embedding VECTOR(768),  -- BERT's hidden size
    created_at TIMESTAMPTZ DEFAULT now()
);

CREATE TABLE relations (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    source_event_id UUID REFERENCES events(id),
    target_event_id UUID REFERENCES events(id),
    relation_type TEXT NOT NULL,
    confidence FLOAT
);

The Aha Moment

Paste in three unrelated paragraphs of history (say, one on WWI causes, one on the French Revolution, one on the fall of Rome) and watch the pipeline correctly keep the triples from each topic separate, with no spurious cross-topic relations — because relation extraction is confidence-thresholded and entity embeddings from unrelated topics don't get spuriously matched.

The Extension

This pipeline is the entire backend of the /api/extract endpoint from Chapter 0's data flow diagram. Everything downstream (Phase 3, Phase 4) consumes its output.

Resources

pgvector docs: https://github.com/pgvector/pgvector
SQLAlchemy + pgvector integration: https://github.com/pgvector/pgvector-python

✅ Checkpoint 2.1: Run your pipeline on a genuinely messy input (a Wikipedia paragraph with footnote markers, inconsistent date formats, nested clauses) and verify it degrades gracefully — no crash, just fewer/lower-confidence triples — rather than throwing an unhandled exception.

Week 8: Evaluation and Hardening

The Concept

An extraction pipeline that "looks right" on your three favorite test paragraphs is not the same as one that's reliable. This week is about building a small held-out evaluation set (hand-labeled, separate from training data) and computing real metrics: precision, recall, and F1 for both entity spans and relation classification.

The Code Challenge

def evaluate_ner(model, eval_sentences, eval_labels):
    # TODO: run inference, compute span-level (not token-level!) precision/recall/F1
    # Span-level means a predicted "B-EVENT I-EVENT" span only counts as
    # correct if it matches the gold span's exact boundaries, not just
    # individual token labels — this is a stricter and more meaningful metric.
    ...

def evaluate_relations(model, eval_pairs, eval_labels):
    # TODO: per-class precision/recall/F1, plus a confusion matrix
    # (CAUSES vs ENABLED_BY vs PRECEDES are semantically close and
    # commonly confused — expect this)
    ...

The Aha Moment

Your confusion matrix will likely show CAUSES and ENABLED_BY bleeding into each other — this is expected and informative, not a bug. It tells you those two categories are genuinely hard to distinguish from surface text alone (a human annotator would disagree with themselves on some of these too), which should inform how confidently the frontend displays that distinction later (e.g., maybe you merge them into one edge style with a subtler visual difference rather than two starkly different arrow types).

The Extension

Phase 2 milestone: a real, measured, imperfect-but-quantified extraction system, feeding real triples into a real database.

Resources

"Named Entity Recognition" evaluation conventions (seqeval library): https://github.com/chakki-works/seqeval

Phase 2 Milestone

You now have a fine-tuned BERT NER model, a relation extraction classifier, a hardened pipeline connecting them, a Postgres+pgvector store, and honest evaluation metrics on held-out data. This is a legitimate, demoable NLP system on its own — worth pausing to appreciate before diving into Phase 3, which is where the project becomes genuinely novel.

CHAPTER 3: The Intelligence — Making Your Engine Think Spatially

(Weeks 9–12 · Prerequisite: Phase 1 and 2 complete. This is the conceptual core of ChronoWeave — take this chapter slowly.)

Why This Chapter Gets the Most Depth

Everything before this point — the Transformer, the NER model, the relation extractor — is, architecturally, "standard" deep learning applied to a specific domain. Impressive to build from scratch, but conceptually well-trodden. This chapter is where ChronoWeave does something genuinely uncommon: using gradient descent as a general-purpose layout algorithm, with coordinates as first-class trainable parameters. Take this slowly. It's the part of the project you'll actually be excited to explain in an interview.

Week 9: Coordinates as Parameters

The Concept

Every optimization problem in ML has the same shape: parameters, a loss function measuring how bad the current parameters are, and an optimizer that nudges parameters to reduce that loss. So far, "parameters" has meant neural network weights. Nothing in that shape requires the parameters to be weights. A parameter is just a tensor with requires_grad=True that appears somewhere in a differentiable computation whose output is your loss.

So: what if the parameters are the (x, y) positions of graph nodes on a canvas, and the loss is a hand-designed function that scores how "good" a layout is?

This reframes graph layout — traditionally solved with physics simulations (force-directed layout, spring-embedder algorithms) — as an optimization problem solvable with the exact same machinery you built in Phase 1.

The Code Challenge

Set up the skeleton, no loss function yet — just prove coordinates can be optimized at all:

import torch

num_nodes = 10
# TODO: initialize positions as a (num_nodes, 2) tensor, random in
# some reasonable range (e.g. -5 to 5), requires_grad=True
positions = ...

# Toy goal: pull every node toward the origin (0,0) — trivial loss,
# just to prove the plumbing works before you write anything smarter
def toy_loss(positions):
    # TODO: sum of squared distances from origin
    return (positions ** 2).sum()

optimizer = torch.optim.Adam([positions], lr=0.1)
for step in range(100):
    optimizer.zero_grad()
    loss = toy_loss(positions)
    loss.backward()
    optimizer.step()
    if step % 20 == 0:
        print(step, loss.item())

# TODO: plot positions before and after — every node should have
# collapsed toward (0,0)

The Aha Moment

Watching ten random points converge toward the origin using torch.optim.Adam — the exact same optimizer class you'd use to train a neural network — is the moment this project's central idea stops being abstract. You just used backpropagation to solve a geometry problem. No physics, no forces, no simulation of springs — just calculus.

The Extension

toy_loss is a stand-in. Next week you replace it with the real Map Clarity Loss — but the training loop around it doesn't change at all.

🧭 The Mentor Says: Resist the urge to jump straight to the full Map Clarity Loss this week. Build up loss terms one at a time, verify each one does what you expect in isolation (attraction only, then repulsion only, then combined), and only then combine everything. Debugging a five-term loss function that's never converged correctly at any point is miserable. Debugging a five-term loss function where you've verified each term separately is straightforward.

Week 10: The Map Clarity Loss

The Concept

A good layout satisfies several competing objectives simultaneously:

Attraction: causally-linked events should be close together (short edges are more readable)
Repulsion: all events should push apart from each other generally, so the graph doesn't collapse into a single point (this is what "attraction" alone would do — see Week 9's toy example)
Non-overlap: no two event boxes should visually overlap
Temporal ordering (optional but powerful): since these are historical events, you may want an additional soft constraint that events with earlier dates trend toward one side of the canvas — turning the layout into something between a pure causal graph and a timeline

Each of these becomes a differentiable term, and the total loss is a weighted sum:

L_total = w1 * L_attraction + w2 * L_repulsion + w3 * L_overlap + w4 * L_temporal

The weights (w1..w4) are hyperparameters you'll tune empirically — this is exactly analogous to loss weighting in multi-task neural network training, another well-known hard problem, so don't be surprised if getting a visually pleasing balance takes real iteration.

The Code Challenge

def attraction_loss(positions, edges):
    """
    edges: list of (i, j) node index pairs that are causally linked
    Pulls linked nodes together — squared distance, like a spring
    """
    i_idx = torch.tensor([e[0] for e in edges])
    j_idx = torch.tensor([e[1] for e in edges])
    # TODO: diff = positions[i_idx] - positions[j_idx]
    # TODO: return (diff ** 2).sum(dim=1).mean()
    ...

def repulsion_loss(positions, min_distance=2.0):
    """
    Pushes ALL pairs of nodes apart if they're closer than min_distance.
    This is O(n^2) -- fine for the node counts ChronoWeave targets
    (tens to low hundreds of events per graph), but know that this term
    is the one that would need approximating (e.g. Barnes-Hut) at scale.
    """
    n = positions.shape[0]
    # TODO: compute pairwise distance matrix
    # (hint: torch.cdist(positions, positions) does this in one call)
    dist_matrix = ...
    # TODO: for pairs closer than min_distance, penalize (min_distance - dist)^2
    # TODO: mask out the diagonal (distance of a node to itself = 0, don't penalize that)
    ...

def overlap_loss(positions, box_size=1.0):
    # TODO: similar to repulsion but specifically penalizes overlap of
    # fixed-size boxes -- can start as a simplified version of repulsion_loss
    # with min_distance = box_size, and refine later once you see real overlaps
    ...

def temporal_loss(positions, dates_normalized):
    """
    dates_normalized: tensor of shape (n,), each event's date mapped to
    [0, 1] (earliest event=0, latest=1)
    Soft-encourages x-coordinate to correlate with date
    """
    # TODO: target_x = dates_normalized * canvas_width
    # TODO: return ((positions[:, 0] - target_x) ** 2).mean()
    ...

def map_clarity_loss(positions, edges, dates_normalized, weights):
    w1, w2, w3, w4 = weights
    return (w1 * attraction_loss(positions, edges)
            + w2 * repulsion_loss(positions)
            + w3 * overlap_loss(positions)
            + w4 * temporal_loss(positions, dates_normalized))

The Aha Moment

Run the full optimization on a real extracted graph (from your Phase 2 pipeline!) with all four terms weighted, and watch causally-connected clusters visually separate from unrelated clusters, while individual nodes within a cluster stay legibly spaced apart. This is the actual "map" in ChronoWeave — and unlike a force-directed layout from a library, you can explain precisely why every node ended up where it did, because you wrote every term of the objective yourself.

The Extension

This is the function whose gradient — with respect to positions, not any neural network weight — drives literally everything the user sees on the canvas, and everything that happens when they drag a node in Chapter 4.

😤 The Struggle: The most common failure mode here is loss term imbalance — e.g. repulsion dominating so hard that attraction can't pull anything together, producing a uniform grid-like scatter with no visible clustering. When this happens, don't guess-and-check weights blindly. Log each individual loss term's value (not just the weighted sum) every N steps, and look at their relative magnitudes. If repulsion_loss is naturally 50x the scale of attraction_loss just from how you defined it (e.g. squared distances over many more pairs), your weights need to correct for that scale difference before they can express your actual priorities between the terms.

Resources

Force-directed graph drawing (for conceptual contrast — read this to understand what you're deliberately doing differently): Fruchterman & Reingold, "Graph Drawing by Force-Directed Placement" (1991) — search for the PDF, it's a classic short paper
torch.cdist docs: https://docs.pytorch.org/docs/stable/generated/torch.cdist.html

✅ Checkpoint 3.1: Take a graph with two clearly separate causal clusters (e.g. WWI causes and, unrelated, causes of the 2008 financial crisis, extracted from two different pasted texts) and verify optimization produces two visually distinct, non-overlapping clusters — not because you told it to, but because attraction pulls each cluster's internal nodes together while repulsion pushes the two clusters apart as aggregate masses.

Week 11: Custom Gradient Hooks and the Coordinate Optimizer

The Concept

Two refinements this week. First: gradient hooks. PyTorch lets you register a function that runs every time a gradient is computed for a specific tensor, which is how you'll implement per-node learning rate decay, gradient clipping for numerically unstable nodes, or debugging instrumentation (logging exactly which nodes have the largest gradients at each step — useful for diagnosing which part of the layout is "fighting" hardest).

Second: formalize your Phase 1 ManualAdam into a purpose-built CoordinateAdam that supports freezing individual coordinates — critical for Chapter 4's drag-and-reoptimize feature, where the dragged node must stay exactly where the user put it while every other node reoptimizes around it.

The Code Challenge

# Gradient hook example
def make_logging_hook(node_names):
    def hook(grad):
        # grad shape: (num_nodes, 2)
        # TODO: find and print the node with the largest gradient norm
        # this step -- useful for seeing which node is "hardest to place"
        ...
        return grad  # must return grad unchanged (or modified) -- don't return None
    return hook

positions.register_hook(make_logging_hook(node_names))

class CoordinateAdam:
    """
    Like your Phase 1 ManualAdam, but supports freezing a subset of
    coordinates (e.g. a node the user just dragged).
    """
    def __init__(self, positions: torch.Tensor, lr=0.05, betas=(0.9, 0.999), eps=1e-8):
        self.positions = positions
        self.lr, self.b1, self.b2, self.eps, self.t = lr, *betas, eps, 0
        self.m = torch.zeros_like(positions)
        self.v = torch.zeros_like(positions)
        # TODO: a boolean mask (num_nodes,) -- True = frozen, don't update
        self.frozen_mask = torch.zeros(positions.shape[0], dtype=torch.bool)

    def freeze(self, node_idx):
        self.frozen_mask[node_idx] = True

    def unfreeze(self, node_idx):
        self.frozen_mask[node_idx] = False

    def step(self):
        self.t += 1
        with torch.no_grad():
            g = self.positions.grad
            # TODO: zero out gradient rows for frozen nodes BEFORE the
            # Adam update, so their moment buffers don't drift either
            g = g.clone()
            g[self.frozen_mask] = 0.0

            self.m = self.b1 * self.m + (1 - self.b1) * g
            self.v = self.b2 * self.v + (1 - self.b2) * g ** 2
            m_hat = self.m / (1 - self.b1 ** self.t)
            v_hat = self.v / (1 - self.b2 ** self.t)
            update = self.lr * m_hat / (v_hat.sqrt() + self.eps)
            # TODO: apply update only to non-frozen rows
            update[self.frozen_mask] = 0.0
            self.positions -= update

    def zero_grad(self):
        if self.positions.grad is not None:
            self.positions.grad.zero_()

The Aha Moment

Freeze one node mid-optimization (simulate a user drag by just calling .freeze(node_idx) and manually setting that row of positions to a fixed target), re-run .step() for another 100 iterations, and watch every other node smoothly re-flow into a new stable configuration around the pinned node — while the pinned node itself doesn't move a single pixel, even though it still has a nonzero gradient every step.

The Extension

This freeze/unfreeze mechanism is exactly the backend logic Chapter 4's WebSocket drag handler calls into. You are not building new logic in Phase 4 — you're wiring this class up to a live event stream.

💡 The Innovation: This is the strongest, most concrete "why is this novel" answer for the whole project: "Standard force-directed layout libraries treat a dragged node as an external constraint bolted onto a physics simulation — usually by literally fixing its position and letting the simulation tick forward. My approach treats the drag as freezing one parameter's gradient within the exact same optimization loop used to generate the original layout, which means the re-layout objective is provably the same objective function, just re-solved under an added constraint — not a different, ad hoc mechanism." That's a real distinction, not a marketing line, and you can defend it because you wrote both halves.

Resources

PyTorch hooks documentation: https://docs.pytorch.org/docs/stable/notes/autograd.html#hooks-for-saved-tensors
register_hook API: https://docs.pytorch.org/docs/stable/generated/torch.Tensor.register_hook.html

Week 12: Debugging Gradient-Based Layout

The Concept

Gradient-based layout fails in ways that are different from — and often more confusing than — a standard training loop failing, because there's no "accuracy" metric to sanity check against, only a loss value and a visual result. This week is a deliberate practice week: you'll intentionally break your Week 10–11 system in several ways and learn to recognize the symptoms.

The Code Challenge — Diagnose Each of These Failure Modes

Everything collapses to one point. (Hint: check the sign and relative magnitude of your attraction vs. repulsion terms — is repulsion actually being computed, or is a masking bug zeroing it out?)
Layout oscillates forever, never converging. (Hint: learning rate too high for this loss surface, OR two loss terms directly fighting with equal and opposite gradients at every step — try lowering lr by 10x before assuming it's a logic bug)
NaN positions after N steps. (Hint: torch.cdist produces a zero distance for a node compared to itself; if your repulsion loss divides by distance anywhere instead of just using squared distance, you'll get a divide-by-zero on the diagonal. Always double-check you've masked the diagonal.)
One node flies off to extreme coordinates while everything else looks fine. (Hint: register a gradient hook — from Week 11 — on positions and print per-node gradient norms; the runaway node almost always has an anomalously large gradient from a bug specific to its edges, e.g. it participates in zero attraction edges so repulsion has nothing to balance against)

The Aha Moment

After deliberately inducing and then fixing all four failure modes above, run your full pipeline end-to-end — real text in, real triples extracted, real layout optimized — and watch it just... work, cleanly, without you needing to touch a single hyperparameter mid-run. That reliability is earned specifically by having debugged each failure mode once on purpose, rather than encountering them for the first time under deadline pressure in Phase 4.

The Extension

Phase 3 milestone: a working, debugged spatial intelligence engine that takes a causal graph and produces a legible 2D layout via gradient descent, with the ability to freeze nodes and re-optimize live.

🧭 The Mentor Says: Keep a running "failure log" as a markdown file next to your code: every bug you hit, what the symptom looked like, and what actually fixed it. Six months from now when this happens again in a different project, that log is worth more than any Stack Overflow search, because it's your debugging vocabulary for this specific class of problem (gradient-based systems with non-standard parameters), which almost nobody else has written down anywhere.

✅ Checkpoint 3.2 — End of Phase 3: Explain to a rubber duck (or a friend, or a mirror) why gradient descent on coordinates is a valid approach to graph layout at all — i.e., why the loss function's gradient with respect to position tells you a meaningful direction to move a node, given that "position" isn't something with an obvious ground truth the way a classification label is. (The core answer: there's no ground truth position, but there's a well-defined relative preference — closer is better for linked nodes, farther is better for unlinked ones — and gradient descent is a general algorithm for finding local optima of any differentiable preference function, not just supervised losses with ground truth targets.)

CHAPTER 4: The Body — Giving Your Engine a Web Presence

(Weeks 13–16 · Prerequisite: Phase 3 complete)

Week 13: FastAPI Backend and WebSockets

The Concept

Two distinct interaction patterns need two different transport mechanisms. The initial "paste text, click Generate" flow is a classic request/response — REST fits fine. But the drag-and-reoptimize flow needs the server to push a stream of intermediate frames to the client as optimization runs (so the user sees nodes flow smoothly into place rather than jumping instantly), which REST can't do — that's what WebSockets are for.

The Code Challenge

from fastapi import FastAPI, WebSocket
from fastapi.responses import JSONResponse

app = FastAPI()

@app.post("/api/extract")
async def extract(payload: dict):
    text = payload["text"]
    # TODO: run ExtractionPipeline (Chapter 2) -> triples
    # TODO: run spatial optimization (Chapter 3) to convergence
    # TODO: return {"nodes": [...], "edges": [...]}
    ...

@app.websocket("/ws/relayout/{graph_id}")
async def relayout(websocket: WebSocket, graph_id: str):
    await websocket.accept()
    # TODO: receive {"dragged_node_id": ..., "new_x": ..., "new_y": ...}
    drag_event = await websocket.receive_json()

    # TODO: load current positions for graph_id, freeze the dragged node
    #       at (new_x, new_y) using CoordinateAdam.freeze() from Chapter 3
    # TODO: run optimization step-by-step (not to convergence in one go!)
    #       and after every K steps, send the current positions:
    for step in range(200):
        # optimizer.step()...
        if step % 5 == 0:
            # TODO: await websocket.send_json({"positions": [...]})
            ...
    await websocket.close()

The Aha Moment

Open your browser's network tab, trigger a drag, and literally watch a stream of JSON frames arrive over the WebSocket connection every few milliseconds — the exact same optimization loop from Chapter 3, just now visible as it happens, one HTTP-adjacent message at a time.

The Extension

This is the wiring that turns Chapter 3's offline optimization script into a live, interactive feature.

Resources

FastAPI WebSockets guide: https://fastapi.tiangolo.com/advanced/websockets/

Week 14: React + D3.js Rendering

The Concept

D3 excels at binding data to SVG elements and handling enter/update/exit transitions smoothly — exactly what you need when node positions update every few WebSocket frames. React manages component state and lifecycle; D3 manages the actual SVG manipulation and smooth transitions between positions. The common pattern: let React own the DOM structure (which nodes/edges exist), let D3 own the transitions (how a node's x/y animates from old position to new).

The Code Challenge

// TODO: a GraphCanvas component that:
// 1. Receives `nodes` and `edges` as props (from the /api/extract response)
// 2. Renders SVG circles for nodes, lines for edges, using D3 scales
//    to map data coordinates to screen coordinates
// 3. Uses d3.transition() to animate position changes smoothly whenever
//    the `nodes` prop updates (i.e., whenever a new WebSocket frame arrives)
// 4. Attaches d3.drag() behavior to each node, which on 'end' fires a
//    callback (e.g. onNodeDragged(nodeId, newX, newY)) that the parent
//    component uses to open the /ws/relayout WebSocket and stream in
//    the live re-layout frames from Week 13

function GraphCanvas({ nodes, edges, onNodeDragged }) {
  // TODO: useRef for the svg element, useEffect that runs D3 rendering
  // logic whenever `nodes` or `edges` change
}

The Aha Moment

Drag a node and watch not just that node move, but every connected node smoothly animate to its new position over the following second or two — not teleporting, not choppy, an actual fluid re-settling. This is the "magic moment" the entire project overview promised in Chapter 0, now real in a browser.

The Extension

Nothing further — this is the product. Everything from Chapter 1 onward has been building toward this specific ten seconds of interaction.

Resources

"Learn D3: Joining Data" (Observable): https://observablehq.com/@d3/learn-d3-joining-data
Amelia Wattenberger, "React + D3" (a well-regarded pattern guide for combining the two): https://wattenberger.com/blog/react-and-d3

😤 The Struggle: React re-rendering and D3 both wanting to own the DOM is a classic source of fights — D3 mutates elements directly, React expects to control them via its virtual DOM diff. The cleanest fix most people converge on: let React render the SVG container and static structure, but hand D3 a ref to manipulate node/edge elements directly inside a useEffect, and never let React's render also try to set those same attributes. Pick one owner per DOM property.

Week 15: Deployment

The Concept

Docker Compose ties your FastAPI backend, Postgres+pgvector database, and (optionally) a separate model-serving container into one reproducible unit you can run locally exactly as it'll run in the cloud — closing the "works on my machine" gap before it costs you a demo day.

The Code Challenge

# docker-compose.yml — TODO fill in the blanks
services:
  db:
    image: pgvector/pgvector:pg16
    environment:
      POSTGRES_PASSWORD: ...
    volumes:
      - pgdata:/var/lib/postgresql/data

  backend:
    build: ./backend
    depends_on:
      - db
    environment:
      DATABASE_URL: postgresql://...@db:5432/...
    ports:
      - "8000:8000"
    # TODO: mount your fine-tuned model weights as a volume rather than
    # baking them into the image -- keeps image builds fast during dev

  frontend:
    build: ./frontend
    ports:
      - "3000:3000"

volumes:
  pgdata:

The Aha Moment

docker-compose up on a completely fresh machine (or a cloud VM) and the whole system — model inference, database, backend, frontend — comes up correctly with zero manual steps beyond that one command. That's the difference between "a project on my laptop" and "a project."

The Extension

Deploy to Render, Fly.io, or a similar platform for a public demo URL — essential for the hackathon-ready, portfolio-ready version of this project.

Resources

Docker Compose docs: https://docs.docker.com/compose/
Render's Docker deployment guide: https://render.com/docs/docker

Week 16: Polish and the Full Demo

The Concept

The last week is deliberately not about new features. It's about the unglamorous 20% that makes a demo feel finished: loading states while extraction/optimization runs, graceful handling of bad input, a couple of pre-baked example texts a judge or interviewer can one-click try instead of needing to paste their own, and a README that explains the project in 30 seconds.

The Code Challenge

Add a loading skeleton/spinner state to the frontend while /api/extract is in flight (this can take several seconds — a live optimization loop is not instant, and users need to know it's working, not frozen)
Add 2–3 pre-loaded example texts (a good default: causes of WWI, causes of the French Revolution, causes of the 2008 financial crisis)
Write a project README with: what it does, a GIF of the drag-and-reoptimize interaction, architecture diagram, how to run it locally

The Aha Moment

Show the finished product to someone who has never seen it, say nothing, and watch them paste in their own text and drag a node without prompting. If they intuitively understand what happened without you explaining it, the product design succeeded.

Phase 4 Milestone — Project Complete

A live, deployed, full-stack application: paste historical text, get an extracted causal graph laid out via gradient descent, drag any node and watch the rest of the graph re-optimize in real time — all running on infrastructure you understand end to end, built from raw tensor operations up.

EPILOGUE: Beyond ChronoWeave — What You Can Build Next

Portfolio Presentation

Lead with the live demo, not the tech stack list. The first ten seconds should be someone pasting text and dragging a node — let the magic moment sell itself before you explain any internals.

Research Paper Possibilities

The "gradient descent as general-purpose graph layout" idea, generalized beyond history specifically, touches on active research areas: differentiable graph drawing, learned layout objectives, and combining symbolic graph constraints with continuous optimization. A write-up comparing your Map Clarity Loss approach against classical force-directed layout on layout-quality metrics (edge crossing count, node overlap, cluster separation) would be a legitimate small research contribution, or at minimum a strong technical blog post.

Scaling the System

Current design targets tens to low-hundreds of nodes per graph (the O(n²) repulsion term is the binding constraint). Scaling further means either approximating repulsion (Barnes-Hut / quadtree methods, the same trick classical force-directed layout libraries use at scale) or moving to a hierarchical layout where clusters are optimized independently and then composed.

APPENDIX A: First Week Setup Guide (Day 1)

# 1. Install Python 3.11+, then:
curl -sSL https://install.python-poetry.org | python3 -
poetry --version   # verify install

# 2. Project scaffold
mkdir chronoweave && cd chronoweave
poetry init --name chronoweave-ml -n
poetry add torch numpy matplotlib jupyter transformers

# 3. Verify PyTorch sees your GPU (if you have one -- CPU is fine for Phase 1)
poetry run python -c "import torch; print(torch.cuda.is_available())"

# 4. Launch a notebook and run the Week 1 autograd exercise before doing
#    anything else -- confirm your environment works before building on it
poetry run jupyter notebook

If you don't have a GPU: everything in Phase 1 and most of Phase 3 runs fine on CPU (small networks, small graphs). Phase 2's BERT fine-tuning is the one place a GPU meaningfully helps — Google Colab's free tier is sufficient for the scale of fine-tuning this project needs.

APPENDIX B: Troubleshooting

Symptom	Likely Cause	Fix
Loss doesn't decrease at all	Forgot `zero_grad()` before `.backward()`	Gradients are accumulating across steps — add `zero_grad()` at the top of the loop
`RuntimeError: ...does not require grad and does not have a grad_fn`	A leaf tensor got overwritten outside `torch.no_grad()`	Check every in-place parameter update is inside `with torch.no_grad():`
Loss becomes `NaN` after N steps	Divide-by-zero, usually in a pairwise-distance loss term where the diagonal (self-distance = 0) isn't masked	Mask the diagonal before dividing by any distance term
Everything collapses to a point during layout optimization	Repulsion term isn't actually contributing (masking bug or weight of 0)	Log each loss term separately, verify repulsion is nonzero and comparable in scale to attraction
NER model predicts `O` for everything	Label misalignment between words and WordPiece subtokens	Spot-check `tokenizer.convert_ids_to_tokens()` against your label array on several examples
WebSocket disconnects mid-reoptimization	Sending too many frames too fast, or an unhandled exception mid-loop killing the coroutine	Throttle to sending every 5th step, wrap the optimization loop in try/except with a graceful `websocket.close()` on error
D3 nodes flicker or don't animate smoothly	React re-render fighting with D3's direct DOM manipulation	Ensure only one of React/D3 sets a given DOM attribute; do D3 updates inside `useEffect`, not render
Docker Compose backend can't reach the database	Using `localhost` instead of the service name in `DATABASE_URL`	Compose networking resolves service names (`db`), not `localhost`, between containers

APPENDIX C: Project Timeline (Text Gantt Chart)

Week:        1  2  3  4  5  6  7  8  9  10 11 12 13 14 15 16
Phase 1      ██ ██ ██ ██
  Autograd    ██
  Manual NN      ██
  Optimizers        ██
  Tiny Xformer         ██
Phase 2                  ██ ██ ██ ██
  NER fine-tune            ██
  Relation extract            ██
  Data pipeline                   ██
  Evaluation                         ██
Phase 3                              ██ ██ ██ ██
  Coords as params                     ██
  Map Clarity Loss                        ██
  Grad hooks + optimizer                     ██
  Debugging practice                            ██
Phase 4                                          ██ ██ ██ ██
  FastAPI + WS                                     ██
  React + D3                                          ██
  Deployment                                             ██
  Polish + demo                                             ██

APPENDIX D: Success Metrics — How Do I Know I Built It Correctly?

Phase 1 (correctness of understanding, not the app):

You can derive, on paper, the gradient of MSE loss with respect to a 2-layer network's weights, without a reference
Your manual Adam implementation converges at a comparable rate to torch.optim.Adam on the same toy problem
You can explain attention's Q/K/V roles without analogy, in precise terms

Phase 2 (quantitative, on your held-out eval set):

NER span-level F1 ≥ 0.75 on your hand-labeled eval set (BERT-base fine-tuned on ~100+ examples should clear this comfortably)
Relation classifier per-class F1 ≥ 0.6 on CAUSES specifically (the class you care most about); lower is acceptable for rarer/harder classes like ENABLED_BY
Pipeline handles at least 3 different genuinely messy real-world text samples without crashing

Phase 3 (behavioral, verified visually + numerically):

Map Clarity Loss converges to a stable value (not oscillating) within 500 optimization steps on a graph of ~20 nodes
Two independently-clustered causal groups in the same graph visually separate without being told to
Freezing a node mid-optimization and resuming produces a stable re-layout where the frozen node's position never changes by more than floating-point rounding error

Phase 4 (product-level):

End-to-end latency from clicking "Generate" to seeing a laid-out graph is under ~10 seconds for a few paragraphs of input
Drag-and-reoptimize visually completes (settles, stops moving) within ~2 seconds
A first-time user, with zero explanation, successfully pastes text and drags a node within 60 seconds of opening the app

APPENDIX E: Interview Prep — Explaining ChronoWeave to a Hiring Manager

The 30-second version:
"I built a system that reads historical text, extracts cause-and-effect relationships between events using a fine-tuned Transformer, and lays those events out on a 2D map using gradient descent instead of a physics engine — meaning the layout is generated by backpropagating a custom loss function with respect to node coordinates, the same way you'd train a neural network, except the 'parameters' are positions on a canvas instead of weights."

Anticipated follow-up questions and how to answer them:

"Why not just use a force-directed graph library?" — Because I needed the layout objective to express more than physical forces can naturally capture: e.g., a soft temporal-ordering constraint alongside causal clustering, with tunable relative weights between competing goals, and the ability to freeze arbitrary subsets of nodes mid-optimization while re-solving the same objective for everyone else. Gradient descent on a custom differentiable loss generalizes to all of that; a spring simulation would need a bespoke mechanism bolted on for each one.
"How did you handle relation extraction being noisy?" — I measured it rather than assumed it: held-out precision/recall/F1 per relation class, a confidence threshold before a triple gets displayed, and I designed the frontend to reflect uncertainty where the model itself is uncertain (e.g., visually softer distinction between relation types the model confuses most, per the confusion matrix).
"What was the hardest bug?" — Talk about one specific failure mode from Chapter 3 Week 12 in detail (e.g., the NaN-from-unmasked-diagonal bug) — specificity here is what separates "I used ML" from "I understand ML."
"Why build the optimizer from scratch instead of using torch.optim.Adam directly?" — Because I needed per-node freezing during live re-layout, which meant controlling exactly how the moment buffers and update step interacted with a frozen mask — not something the standard API exposes, and writing it myself meant I fully understood what I was modifying.
"What would you do differently at scale?" — The repulsion loss term is O(n²); past a few hundred nodes I'd move to an approximate method (Barnes-Hut quadtree, the same technique classical force-directed layouts use), or a hierarchical approach that optimizes clusters independently before composing them.

End of guide. This document covers the full architecture, all four learning phases, and the operational appendices for building ChronoWeave. Treat each chapter's code skeletons as a starting point to fill in and argue with — the goal was never for you to run code someone else wrote, but to be the one who could have written PyTorch's autograd yourself, at least once, in miniature.

Building Your Own Deep Learning Framework: A 3–5 Month Journey From Zero to Tiny Transformer

Biki Kalita — Thu, 23 Jul 2026 14:24:34 +0000

Prologue: The Machine That Learns to Learn

Somewhere in a research lab in the early 2010s, a small group of engineers sat staring at a whiteboard covered in arrows. They weren't drawing a neural network. They were drawing a graph of computation — a map of every multiplication, every addition, every function that data would pass through on its way to becoming a prediction. That whiteboard sketch, refined over years, eventually became the beating heart of PyTorch: a system that doesn't just compute forward, but remembers how it got there so it can compute backward — and learn.

Here's the secret almost nobody tells beginners: PyTorch is not magic. It's not some impenetrable fortress of genius that only PhDs can understand. At its core, it's a few ideas, executed carefully:

Track every operation performed on a number.
Remember how to undo that operation mathematically (its derivative).
Walk backward through that history, multiplying derivatives together (the chain rule).
Use the result to nudge numbers in a direction that reduces error.

That's it. That's the whole trick. Everything else — tensors, GPUs, layers, optimizers, transformers — is engineering built on top of that one idea.

Over the next 3 to 5 months, you are going to rebuild that idea yourself, from the ground up, in Python. Not because the world needs another deep learning framework. But because there is no faster way to actually understand deep learning than to build the thing that makes deep learning possible.

This is not a copy-paste tutorial. I will not hand you a finished library. What I will hand you is a map, a set of checkpoints, the theory you need at each stage, the mistakes you're likely to make, and the tests that will tell you whether you're right. The actual code — the thinking, the debugging, the "aha" moments — that has to be yours. That's where the learning lives.

Let's begin.

Part 1: Why Build This At All?

There are three ways to learn deep learning:

Use a framework. You import torch, call .backward(), and a neural network trains. You learn what deep learning does, but the internals stay a black box.
Read papers and textbooks. You learn the math in isolation, but it often stays abstract — symbols on a page that never quite connect to working code.
Build the framework yourself. You are forced to confront every hidden assumption, every edge case, every "wait, why does this even work?" moment, because nothing works until you make it work.

The third path is slower. It is also the only one that produces the kind of deep, load-bearing understanding that shows up in job interviews, research intuition, and your ability to debug a model that refuses to train. When you've implemented backpropagation by hand, you will never again see a loss.backward() call as a magic incantation. You'll see it as: "gradients flowing backward through a graph I understand, because I've built one."

This project is also a phenomenal way to practice a skill that's rarely taught explicitly: reading the world's collective knowledge and turning it into your own working system. You won't invent autodiff from nothing. You'll read blog posts, skim papers, study existing tiny frameworks like micrograd and tinygrad for inspiration (not for copying), and synthesize that knowledge into code that is unmistakably yours. That synthesis skill — reading, understanding, rebuilding — is exactly what separates engineers who can only follow tutorials from engineers who can solve novel problems.

By the end, you will have a working library that can:

Automatically differentiate arbitrary chains of operations (autodiff)
Represent and manipulate multi-dimensional arrays with broadcasting (a tensor engine)
Compose neural network layers (Linear, activations, normalization)
Compute losses and optimize weights (SGD, Adam)
Train a small transformer to generate text, character by character

And more importantly, you will understand why every piece exists.

Part 2: The Big Picture — What You're Actually Building

Before writing a single line of code, hold the whole system in your head, even blurrily. Deep learning frameworks are built in layers, like geological strata. Each layer depends on the one below it:

Layer 5:  Transformer blocks (attention, feedforward, positional encoding)
Layer 4:  Training loop (forward → loss → backward → optimizer step)
Layer 3:  Losses & Optimizers (MSE, cross-entropy, SGD, Adam)
Layer 2:  Neural network layers (Linear, ReLU, Softmax, LayerNorm)
Layer 1:  Tensor engine (N-dimensional arrays, broadcasting, matmul)
Layer 0:  Autodiff engine (the core: track operations, compute gradients)

Notice the direction of dependency: a transformer is just layers, arranged cleverly. Layers are just tensor operations, arranged into reusable objects. Tensor operations are just autodiff-tracked math. Everything above Layer 0 is organization. Layer 0 is the one true foundation. This is why you'll spend real, unhurried time there before moving on — everything else stands or falls on whether your autodiff engine is correct.

By the end of this project, you should be able to say, in plain language and without hand-waving:

"I built a system where every math operation remembers its inputs and knows its own derivative. When I ask for gradients, the system walks backward through the history of operations it recorded, applying the chain rule at each step, until every parameter knows exactly how much it contributed to the final error — and therefore how it should change."

If you can say that sentence and mean it, mission accomplished. Everything else is detail.

Part 3: The Roadmap — Nine Phases Over 3 to 5 Months

Here is the full journey. Don't worry about memorizing it — we'll walk through each phase in depth below. Treat this as the trail map you'll keep returning to.

Phase	Focus	Approx. Duration
1	Python + NumPy foundations, mathematical warm-up	1–2 weeks
2	Scalar autodiff engine (the heart of the project)	2–3 weeks
3	Tensor engine with broadcasting & matmul	3–4 weeks
4	Neural network layers (Linear, activations, init)	2 weeks
5	Losses and optimizers (SGD, momentum, Adam)	1–2 weeks
6	Training loop + first real experiments (XOR, MNIST-lite)	2 weeks
7	Transformer components (attention, embeddings, LayerNorm)	3–4 weeks
8	Mini language model demo (character-level generation)	2–3 weeks
9	Cleanup, tests, documentation, packaging	1–2 weeks

That totals roughly 17–24 weeks — comfortably inside your 3–5 month window, with room for the inevitable week where nothing works and you have to slow down. Budget for that week. It will happen, probably more than once, and it is not a sign of failure. It's a sign you've hit a real idea.

A sane weekly rhythm: 3–5 focused sessions per week, 1–3 hours each, mixing reading/study with hands-on building. Deep learning concepts reward spaced repetition — you'll understand backprop better the third time you re-derive it than the first.

Phase 1: Python + NumPy Foundations

What you're learning

Comfort with NumPy's array operations, broadcasting rules, and vectorized thinking — because your tensor engine will essentially be a thin, gradient-aware wrapper around NumPy.

Why it matters

If you don't deeply understand how NumPy broadcasts shapes like (3, 1) and (1, 4) into (3, 4), you will not understand why your tensor engine's gradients sometimes come out the wrong shape later. This bug bites everyone, and the fix is always the same: understand broadcasting now, thoroughly, so you recognize the symptom later.

What to study online

NumPy's official broadcasting documentation (read it twice — once now, once after Phase 3)
Any solid "vectorization vs. loops" tutorial that shows the same computation written both ways
A refresher on matrix multiplication, dot products, and the chain rule from calculus (Khan Academy or 3Blue1Brown's "Essence of Calculus" series are excellent for building visual intuition)

What to build yourself

A handful of small NumPy exercises: matrix multiply without np.dot, broadcast a (3,1) array against a (1,4) array and predict the output shape before running it, implement softmax and sigmoid by hand.
A "shape tracer" habit: for every array operation you write, say out loud what shape goes in and what shape comes out.

Tiny illustrative idea

a = np.array([[1], [2], [3]])   # shape (3, 1)
b = np.array([10, 20, 30, 40])  # shape (4,)
# predict the output shape BEFORE running a + b

How to test yourself

Before running any broadcasting expression, write down the predicted output shape on paper. Then check. If you're wrong more than once, stop and reread the broadcasting rules until you can predict shapes reliably.

Common mistakes

Assuming broadcasting "just works" without understanding the alignment rule (dimensions are compared from the right, and must either match or be 1).
Confusing element-wise multiplication (*) with matrix multiplication (@ or np.matmul).

You're ready to move on when…

You can look at two array shapes and confidently predict whether they'll broadcast together, and what shape the result will be — without running code to check.

Mini exercises

Implement softmax for a batch of vectors (shape (batch, classes)) using only NumPy, handling numerical stability (subtract the max before exponentiating).
Write a function that multiplies two matrices using nested Python loops, then verify it matches A @ B.

Phase 2: Scalar Autodiff — The Heart of Everything

This is the phase that matters most. Take it slow. Everything you build later is a generalization of what you build here.

What you're learning

How to build a system where every number "remembers" the operations that created it, so that gradients can be computed automatically via the chain rule. This is the single idea that makes deep learning trainable.

Why it matters

Backpropagation is often taught as an abstract algorithm on paper. Building it yourself turns it into something concrete: a graph of tiny objects, each holding a value, a gradient, and a memory of its parents. Once you've built this for scalars, tensors are "just" the same idea applied to arrays.

What to study online

Andrej Karpathy's "The spelled-out intro to neural networks and backpropagation" video and the micrograd repository. Watch it, understand the ideas, then close the tab and build your own version from memory and understanding — don't transcribe his code.
A clear explanation of computational graphs (search "computational graph backpropagation explained") — focus on the idea of a DAG (directed acyclic graph) and topological sort, because that's how you'll decide the correct order to walk backward through the graph.
The chain rule, revisited specifically in the context of composed functions: if z = f(y) and y = g(x), then dz/dx = dz/dy * dy/dx.

What to build yourself

Design a Scalar (or Value) class that wraps a single float and:

Stores its numeric value
Stores a gradient (grad), initialized to zero
Stores references to the "parent" scalars that created it (if any)
Stores a function that knows how to propagate gradient backward to those parents
Overloads Python operators (__add__, __mul__, __pow__, and later things like tanh or relu) so that writing normal-looking math (c = a * b + a) automatically builds a graph behind the scenes
Implements a .backward() method that performs a topological sort of the graph, then walks it in reverse, accumulating gradients using the chain rule

Tiny illustrative idea

Don't copy this — but here's the shape of the thinking, for a single operation:

def __mul__(self, other):
    out = Scalar(self.value * other.value, parents=(self, other))
    def _backward():
        self.grad += other.value * out.grad
        other.grad += self.value * out.grad
    out._backward_fn = _backward
    return out

Notice the pattern: every operation defines its own tiny local rule for how gradient flows backward to its inputs. That's the whole idea, repeated for every operation you support.

How to test it

This is where you build the habit that will save you for the rest of the project: compare your analytical gradient against a numerical gradient.

The numerical gradient of a function f at point x can be approximated as:

(f(x + h) - f(x - h)) / (2h)

for a small h (like 1e-5). If your hand-derived, backprop-computed gradient doesn't match this numerical approximation within a small tolerance, your backward pass has a bug — full stop, no exceptions. This technique is called gradient checking, and it is the single most important debugging tool in this entire project. Build it early. Use it constantly.

Common mistakes

Forgetting to zero out gradients between training steps (gradients accumulate by design, so you must explicitly reset them, or they'll silently corrupt every subsequent step).
Getting the local derivative wrong for a specific operation (e.g., writing the derivative of x**2 as 2 instead of 2*x).
Forgetting that when a value is used multiple times in a graph (e.g., y = x + x), gradients from both usages must be summed, not overwritten. This is the single most common bug in every autodiff implementation ever written by a beginner.
Building the backward traversal in the wrong order (not respecting topological order), which causes gradients to be computed using stale, not-yet-fully-accumulated values.

You're ready to move on when…

You can build an arbitrary expression using +, *, **, and tanh, call .backward() on the output, and have every intermediate scalar's .grad match a numerically-computed gradient within 1e-4 or so — for at least five different hand-built expressions, including ones where a variable is reused multiple times.

Mini exercises

Build the expression L = (a * b + c) ** 2 for scalar values a, b, c. Call backward and verify all three gradients against numerical gradient checking.
Deliberately introduce the "reused variable" bug (don't accumulate, just overwrite gradients) and observe how the numerical check catches it. This will burn the failure mode into your memory permanently.

Phase 3: The Tensor Engine — Scaling Up From Scalars

What you're learning

How to generalize your scalar autodiff engine to operate on N-dimensional arrays (tensors), including the trickiest and most bug-prone part of any framework: gradient broadcasting.

Why it matters

Real neural networks don't operate on individual numbers — they operate on batches of vectors and matrices, for speed and for the mathematical elegance of representing whole layers as matrix multiplications. This phase is where your framework becomes genuinely useful rather than a toy.

What to study online

Revisit NumPy broadcasting rules, this time thinking specifically about the reverse problem: if a tensor was broadcast during the forward pass (say, a (1,4) bias added to a (32,4) batch), how must the gradient be summed back down to the original smaller shape during the backward pass? (Hint: you sum gradients over exactly the dimensions that were broadcast.)
Matrix calculus basics — specifically the gradient of a matrix multiplication C = A @ B with respect to A and B. Search for "gradient of matrix multiplication backpropagation" and work through the derivation on paper before coding it.
The tinygrad or micrograd source code, again for inspiration and comparison after you've attempted your own design — not as a template to copy line by line.

What to build yourself

A Tensor class wrapping a NumPy array, with the same structure as your Scalar class: value, gradient, parent references, and a local backward function per operation.
Core operations: elementwise +, -, *, /, matrix multiplication (@), sum, mean, reshape, and broadcasting-aware arithmetic.
A .backward() method that performs topological sort and reverse traversal, just like your scalar engine — but now the "local backward rule" for each operation must handle array shapes correctly, including reducing broadcasted gradients back down.

Tiny illustrative idea

The conceptual shape of the fix for broadcast gradients (not literal code to paste, but the idea to implement yourself):

"After computing the gradient at the operation's output shape, if the input was broadcast to get there, sum the gradient over the broadcasted axes until it matches the input's original shape."

How to test it

Gradient checking again — but now applied to whole tensors. Perturb a single entry of a tensor by h, recompute the forward pass, measure the change in the loss, and compare to your backprop-computed gradient at that entry. Do this for a random sample of entries across several test cases: elementwise ops, broadcasting ops, and matrix multiplication.

Also write shape assertions liberally during development: after every backward operation, assert that the gradient's shape exactly matches the corresponding tensor's shape. Shape mismatches are the number one symptom of a broadcasting bug, and catching them immediately (rather than three layers later) will save you hours.

Common mistakes

Forgetting to sum gradients back down after broadcasting (the gradient tensor ends up the wrong shape, or numerically wrong even if the shape happens to match).
Getting the matrix multiplication gradient transposed incorrectly (a very common and very confusing bug — if C = A @ B, then the gradient w.r.t. A involves B transposed, and the gradient w.r.t. B involves A transposed; get this backwards and your loss will often still decrease, just slower or incorrectly, which makes the bug sneaky).
Silent shape bugs: NumPy broadcasting is forgiving, so a wrong-shape gradient often doesn't crash — it just silently gives you wrong numbers. This is why explicit shape assertions matter more here than almost anywhere else in the project.

You're ready to move on when…

You can build a small computation involving matrix multiplication, broadcasting (like adding a bias vector to a batch), and elementwise operations, call backward, and have every gradient match numerical gradient checking — and you understand, in your own words, why broadcasted gradients need to be summed back down.

Mini exercises

Implement y = X @ W + b where X is (batch, in_features), W is (in_features, out_features), and b is (out_features,). Verify all three gradients numerically.
Deliberately break the broadcasting-gradient-reduction step and observe what kind of numerical mismatch results — get familiar with what "wrong" looks like on this specific bug.

Phase 4: Neural Network Layers

What you're learning

How to organize tensor operations into reusable, composable objects — the Module abstraction that every framework uses (PyTorch calls it nn.Module).

Why it matters

This is where your project starts to feel like a real framework. You'll design an API elegant enough that stacking layers to build a network feels natural, not clunky.

What to study online

PyTorch's nn.Module documentation — not to copy the implementation, but to study the interface design: how parameters are registered, how forward() is called, how submodules nest inside each other.
Weight initialization strategies (Xavier/Glorot, Kaiming/He) — read the original reasoning for why initialization scale matters (poor initialization causes vanishing or exploding activations before training even starts).

What to build yourself

A base Module class with a parameters() method that can recursively collect every trainable tensor from itself and its submodules.
A Linear layer (y = xW + b) with properly initialized weights.
Activation functions as their own small modules or functions: ReLU, Tanh, Sigmoid, Softmax.
A way to compose modules sequentially (a Sequential container, or simply nesting modules inside a custom class with a forward method).

How to test it

Forward-pass shape tests: build a small network, push a batch through it, and confirm the output shape is exactly what you expect at every layer.
Initialization sanity checks: after initializing a Linear layer, check that the variance of its weights roughly matches what your chosen initialization scheme predicts (this catches typos in initialization formulas early, before they cause mysterious training failures three phases from now).
Gradient flow sanity check: run a forward and backward pass through a multi-layer network and confirm gradients exist (are non-zero and non-NaN) for every parameter, at every layer, not just the last one. This catches "dead" layers early.

Common mistakes

Initializing weights to all zeros (a classic mistake — with zero weights, every neuron in a layer computes the exact same output and gradient, so they never differentiate from each other, a failure mode called symmetry).
Forgetting to zero gradients before each new backward pass, causing gradients to silently accumulate across training steps.
Building forward() methods that accidentally break the autodiff graph — for example, converting a Tensor to a raw NumPy array partway through a computation, which severs the graph and makes .backward() unable to reach earlier layers.

You're ready to move on when…

You can build a 2–3 layer network with your Module system, push a batch of random data through it, call backward, and confirm every single parameter (from the first layer to the last) receives a sensible, non-zero, non-NaN gradient.

Mini exercises

Build a Sequential container that chains Linear → ReLU → Linear and verify the output shape for a batch input.
Deliberately zero-initialize a layer's weights and observe the symmetry problem directly — confirm that all neurons in that layer produce identical gradients.

Phase 5: Losses and Optimizers

What you're learning

How to measure "wrongness" (loss functions) and how to translate gradients into actual parameter updates (optimizers).

Why it matters

Loss functions define what the network is trying to learn. Optimizers define how it gets there. A subtle bug in either one produces the most frustrating kind of failure: a model that trains, technically, but never quite gets good — and you won't immediately know which piece is at fault.

What to study online

Mean Squared Error and Cross-Entropy loss — read a derivation of why cross-entropy is the natural loss for classification (its connection to maximum likelihood estimation is worth understanding, not just memorizing the formula).
The original SGD-with-momentum intuition (think of it as a ball rolling downhill, accumulating velocity) and the Adam paper's core idea (adaptive per-parameter learning rates using running estimates of the gradient's mean and variance).

What to build yourself

MSELoss and CrossEntropyLoss (careful with the numerically stable combination of softmax + cross-entropy — computing them separately can cause overflow; study how frameworks combine them into one numerically stable operation).
An Optimizer base structure, then SGD (with optional momentum) and Adam, each implementing a step() method that reads .grad off every parameter and updates .value accordingly, plus a zero_grad() method.

How to test it

Compare your loss function's output against a hand-computed value on a tiny example with numbers small enough to check by hand or calculator.
Overfit a tiny synthetic dataset (even just 4–8 data points) with your optimizer and confirm the loss goes essentially to zero. If it can't even overfit a handful of points, something is fundamentally broken before you try anything harder.
For Adam specifically, verify the bias-correction terms are implemented — a very common subtle bug is forgetting bias correction on the first few steps, which causes unstable early training.

Common mistakes

Not zeroing gradients before each step(), so gradients from previous batches contaminate the current update.
Using the wrong sign in the update rule (subtracting instead of adding, or vice versa — always double check: parameters move in the negative gradient direction to reduce loss).
Learning rate too high (loss explodes or oscillates) or too low (loss barely moves) — learn to recognize both failure signatures by deliberately inducing them once.

You're ready to move on when…

You can take a tiny synthetic dataset, run several hundred optimization steps with either SGD or Adam, and watch the loss curve monotonically (or nearly so) decrease toward zero — and if it doesn't, you know how to diagnose why.

Mini exercises

Implement plain SGD and use it to fit a single Linear layer to noisy linear data (y = 3x + 2 + noise) — check that the learned weight and bias converge close to 3 and 2.
Implement Adam and compare its convergence speed against plain SGD on the same tiny problem — observe (don't just read about) why adaptive methods often converge faster on ill-conditioned problems.

Phase 6: The Training Loop and First Real Experiments

What you're learning

How to assemble everything into the actual training loop — the rhythm of forward pass, loss computation, backward pass, and optimizer step that every deep learning system repeats millions of times.

Why it matters

This is the moment your framework stops being a pile of isolated pieces and becomes a system. It's also, often, the first moment something genuinely surprising happens: a model that actually learns a nontrivial function, using code you wrote entirely yourself.

What to study online

Classic explanations of the XOR problem and why it historically mattered (a single-layer perceptron cannot solve it — this is a famous, important lesson in why depth and nonlinearity matter).
General training loop patterns across frameworks — notice the near-universal shape: zero_grad() → forward() → compute loss → backward() → step().

What to build yourself

A full training loop for the XOR problem: 4 data points, a small 2-layer network with a nonlinear activation, and your loss + optimizer from Phase 5.
Basic logging: print or plot the loss every N steps so you can watch it decrease (or fail to).
If you're feeling ambitious, a tiny subset of MNIST (even just a few hundred examples, downsampled) as a slightly harder next test.

How to test it

Watch the loss curve. It should decrease, generally, though not necessarily perfectly monotonically.
Check final predictions against ground truth directly — for XOR, after training, does the network correctly predict 0 for (0,0) and (1,1), and 1 for (0,1) and (1,0)?
This is a good moment to also test overfitting a single batch as a diagnostic reflex you'll use for the rest of your career: if a model can't overfit a tiny amount of data, it has a bug, not a "needs more data" problem.

Common mistakes

Forgetting zero_grad() at the start of each loop iteration (the single most common training-loop bug in every framework, home-built or professional).
Using a network without a nonlinearity between layers, which collapses multiple linear layers into a mathematically equivalent single linear layer — and a single linear layer cannot solve XOR, no matter how long you train it. If your XOR loss refuses to go below a certain plateau, this is the first thing to check.
Learning rate mis-set for the specific optimizer (Adam and SGD often want quite different learning rate scales).

The documentary moment

There will be a specific instant — maybe late at night, maybe after your third attempt at fixing a shape bug — when you run your training loop and watch the loss actually fall, and the XOR network actually gets every example right, using nothing but code you wrote from first principles. That moment is worth the whole project. Let yourself notice it when it happens.

You're ready to move on when…

Your framework can solve XOR reliably, and you can explain, without notes, why a single linear layer fundamentally cannot.

Mini exercises

Train your XOR network and plot the decision boundary it learns (even a crude ASCII or matplotlib grid) — see the nonlinear boundary emerge visually.
Remove the nonlinear activation from your network and confirm it now fails to solve XOR, exactly as theory predicts.

Phase 7: Transformer Components

This is the most ambitious and mathematically rich phase. Budget extra time here, and don't be discouraged if it takes longer than planned — this is genuinely advanced material, and moving slowly here is the difference between "I copied a transformer" and "I understand a transformer."

What you're learning

The architecture that underlies essentially all modern large language models: token embeddings, positional information, self-attention, multi-head attention, layer normalization, and residual connections.

Why it matters

Understanding attention from the ground up — implementing the query/key/value mechanism yourself, watching attention weights sum to 1 via softmax, seeing residual connections stabilize deep networks — turns transformers from a buzzword into a mechanism you genuinely understand. This is the difference that shows in technical interviews and in your ability to read modern ML papers without getting lost.

What to study online

The original "Attention Is All You Need" paper — read it once quickly for the shape of the ideas, then again slowly, focusing specifically on the scaled dot-product attention formula and why the scaling factor (dividing by the square root of the key dimension) exists (it prevents softmax saturation for large dimensions).
A visual walkthrough of self-attention (there are several excellent illustrated blog posts and diagrams online — search for "illustrated transformer" or "illustrated self-attention") to build geometric intuition before diving into the formula.
Layer normalization and residual connections — understand these as stabilization techniques that make deep networks trainable, not as arbitrary architectural decoration.

What to build yourself

An Embedding layer (a lookup table mapping token indices to vectors — this is really just indexed access into a trainable matrix, with the indexing operation needing its own backward rule).
Positional encoding (either the fixed sinusoidal version from the original paper, or a simpler learned positional embedding — the learned version is easier to implement correctly first).
Scaled dot-product self-attention: compute queries, keys, and values via linear projections; compute attention scores as Q @ K^T / sqrt(d_k); apply softmax; multiply by V.
Multi-head attention: run several attention "heads" in parallel on different projected subspaces, then concatenate and project back.
LayerNorm and residual (skip) connections wrapping your attention and feedforward sublayers.
A causal mask for autoregressive generation (so a token at position i can only attend to positions ≤ i, not the future).

Tiny illustrative idea

The conceptual shape of attention, not literal code:

"For each token, compute how much it should 'attend to' every other token, using a similarity score between its query and every key. Normalize those scores with softmax so they form a valid weighting. Use those weights to compute a weighted average over the value vectors — that weighted average becomes the token's new, context-aware representation."

How to test it

Shape tests, relentlessly: attention involves several reshapes and transposes for multi-head splitting, and shape bugs here are extremely common. At every step, know exactly what shape you expect and assert it.
Attention weight sanity check: after softmax, confirm attention weights for each query sum to exactly 1 across the key dimension.
Causal mask sanity check: confirm that after masking, a token's attention weights to future positions are exactly zero, not just small.
Gradient checking one more time, on a small attention block in isolation, before trusting it inside a larger model — attention has enough moving parts that isolating it for testing is worth the extra effort.
Overfit a tiny synthetic sequence task (like copying a short sequence, or predicting the next character in a short repeating pattern) to confirm the whole transformer block can actually learn something before scaling up.

Common mistakes

Forgetting the causal mask, which lets the model "cheat" by looking at future tokens during training — a model trained this way often looks like it's learning beautifully, until you try to generate text with it and discover it never actually learned to predict anything, it just memorized answers using information it shouldn't have had access to.
Getting the transpose wrong when splitting into multiple attention heads, silently mixing information across heads that should stay separate.
Applying LayerNorm at the wrong point relative to the residual connection (pre-norm vs. post-norm architectures behave differently, and mixing conventions inconsistently causes subtle instability).
Numerical instability in softmax over attention scores when the scaling factor is missing or wrong.

You're ready to move on when…

You can build a single transformer block, push a small batch of token sequences through it, confirm shapes and attention-weight sums are correct at every step, confirm the causal mask genuinely blocks future information, and overfit a tiny toy sequence task.

Mini exercises

Implement attention without multi-head splitting first (single head), verify it fully, then generalize to multi-head — resist the urge to implement multi-head attention directly, since debugging the simpler version first will save you real time.
Visualize attention weights (as a heatmap) for a short sequence and manually inspect whether they look sensible before and after training.

Phase 8: The Mini Language Model Demo

What you're learning

How to assemble embeddings, positional information, stacked transformer blocks, and a final projection layer into a complete, trainable, generative language model — and how to actually generate new text from it.

Why it matters

This is the payoff phase. Everything from Phase 2 onward converges here: autodiff, tensors, layers, losses, optimizers, and attention, working together to do something that feels genuinely impressive — a small AI system, built by you, generating text one character at a time.

What to study online

Character-level language modeling as a task — read about why it's a good learning demo (small vocabulary, no tokenizer complexity, fast to train, easy to sanity-check output).
Autoregressive sampling strategies: greedy decoding, temperature sampling, top-k sampling — understand at least greedy and temperature sampling before implementing generation.
Cross-entropy loss specifically as applied to next-token prediction (the standard training objective for language models).

What to build yourself

A small character-level dataset (a public domain text file works well — something like a short story or a chunk of Shakespeare).
A vocabulary built from unique characters, with encode/decode functions between characters and integer indices.
A small GPT-style model: embedding layer + positional encoding + a few stacked transformer blocks + a final linear layer projecting to vocabulary size.
A training loop that samples random chunks of text, predicts the next character at every position, and computes cross-entropy loss.
A generate() function that starts from a seed string and autoregressively samples new characters one at a time, feeding each generated character back in as input for the next step.

How to test it

Training loss should decrease steadily over time — plot it.
Periodically, during training, generate a short sample of text and read it. Early on, it will be gibberish. Partway through, you'll start seeing real word-shapes and plausible letter combinations, even if the content is nonsensical. This progression — noise, to word-shapes, to something reading almost like language — is one of the most satisfying things you'll observe in this entire project.
Compare generation with different temperatures: very low temperature should produce repetitive, "safe" text; higher temperature should produce more varied (and eventually incoherent) text. Seeing this tradeoff directly builds real intuition about sampling.

Common mistakes

Off-by-one errors in constructing input/target pairs for next-character prediction (the target sequence should be the input sequence shifted by one position — get this wrong and the model trains on a nonsensical objective, usually still producing decreasing loss, which makes the bug sneaky).
Forgetting the causal mask here specifically, which lets the model see future characters during training and then produce oddly perfect-looking loss curves that mean nothing, since real generation won't have access to the future.
Training on a dataset too small to generalize, then being surprised that generated text is mostly memorized rather than genuinely learned patterns — a useful phase to observe and understand, not necessarily to "fix."

The documentary moment

This is the second big emotional payoff of the project. The first time you seed your model with a few characters and watch it generate a full paragraph of text that looks like language — even if it's utter nonsense semantically — you are watching a system you built, atom by atom, produce something that feels alive. Sit with that moment. It's earned.

You're ready to move on when…

You have a full training run showing decreasing loss, and generated samples that show clear qualitative improvement from early training to late training — moving from random noise toward recognizable word-shapes and plausible local structure.

Mini exercises

Train on two very different text sources (say, one is repetitive and structured, one is more free-form) and compare how quickly and how well the model picks up each one's patterns.
Implement and compare greedy decoding vs. temperature sampling on the same trained model, and describe in your own words the qualitative difference you observe.

Phase 9: Cleanup, Tests, Documentation, Packaging

What you're learning

How to turn a working research prototype into something that looks and behaves like real, professional software — because a framework nobody else (including future-you) can read or trust isn't really finished.

Why it matters

This phase is often skipped by beginners eager to move on to the next shiny project, but it's exactly the phase that turns "a thing I hacked together" into "a project I can show in an interview, link on my resume, and actually reuse." It also forces you to re-read your own code with fresh eyes, which reliably surfaces bugs you missed the first time.

What to study online

Basic pytest usage, if you haven't used a testing framework before — how to structure test files, write assertions, and run a full test suite with one command.
Good README structure for open-source projects — look at a few well-regarded small ML repos on GitHub for inspiration on structure, not content.
Basic Python packaging (a pyproject.toml or setup.py, so your library can be pip install -e .'d locally).

What to build yourself

A formal test suite covering: gradient checks for every operation, shape tests for every layer, an overfitting test for a tiny dataset, and a small end-to-end test that trains for a few steps and confirms loss decreases.
A README explaining what the project is, how to install it, a quickstart example, and a short explanation of the architecture (this is also a superb exercise in explaining your own system clearly — if you struggle to explain a part, that's a signal you don't fully understand it yet).
Docstrings and type hints across your public API.
A couple of example scripts (XOR training, character-level generation) that a stranger could run to see the library work.

How to test it

Run your full test suite from a clean checkout, ideally in a fresh virtual environment, to confirm there are no hidden dependencies on leftover state from your development process. Ask a friend, if possible, to follow your README's quickstart from scratch and tell you where they got confused — this is the single best way to find documentation gaps.

Common mistakes

Tests that only check "does it run" without checking "is the output correct" (a test that passes even when the math is subtly wrong is worse than no test at all, because it creates false confidence).
A README that describes what you intended to build rather than what actually exists in the code — keep them in sync.

You're ready to call it done when…

A stranger could clone your repository, follow your README, install the package, run the examples, and understand — at least at a high level — how the pieces fit together, without needing to ask you anything.

Part 4: The Debugging Mindset

You will spend a significant fraction of this project's time debugging, not writing new code. That is not a sign you're doing it wrong — it's the nature of building something this foundational. A few principles that will serve you well:

Numerical gradient checking is your best friend. Reach for it constantly, especially any time you add a new operation to your autodiff engine. It is objective, mechanical, and doesn't lie.
Overfit small before you scale up. If your model can't perfectly memorize four data points, it has a bug — it does not need more data, a bigger model, or more training time.
Assert shapes aggressively during development. Most bugs in tensor code are shape bugs. Catching them the instant they occur, rather than three layers downstream, saves enormous time.
Isolate before you integrate. Test each new component (an operation, a layer, an attention block) completely on its own, with its own small test cases, before wiring it into the larger system. Debugging a fully assembled transformer is much harder than debugging one attention block in isolation.
When something is "almost right," be suspicious, not relieved. A model that trains but plateaus at a mediocre loss, or a gradient that's close but not quite matching the numerical check, is very often hiding a real bug — broadcasting summed over the wrong axis, a transpose in the wrong place — rather than being "good enough." Chase these down; they compound.
Reproduce with the smallest possible example. When something breaks in your full training loop, don't debug it there. Strip the problem down to the smallest snippet that reproduces the bug — often just two or three lines — and debug that instead.

Part 5: What Not To Do

Don't copy an existing tiny framework's source code line by line. Reading micrograd, tinygrad, or similar projects for conceptual inspiration is not just fine but encouraged. Transcribing their code without building your own understanding defeats the entire purpose and will leave you unable to debug your own system when it inevitably breaks in a way the original project didn't.
Don't skip gradient checking "because it seems to be working." Loss curves that go down can mask real bugs (a common one: gradients computed with a sign error can still sometimes decrease loss, just less efficiently, hiding the bug for a long time).
Don't jump straight to the transformer phase before your autodiff and tensor engine are solid. Every bug you haven't found in Phase 2 or 3 will resurface, disguised and much harder to diagnose, somewhere deep inside your attention mechanism in Phase 7.
Don't optimize for speed prematurely. Your framework will be slow compared to PyTorch — that's fine and expected. This project is about correctness and understanding, not competing with a production framework backed by compiled CUDA kernels.
Don't work in total isolation. Read forum discussions, ask questions in ML-focused communities when you're stuck, and compare notes with other builders. Building alone doesn't mean learning alone.

Part 6: How to Actually Learn From the Internet While Building This

Since you won't be handed finished code, your primary skill throughout this project is turning scattered online knowledge into working understanding. Here's how to do that well:

Read broadly before you write code, narrowly while you're stuck. Before starting a new phase, skim two or three different explanations of the core concept (a blog post, a paper, a video) to build a rounded mental model. Once you're implementing and hit a specific wall, narrow your search to that exact problem.
Treat existing tiny frameworks as reference material, not templates. Open micrograd or tinygrad after you've made your own honest attempt, specifically to compare your design decisions against theirs — not before, and not to copy.
Papers are denser than blog posts but often clearer once you know what to look for. Read the "Attention Is All You Need" paper's method section only after you already have a rough intuition from a blog post or video — the paper will then click into place much faster.
GitHub issues and forum threads are goldmines for edge cases. If you hit a subtle bug (say, in broadcasting gradients or in causal masking), search for how others have described that exact symptom — someone has almost certainly hit it before you.
Write your own notes as you go. A short paragraph per phase, in your own words, explaining what you built and why, is one of the highest-leverage habits in this entire project. It forces synthesis, and it becomes both your documentation and your future interview prep material.

Epilogue: What You'll Have, and Where To Go Next

By the end of this project, you will have built — with your own hands, your own debugging sessions, your own late-night "oh, THAT'S why" moments — a working deep learning framework that can automatically differentiate arbitrary computations, represent and manipulate tensors with broadcasting, compose neural network layers, optimize them with SGD and Adam, and train a small transformer to generate text.

More importantly, you will have built a kind of understanding that's very hard to get any other way. You will look at a PyTorch stack trace and recognize exactly what kind of shape mismatch or graph-disconnection bug is happening, because you've caused and fixed that exact bug in your own code. You'll read a new architecture paper and be able to mentally sketch how you'd implement its core mechanism, because you've implemented the building blocks it's made from. In interviews, when someone asks "how does backpropagation actually work," you won't recite a definition — you'll explain a system you built.

This project also naturally sets up several strong next steps, whenever you're ready for them:

GPU acceleration: Extend your tensor engine to dispatch operations to a GPU backend, learning the basics of CUDA or a framework like CuPy along the way.
A larger, real transformer: Scale your mini language model up on a real dataset, and explore techniques like learning rate scheduling, gradient clipping, and mixed precision.
Convolutional networks: Apply the same "build it yourself" approach to a CNN, implementing convolution and pooling operations and their gradients from scratch.
Contributing to a real open-source ML framework: With this foundation, reading and contributing to real frameworks like PyTorch or JAX becomes dramatically more approachable — you'll recognize the patterns immediately.

Whatever you build after this, you'll build it with a kind of foundational confidence that's genuinely rare — the confidence of someone who has personally taken deep learning apart, piece by piece, and put it back together again with their own hands.

Now go write your first Scalar class. The journey starts there.

BugVault AI: Never Solve the Same Bug Twice (A VS Code Extension)

Biki Kalita — Sat, 11 Jul 2026 02:59:30 +0000

As developers, we've all been there: staring at an error message, a nagging sense of déjà vu creeping in. "Didn't I fix this last month?" you wonder, before diving back into the same rabbit hole. This cycle of re-solving known issues is a silent productivity killer, draining precious time and mental energy.

Enter BugVault AI, a new open-source VS Code extension designed to put an end to this frustration. BugVault acts as your personal — and team — debugging assistant, intelligently capturing errors, remembering their fixes, and warning you when a familiar foe reappears.

The Problem: Debugging Déjà Vu

Manual documentation is often outdated, Stack Overflow answers are generic, and human memory is fallible. The cost? Hours lost, momentum stalled, and the sheer frustration of tackling the same problem repeatedly. BugVault AI was built to address this core pain point, transforming how developers approach recurring bugs.

How BugVault AI Works: Intelligent Memory for Your Codebase

BugVault AI integrates seamlessly into your VS Code workflow, operating silently in the background to provide a powerful, context-aware debugging experience.

🌟 Key Features for Developers:

Automatic Error Capture: BugVault passively monitors your terminal output, VS Code diagnostics (Problems tab), and build tasks. Errors are fingerprinted and stored without interrupting your flow.
AI-Powered Solutions: When you mark a bug as solved, BugVault leverages VS Code's built-in Language Model API (e.g., GitHub Copilot) to generate a concise, actionable fix description. This solution is highly contextual, drawing from your git diff HEAD and relevant code snippets.
Semantic Memory Matching: Beyond exact text matching, BugVault uses semantic search (powered by Supermemory) to identify similar bugs, even if the error message isn't identical. This means you get relevant solutions faster.
Shared Memory Mode: Collaborate effortlessly. Point BugVault to a central Supermemory instance, and your entire team benefits from fixes discovered by teammates. Knowledge silos become a thing of the past.
Time Saved Counter: Every time BugVault identifies a repeat bug and provides a solution, it logs approximately 15 minutes saved. Watch your productivity grow in the status bar: $(watch) Saved ~X hrs.

A Day in the Life with BugVault AI

Imagine this workflow:

Code Normally: You're focused on writing code. BugVault is quietly capturing potential issues.
Gutter Hint: An error pops up. If BugVault recognizes it, a subtle CodeLens annotation appears above the line: $(bug) Seen Nx · fix: "...".
Confidence Card: A WebView card slides in, showing an AI-powered confidence score and the stored fix. If it's a team fix, it's clearly labeled 👥 Team Fix.
Mark as Solved: You apply the fix. With a quick command, you mark the bug as solved, and BugVault generates and stores the solution for future reference.
Productivity Boost: You move on, knowing that bug won't bother you or your team again.

Getting Started

Prerequisites:

VS Code 1.90+
Node.js & npm
Supermemory Local (for semantic search, can run locally or connect to a shared instance)
AI Access (e.g., GitHub Copilot extension)

Installation (from source):

git clone https://github.com/biki-dev/BugVault.git
cd BugVault
npm install
npm run compile

Open the project in VS Code and press F5 to launch the Extension Development Host.

Configuration:

BugVault is highly configurable. Open VS Code Settings (Ctrl+, / Cmd+,) and search for BugVault to adjust settings like similarityThreshold or enable sharedMemory.enabled.

Contribute to BugVault AI

BugVault AI is open-source and thrives on community contributions. Whether you want to report a bug, suggest a feature, or dive into the codebase, we welcome your involvement.

Repository: https://github.com/biki-dev/BugVault
Issues: https://github.com/biki-dev/BugVault/issues

We've designed the architecture to be modular and easy to understand. Check out the ARCHITECTURE.md (or the Architecture Overview in the README) for a deeper dive into its components.

Join the BugVault Community

Stop solving the same bugs twice. Install BugVault AI today and reclaim your development time. We're excited to see how you use it and contribute to its evolution!

IssueWatch — Never Miss a GitHub Issue That Matters to You

Biki Kalita — Fri, 05 Jun 2026 09:57:39 +0000

This is a submission for the GitHub Finish-Up-A-Thon Challenge

What I Built

IssueWatch is a mobile app built with React Native and Expo that turns the chaos of open source issue tracking into a calm, intelligent feed delivered straight to your phone.

The idea came from a real frustration. I wanted to contribute to open source, but I had no reliable way to know when a good issue dropped on a repository I cared about. By the time I found it, ten people had already commented. I was always late.

So I built the tool I wished existed.

You subscribe to GitHub repositories, pick the labels you care about — good first issue, bug, help wanted — add optional keyword filters, and the app does the rest. The moment a matching issue opens, your phone gets a push notification. Not a generic one — an AI-triaged one that already tells you the severity, the type, and a one-line summary of what the issue is actually about before you even tap.

IssueWatch is for contributors who want to be first. For maintainers who want to stay on top of incoming bugs. For anyone who lives in open source and needs a smarter way to filter the noise.

Demo

Repo

Biki-dev / Opensource-issue-notify

Open-source GitHub issue tracker with real-time notifications, issue monitoring, and mobile alerts.

IssueWatch

About

IssueWatch is a powerful, open-source mobile application and backend system designed to help developers and teams stay on top of their GitHub issues. It provides real-time push notifications for new issues, comments, and activity on tracked repositories, ensuring you never miss a critical update. With features like personal GitHub token integration for enhanced rate limits and private repository access, and a tiered scheduler for efficient monitoring, IssueWatch streamlines your workflow and keeps you informed.

✨ Features

Real-time Notifications: Get instant push notifications for new issues and comments on your tracked GitHub repositories.
Personal GitHub Token Integration: Connect your personal GitHub token to unlock higher API rate limits and track private repositories.
Tiered Scheduler: An intelligent backend scheduler optimizes GitHub API calls based on your token status, ensuring efficient and timely updates.
Issue Tracking & Detail View: Follow specific issues, view their details, and monitor comment activity directly within…

View on GitHub

The Comeback Story

IssueWatch started as a backend-heavy side project — a Node.js server that could poll GitHub and send push notifications. The core scheduler worked. The database models were solid. But the mobile app was incomplete, rough around the edges, and nowhere near ready to show anyone.

Here is what was missing or broken when I picked it back up:

What existed:

Basic Express server with auth routes
MongoDB models for users, repositories, subscriptions, notifications
A working scheduler that polled GitHub on a timer
A bare-bones React Native app with navigation set up

What I finished:

Mobile App — rebuilt from rough screens to a polished product

Designed and built the entire UI system from scratch — custom tab bar with animated indicators, skeleton loaders, haptic feedback, smooth moti animations throughout
Built the Dashboard screen with live stats, an activity chart that reads from the notification database instead of hammering the GitHub API, and expandable issue cards
Built the Subscriptions grid with mute, hide, and edit-labels inline actions
Built the Inbox with swipe-to-dismiss, severity filters, repository filters, and the Follow feature
Built the Following tab with full comment thread rendering, markdown support, maintainer badges, and new comment indicators
Built the Repo Detail screen showing full GitHub metadata, README rendered in markdown, stats, and a one-tap refresh
Built the GitHub Token Settings screen with live rate limit status
Fixed the onboarding flow and authentication edge cases
Added web platform support via platform-specific chart components

Backend — hardened and extended

Added AI triage using OpenRouter and NVIDIA as providers — every new issue is classified by severity, type, effort, and summarized in one sentence
Built a tiered scheduler — default users check every 60 minutes, personal token users every 30, premium every 15
Built the comment poller service — tracks followed issues, detects new comments, and sends targeted push notifications
Added the issue tracker routes and MongoDB model for the Following feature
Fixed duplicate notification bugs with atomic upserts and compound unique indexes
Added AES-256-GCM encryption for stored GitHub tokens
Added metadata fetching — stars, forks, watchers, open issues, PRs, license, topics, README — fetched at subscribe time and cached
Built repository detail and refresh endpoints
Added orphaned subscription cleanup on startup and a daily cron cleanup job
Added push receipt verification running every 15 minutes via Expo's receipt API

The gap between where this project was and where it is now is the entire product. The foundation was there. Everything you can see and use was built in this push to finish it.

My Experience with GitHub Copilot

Copilot was present throughout this build in ways that genuinely changed how fast I could move.

The place it helped most was in the repetitive but critical work — the kind of code that has to be correct but follows clear patterns. Writing MongoDB aggregation pipelines, setting up axios interceptors, structuring the tiered cron scheduler, building out the push notification chunking logic with Expo's SDK — Copilot would get me 80% of the way there on the first suggestion, and I would refine the remaining 20% to fit the exact requirements of the project.

For the React Native UI, Copilot was surprisingly good at completing component patterns once it understood the design system I was building. After writing two or three card components by hand, suggestions for the next one were almost exactly right — matching the shadow styles, the moti animation patterns, and the NativeWind class conventions I had established.

Where I had to stay sharp was in the logic that required deep context about the whole system — like the interaction between the scheduler, the notification deduplication, and the push receipt verification. Copilot would suggest plausible-looking code that missed a subtle constraint. Knowing when to accept, when to modify, and when to ignore entirely was the real skill. Copilot made me faster. It did not replace thinking.

The honest summary: this project would have taken significantly longer without it. Not because I could not write the code — but because the volume of work required to go from an incomplete side project to a finished, shippable product is enormous. Copilot compressed the time on the mechanical parts so I could spend my energy on the parts that actually required judgment.

Built With

Built with React Native, Expo, Node.js, Express, MongoDB, GitHub API, Expo Push Notifications, and OpenRouter AI.

Every notification. Every triage. Every comment update. Delivered to your pocket.

Building a Reusable React Knowledge Graph Component: OKVE v0.3.0

Biki Kalita — Tue, 07 Apr 2026 13:24:32 +0000

I wanted a graph component that felt native in React apps, not a one-off visualization.
So I built OKVE: a TypeScript-first knowledge graph component powered by D3.

In v0.3.0, I focused on product UX:

programmatic camera focus
built-in node search
group filtering
keyboard deselect
stats overlay
PNG export API
In this post I will break down the component API, implementation tradeoffs, and what I learned shipping it in a monorepo with a live demo.

Repo: https://github.com/Biki-dev/OKVE
npm: https://www.npmjs.com/package/@biki-dev/okve

I am looking for product-oriented feedback:

What API feels awkward?
What is missing for production usage?
What should be next: minimap, clustering, virtualization, or something else?

I built an open-source CLI tool that turns your developer portfolio into a terminal experience

Biki Kalita — Fri, 06 Mar 2026 14:03:11 +0000

Ever wanted your developer portfolio to feel like a real hacker terminal?

I just open-sourced own-term — a framework that lets any developer showcase their portfolio directly in the terminal, runnable with a single npx command.

npx own-term

What it does

🎨 6 built-in themes — dark, light, hacker, neo, dracula, nordic
💻 5 prompt styles — git, time, dashboard, minimal, cyberpunk
📦 Interactive project browser — pick a project, see details, open in browser
⚡ Skills with progress bars — or simple pill-style tags
🔌 Plugin system — extend with custom commands
✅ Zero config to start — works out of the box

How to create your own

Create a termfolio.config.ts:

export default {
  name: "Your Name",
  title: "Full-Stack Developer",
  theme: "dark",
  promptStyle: "git",
  links: {
    github: "https://github.com/yourusername",
    email: "you@example.com"
  },
  projects: [
    {
      name: "My Project",
      desc: "Something awesome",
      repo: "https://github.com/you/project",
      tags: ["react", "node"],
      status: "active"
    }
  ],
  skills: {
    languages: [
      { name: "TypeScript", level: 90 },
      { name: "Python", level: 75 }
    ],
    tools: ["Git", "Docker", "VS Code"]
  }
};

Then run it:

npx own-term --config=./termfolio.config.ts

Why I built this

As developers we spend so much time building impressive things — but our portfolios are just another website. I wanted something that felt native to what we actually do: work in terminals.

What's next

Custom ASCII logo rendering
Live theme switching (no restart needed)
Tab completion & command history

Contribute

The project is open source and actively looking for contributors.
Check the issues tab — several are tagged good first issue.

GitHub: https://github.com/Biki-dev/own-term

Drop a comment if you build your own — I'd love to see it! 🚀

DEV Community: Biki Kalita

The Auto-DJ Case Files: Building a Self-Learning Music Brain From Your YouTube Habit

A 3–5 month field guide for turning a browsing habit into an AI orchestra

Prologue: The Case That Started With a Skip Button

Chapter 1: The Case of the Repeating Refrain

Following one song's journey

The architecture, explained like you're new here

Why these particular tools (and not the flashier alternatives)

The living organism

Chapter 2: The Learning Roadmap — Five Months, Six Phases

Phase 0: Recon (3–5 days, before Week 1 really starts)

Phase 1: The Witness — Event Tracking (Weeks 1–3)

Phase 2: The Classifier -- Is This Even Music? (Weeks 4-6)

Phase 3: The Forensic Lab -- Audio & Metadata Enrichment (Weeks 7-10)

Phase 4: The Behavioral Profiler -- Learning What You Actually Like (Weeks 11-14)

Phase 5: The Case Board -- Clustering & Playlist Generation (Weeks 15-18)

Phase 6: The Broadcast -- Web Player & Full Integration (Weeks 19-22)

Chapter 3: The Brain of the System — Meet Your Neural Concierge

How to search, read docs, and debug like you've done this before

Trade-off thinking, one more time with feeling

NLP for mood: zero-shot first, fine-tuning as a deliberate upgrade

The math of implicit feedback, one level deeper

Neural collaborative filtering, sketched a little further

Chapter 4: The Data Story — Where Evidence Lives

The schema, roughly

Bootstrapping with Google Takeout

Simulating data for development

Privacy and local-first design

Chapter 5: The Galaxy of Songs — Playlist Generation Magic

Embeddings, explained like you're teaching a friend at a bar

Clustering and automatic labeling, tied together

Filtering for activity-based playlists

A future reinforcement learning layer

Chapter 6: The Web Dev Spine — Extension, Backend, Frontend, Tied Together

The extension, structurally

REST API design that won't fight you later

Streaming audio, and the ethical note this section owes you

Frontend architecture, briefly

Tying the vector database to the frontend

Chapter 7: The Detective's Logbook — Milestones Across the Whole Case

Chapter 8: Beyond the Guide — Where the Case Reopens

Chapter 9: The Final Note

ChronoWeave: The Documentary

From Framework User to Framework Understander — A 4-Month Journey Through Gradient Descent, Transformers, and the Physics of History

How to Use This Guide

CHAPTER 0: The Grand Vision — Seeing the Complete Picture

The Problem: Why Historical Timelines Are Broken

The Solution: ChronoWeave's Architecture

Data Flow: What Happens When a User Clicks "Generate"

Technology Stack: Why Each Piece Was Chosen

The Journey Ahead

CHAPTER 1: The Foundation — Building Your Neural Engine

Week 1: Environment and the Shape of a Tensor

The Concept

The Code Challenge

The Aha Moment

The Extension

Resources

Week 2: Manual Forward and Backward — No nn.Module

The Concept

The Code Challenge

The Aha Moment

The Extension

Resources

Week 3: Hand-Rolled SGD, Momentum, RMSprop, and Adam

The Concept

The Code Challenge

The Aha Moment

The Extension

Resources

Week 4: A Transformer From Scratch (Tiny, But Real)

The Concept

The Code Challenge

The Aha Moment

The Extension

Resources

Phase 1 Milestone

CHAPTER 2: The Understanding — Teaching Your Engine to Read History

Week 5: Fine-Tuning BERT for Named Entity Recognition

The Concept

Week 2: Manual Forward and Backward — No `nn.Module`