Biki Kalita

Posted on Jul 26

The Auto-DJ Case Files: Building a Self-Learning Music Brain From Your YouTube Habit

#programming #ai #python #machinelearning

A 3–5 month field guide for turning a browsing habit into an AI orchestra

Prologue: The Case That Started With a Skip Button

Every great investigation starts with something small and irritating. Yours starts on a Tuesday night. You're three hours deep into a YouTube rabbit hole — lo-fi beats, a Vietnamese city-pop deep cut, a live session you didn't mean to watch twice — and you think, someone should be paying attention to this.

That someone is about to be you. Not as a listener. As a detective.

Here's the case file: somewhere in your YouTube history is a pattern. A shape. You skip certain songs after four seconds. You replay others three times in a row without noticing. You always end up on synthwave at 11pm and never before 6pm. Right now, that pattern is invisible — smeared across a history page that YouTube barely lets you search, let alone reason about. Your job over the next few months is to build the instrument that makes it visible: a system that watches what you actually do, figures out what it means, and hands you back playlists that feel like they were made by someone who knows you. Because they were. That someone is a machine you built, trained on a dataset of one: you.

This won't be a weekend project. It's closer to building a small orchestra — a browser extension as your field agent, a backend as your evidence locker, a couple of ML models as your forensic lab, and a frontend as the stage where the final performance happens. Every piece depends on every other piece. That's what makes it hard, and it's also what makes it worth doing slowly, in public, with your own hands on every layer.

Let's start where every good documentary starts: by following one song from the crime scene to the credits.

Chapter 1: The Case of the Repeating Refrain

Following one song's journey

It's 9:47pm. You click a YouTube video: "Kiss of Life – Midas Touch (Live Performance)." You watch 41 seconds, skip to the chorus, watch another 90 seconds, then close the tab. Here's what needs to happen for that unremarkable moment to eventually shape a playlist called "Monday Chill":

The Witness (Browser Extension). A content script sitting quietly inside the YouTube tab notices the video element load. It reads the video ID, title, channel name, and duration from the page. It watches the <video> element's timeupdate, pause, seeked, and ended events like a stenographer taking notes: play at 0:00, seek to 1:32, pause at 3:04, total watch time 41s + 90s = 131s out of 210s (62%).
The Tip-Off (Event API). Every so often — batched, not on every tick, because nobody wants to be that extension hammering a server — the extension bundles these observations into a small JSON payload and POSTs it to your backend: {video_id, title, channel, watch_segments, timestamp, session_id}.
The Evidence Locker (Database). Your FastAPI backend receives the tip-off, validates it, and writes a raw event row into PostgreSQL. Nothing clever happens yet — this is just chain-of-custody. You never want to lose the raw signal, because every clever thing you do later will be derived from it, and derived things need to be recomputable.
The Lab Work (ML Pipelines, offline). On a schedule — say, once a night — a pipeline wakes up, looks for new raw events, and does the actual detective work:
- Is this even a music video? (classification)
- What song is this, really? (metadata resolution via title parsing + external lookups)
- What does it sound/feel like? (audio embedding + mood tagging)
- How does this fit with everything else you've watched? (behavioral signal: implicit rating from watch %, skip pattern, replay count, time of day)
The Case Board (Vector store + relational tables). The song's audio embedding gets written to a vector column (via pgvector), its metadata to a normal relational table, and its behavioral score gets updated incrementally — like a detective updating a suspect's file every time new evidence comes in.
The Reveal (Playlist Generator). Periodically, a clustering job looks at the whole case board — all your songs, their embeddings, their mood tags, their time-of-day patterns — and groups them. A cluster of moody, low-energy, evening-heavy tracks with "chill," "lofi," or "acoustic" tags gets surfaced and, because most of its listens happened on Mondays, gets named "Monday Chill."
The Broadcast (Frontend + Player). Your React app queries the backend, gets back a list of playlists with their tracks, and renders them. When you hit play, it doesn't awkwardly embed a YouTube iframe — it either streams audio you've legitimately extracted for personal use, or falls back to a controlled YouTube player, and it logs that playback too, feeding the loop again.

That's the whole organism, end to end. Six weeks from now this will feel obvious. Right now, notice the shape: observe → transmit → store → understand → organize → present, with a feedback loop closing the circle back to step 1. Every phase of this guide builds one of these organs.

The architecture, explained like you're new here

[Browser: YouTube tab]
        |
   content script  (watches video element)
        |
   background script (batches + sends events)
        |
        v
   Event Ingestion API  (FastAPI, /events endpoint)
        |
        v
   Raw Events Table  (PostgreSQL)
        |
   +----+----------------------------+
   |                                 |
   v                                 v
Metadata & Classification      Audio & Behavior Pipelines
(is it music? what song?)      (embeddings, mood, implicit score)
   |                                 |
   +----------------+----------------+
                    v
        Enriched Track Store
   (PostgreSQL relational + pgvector embeddings)
                    |
                    v
          Clustering / Playlist Engine
        (HDBSCAN, labeling, scheduling)
                    |
                    v
             Playlist API (FastAPI)
                    |
                    v
           React Frontend + Player
     (renders playlists, plays audio, logs playback)
                    |
                    +--> feeds back into Raw Events

Plain English: a browser extension is the only part of this system that can see raw YouTube behavior, so it's your sensor. Everything after that is you progressively distilling raw noise into structured meaning — first "what is this," then "what does it sound like," then "what does it mean that you watched it," then "what group of songs does this belong to." A pipeline is just a name for "a sequence of these distillation steps that runs on a schedule instead of live," which matters because audio analysis and ML inference are too slow to run inline while you're browsing.

Why these particular tools (and not the flashier alternatives)

FastAPI, not Django or raw Flask. You'll be exposing a mix of simple CRUD endpoints and ML inference endpoints. FastAPI's automatic request validation via Pydantic means fewer "why did my extension send a malformed payload and silently corrupt my data" nights, and its async support matters once you're streaming audio or calling external APIs (metadata lookups) without blocking.
PostgreSQL + pgvector, not a separate vector database. You could stand up Pinecone or Weaviate. For a personal-scale project (thousands, maybe tens of thousands of tracks — not billions), that's solving a problem you don't have yet, at the cost of one more service to run, secure, and back up. pgvector lets you keep relational metadata (title, channel, mood tags) and embeddings in the same database, so a query like "find songs similar to this one, but only ones tagged 'study' and watched after 9pm" is one SQL query with a vector distance operator, not a join across two systems.
A CNN (or a pretrained audio model) for audio classification, not a giant end-to-end transformer. Audio classification at the "is this speech, music, or noise" and "what does this sound like" level has been solved well by relatively small convolutional architectures operating on spectrograms (think YAMNet, VGGish). You don't need GPT-scale compute to tell that a track is high-energy electronic versus a mellow acoustic set. This is a deliberate choice to keep your compute budget human-sized — you'll likely run this on a laptop or a modest cloud instance, not a GPU cluster.
Transformers for tagging, not for audio itself. Where transformers genuinely earn their complexity is text — parsing messy YouTube titles ("Artist - Song (Official Music Video) [4K]"), zero-shot classifying mood from lyrics or descriptions, and understanding channel/description context. Hugging Face's zero-shot classification pipelines let you throw a candidate label set ("chill," "energetic," "sad," "study," "workout") at a piece of text and get calibrated-ish scores back without training anything — perfect for a first pass, with fine-tuning as a stretch goal.
A browser extension, not scraping Google Takeout alone. Takeout gives you history, but it's after the fact, coarse, and doesn't include watch percentage, skips, or replays — exactly the implicit-feedback signal your recommendation logic needs. The extension is how you get behavioral data, not just occurrence data. Takeout is still useful — as bootstrap data and backfill, which Chapter 4 covers.

Notice the pattern in every choice above: pick the tool that matches the actual scale and shape of your problem, not the tool with the best conference talk. You're building a personal system for one very well-instrumented user (you), not a production service for millions. Keep saying that sentence to yourself whenever you're tempted to add Kubernetes.

The living organism

Here's the mental model to carry through the rest of this guide: this system has a nervous system (the extension + event API, always sensing), a subconscious (the nightly pipelines, working while you sleep, turning raw sensation into structured understanding), and a conscious voice (the frontend, presenting a curated, explainable result). If you break the nervous system, nothing gets sensed. If you break the subconscious, sensations pile up unprocessed. If you break the conscious voice, all that understanding has nowhere to be expressed. You'll spend the next several months building each of these organs, and — this is the important part — you'll build them so that each one can be tested and demoed on its own, before they're wired together. That's the difference between a project that has a working prototype at every milestone and a project that only "works" the week before a deadline, if you're lucky.

Chapter 2: The Learning Roadmap — Five Months, Six Phases

You know basic Python and JavaScript. You don't yet know how these pieces talk to each other, or how to think about audio and behavior as data. So the roadmap is designed to stack skills like sediment layers — each phase leaves you with a thin, ugly, but genuinely working slice of the system, and each phase's slice becomes the foundation the next phase builds on. You will never be staring at a blank canvas with six unconnected concepts in your head. You will always be extending something that already runs.

Treat each phase as 2–4 weeks depending on how much time you have per week. If you have 10+ hours/week, aim for the shorter end; if you're doing this around a full-time job, take the longer end. Five months is the "comfortable, learn deeply" pace. Three months is achievable if you already move fast and cut a few of the "stretch" items.

Phase 0: Recon (3–5 days, before Week 1 really starts)

Goal: Set up the skeleton of every future piece, even if each piece does almost nothing.

Create the repo structure: extension/, backend/, ml/, frontend/, infra/.
Get a FastAPI "hello world" running locally with one route, GET /ping.
Get a bare Chrome extension loaded in developer mode that does nothing but log "extension loaded" to the console on any youtube.com page.
Spin up a local PostgreSQL instance (Docker is your friend here — docker run postgres).

Learning pit stop:

15 min: FastAPI's official "First Steps" tutorial — just enough to run a server.
20 min: Chrome's Extension "Getting Started" tutorial — focus on the manifest and content scripts, skip the rest for now.

You know you're ready when: you can open a YouTube tab, see "extension loaded" in the console, and hit http://localhost:8000/ping in your browser and get {"status": "ok"}. That's it. Two independent heartbeats.

Phase 1: The Witness — Event Tracking (Weeks 1–3)

Goal: A Chrome extension that logs real YouTube watch events, and a backend endpoint that stores them.

Skills gained: DOM event listeners, Chrome extension messaging (content script <-> background script), REST API design basics, relational schema design for event logs.

What you'll build:

A content script that attaches listeners to the YouTube <video> element (play, pause, seeked, ended, timeupdate sampled every few seconds — not every frame, that's overkill and will flood your database).
A background script that batches events (e.g., flush every 30 seconds or every 10 events) and sends them via fetch to your backend.
A POST /events FastAPI endpoint with a Pydantic model validating the payload, writing rows into a raw_events table.
A minimal schema: raw_events(id, video_id, title, channel, event_type, position_seconds, video_duration_seconds, session_id, created_at).

Architecture decision point: Do you send every micro-event (every timeupdate tick) to the server, or do you compute a per-video summary (total watch time, max position reached, number of seeks) in the extension and send one summary per video? Sending raw ticks gives you more analytical flexibility later (you could reconstruct exact skip behavior) at the cost of way more data and server load. Summarizing in-extension is simpler and cheaper but throws away granularity. Recommendation for a learning project: send raw ticks for now — you're optimizing for "what can I learn from this," not production efficiency, and you can always summarize later. You can throttle it (sample every 5 seconds) to keep volume sane.

Learning pit stops:

20 min video: search "Chrome extension message passing background content script" — get comfortable with chrome.runtime.sendMessage.
Skim MDN's HTMLMediaElement events page — you need timeupdate, pause, seeked, ended, play.
10 min: Pydantic's "Models" quickstart — you'll lean on this for every endpoint from here forward.

Skeleton to work from (not a finished file):

// content_script.js -- the witness's notebook
function attachListeners(videoEl, videoId) {
  let lastPosition = 0;

  videoEl.addEventListener('timeupdate', () => {
    // TODO: throttle this -- don't fire on every tick
    // TODO: send {event_type: 'progress', position: videoEl.currentTime}
  });

  videoEl.addEventListener('seeked', () => {
    // TODO: compare videoEl.currentTime to lastPosition
    // to detect skip-forward vs rewind
  });

  videoEl.addEventListener('pause', () => {
    // TODO: send pause event with position
  });
}

// TODO: figure out how to get videoId from the page URL
// TODO: figure out how to detect when YouTube's SPA navigation
//       changes video without a full page reload (hint: MutationObserver
//       on the title element, or listen for yt-navigate-finish)

# backend/main.py -- the evidence intake desk
from fastapi import FastAPI
from pydantic import BaseModel

app = FastAPI()

class RawEvent(BaseModel):
    video_id: str
    title: str
    channel: str | None = None
    event_type: str  # 'play' | 'pause' | 'seeked' | 'ended' | 'progress'
    position_seconds: float
    video_duration_seconds: float | None = None
    session_id: str

@app.post("/events")
def ingest_event(event: RawEvent):
    # TODO: open a DB connection (consider SQLAlchemy or raw psycopg)
    # TODO: INSERT into raw_events
    # TODO: return a lightweight ack, don't do heavy work here --
    #       this endpoint must stay fast, it's on the hot path
    ...

Milestone / "you know you're ready when": you watch three different YouTube videos for real, then query your database and see a believable, timestamped trail of what you actually did -- including at least one skip you remember making. Take a screenshot. This is the first time the system has seen you.

Phase 2: The Classifier -- Is This Even Music? (Weeks 4-6)

Goal: Given a raw event's title/channel/duration, decide whether it's a music video at all (versus a podcast, tutorial, vlog) -- and if it is, extract a clean artist/song guess.

Skills gained: Text classification basics, regex-as-first-resort vs ML-as-fallback thinking, working with the YouTube Data API, zero-shot classification with Hugging Face.

What you'll build:

A nightly (or on-demand) script that reads unprocessed rows from raw_events, groups by video_id, and for each unique video, runs a classification step.
First pass -- heuristics, not ML. Channel name contains "Official," "VEVO," "Music"? Title matches patterns like Artist - Title (Official ...)? These heuristics alone will correctly classify a surprising majority of real YouTube music content. Write these first. Resist the urge to reach for a model before you've squeezed out what regex and metadata can do -- this is a core "how to think like an engineer" lesson: cheap and boring beats fancy and unnecessary, every time it's available.
Second pass -- zero-shot text classification for the ambiguous remainder, using Hugging Face's zero-shot-classification pipeline with candidate labels like ["music video", "podcast", "tutorial", "vlog", "gaming"].
A tracks table that stores the resolved artist/title guess, separate from the raw event log -- this is your first "enriched" table, the first sign of the subconscious turning sensation into understanding.

Architecture decision point: Regex-first-then-ML, or ML-first-then-regex-cleanup? Going regex-first means your model only has to handle genuinely ambiguous cases, which is both faster (you skip inference on 70%+ of videos) and more accurate (you're not asking a general-purpose model to outperform a rule you could write in one line for "channel name ends in VEVO"). This is a recurring theme: use ML for the residual, not the whole problem.

Learning pit stop:

15 min: Hugging Face's zero-shot classification pipeline docs -- run their example locally before writing your own.
Skim: the YouTube Data API "Videos: list" reference -- you'll want snippet.description and snippet.tags for extra signal, and eventually contentDetails.duration.

Skeleton:

# ml/classify.py -- the detective's first read of the case file
MUSIC_CHANNEL_HINTS = ["vevo", "official", "records", "music"]

def heuristic_is_music(title: str, channel: str) -> bool | None:
    # Return True/False if confident, None if unsure (defer to ML)
    channel_lower = channel.lower()
    if any(hint in channel_lower for hint in MUSIC_CHANNEL_HINTS):
        return True
    # TODO: add a regex for "Artist - Title" pattern
    # TODO: add a denylist for obvious non-music (podcast, vlog, tutorial keywords)
    return None

def ml_classify(title: str, description: str) -> str:
    # TODO: load huggingface zero-shot pipeline once, reuse across calls
    #       (loading it per-call will make this painfully slow)
    # candidate_labels = ["music video", "podcast", "tutorial", "vlog", "other"]
    ...

def extract_artist_title(title: str) -> tuple[str | None, str | None]:
    # TODO: try a regex like r"^(.*?)\s*-\s*(.*?)(\(|\[|$)"
    # TODO: fall back to None, None if it doesn't match -- don't force a bad guess
    ...

Milestone: run the classifier over a week's worth of real watch history. Print a summary: "Classified 42 videos: 31 music, 11 other." Manually eyeball 10 of them -- if your accuracy feels roughly 80%+, move on; ML classifiers don't need to be perfect, because Phase 5's clustering step will naturally down-weight noise. Perfectionism here is a trap -- resist polishing this past "good enough to build on."

Phase 3: The Forensic Lab -- Audio & Metadata Enrichment (Weeks 7-10)

Goal: For every confirmed music video, extract audio and metadata rich enough to power mood tagging and similarity search.

Skills gained: Working with yt-dlp, basic digital signal processing (spectrograms, mel scales), running a pretrained audio embedding model, external metadata APIs (MusicBrainz or similar), and your first taste of "architecture decision with a real trade-off."

What you'll build:

A pipeline step that, given a video_id, downloads (or streams) audio using yt-dlp, for local processing only -- not redistribution. This is the moment to have your privacy/ethics conversation with yourself; more in Chapter 4.
A function that converts raw audio into a mel spectrogram -- the standard "image-like" representation that lets you use image-flavored models (CNNs) on sound.
Architecture decision point, spelled out for you: you could (a) use a pretrained audio embedding model like YAMNet or OpenL3 to get a general-purpose embedding vector per track with zero training, or (b) train a small CNN yourself on your own labeled data (mood-tagged tracks) to get embeddings tuned to your taste categories. (a) is faster, works immediately, and is what you should ship first. (b) is the better learning experience and gives you a genuinely custom system -- a great Phase 3.5 stretch goal once (a) is working end to end. Don't skip straight to (b); you'll spend three weeks on a training loop for a payoff you could've had in three days.
Store the resulting embedding vector in a pgvector column on your tracks table.
A metadata enrichment step: look up the resolved artist/title against MusicBrainz's API (free, no key needed for light use) to pull canonical artist name, genre tags if available, and release year.

Learning pit stops:

20 min video: search "mel spectrogram explained simply" -- you need the intuition (frequency content over time, perceptually weighted), not the full DSP math.
Read the yt-dlp README's "Usage and Options" section, specifically audio extraction flags (-x --audio-format).
Skim: YAMNet's model page on TensorFlow Hub -- its usage example is short and you can adapt it directly.

Skeleton:

# ml/audio_features.py -- the forensic lab bench
import yt_dlp
import librosa
import numpy as np

def download_audio(video_id: str, out_dir: str) -> str:
    # TODO: configure yt_dlp options for audio-only extraction
    # ydl_opts = {'format': 'bestaudio/best', 'outtmpl': f'{out_dir}/%(id)s.%(ext)s', ...}
    # NOTE: this is for local, personal analysis only -- see Chapter 4
    ...

def compute_mel_spectrogram(audio_path: str) -> np.ndarray:
    # TODO: y, sr = librosa.load(audio_path)
    # TODO: librosa.feature.melspectrogram(y=y, sr=sr, n_mels=128)
    # TODO: convert to log scale (librosa.power_to_db) -- raw mel values
    #       span orders of magnitude, log scale is what models expect
    ...

def get_embedding(audio_path: str) -> np.ndarray:
    # TODO: load a pretrained model (YAMNet via tensorflow_hub, or
    #       openl3.get_audio_embedding) and run inference
    # TODO: pool frame-level embeddings into a single fixed-length vector
    #       (mean pooling is a fine first choice)
    ...

Architecture decision point (data storage): store raw audio permanently, or delete it after extracting the embedding? Storing it burns disk and raises copyright/privacy questions for no real benefit -- you only need the embedding, not the waveform, for everything downstream. Recommendation: extract-then-delete, keep only the derived vector. Write this down as a design principle now; it'll save you a difficult conversation with yourself in Month 4 when your disk is full of MP3s you don't need.

Milestone: pick five songs you know well, run them through the pipeline, and eyeball their embeddings' pairwise cosine similarity. Two songs by the same artist in the same genre should be closer to each other than to a wildly different genre. If your embeddings pass this smell test, you have a forensic lab that actually works. Screenshot the similarity matrix -- this is the moment the system starts to have taste.

Phase 4: The Behavioral Profiler -- Learning What You Actually Like (Weeks 11-14)

Goal: Turn raw watch behavior into an implicit "how much did you like this" score, and build your first learned model of taste.

Skills gained: Implicit feedback theory, feature engineering from event logs, neural collaborative filtering, thinking in terms of "signal vs noise" in behavioral data.

What you'll build:

A feature engineering step that, per track, computes: percent watched, replay count within a session, skip-within-first-10-seconds flag, time-of-day distribution, day-of-week distribution.
An implicit score -- not a 1-5 star rating (you never gave one), but a derived number. A simple, honest starting formula:

implicit_score = w1 * completion_ratio
                + w2 * log(1 + replay_count)
                - w3 * early_skip_penalty

where completion_ratio = watched_seconds / video_duration, capped at 1.0, early_skip_penalty is 1 if you skipped within the first 10 seconds and never returned, else 0, and w1, w2, w3 are weights you pick by hand at first (try w1=1.0, w2=0.5, w3=0.8) and refine once you have enough data to eyeball whether the resulting rankings feel right. This is implicit feedback in a nutshell: you're inferring preference from behavior, not asking for it directly, and every implicit signal is a proxy, not ground truth -- a song you love but skip because you're not in the mood right now looks identical, in raw data, to a song you dislike. Time-of-day and mood context (Phase 5) is what starts to disambiguate the two.

The math of implicit feedback, briefly: classic explicit recommender systems (star ratings) treat missing data as "unknown." Implicit feedback systems (like this one) treat everything as a signal -- even the absence of a play is information (you had the chance to watch it and didn't). The seminal framing here, worth reading about, is "confidence-weighted preference": you're not just modeling whether you like something, you're modeling how confident the system should be in that preference, where confidence scales with how many times you've interacted with a track.
A first neural collaborative filtering sketch, once you have a few hundred (track, implicit_score, context) rows. Neural CF replaces the classic matrix-factorization approach (learn a vector per user, a vector per item, dot-product them to predict preference) with small embedding layers you can extend to include context features (time of day, mood tag) alongside the learned latent vectors.

Learning pit stops:

25 min: search "implicit feedback recommender systems explained" -- get comfortable with the idea that "no interaction" is still data.
Skim: Keras's Embedding layer docs -- this is the core building block of neural CF.
Optional deeper dive (30 min): the original "Collaborative Filtering for Implicit Feedback Datasets" paper's abstract and intro -- you don't need the full math, just the framing.

Skeleton (Keras):

# ml/behavior_model.py -- teaching the system your taste, one skip at a time
import tensorflow as tf
from tensorflow.keras import layers, Model

def build_neural_cf(n_tracks: int, embedding_dim: int = 32) -> Model:
    track_input = layers.Input(shape=(1,), name="track_id")
    context_input = layers.Input(shape=(4,), name="context_features")  # e.g. hour_of_day, day_of_week, recent_genre_onehot...

    track_embedding = layers.Embedding(n_tracks, embedding_dim)(track_input)
    track_vec = layers.Flatten()(track_embedding)

    # TODO: concatenate track_vec with context_input
    # TODO: pass through 1-2 Dense layers with ReLU
    # TODO: final Dense(1, activation='sigmoid') to predict a
    #       normalized implicit_score in [0, 1]

    ...
    return Model(inputs=[track_input, context_input], outputs=..., name="neural_cf")

def compute_implicit_score(row) -> float:
    completion_ratio = min(row["watched_seconds"] / row["duration_seconds"], 1.0)
    replay_bonus = 0.5 * (row["replay_count"] ** 0.5)  # diminishing returns
    early_skip_penalty = 0.8 if row["early_skip"] else 0.0
    return max(0.0, completion_ratio + replay_bonus - early_skip_penalty)

Architecture decision point: do you train one global model, or treat this as pure feature engineering with no trained model at all (just weighted heuristics)? With a personal dataset of maybe a few thousand tracks and a few weeks of history, you may not have enough data for a neural model to meaningfully outperform a well-tuned heuristic score. Recommendation: ship the heuristic first (it's basically free), and treat the neural CF model as a learning exercise you compare against the heuristic -- a great "does my fancy model actually beat my simple baseline?" experiment, which is one of the most important habits in applied ML.

Milestone: for a handful of tracks you have strong feelings about, check whether your implicit score ranks them the way you'd expect -- your most-replayed song should sit near the top, something you skipped every time near the bottom. If the ranking basically matches your gut, you've built a taste profile that didn't exist an hour ago in any explicit form.

Phase 5: The Case Board -- Clustering & Playlist Generation (Weeks 15-18)

Goal: Group tracks into meaningful clusters and turn each cluster into a named, explainable playlist.

Skills gained: Unsupervised clustering (HDBSCAN, K-Means), cluster labeling strategies, combining embeddings from different modalities (audio + behavior + text mood tags), scheduling recurring jobs.

What you'll build:

A combined feature vector per track: concatenate (or weighted-sum) the audio embedding, a mood-tag one-hot/multi-hot vector (from zero-shot classification against labels like "chill," "energetic," "sad," "focus," "romantic," "aggressive"), and behavioral context features (average time-of-day, day-of-week distribution).
A clustering pass using HDBSCAN over that combined vector space. Why HDBSCAN over plain K-Means: K-Means forces every point into a cluster and requires you to pre-specify the number of clusters -- a bad fit here, because you genuinely don't know how many "moods" exist in your listening history, and forcing an odd one-off track into a cluster it doesn't belong in produces bad playlists. HDBSCAN discovers the number of clusters itself and explicitly labels outliers as noise (cluster = -1) rather than cramming them somewhere.
A cluster-labeling step: for each cluster, look at the most common mood tags among its members, the most common time-of-day, and generate a human name via simple rules ("if dominant mood is 'chill' and dominant time is evening -> name candidates: 'Evening Chill', 'Wind Down'"). This is a fun place to eventually swap in an LLM call ("given these five representative songs and these mood tags, suggest a playlist name") as a stretch goal.
Specialized playlist types layered on top of the general clustering:
- Most-played: pure sort by implicit_score / play_count, no clustering needed.
- Mood playlists: direct output of the HDBSCAN clusters.
- Study / focus: filter tracks where the zero-shot classifier's "instrumental" or "low-vocal-density" signal is high and mood tag includes "calm" or "focus" -- this is a good spot to add a simple vocal-presence heuristic (spectral features like zero-crossing rate can proxy for vocal presence without a dedicated model).
- Travel: filter by tracks played during sessions your extension can tag as "away from home" if you choose to add coarse geolocation (optional, privacy-sensitive -- see Chapter 4) -- or more simply, tracks played during longer, uninterrupted sessions (a proxy for commutes or trips).
- Language: filter using a language-detection pass over the (translated or original) lyrics/title text, using a lightweight library like langdetect or a Hugging Face language-ID model.

Learning pit stops:

20 min: read HDBSCAN's own documentation "How HDBSCAN Works" page -- it's written for exactly this level of newcomer and has excellent visuals.
Skim: scikit-learn's clustering comparison page to see K-Means vs DBSCAN vs HDBSCAN side by side on toy datasets -- seeing the failure modes visually will save you hours of confusion later.

Skeleton:

# ml/cluster.py -- the case board, where scattered evidence becomes a story
import hdbscan
import numpy as np

def build_combined_vector(audio_emb, mood_vec, behavior_features) -> np.ndarray:
    # TODO: consider normalizing each component before concatenating --
    #       otherwise a component with larger raw magnitude (e.g. audio_emb)
    #       will dominate the distance metric HDBSCAN uses
    ...

def cluster_tracks(vectors: np.ndarray, min_cluster_size: int = 5):
    clusterer = hdbscan.HDBSCAN(min_cluster_size=min_cluster_size, metric="euclidean")
    labels = clusterer.fit_predict(vectors)
    # labels == -1 means "noise" -- HDBSCAN is telling you this track
    # doesn't fit cleanly anywhere yet. That's useful information, not a bug.
    return labels

def name_cluster(track_rows_in_cluster) -> str:
    # TODO: find most common mood tag, most common time-of-day bucket
    # TODO: map (mood, time_bucket) -> candidate name via a lookup table
    # TODO: fall back to "Mix #<id>" if nothing matches cleanly
    ...

Architecture decision point: re-cluster from scratch every time, or incrementally assign new tracks to existing clusters? Full re-clustering is simpler to reason about and is what you should build first -- run it nightly over your whole track history. Incremental assignment (using each cluster's centroid or a trained classifier) is faster and preserves playlist identity over time (your "Monday Chill" playlist doesn't get renamed every week), and is a great optimization once the naive approach is working and you notice it's slow or that playlist churn is annoying you.

Milestone: run the full pipeline end to end and get back a JSON structure of named playlists with real tracks in them. Take a screenshot of the raw output -- you now have an AI DJ's set list, even before there's a frontend to look at it.

Phase 6: The Broadcast -- Web Player & Full Integration (Weeks 19-22)

Goal: A React frontend that displays your playlists and actually plays music, wired end-to-end to everything you've built.

Skills gained: React component architecture, working with an audio player (HTML5 <audio> or the YouTube IFrame API), API integration patterns, and the specific discipline of wiring together five already-working pieces without breaking any of them.

What you'll build:

A GET /playlists endpoint that returns your generated playlists with track metadata.
A React app with a playlist grid, a track list per playlist, and a persistent bottom player bar (the "Spotify-like" feel).
Playback, decision point spelled out: you have two real options. (a) Embed the YouTube IFrame Player API and drive it programmatically -- simplest, fully compliant with YouTube's terms, but visually and functionally you're still "a YouTube player with a nicer wrapper," and you can't easily do gapless playback or true background audio. (b) Serve audio you've extracted yourself, from your own backend, for genuinely personal, non-redistributed use -- gives you a true from-scratch audio player experience (waveforms, gapless transitions, real <audio> element control) but carries real copyright and terms-of-service responsibilities you must take seriously and keep strictly personal/local, never exposed publicly. Recommendation: build the IFrame API version first, since it's unambiguously fine to build and ship as a personal tool, and treat the self-hosted audio player as an explicitly-labeled "local-only, for-my-eyes-only experiment" if you choose to explore it, never something you deploy publicly.
A "similar tracks" widget: given the currently playing track's embedding, query pgvector for nearest neighbors (ORDER BY embedding <-> :current_embedding LIMIT 5) and render them as suggestions.
Wire the frontend's play events back into the extension's event pipeline (Chapter 1's feedback loop closes here) -- plays that originate in your own app should count as behavioral signal too.

Learning pit stops:

20 min: the YouTube IFrame Player API "Getting Started" guide -- focus on loadVideoById, playVideo, and the onStateChange event.
Skim: any short React hooks refresher if it's been a while (useState, useEffect, useContext for a simple global "currently playing" state).

Skeleton:

// frontend/src/components/Player.jsx -- the stage where the case gets solved
import { useState, useEffect, useRef } from "react";

export default function Player({ currentTrack }) {
  const playerRef = useRef(null);

  useEffect(() => {
    if (!currentTrack) return;
    // TODO: if using YouTube IFrame API:
    //   playerRef.current.loadVideoById(currentTrack.video_id);
    // TODO: log a 'play' event back to your /events endpoint here,
    //       reusing the same schema Phase 1 established
  }, [currentTrack]);

  return (
    <div className="player-bar">
      {/* TODO: track title, artist, progress bar */}
      {/* TODO: mount the YouTube IFrame player (hidden or small) here */}
    </div>
  );
}

# backend/main.py (addition) -- serving the case board to the courtroom
@app.get("/playlists")
def get_playlists():
    # TODO: query your clusters/playlists table
    # TODO: for each playlist, include a handful of representative tracks
    #       and the playlist's dominant mood tags for display
    ...

@app.get("/tracks/{track_id}/similar")
def similar_tracks(track_id: str, limit: int = 5):
    # TODO: SELECT * FROM tracks ORDER BY embedding <-> (
    #         SELECT embedding FROM tracks WHERE id = :track_id
    #       ) LIMIT :limit
    ...

Milestone -- the big one: open your app, see real named playlists built from your real YouTube behavior, click play, and hear music. Take a screenshot of "Monday Chill" (or whatever your system named it) with real tracks in it. This is the moment described in the opening story finally closes the loop -- you are, quite literally, looking at the output of a detective story you wrote about yourself.

Chapter 3: The Brain of the System — Meet Your Neural Concierge

Phases give you a build order. This chapter zooms into the thinking tools you'll reach for repeatedly — the habits of mind that separate "I copied a tutorial" from "I understand what I built."

How to search, read docs, and debug like you've done this before

You haven't done this before, and that's fine — nobody starts out knowing how to google effectively for a niche integration. Here's the actual process, made explicit:

Name the smallest unit of your confusion. Not "how do I do audio classification" — that's a whole field. Instead: "how do I get a fixed-length vector out of a variable-length audio file using a pretrained model." That sentence is close to a good search query.
Search for the library name plus the verb, not the concept plus the verb. "yt-dlp extract audio python" beats "how to download youtube audio." Library-specific queries land you on docs and GitHub issues where someone already hit your exact wall.
Read the signature before the prose. When you land on a function's docs, look at its parameters and return type first. Half the time that alone answers your question faster than the paragraph above it.
When you're stuck for more than 20-30 minutes, write down what you expected to happen and what actually happened, in one sentence each. This is the single highest-leverage debugging habit there is — most of the time, writing that sentence reveals the bug before you've even searched anything, because it forces you to notice the gap between your mental model and reality.
Reproduce in isolation. If your pipeline breaks on the audio-embedding step, don't debug it inside the full nightly job — copy the failing call into a throwaway script or a notebook cell with one hardcoded input. You want the shortest possible loop between "change something" and "see the result."

You'll use this loop dozens of times across the next five months. It's worth more than any specific fact in this guide.

Trade-off thinking, one more time with feeling

Every phase above included at least one "architecture decision point," and that's not decoration — it's the actual skill this project is teaching you. Real systems are built from a sequence of decisions where there's no objectively correct answer, only a right answer for your constraints. The questions to ask yourself every time are the same four:

What does each option cost me right now (time, complexity, new concepts to learn)?
What does each option cost me later (technical debt, rework, scaling pain)?
Do I actually have the problem this option solves, or am I solving a problem I imagine having?
Can I try the cheap option first and upgrade later, or is this a decision that's expensive to reverse?

Notice that almost every recommendation in this guide follows the same shape: start with the boring, cheap, well-understood option; treat the fancy option as a deliberate, isolated upgrade once you've felt the limits of the simple one. That's not a lack of ambition — it's how you make sure the ambition is spent on the parts of the project that are genuinely novel (your specific taste model, your specific clustering logic) instead of getting burned on generic infrastructure choices that a thousand tutorials have already solved for you.

NLP for mood: zero-shot first, fine-tuning as a deliberate upgrade

You'll use Hugging Face's zero-shot-classification pipeline in two places: initial content classification (Phase 2) and mood tagging (Phase 5). It's worth understanding why zero-shot works at all — the underlying model (typically an NLI, natural-language-inference, model like BART-MNLI) was trained to judge whether one sentence entails another. Zero-shot classification is a clever repurposing: it turns "does this song feel energetic?" into "does the premise 'this text describes energetic music' entail the hypothesis built from your candidate label," and ranks labels by entailment confidence. You get calibrated-ish multi-label scores over any label set you dream up, with zero training.

The catch: zero-shot models are working from text — titles, descriptions, maybe lyrics if you fetch them — not from the audio itself. A song titled "Sunny Day Vibes" will zero-shot-classify as upbeat even if the actual track is a somber ballad using the title ironically. This is exactly why Phase 5's combined vector blends the zero-shot mood tags with the audio embedding — text tells you what the creator labeled the song as, audio tells you what it actually sounds like, and behavior tells you how you actually responded to it. None of the three signals is trustworthy alone; together they triangulate something closer to the truth.

When to fine-tune instead of relying on zero-shot: once you've hand-corrected mood tags on maybe 200-300 tracks (a natural byproduct of using your own app and noticing "this is tagged wrong"), you have enough labeled data to fine-tune a small classifier — even a lightweight one trained on top of frozen text embeddings — specifically on your vocabulary of moods and your taste in how songs get labeled. This is a genuinely great Month 4-5 stretch goal: it's the moment your mood tagger stops being "a generic pretrained model" and starts being "a model that knows your ears."

The math of implicit feedback, one level deeper

Section from Phase 4 gave you a working formula. Here's the conceptual core worth sitting with: in explicit feedback (star ratings), the model's job is to predict a rating, and the loss function punishes wrong ratings directly. In implicit feedback, there's no rating to predict — only binary or continuous signals of engagement. The standard reframe (going back to Hu, Koren & Volinsky's foundational work) is to treat every (user, item) pair as having a binary preference (did they like it, roughly, yes/no) plus a confidence in that preference, where confidence grows with how much engagement evidence you've observed. A song you've watched once for 10 seconds contributes a weak, low-confidence data point. A song you've replayed eleven times contributes a strong, high-confidence one. Your loss function (if you get to training a real model) should weight errors on high-confidence pairs more heavily than errors on low-confidence ones — getting a well-established favorite's ranking wrong is a bigger problem than getting an ambiguous one-play track wrong.

Practically, for a personal-scale project, you don't need to implement the full weighted matrix factorization machinery to benefit from this idea — just make sure your implicit_score formula and your neural CF's context features both account for "how many times have I actually interacted with this" as a first-class signal, not an afterthought.

Neural collaborative filtering, sketched a little further

The Phase 4 skeleton gave you the shape: an embedding layer per track, concatenated with hand-engineered context features, passed through dense layers to a final prediction. A few things worth understanding about why this works before you write more of it:

The embedding layer is learning a compressed "taste coordinate" for each track — not from any explicit attribute you gave it, but purely from patterns in which tracks tend to get high implicit scores together, in similar contexts. Two tracks can end up with similar embeddings even if their audio sounds nothing alike, if you consistently listen to them in the same moods/times.
This is a genuinely different signal from the audio embedding. Audio embeddings capture "sounds like." CF embeddings capture "gets consumed like." A great expansion once both exist: concatenate them (as Phase 5 does) so your clusters reflect both what things sound like and how you actually use them.
With a small personal dataset, expect this model to overfit fast. That's not a failure — it's expected, and it's a good opportunity to learn regularization basics (dropout between dense layers, L2 weight decay on the embedding) and to practice train/validation splitting even when your dataset is uncomfortably small. Keep a held-out slice of your most recent week's data and check whether the model's predicted scores actually track your real subsequent behavior — that's your only honest signal that any of this is working.

Chapter 4: The Data Story — Where Evidence Lives

Every detective needs an evidence locker with a sane filing system. Here's yours.

The schema, roughly

raw_events — the unfiltered sensor feed. Every play, pause, seeked, progress, ended tick from the extension, tied to a video_id and session_id. This table only grows; you never edit it, only append. Treat it the way you'd treat a security camera's footage — the raw record, kept intact, that every later interpretation can be checked against.
tracks — one row per resolved song: video_id, artist, title, channel, is_music (bool), duration_seconds, audio_embedding (a pgvector column), mood_tags (array or jsonb), language, musicbrainz_id (nullable), implicit_score (recomputed periodically), created_at, updated_at.
sessions — optional but genuinely useful: groups of events with a shared session_id, with derived fields like session_start, session_length, dominant_time_bucket (morning/afternoon/evening/night). This is where "travel" and "study" playlist heuristics get their signal.
playlists and playlist_tracks — the output of the clustering step: a playlist has a name, a generation timestamp, dominant mood tags, and a join table linking it to track rows, ordered.

The key design idea: raw data is immutable, derived data is regenerable. If your classification logic improves in Month 4, you should be able to delete and recompute the tracks table's is_music and mood_tags columns from raw_events without losing anything real. This is the same principle behind "extract-then-delete" for audio from Phase 3 — keep the cheap-to-regenerate stuff disposable, keep the expensive-to-recreate stuff (your actual watch history) sacred.

Bootstrapping with Google Takeout

Your extension only sees behavior from the moment you install it forward. If you want historical depth on day one, Google Takeout's YouTube history export gives you a JSON/HTML dump of your watch history going back as far as YouTube has recorded it. It's coarser than extension data — no watch percentage, no skip detection, just "you watched this, at this timestamp" — but it's real, and it's enough to seed your tracks table and get Phase 2's classifier something to chew on in week one, instead of waiting three weeks for organic data to accumulate.

How to use it: request a Takeout export (JSON format), write a one-off import script that reads the export and inserts synthetic raw_events rows with event_type = 'legacy_watch' and no position data, then let Phase 2's classifier run over them exactly as it would over live data. Treat legacy rows as lower-confidence in your implicit-score formula — you don't know how much of the video was actually watched, so don't let a legacy "watched" event count as strongly as a fully-instrumented one.

Simulating data for development

You don't want to wait around watching YouTube for hours just to test whether your clustering code handles 500 tracks reasonably. Write a small synthetic data generator early — even a rough one — that creates plausible-looking raw_events rows: pick from a pool of real song titles (you can hand-curate 100-200 from your own taste, spanning a few genres/moods), assign random-but-plausible watch percentages and timestamps clustered around a few "mood personas" (a synthetic "you" who watches chill music at night and energetic music in the morning). This does two things: it lets you develop and test the ML/clustering code without waiting on real data, and it gives you a known-answer test — since you control the synthetic personas, you can check whether your clustering actually recovers something close to them, which is a much stronger validation than "the code ran without crashing."

Privacy and local-first design

This system is, by design, an intimate profile of your habits — what you listen to, when, how obsessively. A few commitments worth making explicit before you write a line of storage code:

Keep it local by default. Run Postgres on your own machine or a private instance you control, not a shared cloud service, unless you've deliberately decided to and secured it (auth, no public ports, encrypted connections).
Never redistribute extracted audio. Phase 3's "extract-then-delete" isn't just an engineering nicety — it's the difference between "a personal analysis tool" and "a tool with a copyright problem." Keep audio files ephemeral, process-and-discard, never served to anyone but the pipeline that needs them for a few seconds.
Be deliberate about optional signals like geolocation. The "Travel" playlist idea in Phase 5 floated coarse geolocation as one possible signal. If you add it, make it opt-in, store it at low resolution (city-level, not GPS-precise), and be honest with yourself about whether the marginal playlist quality is worth the marginal privacy cost. The session-length proxy (no geolocation needed) gets you most of the value for none of the risk — a good default.
You are both the engineer and the subject. That's unusual and worth sitting with: normally, privacy-by-design is about protecting other people's data from your system. Here, you're the one whose intimate behavioral data is on the line, which makes it a genuinely good, low-stakes place to practice privacy-conscious engineering habits before you ever build something that touches anyone else's data.

Chapter 5: The Galaxy of Songs — Playlist Generation Magic

Embeddings, explained like you're teaching a friend at a bar

Imagine every song as a point of light in a vast dark space. Songs that sound alike, or that you tend to listen to in similar moods, drift close together — not because anyone placed them there by hand, but because the math that produced their coordinates was trained (or, for pretrained models, trained by someone else on a huge amount of audio) to put "things that behave similarly" near each other and "things that behave differently" far apart. That's an embedding: a long list of numbers (say, 128 or 512 of them) that represents a song not as an ID, but as a position in a meaningful space. Two songs' embeddings being close together, measured by something like cosine similarity, is a proxy for "these songs are alike in whatever sense the model was trained to notice."

You have two galaxies in this project: the audio galaxy (positions determined by what things sound like) and the behavioral galaxy (positions determined by how you actually consume them, learned via the collaborative-filtering embeddings from Chapter 3). A track's true "meaning" in your system lives at the intersection — which is exactly why Phase 5 concatenates both into one combined vector before clustering. A cluster in that combined space isn't just "songs that sound similar," and it isn't just "songs you listen to in similar contexts" — it's songs that are similar on both axes simultaneously, which is a much stronger, more personally meaningful notion of "these belong on the same playlist" than either signal alone could produce.

Clustering and automatic labeling, tied together

HDBSCAN doesn't know what a "mood" is. It only knows distances. The intelligence in the system comes from what you fed it (the combined embedding) and how you interpret its output (the labeling heuristic). This division of labor is worth internalizing as a general ML pattern: unsupervised algorithms are excellent at finding structure, and terrible at explaining what the structure means — that translation step is where your domain knowledge (you know what "chill" sounds like to you) has to do the work, usually through a simple, inspectable rule rather than another opaque model. Keep that labeling step simple and debuggable for as long as possible; it's the layer you'll be staring at and tweaking the most, because it's the layer that decides whether a playlist gets a name you'd actually want to click.

Filtering for activity-based playlists

"Study" and "Travel" playlists aren't clusters in the audio/behavior space so much as they're filters layered on top of it — a study playlist is really "the calm, low-vocal-density cluster, further filtered to sessions where you historically kept the same track playing for a long uninterrupted stretch (a proxy for focus, since skip-heavy sessions suggest browsing, not working)." Treat these as compositions of your existing signals rather than new things to model from scratch — it's both less work and more explainable, which matters a lot when you're debugging why a jazz track ended up in "Study" and a moment's thought about "oh, right, low vocal density plus long uninterrupted plays" tells you exactly why.

A future reinforcement learning layer

Once the system exists and you're using it daily, a natural next question emerges: instead of only observing your behavior passively, could the system learn from how you react to its own suggestions? This is where a lightweight RL framing becomes interesting — treat playlist generation as a policy that chooses which tracks to include, treat your subsequent behavior (did you skip the second track in "Monday Chill," did you let the whole playlist play through) as a reward signal, and use something like a contextual bandit (a simpler, more tractable cousin of full RL, well suited to problems where you're repeatedly choosing from a set of options and observing an immediate reward) to nudge future playlist composition toward what you actually engage with. This is explicitly a later idea — Chapter 9 revisits it as a genuine "beyond the guide" project, because it depends on having a stable, working system generating playlists you actually use regularly first. RL on top of a system that doesn't have real usage data yet is RL on noise.

Chapter 6: The Web Dev Spine — Extension, Backend, Frontend, Tied Together

The ML chapters get the glamour, but this is the skeleton everything else hangs on. A quick tour of the parts you haven't already built phase-by-phase, plus the connective tissue between them.

The extension, structurally

A Chrome extension (Manifest V3) has three moving pieces you need to hold in your head as separate execution contexts that can't directly touch each other's variables — they only talk via message passing:

manifest.json — declares permissions (activeTab, host permissions for *.youtube.com), which scripts run where, and whether you have a background service worker.
The content script — runs inside the YouTube page's context, so it can read the DOM and attach listeners to the <video> element, but it's sandboxed from the extension's other state and can't make cross-origin requests as freely.
The background service worker — has broader permissions (can make network requests to your backend without CORS headaches, can persist state across tab navigations) but can't touch the page DOM directly.

The message-passing pattern you'll lean on constantly: content script calls chrome.runtime.sendMessage({type: 'EVENT', payload: {...}}), background script listens via chrome.runtime.onMessage.addListener, batches, and does the actual fetch to your API. Get comfortable with this pattern early — Phase 1 depends on it, and every later extension feature (detecting SPA navigation between videos, badge icon updates showing tracking status) reuses it.

REST API design that won't fight you later

A few conventions worth committing to from Phase 1 onward, because retrofitting them later is annoying:

Version your API from day one (/api/v1/events), even though you're the only consumer. Future-you, three months in, adding a mobile client or a second extension version, will thank present-you.
Keep ingestion endpoints (POST /events) fast and dumb. They should do validation and a write, nothing else. Any classification, embedding, or clustering work belongs in the offline pipelines, not inline in the request path — mixing them means a slow ML call can make your extension feel laggy or drop events under load.
Design read endpoints (GET /playlists, GET /tracks/{id}/similar) around what the frontend actually needs to render, not around your database schema. It's fine, even good, for an endpoint to join and reshape data server-side so the frontend doesn't have to stitch together three separate calls just to draw one screen.

Streaming audio, and the ethical note this section owes you

If you go the self-hosted-audio route from Phase 6's decision point, you're technically capable of building a /stream/{track_id} endpoint that reads an extracted audio file and streams it with proper Range header support so the browser's <audio> element can seek. Build it if you want the learning experience of implementing HTTP range requests (a genuinely useful, transferable skill — it's the same mechanism video platforms use). But hold the line firmly: this stays a local, single-user, never-publicly-deployed feature. The moment "personal tool running on my own machine for my own previously-watched content" turns into "a service anyone else can reach," you've crossed from a personal fair-use-adjacent experiment into redistribution, which is a different, real legal category. If you ever want to share this project publicly (a portfolio demo, a hackathon submission), disable or strip the self-hosted audio path and demo with the YouTube IFrame API version instead — it's not a downgrade, it's the responsible default.

Frontend architecture, briefly

Keep state simple: a top-level currentTrack and currentPlaylist in React context (or a lightweight state library if you want the practice), a PlaylistGrid component that fetches from /playlists on mount, a Player component that owns the actual playback element and logs playback events. The "similar tracks" widget is a nice small component to practice component composition on: it takes a trackId prop, fetches /tracks/{id}/similar on prop change, and renders a small horizontal list — self-contained enough to build and test in isolation before wiring it into the main player view.

Tying the vector database to the frontend

The satisfying part of pgvector is that "find similar tracks" is one query, not a separate service call to a vector database plus a join back to your relational metadata:

SELECT id, artist, title, embedding <-> (
    SELECT embedding FROM tracks WHERE id = :current_track_id
) AS distance
FROM tracks
WHERE id != :current_track_id
ORDER BY distance
LIMIT 5;

That single query is doing something that would otherwise require standing up and syncing two separate systems. It's a small thing, but it's the kind of small thing that, once you've felt the alternative, you start actively choosing simpler architectures for the rest of your career.

Chapter 7: The Detective's Logbook — Milestones Across the Whole Case

You've hit a concrete milestone at the end of every phase in Chapter 2, but it's worth stepping back and seeing the whole arc as one continuous story, because that's genuinely what it is:

Phase 0: two independent heartbeats — an extension that logs, a server that responds.
Phase 1: the system sees you for the first time. A real, timestamped trail of your own behavior, sitting in a database, that didn't exist yesterday.
Phase 2: the system starts telling music apart from everything else — its first act of judgment.
Phase 3: the system develops senses beyond text — it can now "hear," in the limited but real sense of turning sound into a comparable, structured signal.
Phase 4: the system develops a memory of your taste that isn't just "what you told it" but "what you actually did," which is a fundamentally more honest signal than any explicit rating system.
Phase 5: scattered evidence becomes a story — songs organize themselves into groups you didn't manually create, with names that feel earned rather than assigned.
Phase 6: the system finds its voice, and hands the story back to you in a form you can actually use, day to day, the way you'd use Spotify — except every playlist in it is an artifact of your own engineering.

Each of those is worth a screenshot, a commit message that actually describes what changed and why, and — genuinely — a moment of letting yourself feel like this is a big deal. It is. Most people who use recommendation systems every day have no idea how the black box works. You're about to have built one, from the sensor all the way to the speaker.

Chapter 8: Beyond the Guide — Where the Case Reopens

Five months from now, you'll have a working, personal AI DJ. Here's where the story can keep going, roughly in order of "natural next step" to "ambitious departure":

Deploy it somewhere always-on, like a Raspberry Pi or a small home server, so the nightly pipeline runs whether or not your laptop is open. This is a good, contained systems-administration project on its own — cron jobs, systemd services, basic monitoring so you notice if the pipeline silently stops working.
Add voice control — "play something chill" parsed via a small intent-classification step (zero-shot classification again, this time over your playlist names and mood tags) wired to a wake-word library or a simple push-to-talk button in the frontend.
Integrate with the real Spotify API as a second data/playback source, letting your behavioral model learn from both platforms at once, and letting your generated playlists optionally get pushed into actual Spotify playlists you can use anywhere, on any device.
Build the reinforcement learning layer sketched in Chapter 5 — once you have real usage data (skips, replays, completions) on the system's own suggestions, not just your organic YouTube behavior, you have the reward signal that makes a bandit-style approach genuinely meaningful instead of speculative.
Turn it into a SaaS, if you want the entirely different (and genuinely valuable) experience of learning multi-tenant architecture, per-user data isolation, OAuth flows for other people's YouTube accounts, and all the "now it's not just your data anymore" privacy engineering that Chapter 4 flagged as a different category of problem. Don't take this step lightly or quickly — it's a good six-month project in its own right, layered on top of the five months you'll have already spent.

None of these are required to call the project "done." The five-month build is a complete story with a real ending. Everything in this chapter is a sequel hook, not an unfinished obligation.

Chapter 9: The Final Note

You started this case with an irritating, wonderful little observation: somewhere in your YouTube history was a pattern nobody, not even you, could see clearly. Over the next five months, you built the instrument to see it — a browser extension that learned to watch the way you watch, a pipeline that learned to hear the way songs sound, a model that learned your taste from what you actually did instead of what you said, and a clustering engine that turned all of that into playlists with names that feel like they know you, because in the most literal sense, they do.

Along the way you picked up a working fluency in things that sound intimidating from the outside and turn out to be learnable in exactly the incremental way this guide walked you through: DOM events and extension messaging, REST API design, mel spectrograms and audio embeddings, zero-shot NLP classification, the actual math behind implicit feedback, neural collaborative filtering, density-based clustering, vector similarity search, and a real frontend audio player. Each of those, on its own, is a legitimate skill line on a resume. Together, wired into one living system, they're something rarer: a complete, working thing you built end to end, that solves a problem only you have, in a way that only you could have tuned it.

You started with a YouTube habit. You're ending with a machine learning orchestra — one that watches, listens, remembers, and plays, quietly, in the background of your life, getting a little more you every single day it runs.

Case closed. Time to hit play.