How building an AI chapter generator taught me to stop reaching for a vector database first.
When I started building my AI chapter generator, I spent hours researching vector databases. I was convinced I needed one.
I didn't.
The idea
The project was simple on paper. A user uploads a long lecture video, the app transcribes it, figures out where the instructor moves from one topic to the next, and spits out timestamped chapters.
Before I'd written a single line of the actual feature, my brain had already decided what the architecture should look like. Long text plus AI app equals: chunk it, embed it, store it, retrieve it. That's just the shape everyone's trained to see these days.
So that's the road I went down. I spent an evening comparing Pinecone, Chroma and FAISS. I read through chunking strategies, fixed size chunks, overlapping windows, semantic chunking, all of it. I had a RAG pipeline half sketched out in my notes app.
Then I stopped and asked myself something a lot more basic:
What problem am I actually trying to solve?
I didn't have a good answer. So I backed up and started over.
What embeddings actually do
If you're newer to this stuff, here's the plain version.
An embedding turns a piece of text into a list of numbers that captures what it means. Text with similar meaning ends up with similar numbers. That's really the whole trick behind it. It turns "how similar are these two ideas" into "how close are these two points," which is something a computer can check almost instantly, even across millions of documents.
That's a genuinely useful trick, but notice what it's actually for. It's for finding things. If you've got 10,000 support articles and someone asks a question, you can't just feed all 10,000 into the model every time. It wouldn't fit, and even if it somehow did, it'd be painfully slow. Embeddings let you narrow that pile down to the handful of articles that are actually relevant, and only those get handed to the model.
That narrowing down is retrieval. And retrieval turns out to be a different job than reasoning.
Retrieval vs reasoning
This is the part that took me a minute to actually get, and I think it's the bit most people gloss over.
Embeddings solve retrieval. They're good at finding a needle in a haystack you can't fully read.
LLMs handle reasoning. They think through whatever's in front of them.
RAG is called Retrieval Augmented Generation because retrieval is just a preprocessing step that decides what the model gets to look at. It doesn't make the model reason any better once it's looking at the right thing. It just makes sure it's looking at the right thing in the first place.
Which leads to an obvious question. What if the model already has the right thing?
That was basically my situation. The whole transcript was already right there. Nothing was buried somewhere I needed to dig it out of. There was nothing to find. Just something to work through, in order.
But don't long transcripts still need chunking?
Yes, and this tripped me up for a bit, so it's worth spelling out.
A one hour lecture transcript is way too long to comfortably send in one model call. So you still have to split it up somehow. That part's unavoidable.
But there are two different reasons you'd chunk something, and I'd been mushing them together in my head without realizing it.
Chunking for retrieval splits text so you can search it. Each chunk gets embedded and scored against a query, and only the best matches get used.
Chunking for context just splits text because of size limits. Every chunk still gets used, in order, nothing gets filtered out.
I needed the second kind. Not "which piece is most relevant," but "here's the whole thing in pieces, go through all of it, in order." That doesn't need embeddings. It just needs a loop.
The moment it clicked
Once I'd separated retrieval from reasoning in my head, and chunking to search from chunking to fit, things got a lot clearer fast.
I asked myself how a person would do this task. Not an AI, an actual human sitting there watching the lecture.
They wouldn't be scanning through looking for the paragraph most "similar" to some query, because there's no query. Nobody's asking a question. They'd just watch it from start to finish and notice, naturally, when the instructor wrapped up one idea and moved on to the next.
That's when it actually clicked. Chapter generation is about sequence, not similarity.
Embeddings answer "where in this pile does something like X show up." My problem was "where does this one, ordered thing change." Those aren't the same problem wearing different clothes. They're genuinely different problems, and no clever chunking-for-retrieval trick was going to fix that, because retrieval was never actually the job.
What I thought I needed
Transcript
│
▼
Chunk for retrieval
│
▼
Embed each chunk ──► Vector database
│ │
│ (similarity search)│
▼ ▼
Query-relevant chunks
│
▼
LLM
│
▼
Chapters
What I actually built
Transcript
│
▼
Chunk sequentially (for size, not search)
│
▼
LLM reads each chunk in order,
carrying forward a rolling summary
│
▼
Flag topic transitions
│
▼
Merge + align to natural pauses
│
▼
Chapters
Same input, same output, half the pipeline.
What I actually built
Here's the workflow, roughly:
- Transcribe the video.
- Split the transcript into sequential chunks of a few minutes each, keeping order and timestamps intact.
- Feed each chunk to the model along with a short rolling summary of everything before it, and ask it to flag where the topic shifts.
- Merge the transitions that land close together, and snap each boundary to the nearest natural pause so a chapter doesn't start mid sentence.
Step 3 is really the whole trick. Each chunk carries context forward from the ones before it. The model isn't staring at six disconnected fragments trying to guess how they relate, it's walking through the lecture the way a person would, carrying the thread as it goes. That continuity is what actually lets it notice "okay, this is where the topic changed." It's also exactly what a similarity search would throw away, since similarity search treats every chunk as its own isolated thing to be scored.
No vector database, no embedding calls, no retrieval step.
The real reason I dropped embeddings
I could frame this as a story about cost or speed. Fewer moving parts, lower latency, no vector DB bill. Those were real benefits and I did get them. But that's not actually why I made the change, and leading with that would kind of miss the point.
The real reason is simpler. Embeddings would have solved a problem I didn't have. My bottleneck was never "how do I find the relevant part of this transcript," because the whole transcript was the relevant part. Bolting retrieval on top wouldn't have made the output better, it would've just added machinery answering a question nobody was asking, while the actual question, where does the topic change, stayed exactly as unsolved as before.
Cost and latency got better as a side effect of cutting something unnecessary. They weren't the reason it was unnecessary.
Embeddings are still great, when you actually need retrieval
None of this is an argument against embeddings. I still use them all the time, just for a different kind of problem.
Say you're building a support bot on top of 10,000 help articles. Someone asks a question and there's no way to hand the model all 10,000 articles at once, it needs to find the handful that are actually relevant first. That's exactly the retrieval problem embeddings were built for, and it's a great use of a vector database there.
The difference between that and my chapter generator isn't the amount of text involved, both had plenty. The difference is whether the model needs to search across a bunch of separate things, or read through one thing it already has in full.
Needs retrieval: company docs, big knowledge bases, research paper libraries, product manuals, support article collections, long chat history archives.
Doesn't need retrieval: summarizing a single document, translating text, classifying content, pulling structured data out of one file, reviewing a single codebase, generating chapters from one transcript.
If the model already has everything it needs sitting in front of it, embeddings don't help it reason better. There's nothing to find, so there's nothing for retrieval to speed up.
A question worth stealing
Before I add a vector database to anything now, I ask myself one thing, and it's worth stealing if you don't already ask it:
Does my AI need to find information, or does it already have it?
If it already has everything, embeddings won't make the answer any sharper. They'll just add cost, latency, and one more system that can quietly break on you.
Final thoughts
I almost built a whole retrieval pipeline for a problem that had nothing to retrieve, just because that's the default move everyone reaches for. The real lesson here wasn't about embeddings specifically. It was about catching myself solving the problem the tutorials describe instead of the one actually sitting in front of me.
The best architecture isn't the one with the most pieces. It's the one that solves the actual problem with the least stuff bolted on.
Top comments (0)