Giving your AI agent access to a podcast library, via MCP

Xin Wu — Mon, 19 Oct 2026 14:45:08 +0000

I listen to a lot of podcasts. The problem was never finding episodes, it was using them afterwards: a 90-minute conversation contains maybe four ideas worth keeping, and by the next week I can remember the idea but not the episode, the timestamp, or the guest's name.

Notes apps did not solve this. I would write "the bit about pricing floors" and six months later have no way back to the source.

This post is about the architecture that finally worked: treat the podcast library as a queryable store, and expose it to an LLM through the Model Context Protocol so the agent can search it the way it searches a codebase.

Why transcripts alone are not enough

The obvious move is transcription. That gives you text, but raw transcripts are hostile to retrieval:

90 minutes of speech is roughly 13,000 words, most of it filler.
Speaker turns are unlabeled unless you diarize.
A question asked at minute 12 is answered at minute 40, so the useful span does not sit in one chunk.
Names and numbers are transcribed inconsistently, which breaks keyword search.

So the pipeline needs a structure layer before retrieval: summary, key points, and a topic map, all anchored to timestamps. Then retrieval can operate on structure and fall back to transcript spans.

What MCP changes

Without MCP, every client integration is bespoke: one plugin for ChatGPT, one for Cursor, one for a CLI. With MCP, the server exposes tools once and any MCP-capable client can call them.

The tool surface I ended up with is small on purpose:

Tool	Input	Returns
`search_episodes`	query, optional date range	episode IDs with matched spans and timestamps
`get_episode`	episode ID	summary, key points, chapter map
`get_transcript`	episode ID, optional time window	verbatim text for that window
`list_recent`	limit	latest imported episodes

The important constraint is that search_episodes returns spans with timestamps, not whole documents. An agent that receives 13,000 words will summarize the middle and lose the point. An agent that receives six 200-word spans with timestamps can answer precisely and cite.

Connecting it

If you want to try the shape of this without building it, Podwise runs a hosted MCP server at https://mcp.podwise.ai/mcp that exposes a library you have imported there: Apple Podcasts, Spotify, YouTube, RSS and private feeds, with summaries, transcripts, mind maps and search across everything. One OAuth sign-in from Claude, ChatGPT or Cursor and the tools appear. There is a CLI as well (brew install hardhackerlabs/podwise-tap/podwise) if you prefer scripted imports.

The client side is unremarkable: point the client at the server, complete OAuth, and the four tools above show up in the tool list.

What actually works in practice

Three query patterns earned their place:

"Where did someone argue X?" — span search across the library, returns timestamps, no hallucination risk because the answer is quoted.
"Summarize everything this guest said about Y across three appearances." — cross-episode aggregation, which is where a plain transcript search tool falls over.
"What did I listen to last month that mentioned Z?" — recency filter plus search, useful for writing.

What does not work well yet: questions that require understanding audio the transcript missed (heavy accents, crosstalk), and anything that needs the visual context of a video podcast.

The design takeaway