I Built an AI Tool That Transcribes and Summarizes Podcasts Automatically

FTC Disclosure: I'm the developer of PodCrisp, the AI podcast tool discussed in this article.

---

I listen to a lot of podcasts. About 15 hours per week, across business, tech, and design shows. The problem? I kept forgetting key insights from episodes I'd listened to days earlier. I'd hear "Oh, this reminds me of that podcast episode about..." and then draw a complete blank.

Sound familiar?

I tried note-taking apps, bookmarking episodes, even hiring a virtual assistant to take notes. Nothing worked well. So I did what any developer would do — I built my own solution.

Meet PodCrisp: an AI-powered tool that transcribes, summarizes, and extracts actionable insights from any podcast episode automatically.

The Technical Stack

PodCrisp is built on a pipeline architecture. Each stage processes the podcast content and passes enriched data to the next:

RSS Feed → Audio Download → Transcription → LLM Processing → Structured Output

Let me break down each component.

Stage 1: Audio Acquisition

PodCrisp accepts podcast episodes via RSS feed URL or direct audio upload. The system parses standard podcast RSS feeds (RSS 2.0 with iTunes extensions) and extracts:

• Episode audio URL (MP3/AAC/M4A)
• Episode metadata (title, description, publish date, duration)
• Show metadata (title, author, artwork)

import feedparser

def parse_podcast_feed(feed_url: str) -> list[Episode]:
    feed = feedparser.parse(feed_url)
    episodes = []
    for entry in feed.entries:
        audio_url = None
        for link in entry.get("links", []):
            if link.get("type", "").startswith("audio/"):
                audio_url = link["href"]
                break
        if audio_url:
            episodes.append(Episode(
                title=entry.title,
                audio_url=audio_url,
                published=entry.get("published"),
                duration=entry.get("itunes_duration"),
            ))
    return episodes

Stage 2: Transcription

This is where things get interesting. I evaluated several transcription services:

Service	Accuracy	Speed	Cost/hr
OpenAI Whisper API	95%+	~30s/min	$0.36
AssemblyAI	93%+	~20s/min	$0.65
Deepgram	91%+	~15s/min	$0.36
Local Whisper	94%+	Variable	Free

I went with OpenAI's Whisper API for the best balance of accuracy and cost. The key insight was that podcast audio quality varies wildly — some are studio-recorded, others are Zoom calls with background noise. Whisper handles this variation better than alternatives.

import openai

def transcribe_episode(audio_path: str) -> str:
    # Split long audio into 25MB chunks (Whisper API limit)
    chunks = split_audio(audio_path, max_size_mb=25)
    transcripts = []
    for chunk in chunks:
        with open(chunk, "rb") as audio_file:
            transcript = openai.audio.transcriptions.create(
                model="whisper-1",
                file=audio_file,
                response_format="verbose_json",
                timestamp_granularities=["segment"],
            )
            transcripts.append(transcript.text)
    return " ".join(transcripts)

One non-obvious challenge: speaker diarization. Whisper doesn't distinguish between speakers. For the summary, this matters — knowing who said what adds context. I solved this by combining Whisper timestamps with the podcast's chapter markers (when available) to approximate speaker segments.

Stage 3: LLM Processing

This is the magic layer. I send the transcript to a large language model with a carefully crafted prompt that extracts:

1. Summary — A concise 3-5 paragraph overview
2. Key Takeaways — 5-10 actionable insights
3. Quotes — Notable quotes with timestamps
4. Topics — Categorized discussion topics
5. Action Items — Things the listener should do

SUMMARY_PROMPT = (
    "You are a podcast analyst. Given the following transcript, "
    "produce a structured analysis:\n\n"
    "1. SUMMARY: Write a 3-5 paragraph summary capturing the main discussion.\n"
    "2. KEY_TAKEAWAYS: List 5-10 actionable insights or lessons.\n"
    "3. NOTABLE_QUOTES: Extract 3-5 memorable quotes with approximate timestamps.\n"
    "4. TOPICS: Categorize the discussion into 3-5 topic areas.\n"
    "5. ACTION_ITEMS: List specific actions the listener should consider.\n\n"
    "Format your response as JSON matching the provided schema.\n\n"
    "TRANSCRIPT:\n{transcript}\n"
)

The prompt engineering took significant iteration. Early versions would hallucinate quotes or mix up speakers. I found that:

• Including timestamps in the transcript chunk helped the model reference specific moments
• Asking for JSON output with Pydantic validation caught formatting errors
• Splitting long transcripts into 10-minute segments and merging results improved accuracy for hour-long episodes

Stage 4: Structured Output

The final output is a rich, structured document:

{
  "title": "Episode 42: Building in Public",
  "summary": "In this episode, the hosts discuss...",
  "key_takeaways": [
    "Share your progress publicly to build accountability",
    "Engagement metrics matter less than genuine connections",
    "Consistency beats perfection in content creation"
  ],
  "notable_quotes": [
    {
      "text": "The best marketing is just being transparent about what you're building.",
      "timestamp": "12:34",
      "speaker": "Host"
    }
  ],
  "topics": ["marketing", "indie-hacking", "content-strategy"],
  "action_items": [
    "Start a weekly progress update thread",
    "Share one technical challenge you solved this week"
  ]
}

Architecture Decisions

Why Not Just Use ChatGPT on the Transcript?

You could, but there are practical issues:

1. Context limits — A 2-hour podcast transcript is 20,000-30,000 words. That's beyond most models' effective processing range.
2. No structure — ChatGPT gives you a wall of text. PodCrisp gives you categorized, timestamped, actionable output.
3. No persistence — ChatGPT conversations disappear. PodCrisp builds a searchable knowledge base over time.
4. No automation — PodCrisp watches your RSS feeds and processes new episodes automatically.

Handling Scale

PodCrisp processes podcasts using a queue-based architecture:

• Cloudflare Workers handle the API layer and RSS feed polling
• KV storage caches episode metadata and processed summaries
• Background tasks process transcription and LLM calls asynchronously

The biggest cost optimization was caching. Many popular podcasts discuss overlapping topics. PodCrisp caches topic-level summaries so when a new episode covers similar ground, it can build on existing knowledge rather than starting from scratch.

The Result

After three months of daily use, here's what PodCrisp has done for me:

• Processed 200+ episodes across 15 podcasts
• Saved approximately 8 hours/week in note-taking and recall
• Built a searchable knowledge base of podcast insights
• Never forgot a key insight again — full-text search across all summaries

The tool has evolved from a personal productivity hack to a product I'm sharing with other podcast addicts. If you listen to podcasts for learning and want to retain more without spending more time, check out PodCrisp.

What's Next

• Multi-language support — Transcription and summaries in Spanish, French, German, Japanese
• Podcast recommendations — AI-powered suggestions based on your listening patterns and summary interests
• Team sharing — Share episode summaries with your team, assign action items
• Custom analysis — Define your own extraction templates for specific use cases (interview prep, competitive analysis, research)

---

How do you currently take notes on podcasts? What's your biggest pain point? Let me know in the comments — I'm always looking for feature ideas.