How I Architected a Serverless AI Tool for Podcasts

I was struggling with the post-production grind. After recording a 45-minute episode, the real work began: writing show notes, brainstorming titles, and pulling out SEO keywords. I'd spend two hours on admin work for every hour of recording. I tried stitching together different AI APIs and prompt templates in a standard Node.js backend, but managing the latency and context windows became a nightmare. The payloads were huge, the functions were timing out, and the outputs were inconsistent. So I decided to build something.

The Problem with Standard AI Wrappers

The problem with building AI tools for podcasters isn't just the AI part; it's the orchestration and infrastructure. Creators need highly structured outputs—accurate timestamps, concise summaries, and catchy titles. If you just pass a 5,000-word transcript to a standard LLM API, it often hallucinates timestamps and forgets the core theme by the end of the text.

Furthermore, running heavy inference on a standard serverless function often leads to cold starts and HTTP timeouts. When you're dealing with audio transcripts, the JSON payload is large, and the processing time can easily exceed standard 10-second serverless limits. The cost of ignoring this architecture problem is a clunky user experience where creators stare at a loading spinner for a minute, only to get half-baked show notes that require heavy manual editing.

Architecting for the Edge

I needed an architecture that could handle large text payloads, maintain context, and respond quickly without managing bare-metal servers or dealing with cold starts. I chose Cloudflare Workers for the compute layer, specifically leveraging their AI inference capabilities at the edge.

Decision 1: Edge Inference

By running the models close to the user, we reduce network latency and avoid the traditional cold-start penalties of serverless functions. Cloudflare's @cf/meta/llama-3-8b-instruct model is optimized for this environment, allowing us to process large transcript chunks without routing traffic back to a centralized origin server.

Decision 2: Strict Prompt Engineering

The secret to reliable show notes isn't just using a bigger model; it's strict system prompts. I had to force the model to output Markdown with specific headers so the frontend could render them cleanly without complex regex parsing.

Here is the core worker script that handles the show notes generation pipeline:

// Show notes generation pipeline (Cloudflare Workers + AI)
export default {
  async fetch(request, env) {
    const { transcript } = await request.json();

    // Call AI model for structured show notes
    const response = await env.AI.run('@cf/meta/llama-3-8b-instruct', {
      messages: [
        {
          role: 'system',
          content: `Convert podcast transcripts to show notes:
## Episode Summary (2-3 sentences)
## Key Takeaways (5-7 bullet points)
## Timestamps (estimate from content flow)
## Resources Mentioned`
        },
        { role: 'user', content: transcript }
      ],
      max_tokens: 800,
    });

    return new Response(JSON.stringify(response), {
      headers: { 'Content-Type': 'application/json' },
    });
  },
};

Decision 3: Contextual Title Generation

Titles require a completely different approach. They need to be punchy and drive curiosity, not just summarize the content. I built a specific prompt chain for this that separates the tone from the topic.

// Prompt engineering for podcast title generation
function buildTitlePrompt(topic, tone, count) {
  return {
    system: `You are an expert podcast producer. Generate compelling titles.
Rules:
- 5-10 words each
- Use curiosity gaps, numbers, or strong hooks
- Match the tone: ${tone}
- No clickbait that overpromises
- Format as numbered list`,
    user: `Episode topic: ${topic}

Generate ${count} title options that make someone want to click play.`
  };
}

Trade-offs and Constraints

Edge AI is fast, but context windows are still a physical constraint. For episodes longer than 4,000 tokens, I had to implement a map-reduce chunking strategy. The worker splits the transcript into smaller segments, summarizes each chunk individually, and then passes those summaries to a final prompt to generate the complete show notes. It adds a bit of latency, but it completely eliminates the hallucination issues we saw with full-transcript processing.

Practical Results and Lessons Learned

What worked exceptionally well was the edge deployment. It completely eliminated the timeout issues I was facing with traditional serverless platforms. The structured prompts yielded surprisingly consistent Markdown formatting, which meant I didn't have to write fragile parsing logic on the frontend.

What surprised me was how much the tone parameter in the title generator changed the output. Passing "inspirational" versus "analytical" to the system prompt drastically shifted the vocabulary the model chose, making the tool actually useful for different podcast niches.

What would I change? Initially, I tried to do everything in a single massive prompt. Splitting the title generation and show notes generation into distinct, focused API calls improved reliability and made debugging prompt failures much easier. I also learned that giving the model an explicit "do not" list (like "No clickbait that overpromises") is just as important as telling it what to do.

Try the Tool

If you're building podcast tools or just want to see how these edge prompts perform in the wild, I've packaged the core title generation logic into a free tool. You can try the Podcast Title Generator over at PodCrisp to see the exact architecture and prompt structures in action.

---

Full disclosure: I'm the developer of PodCrisp. I built this because I needed it myself.