Build Your Own AI-Powered Meeting Notes System This Weekend

Turn hours of manual note-taking into an automated workflow with Python, Whisper, and Claude

---

We've all been there: you finish a meeting, realize you should have taken better notes, and spend the next 20 minutes trying to reconstruct what was discussed from fragmented memories and a few bullet points.

This weekend, let's fix that permanently. We're going to build an AI-powered meeting notes system that:

1. Transcribes audio recordings using OpenAI's Whisper
2. Generates structured summaries with action items using Claude
3. Saves everything to Markdown files for your second brain

Total time: 2-3 hours. Difficulty: Beginner-friendly.

What You'll Need

• Python 3.9+
• An Anthropic API key (for Claude)
• Basic terminal knowledge
• Audio files of meetings (or a microphone to record new ones)

Project Structure

meeting-notes/
├── transcribe.py
├── summarize.py
├── main.py
├── recordings/
├── transcripts/
└── summaries/

Step 1: Set Up the Environment

mkdir meeting-notes && cd meeting-notes
python -m venv venv
source venv/bin/activate  # Windows: venv\Scripts\activate

pip install openai-whisper anthropic python-dotenv

Create a .env file:

ANTHROPIC_API_KEY=your-key-here

Step 2: Build the Transcription Module

Whisper runs locally, so your audio never leaves your machine. Create transcribe.py:

import whisper
from pathlib import Path

def transcribe_audio(audio_path: str, model_size: str = "base") -> str:
    """
    Transcribe audio file using Whisper.
    Model sizes: tiny, base, small, medium, large
    Larger = more accurate but slower
    """
    model = whisper.load_model(model_size)
    result = model.transcribe(audio_path)
    return result["text"]

def save_transcript(text: str, output_path: str) -> None:
    """Save transcript to file."""
    Path(output_path).parent.mkdir(parents=True, exist_ok=True)
    with open(output_path, "w") as f:
        f.write(text)

if __name__ == "__main__":
    # Quick test
    import sys
    if len(sys.argv) > 1:
        text = transcribe_audio(sys.argv[1])
        print(text[:500] + "..." if len(text) > 500 else text)

Pro tip: Start with base model for speed during development. Switch to medium or large for production quality.

Step 3: Build the AI Summary Module

Now the magic happens. Create summarize.py:

import anthropic
from dotenv import load_dotenv

load_dotenv()

SUMMARY_PROMPT = """You are an expert meeting analyst. Given a transcript, create a structured summary with:

## Meeting Summary
A 2-3 sentence overview of what was discussed.

## Key Points
- Bullet points of main topics covered

## Decisions Made
- Any decisions that were finalized

## Action Items
- [ ] Task description (@owner if mentioned) - deadline if specified

## Questions/Open Items
- Unresolved questions or items needing follow-up

Be concise but thorough. Extract every actionable item mentioned.

TRANSCRIPT:
{transcript}
"""

def summarize_transcript(transcript: str) -> str:
    """Generate structured meeting notes from transcript."""
    client = anthropic.Anthropic()

    message = client.messages.create(
        model="claude-3-5-sonnet-20241022",
        max_tokens=2000,
        messages=[
            {
                "role": "user",
                "content": SUMMARY_PROMPT.format(transcript=transcript)
            }
        ]
    )

    return message.content[0].text

def save_summary(summary: str, output_path: str, metadata: dict = None) -> None:
    """Save summary with optional YAML frontmatter."""
    from pathlib import Path
    from datetime import datetime

    Path(output_path).parent.mkdir(parents=True, exist_ok=True)

    frontmatter = f"""---
date: {datetime.now().strftime('%Y-%m-%d')}
type: meeting-notes
"""
    if metadata:
        for key, value in metadata.items():
            frontmatter += f"{key}: {value}\n"
    frontmatter += "---\n\n"

    with open(output_path, "w") as f:
        f.write(frontmatter + summary)

Step 4: Wire It All Together

Create main.py:

#!/usr/bin/env python3
"""
AI Meeting Notes Generator
Usage: python main.py path/to/recording.mp3
"""

import sys
from pathlib import Path
from datetime import datetime

from transcribe import transcribe_audio, save_transcript
from summarize import summarize_transcript, save_summary

def process_meeting(audio_path: str, title: str = None) -> str:
    """Full pipeline: audio -> transcript -> summary."""

    audio_path = Path(audio_path)
    timestamp = datetime.now().strftime("%Y-%m-%d_%H%M")
    title = title or audio_path.stem

    print(f"Processing: {audio_path.name}")

    # Step 1: Transcribe
    print("Transcribing audio (this may take a few minutes)...")
    transcript = transcribe_audio(str(audio_path), model_size="base")

    transcript_path = f"transcripts/{timestamp}_{title}.txt"
    save_transcript(transcript, transcript_path)
    print(f"Transcript saved: {transcript_path}")

    # Step 2: Summarize
    print("Generating AI summary...")
    summary = summarize_transcript(transcript)

    summary_path = f"summaries/{timestamp}_{title}.md"
    save_summary(summary, summary_path, {"title": title})
    print(f"Summary saved: {summary_path}")

    return summary_path

if __name__ == "__main__":
    if len(sys.argv) < 2:
        print("Usage: python main.py <audio_file> [title]")
        sys.exit(1)

    audio_file = sys.argv[1]
    title = sys.argv[2] if len(sys.argv) > 2 else None

    output = process_meeting(audio_file, title)
    print(f"\nDone! Open {output} to see your notes.")

Step 5: Take It Further

Here are upgrades for next weekend:

Add a Watchdog: Automatically process new recordings:

from watchdog.observers import Observer
from watchdog.events import FileSystemEventHandler

class RecordingHandler(FileSystemEventHandler):
    def on_created(self, event):
        if event.src_path.endswith(('.mp3', '.m4a', '.wav')):
            process_meeting(event.src_path)

Build a Simple Web UI: Use Streamlit for drag-and-drop uploads:

import streamlit as st

uploaded_file = st.file_uploader("Drop your meeting recording")
if uploaded_file:
    # Process and display summary in real-time

Send to Your Task Manager: Auto-create Todoist/Linear tasks from action items using their APIs.

The Real Value

This project isn't just about transcription—tools like Otter.ai already do that. The value is in ownership and customization:

• Your audio stays local (Whisper runs on your machine)
• Summaries are formatted YOUR way
• Everything saves to Markdown for Obsidian/Notion/whatever
• You can tune the prompt for your meeting style
• No monthly subscription

I've been running this for three months. My action item capture rate went from ~60% to nearly 100%. More importantly, I actually attend meetings now instead of frantically scribbling notes.

Key Takeaways

1. Local-first AI is practical now. Whisper runs fine on a decent laptop.
2. Chain simple tools for powerful workflows. Transcription + LLM + file system = complete solution.
3. Own your data. Build systems that work with your existing tools, not proprietary silos.

The code above is a starting point. Fork it, break it, make it yours. That's the whole point of weekend projects.

---

Got a weekend project you've built? Drop it in the comments—I'd love to see what you're automating.