I Built an AI Meeting Summarizer with Spring AI — Here's How (and Why)

🚀 The Problem

Wait, what did we decide in that meeting?

We've all been there. You have a productive meeting, everyone agrees on action items, and by the next day, nobody remembers who was supposed to do what.

I wanted to fix this. So I built a tool that:

• 🗣️ Identifies speakers and what they said
• 📝 Generates a professional meeting title
• ✅ Extracts action items with priorities
• ❓ Highlights unresolved issues
• 🚀 Lists clear next steps

The best part? It works with any audio file — upload an MP3, and get a structured summary in seconds.

And I built it entirely with Java and Spring AI.

🛠️ The Tech Stack

Component           Technology          Purpose
Framework           Spring Boot 3.4.x   Application backbone
AI Integration      Spring AI 1.0.3     Unified AI abstraction
Transcription       OpenAI Whisper      Speech-to-text
Speaker Detection   AssemblyAI          Speaker diarization
Summary Generation  GPT-4o-mini         LLM-powered summarization

Why this stack? Spring AI is the killer feature. It provides a clean, Spring-native way to interact with AI models. No boilerplate HTTP clients. No manual JSON parsing.

🧠 How It Works
Step 1: Transcribe with Whisper
Spring AI provides OpenAiAudioTranscriptionModel. I inject it directly into my service:

@Service
public class TranscriptionService {

    private final OpenAiAudioTranscriptionModel transcriptionModel;

    @Autowired
    public TranscriptionService(OpenAiAudioTranscriptionModel transcriptionModel) {
        this.transcriptionModel = transcriptionModel;
    }

    public String transcribe(File audioFile) {
        AudioTranscriptionPrompt prompt = new AudioTranscriptionPrompt(
            new FileSystemResource(audioFile),
            AudioTranscriptionPrompt.builder()
                .withModel("whisper-1")
                .withResponseFormat(TranscriptResponseFormat.JSON)
                .withLanguage("en")
                .build()
        );

        AudioTranscriptionResponse response = transcriptionModel.call(prompt);
        return response.getResult().getOutput();
    }
}

Step 2: Detect Speakers with AssemblyAI
Spring AI doesn't have built-in speaker diarization, so I integrated AssemblyAI for this.

@Service
public class SpeakerDetectionService {

    private String uploadAudio(File audioFile) throws IOException {
        // Upload to AssemblyAI
        // Returns upload_url
    }

    private String submitTranscription(String audioUrl) {
        // Submit with speaker_labels: true
        // Returns transcript ID
    }

    private List<SpeakerUtterance> getTranscriptWithSpeakers(String transcriptId) {
        // Poll until status is "completed"
        // Returns utterances with speaker labels
    }
}

Why this matters: Speaker detection turns a plain transcript into an actionable meeting summary. Without it, you can't assign tasks to specific people.

Step 3: Generate the Summary with GPT
This is where the magic happens. I use Spring AI's ChatClient with a custom prompt:

@Service
public class SummaryService {

    private final ChatClient chatClient;

    public String generateSummary(String transcript, List<SpeakerUtterance> utterances) {
        String prompt = buildPrompt(transcript, utterances);

        return chatClient.prompt()
            .user(prompt)
            .call()
            .content();
    }

    private String buildPrompt(String transcript, List<SpeakerUtterance> utterances) {
        return """
            You are a professional meeting summarizer with 10 years of experience.

            IMPORTANT: First, generate a concise, professional meeting title (max 5 words).

            Provide your response in this exact format:

            # MEETING TITLE
            [Your concise title here, max 5 words]

            # MEETING OVERVIEW
            One paragraph summary (max 50 words)

            # KEY DECISIONS
            - Decision 1
            - Decision 2

            # ACTION ITEMS
            | Task | Assignee | Deadline | Priority |
            |------|----------|----------|----------|
            | [Task] | [Person] | [Date] | [HIGH/MEDIUM/LOW] |

            # UNRESOLVED ISSUES
            - Issue 1

            # NEXT STEPS
            - Step 1

            Rules:
            1. ONLY use information from the transcript
            2. If a deadline or assignee is missing, write "Not specified"
            3. Keep action items to a maximum of 5
            4. Be concise and actionable
            """.formatted(transcript, speakerInfo);
    }
}

💡 Pro tip: Never hardcode your prompt directly in the service. Store it as a classpath resource and load it with @Value. This way you can tweak the output without a code review.

📡 The Controller
The REST endpoint ties everything together:

@PostMapping("/process")
public ResponseEntity<Map<String, Object>> processMeeting(
        @RequestParam("audio") MultipartFile audioFile) {

    // 1. Save uploaded file to temp location
    File tempFile = saveToTemp(audioFile);

    // 2. Transcribe with Whisper
    String transcript = transcriptionService.transcribe(tempFile);

    // 3. Detect speakers with AssemblyAI
    List<SpeakerUtterance> utterances = speakerDetectionService.detectSpeakers(tempFile);

    // 4. Generate summary with GPT
    String summary = summaryService.generateSummary(transcript, utterances);

    // 5. Parse tasks from the summary
    List<Map<String, String>> tasks = parseTasks(summary);

    // 6. Return structured JSON
    return ResponseEntity.ok(buildResponse(summary, utterances, tasks));
}

📊 What the Output Looks Like
Here's a real response from a test meeting:

{
  "meetingTitle": "Student Attendance Concerns",
  "meetingOverview": "The meeting addressed a decline in student attendance on Fridays...",
  "keyDecisions": [
    "Host a pancake breakfast to encourage attendance.",
    "Put up health posters to address seasonal illness."
  ],
  "tasks": [
    {
      "description": "Organize pancake breakfast",
      "assignee": "Not specified",
      "deadline": "Not specified",
      "priority": "HIGH"
    },
    {
      "description": "Create and display health posters",
      "assignee": "Not specified",
      "deadline": "Not specified",
      "priority": "MEDIUM"
    }
  ],
  "speakerCount": 3
}

🚀 Try It Yourself
The API is live:
meeting-summarizer

💡 What I Learned
1. Spring AI Eliminates Boilerplate
I didn't write a single HTTP client or JSON parser. The ChatClient abstraction handles everything. You can swap the LLM provider by changing one dependency and one properties line.

2. Prompt Engineering Is the Real Skill
The output quality depends entirely on the prompt. I iterated on my prompt more than on any code component. A good prompt with system context makes the difference between generic and actionable output.

3. File-Based Processing Has Privacy Benefits
Unlike bot-based tools (like Fireflies or Otter), my tool processes uploaded files — no one needs to join your meetings. There's growing demand for privacy-first meeting assistants.

4. Speaker Detection Changes Everything
Without speaker labels, you just get a wall of text. With them, you get clear ownership of action items.

🔮 What's Next

• Vector Database: RAG for semantic search across past meetings
• Asana/Jira Integration: Auto-create tasks from action items
• Email Delivery: Automatically send summaries to attendees
• Better Assignee Detection: Infer who each task belongs to from speech

📚 Resources
Spring AI Documentation
AssemblyAI Speaker Diarization
OpenAI Whisper API
GitHub Repository

💬 Let's Connect
If you're building AI tools with Spring AI, I'd love to hear about your experience. Drop a comment.

Thanks for reading! 🚀