My LLM Stand-up Summarizer: It Hallucinates Decisions & Who Said Them

Okay, so I had this brilliant idea, right? Our daily stand-ups are... fine. Necessary, but sometimes a bit long. Everyone gives their updates, we make some quick decisions, and then an hour later, someone asks, "Wait, who was going to check on that API endpoint?" My developer brain immediately screamed, "Automation! LLMs!"

The vision was clear: record the stand-up, feed it to an LLM, and boom – a perfectly concise summary delivered to Slack, attributing actions to people and highlighting key decisions. No more scribbling notes, no more forgotten commitments. It sounded like pure productivity bliss.

The Problem: When AI Gets Creative with Reality

I wired up a simple system using the openai Python library v1.1.1. First pass: Whisper for transcription. That part was actually pretty solid. Second pass: feed the raw transcript to GPT-3.5-turbo (and later, GPT-4-turbo, thinking more brainpower would fix it) with a prompt like, "Summarize this stand-up transcript, including who said what and any decisions made."

Initial results were... fascinating. The summaries were concise, conversational, and totally plausible. But also, often completely wrong on the specifics. My LLM became a tiny, confident fabulist.

"Sarah confirmed the database schema change would be deployed today."

Me, reading the transcript: Sarah said, "I'll look into the database schema change today, but I foresee some blockers."

That's not a confirmation; that's an investigation with a warning! Or even worse: "John decided we'd use the new caching layer." John actually said, "Maybe we should explore the new caching layer?" The LLM wasn't summarizing; it was interpreting and completing things that weren't fully decided, assigning definitive statements to people who made tentative ones. Speaker attribution was a mess, and key decisions were either invented or completely misconstrued. It felt like playing a game of 'telephone' with a very opinionated robot.

What I Tried (and mostly failed at)

My first reaction was to beat the LLM into submission with better prompts. I tried:

1. More verbose instructions: "STRICTLY attribute statements to the correct speaker. ONLY list confirmed decisions. DO NOT invent details." This just made the summaries longer and still subtly wrong. The LLM would stick in phrases like "According to John..." but then still misrepresent John's statement.
2. Role-playing: "You are a diligent meeting secretary. Your goal is to accurately record facts, not interpret them." Again, marginal improvement, but the core issue of hallucinating confirmed decisions persisted.
3. Model hopping: Upgrading to gpt-4-turbo. It was definitely smarter, less prone to grammatical errors, but still had the same fundamental flaw when pushed on explicit attribution and decision extraction. It'd invent specific values for numbers or names that weren't there, or misattribute a question as a statement.
4. Chunking: I even tried breaking the transcript into smaller, 5-minute chunks, summarizing each, and then combining those summaries with another LLM call. This was a messy disaster. Context was lost between chunks, and the final combined summary was even more disjointed and less accurate. It was like getting a jigsaw puzzle where half the pieces were from a different box.
5. JSON output: Asking for a structured JSON output like {"speaker": "John", "action": "look into API"} helped slightly with the structure, but the content within those fields could still be fictional.

I was spending hours each week debugging these summaries, effectively doing the job myself, just with extra steps involving Python and AI tokens. The dream was slowly turning into a chore.

What Actually Started to Fix It (The Hard Way)

The real breakthrough came when I stopped asking the LLM to do everything in one go. I realized the LLM is great at language generation and understanding, but it needs very specific, atomic tasks when you're after factual extraction.

My current, much more robust process involves a few steps before the LLM even sees the full, messy transcript:

1. Pre-processing for Speaker Attribution: Instead of relying on the LLM to figure out who said what from raw text, I now use a simple regex during the Whisper transcription post-processing. Most transcription services output speaker turns. I explicitly tag each sentence with its speaker before feeding it to the LLM.

   python

   Simplified example in my transcript_processor.py

   def tag_speakers(raw_transcript_lines):
   processed_lines = []
   current_speaker = "Unknown"
   for line in raw_transcript_lines:
   # Assuming lines look like: "[00:01:23] Speaker A: I did X."
   speaker_match = re.match(r'^.]\s(Speaker\s[A-Z]):\s(.)$', line)
   if speaker_match:
   current_speaker = speaker_match.group(1)
   content = speaker_match.group(2)
   processed_lines.append(f"SPEAKER:{current_speaker} {content.strip()}")
   else:
   # Handle cases where speaker isn't explicitly tagged by ASR
   processed_lines.append(f"SPEAKER:{current_speaker} {line.strip()}")
   return "\n".join(processed_lines)

   So the LLM now sees: I'll look into the database schema change today... This explicit tagging dramatically improved attribution accuracy. The LLM still needs to parse it, but the information is front-and-center, not implied.

2. Two-Pass Extraction for Decisions: This was the game-changer for reducing decision hallucination. I stopped asking for a summary with decisions and started asking for explicit decision extraction.

*   **Pass 1 (Extraction):** I prompt the LLM to first extract *all potential action items or decisions* in a structured JSON format, explicitly stating: "If it's not a confirmed decision, mark its status as 'potential' or 'discussion'. Include the speaker who mentioned it and the exact proposed action." 

    [
      {"speaker": "Sarah", "action": "look into database schema change", "status": "potential"},
      {"speaker": "Mark", "action": "deploy API hotfix", "status": "confirmed"}
    ]


*   **Pass 2 (Verification/Refinement):** I then take that JSON output, and in a *separate LLM call*, ask it to review and refine. "Review these potential actions. Based on the full transcript (provided again), confirm if the 'status' is correct. If 'potential', suggest what needs to happen to make it 'confirmed'." This second pass acts as a self-correction mechanism, forcing the LLM to cross-reference its own initial extraction with the original source. 

This two-step process, combined with explicit speaker tagging, reduced my decision hallucination rate from about ~30% in early attempts to under 5%. It took about two weeks of evenings and weekends to get this working reliably, using Python 3.10 and openai v1.1.1. It's not perfect, but it's usable.

My takeaway? LLMs are powerful, but they're not magic. When you need factual accuracy, especially around attribution and specific decisions, you have to break down the problem into smaller, unambiguous tasks. Give them structured input, ask for structured output, and don't be afraid to make them do a second pass to verify their own work. It's more code, more tokens, but way less headache than explaining to your team that the AI made up Sarah's commitments again.

Now, about automating the deployment of this... that's a story for another time.