My AI Scrum Buddy Kept Missing Action Items: Here's How I Fixed It

We run a pretty small, nimble team, and like many, we've been trying to leverage AI wherever we can to shave off those repetitive, brain-drain tasks. One area I was particularly excited about was automating our daily scrum note processing.

The Problem: Our AI Assistant Couldn't Find Its Keys

Our team's daily scrum notes are, shall we say, free-form. Some days it's a bulleted list, other days a short paragraph in Slack. I built a little internal AI assistant using the OpenAI API (initially gpt-3.5-turbo) to digest these notes. The dream was simple: reliably pull out any clear action items (who, what, by when) and blockers (what's stuck, who's affected, what's needed).

Initially, it was... well, a bit of a disaster. It was like teaching a toddler to find specific toys in a messy room. Sometimes it'd grab the right thing, other times it'd bring me a sock, or insist there were no toys at all. Accuracy was hovering around 60% on a good day. It would frequently miss obvious action items like, "I'll push the new auth branch to staging by EOD Friday," or incorrectly tag something vague as a blocker. Worse, sometimes it would hallucinate an action item that wasn't even there!

For example, if someone wrote:

"Spent most of yesterday debugging the payment gateway. Looks like a misconfiguration on the dev server. I'll need to coordinate with ops to get this sorted out by tomorrow if possible. Hopefully, we can get that migration done soon too."

The assistant might extract:

{
"action_items": [
{"owner": "(unknown)", "task": "Debug payment gateway", "deadline": "yesterday"},
{"owner": "(unknown)", "task": "Migration", "deadline": "soon"}
],
"blockers": []
}

Completely missing the actual action to coordinate with ops and failing to identify the payment gateway as a blocker, not an action I'd just done.

What I Tried (And What Didn't Quite Stick)

I spent a solid two weeks, probably 16-20 hours total, just tweaking things.

1. Simple Prompting: My initial prompt was naive: "Extract action items and blockers from the following text." As expected, gpt-3.5-turbo did a decent job on very clear, structured input, but our free-form nature threw it off.
2. More Detailed Instructions: I elaborated. "Identify action items including the owner, a clear task description, and a specific deadline. For blockers, identify the issue, who is affected, and what dependency they are waiting on." This improved things marginally, perhaps bringing accuracy to 70%, but it was still inconsistent.
3. Few-Shot Examples: I fed it 3-4 perfect examples of input notes and the desired JSON output. This was promising! For inputs similar to the examples, it performed well. But the moment our team's notes deviated from that pattern (e.g., someone used emojis, or wrote a longer story), it started to degrade again.
4. Switching Models: I upgraded to gpt-4o (using openai==1.30.0). This immediately gave a noticeable bump in reasoning and overall quality. It was better at understanding context and inferring meaning. However, the core problem of reliably extracting structured data from unstructured text for every edge case remained. It was like having a super-smart toddler, but the room was still a mess.
5. Strict JSON Output: I explicitly instructed the model to output a JSON object with specific keys (action_items as a list of dicts, blockers as a list of dicts). This was crucial for parsing the output programmatically, but it didn't solve the extraction reliability itself. Often, it would return empty lists if it wasn't confident, or try its best but still miss things that were clearly there.

What Actually Fixed It: A Little Nudge and a Stronger Net

The real breakthrough wasn't a magic prompt or a new model. It was a combination of gently nudging our team to add minimal structure to their notes and then using a robust validation step.

Here's what we did:

1. Tiny Bit of User Guidance: We asked everyone to add a simple prefix for explicit actions and blockers. So instead of just writing something, they'd add:

   • AI Note: Action - @alice will deploy the new auth service by EOD Friday.
   • AI Note: Blocker - Database migration blocked by staging environment access for Bob.

   This wasn't a strict template, just a keyword. People could still write their free-form updates around these. It felt like a small compromise for a huge gain.

2. Revised, Focused Prompt: The prompt now explicitly looked for these prefixes, essentially turning the AI's task from inference to parsing with intelligence:

   markdown
   You are an AI assistant for a development team. Your task is to extract action items and blockers from daily scrum notes. Only extract items that are explicitly prefixed with "AI Note: Action -" or "AI Note: Blocker -".

   An action item must include:

   • owner: The person responsible (e.g., @alice).
   • task: A clear description of the work.
   • deadline: When the task should be completed, if specified.

   A blocker must include:

   • issue: A brief description of the problem.
   • affected_by: Who is impacted or waiting.
   • waiting_on: What is needed to resolve it (e.g., another team, a specific deployment).

   If no such prefixes are found, return empty lists for both.

   Input notes:
   {{NOTES_HERE}}

   Output a JSON object with two top-level keys: "action_items" (a list of action item objects) and "blockers" (a list of blocker objects).

3. Pydantic Validation (The Strong Net): On the backend, after the LLM returned its JSON, I added a Python script that used Pydantic to strictly validate the structure. If the AI, despite its best efforts, messed up a key name or returned an unexpected type, Pydantic would catch it immediately. This allowed me to log the malformed outputs and flag them for manual review, ensuring that anything processed was always correctly structured.

This hybrid approach shot our reliability through the roof, hitting over 95% accuracy for correctly extracted and structured items. The LLM's job became much clearer, and the light structuring meant less guessing. It's not truly free-form anymore, but the compromise was so minor, and the productivity gain so significant, that our team adopted it without a second thought. Sometimes, the best AI solution involves a little bit of human-AI collaboration on the input side, too.