Okay, so let's talk about something that's been a real pain point in my indie dev life: keeping project trackers updated from messy, unstructured meeting notes. You know the drill – someone scribbles down 'John to check on the database migration next week' or 'Team needs to decide on UI component library' and then it's a manual, tedious process to translate that into a Jira ticket or Trello card with a clear owner and deadline.
I probably don't need to tell you how much of a time sink this is, not to mention the errors and forgotten tasks. My project tracker was always lagging, and I was perpetually annoyed. Enter the dream: an LLM that could just read our meeting notes and spit out perfectly formatted action items with owners, ready to sync. Easy, right? Spoiler: It was anything but.
The Problem: When 'Smart' Isn't Smart Enough
The core issue wasn't just extracting text; it was extracting precise, actionable, owner-attributed text from human-speak. People talk in implied contexts, vague timelines, and often don't explicitly assign ownership. A typical note might say: "Need to investigate CDN options." Who's 'need to'? Everyone? No one? My LLM was constantly struggling with this ambiguity.
What I Tried (and Why It Mostly Tanked)
My initial approach was pretty naive, I'll admit. I thought a simple prompt like, "Extract all action items and their owners from the following meeting notes," would do the trick. It didn't. I'd get back things like:
- Action: "Need to investigate CDN options." Owner: "(not specified)" (or worse, it would hallucinate an owner)
- Action: "Database migration." Owner: "John to check on." (This is an action AND an owner mixed, not clean)
Attempt #1: Just ask nicely. Failed on ambiguity, nuance, and implied context. The LLM would often miss things, or include non-actionable chatter.
Attempt #2: Force a JSON structure. This felt like progress! I'd tell it, "Output a JSON array of objects, each with an 'action' and 'owner' field." I'd get the structure, but the content quality didn't magically improve. I was pulling my hair out when gpt-4-turbo would reliably output owner: null for things like 'John to follow up on API docs' despite John being right there, clear as day. The data was structured, but still unreliable.
Attempt #3: Few-shot prompting. I gave it a few perfect examples of raw notes mapped to ideal JSON output. This helped a little, especially with consistency in formatting, but it didn't solve the core problem of interpretation and reliable extraction across a wider variety of notes. It still struggled with edge cases and creative human language.
Attempt #4: Different LLMs. I tried gpt-3.5-turbo (too flaky), llama2:7b (even flakier locally via Ollama), and finally settled on gpt-4-turbo as the best of the bunch. While gpt-4-turbo was definitely superior, it still wasn't the magic bullet I hoped for. The fundamental problem wasn't the model's intelligence, but how I was asking it to apply it.
I probably easily sunk 10-12 hours into just these prompt engineering iterations, feeling like I was getting nowhere fast.
What Actually Fixed It: The Deconstruct and Validate Strategy
The breakthrough came when I realized I was asking the LLM to do too much in one go. Interpreting ambiguous human language, deciding what's actually an action item, figuring out the owner, and then formatting it perfectly – that's a lot of distinct cognitive steps. LLMs are powerful, but they shine when you break down complex tasks into simpler, sequential steps.
Here's the multi-stage approach that finally worked, turning my ~50% reliability into 90%+:
Step 1: Broad Extraction of Candidates
First, I prompt the LLM to just identify potential action items, without worrying too much about perfect phrasing or ownership yet. The goal here is to cast a wide net.
prompt
You are an assistant tasked with identifying potential action items from meeting notes. Read the following notes and list any sentence or phrase that might represent a task, decision, or follow-up required. Do not try to assign owners or rephrase yet, just list the raw potential items. Separate each item with a newline.
Meeting Notes:
"""
[My messy meeting notes text here]
"""
This gives me a list of raw strings like "John to check on the database migration next week." or "Team needs to decide on UI component library."
Step 2: Per-Item Validation and Refinement
This is the crucial step. For each candidate item from Step 1, I make a separate, focused call to the LLM. I give it the potential item and the original full meeting notes as context. This allows the LLM to focus on just one decision point at a time.
prompt
You are an assistant validating potential action items. Given a potential action item and the full meeting notes for context, determine if it is a concrete, actionable task with a clear owner.
If it IS actionable and has a clear owner:
- Rephrase the action concisely and clearly.
- Identify the specific owner. If multiple people are mentioned, pick the primary one responsible.
- Output as a JSON object: {"action": "[concise action]", "owner": "[owner's name]"}
If it IS NOT actionable, lacks a clear owner, or is a duplicate/redundant:
- Output: {"action": null, "owner": null}
Full Meeting Notes (for context):
"""
[My messy meeting notes text here]
"""
Potential Action Item to Validate:
"""
[Single candidate item from Step 1 here]
"""
This is where the magic happens. By giving it the full context again for each individual item, it can better infer ownership, clarify vague actions, or correctly identify non-actionable statements.
Step 3: Final Aggregation
Finally, I just filter out any {"action": null, "owner": null} results from Step 2 and consolidate the valid JSON objects into a single array. This gives me a clean, structured list of action items.
The Takeaway
This multi-stage approach bumped the processing time from around 3 seconds for a single gpt-4-turbo call to about 7-8 seconds (due to multiple API calls for potentially many candidate items), but the jump in reliability was absolutely worth it. It’s not about making the LLM 'smarter,' but about guiding it through a logical reasoning process. Deconstruct complex problems, validate decisions iteratively, and provide context where it's most needed. Sometimes, the 'simple' solution is actually a series of simple steps, not one giant leap.
I'm now much closer to my dream of fully automated project tracking. It's still not 100% perfect, but it's a hell of a lot better than my human brain trying to decipher scribbled notes at 5 PM on a Friday. And that, my friends, is a win in my book. 💪
Top comments (0)