Hindsight sits in the middle of that loop, not bolted on the side. Every screen in the product either writes into a bank (during seeding, or after a deal closes and an autopsy runs) or reads through one (for briefs, tactic stats, the playbook, the live copilot). There's no code path that answers a question about a deal from raw JSON alone — if Hindsight is down, the app says so, it doesn't quietly fall back to reading files.
The story: A rate-limit regex that ate its own retries
To get anything into Hindsight, I first needed a corpus — forty realistic historical deals, each with a multi-call narrative, generated by an LLM and decomposed into the structured experience format the rest of the system reasons over. That generation step hammers Groq with a few thousand tokens per deal, and Groq's free/on-demand tier has two separate rate limits stacked on top of each other: a per-minute token cap and a per-day token cap. Hit the first one and the server tells you to wait a few seconds. Hit the second one and it tells you to wait the better part of half an hour.
Groq's Python SDK raises a RateLimitError on both, and it doesn't surface Retry-After as a typed field — you have to go dig it out of the error yourself. So I wrote a small helper to parse it out of the exception:
python
def _retry_after_seconds(exc: Exception, default: float) -> float:
match = re.search(r"\d+(?:\.\d+)?s", str(exc))
if match:
return float(match.group(0)[:-1]) + 1.0
return default
This looked correct, and it passed every test I threw at it — because every test I threw at it was a per-minute rate limit. Groq's message for that case reads like "try again in 28.26s", and the regex matched the 28.26s cleanly. It wasn't until a real seeding run tripped the daily quota that the bug showed up. That message reads "try again in 21m13.536s" — and my seconds-only regex still matched, just on the wrong substring. 13.536s instead of 21m13.536s.
No exception, no malformed input, just a number that was technically valid and completely wrong. The retry loop backed off fourteen seconds, hit the same daily-quota wall, backed off again, and kept doing that until MAX_GENERATION_ATTEMPTS ran out and the deal was reported as a failure — not because Groq was actually unavailable, but because my parser had quietly deleted the "21 minutes" part of a "21 minutes and 13 seconds" message.
The fix is one extra optional capture group:
python
def _retry_after_seconds(exc: Exception, default: float) -> float:
"""Groq's 429 response includes a Retry-After header; the SDK doesn't
surface it as a typed field, so parse it out of the error body/message
rather than guessing a backoff that's likely wrong relative to what the
server told us. The message takes two shapes depending on which limit
was hit: 'try again in 28.26s' (per-minute) or 'try again in
21m13.536s' (per-day) — a seconds-only regex silently drops the minutes
component on the second shape, which is the one that actually matters:
retrying a daily-quota 429 after 14s instead of the real ~21 minutes
just burns through MAX_GENERATION_ATTEMPTS doing nothing.
"""
for source in (
getattr(getattr(exc, "response", None), "headers", {}).get("retry-after")
if hasattr(exc, "response") else None,
str(exc),
):
if not source:
continue
match = re.search(r"(?:(\d+)m)?(\d+(?:\.\d+)?)s", str(source))
if match:
minutes = float(match.group(1)) if match.group(1) else 0.0
seconds = float(match.group(2))
return minutes * 60 + seconds + 1.0
return default
That call site, generate_deal(), is where this actually matters: it retries on rate limits by honoring exactly what the server said, and retries separately (with a flat 2-second backoff) on malformed JSON or off-taxonomy values, because those need a fresh sample rather than a wait:
python
def generate_deal(n: int, archetype: str, company_name: str) -> dict:
last_exc: Exception | None = None
for attempt in range(1, MAX_GENERATION_ATTEMPTS + 1):
try:
deal = llm.chat_json([...], temperature=0.95)
_validate_deal_shape(deal)
deal["deal_id"] = f"DEAL-{n:03d}"
return deal
except RateLimitError as exc:
last_exc = exc
time.sleep(_retry_after_seconds(exc, default=15.0 * attempt))
except Exception as exc: # malformed JSON, missing keys, off-taxonomy values
last_exc = exc
time.sleep(2.0)
raise RuntimeError(f"deal {n} failed after {MAX_GENERATION_ATTEMPTS} attempts: {last_exc}")
The reason this bug was so easy to miss is that it never threw. A regex that fails to match at all is loud — you get None, you get an obvious crash, you go fix it in five minutes. A regex that matches the wrong substring is quiet. It returns a plausible-looking float, the caller sleeps for a plausible-looking duration, and everything downstream behaves exactly like a system that's working, just working on the wrong timescale. The only visible symptom was a deal that took six hours to seed instead of six minutes, and a retry counter that kept resetting instead of accumulating toward anything useful. I only found it by adding a log line that printed the parsed sleep duration next to the raw exception message and staring at the mismatch.
What it looks like once memory is actually there
Once the corpus is seeded — every experience tagged with objection:price, tactic:roi_framing, competitor:rivalco, outcome:worked, and matching metadata — recall stops being a plain similarity search and starts being a filtered one. tactic_engine.py groups every experience by (objection_type, tactic_category) and computes real numbers: evidence count, win/fail split, a confidence band, and a recent-vs-older trend. None of that touches an LLM:
python
CONFIDENCE_BANDS = [
(0, 2, "insufficient"),
(3, 4, "low"),
(5, 8, "moderate"),
(9, float("inf"), "high"),
]
The LLM's job is strictly downstream of that: narrate the numbers, never invent them. In reasoning.py, a Strategy step proposes one concrete next move grounded in the evidence summary it's handed, and a Critic step checks that move against the same evidence and can only downgrade the confidence — it has no path to inflate it. If a rep faces a pricing objection with only two comparable past experiences, the system says "not enough evidence yet, use judgment" instead of manufacturing a confident-sounding tactic out of a sample size of two. That refusal to bluff is the whole point of grounding recommendations in agent memory instead of an LLM's instinct for what a good answer sounds like.
The Memory Inspector screen makes this concrete by showing the raw pipeline for any query: what got recalled, with its tags and scores, then the reflection built on top of it. Nothing on that screen is hidden behind a black-box answer — you can see exactly which three past experiences justified a given recommendation, and go check the source deal yourself.
Lessons:
A retry helper that parses server text is a parser, and needs parser-level test cases. I tested the happy path (a well-formed per-minute message) and skipped the case that actually costs you hours (a well-formed per-day message with a different shape). If a string has two documented formats, write a test for both before you write the regex.
Silent-wrong is worse than loud-wrong. A crash on bad input gets fixed immediately. A regex that matches a plausible but incorrect substring produces behavior that looks like success from the outside — the loop runs, the sleep happens, nothing raises — right up until you notice the wall-clock time doesn't add up. Anywhere you're extracting a number from unstructured text to drive a decision, treat a match on the wrong substring as at least as likely as no match at all.
Keep the numbers and the narration in separate modules, on purpose. tactic_engine.py never imports the LLM client and never makes a network call — it's pure computation over structured data, which means every number it produces is something a skeptical reader can recompute by hand from the underlying records. That boundary made a lot of other decisions easier: the Critic step in the reasoning pipeline can downgrade confidence but not invent it, because there's no path for an LLM call to touch the evidence count in the first place.
Structure the memory you retain, not just the memory you recall. Tagging every retained experience with a closed taxonomy (objection type, tactic category, outcome) rather than free text is what makes recall filterable later — "show me only pricing-objection experiences where the tactic worked" is a tag filter, not a hope that semantic similarity happens to surface the right subset. That decision was made at write time, in the retain() call, long before any query needed it.
Idempotent seeding pays for itself the first time a job dies partway through. Because each generated deal is written to disk and retained into its Hindsight bank the moment it finishes — not batched and flushed at the end — a six-hour job that got interrupted by the regex bug above still left every deal generated up to that point fully usable. Re-running the seed picked up where it left off instead of starting over. If you're building anything that ingests into an external memory store over a slow or rate-limited API, don't make success all-or-nothing.
None of this is exotic engineering. It's a rate-limit parser, a deterministic stats module, and a memory layer that keeps them honestly separated. Check our project on GitHub and explore the Hindsight repository if you want to see how the retain/recall/reflect primitives that this whole pipeline builds on are actually implemented.
[1](https://github.com/vectorize-io/hindsight)
[2](https://hindsight.vectorize.io/)
[3](https://vectorize.io/what-is-agent-memory)
[4](https://github.com/Sanjayram3269/deal-intelligence-agent)
Top comments (0)