My Analytics Agent Stopped Repeating Its Wrong Guesses With Hindsight

viraj gubbewad — Sat, 07 Nov 2026 06:37:01 +0000

My Analytics Agent Stopped Repeating Its Wrong Guesses With Hindsight

Every Monday morning, someone in leadership asks why a regional revenue metric dropped, and an analyst spends two hours checking the exact same five tables in the exact same order. When we first wired an LLM tool-calling agent to automate root-cause investigations, it had an infuriating habit: it treated every single alert like its first day on the job.

If the South region experienced three consecutive pricing-tier drops over three weeks, a stateless agent would still burn time querying holiday calendars, competitor promo trackers, and warehouse stockout logs before checking pricing. Worse, if the Monday morning ETL sync stalled at 02:00 UTC, the agent would panic over incomplete tables and report a fake revenue collapse. And no matter how many times the Finance Head asked for a single-line summary with the exact dollar variance, the next session returned a multi-paragraph essay.

To fix this, we built WhyDesk—an autonomous metric investigation system powered by FastAPI, Groq tool-calling, and the Hindsight GitHub repository. Instead of treating memory as a passive chat log, we used Vectorize agent memory to dynamically reorder diagnostic tool execution, skip low-probability checks, flag flaky data pipelines before trusting raw numbers, and enforce stakeholder-specific output formatting.

What the System Does and How It Hangs Together

WhyDesk sits between business stakeholders (Finance Head and Senior Analyst) and five deterministic diagnostic tools that inspect weekly regional performance across South, North, West, and East:

check_holiday_calendar: Audits regional banking holidays and seasonal B2B checkout dips.
check_competitor_promos: Scans market intelligence feeds for aggressive rival discounting.
check_inventory_stockouts: Inspects warehouse SKU depletion and unfulfilled hardware bundle backlogs.
check_pricing_changes: Audits contract repricing, partner promo codes, discount overrides, and unit ACV shifts.
check_etl_pipeline: Verifies warehouse sync completion, job timestamps, and missing row counts.

When an investigation request arrives at POST /api/investigate, WhyDesk does not immediately hand the five tool schemas to the LLM. Instead, it executes a four-stage loop built around the primitives described in what Hindsight does:

User Query + Stakeholder Role
       │
       ▼
1. Hindsight Recall (world, experience, observation) + Reflect
       │
       ▼
2. Memory-Driven Tool Planner (Groq qwen/qwen3-32b -> openai/gpt-oss-120b -> llama-3.3-70b-versatile)
   [Reorders diagnostic tools & enables early stopping once root cause + clean ETL are verified]
       │
       ▼
3. Deterministic Tool Execution + Numeric Grounding Verification
       │
       ▼
4. Synchronous Hindsight Retain (Confirmed cause, ruled-out theories, pipeline reliability, format rule)

Core Technical Story: Turning Memory Into Tool Execution Order

Most agent implementations stuff retrieved text chunks into a prompt and hope the model changes its behavior. In data analysis, passive context is not enough. We needed institutional memory to govern which diagnostic tools run first, which tools get skipped, and how the final answer is shaped.

We organized our Hindsight memory bank (whydesk-bank) around three memory types and configured the bank disposition with high skepticism and literalism (skepticism: 4, literalism: 4, empathy: 2) so confirmed metric verdicts always outweigh unverified guesses:

World facts (world): Confirmed historical root causes and explicitly falsified hypotheses for a region or product line (for example, "In W10 and W12 for the South region, revenue drops were caused by pricing configuration changes; holiday and competitor promo hypotheses were ruled out").
Operational experiences (experience): Lessons learned about data source reliability (for example, "Monday revenue drops in W11 were false alarms caused by an etl_sales_sync warehouse stall; always verify ETL pipeline status before trusting Monday numbers").
Stakeholder observations (observation): Reporting rules per persona (for example, "Stakeholder 'Finance Head' strictly requires a 1-line summary with the exact dollar impact").

1. Pre-Investigation Recall and Reflection (`backend/memory.py`)

Before calling Groq, WhyDeskMemory.recall_context() queries Hindsight across all three memory types and runs reflect() to synthesize a tactical plan for the current investigation:

def recall_context(self, question: str, region: str, stakeholder: str) -> dict:
    query = (
        f"Past metric investigations for region {region}, reliable vs unreliable data sources, "
        f"ruled out causes, and formatting preferences for stakeholder {stakeholder}. "
        f"Question: {question}"
    )
    recall_resp = self.client.recall(
        bank_id=self.bank_id,
        query=query,
        types=["world", "experience", "observation"],
        budget="mid",
        max_tokens=2048,
    )

    memories = [
        {
            "id": getattr(r, "id", ""),
            "text": getattr(r, "text", ""),
            "type": getattr(r, "type", "world"),
            "context": getattr(r, "context", ""),
        }
        for r in (recall_resp.results or [])
    ]

    reflect_resp = self.client.reflect(
        bank_id=self.bank_id,
        query=(
            f"Based on past investigations in {region} and stakeholder {stakeholder}: "
            f"1) Which diagnostic tools should run first? "
            f"2) Which hypotheses were repeatedly ruled out? "
            f"3) Any data pipeline traps to check first? "
            f"4) How should the final response for {stakeholder} be formatted?"
        ),
        budget="low",
    )

    return {
        "query": query,
        "memories": memories,
        "reflection": getattr(reflect_resp, "text", ""),
    }

2. Memory-Driven Tool Ordering and Early Stopping (`backend/agent.py`)

Rather than hardcoding regional rules in Python, backend/agent.py derives the diagnostic execution queue strictly from the recalled memory text and reflection output. On a cold start (0 recalled memories) or when the user switches the UI toggle to Memory off, the agent runs all five tools in the default baseline sequence (check_holiday_calendar -> check_competitor_promos -> check_inventory_stockouts -> check_pricing_changes -> check_etl_pipeline).

When Memory on is active and Hindsight returns prior investigations, the planner scores each diagnostic tool based on recalled pipeline warnings, historical regional root causes, and previously ruled-out checks:

def derive_tool_order_from_memory(memories: list[dict], reflection: str) -> list[str]:
    combined = " ".join(m.get("text", "") for m in memories) + " " + (reflection or "")
    text = combined.lower()

    if not text.strip():
        return list(DEFAULT_TOOL_ORDER)

    priority_scores = {tool: 0 for tool in DEFAULT_TOOL_ORDER}

    if "etl" in text or "pipeline" in text or "sync" in text or "lag" in text:
        priority_scores["check_etl_pipeline"] += 30
    if "pricing" in text or "discount" in text or "promo code" in text or "repricing" in text:
        priority_scores["check_pricing_changes"] += 20
    if "stockout" in text or "inventory" in text:
        priority_scores["check_inventory_stockouts"] += 15
    if "ruled out" in text and "holiday" in text:
        priority_scores["check_holiday_calendar"] -= 10
    if "ruled out" in text and "competitor" in text:
        priority_scores["check_competitor_promos"] -= 10

    return sorted(
        DEFAULT_TOOL_ORDER,
        key=lambda t: (-priority_scores[t], DEFAULT_TOOL_ORDER.index(t)),
    )

Once the prioritized tools run, the agent checks whether the warehouse ETL sync is verified clean and a primary anomaly (status == "anomaly_detected") has been isolated. When both conditions are met under memory-guided execution, the loop stops early and skips the remaining low-probability checks.

3. Synchronous Post-Investigation Retention (`backend/memory.py`)

In an interactive analytics workflow, an analyst often runs follow-up investigations seconds apart. If memory ingestion happens asynchronously in the background, the very next query might miss the finding that was just confirmed. We set retain_async=False on every client.retain() call so verdicts, pipeline notes, and stakeholder preferences are immediately searchable:

self.client.retain(
    bank_id=self.bank_id,
    content=(
        f"In {week} for the {region} region, the confirmed root cause of the revenue anomaly "
        f"was {confirmed_cause} (Impact: {dollar_impact}). "
        f"Ruled-out hypotheses: {', '.join(ruled_out) if ruled_out else 'None'}."
    ),
    context=f"investigation_verdict:{region}",
    timestamp=datetime.now(timezone.utc),
    metadata={"region": region, "week": week, "verdict": "confirmed"},
    retain_async=False,
)

Concrete Results: 5-Run Progression and Memory On vs. Off

To verify that every behavioral change comes strictly from stored memories, we ran a five-investigation progression starting from an empty memory bank:

Run	Scenario & Stakeholder	Recalled Memories	Tools Executed (In Order)	Tools Count	Result & Behavior
Run 1	`W10 South` (-18%) — Finance Head	`0` (Cold Start)	`holiday` -> `competitor` -> `inventory` -> `pricing` -> `etl`	`5 / 5`	Finds 20% discount tier (-$42,500). Retains verdict + Finance Head's 1-line rule.
Run 2	`W11-Mon South` (-64%) — Senior Analyst	`2`	`pricing` -> `etl`	`2 / 5`	Discovers `etl_sales_sync` stalled at 02:00 UTC ($0 real loss). Retains ETL lag warning.
Run 3	`W11 North` (-24%) — Senior Analyst	`3`	`etl` -> `pricing` -> `inventory` -> `competitor` -> `holiday`	`5 / 5`	Confirms `ENT-GATEWAY-02` stockout (-$31,200); explicitly rules out competitor promo & holiday.
Run 4	`W12 South` (-15%) — Senior Analyst	`5`	`etl` -> `pricing`	`2 / 5`	Verifies ETL clean first, then catches reactivated `SOUTH-Q1-25` promo (-$36,800).
Run 5 (Memory ON)	`W13 South` (-19%) — Finance Head	`6`	`check_etl_pipeline` -> `check_pricing_changes`	`2 / 5`	Skips `holiday`, `competitor`, and `inventory`. Returns strict 1-line executive summary.
Run 5 (Memory OFF)	`W13 South` (-19%) — Finance Head	`0` (Bypassed)	`holiday` -> `competitor` -> `inventory` -> `pricing` -> `etl`	`5 / 5`	Reverts to running all 5 checks blindly and returns a generic multi-paragraph breakdown.

On Run 5 with Memory ON, WhyDesk recalls the two prior South pricing anomalies, the W11 Monday ETL stall warning, and the Finance Head formatting preference. It checks check_etl_pipeline first, checks check_pricing_changes second, skips the three ruled-out checks, and outputs:

W13 South revenue dropped 19% (-$47,900 WoW) due to a regional reseller contract repricing that reduced unit ACV; ETL sync verified clean.

Lessons Learned

Synchronous retention (retain_async=False) eliminates race conditions in sequential debugging: When analysts run back-to-back diagnostic queries, waiting for synchronous indexing guarantees that a pipeline warning discovered in one turn immediately protects the next turn.
Retaining falsified hypotheses is just as valuable as retaining root causes: Explicitly recording that competitor promos and holidays were ruled out in a region gives the planner the signal it needs to deprioritize noisy checks.
LLM tool-calling needs deterministic schema repair and numeric validation: Fast inference models occasionally emit malformed tool call payloads or round currency figures in prose. Adding a fallback chain (qwen/qwen3-32b -> openai/gpt-oss-120b -> llama-3.3-70b-versatile) and verifying that every dollar figure in the final response matches the raw tool output kept financial reporting accurate.
Separate data-pipeline health from domain anomalies: Storing ETL stalls as experience memories forced the agent to verify data freshness before blaming pricing or sales execution.

DEV Community: viraj gubbewad

My Analytics Agent Stopped Repeating Its Wrong Guesses With Hindsight

My Analytics Agent Stopped Repeating Its Wrong Guesses With Hindsight

What the System Does and How It Hangs Together

Core Technical Story: Turning Memory Into Tool Execution Order

1. Pre-Investigation Recall and Reflection (backend/memory.py)

2. Memory-Driven Tool Ordering and Early Stopping (backend/agent.py)

3. Synchronous Post-Investigation Retention (backend/memory.py)

Concrete Results: 5-Run Progression and Memory On vs. Off

Lessons Learned

1. Pre-Investigation Recall and Reflection (`backend/memory.py`)

2. Memory-Driven Tool Ordering and Early Stopping (`backend/agent.py`)

3. Synchronous Post-Investigation Retention (`backend/memory.py`)