How We Built Real Decision Intelligence with a Retention Agent

How We Built Real Decision Intelligence with a Retention Agent
AI is everywhere. But most AI systems do the same thing for every user — a generic recommendation, a one-size-fits-all response. For our Microsoft Hackathon project, we set out to build something different: a system that actually thinks before it responds.
That was the AI/ML engineering challenge — designing a RetentionAgent that doesn't just react to churn signals, but reasons through them.

The Core Responsibility
The RetentionAgent is the brain of our Cross-Lifecycle Customer Intelligence system. When a customer shows signs of churning, the agent doesn't fire a generic "we miss you" email. It retrieves everything the system knows about that specific customer — how they browsed, what signals they showed during conversion, what mental models apply to their profile — and generates a strategy built specifically for them.
Designing this agent was the central AI engineering challenge of the entire build.

How the Agent Works
The RetentionAgent follows a four-step reasoning flow every time a churn signal fires:

1. Analyze the churn signal — What is the signal? How strong is it? What does it indicate about this customer's current state?
2. Retrieve customer memory — Pull the customer's stored behavioral profile from the memory bank. This includes world facts, observations, and mental models built during the conversion phase.
3. Match with playbook — Cross-reference the customer profile against known retention strategies. A price-sensitive customer gets a different playbook than a feature-driven one.
4. Generate recommendation — Produce a personalized, evidence-grounded retention strategy with clear reasoning behind every decision. This isn't a lookup table. It's genuine reasoning over structured memory.

The Mental Model Priority Stack
One of the most important design decisions was how the agent prioritizes information. We defined a clear hierarchy:

Mental models first — high-level patterns about this customer type, grounded in research
Observations second — specific behaviors recorded from this customer's actual journey
Raw data last — individual events used only when higher-level signals are ambiguous

This hierarchy ensures the agent reasons from insight, not just noise. It also means the same raw data can produce different recommendations depending on the mental models attached to a customer profile.

The Hard Parts
Avoiding generic outputs was the biggest challenge. LLMs default to safe, average answers. Getting the agent to produce genuinely differentiated strategies required careful prompt engineering, strict directives in the memory bank, and a two-reflect flow that forced the model to critique its own first response before finalizing.
Ensuring consistency was equally critical. The same customer profile should produce the same class of recommendation every time — not a random variation based on how the prompt happened to parse. We addressed this through structured memory retrieval and explicit output constraints.
Handling ambiguity is where the system gets interesting. When a customer's signals are mixed — some hesitant, some decisive — the agent has to reason through the conflict rather than defaulting to one signal. This required the agent to explicitly acknowledge uncertainty in its output rather than hiding it behind a confident-sounding recommendation.

What It Produced
Two customers. Same churn signal. Completely different strategies.

Alice — hesitant, price-sensitive → discount-led offer, social proof messaging
Bhavik — decisive, feature-driven → feature unlock, no discounting

That difference isn't a rule someone hardcoded. It emerged from the agent reasoning over structured memory. That's what real decision intelligence looks like.

What I Learned
Generic AI is easy. Specific AI is hard. Getting a model to produce a recommendation that feels tailored to one person requires serious engineering — in the memory layer, the prompt design, and the reflection flow.
Ambiguity should be surfaced, not hidden. The best AI outputs acknowledge what they don't know. Building that honesty into the agent made the recommendations more trustworthy, not less.
Intelligence lives in the reasoning, not the answer. The final recommendation matters less than whether the agent arrived at it through sound reasoning. If the reasoning is right, the recommendation follows.

Stack
Python · Hindsight API · Groq LLM · FastAPI · React · asyncio

AI isn't just about processing data — it's about making better decisions from it. The RetentionAgent doesn't know more than a human analyst. It just reasons faster, at scale, with memory that never forgets.
Built for the Microsoft Hackathon. Happy to discuss the agent design and reflection flow in the comments.