DEV Community

Sainath Bodige
Sainath Bodige

Posted on

How Hindsight Made My Agent Remember Vendor Promises

What Makes a Memory-Based AI System Trustworthy

A memory-enabled AI system can sound much smarter than a stateless one.

It can remember previous conversations, bring old information into new decisions, and create continuity across interactions.

But persistent memory introduces a harder requirement:

The application has to know what it can safely trust.

PactTrace is a useful example.

It remembers vendor commitments using Hindsight, recalls relevant history when a new quote appears, and uses Groq to compare the current evidence against that history.

The architecture is interesting because trust is not delegated to a single model response.

Several layers work together.

The use case

Vendor promises are often spread across calls, emails, notes, and negotiations.

A vendor might promise:

  • a discount
  • a fee waiver
  • a pricing guarantee
  • a service level
  • a support commitment
  • a renewal term

Months later, the new quote may be reviewed by someone who never saw the original conversation.

PactTrace turns those historical interactions into reusable memory.

The workflow is:

Capture → Remember → Recall → Verify → Act

Layer 1: only extract supported commitments

The first trust boundary appears before anything reaches long-term memory.

PactTrace asks the model to extract only explicit or strongly supported vendor commitments.

The prompt includes:

Do not invent commitments.

If a value, condition, or deadline is not present,
return null.
Enter fullscreen mode Exit fullscreen mode

The response must also satisfy a strict Zod schema.

const extractedCommitmentsSchema = z.strictObject({
  vendor: z.string(),
  summary: z.string(),
  commitments: z.array(
    z.strictObject({
      type: z.string(),
      description: z.string(),
      value: z.string().nullable(),
      condition: z.string().nullable(),
      deadline: z.string().nullable(),
      status: z.enum([
        "promised",
        "active",
        "honored",
        "violated",
        "unknown",
      ]),
    }),
  ),
});
Enter fullscreen mode Exit fullscreen mode

That matters because long-term memory amplifies mistakes.

A hallucination that exists for one response is bad.

A hallucination stored as persistent historical truth can affect future decisions too.

So the safest memory is one that starts with conservative ingestion.

Layer 2: confirm that memory was actually retained

PactTrace stores the original interaction and extracted commitment memories in Hindsight.

It uses synchronous batch retention:

const result = await getHindsightClient().retainBatch(
  bankId,
  items,
  {
    async: false,
    signal: AbortSignal.timeout(60_000),
  },
);
Enter fullscreen mode Exit fullscreen mode

Then it validates the acknowledgement:

if (
  !result.success ||
  result.async ||
  result.items_count !== items.length
) {
  throw new Error(
    "Hindsight did not confirm the complete batch"
  );
}
Enter fullscreen mode Exit fullscreen mode

This seems like a reliability detail, but it affects user trust.

The application should not display “saved” when the provider has only queued a request or accepted part of the expected memory batch.

The UI should reflect confirmed system state.

Layer 3: attach metadata to memory

Each retained item includes metadata such as:

const metadata = {
  vendor: input.vendor,
  source: input.source,
  interaction_id: interactionId,
};
Enter fullscreen mode Exit fullscreen mode

Individual commitment memories add:

  • memory type
  • commitment type
  • status

That metadata later helps isolate vendor-specific history.

Without it, retrieval would depend too heavily on natural-language similarity.

For a business system, “sounds related” is not always enough.

Layer 4: verify recalled memories belong to the vendor

When Hindsight returns results, PactTrace checks the metadata:

if (
  !owner ||
  normalizeVendor(owner) !== normalizedVendor ||
  !memory.text.trim()
) {
  return [];
}
Enter fullscreen mode Exit fullscreen mode

This is one of the most important lines in the retrieval pipeline.

The recall prompt asks for CloudNova memories, but the application does not assume prompt wording alone guarantees isolation.

It checks the stored vendor field too.

That is an example of a broader engineering principle:

Use the model or retrieval system for intelligence, but use application logic for invariants.

Layer 5: force the analysis to remain grounded

PactTrace's comparison model receives the current quote and selected recalled memories.

It is instructed not to invent:

Use only the supplied memories for historical facts
and only the supplied quote for current evidence.

Never invent prices, percentages, dates or conditions.
Enter fullscreen mode Exit fullscreen mode

More importantly, PactTrace validates the output after generation.

if (
  !memory.text.includes(conflict.historicalCommitment) ||
  !input.quote.includes(conflict.currentEvidence)
) {
  throw new Error(
    "Comparison evidence was not grounded in the supplied sources"
  );
}
Enter fullscreen mode Exit fullscreen mode

This means the final answer cannot rely only on the model's confidence.

The quoted evidence has to exist in the actual sources.

The CloudNova case

Suppose a historical interaction says:

CloudNova agreed to waive the onboarding fee and promised a 15% renewal discount if the account exceeds 100 seats.

A later quote says:

CloudNova renewal quote for 130 seats is ₹460000 annually. The quote does not include any renewal discount.

PactTrace recalls the historical promise.

It checks the condition:

130 seats exceeds 100.

Then it checks current evidence:

the quote explicitly says no renewal discount is included.

That supports a potential conflict.

Notice what it does not do.

It does not automatically claim a legal violation.

It does not calculate financial damage.

It does not assume every historical commitment applies to every future document.

The application even sets:

financialImpact: null
Enter fullscreen mode Exit fullscreen mode

because no verified financial-impact calculator is implemented.

That is a trustworthy limitation.

“I don't know” is a feature

The onboarding-fee commitment demonstrates another important rule.

If the new renewal quote says nothing about onboarding, PactTrace does not interpret silence as contradiction.

The result is insufficient evidence.

Likewise, if a historical commitment depends on a condition and the current quote cannot establish whether that condition occurred, the system does not escalate the result.

Unknown or unmet conditions must remain low-severity insufficient evidence.

That behavior is less dramatic than always finding a problem.

It is also much more defensible.

Reliability is part of AI quality

When people discuss trustworthy AI, they often focus entirely on prompting.

PactTrace shows why production reliability matters too.

The system includes:

  • strict schemas
  • server-only provider code
  • timeouts
  • bounded retry behavior
  • vendor filtering
  • conservative deduplication
  • evidence validation
  • explicit uncertainty states
  • sanitized provider failures

Trust comes from the complete system, not just the language model.

Persistent memory raises the standard

Hindsight agent memory gives PactTrace something a normal one-shot quote analyzer does not have:

historical continuity.

That makes the application more useful.

It also raises the standard for engineering.

If an agent remembers information for later decisions, developers need to think carefully about:

  • what enters memory
  • how it is identified
  • how it is recalled
  • how conclusions are grounded
  • how uncertainty is represented

The Vectorize discussion of agent memory versus RAG is useful context for this broader architectural shift.

My main takeaway from PactTrace is:

Trustworthy AI memory is not just about remembering the right thing. It is about refusing to claim more than the evidence supports.

Project: https://github.com/v9vek26/pacttrace

Live application: https://pacttrace.vercel.app/

Top comments (0)