I worked on the data layer for VendorPulse. The problem that hit me early was simple: a single procurement dataset is useless for agent memory.
We started with DataCo (180k shipments: late delivery risk, real vs. scheduled days). It’s great for logistics, but it has no rejection rates, no excuses, and no vendor context. It tells you a delivery was late—not why it matters.
We added the Vendor Performance dataset (2.3M purchases: vendor invoices, freight costs). This gave us price and invoice dates, but lacked seasonality.
Both datasets are accurate, but neither captures what a procurement manager actually remembers:
"Last July, SteelCore was 22 days late during the monsoon with a 12% rejection rate and claimed 'truck breakdown'. They used the exact same excuse in August."
That sentence never exists inside an ERP. To fix this, we had to build a third source: synthetic procurement context.
The Three-Layer Data Model
Here is the three-layer data architecture I implemented:
Procurement KPI → Vendor / PO History (Structured Facts)
Data: PO_ID, vendor name, quantity, price, delivery date.
Storage: SQLite (deterministic query layer).
DataCo → Logistics Context (Macro Patterns)
Data: Aggregate logistics patterns, a 54.8% late delivery baseline, First Class 0% on-time performance, monsoon delay vectors.
Purpose: Provides realistic seasonality.
Synthetic Context → Demo / Historical Narrative (Qualitative Memory)
Data: Excuses, rejection reasons, true cost after penalty, human operational notes.
Purpose: Enables meaningful semantic recall in Hindsight.
The Data Pipeline
I built a pipeline to merge factual numbers with narrative context to create episodic experiences for our memory store:
data_processing_pipeline.py - merging real + synthetic
import pandas as pd
from faker import Faker
dataco = pd.read_csv("DataCoSupplyChainDataset.csv")
vendor = pd.read_csv("vendor_invoice.csv")
Create an episodic experience that Hindsight will store
def build_experience(row):
return {
"vendor": row["Vendor"],
"material": row["Category"],
"month": row["Order_Month"],
"delay": row["Days_Real"] - row["Days_Scheduled"],
"rejection": row["Defect_Rate"],
"excuse": generate_excuse(row["Vendor"], row["Month"]), # e.g., truck breakdown, customs
"true_cost": row["Price"] + (row["Delay"] * penalty_per_day)
}
Why Hindsight for this?
Because real-world procurement data is messy by design. Phrases like "Truck breakdown" and "Vehicle failure in heavy rain" must match semantically. A traditional key-value store or relational DB can't bridge that gap; Hindsight does.
Resources: Hindsight GitHub | Hindsight Documentation
Retention Quality Controls Recall Quality
If you retain low-context tuples like:
JSON
{"vendor": "X", "late": 22}
...you get poor recall.
However, if you retain rich, natural narrative strings like:
Plaintext
"Structural Steel from SteelCore in July monsoon: +22 days late, 12% rejected, excuse truck breakdown, flagged by Apex team"
...you unlock excellent semantic matches for future queries.
Implementation:
Python
retain.py - what I actually store in memory
hindsight.retain(
text=f"Vendor {vendor} delivered {material} in {month}: "
f"{delay} days late, {rejection}% rejected. "
f"Reason: {excuse}. Outcome: {outcome}. Lesson: {lesson}",
metadata={"vendor": vendor, "material": material, "month": month}
)
Key Takeaway
The best memory system will fail if your source data lacks a narrative.
Real datasets give you credible numbers.
Synthetic context gives you recallable narrative.
You need both. DataCo and Vendor KPIs provide baseline credibility, while synthetic excuses provide semantic recallability.
The Result
When we query Hindsight for "Structural Steel October high priority", it doesn't just return average historical delay—it surfaces the July and August failures linked to repeated excuses.
The baseline evaluation sees a 4.1 rating. Agent memory sees the pattern.
Note: The human procurement manager remains the ultimate decision-maker—we never auto-purchase. We simply ensure the next evaluation remembers last July.
Top comments (0)