How Hindsight Turned a Field Removal Into Evidence

The dangerous part of removing an API field isn't the diff. It's knowing who still depends on it.

I built API Sentinel around that problem: instead of treating an API change as an isolated text comparison, I wanted the compatibility check to remember what consumer applications had told us before. Hindsight became the memory layer that made that possible.

The problem I wanted to solve

Consider a Course API with fields such as:

Course
├── id
├── title
├── description
├── category
└── instructor

At some point, someone proposes:

Remove description from Course API

From the provider's point of view, this can look like a small cleanup. From a consumer's point of view, it can be a breaking change.

The problem is that an API compatibility tool usually sees the proposed change, the API contract, and perhaps a source-code diff. It does not automatically know that an E-Learning application has been using description for months.

That missing context is what I wanted to capture.

API Sentinel is a Spring Boot application backed by MySQL, with a Python/Flask agent handling the memory and LLM workflow. The Python agent talks to Hindsight GitHub for persistent memory and to Groq for the final compatibility explanation.

The important architectural decision was to separate the system into two kinds of state:

MySQL stores API endpoints and proposed changes.

Hindsight stores knowledge about consumers and previous compatibility analyses.

That distinction matters because an API dependency is not just another database row. It is context that should be retrievable when a later change makes it relevant.

The flow is deliberately simple

The system has four important steps.

First, a consumer registers a dependency:

POST /api/ai/remember

with:

{
"application": "E-Learning App",
"api": "Course API",
"field": "description"
}

Spring Boot forwards that request to the Python agent.

The agent stores the relationship in Hindsight under the api-consumer-dependency context.

Later, a developer submits:

POST /api/ai/analyze

with:

{
"change": "Remove description from Course API"
}

The agent extracts the field name, asks Hindsight which consumers depend on it, filters the returned memories for direct dependency statements, and only then asks the LLM to explain the impact.

Finally, the compatibility analysis itself is retained in Hindsight.

So the system is not just:

API change -> LLM -> answer

It is:

API change
|
v
identify changed field
|
v
recall historical dependencies
|
v
filter relevant memories
|
v
build compatibility context
|
v
LLM analysis
|
v
remember the analysis

That middle section is the part I cared about most.

Why I didn't send the whole memory store to the LLM

One tempting implementation would be to retrieve a large amount of historical memory and put everything into the prompt.

I avoided that.

The agent first identifies the field being removed:

match = re.search(
r"remove\s+([A-Za-z0-9_]+)",
change,
re.IGNORECASE
)

if not match:
return (
"UNKNOWN",
"Could not identify the API field being removed."
)

field = match.group(1)

For a request like:

Remove description from Course API

the field becomes:

description

That gives the memory query a concrete target.

The agent then asks Hindsight:

result = hindsight_client.recall(
bank_id=BANK_ID,
query=(
f"Which API consumers depend on the {field} field? "
f"Return only direct consumer dependencies."
)
)

I like this boundary because the LLM isn't being asked to discover the dependency from an arbitrary pile of text. The retrieval step is responsible for finding historical evidence; the LLM is responsible for explaining what that evidence means.

That is a useful separation of concerns.

The memory itself is intentionally boring

The dependency stored in Hindsight is simple:

client.retain(
bank_id=BANK_ID,
content=(
f"API: {api_name}. "
f"Consumer: {consumer}. "
f"Dependency: {consumer} depends on the {field} field."
),
context="api-consumer-dependency"
)

There is no elaborate graph schema hiding behind it.

For the E-Learning example, the memory effectively says:

API: Course API.
Consumer: E-Learning App.
Dependency: E-Learning App depends on the description field.

That simplicity turned out to be useful. The memory has a clear statement that can be retrieved and inspected, while the application-specific reasoning remains outside the storage layer.

The Hindsight documentation describes persistent agent memory as a way to retain information and retrieve it later. In API Sentinel, I use that capability for something fairly concrete: preserving the relationship between an API field and a consumer application across separate interactions.

The interesting part happens when the field disappears

Once the memory comes back, I don't immediately trust every result.

The agent filters the recalled memories:

for memory in result.results:

text = memory.text.lower()
field_lower = field.lower()

if (
    field_lower in text
    and (
        "depends on" in text
        or "dependency on" in text
        or "relies on" in text
        or "rely on" in text
    )
):
    matching_memories.append(memory.text)

This is intentionally conservative.

If the recalled memory does not mention the field and a direct dependency relationship, it is not treated as evidence of an affected consumer.

The matching memories are then deduplicated:

matching_memories = list(dict.fromkeys(matching_memories))

If at least one dependency is found, the compatibility status becomes:

POTENTIALLY_BREAKING

Otherwise:

NO_KNOWN_IMPACT

That distinction is important. "No known impact" is not the same thing as "safe." It means the memory layer did not contain a known dependency matching the field.

I wanted that uncertainty to remain visible instead of turning absence of evidence into a confident compatibility claim.

The LLM gets evidence, not authority

The final prompt gives the model the proposed change, the field, the compatibility status, and the relevant historical memories:

prompt = f"""
You are an API compatibility analysis assistant.

Proposed API change:
{change}

Field being changed:
{field}

Compatibility status:
{status}

Relevant historical memories from Hindsight:
{memory_text}

Provide a concise developer-focused explanation.

Include:

1. Compatibility assessment
2. Affected consumers
3. Why the change may cause problems
4. A recommended migration action

Do not invent consumers or dependencies that are not present in the Hindsight memories.
"""

That last instruction is one of the most important lines in the system.

The model should explain the evidence, not manufacture more of it.

For the example, the useful chain is:

Proposed change:
Remove description from Course API

Historical evidence:
E-Learning App depends on description

Compatibility result:
POTENTIALLY_BREAKING

The LLM can turn that into a developer-readable explanation and migration guidance, but the affected consumer comes from stored evidence.

This is also why I chose a relatively low temperature:

temperature=0.2

The goal isn't creative generation. It's a consistent explanation of a compatibility finding.

I also made the analysis itself part of memory

After the LLM responds, API Sentinel stores the result:

hindsight_client.retain(
bank_id=BANK_ID,
content=(
f"API compatibility analysis: "
f"Proposed change: {change}. "
f"Field: {field}. "
f"Result: {status}. "
f"AI analysis: {ai_reason}"
),
context="api-compatibility-analysis"
)

This gives the system a second kind of historical knowledge.

The first kind is:

E-Learning App depends on description

The second is:

A previous compatibility analysis concluded that
removing description was potentially breaking.

That means compatibility work doesn't have to disappear after the HTTP response is returned.

The broader idea of agent memory is useful here because the agent's context can extend beyond one request. For API compatibility, that maps naturally to historical relationships and decisions.

How the Java and Python pieces fit together

I kept the Spring Boot layer responsible for application-facing APIs and persistence.

The controller exposes the memory and analysis endpoints:

@PostMapping("/analyze")
public String analyze(@RequestBody Map request) {

String change = request.get("change");

return pythonAgentService.analyzeChange(change);

}

@PostMapping("/remember")
public String remember(@RequestBody Map request) {

String application = request.get("application");
String api = request.get("api");
String field = request.get("field");

return pythonAgentService.rememberDependency(
        application,
        api,
        field
);

}

The Java service then forwards those calls to the Flask agent on port 5000:

return restClient.post()
.uri("/analyze")
.body(request)
.retrieve()
.body(String.class);

That split gives me a clean boundary:

Browser
|
v
Spring Boot :8080
|
v
Python Agent :5000
|
+--> Hindsight
|
+--> Groq

The API endpoint and change entities remain normal JPA models, while the agent-specific memory workflow stays in Python.

What the developer actually sees

From the UI, the interaction is intentionally small.

The developer enters:

Remove description from Course API

and clicks:

Analyze Compatibility

The browser calls:

const response = await fetch("/api/ai/analyze", {
method: "POST",
headers: {
"Content-Type": "application/json"
},
body: JSON.stringify({
change: change
})
});

The result exposes two pieces of information:

Compatibility Status
POTENTIALLY_BREAKING

and an AI-generated explanation.

The UI also explicitly identifies Hindsight as the memory source. I wanted that to be visible because the important difference between a stateless LLM call and this workflow is where the context came from.

What I learned

1. API compatibility is partly a history problem

A schema diff can tell me that a field disappeared.

It cannot, by itself, tell me which internal application still depends on that field.

Once I started thinking about compatibility as historical knowledge rather than only structural comparison, persistent memory became a natural part of the architecture.

1. Retrieval should happen before reasoning

I don't want the model guessing which applications might be affected.

I want the system to retrieve known dependencies first and then give the model those dependencies as evidence.

That makes the LLM's role narrower and easier to reason about.

1. "No known impact" needs careful wording

If Hindsight returns no matching dependency, API Sentinel reports NO_KNOWN_IMPACT.

It does not report "definitely safe."

That distinction matters in production because a memory system can only know what has been registered or otherwise captured. An unknown consumer remains possible.

1. Persistent memory is useful when the remembered fact has a clear lifecycle

"Remember everything" isn't a useful architecture.

In this system, I have two explicit memory contexts:

api-consumer-dependency
api-compatibility-analysis

That gives the stored information a purpose and makes retrieval easier to reason about.

1. The next hard problem is discovering dependencies automatically

The current architecture expects a consumer to register its dependency explicitly.

That's useful, but it leaves an obvious next step: automatically discovering dependencies from OpenAPI specifications, API gateway logs, runtime instrumentation, static analysis, or CI pipelines.

That is where I would take the system next.

The part I would keep

The most important design decision in API Sentinel isn't the LLM call.

It's the decision to preserve the relationship:

Consumer
|
| depends on
v
API field

and make that relationship available when the field eventually changes.

Removing description from the Course API is still just one line of an API diff.

With historical memory, that line can be connected to evidence about the applications that use it.

That's the difference I was trying to build: not an assistant that merely explains an API diff, but an analysis flow that can remember why a seemingly small change matters.