Giving a Satellite-Analysis Agent a memory

Kamal Misra Boddu — Mon, 28 Sep 2026 15:37:46 +0000

The first thing I noticed while building SatQuery AI was that the hard part wasn't understanding a satellite-image question—it was remembering what the question was about five messages later.
That sounds trivial in a normal chatbot. In Earth-observation software, it isn't.
A conversation about satellite imagery carries state that is easy for a human to keep in mind but surprisingly easy for a stateless AI system to lose: the selected region, the imagery being discussed, the time period, what “change” means in the current conversation, which objects we are tracking, and what the user asked us to analyze previously.
SatQuery AI is designed around a simple interaction loop:
Ask → Understand → Analyze → Verify → Visualize → Explain
The addition of persistent agent memory changed how we could implement that loop. Instead of treating every natural-language request as an isolated analysis job, we could let later requests build on earlier analytical context.
For this, we use Hindsight as the memory layer.
Hindsight on GitHub⁠�
Hindsight documentation⁠�
What is agent memory? — Vectorize⁠�
The system we were actually building
At a high level, SatQuery AI sits between a conversational interface and a collection of Earth-observation analysis workflows.
A user might start with:
“Show me where vegetation decreased in this region.”
The system first has to understand what that means operationally. The query contains several pieces of information:
Target: vegetation
Operation: decrease/change detection
Region: the selected area
Time dimension: potentially multiple observations
Expected evidence: changed areas, percentages, visual regions, etc.
That interpretation then determines which analysis workflow should be used.
Depending on the request, that workflow might involve object detection, segmentation, change detection, vegetation analysis, land-use/land-cover analysis, object counting, image comparison, or broader geospatial processing.
The resulting evidence is important. We don't want the language model to simply produce:
“Vegetation decreased significantly.”
We want the underlying analysis to provide evidence such as detected regions, counts, change areas, percentages, confidence information, or geospatial information. That evidence is then visualized on the imagery or map and explained in natural language.
Conceptually, the architecture looks like this:
User
│
▼
Natural-language query
│
▼
Query understanding
│
├──────────────► Hindsight memory
│ │
│ ▼
│ Relevant context
│
▼
Analysis routing
│
├── Object detection
├── Segmentation
├── Change detection
├── Vegetation analysis
└── Geospatial analysis
│
▼
Evidence / results
│
▼
Visualization
│
▼
Natural-language explanation
The important addition is that memory doesn't replace the analysis pipeline.
It sits alongside the reasoning layer and helps the system understand what the current question refers to.
Why memory is unusually important for satellite analysis
In a normal conversation, “What about the buildings?” is relatively easy to interpret if the previous message was about buildings.
In satellite analysis, that same sentence can be much more ambiguous.
Suppose I ask:
“Analyze this region and find areas where vegetation decreased.”
The system performs the analysis and shows several regions.
I might then ask:
“How much did it decrease?”
I haven't specified the region again.
Then:
“What about the buildings?”
And finally:
“Compare that with the previous result.”
These aren't four independent questions. They're part of one analytical thread.
A stateless system can process each request individually, but then we have to repeatedly pass context into every request or force the user to restate it.
That is exactly the kind of interaction I wanted to avoid.
The useful unit isn't just the current prompt. It's the analytical conversation.
Before memory: every question starts from zero
Imagine this interaction.
User:
“Analyze the selected region for vegetation loss.”
SatQuery AI:
“Vegetation loss was detected in several areas.”
The user follows up:
User:
“Show me the percentage.”
A stateless implementation has to know which region, imagery, comparison and analysis the user means.
Without retained context, we end up with something like:
User:
"Show me the percentage."

System:
"Which region and satellite images should I analyze?"
The user now has to repeat information that was obvious from the conversation.
The alternative is to stuff the entire conversation into every subsequent request and hope the model correctly extracts the relevant information. That can work, but it mixes conversation history with the more deliberate concept of agent memory.
For SatQuery AI, I wanted memory to be a first-class part of the system.
After Hindsight: the conversation becomes cumulative
With Hindsight providing the memory layer, the interaction can instead behave like this:
User:
“Analyze this region for vegetation loss between the two images.”
SatQuery AI:
“I found vegetation decrease across the selected region. The affected areas are highlighted on the map.”
User:
“Show me the percentage.”
The system can use the remembered analytical context to understand that “the percentage” refers to the vegetation-change analysis that just happened.
Then:
User:
“Now detect buildings in the same region.”
The important phrase is “same region.”
The user doesn't need to redefine it. The previous context can inform the interpretation of the new query.
Then:
User:
“Compare those results with the vegetation changes.”
Now the conversation contains a chain of related analytical operations rather than isolated requests.
A simplified representation of the memory interaction looks like this:

Pseudocode — illustrative, not project code

context = memory.retrieve_relevant_context(user_query)

analysis_request = understand_query(
query=user_query,
context=context
)

result = run_analysis(analysis_request)

memory.store(
query=user_query,
analysis_context=analysis_request,
result_summary=result.summary
)
The key idea isn't the syntax. It's the separation of responsibilities.
The analysis pipeline remains responsible for producing evidence. Memory remains responsible for preserving useful conversational context.
What we actually want to remember
One lesson I learned quickly is that “remember everything” is not a useful memory strategy.
For satellite analysis, useful context can include things such as:
the region currently being discussed
previous analytical tasks
the object or phenomenon being investigated
previous questions
interpretation preferences
relationships between successive analyses
For example, if a user has established that they are interested in vegetation changes in a particular region, a subsequent question like:
“Can you show only the major changes?”
should be interpreted relative to that analytical thread.
Conceptually, our reasoning layer can use memory like this:

Pseudocode — illustrative only

current_query = "Show only the major changes"

relevant_memory = hindsight.retrieve(current_query)

interpreted_query = query_understanding(
current_query,
relevant_memory
)
The important distinction is that memory isn't the source of truth for the satellite measurements.
If the system says vegetation decreased by a particular percentage, that number should come from the analysis pipeline—not from something remembered from an earlier conversation.
That separation matters.
Memory and evidence are different things
This became one of the most important architectural boundaries for me.
Consider two statements:
“The user is analyzing Region A.”
and:
“Vegetation decreased by 17%.”
The first can be conversational context.
The second is an analytical result that should be grounded in the underlying image-processing pipeline.
I don't want memory accidentally becoming a database of facts that the model blindly trusts.
Instead, the flow should look more like:
Memory
│
▼
"What is the user referring to?"
│
▼
Analysis pipeline
│
▼
"What does the imagery actually show?"
│
▼
Evidence
│
▼
Explanation
This distinction makes the system easier to reason about and easier to debug.
If the visualization is wrong, I can inspect the geospatial or computer-vision pipeline.
If the system misunderstands “that region,” I can inspect query interpretation and memory retrieval.
Those are different failure modes.
The visualization closes the loop
Another important part of SatQuery AI is that the answer isn't just text.
Suppose the system detects vegetation decrease.
The analysis can produce regions representing the detected change. Those results can then be visualized directly over satellite imagery or a map.
The language response becomes an explanation of that evidence:
“The analysis identified vegetation decrease primarily in the western portion of the selected region. The highlighted areas correspond to the detected changes.”
This gives the user three connected layers:
Conversation — what they asked.
Evidence — what the analysis produced.
Visualization — where that evidence exists spatially.
Memory connects the conversation across multiple analytical operations.
A limitation we had to take seriously
Persistent memory introduces a new failure mode: remembering the wrong thing can be worse than remembering nothing.
Imagine the user switches from Region A to Region B and then asks:
“Find the buildings here.”
If memory incorrectly retrieves Region A as the active context, the system could perform a perfectly valid building-detection operation on the wrong location.
That's not a model-quality problem alone. It's a system-design problem.
This is why I don't think memory should silently override explicit information from the current query.
A useful principle is:
Current explicit context
>
Relevant remembered context
>
Older conversational context
Memory should help resolve ambiguity, not create it.
That was one of the clearest lessons from thinking about Hindsight in this architecture: agent memory is contextual infrastructure, not an authority on reality.
Where Hindsight fits
I think of Hindsight as sitting between the conversational interface and the analytical execution layer.
It helps answer questions such as:
“What is this user referring to?”
It should not answer:
“What does the satellite image actually contain?”
That second question belongs to the remote-sensing and computer-vision pipeline.
This distinction keeps the architecture relatively clean:
┌──────────────────┐
│ User Query │
└────────┬─────────┘
│
▼
┌──────────────────┐
│ Query Understanding│
└────────┬─────────┘
│
┌─────────┴─────────┐
▼ ▼
┌──────────────┐ ┌──────────────┐
│ Hindsight │ │ Current Query│
│ Memory │ │ Context │
└──────┬───────┘ └──────┬───────┘
│ │
└─────────┬─────────┘
▼
┌──────────────────┐
│ Analysis Routing │
└────────┬─────────┘
▼
┌──────────────────┐
│ EO/CV Workflows │
└────────┬─────────┘
▼
┌──────────────────┐
│ Evidence + Maps │
└────────┬─────────┘
▼
┌──────────────────┐
│ Explanation │
└──────────────────┘
That placement is important. Hindsight isn't another computer-vision model and it isn't a replacement for geospatial processing. It's part of the conversational intelligence layer.
What I would build differently next time
The tempting approach with conversational systems is to add memory after everything else works.
I think that's backwards for stateful analytical applications.
Once users start asking follow-up questions, context becomes part of the product behavior itself.
I'd also be careful about deciding what deserves long-term memory. Not every sentence in a conversation has equal value. A useful memory system needs to distinguish durable analytical context from temporary conversational noise.
Most importantly, I'd treat memory retrieval as something that requires verification.
If the current request contains explicit information that conflicts with remembered context, the current request should win. If the retrieved memory is ambiguous, the system should have a way to ask for clarification rather than confidently choosing the wrong region or analysis.
What I learned
Three engineering lessons stand out.

Conversational state is part of the application architecture For analytical agents, conversation history isn't merely UI context. It can directly affect which computation gets executed.
Memory and evidence should remain separate Hindsight can help us understand what the user means. It shouldn't become the authority for what the satellite imagery shows.
Ambiguity needs a safer fallback A memory system that confidently retrieves the wrong context can produce technically correct analysis for the wrong problem. Explicit user input should take precedence over remembered assumptions.
Follow-up questions are a real workload Users naturally ask questions incrementally: “Find vegetation loss.” “How much?” “Where exactly?” “What about buildings?” “Compare that with the previous result.” Designing for that sequence produces a very different system from designing for one prompt at a time. The bigger picture The interesting part of SatQuery AI isn't simply putting a language model in front of satellite-image processing. The engineering challenge is connecting language, spatial reasoning, computer vision, evidence and conversation state without allowing one layer to pretend it knows what another layer actually measured. Hindsight gives us a way to make the conversational side persistent. That means a user can move from: “Analyze vegetation change.” to: “How much changed?” to: “Show the major areas.” to: “Now detect buildings there.” without reconstructing the entire analytical context every time. For me, that's the difference between a chatbot that happens to analyze satellite imagery and an analytical system that can actually support an ongoing investigation. The images still provide the evidence. The analysis pipeline still does the measurement. The map still shows where the result occurred. Memory makes the conversation capable of carrying that investigation forward.

DEV Community: Kamal Misra Boddu

Giving a Satellite-Analysis Agent a memory

Pseudocode — illustrative, not project code

Pseudocode — illustrative only