I wanted a deploy check to answer a question that ordinary code review often cannot: have we made this exact kind of change before, and what happened afterward?
Foresight is a deploy-safety service for payment changes. Its most useful example is deliberately concrete: a pull request lowers GATEWAY_RETRY_TIMEOUT_MS from 30,000 milliseconds to 8,000. A conventional reviewer can see the diff. Foresight can also retrieve the incident history attached to that setting and make the prior outcome part of the current decision.
That second part is Hindsight’s job. Foresight retains incidents, deploys, fixes, and outcomes in a Hindsight bank, recalls a small set of memories for each proposed change, and asks a language model to produce a structured risk brief grounded in those memories. The distinction matters: without recall, the example returns generic canary advice; with recall, the seeded scenario returns a cited HOLD and a reason tied to past failures.
The application has a Next.js UI and CLI/GitHub Action clients in front of a FastAPI and SQLModel backend. The backend sends memory operations to Hindsight and uses Groq for live JSON reasoning. The article below focuses on the path where those pieces meet: a deploy check.
Give the change a history
The repository models a fictional payment platform called PayNest. Its sample data is synthetic, as the README states; these incidents are not claims about production outages. They make the system’s behavior repeatable and let the demo show how prior experience changes a deploy review.
The first seeded incident, INC-2024-001, describes a timeout reduction from 30 seconds to 8 seconds. NPCI processing takes about 12 seconds during peak load, so the gateway times out and retries while the original transaction is still pending. The incident record includes the 8-second timeout in its logs, a missing idempotency key in the retry envelope, and the resulting double debit.
Its fix attempts are more valuable than a bare incident title. Increasing the maximum retry attempts from three to five failed because it worsened load on NPCI. Rolling the timeout back to 30 seconds and adding an idempotency header worked; the seeded data says the incident resolved in 38 minutes.
The second incident, INC-2024-002, adds a recurrence. Config drift reintroduced the shortened timeout, and the async retry path bypassed the idempotency check. Restarting gateway nodes helped temporarily, but the seed data says the relief lasted nine days before auto-scaled nodes brought the bad configuration back. The durable fix was to enforce an idempotency-key check before retry dispatch and keep the timeout at 25 seconds or more.
Those details are retained, rather than kept only in a demo prompt. During replay, the demo service sorts incident and deploy records chronologically, then writes incident content, tags, and metadata to the bank. The incident content includes the alert, logs, Slack excerpt, root-cause class, and serialized fix attempts. Tags include service, severity, and root-cause class. Deploy memories include the changed configuration and recorded outcome.
In live mode, MemoryService passes those fields through to the Hindsight SDK:
self.client.retain(
bank_id=self.bank_id,
content=content,
document_id=document_id,
tags=tags,
metadata=metadata,
)
The live integration creates a dedicated bank, foresight-paynest, with a mission focused on causal links between changes, outages, and fix efficacy. When an incident is resolved, Foresight also builds a postmortem and retains it with identifiers and service metadata. When a deploy check receives an actual outcome, the outcome endpoint stores the record in SQLite and retains a corresponding Hindsight memory. That is the path by which future checks can learn from what happened after the current review.
Recall against the proposed change
The deploy-check endpoint is POST /api/v1/deploys/check. With memory enabled, Foresight builds a query from the service, title, description, configuration changes, and up to 300 characters of the diff. It asks the memory service for at most ten results:
query_parts = [request.service, request.title, request.description or ""]
if request.config_changes:
query_parts.append(" ".join(request.config_changes))
if request.diff:
query_parts.append(request.diff[:300])
query = " ".join(query_parts)
recalled_memories = self.memory_service.recall(query=query, limit=10)
In live mode, the memory service calls the Hindsight Python client with the configured bank ID and query. It maps the returned results into Foresight’s internal shape: ID, text, score, tags, and metadata. It also handles the synchronous client call when invoked from a running event loop by putting that call in a worker thread. Recall failures are surfaced as errors rather than quietly represented as an empty history.
The recalled items are then included in the LLM prompt alongside the proposed change. The prompt asks for a JSON response containing a risk score, verdict, reasons, similar incidents, canary plan, rollback plan, and citations. It explicitly tells the model to use only incident or memory IDs present in the recalled set.
Foresight checks the structured output before returning it. It drops citation and similar-incident objects whose IDs are not in the recalled results, and if the model omitted citations while memories were available, it supplies citations from the retrieved items. That check makes the structured evidence trail more useful: a reader can inspect the memory IDs Foresight actually retrieved instead of accepting an invented incident reference.
The citation filter is small and explicit:
cit_id = cit_dict.get("id")
if cit_id in recalled_ids:
valid_citations.append(MemoryCitationItem.model_validate(cit_dict))
The deploy check is not the only recall path. The incident workflow queries Hindsight with the service, incident title, alerts, and logs, then asks the LLM to rank fixes with outcomes such as worked, failed, or temporary. The /api/v1/ask route also recalls memories before synthesizing an answer. Across these paths, Hindsight supplies persistent operational context; Foresight decides how that context enters a specific workflow.
What changes when memory is on
The UI and API expose a memory toggle, which makes the difference observable. With memory disabled, the deploy check returns a deterministic generic response: risk score 45, verdict CANARY, no similar incidents, and no memory citations. The advice is to use standard testing and a small canary. It does not mention NPCI’s latency, the double debit, or the failed retry increase.
With memory enabled in the seeded offline scenario, the fallback returns risk score 87 and verdict HOLD. The response cites the recalled incident IDs and explains that 8 seconds is shorter than the 12-to-15-second payment processor latency represented in the seed data. It mentions that increasing retries made the earlier storm worse, that a restart only held for nine days, and that the safer correction was a longer timeout plus idempotency enforcement. The proposed plan is specific: do not proceed without a canary, enforce idempotency checks, watch gateway timeouts and duplicate-debit rates, and restore the timeout to 30 seconds if rollback is needed.
The exact numbers are code behavior for the demo, not benchmark results. In offline mode, Hindsight is represented by a local mock memory list loaded from the seed files, and LLMClient validates the deterministic fallback object instead of calling Groq. The live path calls Hindsight and Groq; its verdict depends on the retrieved memories and model output. The repository’s test for the Friday 5:40 PM scenario asserts a HOLD, a score of at least 80, citations, and similar incidents, while the memory-disabled test asserts that citations and similar incidents are empty. Those tests define observable behavior; they do not establish a production false-positive rate.
Three lessons from wiring memory into a deploy gate
1. Store outcomes, not just symptoms. An incident description says what broke. A fix outcome tells the next review what to avoid or repeat. Here, “increase retries” is explicitly marked failed, while rollback plus idempotency enforcement is marked worked. The temporary restart carries its nine-day lifetime. That distinction gives the next decision more signal than a list of outage titles.
2. Treat retrieval as part of the decision interface. The query is built from the service, change title, description, config keys, and a diff excerpt. That is a practical boundary: it gives the retrieval layer multiple clues without sending the entire diff into the recall query. The results are then shown as citations, so a reviewer can see why the decision changed.
3. Validate references after generation. Asking a model to cite only retrieved IDs is useful, but it is not a complete guard. Foresight also filters the structured citation and similar-incident arrays against the actual recalled IDs. This keeps those UI elements tied to retrieval, even if a model returns an unsupported ID.
4. Separate the memory store from the reasoning step. In the deploy path, Foresight calls Hindsight’s retain and recall operations; the Groq-backed LLM produces the risk brief. The repository does not call Hindsight’s reflect operation for this decision. Keeping that boundary explicit makes it easier to reason about which component searched history and which component interpreted it.
The trade-off I would keep visible
The offline story is useful for a repeatable demo, but it is not a substitute for measuring the live path. The mock recall implementation uses local text matching and a special retry/timeout boost. The 87-point score and HOLD are prewritten fallback behavior for the scenario, not a result learned from Hindsight’s retrieval quality or generated by a live LLM. The seeded PayNest incidents are synthetic too.
There is another limit in the citation guard: Foresight filters structured citation and incident IDs, but it does not independently verify every sentence in the model-generated summary and reasons against the recalled text. The prompt constrains the model, and the citations improve inspectability, but the code does not prove each natural-language claim. A production deployment gate should treat that as a review surface and evaluate the live retrieval and generation path on real, permissioned incident history.
For this project, the central insight is simple: Hindsight lets a deploy check carry outcomes forward. The risky change is no longer just “timeout equals 8,000.” It is “timeout equals 8,000, and we have seen what happened when that value was shorter than the payment path.”
Top comments (0)