Why I Put Hindsight Between CI Logs and Recommendations

Why I Put Hindsight Between CI Logs and Recommendations

A CI build fails at 02:13. The log points to "PaymentIntegrationTest" and reports a connection timeout. A developer reruns the job, checks the stack trace, changes a timeout, and tries again. The failure may disappear—or return next week. The difficult part is not always reading the current log. It is knowing what the team already tried the last time a similar failure appeared.

That is the architectural problem behind FlakeTrail: an investigation should not end when an answer is displayed. Its outcome should become useful context for the next investigation.

The key decision was to put Hindsight between the incoming CI failure and the recommendation layer. That gives FlakeTrail a place to retrieve earlier investigation outcomes and later preserve what developers learned.

The architecture follows the investigation lifecycle

FlakeTrail has a React and Vite frontend, a Node.js and Express backend, a shared TypeScript contract layer, and Hindsight as its persistent investigation-memory service. The backend is divided into focused modules rather than putting log parsing, memory access, and recommendation logic in one route.

The flow is:

1. A developer uploads or pastes a CI log in the frontend.
2. The backend validates the request and parses the log.
3. The parser redacts secrets and extracts useful failure details.
4. The backend asks Hindsight for similar historical investigations.
5. The reasoning module combines the current failure with the retrieved history.
6. The frontend displays the analysis and the developer records the actual outcome.
7. The backend validates that outcome and sends the investigation record to Hindsight.

The separation keeps responsibilities clear: parsing extracts evidence, Hindsight retrieves history, reasoning interprets it, and backend validation controls which outcomes can be called verified.

Step 1: Make the log safe and useful

Raw CI logs are noisy. They can contain stack traces, file paths, runner details, credentials, tokens, and unrelated output. Sending the entire log to every downstream service would increase exposure and make retrieval less focused.

FlakeTrail's parser calls "redactSecrets()" before extracting fields. The current implementation includes patterns for authorization headers, API keys, passwords, GitHub and GitLab tokens, private keys, and database connection strings. It then looks for details such as the test name, error message, error type, stack trace, file paths, and CI job or branch information.

It also creates a failure signature from the test identity and a normalized error message. The purpose is to give the memory lookup a compact representation of the failure pattern rather than relying only on a long, variable log.

Redaction must happen before external calls, not after. Pattern-based rules still need tests as new log formats and secret formats are introduced.

Step 2: Retrieve history before producing a recommendation

The "POST /api/analyze" route parses the incoming request, calls the parser, and then invokes "recallSimilarTracked()" using the generated failure signature. The route also supplies test and CI-job tags to the service interface. The service limits returned history to five results before passing it to the reasoning module.

An important detail in the Hindsight service is how it preserves the status of each recalled investigation:

const status: IncidentStatus =
m.metadata?.status === 'verified' ||
m.metadata?.status === 'failed' ||
m.metadata?.status === 'passed_unmerged'
? m.metadata.status
: 'proposed';

This is a small piece of code with an important architectural purpose. A previous investigation marked "verified" is not equivalent to one marked "failed" or "passed_unmerged". FlakeTrail preserves that distinction instead of treating every similar memory as an equally successful solution.

The service also limits recall to five results. That keeps the context passed downstream controlled instead of sending an unbounded history dump with every analysis.

If Hindsight recall fails, the route logs a warning and continues with empty history. This avoids making a memory-service outage a hard stop, but the result must be treated as history-free—not as proof that no similar failure exists.

Step 3: Treat outcomes differently

A list of similar error messages is not enough. The system needs to know what happened after the previous developer investigated the problem.

FlakeTrail uses four statuses:

• "proposed": a hypothesis or suggestion that has not been confirmed.
• "failed": an attempted approach that did not solve the issue.
• "passed_unmerged": a change passed CI but has not been merged.
• "verified": CI passed and the change was merged.

The distinction is enforced by backend validation. A request cannot mark an incident as "verified" unless the CI result is "passed" and "merged" is "true". This rule is more important than a prompt instruction because the API, not the language model, owns the final status decision.

The reasoning module then uses those labels. It gives priority to previously verified approaches, treats failed attempts as things to avoid repeating, and keeps passed-but-unmerged changes separate from confirmed fixes. When no useful history exists, it says that the failure needs to be investigated from first principles instead of pretending that memory found an answer. In the linked branch, "analyzeFailure()" is a status-aware TypeScript decision layer that returns structured analysis fields; it should be presented as investigation guidance, not as proof of autonomous root-cause discovery.

Step 4: Write back the result

The investigation is not complete when the recommendation appears. The developer still has to try an approach, run CI, and record what happened.

When the developer submits an outcome, "POST /api/incidents/:id/resolve" validates the payload and updates the incident. If Hindsight is configured, the route calls "retainInvestigation()" with the test name, error message, failure signature, attempted fix, outcome, CI result, merge state, pull-request link, notes, and timestamp.

Before RETAIN, the service builds a concise investigation record and runs it through secret redaction again. It also attaches searchable metadata such as the status, test name, failure signature, and incident ID. The service rejects incomplete records and does not store a "proposed" hypothesis as a completed investigation.

This creates a useful feedback loop: RECALL brings prior experience into the current investigation; the developer's outcome then becomes material for a future RECALL. Hindsight is not just a storage destination at the end of the workflow. It is part of the application's control flow.

REFLECT is available through a separate optional endpoint. The normal "/api/analyze" path uses RECALL and does not call REFLECT. Keeping the operations separate makes the default analysis path explicit and avoids silently adding an extra memory operation to every request.

A before-and-after example

Consider an illustrative sequence involving a test that fails with a timeout.

Without investigation memory: the next developer sees the timeout, searches the current log, and may try increasing the timeout again. The earlier failed attempt and its result are easy to miss if they live in an old chat or pull request.

With FlakeTrail: the failure signature is used to retrieve related investigations. The reasoning layer can see that a timeout change previously failed, while a different approach may have reached "verified" status. It can surface the failed attempt as something not to repeat and point the developer toward the verified approach. The developer still has to reproduce the issue, apply the change, and verify it in CI.

The system does not automatically fix the pipeline; it lets the next investigation start with evidence from the previous one.

What this architecture does not solve yet

The current MVP has limitations. The incident route stores active incident objects in an in-memory map, so that local incident list is not a durable database-backed history across backend restarts. Hindsight stores retained investigation outcomes, but that does not automatically make every part of the application state persistent.

The parser is pattern-based, so unfamiliar log formats can produce incomplete fields. Although the route passes test and CI tags, the current Hindsight query is built from the failure signature; tag-aware filtering is not active in that call. Similar signatures can also hide different causes, so a retrieved fix is evidence to inspect, not permission to apply blindly.

These limitations shape the product's safety rule: FlakeTrail recommends the next investigation step; it does not change code, suppress tests, or merge a fix on its own.

The lesson

The most important architectural choice was not the frontend framework or the API structure. It was treating a CI investigation as a lifecycle with a memory read before reasoning and a verified write after human action.

That design turns a sequence of isolated debugging sessions into a growing record of what was tried and what actually worked. Hindsight gives FlakeTrail the memory layer; backend validation and explicit outcome recording keep that memory tied to developer-confirmed results.

The project source is available at "github.com/amithsharma91/FlackTrail" (https://github.com/amithsharma91/FlackTrail). The key implementation points are the "log parser" (https://github.com/amithsharma91/FlackTrail/blob/feature/flaketrail-mvp/backend/src/parser/index.ts), "analysis route" (https://github.com/amithsharma91/FlackTrail/blob/feature/flaketrail-mvp/backend/src/routes/analyze.ts), "Hindsight service" (https://github.com/amithsharma91/FlackTrail/blob/feature/flaketrail-mvp/backend/src/hindsight/service.ts), and "reasoning module" (https://github.com/amithsharma91/FlackTrail/blob/feature/flaketrail-mvp/backend/src/reasoning/index.ts).

The goal is simple: the next person who sees a familiar CI failure should not have to rediscover everything the team already learned.