What I Redact Before Sending CI Logs to Hindsight

A CI log can contain two things at the same time: the exact error a developer needs to investigate and credentials that should never leave the system.

For example, a failed test might show a payment gateway timeout, a useful stack trace, and an API key printed by a debugging statement. Sending that entire log to an external AI or memory service without preprocessing creates an unnecessary security risk.

While working on FlakeTrail, I focused on this part of the workflow: extracting useful debugging context while reducing the amount of sensitive information that reaches downstream services.

The goal is not to claim that a regular expression can detect every secret. It is to make redaction an explicit step before analysis and memory operations.

Why CI logs need preprocessing

CI logs are written for debugging, not necessarily for safe external processing. They can contain:

• Authorization headers and access tokens.
• API keys and passwords.
• Database connection strings.
• Private-key material.
• Test names, file paths, stack traces, and error messages.
• Runner or environment information.

Removing the entire log is not a practical solution. The error message and stack trace may be exactly what helps identify a recurring failure.

FlakeTrail therefore uses a preprocessing step that attempts to remove common secret patterns while preserving relevant debugging context.

Step 1: Redact before extracting the failure

The parser exposes a redactSecrets() function. It applies a sequence of replacements to the incoming text.

For example, the implementation includes rules for authorization headers, API keys, tokens, passwords, and other credential formats.

One part of the implementation looks like this:

// Redact Authorization headers
redacted = redacted.replace(
  /Authorization:\s*Bearer\s+\S+/gi,
  'Authorization: [REDACTED]'
);

// Redact API keys
redacted = redacted.replace(
  /(API_KEY\s*[:=]\s*)\S+/gi,
  '$1[REDACTED]'
);

// Redact passwords
redacted = redacted.replace(
  /(password\s*[:=]\s*)\S+/gi,
  '$1[REDACTED]'
);

The important design decision is that redaction happens inside the parser before the extracted information is returned to the analysis route.

The implementation also includes patterns for GitHub tokens, GitLab tokens, private keys, and database connection strings containing credentials.

This is a practical first layer, not a guarantee that all possible secrets will be detected. New credential formats and unusual log structures can require additional rules.

Step 2: Preserve information that helps debugging

After redaction, FlakeTrail extracts structured information from the remaining log.

The parser looks for:

• Test identity.
• Error message.
• Error type.
• Stack trace.
• File paths.
• CI job ID and branch information, when recognizable.
• A generated failure signature.

The parser also returns a redacted log excerpt limited to 2,000 characters. The extracted error message is limited to 500 characters.

This makes it easier for the rest of the application to work with a focused representation rather than repeatedly processing an unrestricted log.

For example, consider this illustrative input:

FAILED payments/service_test.py::test_payment_retry
Error: timeout after 30s waiting for payment gateway mock
API_KEY=sk-live-example-secret

The useful debugging context is the failing test and timeout message. The credential is not required to investigate that failure.

After preprocessing, the API response should preserve the test and error context without returning the original credential.

Step 3: Create a consistent failure identity

A similar failure may appear in different runs with changing timestamps, request IDs, or other numeric values. Comparing entire log files would make matching difficult.

FlakeTrail therefore creates a failure signature using the test name and a normalized error message. The normalization step lowercases the message, replaces digits with a common placeholder, removes punctuation, and standardizes whitespace.

For example, these two messages:

Timeout after 30 seconds
Timeout after 45 seconds

can become equivalent after numeric normalization, assuming the test identity is also the same.

This gives FlakeTrail a compact identifier that can be used during Hindsight RECALL.

It is important to understand what this signature does not mean. Two failures with similar messages are not necessarily caused by the same bug. A signature is a retrieval aid, not proof of a shared root cause.

Step 4: Connect preprocessing to Hindsight

The analysis route follows a clear order:

1. Validate the incoming request.
2. Parse and redact the CI log.
3. Generate the failure signature.
4. Use Hindsight RECALL to retrieve related investigations.
5. Pass the current parsed failure and historical investigations to the reasoning layer.

This ordering matters. The application does not need to send an unprocessed raw log to Hindsight just to retrieve related incidents.

When a developer later records an investigation outcome, the Hindsight service builds a concise memory record from the investigation details. That record is also passed through redactSecrets() before RETAIN.

This means the parser's security responsibility is not limited to the first analysis request. The memory write path also applies redaction before sending the investigation record to Hindsight.

Step 5: Test the security behavior

A security feature should not exist only as a comment in the source code. It needs tests.

FlakeTrail's parser test suite includes cases that check redaction of API keys, authorization headers, passwords, private keys, and database connection strings. It also checks that useful debugging information is preserved.

The backend API tests add another layer. They submit logs containing example secrets and verify that those values do not appear in the parsed API response.

The tests also cover empty logs, malformed output, missing stack traces, large log excerpts, and consistent signatures across whitespace-only differences.

This is useful because a redaction rule can accidentally remove information that a developer needs. Testing both sides matters: the secret should disappear, but the diagnostic context should remain.

A before-and-after example

Before preprocessing: A raw CI log contains a timeout error, a stack trace, and a credential. Sending it directly to external services could expose the credential.

After preprocessing: FlakeTrail redacts recognized secret patterns, extracts the failing test and error context, and generates a failure signature. Hindsight can then be used to retrieve related investigation history without requiring the original unprocessed log as the retrieval input.

The difference is not just cleaner text. It is a more controlled boundary between incoming CI data and external processing.

What still needs improvement

Pattern-based redaction has limits. A secret may appear in an unfamiliar format, be split across log lines, or be embedded in encoded data. A rule that catches API_KEY=value does not automatically catch every possible credential representation.

The parser also uses lightweight patterns to identify test names and errors. Different testing frameworks produce different output formats, so extraction may be incomplete when a new format appears.

A production-grade version should add broader secret-detection coverage, test against a larger collection of real-world log formats, and consider structured CI artifacts where available. Sensitive test fixtures should use synthetic values rather than real credentials.

These limitations are why I would not describe the current parser as a complete data-loss-prevention system. It is a useful preprocessing layer that reduces common exposure risks and makes logs easier to analyze, but it still needs ongoing validation.

The lesson

The most useful CI log is not necessarily the longest log. It is the one that preserves the evidence needed to investigate a failure without unnecessarily exposing sensitive information.

In FlakeTrail, parsing and redaction happen before Hindsight retrieval, and the RETAIN path applies redaction again before storing an investigation outcome.

That makes secure preprocessing part of the application's architecture rather than an optional cleanup step.

The project source is available at FlakeTrail on GitHub. The implementation can be reviewed in the CI parser, parser tests, and Hindsight memory service.

A debugging assistant should preserve what developers need to understand a failure—not everything that happened to appear in the log.