DEV Community

Cover image for False Green: My Incident Agent Said “Healthy” Without Live Evidence
Michael Neang
Michael Neang

Posted on

False Green: My Incident Agent Said “Healthy” Without Live Evidence

#devchallenge #bugsmash #agents #ai

Summer Bug Smash: Clear the Lineup 🐛🛹

This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.

The pipeline was healthy.

At least, that was what my incident agent told me.

There was one problem: it could not see the live Fivetran pipeline.

The live check had failed, so the app fell back to cached evidence. That snapshot still said connected and on_schedule.

The final verdict was still:

Healthy.

Nothing crashed.

The request succeeded.

The fallback data looked healthy.

The conclusion was still wrong.

The application had taken the last known state and quietly treated it like the current one.

That was the false green.

Current confidence requires current evidence.

Project Overview

Pipeline Rescue Agent is a Next.js incident-investigation app that combines connector-health and data-freshness signals to decide what still needs investigation.

Gemini runs later in the recovery-planning flow. The false-green verdict occurred in deterministic application logic before the model was involved.

The question at this decision point was simple:

Is this pipeline healthy right now?

I found a case where the application was more confident than its evidence allowed.

Bug Fix or Performance Improvement

When live Fivetran telemetry became unavailable, Pipeline Rescue Agent fell back to cached connector evidence.

I wanted to keep that fallback. The last known connector state can still be useful during an incident.

The failed live path was the trigger condition.

It was not the application defect.

The defect was what happened afterward: cached evidence was still allowed to support a current Healthy verdict.

Before the fix, the decision state looked like this:

evidence.mode = cached_fivetran_evidence
evidence.live = false

fivetran.mcp_ok = false
fivetran.setup_state = connected
fivetran.update_state = on_schedule

pipeline.health_verdict = healthy
false_green.detected = true
Enter fullscreen mode Exit fullscreen mode

Debugging chronology

There are three intentional repository states in this investigation:

State Purpose
6194a3b14efd33a0d5f8235f703187005123a0f9 Historical tagged baseline, before Sentry instrumentation
b3a45110 Sentry instrumentation added while the decision bug was still intact
b05d2cccf93713f6a52a4a4ec6fa6b1600cbec86 Final fixed state

The sequence was deliberate:

reproduce
→ instrument the still-broken decision
→ inspect
→ diagnose
→ fix
→ verify
Enter fullscreen mode Exit fullscreen mode

Pre-fix Sentry decision span showing cached Fivetran evidence, live evidence false, MCP unavailable, connected and on-schedule status, and a Healthy pipeline verdict

Pre-fix Sentry decision span: cached, non-live evidence still produced a Healthy verdict.

Almost every individual status value looked reasonable.

The /api/investigate request itself completed successfully. The fallback returned data. The cached connector status said connected and on_schedule. There were no active warnings or tasks.

The endpoint completed successfully; the semantic failure itself raised no exception.

The contradiction only appeared when evidence provenance and the final decision were inspected together:

cached evidence
+ live = false
+ MCP unavailable
→ Healthy
Enter fullscreen mode Exit fullscreen mode

The cache itself was doing its job.

The health gate was asking the cache to prove something it could not prove.

Cached evidence could answer:

What was last known?

It could not answer:

Is the pipeline healthy right now?

Why this mattered

Healthy was not only a displayed status.

The connector decision feeds the agent's final pipelineStatus, which getLikelyIssue uses when choosing what the investigation should examine next.

When connector health cannot be established:

Needs review
→ likely issue: Connector health
Enter fullscreen mode Exit fullscreen mode

When the connector is considered healthy while destination data is stale:

Healthy + stale data
→ likely issue: Upstream source freshness
Enter fullscreen mode Exit fullscreen mode

A false green did not merely change the displayed status; it could change what the investigation examined next.

I did not need the application to assume Fivetran was broken. I needed it to stop clearing connector health when current connector evidence was unavailable.

Sentry Exposed the Contradiction

I first reproduced the suspicious Healthy result through /api/investigate.

Before fixing it, I added a custom Sentry pipeline.decision span containing the runtime state behind that decision:

evidence.mode
evidence.live
fivetran.mcp_ok
fivetran.setup_state
fivetran.update_state
pipeline.health_verdict
false_green.detected
Enter fullscreen mode Exit fullscreen mode

This was a silent correctness bug.

The request worked. The fallback worked. The values looked plausible. The failure was in the conclusion drawn from them.

Sentry put the healthy-looking status content, its non-live provenance, and the final verdict in the same trace.

That changed the debugging question from:

Why does this endpoint say Healthy?

to:

Why is non-live evidence allowed to support Healthy?

Seer and the root cause

I used Seer to analyze the captured failing trace.

It highlighted evidence currentness as the likely missing decision constraint: healthy-looking cached values were reaching a verdict path that did not enforce live provenance.

Seer root-cause analysis identifying that healthy-looking cached Fivetran status was evaluated without enforcing evidence currentness or provenance

Seer RCA: the failing trace pointed to evidence currentness/provenance as the missing decision constraint.

I treated that as a hypothesis and verified it against the source.

I also did not copy the generated suggestion literally.

Pipeline Rescue Agent supports two legitimate live evidence paths:

mcp_live
live
Enter fullscreen mode Exit fullscreen mode

A direct live Fivetran API response can still be current evidence even when the MCP path is unavailable.

So the correct invariant was not:

Healthy requires MCP success
Enter fullscreen mode Exit fullscreen mode

It was:

Healthy requires live evidence
Enter fullscreen mode Exit fullscreen mode

The Missing Invariant

The bug reduced to one safety property:

A Healthy verdict requires live evidence.

Healthy ⇒ evidence.live = true
Enter fullscreen mode Exit fullscreen mode

The failing Sentry trace violated it:

evidence.live = false
pipeline.health_verdict = healthy
Enter fullscreen mode Exit fullscreen mode

The cache could remain; only the decision boundary had to change.

Code

PR: Fix false-green pipeline health verdicts from cached Fivetran evidence

I isolated the health decision in a small helper used by the production route:

export function evaluateFivetranHealth({
  mode,
  setupState,
  updateState,
  hasWarnings,
  hasTasks,
  paused,
}) {
  const hasLiveEvidence = mode === "mcp_live" || mode === "live";

  const isHealthy =
    hasLiveEvidence &&
    setupState === "connected" &&
    updateState === "on_schedule" &&
    !hasWarnings &&
    !hasTasks &&
    paused === false;

  return {
    hasLiveEvidence,
    isHealthy,
  };
}
Enter fullscreen mode Exit fullscreen mode

The important distinction is small:

Evidence What it may establish
Live May certify current health
Cached Historical context only; cannot independently certify current health

Historical evidence can still inform the investigation.

Only current evidence may certify current health.

My Improvements

I checked the surrounding investigation flow for the same provenance assumption. Cached Fivetran evidence no longer appears as a successful current connector check, no longer counts as live evidence, and is no longer described in recovery reasoning as if it represented current health. The production rule now lives in the small pure helper above, which also makes the decision boundary directly testable.

After: Same Non-Live Evidence Class, Safer Decision

The final Sentry trace from commit b05d2cccf93713f6a52a4a4ec6fa6b1600cbec86 shows:

evidence.mode = cached_fivetran_evidence
evidence.live = false

fivetran.mcp_ok = false
fivetran.setup_state = connected
fivetran.update_state = on_schedule

provenance.guard_applied = true
pipeline.health_verdict = needs_review
Enter fullscreen mode Exit fullscreen mode

Final Sentry decision span showing cached non-live Fivetran evidence with the provenance guard applied and the pipeline verdict changed to Needs review

Final Sentry decision span: the same non-live evidence class now produces Needs review.

The status values still look superficially healthy.

The live telemetry is still unavailable.

The fix did not make unavailable telemetry available. It changed what conclusion the application was allowed to derive from that uncertainty.

cached + non-live
→ Needs review
Enter fullscreen mode Exit fullscreen mode

Historical RED → Final GREEN

I wanted stronger verification than:

The patched application passes a test written after patching it.

So I moved the decisive assertion outside the repository and ran it unchanged against the tagged historical application and the final branch.

The script imports no application decision logic. It intentionally reads only response fields available in both versions:

timeline[].evidence.mode
agentRun.decision.pipelineStatus
Enter fullscreen mode Exit fullscreen mode

It also refuses to count a run if the expected cached-fallback condition was not reproduced.

In simplified form, the external assertion enforced three conditions:

if (mode !== "cached_fivetran_evidence") {
  throw new Error("TEST INVALID: cached fallback was not reproduced");
}

if (verdict === "Healthy") {
  process.exit(1);
}

if (verdict !== "Needs review") {
  throw new Error(`Unexpected verdict: ${verdict}`);
}
Enter fullscreen mode Exit fullscreen mode

I ran that same external assertion against the historical baseline:

Commit:
6194a3b14efd33a0d5f8235f703187005123a0f9

Evidence mode:
cached_fivetran_evidence

Pipeline verdict:
Healthy

Result:
FAIL

Exit code:
1
Enter fullscreen mode Exit fullscreen mode

Then I ran it against the final branch:

Commit:
b05d2cccf93713f6a52a4a4ec6fa6b1600cbec86

Evidence mode:
cached_fivetran_evidence

Pipeline verdict:
Needs review

Result:
PASS

Exit code:
0
Enter fullscreen mode Exit fullscreen mode

Terminal comparison showing the historical baseline returning Healthy and failing the external black-box check, while the final branch returns Needs review and passes

External black-box check: historical baseline fails; final branch passes.

This comparison does not claim that every low-level cause of telemetry unavailability was identical between the two executions.

That is not what the assertion tests.

The controlled application condition is:

Fivetran evidence is cached rather than live.

The question is:

Can that evidence still certify current health?

Historical application:

Yes.

Final application:

No.

Decision-Boundary Positive Control

There was one more bad fix I wanted to rule out:

What if Healthy simply became impossible?

The same pure predicate imported by the production route was checked on both sides of the boundary:

Scenario Expected Result
cached_fivetran_evidence + clean status Healthy = false PASS
mcp_live + clean status Healthy = true PASS
live + clean status Healthy = true PASS

This is intentionally a decision-boundary check.

It does not claim to contact a live MCP server or the live Fivetran API.

Its purpose is narrower: both supported live evidence modes remain eligible for Healthy, while cached evidence does not.

The non-live application behavior is separately verified through /api/investigate and the external black-box assertion.

Verification

Final branch verification:

npm run test:health-invariant
PASS

PIPELINE_RESCUE_URL=http://localhost:3001 npm run test:false-green
PASS

External black-box check
PASS
exit 0

npm run lint
PASS

npm run build
PASS
Enter fullscreen mode Exit fullscreen mode

The HTTP regression check confirms:

Evidence mode: cached_fivetran_evidence
Live evidence: false
Pipeline verdict: Needs review

PASS: non-live cached evidence cannot produce a Healthy verdict.
Enter fullscreen mode Exit fullscreen mode

Best Use of Sentry

This was exactly where Sentry mattered: /api/investigate completed successfully, so ordinary error monitoring did not expose the semantic failure.

The pipeline.decision span made provenance, liveness, connector state, and final verdict observable together inside that successful request.

Sentry did not merely verify the repair after the fact: I captured the contradiction from the still-buggy release, used that failing trace as runtime context for Seer, and then inspected the same decision again after the fix.

Seer then helped narrow that runtime contradiction to evidence currentness, which I verified against the source before implementing the narrower application invariant.

After the fix, the final trace shows:

evidence.live = false
provenance.guard_applied = true
pipeline.health_verdict = needs_review
Enter fullscreen mode Exit fullscreen mode

Sentry did not replace the debugging.

It exposed the hidden decision state that ordinary success/failure monitoring would not have explained.

Final Result 

BEFORE

No live Fivetran telemetry
→ cached healthy-looking evidence
→ Healthy


AFTER

No live Fivetran telemetry
→ cached evidence retained as context
→ live-evidence guard
→ Needs review
Enter fullscreen mode Exit fullscreen mode

The cache can still tell an investigator what was last known.

It just no longer gets to answer:

Is the pipeline healthy right now?

Current confidence requires current evidence.

PR: github.com/mneang/pipeline-rescue-agent/pull/1

Top comments (0)