From Logs to Issues: An Agentic Pipeline for Flaky Test Analysis

Flaky tests are the silent killer of developer velocity. They fail randomly, waste hours of debugging, and erode trust in CI. As a developer, I've spent too many mornings scrolling through GitHub Actions logs trying to figure out if a failure is a real bug or just a network blip.
So I built FlakeFixer — a tool that monitors GitHub Actions workflows, extracts failure logs, uses an AI agent to classify the root cause (race condition, network timeout, infrastructure blip, dependency flake), and automatically creates a GitHub Issue with a plain-English analysis and suggested mitigation.
In this post, I'll walk you through the architecture, the tech stack, and how you can build your own.
The Problem
Flaky tests are tests that pass and fail randomly without any code changes. They're common in CI pipelines, especially when tests rely on external services, timing, or shared state.
Manually debugging flaky tests is painful because:

• Logs are huge and noisy.
• The actual error is often buried in a stack trace.
• The root cause might be an infrastructure hiccup, not a code bug.
• I wanted a system that could automatically:
• Fetch failed CI runs.
• Extract the relevant log snippet.
• Use an LLM to reason about the failure and categorize it.
• Post a concise analysis as a GitHub Issue.

That's exactly what FlakeFixer does.
High-Level Architecture
The system has three main components:
Data Ingestion — Python script that uses PyGithub to list failed workflow runs and download logs.
Agentic Analysis— LangChain + Pydantic output parser that classifies the failure and generates a mitigation suggestion.
Reporting — Automatically creates GitHub Issues with the analysis, labeled by flake category.

Building the Flaky Test Lab
Before building the tool, I needed a way to generate flaky failures on demand. I created a companion repo called ci-flake-lab that intentionally produces several types of flaky failures:
Race condition — a test that fails when threads access a shared list.
Network timeout — a test that tries to connect to a non-routable IP.
Infrastructure blip — a test that randomly raises OSError: No space left on device.
Unknown— failures that don't cleanly fit a known category, to test how the agent handles ambiguity.

I used GitHub Actions environment variables to switch the flake type on each run, with a matrix strategy so multiple flake types run in parallel. This gave me a steady stream of failed logs to test the pipeline against- you can see the resulting auto-filed issues in the repo, each tagged with flaky-test and its specific category.
[Insert your screenshot of the ci-flake-lab issues list here]
Here's the workflow YAML:

name: Flake Farm
on: [push, workflow_dispatch]

jobs:
  flaky-tests:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        flake-type: [RACE, NETWORK, INFRA, UNKNOWN]
      fail-fast: false
    env:
      FLAKE_${{ matrix.flake-type }}: 1
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.10'
      - run: pip install pytest
      - run: pytest test_app.py -v

Data Ingestion with PyGithub
The ingestion module uses PyGithub to authenticate and list failed workflow runs, then downloads the log archive and extracts the raw text.

from github import Github
import requests, zipfile, io

class IngestionPipeline:
    def __init__(self, token, repo_name):
        self.gh = Github(token)
        self.repo = self.gh.get_repo(repo_name)

    def get_failed_runs(self, workflow_id, max_runs=20):
        workflow = self.repo.get_workflow(workflow_id)
        return list(workflow.get_runs(status="failure"))[:max_runs]

    def download_logs(self, run_id):
        run = self.repo.get_workflow_run(run_id)
        headers = {"Authorization": f"token {token}"}
        resp = requests.get(run.logs_url, headers=headers, allow_redirects=True)
        with zipfile.ZipFile(io.BytesIO(resp.content)) as zf:
            full_log = ""
            for name in zf.namelist():
                if name.endswith(".txt"):
                    full_log += zf.read(name).decode("utf-8", errors="replace") + "\n"
        return full_log

This gives me the raw log text. In the full version, I also trim the log to the most relevant parts (the last 100 lines and lines containing FAILED or ERROR) to keep the LLM prompt within token limits.
The Agentic Analysis Engine
For the AI agent, I used LangChain with a Pydantic output parser to force the LLM to return structured JSON. This ensures I can reliably parse the output and use it to create issues.
The prompt asks the LLM to classify the failure and provide an explanation and mitigation.

from langchain_core.prompts import ChatPromptTemplate
from langchain_openai import ChatOpenAI
from langchain.output_parsers import PydanticOutputParser
from pydantic import BaseModel, Field

class FlakeAnalysis(BaseModel):
    flake_category: str = Field(description="race_condition, network_timeout, infrastructure_blip, unknown")
    explanation: str
    mitigation: str
    confidence: float

class FlakeAnalyzer:
    def __init__(self):
        self.llm = ChatOpenAI(model="gpt-4o-mini", temperature=0.1)
        self.parser = PydanticOutputParser(pydantic_object=FlakeAnalysis)

    def analyze(self, log_snippet):
        prompt = ChatPromptTemplate.from_template("""
You are a flaky test analyst. Given the CI failure log, classify the root cause.
Log:
{log}

{format_instructions}
""")
        formatted = prompt.format_prompt(
            log=log_snippet[:6000],
            format_instructions=self.parser.get_format_instructions()
        )
        response = self.llm.invoke(formatted.to_messages())
        return self.parser.parse(response.content)

Here's an example of what the agent returns:

{
  "flake_category": "network_timeout",
  "explanation": "The test attempted to connect to an external service but timed out. This is likely an infrastructure or network issue, not a code bug.",
  "mitigation": "Add retry logic with exponential backoff and increase the timeout. Consider mocking external services in unit tests.",
  "confidence": 0.9
}

Automated Reporting — Creating GitHub Issues
The final piece is the reporting layer. Once the agent returns the analysis, I use PyGithub to create a GitHub Issue with the analysis, labels, and a log snippet.

from github import Github
class Reporter:
    def __init__(self, token, repo_name):
        self.gh = Github(token)
        self.repo = self.gh.get_repo(repo_name)

    def create_flake_issue(self, analysis, run_id, log_snippet):
        title = f"[Flake] {analysis.flake_category.replace('_',' ').title()} (run #{run_id})"
        body = f"""## Flaky Test Analysis
**Category:** `{analysis.flake_category}`
**Confidence:** {analysis.confidence:.2f}

### Root Cause
{analysis.explanation}

### Suggested Mitigation
{analysis.mitigation}

<details><summary>Log snippet</summary>


{log_snippet[:1500]}

</details>

*Auto-generated by FlakeFixer agent.*
"""
        labels = ["flaky-test", analysis.flake_category]
        issue = self.repo.create_issue(title=title, body=body, labels=labels)
        return issue.html_url

You can see the real output of this in the ci-flake-lab issues tab — every one of those [Flake] ... issues was filed automatically by FlakeFixer, not by hand.

Open Source Contribution
While building FlakeFixer, I noticed that pytest-github-actions-annotate-failures did not truncate long annotation messages, causing clutter in PRs. I opened a PR to add truncation for annotation output. This was my first contribution to that project, and it reinforced the importance of clean, readable CI output.

Next steps for FlakeFixer:
Add a vector database to store and retrieve similar historical failures.
Fine-tune a small language model on my labeled flake dataset.
Generate weekly flake summary reports.
Conclusion
Flaky tests don't have to be a black hole of developer time. By combining GitHub Actions, LLMs, and automation, we can automatically triage and document flaky failures, giving maintainers back their time and sanity.
If you're interested in the code or want to see a demo, check out the repos:
https://github.com/Haadiyah-Zafar/FlakeFixer
https://github.com/Haadiyah-Zafar/ci-flake-lab
Let me know in the comments if you've tackled similar problems or have ideas for improvement!