DEV Community

Cover image for Cypress — How to Create Automatic Weekly Flake Alerting
David Ingraham
David Ingraham

Posted on

Cypress — How to Create Automatic Weekly Flake Alerting

Flaky tests waste time, erode confidence, and make debugging a nightmare — especially when running in parallel across multiple CI machines.

Cypress Cloud provides fantastic flake detection and historical insights, but what if you don’t have access to it? You’re not out of luck.

In this guide, I’ll show you how to build your own automated flake alerting system that runs weekly, requires no manual effort, and integrates with your existing workflow.

By the end, you’ll have a setup that:

  • Tracks flake across different environments
  • Detects trends over time
  • Automatically alerts your team when thresholds are exceeded

Let’s dive in.

Prerequisite: Merge Report Infrastructure

This tutorial builds upon an earlier post How to Create a Custom Merge Report in Cypress. If you haven’t implemented that yet, start there — we’ll extend that implementation to save, merge and analyze flake data.

1. Save the Flake Results

Once your custom merge-report script is running, let’s expand it to track flake history over time

Add this at the bottom of your merge script:

  // Flake Detection
  const flakeHistoryPath = path.join(resultsDir, 'flake-history.json')
  const flakeHistory = {}

  // Optional: Differentiate by environment
  const env = mergedResults.baseUrl.includes('localhost') ? 'local' : 'dev'
  delete mergedResults.baseUrl

  for (const { test } of flakyTests) {
  flakeHistory[test] = {
    env,
    lastSeen: new Date().toISOString(),
  }

  // Save file
  fs.writeFileSync(flakeHistoryPath, JSON.stringify(flakeHistory, null, 2))
  console.log(`Flake history created at ${flakeHistoryPath}`)
Enter fullscreen mode Exit fullscreen mode

💡 Tip: Tracking the env (like dev or local) helps you isolate which environments are producing flaky results.

Now each test run will produce a flake-history.json file.

Let’s persist it for future aggregation.

2. Add a Weekly Aggregator Pipeline

We’ll now persist the flake history files and process them weekly.

2.1 Persist Flake Files

In your merge workflow, upload the artifact after the script runs. The following example uses Github Actions but any CI/CD tool works.

- name: Upload Cypress Flake Results
  uses: actions/upload-artifact@v4
  with:
    name: flake-history-${{ github.run_id }}
    path: cypress/results/flake-history.json
    retention-days: 7
Enter fullscreen mode Exit fullscreen mode

2.2 Weekly Aggregation Workflow

Next, set up a new pipeline workflow that runs weekly. It will fetch all flake files from our recent runs and prepare them for merging

name: Cypress - Flake Aggregator

on:
schedule:
- cron: '0 9 * * MON' # 3am MDT every Monday, tweak as desired
workflow_dispatch:

jobs:
  aggregate-flake-files:
  runs-on: your-runner # Your runner
  steps:
    - name: Checkout
      uses: actions/checkout@v4

  # 1. Get Workflow Run IDs from past 7 days
    - name: Get Recent Test Workflow Runs
      id: get-cypress-runs
      uses: actions/github-script@v7
      with:
          script: |
            const workflowNames = ['Cypress Tests'] // Your Cypress GH Workflow names
            const sinceDate = new Date(Date.now() - 7 * 24 * 60 * 60 * 1000) // 7 days ago
            const allRunIds = []

            const workflowsResp = await github.rest.actions.listRepoWorkflows({
              owner: context.repo.owner,
              repo: context.repo.repo,
            })

            for (const name of workflowNames) {
              const wf = workflowsResp.data.workflows.find(wf => wf.name === name)
              if (!wf) continue

              let page = 1
              let keepFetching = true

              while (keepFetching) {
                const runsResp = await github.rest.actions.listWorkflowRuns({
                  owner: context.repo.owner,
                  repo: context.repo.repo,
                  workflow_id: wf.id,
                  per_page: 100,
                  page,
                })

                // Fetch all workflows within the last week and persist run ids
                const recentRuns = runsResp.data.workflow_runs.filter(run =>
                  new Date(run.created_at) >= sinceDate
                )

                recentRuns.forEach(run => allRunIds.push(run.id))

                // Stop if there are no more recent runs or we hit end of pages
                if (
                  recentRuns.length < runsResp.data.workflow_runs.length ||
                  runsResp.data.workflow_runs.length === 0
                ) {
                  keepFetching = false
                } else {
                  page++
                }
              }
            }
            return allRunIds
          result-encoding: string

    # 2. Download Artifacts
    - name: Download Flake Artifacts
      run: |
        mkdir -p cypress/flakes // Wherever you stored the results
        echo '[${{ steps.get-cypress-runs.outputs.result }}]' | jq -r '.[]' | while read RUN_ID; do
          artifact_name="flake-history-$RUN_ID"
          echo "Attempting to download $artifact_name from run $RUN_ID"
          if gh run download $RUN_ID -n $artifact_name -D cypress/flakes/; then
            // Rename each file so it's unique
            mv "cypress/flakes/flake-history.json" "cypress/flakes/flake-history-$RUN_ID.json"
          else
            echo "Missing: $artifact_name"
          fi
        done

    # 3. Merge All Flake Files
    - name: Merge Flake History Files
      run: node cypress/merge-flake-history.js

    # 4. Optionally Upload Aggregated Report
    - name: Upload Aggregated Report
      uses: actions/upload-artifact@v4
      with:
        name: aggregated-flake-history
        path: cypress/merged-flake-history.json
        retention-days: 7
Enter fullscreen mode Exit fullscreen mode

Now, every Monday this job collects all flake history files from the past week and merges them.

Here’s the complete process at a glance, from Cypress test runs to automated alerts:

3. Merge the Flake Results

With the workflow in place, create the new merge-flake-history.js script. This script will loop through all the flake files from the past week and aggregate counts by test name and environment.

/* eslint-disable no-undef */
const fs = require('fs')
const path = require('path')

const mergedFlakeHistoryPath = path.join(__dirname, 'merged-flake-history.json')

const mergeFlakeHistories = async () => {
  const flakesDir = path.join(__dirname, 'flakes')
  const flakeFiles = fs.readdirSync(flakesDir).filter((f) => f.endsWith('.json'))

  console.log(`Found ${flakeFiles.length} flake files to analyze`)

  const merged = {}

  // Loop through all flake files
  for (const file of flakeFiles) {
    const contents = fs.readFileSync(path.join(flakesDir, file), 'utf-8')
    const flakeData = JSON.parse(contents)

    // For each flaky test, add to a new merged object 
    for (const testName in flakeData) {
      const { env, lastSeen } = flakeData[testName]

      if (!merged[testName]) {
        merged[testName] = {}
      }

      if (!merged[testName][env]) {
        merged[testName][env] = {
          count: 1,
          lastSeen,
        }
      } else {
        // Count amount of times the test has flaked across all files
        merged[testName][env].count += 1

        const existingDate = new Date(merged[testName][env].lastSeen)
        const newDate = new Date(lastSeen)

        // Use the earliest lastSeen timestamp
        if (newDate < existingDate) {
          merged[testName][env].lastSeen = lastSeen
        }
      }
    }
  }

  // Persist merge flake report
  fs.writeFileSync(mergedFlakeHistoryPath, JSON.stringify(merged, null, 2))
  console.log(`Merged flake history written to ${mergedFlakeHistoryPath}`)
}

mergeFlakeHistories()
Enter fullscreen mode Exit fullscreen mode

Example output:

{
  "Checkout Flow - should display cart": {
    "local": { "count": 2, "lastSeen": "2025-08-05T14:58:03.908Z" }
  },
  "Login - valid credentials": {
    "local": { "count": 10, "lastSeen": "2025-08-05T14:58:03.913Z" },
    "dev":   { "count": 4,  "lastSeen": "2025-08-07T12:40:06.156Z" }
  }
}
Enter fullscreen mode Exit fullscreen mode

4. Alert When Thresholds are Exceeded

Finally, with our merged report, we can trigger alerts when flake counts pass defined thresholds.

Here’s an example using an integration using Linear.

// Define thresholds based on env
const LOCAL_FLAKE_THRESHOLD = 7
const DEV_FLAKE_THRESHOLD = 3
// Any integration secrets
const LINEAR_API_KEY = process.env.LINEAR_API_KEY
const LINEAR_TEAM_ID = process.env.LINEAR_TEAM_ID

// Linear integration
const createLinearTicket = async (testName, count, env, lastSeen) => {
  const body = {
    query: `
      mutation {
        issueCreate(input: {
          title: "Cypress Flaky Test: ${testName}",
          description: "This test flaked ${count} times in the last week on ${env}. First observed: ${lastSeen}",
          teamId: "${LINEAR_TEAM_ID}"
        }) {
          success
        }
      }
    `,
  }

  try {
    const response = await fetch('https://api.linear.app/graphql', {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json',
        Authorization: LINEAR_API_KEY,
      },
      body: JSON.stringify(body),
    })

    const json = await response.json()
    if (json.data.issueCreate.success) {
      console.log(`Linear ticket created for "${testName}"`)
    } else {
      console.error(`Failed to create Linear ticket for "${testName}"`, json)
    }
  } catch (err) {
    console.error(`Error creating Linear ticket for "${testName}"`, err)
  }
}

// After merging results in mergeFlakeHistories 
for (const testName in merged) {
  for (const env in merged[testName]) {
    const { count, lastSeen } = merged[testName][env]
    // Optionally evaluate threshold based on env
    const threshold = env === 'local' ? LOCAL_FLAKE_THRESHOLD : DEV_FLAKE_THRESHOLD

    // Create Linear ticket if flake detected
    if (count >= threshold) {
      console.log(`Flake threshold exceeded: ${testName} (${count} on ${env})`)
      await createLinearTicket(testName, count, env, lastSeen)
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

💡 Tip: Swap out Linear for Jira, Slack, or any other alerting tool your team uses.

A simple, example Linear ticket that’s automatically created might look like this:

Conclusion

With this setup, automatically every week your team gets actionable alerts about the flakiest tests — without anyone manually digging through logs.

You’ll:

  • Automatically collect and merge flake data
  • Detect patterns per environment
  • Alert only when thresholds are breached
  • Keep your test suite healthier over time

You don’t need a full observability platform to track flake — just a clever CI workflow and a few scripts. For teams needing more depth, Cypress Cloud offers free trials.

Happy testing!

Top comments (0)