I'm building a flaky-test finder with Spring Boot, Go and SQS (v0.1.0)

A flaky test passes and fails on the same code. Nothing changed, but the build is red. You click "re-run", it goes green, and you move on.

I read about a dev team whose tests kept failing on the same commit. They spent a lot of time re-running and investigating why it happened.

So I started FlakeSince, a free, self-hosted tool that finds flaky tests in CI. Most free tools tell you a test is flaky today. FlakeSince keeps the pass/fail history of every normal CI run, so it can also estimate since which commit a test became flaky, without re-running anything.

How it works

1. A GitHub Action uploads the JUnit XML files from your test run (Gradle, Maven Surefire, pytest or go-junit-report).
2. A small Go ingest service checks the API key, parses the XML and puts one message per CI run on a queue. It answers 202 Accepted as soon as the run is safely queued.
3. Amazon SQS holds the message. Locally I use ElasticMQ, which speaks the SQS protocol, so you don't need an AWS account to develop.
4. A Spring Boot API (Java 25) reads the queue and saves each run to PostgreSQL. If saving fails 3 times, the message goes to a dead-letter queue instead of being lost.

Using it in a workflow looks like this:

- name: Upload test results to FlakeSince
  if: always()   # upload even when tests fail
  uses: TheStrawberry-09/flakesince/action@v0
  with:
    url: ${{ secrets.FLAKESINCE_URL }}
    api-key: ${{ secrets.FLAKESINCE_API_KEY }}
    paths: '**/build/test-results/**/*.xml'

A failed upload only adds a warning, so FlakeSince never turns your CI red by itself.

Decision 1: why a separate Go service in front of Spring Boot?

The simplest design is one Spring Boot app that receives uploads and saves them. I didn't do that, because of two facts about CI uploads:

• They arrive in bursts. Many repositories finish CI at the same moment.
• They must not be lost, even when the main API is busy, restarting or down.

If Spring receives uploads directly, every API restart means lost test results. So I put a small service and a queue in front of it. Uploads keep being accepted even when the API is down, and the API catches up when it comes back.

I considered writing that front service in Java too, which would mean one language in the repo. I chose Go because the service is small and does one job: it starts in milliseconds, uses little memory, builds to a single binary, and the standard library already has a production-ready HTTP server and XML parser.

The cost is real: two languages, two CI workflows, and a message format (RunMessage in Go, RunUploadRequest in Java) that must stay in sync. I wrote this down as an architecture decision record (ADR 001) so future me remembers why.

Who checks the API key?

This led to a second question. The API key hashes live in a table that Spring owns. I had three options:

• Go asks Spring to check the key. But if Spring is down, uploads fail, and the queue becomes pointless.
• Spring checks the key later, when it reads the queue. But then anyone can fill the queue, and a wrong key still gets 202, so users never find out their setup is broken.
• Go reads the key table directly.

I picked the third. Go looks up the SHA-256 hash of the key with its own read-only database role, ingest_reader, which can read only three columns. If that password leaked, nothing else would be exposed. Valid keys are cached for 60 seconds, so a revoked key can keep working for up to a minute. That's a trade-off I accepted and wrote down in ADR 002.

Decision 2: saving each run exactly once

SQS guarantees at-least-once delivery: the same message can arrive twice. If the API saved it twice, one CI run would count as two, and the flaky statistics would be wrong.

The fix has two layers. First, the database refuses duplicates:

UNIQUE (project_id, provider_run_id, run_attempt)

Second, the ingest code checks before saving and treats a duplicate as a success, not an error:

Optional<CiRun> existing = runs.findByProjectIdAndProviderRunIdAndRunAttempt(
        project.getId(), upload.providerRunId(), upload.runAttempt());
if (existing.isPresent()) {
    return new IngestResult(existing.get().getId(), true, 0);   // duplicate: already saved
}

Including run_attempt matters. When you click "re-run" in GitHub Actions, the run ID stays the same but the attempt number goes up. Those re-runs are exactly the data a flaky-test tool needs, so they must be saved as separate attempts, not thrown away as duplicates.

The queue listener deletes a message only after it is saved. If anything throws, the message stays in SQS and is retried, and after 3 tries it moves to the dead-letter queue:

IngestResult result = ingestService.ingest(run);
sqs.deleteMessage(r -> r.queueUrl(queueUrl).receiptHandle(message.receiptHandle()));

The listener runs on a Java 25 virtual thread with SQS long polling, so it waits cheaply when the queue is empty.

It runs

Here's the action uploading results from my sample repo, flaky-demo:

What's next

v0.1.0 collects the history. The next steps are the interesting part:

• Flaky test detection: a test that both passed and failed on the same commit is flaky.
• "Flaky since commit X": using the history to estimate when the flakiness started.
• A dashboard and a hosted demo.

The code is Apache-2.0 on GitHub: github.com/TheStrawberry-09/flakesince

This is my first time making an open-source project, so a lot of things are still new to me. I used AI to help me find information, draft code and check bugs in logs.
Feel free to give feedback! I’ll keep updating this project, and I’ll add a mascot too.
Hope you enjoy it. Thank you!