DEV Community: Artem

Catching Flaky Tests Before They Hit CI: Meet FlakyDetector

Artem — Sun, 28 Jun 2026 19:36:52 +0000

By Artem Alimpiev — Python Backend Developer

Every engineering team has that test.

The one that passes locally, passes on your teammate’s machine, passes three times in CI… and then suddenly fails at 2 AM for absolutely no reason.

So somebody reruns the pipeline.

Again.

And again.

Eventually the build goes green, everyone moves on, and the flaky test quietly stays in the repository like a cursed artifact nobody wants to touch.

I’ve seen this happen too many times in real projects. And honestly, the bigger the infrastructure becomes, the worse the problem gets.

At some point I started asking myself a simple question:

Why are we still detecting flaky tests after they break CI?

That question eventually turned into FlakyDetector — an AST + Machine Learning powered system for detecting flaky tests before they hit your pipelines.

GitHub Repository:
FlakyDetector GitHub Repository

Flaky Tests Are More Expensive Than Most Teams Realize

People often treat flaky tests as “just annoying.”

But flaky tests are actually infrastructure debt.

They:

slow down releases
waste CI resources
destroy trust in automation
create false negatives
normalize ignoring failed builds

And that last one is especially dangerous.

Because once engineers stop trusting CI, the entire feedback loop starts collapsing.

Google engineers once reported that a significant percentage of test failures inside large CI systems were caused by flakiness rather than real regressions.

Think about that for a second.

Imagine debugging failures that aren’t even real bugs.

Now combine that with:

async systems
distributed services
parallel test execution
unstable timing
shared global state
external APIs

Suddenly your test suite behaves less like deterministic engineering and more like a physics experiment.

Most Flaky Detection Tools React Too Late

The majority of flaky-test tooling works reactively.

Usually the workflow looks like this:

Write test ↓ Push code ↓ CI randomly fails ↓ Retry pipeline ↓ Lose 40 minutes of engineering time

Traditional systems rely on:

rerun statistics
CI telemetry
historical failure tracking
probabilistic heuristics

Useful? Absolutely.

Preventive? Not really.

I wanted something different.

I wanted a system that could analyze the source code itself and detect risky patterns before the tests ever started failing in production CI environments.

That’s where AST analysis became incredibly interesting.

AST Analysis: Looking at Code Structurally

If you’ve never worked with Python ASTs directly, here’s the simple explanation.

Python code isn’t just text.

Under the hood, Python converts code into an Abstract Syntax Tree — a structured representation the interpreter can reason about.

For example:

time.sleep(5)

isn’t stored as plain text internally.

It becomes semantic structure:

function call
module reference
execution dependency
timing behavior

And once you have structure, you can detect patterns.

That changes everything.

FlakyDetector scans Python test suites and searches for architectural anti-patterns associated with nondeterministic behavior.

Examples include:

`time.sleep()
datetime.now()`
mutable global state
unmocked network requests
dangerous fixture scopes
resource leakage
high cyclomatic complexity

Or in testing terminology:

Test Smells.

Yes, that’s a real technical term. And yes, it sounds slightly ridiculous.

Why I Added Machine Learning

At first, FlakyDetector started as a pure AST rule engine.

But I quickly ran into a problem.

Real flaky behavior is rarely caused by a single issue.

Usually it’s a combination of:

timing dependencies
complexity
state mutations
async interactions
fixture misuse

That means simple rule matching eventually hits limits.

So I added an ML classification layer using CatBoost.

The pipeline now looks roughly like this:

Python Test Code ↓ AST Parsing ↓ Feature Extraction ↓ 42-Dimensional Feature Vector ↓ CatBoost Classification ↓ Flaky Probability + Severity

And honestly, CatBoost turned out to be a surprisingly strong fit.

Most developers associate CatBoost with recommendation systems or business analytics. But it’s extremely good at structured tabular feature classification.

Which is exactly what AST-derived metrics become.

The 42-Feature Detection System

This is where the project became much more than “another linter.”

FlakyDetector currently extracts a 42-dimensional feature space.

The features include:

Feature Group Count AST Features 16 Category Features 9 Fixture Analysis 5 Derived Metrics 3 Confidence Scores 8 Test Smells

Examples include:

timing dependency counters
network interaction detection
mutation ratios
fixture scope analysis
pattern diversity metrics
cyclomatic complexity scoring

The system then classifies the probability of flakiness and assigns severity levels.

But the important part is this:

The model is explainable.

That matters a lot in developer tooling.

Nobody wants a black-box AI saying:

“Your test is dangerous. Trust me.”

So FlakyDetector exposes:

confidence levels
detected anti-patterns
feature importance
severity categories

The goal is not just prediction.

The goal is understanding.

The Architecture: Clean, Fast, and CI-Friendly

The project follows a Hexagonal Architecture approach.

That means the core analysis engine stays isolated from:

UI
infrastructure
APIs
storage layers
CI integrations

The stack currently includes:

Python 3.12
FastAPI
Pydantic v2
CatBoost
React + Vite
ChromaDB
Ollama
Docker
GitHub Actions

I also focused heavily on developer experience.

The project uses:

uv for extremely fast dependency management
ruff for linting and formatting
pyright strict typing
pre-commit
Dockerized infrastructure

Because let’s be honest:

Nobody wants reliability tooling that itself becomes maintenance debt.

GitHub Actions Integration Is Where It Gets Practical

This is probably the most important feature for real teams.

FlakyDetector can block problematic tests directly inside CI pipelines.

Example:

- name: Run FlakyDetector run: uv run python scripts/scan_folder.py ./tests --fail-on-critical

If the system detects critical anti-patterns, the pipeline fails immediately.

That means developers catch instability risks during code review instead of after the test suite starts randomly exploding three weeks later.

This is especially useful for:

fintech platforms
SaaS systems
async Python services
microservice environments
ML infrastructure
large monorepos

The RAG + LLM Layer Sounds Weird… But It’s Useful

One experimental feature I added integrates:

ChromaDB
vector search
local LLMs via Ollama
semantic instability analysis

At first glance it sounds like:

“Congratulations, we added AI to flaky tests.”

Which honestly made me laugh too.

But there’s a practical reason behind it.

Large repositories often contain repeated instability patterns across multiple teams.

The semantic search layer allows engineers to find tests with similar architectural problems.

For example:

“Show me tests similar to this flaky async Redis integration.”

That becomes surprisingly powerful in enterprise-scale repositories.

It’s still evolving, but I think AI-assisted reliability engineering is going to become much more important over the next few years.

Why I Think This Problem Matters

Modern engineering teams invest enormous effort into:

observability
security
performance
type safety
infrastructure automation

But test reliability is still oddly under-engineered.

We accept flaky tests as “normal.”

That’s strange if you think about it.

Because unstable tests don’t just waste CI minutes.

They quietly destroy confidence in the engineering process itself.

FlakyDetector is my attempt to treat flaky testing as an architectural problem instead of random chaos.

And honestly, I think the industry needs more tools that shift reliability checks earlier into the development lifecycle.

The same way:

CodeQL changed security scanning
mypy changed Python typing
Ruff changed linting performance

Static reliability analysis could become a completely normal part of CI pipelines.

Try It Yourself

Quick setup:

git clone https://github.com/Artem7898/flakydetector cd flakydetector 

uv venv --python 3.12 

source .venv/bin/activate 

uv pip install -e ".[dev]" 

python scripts/train_model.py

Run a scan:

uv run python scripts/scan_folder.py ./tests/

And yes…

There’s a good chance it finds something uncomfortable in your test suite.

One Final Question

How many flaky tests are currently sitting in your repository pretending to be “temporary”?

And how much engineering time are they silently burning every single week?

That’s probably worth thinking about.

If this article was useful — follow me for more deep dives into Python backend engineering, CI/CD systems, AST tooling, static analysis, and modern developer infrastructure.

I’d genuinely love to hear how your team deals with flaky tests today.

Retry buttons?
Quarantine lists?
Pure denial?

Write in the comments — I’m curious.

Django Nova: Bringing Strict Typing and Async-First Architecture to Django Models

Artem — Mon, 22 Jun 2026 16:59:30 +0000

Django is one of those frameworks engineers keep returning to.

Fast MVPs. Mature ecosystem. Battle-tested ORM. Admin panel that feels like cheating.

And yet, every experienced Django developer eventually hits the same wall.

Not performance.

Not scalability.

Architecture.

Specifically: type safety, validation duplication, async chaos, and runtime surprises that only appear at 2AM in production when your logs start looking like modern art.

That frustration is exactly why I built Django Nova.

PyPI
GitHub

And no — this isn’t “yet another Django helper library.”

The goal was much more ambitious:

Make Django predictable, typed, async-first, and safe enough for large-scale engineering teams without abandoning Django itself.

Sounds dramatic?

Maybe.

But let’s break down why this problem even exists.

The Hidden Problem Inside Django Nobody Talks About

Django was born in 2005.

Back then, Python didn’t have:

- typing
asyncio
Pydantic
modern validation systems
static analysis tooling like Pyright or Ruff

The framework evolved, but its core architecture still carries assumptions from a pre-type-hint era.

And this creates a weird situation.

Your stack today probably looks like this:

Django + DRF + Pydantic + Celery + async APIs + mypy

But internally?

You’re still fighting:

duplicated schemas
weak runtime guarantees
dynamic model fields
serializer inconsistencies
sync/async boundary issues
hidden ORM side effects

You validate data in:

Django Forms
DRF Serializers
Pydantic models
database constraints

Four layers.

Same data.

Different logic.

And eventually they drift apart.

One field becomes optional in one place but required in another.

One validator changes but another doesn’t.

One API silently accepts invalid payloads.

Congratulations — you’ve created distributed chaos inside a monolith.

So What Is Django Nova?

At its core, Django Nova is an async-first typed architecture layer for Django 5+.

It combines ideas from:

Pydantic v2
FastAPI
modern typed Python
declarative validation systems
repository/service architectures

but keeps Django where Django is strongest:

ORM
migrations
admin
ecosystem

The philosophy is simple:

Keep Django’s productivity. Remove Django’s unpredictability.

That means:

strongly typed models
unified validation
async-compatible workflows
safer service boundaries
explicit contracts between layers

And honestly?

That changes the developer experience more than people expect.

Why Validation Duplication Is a Bigger Disaster Than Performance

Most teams obsess over milliseconds.

But production incidents usually come from something much simpler:

Bad data.

Imagine this payload:

{ "email": "not-an-email", "age": -5 }

Now imagine:

frontend validates differently
DRF serializer partially validates
model allows save()
Celery task crashes later
analytics pipeline breaks silently

This is how engineering debt grows.

Not with explosions.

With tiny inconsistencies.

Django Nova uses a unified validation pipeline inspired by Pydantic v2’s strict parsing model.

Instead of scattering validation everywhere, you define contracts once.

Example:

class UserCreate(Schema): email: str age: int @field_validator("age") def validate_age(cls, v): if v < 0: raise ValueError("Age must be positive") return v

Now validation becomes:

reusable
typed
deterministic
IDE-friendly
testable

And yes, your autocomplete suddenly feels like magic.

Under the Hood: How Django Nova Actually Works

This is where things get interesting.

Django Nova isn’t trying to replace Django ORM.

That would be pointless.

Instead, it wraps Django’s weakest architectural seams with stricter abstractions.

The project focuses on:

1. Typed schemas

Inspired by Pydantic v2 internals.

Instead of dynamic serializer behavior, Nova introduces explicit contracts between:

API layer
services
ORM
background jobs

This dramatically reduces runtime ambiguity.

2. Async-first architecture

Django added async support gradually.

But anyone who has mixed sync ORM calls with async views knows the pain:

SynchronousOnlyOperation

The classic “surprise, you touched the database wrong” moment.

Nova encourages explicit async boundaries and cleaner async orchestration patterns.

That matters because modern apps increasingly rely on:

WebSockets
streaming APIs
AI inference
event-driven systems
real-time dashboards

Async is no longer “optional future architecture.”

It’s normal backend engineering.

3. Service-oriented structure

Typical Django apps become “fat models + mystery utils.py.”

Nova pushes projects toward explicit service layers:

UserService.create_user() PaymentService.capture() OrderService.complete()

This sounds boring until your project reaches 100k+ LOC.

Then suddenly architecture matters more than syntax.

Performance? Yes, That Too

Let’s talk numbers.

One misconception about typed systems is that they’re “slow.”

Actually, modern typed validation engines can outperform traditional serializer-heavy pipelines because they:

avoid excessive dynamic introspection
reduce runtime ambiguity
optimize parsing paths
leverage compiled validation internals

Especially with Python 3.12+, the gap becomes noticeable.

Django Nova targets:

Python 3.12+
Django 5+
async-native workflows

Meaning it’s built for modern CPython optimizations from day one.

And unlike many “enterprise abstractions,” it doesn’t require rewriting your entire stack.

You can introduce it incrementally.

Which is probably the only reason developers will actually adopt it.

Because let’s be honest:

Nobody wants another “complete rewrite migration strategy.”

Where Django Nova Fits in the Market

This is the interesting part.

Django Nova sits somewhere between:

Framework Philosophy
Django REST Framework flexible but highly dynamic
FastAPI typed and async-first
Pydantic validation-focused
Django Ninja API ergonomics
Litestar modern async architecture

Nova’s niche is different.

It’s not trying to replace Django with a new framework.

It’s trying to modernize Django’s engineering model itself.

That distinction matters.

Because many companies want Django:

mature ecosystem
stable ORM
admin panel
hiring availability
documentation
deployment simplicity

But they also want:

typing
async workflows
cleaner contracts
safer large-team development

Nova tries to bridge that gap.

A Real-World Use Case

Imagine you’re building:

crypto exchange backend
fintech APIs
AI orchestration platform
ERP system
marketplace
high-volume SaaS

Now imagine 15 developers touching the same models.

Without strict contracts, entropy appears fast.

One service expects:

Decimal

Another passes:

float

One async task mutates state unexpectedly.

One serializer silently casts types.

Three months later?

Nobody trusts production behavior anymore.

Django Nova exists to reduce that uncertainty.

Not through “magic.”

Through stricter architecture.

Why This Matters More in the AI Era

Here’s something people underestimate.

AI systems amplify bad backend architecture.

LLMs generate unpredictable payloads.

Agents call APIs dynamically.

Event streams become noisy.

Suddenly validation quality matters a lot.

Typed contracts stop being “nice engineering aesthetics.”

They become survival tools.

That’s one reason modern backend engineering is moving toward:

schema-first design
typed APIs
explicit contracts
deterministic validation

Nova follows that trend directly.

The Most Important Part: Developer Experience

This might sound silly, but good architecture changes how coding feels.

You stop guessing.

You stop checking documentation every 10 minutes.

Your IDE becomes useful instead of decorative.

Refactors become safer.

Autocomplete becomes smarter.

Runtime bugs move into editor-time feedback.

And yes — that’s a huge productivity upgrade.

Especially for teams.

Open Source, GitHub, and Why Stars Actually Matter

Django Nova is fully open source.

GitHub:
PyPI:

And honestly?

Open source projects live or die from community momentum.

A GitHub star sounds trivial, but it affects:

visibility
contributors
ecosystem trust
adoption
maintainer motivation

So if the architecture resonates with you:

give the repo a star
open issues
contribute code
benchmark it
challenge design decisions

That’s how strong engineering ecosystems are built.

Where This Could Go Next

The most exciting future for Django probably isn’t replacing it.

It’s evolving it.

Over the next few years, I expect the Python backend ecosystem to move heavily toward:

stricter typing
async-native architectures
schema-driven systems
compile-time guarantees
service-oriented monoliths

Frameworks that ignore this trend may slowly become harder to scale organizationally.

Not technically.

Humanly.

Because modern engineering complexity is less about servers…

…and more about coordination between developers.

That’s the real scalability problem.

And typed architecture helps solve it.

Final Thought

Django made web development accessible.

But modern backend systems demand stronger guarantees than Django originally provided.

Django Nova is my attempt to close that gap without sacrificing what made Django great in the first place.

The real question isn’t:

“Do we need more tooling?”

It’s:

“Can large Python systems stay maintainable without stronger contracts?”

I think the answer is becoming obvious.

What do you think?

Would you adopt a typed async-first architecture layer inside Django — or do you prefer keeping Django максимально flexible and dynamic?

I’m genuinely curious how teams are solving this today.

🙌 If this article was useful — follow for more deep technical breakdowns.
Leave a comment if you’ve faced similar problems in Django architecture.
And if the project looks interesting, consider starring the repo or contributing on GitHub — it genuinely helps the project grow.