DEV Community: Oluwagbade Odimayo

The Pipeline Nobody Builds After DESeq2 Finishes

Oluwagbade Odimayo — Tue, 26 May 2026 07:45:03 +0000

A production-grade differential expression pipeline with biological validation, FastAPI, and an interactive volcano plot dashboard, built on real NCBI GEO breast cancer data

GitHub: github.com/gbadedata/ngs-results-explorer
Dataset: GSE183947, Human breast cancer RNA-Seq, NCBI GEO
Stack: Python 3.12, FastAPI, Plotly Dash, pandas, pyarrow, pytest, GitHub Actions

Here is something that nobody tells you when you start learning bioinformatics data engineering.

Running the analysis is the easy part.

DESeq2 takes your count matrix and returns a table. ERBB2 shows a log2 fold change of 4.21, a p-value of 1.2e-15, a base mean of 892.3. The software ran. The numbers came out. And then you are completely on your own. How do you validate that those numbers are actually correct? How do you query ten thousand genes in under a second? How do you give a bench scientist access to these results without making them learn R? How do you know, six months later, why two records were excluded from the analysis?

These questions are not bioinformatics questions. They are data engineering questions. And they are the questions that production genomics platforms spend most of their engineering time solving.

NGS Results Explorer is my attempt to build the full answer to all of them, from scratch, on real data, with everything documented.

This is the complete technical walkthrough.

Why Breast Cancer Data From NCBI GEO

The dataset is GSE183947, a published RNA-Seq study comparing human breast cancer tumour tissue against matched normal tissue. It is publicly available on NCBI GEO. The published findings are well-characterised in the literature, which means I can validate whether the pipeline is producing biologically correct results rather than just technically correct results.

When the pipeline runs correctly, ERBB2 should be upregulated. So should MYC, ESR1, and CCNE1. BRCA1 and BRCA2 should be downregulated. PTEN should be downregulated. These are not controversial findings. If those genes are not in the correct direction with the correct magnitude, something is wrong with the pipeline, not the biology.

This is important. A lot of portfolio projects use synthetic data where there is no ground truth. You can produce any output and call it correct. Using real published data with known results means the biology itself validates the engineering.

The Architecture Before a Single Line of Code

Before writing anything, I drew out the full data flow. This discipline matters more than most tutorials suggest. When you design the architecture first, you discover constraints you would not have seen mid-implementation.

NCBI GEO (GSE183947)
     |
     v
GEO Fetcher          -- requests, tenacity, NCBI rate limiting, local JSON cache
     |
     v
Raw gene records     -- gene_id, gene_name, log2fc, pvalue, padj, base_mean
     |
     v
Validation Engine    -- 9 rules, quarantine-not-delete
     |-- Passed (51 genes) --> clean records
     |-- Failed  (2 genes) --> quarantine CSV + rejection reason + UTC timestamp
          |
          v
Data Processor       -- significance flags, regulation direction,
                        neg_log10_pvalue, log2FC bins, Parquet output
          |
          |--------------------------|
          v                          v
FastAPI REST API             Plotly Dash Dashboard
8 endpoints                  Volcano plot
Swagger /docs                Gene table
Pydantic v2 schemas          QC summary
Pagination + filters         Interactive filters
          |
          v
GitHub Actions CI    -- flake8, pytest, 35 tests

Five layers. Each layer has one job. Each layer hands off to the next through a clean interface. No layer reaches backwards into a previous layer. This is the structure that makes the whole thing testable, debuggable, and extensible.

The Fetcher: NCBI Rate Limits and Why Tenacity Matters

NCBI allows three API requests per second for anonymous users. Exceed that limit and your IP gets temporarily blocked. This sounds like a footnote but it is actually one of the first real engineering constraints you hit when working with genomics data at scale.

The solution is a 340-millisecond delay between requests, which keeps the rate comfortably below three per second, combined with exponential back-off retry logic via the tenacity library.

from tenacity import retry, stop_after_attempt, wait_exponential

@retry(
    stop=stop_after_attempt(3),
    wait=wait_exponential(multiplier=1, min=2, max=10),
)
def _fetch_url(url: str) -> requests.Response:
    headers = {"User-Agent": f"NGSResultsExplorer/1.0 ({settings.ncbi_email})"}
    response = requests.get(url, headers=headers, timeout=30)
    response.raise_for_status()
    return response

The @retry decorator handles the retry logic automatically. If the first request fails, tenacity waits two seconds, then four, then eight. You write the function once and the library handles the failure modes.

The fetcher also implements a local JSON cache. After the first successful fetch, the raw data is written to data/raw/GSE183947_raw.json. Every subsequent run loads from cache rather than hitting the API again. This matters during development when you are running the pipeline dozens of times and do not want to burn your API quota on each iteration.

cache_file = os.path.join(settings.raw_data_path, f"{accession}_raw.json")

if os.path.exists(cache_file):
    log.info("loading_from_cache", file=cache_file)
    with open(cache_file) as f:
        return json.load(f)

The immutability of the cache is intentional. The raw data directory is in .gitignore and written once. If I later discover a bug in the transformation logic, I can reprocess the original data without re-fetching. This is the same principle as an immutable Bronze zone in a medallion architecture.

The Validation Engine: Nine Rules and Why Quarantine-Not-Delete Is the Only Correct Approach

This is the component that took the most thought to design correctly.

When a record fails validation, you have three options. Delete it silently. Reject the entire batch. Route it to a quarantine zone with a documented rejection reason.

Silent deletion is the worst option in any regulated or semi-regulated environment. You lose data and you lose the ability to explain why data is missing. Six months later, when a scientist asks why a particular gene is absent from the results, you have no answer.

Batch rejection is too aggressive. If one record in a batch of fifty has a missing gene symbol, rejecting all fifty throws away forty-nine correct records to punish one bad one.

Quarantine-not-delete is the correct approach. Failed records are preserved exactly as received, with the rejection reason attached, with a UTC timestamp, in a dedicated CSV file. Scientists can inspect the quarantine zone, identify the upstream data quality problem, and decide whether to fix and reprocess.

Here is the structure of each validation rule:

from dataclasses import dataclass

@dataclass
class ValidationResult:
    passed: bool
    rule: str
    reason: str

def rule_gene_id_format(record: dict) -> ValidationResult:
    gene_id = record.get("gene_id", "")
    passed = bool(ENSEMBL_PATTERN.match(gene_id))
    return ValidationResult(
        passed=passed,
        rule="gene_id_format",
        reason="" if passed else (
            f"gene_id '{gene_id}' does not match ENSG + 11 digits pattern"
        ),
    )

Every rule is a pure function. It takes a dictionary and returns a structured result. This makes each rule independently testable. You do not need to set up the entire pipeline to test whether the Ensembl ID format rule correctly rejects INVALID_ID_001. You call the function directly in a unit test.

The nine rules cover four categories:

Identity rules: gene_id not null, Ensembl ID format (ENSG followed by exactly eleven digits), gene_name not null.

Statistics rules: pvalue in [0, 1], padj in [0, 1], log2FC in [-50, 50].

Metrics rules: base_mean greater than or equal to zero.

Completeness rules: condition present, accession present.

In our run, two records were quarantined. One had the gene_id INVALID_ID_001 which failed the Ensembl format check. One had an empty gene_name field. Both are real data quality issues that occur in public GEO submissions.

INVALID_ID_001  |  gene_id 'INVALID_ID_001' does not match ENSG + 11 digits pattern
ENSG00000099999 |  gene_name is null or empty

Without the validation engine, both records would have silently entered the downstream analysis. The first would have caused join failures on any operation that assumed Ensembl IDs. The second would have produced unlabelled data points on the volcano plot.

The Two-Threshold Significance Criterion

This is one of the most important design decisions in the processor, and it reflects real RNA-Seq analysis practice.

A gene is considered differentially expressed if and only if two conditions are both true simultaneously:

The adjusted p-value is below 0.05 (standard FDR threshold)
The absolute log2 fold change is greater than 1.0 (corresponding to a two-fold change)

Why both? Because each threshold alone is insufficient.

Statistical significance alone (padj < 0.05) will flag genes whose expression difference is real but biologically trivial. A gene that changes by 3% across all samples with extremely low variance will be statistically significant. Nobody cares about a 3% change.

Biological significance alone (|log2FC| > 1.0) will flag genes whose fold change is large but so variable across samples that the estimate is unreliable. A gene that goes from near-zero to slightly-above-zero in three samples will show an enormous fold change with a terrible p-value.

The combination filters to genes with both a meaningful effect size and high statistical confidence. This is why the field converged on both thresholds.

PADJ_THRESHOLD = 0.05
LOG2FC_THRESHOLD = 1.0

significant = (padj < PADJ_THRESHOLD) and (abs(log2fc) > LOG2FC_THRESHOLD)

if significant and log2fc > 0:
    regulation = "upregulated"
elif significant and log2fc < 0:
    regulation = "downregulated"
else:
    regulation = "not_significant"

Note the strict inequality on the padj threshold: padj < 0.05, not padj <= 0.05. A gene with padj exactly equal to 0.05 is not considered significant. This is the correct interpretation of the threshold.

The Volcano Plot Coordinate

The volcano plot is the standard visualisation for RNA-Seq DE results. Understanding how the coordinates are computed is important for building it correctly.

The x-axis is log2 fold change. This is already in the data.

The y-axis is the negative base-10 logarithm of the p-value. This transformation is what gives the volcano its shape. Genes with very small p-values (very significant) get very large negative logarithms, which means they plot at the top of the chart. Genes with large p-values plot near the bottom.

pvalue_floor = max(pvalue, 1e-300)
neg_log10_pvalue = round(-math.log10(pvalue_floor), 4)

The floor at 1e-300 prevents numerical overflow. Some genes in large RNA-Seq studies have p-values so small that the logarithm would overflow a float. The floor is set well below any biologically relevant threshold, so it does not affect interpretation.

The result: ERBB2, with a p-value of 1.2e-15, gets a y-coordinate of approximately 14.9. MYC, with a p-value of 1.8e-22, plots at approximately 21.7. The most significant genes rise to the top of the plot exactly as expected.

The FastAPI Layer: Async, Schemas, and Why the Health Endpoint Is Not Optional

The API is built with FastAPI using async/await throughout. For a web API that primarily waits for data from disk, async matters. A synchronous Flask application blocks while waiting for the data load to complete. FastAPI's async design allows multiple requests to be handled concurrently without blocking.

Every endpoint has a Pydantic v2 response schema. This is not optional boilerplate. When FastAPI serialises a response, it validates every field against the schema. If a field is missing or has the wrong type, FastAPI raises a validation error at the API boundary rather than returning malformed JSON that propagates silently to the client.

class GeneRecord(BaseModel):
    gene_id: str
    gene_name: str
    log2fc: float
    pvalue: float
    padj: float
    base_mean: float
    regulation: str
    significant: bool
    neg_log10_pvalue: float
    abs_log2fc: float
    log2fc_bin: str
    completeness_score: float

The /health endpoint deserves specific discussion because many tutorials treat it as a throwaway diagnostic tool. In production, it is a critical operational component.

Load balancers poll /health continuously to determine whether a service is functioning. If /health returns a non-200 response, the load balancer stops routing traffic to that instance. Our implementation checks data availability and returns a structured response that includes the total gene count and dataset accession.

@app.get("/health", response_model=HealthResponse, tags=["System"])
def health():
    try:
        df = get_df()
        summary = get_summary()
        return HealthResponse(
            status="healthy",
            version=settings.app_version,
            total_genes=len(df),
            significant_genes=int(df["significant"].sum()),
            accession=summary["accession"],
        )
    except Exception as e:
        log.error("health_check_failed", error=str(e))
        raise HTTPException(status_code=503, detail="Data unavailable")

If the data fails to load, the endpoint returns 503 with a status: degraded signal rather than crashing. This is graceful degradation. The API reports its own health problem rather than failing silently.

The Volcano Plot Dashboard

The dashboard is the most visually impressive component and the one that most directly mirrors what commercial NGS platforms like Basepair sell to researchers.

The key design decisions for the volcano plot:

Point colour by regulation direction. Red for upregulated, blue for downregulated, grey for not significant. This colour convention is standard in the field and any RNA-Seq biologist will immediately understand it.

Point size proportional to absolute fold change. Genes with larger effect sizes are visually larger. This encodes an additional dimension of information without adding clutter.

Threshold lines. Dashed lines at padj = 0.05 on the y-axis and at |log2FC| = 1.0 on the x-axis divide the plot into four quadrants. Only the upper-left (downregulated significant) and upper-right (upregulated significant) quadrants contain genes that meet both thresholds.

Gene labels on the top hits. The eight most significant genes are labelled with their gene symbols. This is what biologists actually want to see. The famous genes should be identifiable at a glance.

Hover tooltips. Every data point shows gene name, log2FC, -log10(p), and adjusted p-value on hover. This allows drilling into any gene without leaving the dashboard.

The two filter controls govern the entire dashboard simultaneously via Dash callbacks. When the regulation dropdown is changed, both the volcano plot and the gene table update at the same time. When the log2FC slider is moved, genes below the threshold disappear from both panels. This is how production dashboards work.

Structured Logging: The Difference Between Debugging and Guessing

Every log entry in this pipeline is a JSON object. Not a human-readable sentence.

log.info(
    "validation_complete",
    total=total,
    passed=len(clean),
    quarantined=len(quarantined),
    pass_rate=f"{pass_rate}%",
    quarantine_rate=f"{quarantine_rate}%",
    avg_completeness=avg_completeness,
)

The output looks like this:

2026-05-22 13:34:08 [info] validation_complete  avg_completeness=1.0 pass_rate=96.23% passed=51 quarantine_rate=3.77% quarantined=2 total=53

Every field is a key-value pair. In a production environment using CloudWatch, you can write a metric filter that extracts the quarantine_rate field and triggers an alarm when it exceeds five percent. You cannot do that with a free-text log line that says "2 records failed validation."

The second pattern worth noting is the correlation ID. Every pipeline run gets a unique UUID attached to all its log entries. When multiple pipeline runs are executing concurrently and their log lines are interleaved, you can filter by correlation ID to see exactly what happened in a specific run, in order.

The Results: Biological Validation

51 records passed validation. 2 were quarantined. 20 of the 51 were statistically significant (padj < 0.05 and |log2FC| > 1.0).

The top upregulated genes: CCNE1 (log2FC = 4.56), ERBB2 (4.21), MYC (3.89), ESR1 (3.67), CDC20 (3.23).

The top downregulated genes: BRCA1 (log2FC = -3.14), PTEN (-3.01), NF1 (-2.87), BRCA2 (-2.56), APC (-2.45).

Every one of these is biologically correct. ERBB2 encodes the HER2 receptor, which is amplified in approximately 20% of breast cancers. The HER2-amplified subtype is one of the most studied breast cancer subtypes in the world. BRCA1 and BRCA2 are the most studied inherited breast cancer predisposition genes in the field. PTEN loss is a well-established driver of the PI3K pathway central to breast cancer progression.

The pipeline produced scientifically coherent results. This is not a given. If the validation logic had incorrectly quarantined records, or the significance thresholds had been applied incorrectly, the gene list would not match the published literature. The biology validates the engineering.

Tests: 35 Passing, 84% Coverage on the Most Critical Module

The test suite has 35 unit tests across the validator and processor. The validator has 84% coverage because it is the most critical module. A bug in the validation logic corrupts everything downstream.

class TestGeneIdFormat:
    def test_valid_ensembl_id(self, valid_record):
        result = rule_gene_id_format(valid_record)
        assert result.passed is True

    def test_invalid_format(self, invalid_id_record):
        result = rule_gene_id_format(invalid_id_record)
        assert result.passed is False
        assert "ENSG" in result.reason

    def test_too_short(self, valid_record):
        valid_record["gene_id"] = "ENSG0000014"
        result = rule_gene_id_format(valid_record)
        assert result.passed is False

Because each validation rule is a pure function, every test is a direct function call with no mocking required. You can test edge cases exhaustively without complex setup. This is why the pure function design matters beyond aesthetics.

The GitHub Actions CI pipeline runs flake8 and pytest on every push. A green CI badge means the code meets a defined quality standard automatically, not just when someone remembers to run the tests locally.

What This Taught Me About Production Bioinformatics

Three things stand out from building this.

Data quality is the most underestimated problem. Two records were quarantined from a dataset of 53. That is nearly four percent. In a dataset of 50,000 records, that is 2,000 problematic records that would have silently corrupted downstream analysis. The validation engine is not a nice-to-have. It is the difference between trustworthy results and untrustworthy results.

The biological context makes the engineering meaningful. Writing a generic data validation library is one thing. Writing one where the rules reflect actual biological constraints (Ensembl ID format, p-value bounds, base mean non-negativity) connects the engineering to the science. The rules are not arbitrary. Each one catches a specific class of error that has been observed in real sequencing data submissions.

The data engineering layer is where platforms compete. DESeq2 is free and runs anywhere. The value in commercial NGS platforms is not the statistics. It is the pipeline around the statistics: the validation, the annotation, the queryability, the accessibility to scientists who cannot code. Understanding that distinction is what separates a bioinformatician who can run tools from a bioinformatics data engineer who can build platforms.

Running It Yourself

git clone https://github.com/gbadedata/ngs-results-explorer.git
cd ngs-results-explorer
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
cp .env.example .env  # add your email

python3 -m src.fetcher
python3 -m src.validator
python3 -m src.processor

# API at http://localhost:8000/docs
uvicorn src.api:app --reload --port 8000

# Dashboard at http://localhost:8052
python3 -m src.dashboard

# Tests
pytest -v

The fetcher will pull from NCBI GEO on first run and cache locally. Every subsequent run loads from cache. The entire pipeline from fetch to dashboard takes under five seconds on a local machine after the first run.

What Is Next

The natural extension of this project is connecting the data engineering layer to a real DESeq2 output. Right now the pipeline ingests pre-processed DE results. The next step is building the upstream: taking raw count matrices from NCBI GEO, running differential expression programmatically via rpy2, and feeding the results directly into this pipeline.

The second extension is adding a gene ontology enrichment analysis layer. After identifying the significant genes, the natural question is: what biological processes are enriched in this gene set? Adding GO term analysis and visualisation would complete the picture from raw counts to biological interpretation.

If you build on this, I would genuinely like to know what you find. The repository is at github.com/gbadedata/ngs-results-explorer.

Oluwagbade Odimayo is a data professional with an MSc in Applied Data Science and a background in Genetics and Molecular Biology. This project is part of a portfolio of production-grade genomics data engineering work.

1000 Genomes Data Is Free. Making It Useful Is the Hard Part

Oluwagbade Odimayo — Tue, 26 May 2026 07:32:37 +0000

A complete technical walkthrough of VariantLens - a production-grade VCF parsing, validation, annotation, and visualisation pipeline built on live chromosome 22 data from the 1000 Genomes Project Phase 3

GitHub: github.com/gbadedata/variantlens
Dataset: 1000 Genomes Project Phase 3, Chromosome 22, GRCh37/hg19
Stack: Python 3.12, FastAPI, Plotly Dash, pandas, pyarrow, pytest, GitHub Actions

The 1000 Genomes Project sequenced 2,504 human beings from 26 populations across the world and made every variant freely downloadable. The chromosome 22 file alone contains approximately one million single nucleotide polymorphisms and insertions and deletions, each described in a format called VCF that was designed in 2010 and has not fundamentally changed since.

I built a pipeline that streams this data live from the European Bioinformatics Institute FTP server, parses each variant into structured records, applies nine biological validation rules, annotates every clean variant with functional consequence and clinical gene context, and serves the results through a REST API and an interactive dashboard.

This article is the full technical account of how and why.

The VCF Format: What It Is and Why Parsing It Correctly Is Non-Trivial

VCF stands for Variant Call Format. Every variant caller, GATK, FreeBayes, bcftools, outputs VCF. Understanding the format is prerequisite to understanding everything else in this article.

A VCF file has two sections. The header section starts with ## and contains metadata about the file: the reference genome, the tools used, the INFO field definitions. The data section starts after the #CHROM line and contains one row per variant.

Each data row has eight mandatory tab-separated fields:

Field	Example	Description
CHROM	22	Chromosome
POS	16050075	1-based position
ID	.	rsID or dot if novel
REF	A	Reference allele
ALT	G	Alternate allele
QUAL	100	Phred quality score
FILTER	PASS	Filter status
INFO	AF=0.0002;DP=8012	Semicolon-separated metadata

After the eight mandatory fields, the 1000 Genomes VCF adds a FORMAT field and then 2,504 sample columns, one per individual. This means a single data row in the 1000 Genomes VCF has 2,512 tab-separated fields.

The naive approach is to split every line on tabs and process all 2,512 fields. This is slow and wastes memory. The correct approach is to split on tabs and take only the first eight fields, discarding the sample columns entirely since we only need population-level statistics, which are encoded in the INFO field as AF (allele frequency).

def parse_vcf_line(line: str, line_index: int = 0) -> dict:
    fields = line.strip().split("\t")

    if len(fields) < 8:
        return {
            "parse_error": True,
            "error_reason": f"Expected at least 8 fields, got {len(fields)}",
            "raw_line": line[:200],
            "line_index": line_index,
        }

    chrom = fields[0].lstrip("chr")  # normalise chr22 to 22
    pos_str = fields[1]
    variant_id = fields[2]
    ref = fields[3]
    alt = fields[4]
    qual_str = fields[5]
    filter_status = fields[6]
    info_str = fields[7]
    # fields[8:] are FORMAT and 2,504 sample columns -- discarded

The lstrip("chr") normalises chromosome identifiers. Some VCF files prefix chromosomes with chr (chr22, chrX) and some do not (22, X). Normalising to the unprefixed form makes downstream logic simpler.

The INFO field parser deserves its own function because the format is complex. Values are semicolon-separated key=value pairs, but some entries are flags with no value.

def parse_info_field(info_str: str) -> dict:
    result = {}
    if not info_str or info_str == ".":
        return result

    for token in info_str.split(";"):
        token = token.strip()
        if not token:
            continue
        if "=" in token:
            key, _, value = token.partition("=")
            result[key.strip()] = value.strip()
        else:
            result[token] = True  # flag with no value

    return result

The key INFO subfields we extract are AF (allele frequency across all 2,504 individuals), DP (total read depth at this position), MQ (root mean square mapping quality), AN (total alleles in called genotypes), AC (allele count), and VT (variant type annotation from 1000 Genomes).

Streaming a Gzipped File Over HTTP Without Loading It Into Memory

The chromosome 22 VCF from the 1000 Genomes EBI mirror is approximately 11GB when uncompressed and roughly 1.2GB as a gzip file. Downloading the entire file before parsing is unnecessary and slow.

Python's requests library supports streaming responses. Combined with the gzip module, you can decompress and process the file line by line without ever holding more than a few lines in memory.

def _fetch_vcf_lines(url: str, max_variants: int) -> tuple:
    headers = {"User-Agent": f"VariantLens/1.0 ({settings.ncbi_email})"}

    variant_lines = []
    header_lines = []

    with requests.get(url, headers=headers, stream=True, timeout=60) as resp:
        resp.raise_for_status()
        with gzip.GzipFile(fileobj=resp.raw) as gz:
            for raw_line in gz:
                try:
                    line = raw_line.decode("utf-8").rstrip("\n")
                except UnicodeDecodeError:
                    continue

                if line.startswith("##"):
                    header_lines.append(line)
                    continue
                if line.startswith("#CHROM"):
                    header_lines.append(line)
                    continue
                if not line.strip():
                    continue

                variant_lines.append(line)
                if len(variant_lines) >= max_variants:
                    break

    return variant_lines, header_lines

The stream=True argument tells requests not to download the body immediately. The gzip.GzipFile(fileobj=resp.raw) wraps the raw socket in a gzip decompressor. Every iteration of the for loop decodes one line of the VCF, processes it, and moves on. Once we have collected max_variants data lines, we break and close the connection.

The result: streaming 500 variants from a multi-gigabyte file takes under two seconds. The memory footprint stays constant regardless of the total file size.

The Ts/Tv Ratio: The Most Important Number in SNP Calling Quality

Before building the validation engine, it is worth understanding the single most important quality metric in variant calling, because this pipeline computes and displays it as a dashboard KPI.

The transition-to-transversion ratio, universally called Ts/Tv, measures the ratio of two types of SNP substitutions in a dataset.

Transitions are substitutions between chemically similar bases. Purines substitute for purines (A changes to G, G changes to A) and pyrimidines substitute for pyrimidines (C changes to T, T changes to C). These are A-to-G, G-to-A, C-to-T, T-to-C.

Transversions are substitutions between chemically dissimilar bases. A purine changes to a pyrimidine or vice versa. These are A-to-C, A-to-T, G-to-C, G-to-T, and their reverses.

In real human genomic data, transitions are approximately twice as common as transversions because of the underlying chemistry of DNA mutation. Transitions are more likely to occur spontaneously and more likely to be neutral (synonymous at the protein level). The expected Ts/Tv ratio for high-quality whole-genome SNP calls is 2.0 to 2.1. Exome calls are typically 2.5 to 3.0 because exonic regions are under stronger selection against transversions.

If your Ts/Tv ratio is below 1.8, you have a problem. The most common causes are a high false-positive rate in the variant caller (random errors are equally likely to be transitions or transversions, which would push the ratio toward 0.5) or contamination from another species.

The 1000 Genomes data produces a Ts/Tv ratio of 1.75 in our pipeline, slightly below the expected range but within the plausible zone for a subset of variants. This number is computed automatically and displayed on the dashboard.

def is_transition(ref: str, alt: str) -> bool:
    transitions = {
        frozenset({"A", "G"}),
        frozenset({"C", "T"}),
    }
    pair = frozenset({ref.upper(), alt.upper()})
    return pair in transitions

Using frozenset is elegant here. The pair A/G is the same as G/A for the purposes of this classification. frozenset({"A", "G"}) == frozenset({"G", "A"}) is True. No need for two separate conditions.

The Nine Validation Rules

The validation engine applies nine rules to every parsed VCF record. Same architecture as the validation engine in NGS Results Explorer: pure functions, structured results, quarantine-not-delete.

The rules reflect real data quality issues that occur in production VCF submissions:

ALL_RULES = [
    rule_chromosome_valid,      # must be a recognised human chromosome
    rule_position_positive,     # must be a positive 1-based integer
    rule_ref_allele_valid,      # non-empty, valid nucleotide bases only
    rule_alt_allele_valid,      # non-empty, differs from REF
    rule_quality_score_valid,   # QUAL in [0, 100000]
    rule_filter_field_present,  # FILTER must exist
    rule_allele_frequency_valid, # AF in [0, 1] if present
    rule_depth_non_negative,    # DP >= 0 if present
    rule_ref_not_equal_alt,     # REF != ALT
]

The last rule deserves explanation. A VCF record where REF equals ALT is not a variant. It is a monomorphic site: a position where the sequenced individual has the same allele as the reference genome. These are sometimes emitted by variant callers when running in GVCF mode (genomic VCF, which records all sites including non-variant ones). If you are processing a VCF that was not intended to be a GVCF, these records are errors.

We deliberately inject five bad records into the pipeline to test the validation engine:

bad_records = [
    "22\t-1\tBAD_POS_001\tA\tG\t500.0\tPASS\tAF=0.01;DP=50",       # negative position
    "22\t25000000\tBAD_REF_001\t\tG\t300.0\tPASS\tAF=0.02;DP=80",   # empty REF
    "22\t26000000\tBAD_QUAL_001\tC\tT\t-50.0\tPASS\tAF=0.03;DP=60", # negative QUAL
    "22\t27000000\tBAD_AF_001\tA\tC\t800.0\tPASS\tAF=1.5;DP=120",   # AF > 1
    "22\t28000000\tBAD_SAME_001\tG\tG\t600.0\tPASS\tAF=0.01;DP=90", # REF == ALT
]

All five are correctly quarantined with precise rejection reasons. 500 of 505 records pass.

The Annotation Engine: From Coordinates to Biology

Annotation is where a raw VCF record becomes interpretable. A variant at position 30,100,000 on chromosome 22 means nothing to a biologist. A variant at position 30,100,000 in the NF2 gene, classified as a missense substitution, in a gene associated with neurofibromatosis type 2, in a sample with a minor allele frequency of 0.001, is a candidate for clinical follow-up.

The annotation engine computes seven fields for every clean variant:

variant_class classifies the variant as SNP, insertion, deletion, MNP (multi-nucleotide polymorphism), or complex based on the lengths of the REF and ALT alleles.

def classify_variant_type(ref: str, alt: str) -> str:
    alt_first = alt.split(",")[0].strip()
    ref_len = len(ref)
    alt_len = len(alt_first)

    if ref_len == 1 and alt_len == 1:
        return "SNP"
    elif alt_len > ref_len:
        return "insertion"
    elif ref_len > alt_len:
        return "deletion"
    elif ref_len == alt_len and ref_len > 1:
        return "MNP"
    else:
        return "complex"

ts_tv classifies each SNP as a transition or transversion using the function shown earlier. Non-SNP variants get not_applicable.

consequence assigns a functional consequence category. In a full clinical pipeline, this uses VEP (Ensembl Variant Effect Predictor) or SnpEff with transcript databases. VariantLens applies rule-based classification: frameshift if the indel length is not divisible by three, inframe if it is, synonymous or missense for SNPs based on a deterministic random assignment seeded by the variant coordinates.

gene assigns a gene name based on chromosomal position using a coordinate map derived from Ensembl GRCh37 release 87. The map covers the clinically significant genes on chromosome 22.

CHR22_GENE_MAP = [
    (17000000, 20500000, "DGCR8", "DiGeorge syndrome"),
    (19700000, 19800000, "TBX1", "DiGeorge/velocardiofacial syndrome"),
    (21700000, 21900000, "CRKL", "DiGeorge syndrome"),
    (23400000, 23700000, "BCR", "Chronic myelogenous leukaemia"),
    (23500000, 23700000, "ABL1", "Chronic myelogenous leukaemia"),
    (30000000, 30200000, "NF2", "Neurofibromatosis type 2"),
    (41900000, 42100000, "SMAD4", "Juvenile polyposis, colorectal cancer"),
    # ...
]

BCR and ABL1 appear on chromosome 22. In chronic myelogenous leukaemia, a chromosomal translocation fuses BCR on chromosome 22 with ABL1 on chromosome 9, creating the Philadelphia chromosome and the BCR-ABL1 fusion oncogene. This fusion is the target of imatinib (Gleevec), one of the most celebrated successes in targeted cancer therapy. Any variant near these coordinates is clinically relevant.

af_tier classifies allele frequency into three tiers reflecting population genetics conventions:

def annotate_af_tier(af: float) -> str:
    if af < 0.01:
        return "rare"         # < 1% population frequency
    elif af < 0.05:
        return "low_frequency" # 1-5%
    else:
        return "common"       # > 5% -- above GWAS MAF threshold

In our dataset, 446 of 500 variants (89.2%) are rare. This is the expected distribution. The 1000 Genomes Project captured the full spectrum of human variation, and the vast majority of it is rare variation that exists in only a small fraction of the population.

dbsnp_status marks each variant as known (has an rsID from the dbSNP database) or novel. Novel variants are more likely to be either sequencing artefacts or genuinely rare private variants. In a clinical setting, novel variants require more careful interpretation.

filter_outcome summarises whether the variant passed the variant caller's filter as pass or filtered.

Parquet: Why Columnar Storage Matters for Variant Data

After validation and annotation, the 500 clean records are saved to Parquet format.

Parquet is a columnar storage format. Traditional row-based formats like CSV store all fields for record one, then all fields for record two, and so on. Parquet stores all values of field one together, then all values of field two, and so on.

This distinction matters enormously for analytical queries. Consider a query that asks: how many variants have a consequence of missense? With CSV, the database must read every field of every record to find the consequence column. With Parquet, the database reads only the consequence column, skipping all other columns entirely.

For a table with 30 fields, Parquet can be 30 times faster for this type of column-selective query. At production scale, with millions of variants across thousands of samples, this difference is between a dashboard that responds in milliseconds and one that times out.

df.to_parquet(parquet_path, index=False)

The single line above writes the entire annotated DataFrame to a Parquet file that can be read by pandas, Apache Spark, Athena, BigQuery, and any other modern analytical engine. The format also stores column statistics (min, max, null count) in the file footer, which allows query engines to skip entire file sections when those statistics rule out matching records.

The API: Eight Endpoints and One That Only Genomics Platforms Have

The API has eight endpoints. Seven of them are standard data engineering fare: health, summary, paginated list, single record lookup, distributions, and quarantine.

The eighth is the /clinical endpoint, and it is specific to genomics platforms.

@app.get("/clinical", response_model=list[ClinicalVariant], tags=["Clinical"])
def clinical_variants():
    """
    Variants located in disease-associated genes on chromosome 22.
    Genes include BCR, ABL1, NF2, SMAD4, EP300, TBX1, SHANK3.
    """
    summary = get_summary()
    return summary.get("clinical_variants", [])

This endpoint returns every variant that falls within a gene known to be associated with a human disease. In a clinical genomics workflow, these are the variants that require manual review by a molecular pathologist. Surfacing them through a dedicated endpoint reflects real clinical sequencing practice.

The /variants endpoint has five filter parameters:

@app.get("/variants", response_model=list[VariantRecord], tags=["Variants"])
def list_variants(
    variant_class: Optional[str] = Query(None),
    consequence: Optional[str] = Query(None),
    af_tier: Optional[str] = Query(None),
    gene: Optional[str] = Query(None),
    clinical_only: bool = Query(False),
    pass_only: bool = Query(True),
    limit: int = Query(50, ge=1, le=500),
    offset: int = Query(0, ge=0),
):

The pass_only parameter defaults to True, which means the default response only includes variants that passed the variant caller's filter. This is the correct default for clinical use. Filtered variants exist in the dataset but require explicit opt-in to access.

The Dashboard: Six Panels Built for Genomics Researchers

The dashboard is designed for the actual audience: bioinformaticians and molecular biologists who understand what they are looking at.

The consequence distribution chart uses a biologically meaningful colour scheme. Synonymous variants are green because they do not change the protein sequence. Missense variants are orange because they change one amino acid and may or may not be functional. Nonsense and frameshift variants are red because they disrupt the protein entirely. This colour convention aligns with the clinical variant classification used in medical genetics.

The allele frequency tier chart uses blue for rare, yellow for low-frequency, and red for common. The dominance of blue in the chart immediately communicates that this is population genomics data, where most variants are rare private variants. A dataset with a different balance (more common variants) would suggest either a selected population or a different sequencing strategy.

The QUAL score histogram is the distribution of Phred-scaled quality scores across all 500 variants. A dataset with many low-quality variants would show a histogram shifted toward the left. The 1000 Genomes data is high quality and shows a distribution concentrated at higher scores.

The variant class donut shows the SNP/indel composition. 96.8% SNPs and 3.2% indels, which is consistent with the expected composition for chromosome 22 from the 1000 Genomes Project.

The QC summary panel shows every operational metric: pass rate, quarantine count, Ts/Tv ratio, completeness score, reference genome, and data source. This is the panel that tells an operator immediately whether the pipeline run was healthy.

The Results: What the Data Showed

505 records ingested (500 live from 1000 Genomes, 5 injected test cases).

500 passed validation. 5 quarantined, each with a different failure mode: negative position, empty REF, negative QUAL, AF above 1.0, REF equal to ALT.

Of the 500 clean variants:

484 SNPs (96.8%)
16 indels (3.2%)
Ts/Tv ratio: 1.75
Rare variants (AF < 1%): 446 (89.2%)
Low-frequency variants (AF 1-5%): 21 (4.2%)
Common variants (AF > 5%): 33 (6.6%)

The consequence distribution: synonymous 388, missense 46, splice_region 40, frameshift 15, nonsense 10, inframe_insertion 1.

The rare-variant dominance at 89.2% is the number that tells you this is real population genomics data. In a synthetic dataset, you would not see this distribution naturally.

Tests: 50 Passing Across Three Modules

The test suite covers three modules: the annotator (22 tests), the parser (20 tests), and the validator (8 tests).

The parser tests verify correct behaviour across VCF edge cases. What happens when QUAL is a dot? What happens when the chromosome is prefixed with chr? What happens when a line has fewer than eight fields?

class TestParseVcfLine:
    def test_chr_prefix_normalised(self):
        line = "chr22\t16050075\t.\tA\tG\t100.0\tPASS\tAF=0.001"
        result = parse_vcf_line(line)
        assert result["chrom"] == "22"

    def test_missing_qual(self):
        line = "22\t100\t.\tA\tG\t.\tPASS\tAF=0.01"
        result = parse_vcf_line(line)
        assert result["qual"] is None
        assert result["parse_error"] is False

A missing QUAL field (represented as . in VCF) is not an error. It is a valid representation of an unknown quality. The parser returns None for the qual field and False for the parse_error flag.

The annotator tests verify biological correctness. The test for NF2 gene detection is particularly important:

def test_clinical_gene_detection(self, valid_record):
    valid_record["pos"] = 30100000
    result = annotate_record(valid_record)
    assert result["is_clinical_gene"] is True
    assert result["gene"] == "NF2"

This test confirms that a variant at position 30,100,000 on chromosome 22 is correctly identified as being in the NF2 gene and flagged as clinically significant. If the gene coordinate map had a bug, this test would catch it.

What I Learned About Building for Genomics

Four things stand out.

The VCF format is deceptively complex. The eight mandatory fields are straightforward. The INFO field alone can contain dozens of subfields with different value types, multi-allelic representations, and tool-specific annotations. A robust parser needs to handle all of this gracefully, including the case where an expected subfield is absent.

Streaming matters more than you think. The naive approach of downloading the entire file before processing would work for small datasets. For the 1000 Genomes chromosome 22 VCF at 11GB uncompressed, it would take minutes and require significant memory. Streaming reduces the latency to seconds and the memory footprint to a constant.

Biological context is not optional. The Ts/Tv ratio, the allele frequency tiers, the clinical gene map, the consequence classification: these are not decorative. They are the domain knowledge that separates a genomics data platform from a generic data platform. Building them correctly requires understanding the biology, not just the engineering.

The data engineering is where platforms differentiate. GATK is free. The 1000 Genomes data is free. The variant calling is done. The value in a commercial variant analysis platform is everything that sits downstream: the parsing, the validation, the annotation, the queryability, the clinical reporting, the dashboard. That is the data engineering problem, and it is hard.

Running It Yourself

git clone https://github.com/gbadedata/variantlens.git
cd variantlens
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
cp .env.example .env  # add your email

python3 -m src.fetcher      # streams from 1000 Genomes live
python3 -m src.parser       # parses VCF, computes Ts/Tv
python3 -m src.validator    # validates, writes quarantine
python3 -m src.annotator    # adds consequence, gene, AF tier
python3 -m src.processor    # builds Parquet, writes summary

# API at http://localhost:8001/docs
uvicorn src.api:app --reload --port 8001

# Dashboard at http://localhost:8050
python3 -m src.dashboard

pytest -v

The fetcher streams from the live EBI server on first run and caches locally. Subsequent runs use the cache.

Where This Goes Next

The obvious extension is full VEP annotation. Running the Ensembl Variant Effect Predictor against the pipeline output would replace the rule-based consequence classification with transcript-level annotation, canonical splice site detection, and population frequency data from gnomAD. This would bring the annotation quality to clinical-grade.

The second extension is multi-chromosome support. The current pipeline is hardcoded to chromosome 22. Generalising to any chromosome, or to whole-genome VCF input, would require partitioning the output by chromosome and building an index for fast position-based queries.

The third extension is clinical report generation. The /clinical endpoint already identifies variants in disease-associated genes. Converting this to a structured PDF clinical variant report, with ACMG classification guidelines applied automatically, would bring the project into clinical utility territory.

If this project is useful to you or you build on it, I would like to know. The repository is at github.com/gbadedata/variantlens.

From Broken Repo to Policy-Gated Deployment Platform: Building SwiftDeploy from Scratch

Oluwagbade Odimayo — Thu, 07 May 2026 01:20:22 +0000

How I built a manifest-driven CLI that generates its own infrastructure, enforces environment policy through OPA, observes the running system in real time, and audits every decision it makes

Most deployment tools ask you to write configuration files. SwiftDeploy asks you to describe your intent once, then writes the configuration files for you - and refuses to deploy unless the environment is safe enough to proceed.

This post covers the complete journey of building SwiftDeploy across two stages. Stage A established the foundation: a declarative CLI that generates Docker Compose and Nginx configuration from a single manifest, manages the container lifecycle, and supports stable/canary promotion. Stage B added the intelligence layer: Prometheus metrics, an Open Policy Agent sidecar that gates every deployment and promotion, a live status dashboard, and an append-only audit trail.

A reader who follows this post from beginning to end should be able to replicate everything.

The Problem SwiftDeploy Solves

Consider a typical deployment workflow without tooling:

Write a docker-compose.yml by hand
Write an nginx.conf by hand
Manually update both files every time a port, timeout, or service name changes
Hope you didn't introduce a drift between the two files
Have no policy gate before deployment
Have no audit trail of what changed and when

SwiftDeploy inverts this. You edit exactly one file - manifest.yaml. The CLI derives everything else from it. If you delete the generated files and run swiftdeploy init, they come back identically. The manifest is the single source of truth, and the tool enforces that guarantee mechanically.

Architecture

Here is the complete system after both stages:

manifest.yaml  ← the only file you edit manually
      │
      ▼
swiftdeploy CLI
      │
      ├── OPA policy check (pre-deploy / pre-promote)
      │         │
      │         ▼
      │   policies/*.rego
      │   + policy_limits from manifest
      │
      ▼
templates/
      ├── docker-compose.yml.tpl
      └── nginx.conf.tpl
            │
            ▼
┌──────────────────────────────────────────┐
│              Docker Network              │
│                                          │
│  ┌──────────────┐   ┌─────────────────┐ │
│  │     App      │   │      Nginx      │ │
│  │   :3000      │◄──│  :8080 (public) │ │
│  │  /metrics    │   └─────────────────┘ │
│  │  /healthz    │                       │
│  │  /chaos      │   ┌─────────────────┐ │
│  └──────────────┘   │       OPA       │ │
│                     │ :8181 (loopback)│ │
│                     └─────────────────┘ │
└──────────────────────────────────────────┘
      │
      ▼
history.jsonl  ← append-only audit trail
      │
      ▼
audit_report.md  ← generated on demand

Key isolation rules:

The app is never exposed directly to the host - all traffic goes through Nginx
OPA is bound to 127.0.0.1:8181 only - not reachable via the Nginx port or from the internet
The CLI is the only component that talks to OPA
The app has no knowledge of policy decisions

Stage A: The Engine

The Manifest

Everything starts with manifest.yaml. This file describes the entire deployment intent:

services:
  image: swift-deploy-1-node:latest
  port: 3000
  mode: stable
  version: "1.0.0"
  restart_policy: unless-stopped

nginx:
  image: nginx:latest
  port: 8080
  proxy_timeout: 10
  contact: o.odimayo@gbadedata.com

opa:
  image: openpolicyagent/opa:latest
  port: 8181
  policies_dir: policies
  decision_timeout_seconds: 5

network:
  name: swiftdeploy-net
  driver_type: bridge

logs:
  volume_name: swiftdeploy-logs

policy_limits:
  infrastructure:
    min_disk_free_gb: 10
    max_cpu_load: 2.0
  canary:
    max_error_rate: 0.01
    max_p99_latency_ms: 500

audit:
  history_file: history.jsonl
  report_file: audit_report.md

Every value that the generated configuration needs comes from here. No hardcoded values exist in any template or policy file.

Template-based Config Generation

The CLI uses Jinja2 to render docker-compose.yml and nginx.conf from templates. This is how swiftdeploy init works:

def render_template(template_name, output_path, manifest):
    env = Environment(
        loader=FileSystemLoader(TEMPLATE_DIR, encoding="utf-8-sig"),
        autoescape=False,
        trim_blocks=True,
        lstrip_blocks=True,
    )
    template = env.get_template(template_name)
    rendered = template.render(**manifest).lstrip("\ufeff")
    output_path.write_bytes(rendered.encode("utf-8"))

The template for the app service in docker-compose.yml.tpl looks like this:

services:
  app:
    image: {{ services.image }}
    container_name: swiftdeploy-app
    restart: {{ services.restart_policy }}
    user: "10001:10001"
    cap_drop:
      - ALL
    security_opt:
      - no-new-privileges:true
    environment:
      MODE: "{{ services.mode }}"
      APP_VERSION: "{{ services.version }}"
      APP_PORT: "{{ services.port }}"
    expose:
      - "{{ services.port }}"

Notice expose instead of ports. This is deliberate - the app is reachable inside the Docker network but not from the host machine. All external traffic must go through Nginx.

The Five Validation Checks

Before any deployment, the CLI runs five pre-flight checks:

[PASS] manifest.yaml exists and is valid YAML
[PASS] All required fields are present and non-empty
[PASS] Docker image exists locally: swift-deploy-1-node:latest
[PASS] Nginx port is free on host: 8080
[PASS] Generated nginx.conf is syntactically valid

The Nginx syntax check is particularly interesting - it runs nginx -t inside a temporary container with a host mapping for the app upstream name, so DNS resolution works without the full stack being active:

command = [
    "docker", "run", "--rm",
    "--add-host", "app:127.0.0.1",
    "-v", f"{NGINX_FILE}:/etc/nginx/nginx.conf:ro",
    nginx_image, "nginx", "-t",
]

The Application Service

The app is a Python Flask service that exposes four endpoints:

GET / - welcome message with mode, version, and timestamp
GET /healthz - liveness check with uptime in seconds
GET /metrics - Prometheus-format metrics (added in Stage B)
POST /chaos - fault injection, canary mode only

Stable and Canary Promotion

Promotion updates the manifest in-place, regenerates the Compose file, and restarts only the app container - Nginx is untouched:

def cmd_promote(args):
    target_mode = args.mode.lower()
    manifest = load_manifest()

    # policy check runs here for promote stable (Stage B)

    manifest["services"]["mode"] = target_mode
    write_manifest(manifest)

    render_template("docker-compose.yml.tpl", COMPOSE_FILE, manifest)

    run_command([
        "docker", "compose", "up", "-d",
        "--no-deps", "--force-recreate", "app"
    ])

The same image runs in both modes. The difference is the MODE environment variable injected by the generated Compose file - no rebuild required.

Nginx: Structured Logging and JSON Errors

The generated nginx.conf enforces three important behaviours.

Required access log format:

log_format swiftdeploy '$time_iso8601 | $status | ${request_time}s | $upstream_addr | $request';

Example output:

2026-05-06T23:04:50+00:00 | 200 | 0.001s | 172.18.0.2:3000 | GET / HTTP/1.1

JSON error responses for upstream failures:

error_page 502 = @error502;
location @error502 {
    return 502 '{"error":"bad gateway","code":502,"service":"swiftdeploy","contact":"o.odimayo@gbadedata.com"}';
}

Platform headers on every response:

add_header X-Deployed-By swiftdeploy always;
proxy_pass_header X-Mode;

Security Hardening

Every container runs with least privilege:

user: "10001:10001"   # app
cap_drop:
  - ALL
security_opt:
  - no-new-privileges:true

Images are built from python:3.12-slim and verified under 300MB. No secrets are baked into any image - all configuration is injected at runtime via environment variables.

Stage B: The Intelligence Layer

Part 1: Instrumentation

The first requirement was a /metrics endpoint in Prometheus text format. I used the prometheus_client library and Flask's request hooks to instrument every endpoint automatically:

HTTP_REQUESTS_TOTAL = Counter(
    "http_requests_total",
    "Total HTTP requests",
    ["method", "path", "status_code"],
)

HTTP_REQUEST_DURATION_SECONDS = Histogram(
    "http_request_duration_seconds",
    "HTTP request latency in seconds",
    ["method", "path"],
    buckets=(0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2, 5),
)

APP_MODE = Gauge("app_mode", "Application mode: 0=stable, 1=canary")
CHAOS_ACTIVE = Gauge("chaos_active", "Chaos state: 0=none, 1=slow, 2=error")
APP_UPTIME_SECONDS = Gauge("app_uptime_seconds", "Application uptime in seconds")

The after_request hook records every request automatically:

@app.after_request
def after_request(response):
    duration = time.monotonic() - getattr(g, "request_started_at", time.monotonic())

    HTTP_REQUESTS_TOTAL.labels(
        method=request.method,
        path=request.path,
        status_code=str(response.status_code),
    ).inc()

    HTTP_REQUEST_DURATION_SECONDS.labels(
        method=request.method,
        path=request.path,
    ).observe(duration)

    return response

The /metrics endpoint itself is just two lines:

@app.get("/metrics")
def metrics():
    update_runtime_metrics()
    return Response(generate_latest(), mimetype=CONTENT_TYPE_LATEST)

Part 2: The Policy Engine

This was the most architecturally significant part of Stage B. The requirement was absolute: the CLI must not make any allow/deny decision itself. All decision logic lives exclusively in OPA.

Why this matters

If the CLI embeds policy logic in Python, changing a threshold requires deploying new CLI code. When OPA owns the decisions, changing a threshold means editing one line in manifest.yaml. The operational difference is enormous - and the auditing story is much cleaner.

OPA in the Docker Compose template

OPA runs as a fourth service in the generated stack:

opa:
  image: {{ opa.image }}
  container_name: swiftdeploy-opa
  command:
    - "run"
    - "--server"
    - "--addr=0.0.0.0:{{ opa.port }}"
    - "/policies"
  ports:
    - "127.0.0.1:{{ opa.port }}:{{ opa.port }}"
  volumes:
    - ./{{ opa.policies_dir }}:/policies:ro
  cap_drop:
    - ALL
  security_opt:
    - no-new-privileges:true

The critical detail: 127.0.0.1:8181:8181 binds OPA only to the host loopback interface. It is reachable by the CLI but not via the Nginx port. The OPA API cannot be queried or probed from the internet.

Policy organisation

Each domain owns exactly one question and one set of input data. A change to the canary policy never touches the infrastructure policy.

Infrastructure policy (policies/infrastructure.rego) - answers: is this host safe to deploy on?

package swiftdeploy.infrastructure

default allow := false

disk_ok if {
    input.stats.disk_free_gb >= input.limits.min_disk_free_gb
}

cpu_ok if {
    input.stats.cpu_load <= input.limits.max_cpu_load
}

allow if {
    input.context == "pre_deploy"
    disk_ok
    cpu_ok
}

reasons contains msg if {
    not disk_ok
    msg := sprintf(
        "Disk free %vGB is below required minimum %vGB",
        [input.stats.disk_free_gb, input.limits.min_disk_free_gb],
    )
}

reasons contains msg if {
    not cpu_ok
    msg := sprintf(
        "CPU load %.2f exceeds allowed maximum %.2f",
        [input.stats.cpu_load, input.limits.max_cpu_load],
    )
}

decision := {
    "domain": "infrastructure",
    "question": "pre_deploy",
    "allow": allow,
    "reasons": reasons,
}

Canary safety policy (policies/canary.rego) - answers: is the canary healthy enough to promote to stable?

package swiftdeploy.canary

default allow := false

error_rate_ok if {
    input.context == "pre_promote"
    input.metrics.error_rate <= input.limits.max_error_rate
}

p99_latency_ok if {
    input.context == "pre_promote"
    input.metrics.p99_latency_ms <= input.limits.max_p99_latency_ms
}

allow if {
    input.context == "pre_promote"
    error_rate_ok
    p99_latency_ok
}

decision := {
    "domain": "canary",
    "question": "pre_promote",
    "allow": allow,
    "reasons": reasons,
}

Thresholds in the manifest, not in Rego

Neither policy file contains a hardcoded number. The limits come from input.limits, which the CLI sends as part of the JSON payload. The actual values live in manifest.yaml under policy_limits. Tune thresholds by editing the manifest - no Rego files need to be touched.

The pre-deploy gate

Before starting the stack, the CLI collects host stats and queries OPA:

def pre_deploy_policy_check(manifest):
    stats = get_host_stats()
    limits = manifest["policy_limits"]["infrastructure"]

    payload = {
        "input": {
            "context": "pre_deploy",
            "stats": stats,
            "limits": limits,
        }
    }

    decision = call_opa(manifest, "swiftdeploy/infrastructure/decision", payload)
    print_policy_decision(decision)

    if not decision.get("allow"):
        append_history(manifest, "policy_violation", {
            "domain": decision.get("domain"),
            "reasons": decision.get("reasons", []),
            "stats": stats,
        })
        return False

    return True

To prove the hard gate, I temporarily set min_disk_free_gb: 99999 and ran swiftdeploy deploy:

[POLICY][FAIL] infrastructure.pre_deploy
 - Disk free 2GB is below required minimum 99999GB
[FAIL] Deployment blocked by policy.

The stack never starts. The block is absolute.

The pre-promote gate

Before promoting from canary to stable, the CLI scrapes /metrics, calculates the current error rate and P99 latency, and queries OPA:

def pre_promote_policy_check(manifest):
    metrics_text = scrape_metrics(manifest)
    samples = parse_prometheus_metrics(metrics_text)
    observed = calculate_observed_metrics(samples)

    limits = manifest["policy_limits"]["canary"]

    payload = {
        "input": {
            "context": "pre_promote",
            "metrics": {
                "error_rate": observed["error_rate"],
                "p99_latency_ms": observed["p99_latency_ms"],
            },
            "limits": limits,
        }
    }

    decision = call_opa(manifest, "swiftdeploy/canary/decision", payload)
    print_policy_decision(decision)

    return bool(decision.get("allow"))

OPA failure handling

The CLI handles every distinct failure mode explicitly:

Failure	Message
Connection refused	`OPA unavailable at http://127.0.0.1:8181`
Request timed out	`OPA decision timed out after 5s`
Non-200 response	`OPA returned HTTP 503`
Non-JSON response	`OPA returned non-JSON response`
Missing result	`OPA response did not include a decision result`

In every case the operation is blocked and the event is recorded in the audit trail.

Part 3: The Chaos - What Happened When I Injected Failures

With the canary safety gate in place, I needed to prove it actually blocks unsafe promotions.

Deploy stable, promote to canary:

[PASS] Health check passed: mode=stable, version=1.0.0
[PASS] Promotion confirmed through /healthz: mode=canary

Inject 50% error chaos:

curl -X POST http://127.0.0.1:8080/chaos \
  -H "Content-Type: application/json" \
  -d '{"mode":"error","rate":0.5}'

Generate traffic to build up the error rate:

500  200  500  200  500  500  200  500  200  200

Run the status dashboard - this is where it gets interesting:

SwiftDeploy Status
==================
Timestamp: 2026-05-06T23:12:26.328363+00:00
Mode: canary
Chaos: error
Req/s: 0.00
Error rate: 38.10%
P99 latency: 5.00ms
Uptime: 128.42s

Policy Compliance
-----------------
[PASS] infrastructure.pre_deploy
  - Infrastructure policy passed
[FAIL] canary.pre_promote
  - Error rate 0.380952 exceeds allowed maximum 0.01

The status dashboard scrapes /metrics, calculates error rate from raw Prometheus counters, and queries both OPA domains independently on every interval. The infrastructure policy passes (the host is healthy) while the canary policy fails (the service is broken).

Attempt promotion — blocked:

[POLICY][FAIL] canary.pre_promote
 - Error rate 0.380952 exceeds allowed maximum 0.01
[FAIL] Promotion blocked by policy.

This is the safety guarantee the system is designed to provide. A broken canary cannot accidentally become the stable deployment.

Recover, generate clean traffic, promote successfully:

[POLICY][PASS] canary.pre_promote
 - Canary safety policy passed
[PASS] Promotion confirmed through /healthz: mode=stable

Part 4: The Status Dashboard

swiftdeploy status is a live-refreshing terminal dashboard. It runs continuously, scraping /metrics on every interval and appending every result to history.jsonl.

The interesting engineering challenge was calculating P99 latency from raw Prometheus histogram buckets without a Prometheus server. Prometheus histograms store cumulative bucket counts - to find P99, you need to find the smallest bucket whose cumulative count exceeds 99% of total requests:

def calculate_p99_from_buckets(bucket_totals):
    total_count = bucket_totals.get("+Inf", 0.0)
    if total_count == 0:
        return 0.0

    target = total_count * 0.99
    numeric_buckets = sorted(
        [(float(le), count)
         for le, count in bucket_totals.items()
         if le != "+Inf"],
        key=lambda x: x[0]
    )

    for upper_bound, count in numeric_buckets:
        if count >= target:
            return upper_bound * 1000  # convert to milliseconds

    return numeric_buckets[-1][0] * 1000

Health check and metrics paths are excluded from all calculations to avoid skewing the error rate and latency numbers.

Part 5: The Audit Trail

Every significant event is written to history.jsonl as a JSON line:

{"timestamp": "2026-05-06T23:40:04+00:00", "event_type": "deploy", "data": {"mode": "stable", "version": "1.0.0"}}
{"timestamp": "2026-05-06T23:42:53+00:00", "event_type": "policy_violation", "data": {"domain": "canary", "reasons": ["Error rate 0.380952 exceeds allowed maximum 0.01"]}}
{"timestamp": "2026-05-06T23:45:32+00:00", "event_type": "mode_change", "data": {"mode": "stable"}}

Running swiftdeploy audit generates audit_report.md - a GitHub Flavored Markdown report with a deployment timeline and violations table that renders directly on GitHub.

Complete CLI Reference

Command	What it does
`swiftdeploy init`	Parse manifest, generate `docker-compose.yml` and `nginx.conf`
`swiftdeploy validate`	Run 5 pre-flight checks, exit non-zero on any failure
`swiftdeploy deploy`	init → validate → OPA infra check → compose up → health gate
`swiftdeploy promote canary`	Update manifest, regenerate Compose, restart app only
`swiftdeploy promote stable`	OPA canary check → update manifest → regenerate → restart app
`swiftdeploy status`	Live metrics + policy compliance dashboard, appends to history
`swiftdeploy status --once`	Single scrape and exit
`swiftdeploy audit`	Parse `history.jsonl`, generate `audit_report.md`
`swiftdeploy teardown`	Remove containers, networks, and volumes
`swiftdeploy teardown --clean`	Teardown + delete generated config files

Lessons Learned

1. The manifest discipline pays off at every stage

Every time I was tempted to hardcode a value - a port, a timeout, a threshold - I put it in the manifest instead. This cost five extra minutes each time and saved hours. Any value can be changed in the manifest and the entire system adapts without touching any other file.

2. Separation of concerns is a survival strategy, not just a principle

When the CLI makes policy decisions in Python, changing a threshold means deploying new CLI code. When OPA owns the decisions, changing a threshold means editing one line in manifest.yaml. The operational difference is enormous. More importantly, the policy files can be reviewed, versioned, and audited independently of the tool that enforces them.

3. Every failure mode deserves its own message

The first version of the OPA client raised a generic exception on any failure. Connection refused, timeout, bad JSON, missing result - all looked the same. Adding distinct handling for each case costs twenty lines of code and saves hours of debugging in production.

4. Prometheus counters persist for the process lifetime

After recovering from error chaos and generating clean traffic, promote stable was still blocked. The reason: Prometheus counters never reset. The old error counts were still in the counter from before the recovery. Restarting the container resets them because it starts a fresh process. In production, you would use a time-windowed approach with a proper Prometheus server and PromQL range queries.

5. OPA isolation is a hard security requirement

If OPA were reachable via the Nginx port, anyone could query your policy engine, discover your exact thresholds, and craft traffic that stays just below detection limits. Binding OPA to the loopback interface and keeping it off the public network is not a preference - it is the minimum viable security posture for a policy engine.

6. Generated files should never be committed

Committing docker-compose.yml and nginx.conf to Git creates a false source of truth. Developers start editing the generated file instead of the manifest, the template drifts from reality, and the tool becomes meaningless. Keeping generated files in .gitignore enforces the discipline mechanically.

7. The Windows BOM problem is real

On Windows, writing YAML with Python's yaml.safe_dump can produce a file with a UTF-8 BOM prefix. When that file is later read by the Jinja template loader, the BOM gets rendered into the first key name, producing "\xEF\xBB\xBFservices" instead of services. The fix is to always write files using write_bytes(content.encode("utf-8")) rather than write_text(..., encoding="utf-8"). The difference is subtle and the debugging is painful.

Running It Yourself

# Clone the repository
git clone https://github.com/gbadedata/swiftdeploy.git
cd swiftdeploy

# Set up Python environment
python -m venv .venv
source .venv/bin/activate       # Linux/macOS
# .\.venv\Scripts\Activate.ps1  # Windows PowerShell

pip install pyyaml jinja2 requests

# Build the app image
docker build -t swift-deploy-1-node:latest .

# Full lifecycle
python ./swiftdeploy deploy
python ./swiftdeploy promote canary
python ./swiftdeploy status --once
python ./swiftdeploy promote stable
python ./swiftdeploy audit

# Clean up
python ./swiftdeploy teardown --clean

The full source code, Rego policies, Jinja templates, and screenshots are at:
https://github.com/gbadedata/swiftdeploy

Final Thought

The most important insight from this project is that a deployment tool should be more than a script that runs docker compose up. It should be a control plane - one that enforces standards before acting, surfaces observable state while running, and leaves an auditable record of every decision it makes.

SwiftDeploy is deliberately local-first and small in scope. But the patterns it demonstrates - manifest-driven generation, policy-gated lifecycle, metrics-based promotion gates, and append-only audit trails - are the same patterns that underpin tools like Argo CD, Flux, and every serious production deployment system.

The small version teaches you the patterns. The patterns scale to any size.

Source code: github.com/gbadedata/swiftdeploy

I Built a Real-Time DDoS Detection Engine from Scratch - Here's Every Decision I Made

Oluwagbade Odimayo — Thu, 07 May 2026 00:45:38 +0000

No Fail2Ban. No rate-limiting libraries. No shortcuts. Just Python, a deque, and statistics.

There is a moment every engineer dreads.

You are staring at your monitoring dashboard. The request graph is vertical. Your server is on its knees. Legitimate users are getting timeouts. And somewhere out there, an attacker is running a script they downloaded in five minutes - while your defence took you zero minutes to build, because you had none.

That was the situation at cloud.ng, a rapidly growing cloud storage platform powered by Nextcloud. After a wave of suspicious activity, I was handed a mandate: build an anomaly detection engine that watches all incoming HTTP traffic in real time, learns what normal looks like, and automatically responds when something deviates.

No off-the-shelf tools. No Fail2Ban. Build it from scratch.

This is the full story - every decision, every line of reasoning, every tradeoff.

First, Understand the Enemy

A DDoS attack - Distributed Denial of Service - works by exhausting your server's resources. Your server has a finite ability to handle connections. When an attacker sends 10,000 requests per second, your server spends all its time handling garbage traffic and has nothing left for real users.

But here is the subtler problem that most tutorials gloss over: you cannot just block "high traffic" IPs.

Traffic is not constant. A cloud storage platform gets hammered at 9 AM when everyone starts their workday. It goes quiet at 3 AM. If you set a fixed threshold - say, "block any IP sending more than 50 requests per minute" - you will:

Miss attacks during busy hours when 50 req/min is perfectly normal
Flag innocent users during quiet hours when 50 req/min is genuinely suspicious

What you need is a system that learns what normal looks like for right now - and flags deviations from that learned baseline. That is a statistics problem, not a configuration problem.

This insight shaped every architectural decision I made.

The Architecture

Before writing a single line of code, I mapped out the full system:

┌─────────────┐     HTTP      ┌─────────────────┐     proxy     ┌──────────────┐
│   Internet  │ ────────────► │   Nginx          │ ────────────► │  Nextcloud   │
│  (clients)  │               │ (reverse proxy)  │               │  (app)       │
└─────────────┘               └─────────────────┘               └──────────────┘
                                       │
                                       │ JSON access logs
                                       ▼
                              ┌─────────────────┐  (named Docker volume)
                              │  hng-access.log  │
                              └─────────────────┘
                                       │
                                       │ tail -f (continuous)
                                       ▼
                         ┌────────────────────────┐
                         │    Detector Daemon      │
                         │                         │
                         │  monitor.py   ◄── reads log
                         │  baseline.py  ◄── learns normal
                         │  detector.py  ◄── spots anomalies
                         │  blocker.py   ◄── calls iptables
                         │  unbanner.py  ◄── releases bans
                         │  notifier.py  ◄── pings Slack
                         │  dashboard.py ◄── live metrics UI
                         └────────────────────────┘
                                    │         │
                               iptables    Slack
                              DROP rule   #hng-alerts

Nginx sits in front of Nextcloud and writes every HTTP request as a JSON log line. The detector daemon reads that file continuously, processes each entry, and takes action when needed.

The key design principle: the daemon runs forever as a systemd service. It is not a cron job. It is not a script you run manually. It is always watching.

Part 1: The Foundation - JSON Logs with All the Right Fields

Everything starts with Nginx writing structured logs. Without good data, nothing else works.

Here is the log format I configured:

log_format json_logs escape=json
    '{'
    '"source_ip":"$remote_addr",'
    '"timestamp":"$time_iso8601",'
    '"method":"$request_method",'
    '"path":"$request_uri",'
    '"status":$status,'
    '"response_size":$body_bytes_sent,'
    '"http_host":"$host",'
    '"user_agent":"$http_user_agent"'
    '}';

access_log /var/log/nginx/hng-access.log json_logs;

Every log line looks like this:

{
  "source_ip": "1.2.3.4",
  "timestamp": "2026-04-29T21:48:35+00:00",
  "method": "GET",
  "path": "/",
  "status": 200,
  "response_size": 6674,
  "http_host": "cloud.ng",
  "user_agent": "Mozilla/5.0..."
}

One critical detail: I configured Nginx to extract the real client IP from the X-Forwarded-For header. Since Nginx is a Docker container proxying to another Docker container, without this, every request would appear to come from the Docker gateway IP - completely useless for per-IP detection.

set_real_ip_from  172.16.0.0/12;
real_ip_header    X-Forwarded-For;
real_ip_recursive on;

The logs are written to a named Docker volume called HNG-nginx-logs. Both Nginx (read-write) and Nextcloud (read-only) mount this volume. The detector daemon reads from the host path of the same volume.

Part 2: The Log Tailer - Watching the File Live

The first module, monitor.py, does one thing: follow the log file and yield parsed entries.

def tail_log(log_path: str):
    """
    Generator that yields one parsed dict per HTTP request.
    Works like `tail -f` - blocks waiting for new lines.
    """
    # Wait for the file to exist (Nginx may not have written anything yet)
    while not os.path.exists(log_path):
        time.sleep(2)

    with open(log_path, 'r', encoding='utf-8') as f:
        # Seek to END — we only care about new requests, not history
        f.seek(0, 2)

        while True:
            line = f.readline()
            if not line:
                time.sleep(0.05)   # 50ms poll — barely any CPU cost
                continue
            entry = _parse(line.strip())
            if entry:
                yield entry

The f.seek(0, 2) is important. Without it, on every restart, the daemon would replay the entire log history and potentially re-ban IPs that have already served their time.

The 0.05 second sleep between empty reads means the daemon uses almost zero CPU when traffic is quiet, but responds to new log entries within 50 milliseconds.

Part 3: The Sliding Window - The Heart of Rate Detection

Here is where most tutorials get it wrong.

The wrong way: Count requests per minute. Reset the counter every 60 seconds.

The problem: this creates "bucket boundaries". An attacker can send 59 requests at second 59, wait 2 seconds, send 59 more at second 61, and your per-minute counter never sees more than 59. They are attacking continuously but your counter says they are fine.

The right way: A true rolling window using collections.deque.

from collections import deque
import time

class SlidingWindowCounter:
    """
    Each entry in the deque is a float timestamp of one request.
    The deque always contains only the requests from the last window_seconds.
    """

    def __init__(self, window_seconds: int = 60):
        self.window_seconds = window_seconds
        self._timestamps = deque()    # timestamps of individual requests
        self._err_timestamps = deque()  # timestamps of error requests (4xx/5xx)

    def add(self, ts: float, is_error: bool = False):
        # New request joins from the RIGHT
        self._timestamps.append(ts)
        if is_error:
            self._err_timestamps.append(ts)
        self._evict(ts)

    def _evict(self, now: float):
        cutoff = now - self.window_seconds
        # Stale requests fall off the LEFT
        while self._timestamps and self._timestamps[0] < cutoff:
            self._timestamps.popleft()   # O(1) — this is why deque, not list
        while self._err_timestamps and self._err_timestamps[0] < cutoff:
            self._err_timestamps.popleft()

    def rate(self) -> float:
        # Exact rolling rate — no bucketing, no approximation
        return len(self._timestamps) / self.window_seconds

    def error_rate(self) -> float:
        return len(self._err_timestamps) / self.window_seconds

Picture the deque as a conveyor belt. Requests board at the right. Requests older than 60 seconds fall off the left automatically. At any moment, dividing the belt's occupancy by 60 gives the exact requests-per-second for the last minute.

Why O(1) matters: list.pop(0) shifts every element — O(n). deque.popleft() is O(1). At 10,000 requests per second across thousands of IPs, the difference between O(1) and O(n) is the difference between a daemon that keeps up and one that falls behind and misses attacks.

Every IP gets its own SlidingWindowCounter. There is also one global counter tracking all traffic combined - this catches distributed attacks where no single IP is hitting hard, but collectively they are overwhelming the server.

Part 4: The Baseline - Teaching the System What Normal Looks Like

The sliding window answers what is happening now. The baseline answers what normally happens.

I implemented a rolling 30-minute baseline of per-second request counts.

Timeline ──────────────────────────────────────────────────────►
         │ 1req │ 0req │ 3req │ 2req │ ... │ 5req │ 2req │ NOW │
         └──────────────────────────────────────────────────────┘
              ◄────────── 30 minutes = 1800 seconds ──────────►

Each slot represents one completed second. Every 60 seconds, the system recomputes:

def _recalculate(self):
    samples = [count for _, count in self._window]

    n        = len(samples)
    mean     = sum(samples) / n
    variance = sum((x - mean) ** 2 for x in samples) / n
    stddev   = math.sqrt(variance)

    # Floor values prevent division-by-zero at startup
    self.effective_mean   = max(mean,   self.floor_mean)    # config: 0.1
    self.effective_stddev = max(stddev, self.floor_stddev)  # config: 0.1

The Hour-Slot System

Here is the part that makes the baseline actually smart.

Traffic at 9 AM is genuinely different from traffic at 3 AM. If you use a single rolling window across all time, the baseline at 3 AM will reflect the busy afternoon - making it too permissive, missing real attacks.

I maintain per-hour slots:

hour = datetime.fromtimestamp(ts).hour
self._hour_slots[hour].append(count)   # {0: [...], 1: [...], ..., 23: [...]}

During recalculation:

hour_data = self._hour_slots.get(current_hour, [])

if len(hour_data) >= 60:
    # Enough data for this hour — use it (more accurate)
    samples = hour_data
else:
    # Fall back to the full rolling window
    samples = [count for _, count in self._window]

Once the system has been running for an hour, the 3 AM baseline reflects actual 3 AM traffic. The 9 AM baseline reflects actual 9 AM traffic. The system adapts to time-of-day patterns automatically, with zero configuration.

This is also why effective_mean is never, ever hardcoded. It is always computed from real traffic data.

Part 5: The Detection Logic - Making the Call

Now the payoff. We have:

current_rate — from the IP's sliding window
effective_mean — what is normal right now
effective_stddev — how much variation is normal

We compute a z-score:

z = (current_rate − effective_mean) / effective_stddev

The z-score is the number of standard deviations the current rate sits above the mean. In a normal distribution:

z = 1.0 → slightly above average (84th percentile)
z = 2.0 → notably above average (97th percentile)
z = 3.0 → very far above average (99.87th percentile)

At z ≥ 3.0, something is almost certainly not normal.

But z-score alone has a weakness: if traffic is so explosive that the baseline itself has not had time to update yet, the stddev might be large and the z-score misleadingly small. So I add a second condition:

# Flag if EITHER condition fires — whichever comes first
is_anomalous = (zscore >= 3.0) or (rate >= 5 * effective_mean)

The 5× multiplier catches the "traffic just went vertical" scenario that z-score might miss.

The Error Surge Tightener

Sometimes attackers probe rather than flood. They scan your endpoints looking for vulnerabilities, generating lots of 4xx and 5xx errors without necessarily hitting the raw request rate threshold.

The system detects this:

if ip_error_rate >= 3 * baseline_error_mean:
    # This IP is generating errors at 3× the normal rate
    # Tighten the detection threshold — flag it sooner
    z_threshold = max(z_threshold - 1.5, 1.0)

An IP generating suspicious error patterns gets caught at a lower threshold than regular traffic. This fires independently of the raw rate - it catches the probers.

Part 6: Blocking with iptables - The Linux Firewall

When an IP is flagged, the daemon calls the Linux kernel's firewall directly:

subprocess.run(
    ['iptables', '-I', 'INPUT', '-s', offending_ip, '-j', 'DROP'],
    check=True
)

-I INPUT inserts the rule at the top of the INPUT chain — it gets evaluated first, before anything else.
-j DROP silently discards the packet. No TCP RST. No HTTP response. The attacker's requests just vanish into nothing.

This is more effective than returning a 403 for two reasons:

It consumes zero application resources - the kernel drops the packet before it ever reaches Nginx
The attacker gets no feedback - they cannot even confirm the server is alive

The Backoff Schedule

Permanent bans on first offence are both unfair and operationally risky. You might ban a legitimate user with a buggy client, or a friendly scraper, and then spend hours investigating why a partner's integration broke.

The system uses a progressive backoff:

Offence	Duration	Reasoning
1st	10 minutes	Could be a misconfigured client
2nd	30 minutes	Likely intentional, more time needed
3rd	2 hours	Persistent offender
4th+	Permanent	They know what they are doing

A background Unbanner thread wakes every 10 seconds:

def _check_expiry(self):
    now = time.time()
    for ban in self.blocker.get_banned_ips():
        if ban['duration'] == -1:
            continue  # Permanent — never release
        if now - ban['banned_at'] >= ban['duration']:
            self.blocker.unban(ban['ip'])        # removes iptables rule
            self.notifier.send_unban(ban['ip'])   # notifies Slack

The _ban_history dict persists across unbans — so if an IP gets a 10-minute ban, behaves, gets released, then attacks again, it goes straight to the 30-minute tier. The system remembers repeat offenders even after their ban expires.

Part 7: Slack Alerts - Real-Time Visibility

Every significant event fires a Slack message to #hng-alerts:

Ban alert:

🚨 IP BANNED
• Condition: ip_rate_anomaly
• IP address: 1.2.3.4
• Current rate: 12.4000 req/s
• Baseline mean: 0.4200 req/s
• Z-score: 28.57
• Ban duration: 600s
• Timestamp: 2026-04-29T21:48:35

Global anomaly alert (no block - just awareness):

⚠ GLOBAL TRAFFIC ANOMALY
• Condition: global_rate_anomaly
• Current global rate: 48.3000 req/s
• Baseline mean: 3.2000 req/s
• Z-score: 14.09
• Action: Slack alert only (no IP block)

All Slack calls run in daemon threads - the detection loop never waits for a network call.

Part 8: The Live Dashboard

The system serves a live metrics dashboard at http://dashboard.gbadedata.com. It auto-refreshes every 3 seconds via a simple JavaScript setInterval call hitting a Flask /api/metrics endpoint.

The dashboard shows:

Global requests per second (live)
Baseline effective_mean and stddev
CPU and memory usage
All currently banned IPs with conditions, durations, and ban times
Top 10 source IPs ranked by current request rate
System uptime

No frontend framework. No build step. Flask serves an HTML string with embedded JavaScript. The entire dashboard is ~80 lines of code.

Part 9: The Audit Log

Every ban, unban, and baseline recalculation writes a structured entry to /var/log/detector-audit.log:

[2026-04-29T21:48:35.998876] BAN 172.18.0.1 | ip_rate_anomaly | rate=0.4000 | baseline=0.1000 | duration=600
[2026-04-29T21:58:38.252357] UNBAN 172.18.0.1 | ban_expired | rate=0.0000 | baseline=0.0000 | duration=600
[2026-04-30T12:20:17.038296] BASELINE_RECALC  | source=rolling_window (2 samples) | rate=10.5000 | baseline=10.5000 | duration=0

This is your paper trail. If someone asks "why was our partner's IP blocked at 3 AM on Tuesday?" - the answer is in the audit log, timestamped to the millisecond.

What I Would Do Differently

1. Use a proper WSGI server for the dashboard. Flask's development server works, but in production I would swap it for Gunicorn behind the existing Nginx setup.

2. Persist ban history across restarts. Currently, _ban_history lives in memory. If the daemon restarts, it forgets who was a repeat offender. A small SQLite database would fix this with minimal complexity.

3. Add IP allowlisting. Right now, any IP can be banned - including monitoring services, CDN health checkers, and your own office's IP. A config-driven allowlist would prevent embarrassing self-bans.

4. Ship metrics to Prometheus. The dashboard is useful for humans. For alerting, on-call tools, and long-term trend analysis, exposing a /metrics endpoint would integrate with the rest of a modern observability stack.

The Numbers

After running for 24 hours on the live server:

Processed thousands of HTTP requests continuously
Detected and blocked anomalous IPs within under 10 seconds of the first suspicious request
Auto-released bans correctly on the configured schedule
Dashboard refreshed without interruption
Zero false positives on legitimate traffic once the baseline had enough data

Final Thoughts

The most important lesson from this project is not about Python or iptables or deques.

It is about measurement before action.

Every security tool that uses hardcoded thresholds is making a guess. This system does not guess - it measures, it learns, and it acts on what it has observed. The z-score is not magic. It is just a way of asking: "is what I am seeing now consistent with what I have seen in the past?"

When the answer is no - decisively, statistically, not-even-close no - you block it.

That is the whole idea.

Full source code: https://github.com/gbadedata/hng14-stage3

Live dashboard: http://dashboard.gbadedata.com