DEV Community: Lenard Francis

I Spent Years Balancing Ledgers. Now I Balance Redis Connections.

Lenard Francis — Wed, 03 Jun 2026 09:57:54 +0000

I spent my career in accounting and finance before building infrastructure in Zimbabwe.
In accounting, every transaction has three properties:
Authorization — no entry without approval
Immutability — once recorded, never altered
Reconciliation — every debit has a corresponding credit, provable by audit
When I started building FastAPI AlertEngine, I applied the same discipline to production incidents. The result is not a monitoring tool. It's an operational governance system.

Monitoring Tools Are for Forensics. Governance Tools Are for Control.

Monitoring tools tell you what broke after it broke. Datadog, Grafana, Sentry — they produce beautiful post-mortems.
Governance tools enforce that nothing executes without authorization, and they prove it afterward.
Most teams conflate the two. They buy monitoring, assume governance, and get surprised when auditors ask: "Who approved that deploy?"
AlertEngine separates them explicitly:
plain
Detection → Policy (deterministic, no AI)
Diagnosis → AI (explains, recommends, does not decide)
Authorization → Human (engineer taps approve)
Execution → Webhook (your infrastructure, your control)
Audit → Ledger (immutable, replayable, actor-attributed)
This is not a feature list. It's an architectural hierarchy enforced by code.

The Zimbabwe Constraint

Engineers in Zimbabwe aren't always at laptops when things break. WhatsApp is ubiquituous and can be the operational control plane.
That constraint produces something better than a dashboard: alerts that find you, with a single tap to authorise recovery. No SSH. No runbooks. No "log into Grafana and interpret the graph."
Just: "Something broke. Here's why. Tap approve. Nothing runs without you."

The Ledger Philosophy
In finance, a ledger has two sides: what happened, and who authorized it.
AlertEngine's audit trail has the same structure:
JSON
{
"timestamp": 1717344000,
"incident_id": "inc-abc123-1685000000",
"stage": "AUTHORIZED",
"actor": "engineer",
"decision": "approve",
"reason": "Database connection pool exhausted — restart recommended",
"confidence": 0.87,
"policy_version": "1.0.0",
"tenant_id": "tenant-xyz789"
}
Every entry is append-only. Every entry has an actor. Every entry is replayable.
This is not logging. Logging tells you what the system did. A ledger tells you who authorized it and why.
Policy Is the Floor. AI Is the Ceiling.
The most important architectural decision in AlertEngine is this:
Claude cannot trigger a state transition.
Policy decides whether an incident exists. Policy decides when a system has recovered. Claude diagnoses and explains — but the state machine doesn't listen to Claude. It listens to incident_policy.py.
When health metrics recover, the pipeline doesn't ask Claude what to do. It calls should_recover(score, err) and if the threshold is met, it transitions to RECOVERED with actor="policy". Claude's recommendation is irrelevant.

This means:

A confident wrong AI diagnosis cannot cause an incident to escalate
A policy recovery override is logged as actor: "policy" — auditors can see exactly when and why
Changing thresholds is a one-line edit in one file, versioned, and logged in every subsequent audit entry
The audit trail never lies about who made the decision

Why This Matters Now

Three forces are converging:

Regulators are tightening. SOC 2, PCI DSS, HIPAA, GDPR — all require documented authorisation for production changes. "The AI did it" is not a compliant answer.
AI is getting faster. Claude can diagnose an incident in 3 seconds. Without governance, the temptation is to let it act autonomously. That's how you get a confident wrong diagnosis: restarting your database at peak traffic.
Engineers are burning out. 3 AM alerts with no context, no authorisation trail, and no proof of what happened. The answer isn't better dashboards — it's better workflows. AlertEngine addresses all three: policy gates prevent AI from acting alone, human authorisation prevents burnout, and the audit trail prevents regulatory surprises.

The Honest Part

I'm also building a payment orchestration platform for the African "hustler" context. Getting infrastructure funding in Zimbabwe is genuinely hard.
So I packaged the operational governance layer as a standalone product. It solves a real problem — I needed it myself at 2am. It also funds the bigger build.
That felt worth being honest about.

The Code
The orchestrator is source-available. Every claim in this post is verifiable:
orchestrator/pipeline.py — policy hierarchy, actor="policy" on recovery override
orchestrator/incident_policy.py — single POLICY dict, versioned, env-configurable
orchestrator/audit.py — append-only Redis LIST, full actor attribution, replayable
Read the code. Audit the architecture. Then decide if your infrastructure deserves the same discipline as your accounting.
GitHub: github.com/Tandem-Media/fastapi-alertengine
Install:
bash
pip install fastapi-alertengine
Managed orchestrator: anchorflowalertengine@outlook.com
Built in Harare, Zimbabwe. 🇿🇼

From Eclipses to P95 Latency: What the Joseon Dynasty Can Teach Us About Incident Response

Lenard Francis — Sun, 31 May 2026 16:22:51 +0000

The Joseon Dynasty ruled Korea for more than five centuries, from 1392 to 1897.

That is longer than the United States has existed. Longer than the printing press has been in widespread use. Five hundred years of one government, one bureaucracy, one record-keeping system.

And they documented everything.

The 朝鮮王朝實錄 (Veritable Records of the Joseon Dynasty) is one of the most extensive continuous historical records ever produced. Royal decrees, diplomatic correspondence, criminal cases, military campaigns, natural disasters, celestial observations, agricultural conditions, and administrative decisions were meticulously recorded.

Every eclipse. Every comet. Every drought. Every flood. Every tiger that wandered into a village.

At first glance, it reads like a civilisation obsessed with omens. Look closer and it begins to resemble something else: an accountability system operating at a national scale.

The Mandate of Heaven as an Accountability Mechanism

Joseon inherited the concept of the Mandate of Heaven from Chinese political philosophy.

The basic premise was simple: Heaven's approval of a ruler could be inferred from events in the natural world. Stable harvests, favorable weather, and orderly skies suggested good governance. Floods, eclipses, unusual celestial events, and other disruptions demanded attention.

Whether or not one accepted the underlying cosmology, the system functioned as a powerful accountability mechanism.

When an eclipse occurred, someone had to observe it. Someone had to record it. Officials had to discuss its significance. The court had to determine whether action was required. The response had to be documented. And the entire process became part of a permanent historical record.

A king could not plausibly claim ignorance of a reported eclipse.
An official could not quietly invent a justification for a policy years later. The record existed. The deliberation existed. The decision existed.

Accountability was structural.

https://ajin.im/is/building/omen.ops/

A Dynasty Rendered as Telemetry

Recently, Ajin built omen.ops, a project that renders the Veritable Records as a modern observability dashboard.

It is one of the most serious pieces of digital scholarship I have encountered. Every entry is sourced, annotated, and presented with the gravity the original record-keepers intended. Rather than merely digitising the records, the project reinterprets them through the lens of modern operations, observability, and incident management.

Suddenly, centuries-old historical events look remarkably familiar. Eclipses appear as system alerts. Comets register as anomaly spikes. Droughts become degradation events. The Mandate of Heaven itself is represented as a system health score.

The effect is both humorous and strangely illuminating.
A guest star observed over thirteen consecutive night-watches in 1592 appears as a P1 incident. The court astronomers of the Gwansanggam—the royal bureau responsible for celestial observation—tracked its position relative to known stars, recorded its persistence, and noted the absence of any established remediation procedure.

In modern operations language, the alert was acknowledged, classified, documented, and ultimately deemed unactionable.

Every engineer who has ever been paged at 3 AM has encountered the same category of problem.The dashboard also presents a derived metric called the Mandate Volatility Index. It compresses centuries of recorded anomalies into a single score relative to a reign's baseline conditions.

The historical court and the modern SRE team face the same challenge: overwhelming amounts of signal with limited human attention. Different centuries. Different tools. Same problem.
They needed summaries. We use dashboards. They had volatility indexes. We have P95 latency graphs.

The Tiger Incident

The most memorable entry may not involve the heavens at all.

In 1571, a white-browed tiger reportedly killed hundreds of people and livestock near the capital. The response escalated rapidly. The court mobilised a specialised tiger-catching commander and launched a coordinated hunt.

Then a secondary problem emerged. The soldiers sent to eliminate the tiger began looting civilians. The court was suddenly managing two incidents instead of one. The tiger threat was eventually mitigated. Multiple animals were killed. The military operation was scaled back. Reports of civilian misconduct were documented.

Incident opened. Response initiated. Unexpected side effects detected. Mitigation adjusted. Incident closed. The terminology changes. The workflow does not.

The Governance Problem Hasn't Changed

The tools have changed completely. The governance problem has not. What fascinates me about the Joseon system is not the astronomy. It is the process.

Observe - Record - Deliberate - Authorize - Act - Document

That sequence appears repeatedly throughout the Veritable Records. It is also the sequence behind every mature operational system. Modern monitoring platforms are excellent at collecting signals. They can detect latency spikes, memory pressure, queue backlogs, failed deployments, and infrastructure degradation in seconds.

What many systems still struggle with is everything that happens after detection. Who saw the alert? Who approved the response?
What evidence was considered? Why was a particular action taken?
Can someone reconstruct the decision six months later?
Detection is only the beginning. Accountability begins when decisions become traceable.

Building for Active Control

This idea sits at the centre of what I am building with FastAPI AlertEngine.

FastAPI AlertEngine is incident intelligence for FastAPI services. A free SDK adds health monitoring to your application with a single line of code. When degradation is detected, a managed orchestrator investigates the likely cause and sends you a WhatsApp or Telegram notification containing a single-use approval link. Nothing executes without your authorisation.

The goal is not simply to collect more alerts. Most organisations already have more alerts than they can reasonably process. The goal is to preserve the decision chain.

Observe the signal. Capture the evidence. Present the context.
Recommend an action. Require authorisation. Execute the response.
Record everything.

The Joseon court performed this process with a brush, ink, astronomers, and royal historians. We perform it with telemetry pipelines, machine reasoning, and APIs. The difference is speed. The court might take days to process an eclipse.
We can detect a P95 latency spike, identify likely causes, generate remediation options, and request approval in seconds.

But the underlying governance problem remains unchanged.
How do you ensure that the people responsible for a system are confronted with evidence, required to make a decision, and unable to rewrite history afterward?

Five hundred years ago, the Joseon Dynasty answered that question with brush and ink. We are still figuring it out with Redis and JWT tokens.

Explore the historical dashboard: https://ajin.im/is/building/omen.ops/

If you run FastAPI in production and want incident response that asks before it acts, the free SDK is available through FastAPI AlertEngine.

How I Taught My Incident Alerts to Say "This Broke 3 Minutes After Your Last Deploy"

Lenard Francis — Sat, 30 May 2026 19:50:31 +0000

You're staring at a P95 latency spike.

The alert says: "Database pool exhausted. P95: 2847ms."You know what broke. You don't know why.
So you open your git log, check when the spike started, scroll through commits, and try to figure out what changed in the 10 minutes before everything went sideways.
That archaeology takes 20 minutes on a good day. At 2am it takes longer.

The Problem with Context-Free Alerts
Most incident alerts are great at telling you the “what”. None of them tell you the “when” in relation to your codebase.
The question every engineer asks during an incident isn't "what is the P95?" — they already know that. It's "Did we just deploy something?"

The Insight: Incidents Have a Deployment Shadow
The way I see it, the majority of production incidents fall into one of two categories:
• Infrastructure events — upstream dependency failure, Redis outage, traffic spike
• Deployment shadows — something changed in the last deploy that didn't show up in testing

For category 2, the fastest path to resolution is knowing exactly what changed and when — down to the commit level.
If your alert says:
Database pool exhausted (P95: 2847ms)
Recent deployments before incident:
3m ago — a1b2c3d: "Fix checkout query isolation level" (John, +12/-3)
1 recent commit touched database/query files
You've just saved 20 minutes of log archaeology.

How to Build It
The implementation is simpler than it sounds. Three components:
• A commit store — Redis sorted set, scored by timestamp
• A GitHub webhook — receives push events, stores commits
• An incident correlator — maps incident start time to nearby commits

The Commit Store
def store_commit(tenant_id, sha, message, author, timestamp, files_changed):
key = f"orchestrator:commits:{tenant_id}"
redis.zadd(key, {entry: timestamp})
redis.expire(key, 86400 * 7) # 7 day TTL
A Redis sorted set gives you O(log N) insertion and O(log N + K) range queries — perfect for "give me commits in the 10 minutes before this timestamp."

The GitHub Webhook
@app.post("/commits/webhook")
async def github_webhook(request: Request):
body = await request.json()
for commit in body.get("commits", []):
store_commit(...)

Injecting Context into AI Diagnosis
Without commit context, Claude sees raw metrics. With commit context, Claude sees the metrics AND what changed 3 minutes before the incident — shifting the diagnosis from "likely database connection issue" to "checkout query isolation level change likely caused connection pool exhaustion."
That's a different quality of diagnosis entirely.

What the WhatsApp Message Looks Like
⚠️ Action Recommended
Service: Payment API
Issue: Database pool exhausted — P95 2.8s
Likely cause: Checkout query isolation level change
(commit a1b2c3d, 3m ago)
Confidence: 87%
👉 Approve fix: [link]
Nothing will run without your approval.

Three Setup Options
• GitHub webhook (recommended) — POST /commits/webhook with header X-AlertEngine-Tenant-ID
• Manual push from CI — curl from your GitHub Actions workflow
• GitHub API polling — set GITHUB_TOKEN and GITHUB_REPO, AlertEngine fetches automatically

The Broader Pattern
This feature is an instance of a broader pattern: enrich your incident context with everything that changed recently, not just the metrics at the moment of failure.
Future extensions of the same idea:
• Feature flag changes in the 10 minutes before an incident
• Infrastructure changes (Terraform applies, Docker image updates)
• Database migration executions
• Config changes

The alert that says, "Here's what broke, here's what changed right before it broke, here's the fix"—that's the alert worth building for.

─────────────────────────────────────────
This is now live in FastAPI AlertEngine as commit_context.py.
GitHub: github.com/Tandem-Media/fastapi-alertengine
Docs: tandem-media.github.io/fastapi-alertengine/
pip install fastapi-alertengine

Why P95 Latency Is the Only Metric That Matters at 3 AM

Lenard Francis — Thu, 21 May 2026 18:58:30 +0000

If your checkout endpoint serves 10,000 requests per minute, a 5% latency spike means 500 users are having a bad experience every minute.

Averages compress that pain into a single comfortable number.
P95 latency — the latency at the 95th percentile — tells you what your slowest users are actually experiencing.

It's the metric that catches the spike average hides.
This is why I track P95 as the primary health signal, not averages.

How Latency Spikes Actually Propagate
A latency spike rarely starts in your application.It usually starts somewhere else and cascades inward.

The typical pattern looks like this:

Slow upstream dependency
↓
Connection pool saturation
↓
Request queue growth
↓
Latency spike propagation
↓
Timeouts and failures

The Cascade Pattern
An upstream dependency (database, payment gateway, third-party API) slows down
Your FastAPI app keeps accepting requests while waiting for responses.
Your connection pool fills up – new requests queue behind existing ones.
Queue depth grows, memory pressure builds
Response times climb across all endpoints, not just the affected one. Eventually requests start timing out or failing entirely

By stage 3, you have a problem. By stage 5, your customers know about it before you do.
The cascade failure pattern is particularly nasty.A slow database query holds a connection.

That held connection blocks another request. That blocked request ties up execution capacity. Multiply that by concurrent users and you get full service degradation from a single slow dependency.

Under async workloads, the failure mode becomes especially deceptive because the application continues accepting requests while upstream awaits accumulation in the background.

High Traffic Spikes Make This Worse. Under normal load, a slow upstream dependency is annoying.
Under a traffic spike, it's catastrophic.

Here's why:

Connection pool saturation happens faster. If you have 20 database connections and traffic doubles, you hit the ceiling twice as fast.
Queue depth explodes. Requests piling up behind a slow dependency compound each other's wait time.
Memory pressure builds. Each queued request holds state. Enough of them and you drift toward OOM territory.
Recovery is non-linear. Once a connection pool is saturated, it often stays saturated even after the upstream issue resolves — because the backlog keeps it full.

The cruel irony is that traffic spikes happen when your service matters most.

A flash sale. A viral moment. A major announcement.
Exactly the wrong time to be debugging latency from a dashboard.

What Didn't Work For Me

Monitoring sounds easy in theory. In practice, most setups failed me in one of four ways.

Prometheus + Grafana. Powerful, but operationally heavy.

Setting up exporters, configuring dashboards, maintaining the stack — all before writing a single alert rule.

And when the alert fires at 3am, one still has to log in and interpret charts under pressure.

Simple Health Checks

GET /health → 200 OK tells you the service is alive.
It doesn't tell you it's running at 8x normal latency while technically responding.

Average Latency Monitoring

Averages mask the spikes that actually hurt users.

In one case, a payment provider slowdown pushed P95 latency from roughly 180 ms to over 2 seconds within minutes — while average latency still looked acceptable.

By the time averages reflected the issue, checkout failures had already started.

Alert Fatigue

I added more monitors to catch more things. Which meant more alerts. Most of them were noise. When everything is urgent, nothing is. Monitoring systems usually optimise for data collection.

Operators actually need decision compression.

What I Built Instead

I wanted something that:
Tracked P95, not averages
Produced a single health score instead of 15 metrics to interpret
Caught degradation trends early, before full failure
Required zero config to add to an existing FastAPI app

The result is a FastAPI middleware that continuously computes degradation signals directly from live request traffic.

from fastapi import FastAPI
from fastapi_alertengine import instrument

app = FastAPI()
instrument(app)

The middleware exposes a structured /health/alerts endpoint:

{
"status": "warning",
"health_score": {
"score": 61,
"trend": "degrading"
},
"metrics": {
"overall_p95_ms": 1847.3,
"error_rate": 0.08,
"anomaly_score": 0.9
}
}

One status. One score. One trend direction. No dashboards to configure. No agents to run. No Prometheus exporters.

The Human-in-the-Loop Layer

Once I had a reliable health signal, the next question was:
What do I do with it?

I built a managed orchestration layer that polls /health/alerts every 5 seconds. When the score drops below the threshold, it:

Runs Claude AI diagnosis on the metric context
Sends a WhatsApp or Telegram message (or Slack) with a plain-English summary
Generates a single-use recovery link

Most AI incident tooling jumps straight to autonomous remediation. I intentionally didn't.

Production systems deserve human authorisation before recovery actions execute. I read the diagnosis, preview the recovery action, and tap approve – all from my phone.

Nothing executes automatically. Every action is logged immutably.

I built the mobile-first delivery because I work in Zimbabwe, where engineers aren't always at laptops when things break.

WhatsApp is the operational control plane here.

That constraint produced something better than I expected:

Alerts that find you, rather than dashboards you have to find.

The Open Source Core
The telemetry middleware is free and MIT licensed.
pip install fastapi-alertengine

The managed orchestration layer (AI diagnosis, WhatsApp/Telegram alerts, and human-authorised recovery) is a commercial service.

GitHub: https://github.com/Tandem-Media/fastapi-alertengine

YouTube:

Most monitoring stacks are good at detecting incidents.
Very few are good at reducing operator uncertainty during one.
How are you handling that gap today?