DEV Community: Samuel Mutemi

The Two Joints: Where Agentic Engineering Breaks

Samuel Mutemi — Tue, 18 Aug 2026 18:30:35 +0000

Samuel Mutemi

Aug 2

Agentic Engineering Is Not Vibe Coding: The Three-Skill Loop I Use to Ship Distributed Systems

#webdev #ai #programming #automation

11 min read

Samuel Mutemi

Aug 4

Where Does Truth Live? A Field Guide to Agentic Engineering Methodologies

#agents #programming #ai #productivity

16 min read

Intent — the spec says what correct means.

Verification — tests check whether you got it.

Correction — a loop closes the gap.

This post is what happened when I stopped running those as three separate practices and wired them into one system.

The layers were the easy part.

The hard part is where they join, and there are exactly two joints.

The Real Problem: I Was the Wiring

Running three methodologies "concurrently" looked like this:

The spec lived in GitHub Issues.
The tests lived in the repo.
The loop lived in a skill that re-ran an agent until a task list emptied.

All three were real.

The connection between them was me, noticing things.

I noticed that a failing test meant a PRD assumption was wrong.

I noticed that slices five through nine came from a version of the spec that no longer existed.

I noticed that an agent had "fixed" a failing test by making the test weaker.

That's not a system.

That's three components and an operator, and the operator is the part that doesn't scale.

Each layer had a trigger, an input and an output.

None of them had a defined interface to the layer next door, so the interface defaulted to my attention.

So the goal isn't:

"Adopt all three."

It's:

Give the handoffs between layers a name, a trigger and an artifact, so they happen without me.

The Flow

The system looks like this:

                         SPEC
                    (what correct means)
                           |
                           v
                 +---------+---------+
                 |                   |
                 v                   v
            work slices        tests written
                              and FROZEN first
                 |                   |
                 v                   |
          agent implements           |
                 |                   |
                 +---------+---------+
                           |
                           v
                      run the tests
                           |
                 +---------+---------+
                 |                   |
                 v                   v
             code is wrong       spec looks wrong
             INNER LOOP          OUTER LOOP
             retry N times       STOP, no retries
             agent fixes it      human decides
                 |                   |
                 v                   v
             human gate,        amend the spec,
             by blast radius    re-derive the rest

Two loops.

The difference between them is the whole design:

The inner loop fixes the code.
Runs unsupervised, on a retry budget, allowed to act.

The outer loop fixes the spec.
Runs on escalation, no retry budget, allowed only to propose.

Changing the spec means changing intent, and intent isn't something an agent gets to change by itself.

A loop that can edit its own target will move the target instead of solving the problem.

Every time.

The rest of this post is those two joints.

Joint One: Your Tests Can't Just Be a Copy of Your Spec

Part two ended on this claim, and it's harder than it sounds:

A feedback loop with a bad sensor is worse than no loop, because it converges, confidently, at machine speed, on the wrong thing.

The obvious move is to generate your tests from your spec.

That's the spec-driven pitch, and it works.

But it buys you exactly one thing:

It catches the code disagreeing with the spec.

Now notice what it can't catch.

If the spec is wrong about the real world, a test generated from that spec is wrong in the same direction.

It passes.

Green checkmark.

Everyone goes home.

Safety engineering has a name for this: common-cause failure.

Two backups don't help if they share a design flaw, which is why aircraft use sensors built on different physical principles instead of just duplicating one.

Your spec-generated test suite is one channel, and its design is the spec.

What Can the Test Actually Catch?

What went wrong	Example	Can a spec-generated test catch it?
Code doesn't match the spec	PRD says retry 3 times, code retries once	Yes. This is its job
Spec contradicts itself	Requires strict ordering on a queue that doesn't guarantee it	Yes, cheaply
Spec is silent	Says nothing about ordering, so the agent picks one	No. No test exists for an unstated rule
Spec is wrong about the world	"The provider's charge endpoint is idempotent." It isn't	No. The test asserts the same falsehood
Spec is right, the requirement was bad	Flawless build of the wrong feature	No, and no test ever will

Rows three and four are where money gets lost.

The fix isn't a better prompt.

It's a second kind of test.

Two Kinds of Tests, and Only One Can Challenge the Spec

I now split tests by what they're measured against, not by unit/integration/e2e.

Spec Tests

Generated from the spec.

Measured against intent.

Property tests over the stated invariants, assertions on the out-of-scope list.

Cheap, generated, and I make a lot of them.

They ask:

Does the code do what we said?

Reality Tests

Not generated from the spec.

Measured against the world.

Provider sandbox checks, fault injection, replays of real incidents, reconciliation against the provider's ledger.

Expensive, mostly hand-written, they accumulate slowly.

They ask:

Does this survive contact?

The line that reorganized my thinking:

Only a test that didn't come from the spec can prove the spec wrong.

Anything derived from the target can confirm the target.

Nothing derived from it can contradict it.

Which gives a hard rule:

A failing spec test is never grounds for changing the spec. It means the code is wrong. Only a failing reality test is allowed to say the spec is wrong.

That sounds like bureaucracy until the first time an agent responds to a hard failing test by suggesting you amend the PRD.

Which it will, because that's the cheapest path to green.

Three Habits That Keep the Spec Tests Honest

1. Write the Tests Before the Code — and Freeze Them

Not the same session.

Not after there's an implementation to look at.

Sequencing buys independence almost free.

The agent writing the code may not edit its own tests.

If a slice genuinely needs a test changed, that's not a code change.

It's a claim about intent, and it escalates.

This one rule killed my worst failure mode.

2. Hunt the Silences

The clarifying-questions pass from /to-prd was aimed at my rough prompt.

I now run it again on the finished spec, asking only:

What does this not say that an implementer would have to guess?

Each answer becomes a spec edit or a recorded "don't care."

Both are durable.

An agent's undocumented assumptions are the most dangerous thing in the system, because they're the only part that isn't written down anywhere.

Joint Two: What Happens When the Loop Decides the Target Was Wrong

The Failure This Exists to Prevent

A loop with a retry budget and no escape hatch doesn't stop when the spec is wrong.

It keeps going, because that's what loops do.

And since it can't reach the spec, it reduces the error the only other way available:

by weakening the test.

Relax the assertion.

Widen the tolerance.

Skip the case with a plausible comment.

Special-case the failing input.

Each move is locally reasonable, and each one lies to you afterward.

A loop that can weaken its own tests will always converge. That's not a feature, that's the bug.

So the outer loop isn't mainly a correction mechanism.

It's a stop button.

And stopping is its most important capability.

The Stop, and the Four Verdicts

When a reality test fails in a way the inner loop can't fix, the loop halts that branch.

Zero retries.

Because retrying a spec error just buys more attempts at the wrong problem.

Then it files a spec challenge that blocks the parent PRD.

For example:

spec_challenge:
  spec: PRD-412 @ a3f19c2

  assumption: "provider charge endpoint is idempotent on our key"

  contradicted_by:
    test: reality/provider-sandbox-conformance
    # must be a reality test

    observed: "duplicate charge on retry with identical key"

    reproduced: "5 of 5 runs"

  blast_radius:
    slices_blocked: [4, 5, 7]
    slices_already_merged_on_this_assumption: [1, 2]

  smallest_fix_proposed: |
    key on provider_ref only; add 15m reconciliation sweep
    against provider ledger.

Four fields do the work.

1. Test Class

Because it decides who's even allowed to file this.

2. Slices Already Merged

Because that's the expensive number and I want it before I decide anything.

3. Smallest Fix

Because "smallest" is load-bearing.

An unconstrained agent will propose rewriting the spec around its discovery.

4. Reproduced

Because "failed once" and "failed five out of five" are different conversations.

Then a human decides.

There are exactly four answers, and naming them turned a judgment call into triage:

The spec was wrong. Amend it, cascade downstream. The intended path.
The test was wrong. Fix the test. If it was a spec test, that usually means the requirement was ambiguous, so the spec gets touched too.
The world is wrong. The provider's docs lie. The spec gets a workaround with an expiry date and a ticket, because undated workarounds become permanent architecture.
The requirement was wrong. Not the wording, the requirement. This leaves engineering entirely. Good systems just surface it early and cheaply.

Stamp Everything With the Spec Version

An amendment describes what changed, not the whole world again.

Then it propagates, in three tiers with very different costs.

Not Started Yet

Regenerate.

Free.

This is the entire payoff of spec-driven work.

In Flight

Halt, throw away, re-derive.

Costs tokens and nothing else.

Implementations are disposable; that was the bet.

Already Merged

The one nobody warns you about.

It isn't automatically wrong, but it is now unverified and has to be re-checked against the amended spec.

So every derived artifact carries the spec revision it came from:

Issues
Generated tests
Commit trailers

For example:

Derived-From: PRD-412@a3f19c2

Unglamorous, and it converts the worst question in the system from an archaeology dig into a search.

Which merged work came from a spec that has since changed?

Without stamps, the honest answer after three amendments is:

"Some of it, let me read the git log."

With them, you get a list.

And lists can be worked.

One More Guardrail

If the same assumption gets amended twice in one feature, I stop.

Two amendments mean I'm not correcting my model of the system.

I'm searching for one.

That's a spike, run deliberately in a throwaway branch, with nothing derived from it until it finishes.

A target that moves every time a test complains isn't a target.

What Got Worse

It's worth being honest about what this architecture doesn't solve.

Reality Tests Are Expensive

Reality tests are expensive and don't generate well.

That's the honest asymmetry here.

The channel I need most for correctness is the one that can't be derived from the artifact I have, by definition.

Fault injection.

Real sandboxes.

Recorded incidents.

It accumulates at human speed.

Anyone selling fully generated verification is selling you one channel and calling it two.

Escalation Fatigue

Which is the same bottleneck again.

If the loop escalates too readily, spec challenges pile up and get rubber-stamped exactly the way oversized diffs do.

I've moved the constraint from reviewing code to deciding about intent.

That's a better place for it, far more leverage per decision, but part one's point holds and it's recursive:

You don't eliminate a bottleneck, you relocate it. Then you go find it again.

More Artifacts to Keep True

Spec.

Spec tests.

Reality tests.

Version stamps.

Part two warned that a hybrid fails as a stale spec that agents still trust.

Version stamps make that detectable, not impossible.

What I Still Don't Have

Provenance Tooling

No tooling for any of the provenance.

Commit trailers and issue labels, which is to say conventions and discipline, which is to say it will decay.

This feels like something that should exist.

If it does, tell me.

The Next Generation of Senior Engineers

No answer to part one's open question.

If the mechanical work is where judgment used to get manufactured, and that work is now agentic, where do the next senior engineers come from?

Deciding spec challenges is excellent practice for exactly the skill that matters, and it's also the task I'd hand to the most experienced person in the room.

Which means it isn't a training ground.

I'm suspicious of anyone who claims to have solved this.

Where This Leaves the Series

Part one: your job moved from building the system to designing the system that builds the system.

Part two: sort the methodologies by where truth lives between runs, and you get three layers.

Part three: the layers were the easy part, and a hybrid is defined by its joints, not its parts.

If one thing survives out of all of it:

Both joints are the same problem stated twice.

Independence has to be engineered in.

It is never the default.

A test derived from the spec can't correct the spec, and a loop that can edit its target will move the target instead of doing the work.

And part two's caveat applies here more than anywhere.

A lot of this is model-specific error correction with a good name on it.

The rule about tests exists because today's models will happily weaken one to reach green.

The zero-retry rule exists because they'll happily rewrite a target to hit it.

If a future model reliably refuses both, some of this becomes scar tissue and should be cut.

Notice which of your rituals are load-bearing.

Retune when you change models.

And if you've built the same shape and hit a different joint, tell me, because I'm fairly sure I've only found two of them.

Part three of a series on agentic engineering in production. Part one covered the three-skill workflow and where the bottleneck goes when code stops being the constraint. Part two mapped the methodology landscape by asking where truth lives between agent runs.

Where Does Truth Live? A Field Guide to Agentic Engineering Methodologies

Samuel Mutemi — Tue, 04 Aug 2026 13:48:42 +0000

Last time I wrote about the three-skill workflow I use to ship payment infrastructure, and about where the bottleneck went once code stopped being the bottleneck.

Samuel Mutemi

Aug 2

Agentic Engineering Is Not Vibe Coding: The Three-Skill Loop I Use to Ship Distributed Systems

#webdev #ai #programming #automation

11 min read

I thought it was prudent to answer a question I often get when discussing my workflow; 'Why this approach?'. There are a dozen named methodologies in this space now, several of them with more GitHub stars than my entire company has customers, and I picked a particular combination for particular reasons.

So this post is the survey I wish I'd had when I started. I'll walk the landscape first, honestly, including the parts of it I rejected. Then I'll tell you why I landed on spec-driven development plus loop engineering, and what each half is actually doing.

The organizing question

Here's the lens that made this landscape legible to me.

Context windows are volatile memory. They fill, they compact, they lose the thread. Every session ends and takes its understanding with it. So every agentic methodology, every single one, whether or not its authors frame it this way, is an answer to one question:

Where does truth live between agent runs?

That's it. That's the whole taxonomy. Sort the methodologies by where they park durable state and the differences between them stop being marketing and start being architecture.

Methodology	Truth lives in	Fails when
Vibe coding	The chat session	The session ends
Context engineering	The assembled context / repo files	Assembly is wrong or stale
Spec-driven development	A durable spec artifact	The spec drifts from reality
Loop-based autonomy	The filesystem + git history	The stop condition is wrong
TDD governance	The test suite	Tests encode the wrong contract
Multi-agent orchestration	The orchestrator's task graph	A handoff silently corrupts

Let's go through them.

1. Prompt-native development (vibe coding)

Truth lives in the chat. You describe, you get code, you eyeball it, you continue.

I covered this in part one so I won't relitigate it. I'll just note that it remains the correct tool for spikes, throwaway scripts, and exploring an unfamiliar API, and that its defining property is that nothing survives the session. When the context compacts, your intent compacts with it.

Where it lives on the map: the origin point. Everything else is a strategy for surviving what this one loses.

2. Context engineering (and its cousin, harness engineering)

Truth lives in the assembled context: AGENTS.md, CLAUDE.md, repo conventions, retrieval strategy, file layout designed for agent navigation.

This is the discipline of making sure that whatever the agent reads at the top of a run actually orients it correctly. There's now real research on it, a body of work on structured context engineering for file-native agentic systems, and an emerging line on "harness engineering", which studies the scaffolding around the model rather than the model itself.

I don't think context engineering is a rival methodology. I think it's a substrate. Every other approach on this list is doing context engineering whether it admits it or not, a spec is context, a task list is context, a test failure is context. The people who talk about it explicitly are just being honest about the mechanism.

My take: necessary, not sufficient. A perfectly engineered context still doesn't tell you what to build, or how you'll know it worked.

3. Spec-driven development

Truth lives in a specification artifact that outlives every session.

This is the biggest cluster by far, and it's worth being precise, because "SDD" now names at least four meaningfully different things. The common thread is that a structured specification, usually Markdown, sometimes machine-readable, becomes the authoritative source from which implementation, tests, and docs are derived, rather than documentation written retrospectively.

The variants differ mostly in how much ceremony they impose and how far they push the spec toward being literal source code.

Lean and constitution-driven, GitHub Spec Kit

A four-phase workflow: specify, plan, tasks, implement. Every spec inherits from a project-wide "constitution" that encodes durable rules, stack, conventions, non-negotiables. GitHub's distribution has made it the highest-star option in the category.

Good: minimal ceremony, tool-agnostic, the constitution idea is genuinely excellent. Less good: it's optimized for greenfield and for reasonably-sized changes; small edits fight the phase structure.

Lightweight and brownfield-first, OpenSpec

Purpose-built for modifications rather than net-new work, using delta markers, ADDED, MODIFIED, REMOVED, so a change proposal describes what shifts rather than restating the world.

If your reality is a mature codebase where every task is an amendment, this is the most honest model on the list. It's also the cheapest to run.

Full agile simulation, BMAD-METHOD

The maximalist option: a dozen-plus specialized agent personas, Analyst, PM, Architect, Scrum Master, Developer, QA, passing artifacts down a pipeline that mirrors a full SDLC.

I want to be fair to BMAD, because it's genuinely impressive and it solves a real problem for teams that already think in PRDs and sprint stories. But the critique I find most convincing is structural: a persona pipeline is only as good as its weakest handoff. When the Architect makes an assumption the PM never documented, the Scrum Master faithfully propagates it into stories and the Developer implements it with total confidence. You discover it in QA, or in production. More personas means more handoff surface, and handoff failures are a nasty debugging class because every individual agent behaved reasonably.

The cost is also non-trivial. One consultancy reports BMAD runs averaging in the low tens of thousands of tokens per workflow and monthly frontier-model bills in the high hundreds to low thousands of dollars per developer. Your mileage will vary enormously, but it's not a rounding error.

Environment-native, Kiro, and the platform tier

AWS Kiro is a full IDE with the spec workflow built in from the ground up rather than layered on top. Same for various "agentic development environment" platforms that coordinate agents around a shared living spec.

The trade is explicit: tighter integration in exchange for moving into someone's environment. That's a real cost if your team's tooling is already settled, and a real benefit if it isn't.

Spec-as-source, Tessl and friends

The radical position: the spec is the primary artifact and the code is a build output. Edit the spec, regenerate. The analogy people reach for is Terraform or a SQL query planner, you write intent, the system produces the plan.

I find this directionally correct and practically premature, and I'll come back to why.

4. Loop-based autonomy, the Ralph technique

Truth lives on disk: a task list, per-task specs, logs, and git history.

Geoffrey Huntley named this one in mid-2025 after Ralph Wiggum, on the theory that any single iteration is a plain agent run that might get something slightly wrong, and the loop wins through persistence and a stable source of truth rather than one brilliant prompt. The minimal form is almost insultingly simple, a bash while loop that re-invokes a non-interactive agent until a todo list is exhausted.

The insight that makes it work is the one people miss: each iteration gets a fresh context. A single long session accumulates fatigue, the window fills, compaction drops details, the model loses the thread. A series of short sessions that re-orient from files on disk stays sharp. Progress doesn't live in the chat; it lives in committed code, a task file, and a log the next iteration reads on boot.

There are now many implementations, TUIs for visibility, a Vercel Labs wrapper that loops until a verifyCompletion function passes, and enterprise write-ups running Ralph across dependency-ordered backlogs inside test-gated SDLCs.

The catch, and it's a big one: canonical Ralph runs in --yolo mode. Full permissions, no confirmations. That mandates real sandboxing, and it means the technique in its pure form is unusable on anything where a bad iteration can move money or drop a table.

But note what Ralph actually is. It isn't a replacement for structured development, it's an execution engine for a structure you already have. Its own documentation says as much. That distinction turns out to matter a lot.

5. TDD governance

Truth lives in the test suite. Tests are written first and act as executable specifications, so correctness is defined before generation begins.

This has picked up serious academic attention, work presented at EASE 2026 on TDD governance for multi-agent code generation, and systems like TDFlow building agentic workflows around test-driven cycles. The motivating argument is one every distributed-systems person will recognize: LLMs are non-deterministic, identical prompts produce different outputs, and in multi-agent settings a small logic error propagates across the whole workflow. Automated guardrails aren't a nicety.

The research also finds that test cases reduce ambiguity by functioning as executable specs, which is the same claim SDD makes about prose specs, arrived at from a different direction.

Where it's weak: tests specify behavior beautifully and architecture terribly. A test suite cannot tell you that a module shouldn't know about another module, or that this queue is at-least-once, or why a constraint exists. It's a verification layer, not an intent layer.

6. Multi-agent orchestration

Truth lives in the orchestrator's task graph and shared memory.

Anthropic's 2026 agentic coding trends work describes the shape: an orchestrator coordinating specialized agents in parallel, each with dedicated context, results synthesized into integrated output, as against single-agent workflows that process sequentially through one window.

Addy Osmani's tiering is the most useful practical framing I've found:

Conductor tier, subagents inside one session. No extra tooling. Your context window is the ceiling.
Local orchestration tier, multiple agents in isolated git worktrees, with dashboards and merge control. Best at roughly 3–10 agents on a codebase you know.
Cloud async tier, assign, close laptop, return to a pull request.

The honest counterweight: production-scale orchestration is a multi-quarter engineering effort, observability tooling doesn't work on non-deterministic flows out of the box, and Thoughtworks' radar has been flagging the accumulation of cognitive debt as AI-assisted development scales, the organizational version of the review bottleneck I wrote about in part one.

My take: orchestration is an execution strategy, not a methodology. It answers "how many at once," not "toward what."

The axes that actually matter

Strip the branding and I think you're choosing along five axes:

Axis	The question
Durability	Does intent survive a context compaction?
Re-derivability	Can you regenerate the work from the artifact, or only document it?
Correction cost	When you're wrong, do you fix one thing or N things?
Human position	Where does judgment enter, before, during, or after generation?
Overhead	What does one cycle cost in tokens, wall-clock, and ceremony?

Re-derivability is the one I'd underlined three times if this were paper. Most teams evaluating SDD are asking whether the spec is good documentation. That's the wrong question, and it's the question that gets you a spec nobody maintains.

Why I landed on spec-driven plus loop engineering

Here's my actual reasoning, and it's less about picking winners than about noticing that two of these categories are answering different questions and therefore compose.

The spec is an anchor, and anchors let you re-derive

The property I care about most is durability that enables regeneration.

When I have a well-specified PRD, I'm not holding a description of what got built. I'm holding the thing the work is generated from. That gives me three capabilities I refuse to give up:

One, I can iterate on the same requirement many times. Not "revise the code," revise the requirement, and let everything downstream follow. The unit of change moves up a level.

Two, I can fan out. One artifact spawns many tasks. The PRD is the source; the issues are derived; the implementations are derived from those. That's a tree with a single root I control, not a pile of chat sessions I have to reconcile.

Three, and this is the one that sold me, I can throw implementations away. When an agent goes off the rails on slice four because it invented an assumption nobody wrote down, I don't debug the agent's reasoning or nurse the code back to health. I go to the document, fix the assumption, and regenerate. Completely. As many times as I need to.

The mental shift: implementations became disposable and the specification became the asset. That inverts the relationship I was trained on, where code was the thing you protected and documentation was the thing that rotted.

Notice this is the spec-as-source thesis, but I take it partway, deliberately. I don't accept "the spec compiles to code and you never read the output." In payments I have to be able to read, own, and be on call for what shipped. What I accept is the weaker, more useful claim: the spec is what work is re-derived from. Code stays reviewable and owned. It just stops being precious.

The loop is how the anchor gets corrected

Here's what spec-driven development does not do: guarantee that the spec is right.

A specification is a hypothesis. It's a good hypothesis, it's been through clarifying questions and human review, but it is written before contact with the actual system, and mine are regularly wrong in ways I could not have predicted. Spec-driven development on its own, with no feedback mechanism, is waterfall with better tooling. Same failure mode, faster.

That's what loop engineering is for, and I want to be precise about what I take from it. Not --yolo autonomy. I took the structural insights:

Fresh context per iteration beats one long degrading session
State on disk, task files, logs, git history, is the memory layer, not the chat
An explicit stop condition so the loop ends on a signal rather than a guess
Verification delegated into the loop so the agent finds its own errors before I do

Applied on top of a spec, the loop stops being brute force and becomes something much more useful: a mechanism that invalidates spec assumptions and pushes corrections back upstream. Validation fails on slice two, the assumption behind it was wrong, the PRD gets edited, and slices three through seven re-derive before an agent ever touches them.

That's the cascade I described in part one. It only works because both halves are present.

Stated as control theory, because that's what it is

For anyone who builds distributed systems, this framing will land immediately:

        spec  ────────▶  SETPOINT     (what "correct" means)
                              │
                              ▼
       agent  ────────▶  ACTUATOR     (does the thing)
                              │
                              ▼
        loop  ────────▶  FEEDBACK     (measures the gap, corrects)
                              │
                              └──────▶ error signal updates the SETPOINT
                                       when the setpoint was the problem

Spec without loop is open-loop control. Dead reckoning. Fine until reality diverges from your model, and you find out late.
Loop without spec is a controller with no setpoint. It will converge on something, reliably, persistently, cheerfully, just not necessarily on what you wanted. Or it oscillates.
Both is closed-loop control with a defined target and an error signal that can correct the target itself.

That's not a novel idea. It's the oldest idea in engineering, applied to a new actuator. Which is roughly why I trust it more than the approaches that feel newer.

What I took from the others anyway

Methodologies aren't teams. You don't have to pick one and wear the jersey:

From Spec Kit, the constitution idea. Durable project-level rules every spec inherits, so I'm not restating non-negotiables in every PRD.
From OpenSpec, delta thinking. Most of my work is amendment, not creation, and specs that describe changes beat specs that restate the world.
From TDD governance, tests as the loop's ground truth. Property-based and contract tests are what make the agent's self-correction meaningful rather than self-congratulatory.
From orchestration, parallelism bounded by the dependency graph, and worktree isolation. As an execution strategy underneath the loop, not as the loop.
From the security research, this one's my favourite. There's work on "constitutional" SDD that embeds non-negotiable security constraints, derived from CWE/MITRE Top 25 and regulatory frameworks, into the specification layer, so generated code satisfies them by construction rather than by inspection. If you read part one, you know why that lands for me. It's the compliance bottleneck, addressed at the only place it can be addressed cheaply: upstream, in the artifact.

And what I rejected, plainly

BMAD, the handoff surface. In a domain where a silently propagated wrong assumption means a double-charge, I want fewer inter-agent boundaries, not more. Also the token bill.
Canonical Ralph, I took the pattern and left the --yolo. Human gates on money-moving paths are non-negotiable, which is incompatible with the pure form.
Kiro and the platform tier, I'm not moving environments for a brownfield payments codebase. That's a fine trade for other people; it isn't for me.
TDD governance as a whole methodology, rejected, because tests can't carry architectural intent. But hold this one loosely; I come back to it at the end, and my position on it has changed.

The strongest objection to my position

I'd rather state this than have it stated at me.

The most rigorous comparison I've found of these tools flags two things that should give any SDD advocate pause. First: it's genuinely unclear when SDD adds value versus overhead, for trivial changes, the heavier frameworks are a sledgehammer cracking a nut. Second, and sharper: the historical parallel to Model-Driven Development, which made structurally similar promises in the 2000s and did not survive contact with real software.

I think the counterargument is that MDD failed partly because the generation step was rigid and the abstraction leaked badly, you got code you couldn't touch from models you couldn't express real systems in. LLM-generated implementations are readable, editable, and idiomatic, and specs are prose. That's a materially different failure surface.

But I hold it loosely. The honest version is that the Stack Overflow numbers keep telling us adoption isn't the problem, most developers use these tools and far fewer trust their output. Every methodology on this list is a bet about how to close that gap. Mine is a bet too.

There is no correct answer, and that isn't a cop-out

I want to be careful here, because "it depends" is what people say when they haven't thought about something. I have thought about this, and it still depends, for two reasons that are worth separating.

The first is that your constraints genuinely differ from mine. Blast radius, brownfield versus greenfield, team size, regulatory posture, how much of your work is amendment. Those are real inputs and they point at different answers.

But there's a softer input that I think gets dismissed too easily: your implementation preferences are a real engineering constraint, not a personality quirk. A methodology you abandon in week three has negative value, you paid the setup cost and got none of the compounding. If a seven-persona pipeline makes you want to close the laptop, that's data. Ceremony you won't sustain is worse than ceremony you never adopted.

The second reason is that the ground moves, and it moves fast. Not just the tooling, the models. They differ from each other in ways that matter to methodology design: instruction-following, how gracefully they degrade over long context, whether they ask or assume when a spec is ambiguous, what their characteristic failure modes even are.

Which leads to an uncomfortable observation about this entire field guide:

A large fraction of what we call methodology is really model-specific error correction with a good name on it. Some of these practices exist to compensate for a particular failure mode of a particular generation of model. When the model changes, the compensation may become unnecessary, or insufficient.

So treat everything above, mine included, as a snapshot. Retune when you switch models. Notice which of your rituals are load-bearing and which are scar tissue.

But the shape of the near-optimum is visible

Here's where I'll be less hedging. I don't think there's a perfect methodology. I do think the shape of a very good one is now clear, and it isn't any single entry on this list.

It's three layers doing three genuinely distinct jobs, with no overlap between them:

Layer	Methodology	Job
Intent	Spec-driven development	Defines what correct means, and why. Durable, re-derivable.
Verification	TDD governance	Determines whether it's correct, mechanically, in a way no agent can talk past.
Correction	Loop engineering	Closes the gap, and pushes errors back up to Intent when the intent was the problem.

Earlier in this post I drew the control loop with two components. That was incomplete, and deliberately so, because I wanted to arrive here. The full picture:

        spec  ────────▶  SETPOINT      what "correct" means
                              │
                              ▼
       agent  ────────▶  ACTUATOR      does the thing
                              │
                              ▼
       tests  ────────▶  SENSOR        measures reality, honestly
                              │
                              ▼
        loop  ────────▶  CONTROLLER    computes the error, acts on it
                              │
                              └───────▶ and when the error is in the SETPOINT
                                        itself, corrects upstream

Adding the sensor is not a garnish, and this is the part I got wrong for a while. A feedback loop with an untrustworthy sensor is worse than no loop at all, because it converges, confidently, persistently, at machine speed, on the wrong thing. That is precisely the failure mode of an agent writing its own tests from the same misunderstanding that produced its code. It measures, it agrees with itself, it proceeds.

That's why I've moved TDD governance from "component I borrowed" to co-equal layer. Spec-driven development gives the loop something to converge toward. Test governance gives it something to converge by. Remove either and the third stops working.

Composing rather than choosing

If the three layers are the skeleton, the individual frameworks become a parts bin rather than a set of competing religions:

Take	From	Because
Project constitution	Spec Kit	Non-negotiables shouldn't be restated per feature
Delta specs	OpenSpec	Most real work is amendment, not creation
Fresh context per iteration, state on disk	Ralph	Long sessions degrade; files don't
Test-first gating, adversarially derived	TDD governance	The sensor has to be independent of the actuator
Worktree parallelism	Orchestration tooling	Execution strategy under the loop, not the loop
Constraints in the spec layer	Constitutional SDD	Compliance is cheapest upstream

Two warnings before you go shopping.

Every layer is an artifact you now have to keep true. Composition has a maintenance cost that compounds, and the failure mode of a hybrid is a spec that no longer describes the system, gating agents that trust it. Adopt a layer only if you'll maintain it.

And don't confuse composition with accumulation. Bolting six frameworks together gets you the union of their ceremony and the intersection of their benefits. The point is to take one thing from each that does a job nothing else does, and to be able to say what that job is.

So, four questions, and they're about composing rather than picking:

Amendment or creation? → delta specs versus phase-structured ones.
What's your cost of being wrong? → determines how much sensor you need and where the human gates go.
Re-derive or just document? → if you'll never regenerate from the artifact, you're maintaining documentation, and it will rot the way documentation always has.
What ceremony budget will you actually sustain? → answered honestly, not aspirationally.

What I'm building next

The obvious move, once you see the three layers, is to stop treating them as three methodologies you're running concurrently and start treating them as one strategy with a single control flow. Intent, verification, and correction as first-class stages in one system rather than three practices you're manually keeping in sync.

That's what I've been working on. It's a hybrid, spec-driven at the anchor, test-governed at the gate, loop-driven at the correction step, and the interesting problems turn out to be at the seams: how the sensor gets derived from the spec without inheriting its blind spots, and what exactly happens when the loop concludes the setpoint was wrong.

That's the next post.

In the meantime: if you're running a combination I haven't covered, especially if you've made a persona-pipeline approach work at scale, because I would like to be wrong about that, tell me what broke and what held.

Part two of a series on agentic engineering in production. Part one covered the three-skill workflow and where the bottleneck goes when code stops being the constraint. Part three is the hybrid.

Agentic Engineering Is Not Vibe Coding: The Three-Skill Loop I Use to Ship Distributed Systems

Samuel Mutemi — Sun, 02 Aug 2026 17:59:55 +0000

There is a category error running loose in our industry right now, and it is costing teams real money.

The error is treating vibe coding and agentic engineering as the same activity performed at different levels of enthusiasm. They are not the same activity. They have different units of work, different failure modes, different artifacts, and, this is the part that matters, different economics when you point them at a production payment system that moves other people's money.

I build distributed systems for a living. Lately most of that has been payment infrastructure, which is the least forgiving place I know of to be wrong. Over the last while I have converged on a workflow that is roughly two-thirds spec-driven development and one-third loop engineering, implemented as three Claude Code skills that hand off to each other.

This post is that workflow, plus the things it broke that nobody warned me about.

The distinction, stated sharply

	Vibe coding	Agentic engineering
Optimizes for	Time-to-first-output	Time-to-verified-increment
Unit of work	A file, a function, "the thing I asked for"	An independently testable vertical slice
Where intent lives	In a prompt that scrolls away	In a durable, reviewed artifact
Verification	Visual, manual, by you	Automated, delegated, adversarial
Context	Hoped for	Engineered from a source of truth
Characteristic failure	Shipping something you can't explain	Caught by the loop, not by you at 2am

Vibe coding is a legitimate mode. I use it constantly for spikes, throwaway scripts, and exploring an unfamiliar API. It is a terrible mode for anything with an idempotency requirement.

The rest of this post is about the second column.

Spec-driven development plus loop engineering

Two ideas are doing the heavy lifting here, so let me define them the way I actually use them.

Spec-driven development means intent lives in an artifact that outlives any single agent session. Not in a prompt. Not in your head. Not in a Slack thread. In a document an agent can re-read at the top of every run, that a human reviewed and approved, and that changes through a visible edit rather than a vibe shift.

Loop engineering means deliberately designing the feedback loop the agent operates inside. What can it execute? What tells it that it is wrong, quickly and unambiguously? Where must it stop and escalate to a human?

A model without a tight loop is a very expensive autocomplete. A model with a tight loop is a colleague who never gets bored of running the test suite.

Neither is a prompting trick. Both are systems design applied one level up from the code.

The workflow: three skills, one handoff chain

  rough prompt
       │
       ▼
┌──────────────┐   clarifying questions ──▶ human
│   /to-prd     │◀──────────────────────────  ┘
└──────┬───────┘
       │  PRD: user stories · outcomes · invariants · OUT OF SCOPE
       ▼        (posted to GitHub Issues, reviewed before any code exists)
┌──────────────┐
│  /to-issues   │
└──────┬───────┘
       │  vertical slices, each independently testable, each linking the PRD
       ▼
┌──────────────┐        ┌─────────────────┐
│ engineering  │───────▶│ parallel slices │
│    skill     │        │  1   2   3   4  │
└──────┬───────┘        └────────┬────────┘
       │                         │
       │      ┌──────────────────┘
       │      ▼
       │  validation fails ──▶ assumption was wrong
       │      │
       │      └──▶ PRD updated ──▶ UNDONE slices re-derived  ← the outer loop
       ▼
  human gate (blast radius, not difficulty)

1. `/to-prd`, turn a vague intent into a specification

Every feature and every non-trivial bug starts here. I give it a rough prompt. Sometimes embarrassingly rough, a sentence and a link to a Sentry issue.

The skill then does four things:

Extrapolates the prompt into candidate requirements
Reads the actual codebase to ground those requirements in what exists
Asks me clarifying questions
Emits a full PRD, user stories, outcomes, explicit out-of-scope, and posts it to GitHub Issues

Step three is the entire value proposition, and it took me a while to see that.

The clarifying questions convert ambiguity into a decision at minute three instead of a defect at week six. When I write "add retry logic to the settlement webhook," a vibe-coded session gives me retry logic. /to-prd comes back and asks:

Idempotent on the provider's key, or ours?
What happens to ordering when attempt two lands before attempt one?
Does a poisoned message go to a DLQ, or block the partition?
Is partial settlement a valid terminal state?

I did not know the answer to two of those. That is the point. The gap between "what I asked for" and "what I meant" is where production incidents are manufactured, and a specification pass is a machine for finding that gap early.

The out-of-scope section is load-bearing

I want to be emphatic about this one. Agents are enthusiastic. Given a settlement webhook to fix, an unconstrained agent will refactor your retry utility, introduce a circuit breaker, and rename three things on the way past.

An explicit out-of-scope list is the cheapest guardrail in the entire workflow. It is the difference between a 200-line diff and a 2,000-line diff that nobody can review.

Posting to GitHub Issues rather than a local file is deliberate: it puts the spec where the team already lives, gives it a URL, makes it commentable, and makes review of the specification a first-class event, before a single line of implementation exists.

2. `/to-issues`, decompose into vertical slices

This skill takes the approved PRD and breaks it into implementation issues. The decomposition rule is the important part: vertical, never horizontal.

HORIZONTAL (org-chart slicing)        VERTICAL (what I actually use)

┌───────────────────────────┐         ┌────┐┌────┐┌────┐┌────┐
│  frontend ticket          │         │ UI ││ UI ││ UI ││ UI │
├───────────────────────────┤         ├────┤├────┤├────┤├────┤
│  backend ticket           │         │ API││ API││ API││ API│
├───────────────────────────┤         ├────┤├────┤├────┤├────┤
│  migration ticket         │         │ DB ││ DB ││ DB ││ DB │
└───────────────────────────┘         └────┘└────┘└────┘└────┘
 nothing testable until all             each slice testable
 three land simultaneously              on its own, on merge

Horizontal slicing is an artifact of the org chart, not of engineering. Its defining property is that nothing is testable until everything lands. You accumulate work in progress for two weeks and discover on integration day that the API contract you both agreed to meant two different things.

Vertical slicing means every slice cuts through the whole stack: thin, but complete. Each push can be validated on its own. The next push iterates on top of it.

This matters more with agents than it ever did with humans, for a specific reason:

The validation signal is the cheapest resource you have, and slicing determines how often you get one. If a slice is independently testable, the agent self-corrects and the loop closes without you. If it isn't, you are the test suite, and you've just reintroduced yourself as the bottleneck in the exact place you were trying to remove yourself from.

The second benefit is the one I underrated at the start. When validating slice two reveals that a PRD assumption was wrong, that corrected context cascades downstream to slices three through seven before an agent ever touches them. You are not fixing seven implementations. You are fixing one document and re-deriving the remaining work from it.

A wrong assumption caught in slice two costs an edit. The same assumption caught at integration costs a rewrite.

That cascade is the loop engineering part. The inner loop is agent writes code, tests fail, agent fixes code. The outer loop is validation invalidates a spec assumption, spec updates, undone work re-derives.

3. The engineering skill, execute, in parallel, with gates

The third skill picks up the generated issues, each referencing the parent PRD, and executes them.

Parallelism where the dependency graph allows it. Independent slices run concurrently. This is where the wall-clock gains actually come from, not from the model typing faster than you, but from four slices progressing at once while you're in a meeting.

Human-in-the-loop markers. Some issues are flagged as requiring a human. The skill prepares, stops, presents, and waits. Everything else runs AFK.

My gating rule is blast radius, not difficulty:

Runs unsupervised	Requires me in the room
Gnarly reconciliation algorithm	Schema migration
Complex parsing / transformation	Any money-moving code path
Test scaffolding, fixtures	Authorization boundary changes
Refactors covered by property tests	Anything whose failure is worse than a revert

An agent can write a hard reconciliation algorithm unsupervised, because a wrong one fails a property test. Difficulty is a bad proxy. Cost-of-being-wrong is the right one.

Why payments make all of this non-optional

You can run a sloppy version of this on a CRUD app and mostly get away with it. Distributed systems that move money will not extend you that courtesy.

Here is the failure mode that made me take specification seriously:

// Looks fine. Passes review. Passes its tests.
async function settle(payment: Payment) {
  return retry(3, () =>
    provider.charge({
      amount: payment.amount,
      currency: payment.currency,
    })
  );
}

That code double-charges a customer under a network partition, because the retry has no idempotency key and the underlying delivery guarantee was at-least-once all along. A timeout is ambiguous, the charge may well have committed.

// The invariant, made explicit in code because it was explicit in the spec.
async function settle(payment: Payment) {
  const key = idempotencyKey(payment.id, payment.attemptEpoch);

  return retry(3, () =>
    provider.charge({
      amount: payment.amount,
      currency: payment.currency,
      idempotencyKey: key, // stable across retries, this is the whole ballgame
    })
  );
}

Nothing about the first version looks wrong on review. It is wrong at the level of invariants, and invariants are invisible in a diff.

Which means the specification has to carry more than behavior. My PRDs for anything on a payment path encode non-functional requirements explicitly:

invariants:
  idempotency:
    key: provider_ref + attempt_epoch
    namespace: ours
    ttl: 24h
    on_collision: return prior result, do not re-execute
  ordering:
    transport_guarantee: at-least-once, unordered
    code_may_assume: nothing
  failure_modes:
    timeout: ambiguous, reconcile, never blind-retry
    poison_message: DLQ after 3, alert, never block partition
    partial_settlement: valid terminal state, requires ledger entry
  reconciliation:
    cadence: every 15m against provider ledger
    drift_threshold: 0, any delta pages
  observability:
    trace: payment_id propagated end to end

And verification has to be adversarial rather than confirmatory. Example-based tests written from the same understanding that produced the code will faithfully confirm that understanding, including the parts that are wrong. Property-based tests, fault injection, and contract tests are ground truth an agent cannot talk its way past. They are what make self-correction meaningful instead of self-congratulatory.

What actually got better

Being specific, because "10x productivity" is a claim I don't believe from anyone:

Ambiguity surfaces before implementation, not after. The clarifying-questions pass is the highest-leverage part of the workflow and costs about four minutes.
Wall-clock time on multi-slice features dropped substantially, mostly parallelism, plus not context-switching myself.
The spec exists. For the first time in my career, the design doc isn't a thing we wrote after shipping to satisfy an auditor. It's load-bearing, so it gets maintained.
Onboarding is different. A new engineer reads the PRD chain and understands why, not just what.
My worst days got better. Tedious, well-understood, mechanically tiresome work now happens without my attention.

Now the honest part: I moved the bottleneck, I did not remove it

This is the section I most want people to sit with, because the industry conversation is stuck on generation speed, and generation speed stopped being the interesting variable a while ago.

Code review does not scale with code generation

Generation is now nearly free. Comprehension is exactly as expensive as it was in 2019.

A senior engineer can meaningfully review some finite amount of code per day. The number is smaller than any of us admit, and it collapses further when the reviewer didn't write the surrounding context. Point an agentic workflow at that reviewer and you don't get a linear slowdown, you get rubber-stamping, which is strictly worse than no review, because it manufactures false confidence and diffuses accountability across a team that now collectively believes someone looked.

What helps, in order of effectiveness for me:

Vertical slices keep the review surface human-sized. A 200-line slice with a clear spec is reviewable. A 2,000-line feature drop is theater.
Review against the spec, not the diff. The question is "does this satisfy the stated invariants," not "would I have typed this." The second question is unanswerable at volume and mostly ego anyway.
A reviewing agent with a different context than the implementing agent. Same context reproduces the same blind spot. Different context, derived from the PRD rather than the implementation, catches things.
Hard human gates where being wrong is expensive. Non-negotiable, no exceptions for velocity.

The answer is agentic validation, not more human eyes

If generation is agentic and verification is manual, you have built a funnel that ends at a person.

The only structurally sound response is to make verification agentic too: test generation derived independently from the spec, fault injection, contract conformance, invariant checking, spec-conformance review as a distinct automated pass.

The humans move up the stack. You review the specification and the invariants. Agents review conformance to them. That's a real change in what the job is, and I don't think it's a demotion.

The backlog moved downstream, and that is the actual story

Here's the thing nobody put in the launch demo.

Theory of constraints has been telling us this for forty years: you do not eliminate a bottleneck, you relocate it. Code was the constraint. Now it isn't. So go look at where it went.

BEFORE                          AFTER

 product   ▓▓                    product   ▓▓
 eng       ▓▓▓▓▓▓▓▓▓▓  ← here    eng       ▓
 QA        ▓▓▓                   QA        ▓▓▓▓▓▓
 security  ▓▓                    security  ▓▓▓▓▓
 compliance▓▓▓                   compliance▓▓▓▓▓▓▓▓▓▓▓  ← here now
 ops       ▓▓                    ops       ▓▓▓▓▓▓

It went to QA, handed features faster than it can write test plans. To compliance, where PCI-DSS scope assessments, SOC 2 evidence collection, and data protection reviews run on a human cadence that did not just get 5x faster. To security review. To SRE and ops, absorbing a deployment frequency sized for a slower team. To change advisory boards, release trains, and every governance process that assumed engineering was the slow part.

If you make engineering agentic and change nothing else, you have built a machine that generates work-in-progress inventory for teams downstream of you. WIP is not throughput. In some organizations it is a liability with a carrying cost.

The response is not to slow engineering down. It's to recognize that agentic practice is an organizational transformation wearing a developer-tooling costume:

Compliance evidence generated continuously as a build artifact, not assembled in a panic before an audit
Policy-as-code, with control mapping evaluated in CI
Agent-drafted change records and risk assessments, human-approved
Compliance and QA present at the PRD stage, requirements encoded up front rather than assessed as a veto at the end

That last bullet has the best return and the most organizational friction. Shifting compliance left is far easier when the spec is already a machine-readable artifact your agents produced. It's one of the genuinely underrated second-order benefits of spec-driven work.

And a few costs I have not solved (click to expand)

Where does judgment come from now? If agents do the mechanical work, junior engineers don't get the reps that used to produce senior engineers. I don't have an answer. I don't think the industry does either.

Spec drift. A PRD that no longer describes the system is worse than no PRD, because agents trust it.

Cost. Four parallel agents burn tokens and CI minutes. Often worth it. Never free.

Ownership. Who is on call for code nobody wrote? The answer has to be a person, and the workflow has to make that person feel it.

Nondeterminism. Same prompt, different output. Your process must be reproducible even when generation isn't, which is precisely why the spec, the slices, and the gates are the durable parts, and the generation is the fungible part.

What I actually want you to take away

The shift is not that a model can write your code. It's that your job moved from authoring the system to designing the system that authors the system.

The skills you're compensated for are increasingly: specification, decomposition, invariant definition, verification design, and knowing where a human must stand in the loop. That's architecture work. It was always the valuable part. It's now the only part that doesn't commoditize.

So, two things to go do this week:

1. Find your most repetitive class of work, the feature shape you've built eleven times. Don't write a code-generation skill for it. Write the specification skill first. Make it ask you clarifying questions. You'll be unsettled by how many you can't answer.

2. Go measure where your constraint actually is right now. Not where it was when you last thought about it. If your engineers ship in two days and your compliance review takes three weeks, then every hour you spend making generation faster is an hour spent making the queue longer.

Make it agentic end to end, or don't be surprised when the bottleneck simply moves in next door.

I write about distributed systems, payment infrastructure, and agentic engineering practice. If you're running a variation of this workflow, especially in a regulated environment, I want to hear what broke. That's the most useful thing anyone can share right now.

What Payments Infrastructure Taught Me About Building Systems That Don't Break

Samuel Mutemi — Fri, 31 Jul 2026 09:45:31 +0000

Idempotency, vendor failure, monitoring that catches the invisible outages, and the tradeoffs nobody warns you about, lessons from scaling payments infrastructure.

Most software fails quietly. A page renders slowly, a recommendation is a little off, a report is stale by an hour. Users shrug and move on.

Payments doesn't work like that. When payments break, someone's money is in a place neither of you can account for, and the clock starts ticking on their patience. There's no graceful degradation. Either the money moved, or it didn't, and someone needs to know which.

I've spent a good chunk of my career building and scaling payments infrastructure, and it has quietly rewired how I think about engineering in general. Here's what stuck.

📋 The short version

#	Lesson	One-line summary
1	Idempotency	You will receive the same request twice. Design for it.
2	Vendor failure	Gateways are vendors. Ask "when," not "if."
3	Monitoring	Never learn about an outage from a customer.
4	The unglamorous stuff	Ledgers, reconciliation, state machines, refunds.
5	Tradeoffs	Every lesson above fights at least one other.

1. 🔁 Idempotency isn't a feature. It's a foundation.

The first hard lesson: you will receive the same request twice. Not "might." Will.

A client times out waiting for your response and retries. A user double-taps a button on a bad connection. A queue consumer crashes after processing but before acknowledging. A gateway sends the same webhook four times because it never got a 200 back. None of these are exotic failure modes, they're Tuesday.

If your system treats every incoming call as a new instruction, every one of those scenarios becomes a double charge. And a double charge isn't a bug you fix quietly in the next release. It's a support ticket, a refund, a reconciliation entry, and a customer who now checks their statement every time they use you.

The fix is conceptually simple and operationally demanding: every operation that moves money must be uniquely identifiable and safely repeatable. The caller supplies an idempotency key. You store it before you do anything else. If you see it again, you return the original result, not a new attempt, not an error, the same answer you gave the first time.

The naive version everyone writes first:

# ❌ Race condition hiding in plain sight
existing = db.find_by_idempotency_key(key)
if existing:
    return existing.result

result = charge_customer(amount)      # two concurrent requests
db.save(key, result)                  # both get here
return result

Two identical requests arriving 5ms apart both pass the if. Let the database enforce it instead:

# ✅ The unique constraint does the real work
try:
    with db.transaction():
        record = db.insert_idempotency_record(key, status="in_progress")
except UniqueConstraintViolation:
    return await_or_return_existing(key)

result = charge_customer(amount)
db.update(record, status="complete", result=result)
return result

The parts that actually take work:

Idempotency has to be atomic with the operation itself. If you record the key in one transaction and charge in another, you've just moved the race condition somewhere less obvious.
Concurrent duplicates are the hard case. A naive "check then write" won't catch them. You need a unique constraint or a lock doing the real work, not application logic hoping for the best.
Keys need a sensible lifetime. Too short and legitimate retries slip through. Too long and you're storing a permanent record of every request you've ever seen.
It has to extend past your API boundary. Your internal queues, your workers, your webhook handlers, every hop is another opportunity to duplicate. Idempotency at the edge with at-least-once processing behind it just moves the problem inward.

Idempotency is really just the discipline of making your system's behavior depend on intent rather than on delivery, and delivery is the thing you don't control.

2. ⚠️ Every gateway you integrate is a vendor, and vendors fail

When you integrate a payment gateway, you're not adding a library. You're taking on a dependency whose code you can't read, whose deploys you don't schedule, whose incidents you learn about from Twitter, and whose SLA is a document, not a guarantee.

I stopped asking "what if this provider goes down" a long time ago. The right question is "what do we do when this provider goes down," and it's a question with a schedule attached, because it will happen this quarter.

That shift in framing changes the architecture:

🧱 You isolate them

Every provider sits behind your own interface, speaking your own domain language. Your core system should never know that Provider A calls it a transaction_reference and Provider B calls it a paymentId.

         ┌──────────────────────┐
         │   Payment Service    │   ← speaks YOUR domain language
         └──────────┬───────────┘
                    │
         ┌──────────▼───────────┐
         │  Provider Interface  │   ← translation lives here
         └──┬────────┬────────┬─┘
            │        │        │
        ┌───▼──┐ ┌───▼──┐ ┌───▼──┐
        │  A   │ │  B   │ │  C   │   ← vendors. they will fail.
        └──────┘ └──────┘ └──────┘

That translation layer is annoying to build, and it is the thing that lets you swap a provider under load instead of during a two-month migration.

⏱️ You assume timeouts, not errors

A clean error is a gift, it tells you what happened. The genuinely dangerous response is no response.

Your request may have been processed. It may not have been. You don't know, and you can't just retry blindly. This is where reconciliation and status polling stop being nice-to-haves: when the network gives you ambiguity, you need a way to go ask "did this actually happen?" and get an authoritative answer.

🔌 You add circuit breakers

When a provider starts failing, hammering it with retries makes their recovery slower and your queues longer. Fail fast, back off, route elsewhere, come back later.

🎲 You retry with jitter, and only where it's safe

delay = min(base * (2 ** attempt), max_delay)
delay = random.uniform(0, delay)   # ← the line people skip

Exponential backoff without jitter just means all your retries collide again in a synchronized wave. And "safe to retry" is only true because you did the work in lesson one.

🔀 You route around damage

If you're on multiple providers, the point isn't just pricing leverage, it's that you can shift volume when one degrades.

But automatic failover is only trustworthy if you can tell the difference between "provider is down" and "provider is correctly declining these transactions." Failing over on legitimate declines is a great way to turn a small problem into a fraud incident.

🧪 You never trust the sandbox

Test environments are clean, fast, and always available. Production is none of those things. The behaviors that hurt you, partial failures, delayed settlement, out-of-order webhooks, undocumented status codes, almost never show up until real traffic does.

3. 📡 You should never learn about an outage from a customer

If a customer is telling you something is broken, you've already lost twice: once for the failure, and once for not knowing about it first.

Monitoring in payments has to work on two levels, and most teams only build one.

Level	What you measure	What it tells you
Technical	Latency, error rates, queue depth, saturation, provider response times, retry counts	The system is unhealthy
Business	Success rate by provider / channel / card type / country, volume vs. last week, transactions stuck pending, settlement vs. ledger	Money is not moving correctly

The reason you need both is that the worst payments incidents don't look like outages.

Every service is up. Latency is fine. Error rates are flat. And a specific bank's cards have been silently failing for forty minutes because a provider quietly changed something.

Infrastructure metrics will never catch that. A success-rate alert segmented by issuer catches it in five minutes.

A few things I now consider non-negotiable:

Alert on trends, not just thresholds. A drop from 94% to 71% success is an emergency even though 71% isn't zero.
Every alert needs a runbook. An alert that fires at 3am with no attached "here's what to check and who to call" is just anxiety with a pager.
Traceability end to end. When someone asks about one specific transaction, you should be able to reconstruct its entire life, every state change, every provider call, every retry, without SSH-ing into anything.
Ops needs their own tools. Your support and operations teams shouldn't be filing engineering tickets to answer routine customer questions. Give them read access to transaction state and safe, audited actions. This buys back an astonishing amount of engineering time.

Availability is the other half of this. In most products, downtime costs you some engagement. In payments, downtime is transactions that didn't happen, revenue that evaporates and doesn't come back, plus something more expensive: trust that erodes a little each time and doesn't rebuild at the same rate. People will forgive a slow app. They get quietly nervous about a payment system that failed on them once, and they stop reaching for it first.

4. 🧾 The things nobody warns you about

A few more lessons that cost me something to learn.

Your ledger is the source of truth, not your provider's dashboard

Build a proper double-entry, append-only ledger early. Never update a balance in place. Never delete an entry. Corrections are new entries.

-- ❌ Where did the money go? Nobody knows.
UPDATE accounts SET balance = balance - 500 WHERE id = 42;

-- ✅ Append-only. The history IS the balance.
INSERT INTO ledger_entries (account_id, amount_minor, currency, direction, ref)
VALUES (42, 50000, 'KES', 'debit',  'txn_9f2a'),
       (17, 50000, 'KES', 'credit', 'txn_9f2a');

When something eventually goes wrong, and it will, the only thing that saves you is an immutable record of what your system believed at every point in time.

Reconcile continuously, not monthly

Automated comparison between your ledger and each provider's settlement reports, running daily at minimum. Discrepancies compound. A mismatch found the next morning is a fix; the same mismatch found at quarter-end is an investigation.

Never use floating point for money

0.1 + 0.2 === 0.3   // false. this is your revenue.

Store minor units as integers. Always attach a currency. Decide your rounding rules explicitly and write them down, because rounding disagreements between you and a provider will eventually surface as a real, unexplainable gap.

Model payment state as an explicit state machine

Not a boolean. Not a status string that anyone can set to anything.

initiated ──► pending ──┬─► succeeded ──► refunded
                        │
                        ├─► failed
                        │
                        └─► expired

# succeeded ──► pending is NOT a valid transition.
# Enforce that in ONE place, not in seven services.

Half the weird bugs in payments are illegal state transitions that nobody thought to prevent.

Webhooks arrive out of order, late, or twice

Sometimes the "success" event lands before the "processing" event. Handle each one as a fact about a point in time, not as an instruction to advance a status.

Design the unhappy paths first

Refunds, partial refunds, reversals, chargebacks, disputes, expired authorizations. Teams build the happy path, ship it, and then discover that refunds don't fit the data model at all. The unhappy paths are where the actual complexity lives, and retrofitting them is far more expensive than designing for them.

Compliance is an architectural constraint, not a checklist

What you're allowed to store, where you're allowed to store it, who can see it, how long you keep it. Discovering these requirements after you've built is a rewrite.

5. ⚖️ Everything above is in tension with everything else

Here's the part that took me longest to accept: none of these lessons is free, and several of them actively fight each other.

What you gain	What it costs you
Aggressive retries → higher success rates	More duplicate risk
Multi-provider redundancy → resilience	Integration surface, reconciliation complexity, on-call load
Comprehensive monitoring → catch real problems	Alert fatigue that slows your real response
Meticulous ledger → certainty	Write throughput

There's no configuration where you max out every dial. The engineering is in deciding, deliberately and with your eyes open, where you sit on each of these, and being honest that it's a choice with a cost, rather than pretending you got everything.

🎯 If you're starting on this today

Build for the failure cases before you build for scale. Scale problems announce themselves loudly and you get to fix them in daylight. Correctness problems hide, compound quietly, and surface at the worst possible time, usually as a number that doesn't add up and a customer who wants to know where their money went.

Get the boring parts right:

[ ] Idempotency keys, enforced at the database level
[ ] An append-only, double-entry ledger
[ ] Monitoring on business metrics, not just infrastructure
[ ] An explicit state machine for payment status
[ ] Automated daily reconciliation
[ ] Refunds and chargebacks in the data model from day one

Everything else is much easier to build on top of that than it is to bolt on afterward.

Have you hit any of these the hard way? I'd like to hear which one bit you, the comments are open. 👇

Microservices or Micro-progress? The Perils of Pre-Scaling Too Soon

Samuel Mutemi — Fri, 15 Aug 2025 21:28:09 +0000

The Grand Illusion of "Future-Proof" Code

Ah, the siren song of scalability. It whispers sweet nothings into our ears: "Build it right the first time," "You’ll thank yourself later," "What if you get 10 million users tomorrow?" And before you know it, you’ve spent three months architecting a dazzling microservices masterpiece, only to realize your "scalable" app has exactly one user: you, refreshing the page in incognito mode.

This was me. I was that developer. Convinced that my side project needed a Kubernetes cluster, a distributed notification system, and a payment service that could handle Stripe-level traffic, despite the fact that my MVP was still just a glorified to-do list.

The Hard Truth: Scale Doesn’t Matter If Nobody Cares

Here’s the uncomfortable reality: You cannot optimize for problems you don’t have yet.

Are you drowning in user traffic? → No? Then why are you building a CDN?
Do you have 10,000 concurrent payment requests? → No? Then why are you overengineering Stripe?
Are your notifications crashing under load? → No? Then why did you build a pub-sub system when Novu’s free tier exists?

I had to face the music: I wasn’t building for scale, I was procrasti-scaling. Avoiding the real work (making something people actually wanted) by obsessing over infrastructure that might matter someday.

The Pivot: From Over-Engineered to "Good Enough"

So I did the unthinkable: I deleted months of work and replaced it with:

Supabase (Auth + DB) → Free tier. Works. Done.
Novu (Notifications) → Free tier. Works. Done.
Stripe (Payments) → Basic integration. Works. Done.

Suddenly, I was making real progress - not on infrastructure, but on the actual product. And guess what? If (big if) my app ever outgrows these tools, I’ll cross that bridge when I get there.

The Lesson: Build Stupid First

Your first version should be embarrassingly simple. Not because you’re lazy, but because:

You don’t yet know what needs scaling. (Most things won’t.)
Premature optimization is the root of all evil. (Or at least wasted time.)
Your biggest risk isn’t scale, it’s building something nobody wants.

So next time you catch yourself designing a Kafka pipeline for your cat blog, ask:

"Is this solving a real problem, or just my fear of hypothetical ones?"

Because in the end, a working app with limits beats an unfinished "scalable" one every time.

Now go build something stupid. You can always make it smart later. 🚀

Setting Up Keycloak for Passwordless Authentication

Samuel Mutemi — Wed, 28 May 2025 09:57:59 +0000

Passwordless authentication is becoming a must-have for modern applications, no more forgotten passwords, just seamless access via magic links, biometrics, or security keys. Keycloak, the popular open-source identity and access management solution, makes implementing passwordless auth surprisingly straightforward.

In this guide, we’ll walk through configuring Keycloak to support email-based magic links (a common passwordless approach). Let’s dive in!

Prerequisites

A running Keycloak instance (v20+)
SMTP server access (for sending magic links)
Basic familiarity with Keycloak admin console

Step 1: Enable Email Verification

Since passwordless auth relies on email links, we first need to ensure Keycloak can send emails.

Configure SMTP settings
- Go to Realm Settings → Email
- Fill in your SMTP server details (e.g., Gmail, SendGrid, Postmark)

   Host: smtp.example.com  
   Port: 587  
   From: no-reply@yourdomain.com  
   Enable SSL/TLS: Yes  
   Authentication: Enabled (provide credentials)

Test email delivery
- Click Test connection to verify everything works.

Step 2: Set Up Passwordless Authentication Flow

Keycloak uses authentication flows to define login steps. We’ll customize the default flow.

Create a new authentication flow
- Navigate to Authentication → Flows
- Click New flow, name it (e.g., "Passwordless Email")
Add required steps
- Under your new flow, add these executions:
  - Username Form (for email input)
  - Send Email Verification Link (replaces password check)
  - Conditional User Role (optional, for additional security)
Disable password requirement
- Go to Realm Settings → Login
- Disable "Password" as a required credential

Step 3: Customize the Magic Link Email

Keycloak sends a verification email, let’s make it user-friendly.

Edit the email template
- Go to Realm Settings → Email → Templates
- Modify "Verify Email" to include a clear call-to-action:

   <p>Click below to log in:</p>  
   <a href="${url}" style="background: #2563eb; color: white; padding: 10px 20px; text-decoration: none; border-radius: 5px;">Sign In Instantly</a>

Set link expiration
- Under Realm Settings → Tokens, adjust "Email Verification Link Lifespan" (e.g., 15 minutes).

Step 4: Test the Flow

Try logging in as a test user.
Instead of a password field, you’ll see an email input.
After submitting, check your inbox for the magic link.
Clicking it should log you in directly!

Bonus: Adding WebAuthn (Biometric Auth)

For a more advanced passwordless experience, enable WebAuthn (for security keys/biometrics):

Go to Authentication → Flows
Add "WebAuthn Authenticator" as an alternative.

Final Thoughts

Keycloak makes passwordless auth surprisingly simple. With just a few tweaks, you can replace clunky passwords with secure, user-friendly magic links or biometric logins.

Need help? Check out the official Keycloak docs or drop a question below!

AI in Corporate: The Coming Wave of Disastrous Blunders

Samuel Mutemi — Wed, 14 May 2025 18:50:38 +0000

How AI-Powered Tools Will Accidentally Wreck Prod (And Maybe Some Companies)

We’re in the early days of AI integration in corporate environments, and I’m willing to bet we’re about to see an avalanche of catastrophic mistakes.

Picture this:

A well-meaning infrastructure engineer, eager to speed up their workflow, starts using an AI-powered IDE like Cursor with the "auto-perform commands" feature enabled. They explain a problem—maybe not perfectly, maybe missing some key context—and the AI, confident as ever, starts running terminal commands the engineer has never even seen before.

Before they know it:

Database tables are dropped.
Configs are overwritten.
Prod is a smoking crater.

And the worst part? No one fully understands what happened because the AI made decisions based on incomplete or misinterpreted context.

Why This Will Happen Over and Over

Over-Trust in AI’s “Understanding”
- AI doesn’t reason—it predicts. If your prompt is ambiguous, it will still generate something, and that something might be rm -rf in the wrong directory.
The Illusion of Control
- Tools that auto-execute commands (like GitHub Copilot with shell gen, Cursor’s AI agent, etc.) remove the human review step. Engineers might assume the AI "gets it" until it very clearly doesn’t.
Silent Failures
- Unlike a human who might say "Wait, this looks dangerous," AI will happily run destructive commands with confidence. By the time logs are checked, it’s too late.

The Fallout: AI Will Kill Some Companies

We’ve already seen AI blunders:

Legal briefs citing fake cases (because the AI hallucinated precedents).
Customer service bots going rogue (see: Air Canada’s chatbot inventing refund policies).

Now imagine:

A fintech AI misinterprets a deployment script and wipes transaction records.
A cloud AI "optimizes" costs by deleting "unused" resources… like prod databases.

Some companies won’t recover from these mistakes—especially if they happen at scale.

How to Survive the AI Wild West

Treat AI Like a Junior Dev Who Lies Sometimes
- Always review generated code/commands before execution.
- Sandbox everything before it touches prod.
Disable Auto-Execute (For Now)
- Tools that let AI run commands directly are dangerous. Keep it in "suggestion mode" until guardrails improve.
Assume It Will Fail
- Log every AI-generated action.
- Build rollback plans for when (not if) AI breaks something.

The Bottom Line

AI is powerful, but we’re in the "move fast and break things" phase—except now, the "things" might be entire companies.

Brace for the chaos. The first wave of AI-induced disasters is coming.

What’s the worst AI blunder you’ve seen (or caused)? Drop your horror stories below. 👇

AI Won’t Replace Developers Anytime Soon—Here’s Why

Samuel Mutemi — Wed, 07 May 2025 18:15:53 +0000

(Critiques of the distant future, please consider the date of publication 😅)

The Promise (and Pitfalls) of AI-Generated Code

AI coding assistants like Cursor and Claude have been game-changers for developer productivity. They can scaffold entire applications, debug tricky issues, and even explain complex concepts in seconds. But can they fully replace software engineers?

Not yet—and here’s a funny (and frustrating) example of why.

The Case of the Broken Countdown Timer

I recently asked Cursor to generate a Next.js landing page with a countdown timer to my product launch. It did most of the work well—the UI looked great, the logic seemed sound, but when I tested it… the timer was stuck.

I alerted the AI, but instead of fixing the issue, it just:

Repeated the same code
Gave me a generic checklist (e.g., "Check if the date is correct")
Missed the glaring problem

Next, I pasted the code into Claude. It thought it was a hydration issue (a reasonable guess in Next.js) and tweaked the code—but the timer still didn’t work.

The Obvious Bug AI Missed

After some manual debugging (thankfully, I know JavaScript), I spotted the issue:

function getTimeLeft() {
  const launchDate = new Date(); // 🚨 Problem: This resets EVERY render!
  launchDate.setDate(launchDate.getDate() + 35); // 5 weeks from today
  const now = new Date();
  const diff = launchDate.getTime() - now.getTime();
  // ... rest of the logic
}

The bug? launchDate was being recalculated on every render, meaning the countdown always showed 0 (since now and launchDate were effectively the same time).

The fix? **Make launchDate a static field (From what future date are we counting down?):

const LAUNCH_DATE = new Date("2025-06-01");

function getTimeLeft() {
  const now = new Date();
  const diff = LAUNCH_DATE.getTime() - now.getTime();
  // ... rest of the logic
}

Why AI Still Needs Human Oversight

AI Lacks Deep Context
- It didn’t realize launchDate should be static.
- It followed patterns but didn’t understand the intent.
Debugging Requires Reasoning, Not Just Repetition
- Both assistants gave plausible suggestions but didn’t diagnose the root cause.
Trivial Mistakes Are Hard for AI to Spot
- Humans recognize "obvious" errors faster because we think in terms of goals, not just syntax.

The Verdict: AI is a Powerful Assistant, Not a Replacement

AI has made incredible progress, but it still:

✔ Struggles with nuanced logic

✔ Misses simple but critical bugs

✔ Needs human guidance for real-world scenarios

So, developers, rest easy, your job is safe (for now). AI is a tool, not a replacement. And honestly? That’s a good thing.

Mistakes Are Part of the Dev Environment: It's How You Handle Them That Counts

Samuel Mutemi — Wed, 07 May 2025 08:26:35 +0000

“To err is human, to debug divine.”

If you’ve been writing code for any stretch of time, you’ve probably stared at your screen wondering how something so small could break so much. Whether it’s a misplaced semicolon, an off-by-one error, or deploying to production with test credentials (guilty 😅), every developer—junior or senior—has stories of mistakes they’ve made.

Let’s face it: mistakes are not just part of the development process—they are the development process.

The Illusion of Perfection
There's a common myth, especially among beginners, that good developers don't make mistakes. That once you’re “senior,” you write perfect code the first time.

Spoiler: Nobody does.

In fact, senior developers often make bigger mistakes—but they catch and fix them faster because they’ve learned how to deal with them.

Why Mistakes Matter
Mistakes aren’t just inevitable—they’re useful. They reveal assumptions, force you to dig deeper into how systems work, and often lead to better solutions than you would have considered otherwise. Here’s what mistakes teach us:

Clarity: They expose where our understanding is shallow.
Resilience: They teach us how to handle failure calmly.
Humility: They remind us that there’s always more to learn.

A Culture of Blame vs. a Culture of Learning
One of the worst things a dev team can do is foster a culture where mistakes are punished. This leads to fear, cover-ups, and stagnation.

Great teams treat mistakes as data points. When something breaks, the question isn't “Who messed up?”—it’s “How can we make this less likely to happen again?”

That might mean:

Improving test coverage.
Setting up guardrails (like staging environments or automated linting).
Writing clearer documentation.
Doing post-mortems without finger-pointing.

Your Mistake Recovery Toolbox
Here are a few practical things every developer should keep in their mental toolkit when mistakes happen:

Stay Calm
Panicking never helps. Take a breath, grab a coffee, and remember: you are not your bug.
Reproduce It
If you can reproduce the bug, you can fix the bug. Step-by-step isolation is often the fastest way forward.
Use Version Control
If you’re not using Git (or similar), start now. Version control is the ultimate undo button.
Write Tests Around It
Write a test that fails because of the bug. Then fix it. This not only confirms the fix—it prevents regressions.
Learn and Share
Write about it, tweet it, blog it, bring it up in retros. Your mistake could prevent someone else’s.

Some Famous Dev Mistakes
GitHub once deleted part of their production database.

AWS once took down a major part of the internet with a typo.

Google once lost an entire day of email due to a bad config.

These are massive companies with brilliant engineers. Mistakes still happen. What sets them apart is how they respond.

Final Thoughts
Programming is a constant process of learning, breaking, fixing, and improving. Mistakes are not bugs in you—they’re features of the process.

So next time you bork the deployment or introduce a hard-to-find bug, don’t beat yourself up. Reflect, fix, learn—and maybe even laugh about it later.

Because at the end of the day, real growth in software engineering happens not in the absence of mistakes, but in how we respond to them.

Quantum Computing is About to Pwn Your Encryption – Time to Wake Up!

Samuel Mutemi — Mon, 05 May 2025 11:37:27 +0000

"Harvest Now, Decrypt Later" – The World’s Slowest Heist

Imagine a burglar breaking into your house, but instead of stealing your TV, they take a photocopy of your safe’s lock and say:

"I’ll crack this later when I invent lock-picking lasers."

That’s essentially what hackers are doing right now with "Harvest Now, Decrypt Later" (HNDL) attacks. They’re hoarding encrypted data (your emails, bank details, even those embarrassing selfies) and waiting for quantum computers to crack them open like a cheap piñata.

You: "But quantum computing isn’t ready yet!"

Hackers: "We can wait. Your data isn’t going anywhere."

Shor’s Algorithm: The Math Bully That Eats RSA for Breakfast

Current encryption relies on math problems so hard that even supercomputers cry trying to solve them. But quantum computers? They cheat.

RSA Encryption: "It’ll take a billion years to factor this large prime!"
Shor’s Algorithm: "Hold my qubit." 💥

What’s at risk?

✔ Your HTTPS connections (bye-bye, secure banking).

✔ Your SSH keys (hope you like unexpected server guests).

✔ Bitcoin & blockchain (unless they upgrade fast, quantum miners will be the new crypto whales).

Grover’s Algorithm: The Unwanted Gym Bro of Encryption

While Shor’s algorithm destroys RSA & ECC, Grover’s algorithm is more of a persistent annoyance to symmetric encryption like AES:

AES-256? Still strong, but now with only AES-128-level security.
SHA-256? Collision attacks just got way easier.

Translation: Your encryption just lost half its gains. Time to hit the quantum-resistant crypto gym. 💪

Post-Quantum Cryptography: The Superhero We Need (But Don’t Deserve)

NIST has been working on quantum-proof algorithms, because apparently, we can’t just unplug the quantum computers and call it a day. Here’s the new lineup:

1. CRYSTALS-Kyber – The New RSA (But Fancier)

Good for: Key exchanges (so quantum hackers can’t eavesdrop).
Bad for: People who miss the good ol’ days of RSA (which, let’s be honest, were never that good).

2. CRYSTALS-Dilithium – Like ECDSA, But Won’t Die in 5 Years

Replaces: Digital signatures (so your GitHub commits stay legit).
Bonus: Sounds like a Power Rangers weapon. "Go go Dilithium Signatures!"

3. SPHINCS+ – The Backup That Nobody Wants to Use

How it works: Hash-based, so even if quantum breaks everything else, this still stands.
Downside: Bigger, slower, like that one relative who still uses a flip phone.

What You Should Do Before Quantum Hackers Ruin Your Day

Stop pretending this isn’t happening.
- "Quantum computing is decades away!" – People who will be hacked in 5 years.
Check if your crypto is already obsolete.
- Still using RSA-2048? Start planning your migration yesterday.
Demand quantum-safe encryption in your tools.
- Ask your cloud provider: "Hey, when are you adding Kyber support?"
- If they say "What’s Kyber?" – panic.
Prepare for the inevitable "Oh crap" moment.
- Because someday, a headline will say: "Quantum computer just broke Bitcoin", and you don’t want to be scrambling then.

Final Thought: Don’t Be the Last One Using Broken Crypto

Quantum computing is coming, and it doesn’t care if your security team is ready. The good news? We have solutions now. The bad news? Most people won’t act until it’s too late.

Will you be the early adopter sipping coffee while others panic? Or the one rewriting your entire auth system at 3 AM after the quantum apocalypse hits?

The choice is yours.

"But I don’t even understand quantum mechanics!" – Don’t worry, neither do most quantum physicists. Just start learning post-quantum crypto today.* 😉

Passwords Are a Ticking Timebomb—And These Breaches Prove It

Samuel Mutemi — Mon, 05 May 2025 11:27:16 +0000

Passwords have been the default authentication method for decades, but their flaws are more dangerous than ever. High-profile breaches and cyberattacks consistently expose how fragile password-based security really is. Below, we’ll examine two major case studies that highlight why passwords are failing us—and what we should use instead.

Case Study 1: The LinkedIn Breach (2012, 2016, and Beyond)

What Happened?

In 2012, LinkedIn suffered a breach that exposed 164 million email and password combinations.
Hackers didn’t just steal passwords—they cracked weak hashes (SHA-1 without salting), revealing plaintext credentials.
In 2016, another batch of 117 million passwords from the same breach resurfaced on the dark web.

Why Passwords Failed

Weak Hashing: LinkedIn stored passwords with weak encryption, making them easy to crack.
Password Reuse: Many users reused the same passwords across multiple sites, leading to credential stuffing attacks on other platforms.
Delayed Impact: Even years later, these passwords were still being used in attacks.

The Aftermath

LinkedIn forced password resets, but the damage was done.
Many users who reused passwords saw their other accounts (email, banking, social media) compromised.

How It Could Have Been Prevented

✅ Passwordless Auth: Passkeys or biometric logins would have made stolen credentials useless.

✅ Better Hashing: Modern algorithms (bcrypt, Argon2) could have slowed down cracking.

✅ MFA Enforcement: Even with leaked passwords, MFA would have blocked unauthorized access.

Case Study 2: The Colonial Pipeline Ransomware Attack (2021)

What Happened?

Hackers breached Colonial Pipeline, a major U.S. fuel supplier, causing a six-day shutdown and fuel shortages across the East Coast.
The attack started with a single compromised password to an old VPN account that lacked multi-factor authentication (MFA).

Why Passwords Failed

No MFA: A single weak password was all hackers needed to infiltrate the network.
Legacy Account Exposure: The VPN account was no longer in use but wasn’t deactivated.
Password Reuse: The password may have been reused or easily guessed (though exact details weren’t disclosed).

The Aftermath

Colonial Pipeline paid $4.4 million in Bitcoin as ransom.
The U.S. government recovered some funds, but the incident highlighted how passwords alone are insufficient for critical infrastructure.

How It Could Have Been Prevented

✅ Passwordless VPN Access: A hardware security key (YubiKey) or certificate-based auth would have prevented the breach.

✅ Strict MFA Policies: Even a simple TOTP (Google Authenticator) check would have stopped the attack.

✅ Automated Account Deactivation: Unused accounts should be disabled automatically.

The Way Forward: Killing the Password for Good

These cases prove that passwords alone are a security liability. Here’s what we should adopt instead:

1. Passkeys (FIDO2 / WebAuthn)

No passwords, just biometrics or hardware keys.
Immune to phishing & credential stuffing.

2. Universal MFA Adoption

Mandate MFA everywhere, especially for remote access and admin accounts.

3. Better Credential Management

Password managers for generating and storing strong passwords (if still needed).
Regular audits to deactivate unused accounts.

Final Thoughts

Passwords are outdated, insecure, and costly. The LinkedIn and Colonial Pipeline breaches show just how dangerous reliance on passwords can be. The sooner we move to passwordless authentication, the safer we’ll all be.

Are you still using passwords, or have you switched to passkeys/MFA? Share your experience below! 🔐

Stop Using Keycloak Like a Basic Auth Server—Try These Features

Samuel Mutemi — Tue, 29 Apr 2025 14:20:02 +0000

Keycloak is widely recognized as a powerful open-source Identity and Access Management (IAM) solution, offering SSO, OAuth, and user federation out of the box. But beyond the basics, Keycloak has several underrated features that can significantly enhance security, customization, and usability.

In this post, we’ll explore some of these lesser-known Keycloak features that can save you time, improve security, and unlock new capabilities.

1. Fine-Grained Admin Permissions (Client Policies & Admin Fine-Grained AuthZ)

Most Keycloak admins know about realm roles, but did you know you can delegate admin permissions with surgical precision?

Client Policies: Define rules for client registrations (e.g., enforce HTTPS redirect URIs).
Fine-Grained Admin Permissions: Restrict admin console access (e.g., allow a user to manage only specific clients).

🔹 Use Case: Securely delegate administration without giving full realm access.

2. Dynamic Client Registration (DCR) & Initial Access Tokens

Instead of manually configuring every client, Keycloak supports dynamic client registration via REST API.

Initial Access Tokens: Generate short-lived tokens to allow self-service client registration.
Client Registration Policies: Enforce constraints (e.g., allowed redirect URIs).

🔹 Use Case: SaaS platforms where tenants need to onboard their own apps securely.

3. WebAuthn & Passwordless Authentication

While OTP is common, Keycloak supports FIDO2/WebAuthn for phishing-resistant logins.

Biometric & Security Key Auth: Users can log in via Face ID, Touch ID, or YubiKey.
Conditional Policies: Require WebAuthn only for high-risk actions.

🔹 Use Case: Financial apps or internal systems needing stronger MFA.

4. Token Exchange (Impersonation & Delegation)

Keycloak allows token exchange, letting one token be swapped for another under controlled conditions.

Impersonation: Admins can act on behalf of users (for debugging).
Delegation: A service can obtain a token for another service securely.

🔹 Use Case: Microservices architectures where services need to call each other.

5. Custom User Attributes & Declarative User Profiles

Keycloak supports custom user metadata, but with Declarative User Profiles (DUP), you can enforce validation.

Define Required Fields: Mandate certain attributes at registration.
Validation Rules: Enforce email formats, phone numbers, etc.

🔹 Use Case: Compliance-heavy industries needing structured user data.

6. Event Listeners & Webhooks

Keycloak emits real-time events (logins, token exchanges, failures). You can:

Forward Events to Kafka/RabbitMQ: For SIEM integration.
Trigger Webhooks: Notify external systems on user actions.

🔹 Use Case: Fraud detection, audit logging, or real-time analytics.

7. Script-Based Authentication Flows

Instead of hardcoding auth logic, Keycloak supports JavaScript/Python scripts in auth flows.

Custom Validation: Check user attributes before login.
Conditional Steps: Skip MFA for trusted IPs.

🔹 Use Case: Adaptive authentication without custom code.

8. Lightweight Directory Services (LDAP) with Write-Back

Keycloak can sync with LDAP/Active Directory bidirectionally.

User Write-Back: Changes in Keycloak (e.g., password updates) sync back to LDAP.
On-Demand Sync: Avoid full syncs with lazy loading.

🔹 Use Case: Enterprises migrating from legacy LDAP systems.

9. Built-In Token Revocation & Offline Sessions

Revoking tokens is usually manual, but Keycloak offers:

Not-Before Policy: Invalidate all tokens issued before a certain time.
Offline Session Limits: Control how long refresh tokens last.

🔹 Use Case: Responding to breaches or employee offboarding.

10. Themeable Emails & Localization

Most Keycloak emails look generic, but you can:

Customize Templates: Branded password reset emails.
Multi-Language Support: Auto-send emails in the user’s language.

🔹 Use Case: Improving UX for global user bases.

Final Thoughts

Keycloak is far more than just an OAuth server—it’s a swiss-army knife for IAM. By leveraging these lesser-known features, you can:

✅ Enhance security (WebAuthn, token exchange)

✅ Automate workflows (dynamic client registration)

✅ Improve UX (custom emails, declarative profiles)

Have you used any of these features? Any hidden gems I missed? Let me know in the comments! 🚀

Further Reading:

Would you like a deep dive into any of these features? Let me know!

DEV Community: Samuel Mutemi

The Two Joints: Where Agentic Engineering Breaks

Agentic Engineering Is Not Vibe Coding: The Three-Skill Loop I Use to Ship Distributed Systems

Where Does Truth Live? A Field Guide to Agentic Engineering Methodologies

The Real Problem: I Was the Wiring

The Flow

Joint One: Your Tests Can't Just Be a Copy of Your Spec

What Can the Test Actually Catch?

Two Kinds of Tests, and Only One Can Challenge the Spec

Spec Tests

Reality Tests

Three Habits That Keep the Spec Tests Honest

1. Write the Tests Before the Code — and Freeze Them

2. Hunt the Silences

Joint Two: What Happens When the Loop Decides the Target Was Wrong

The Failure This Exists to Prevent

The Stop, and the Four Verdicts

1. Test Class

2. Slices Already Merged

3. Smallest Fix

4. Reproduced

Stamp Everything With the Spec Version

Not Started Yet

In Flight

Already Merged

One More Guardrail

What Got Worse

Reality Tests Are Expensive

Escalation Fatigue

More Artifacts to Keep True

What I Still Don't Have

Provenance Tooling

The Next Generation of Senior Engineers

Where This Leaves the Series

Where Does Truth Live? A Field Guide to Agentic Engineering Methodologies

Agentic Engineering Is Not Vibe Coding: The Three-Skill Loop I Use to Ship Distributed Systems

The organizing question

1. Prompt-native development (vibe coding)

2. Context engineering (and its cousin, harness engineering)

3. Spec-driven development

Lean and constitution-driven, GitHub Spec Kit

Lightweight and brownfield-first, OpenSpec

Full agile simulation, BMAD-METHOD

Environment-native, Kiro, and the platform tier

Spec-as-source, Tessl and friends

4. Loop-based autonomy, the Ralph technique

5. TDD governance

6. Multi-agent orchestration

The axes that actually matter

Why I landed on spec-driven plus loop engineering

The spec is an anchor, and anchors let you re-derive

The loop is how the anchor gets corrected

Stated as control theory, because that's what it is

What I took from the others anyway

And what I rejected, plainly

The strongest objection to my position

There is no correct answer, and that isn't a cop-out

But the shape of the near-optimum is visible

Composing rather than choosing

What I'm building next

Agentic Engineering Is Not Vibe Coding: The Three-Skill Loop I Use to Ship Distributed Systems

The distinction, stated sharply

Spec-driven development plus loop engineering

The workflow: three skills, one handoff chain

1. /to-prd, turn a vague intent into a specification

The out-of-scope section is load-bearing

2. /to-issues, decompose into vertical slices

3. The engineering skill, execute, in parallel, with gates

Why payments make all of this non-optional

What actually got better

Now the honest part: I moved the bottleneck, I did not remove it

Code review does not scale with code generation

The answer is agentic validation, not more human eyes

The backlog moved downstream, and that is the actual story

What I actually want you to take away

What Payments Infrastructure Taught Me About Building Systems That Don't Break

📋 The short version

1. 🔁 Idempotency isn't a feature. It's a foundation.

2. ⚠️ Every gateway you integrate is a vendor, and vendors fail

🧱 You isolate them

1. `/to-prd`, turn a vague intent into a specification

2. `/to-issues`, decompose into vertical slices