DEV Community: Miguel Shinyenyi

You Cannot Stop What Arrives

Miguel Shinyenyi — Thu, 15 Oct 2026 01:51:52 +0000

What I notice

I cannot stop my thoughts. Try it yourself. Set a timer for thirty seconds and think of nothing. A thought will arrive before you pick it. This is my own observation. I have not checked the research on it.

Some thoughts arrive with weight. One kind is a standard I did not choose. It asks me to be someone before I have done anything. I cannot meet that kind by working, because it never says what finished looks like.

This piece started from a line in the Marvel series VisionQuest, about holding someone's memories without their feelings. I have not tried to explain the show. I took the line as a place to start thinking.

Three answers to an arrival

I see three ways to answer a thought that weighs too much.

Act on it at once, as if it were an order.
Shrink so the test never comes. If I never try, I cannot fall short.
Record what is true, decide what I can, and leave the rest marked as unknown.

The second answer is the one I want to look at. It feels like relief. It also removes the one thing that would tell me whether the weight was real.

The same shape in a settlement engine

My settlement engine cannot choose what arrives either. A payment gateway times out. Postgres deadlocks two transactions against each other. Neither event waits for permission. What I control is the state the engine writes next.

Here is the first decision, trimmed from SettlementService.createSettlement:

try {
    gatewayResult = externalSettlementGateway.execute(executionRequest);
} catch (Exception gatewayFailure) {
    gatewayResult = new GatewayResult(SettlementOutcome.UNKNOWN, null);
}
return finalizeWithRetry(executionRequest.settlementId(), gatewayResult);

A timeout means the engine does not know whether the gateway processed the payment. The code says so. It writes UNKNOWN, not CONFIRMED and not FAILED. The comment in the file gives the reason: leaving the settlement PENDING would leave its idempotency key IN_PROGRESS, and every retry with that key would get a 409 with no way to resolve it.

Here is the second decision, trimmed from finalizeWithRetry:

for (int attempt = 1; attempt <= MAX_FINALIZE_ATTEMPTS; attempt++) {
    try {
        return settlementTransactions.finalizeSettlement(settlementId, gatewayResult);
    } catch (TransientDataAccessException transientFailure) {
        if (attempt == MAX_FINALIZE_ATTEMPTS) {
            return finalizeAsUnknownAfterExhaustingRetries(settlementId, transientFailure);
        }
        meterRegistry.counter("settlement.finalize.retries").increment();
        sleepBriefly(attempt);
    }
}

The last branch is the part that matters. A comment above it says the Phase 9 load test found a bug. When the five attempts ran out with no fallback, the settlement stayed PENDING and its key stayed IN_PROGRESS forever. The failure was not the deadlock. Deadlocks were expected and retried. The failure was that the engine reached a point where it made no decision, and a state with no decision never ends.

The fix gave the engine a second budget of five attempts that writes UNKNOWN. UNKNOWN is a smaller claim than CONFIRMED. It is also true, and a later reconciliation can resolve it.

What I take from the comparison

I do not need to claim greatness, and I do not need to claim failure. Both are large claims, and the evidence for either does not exist yet. The smaller true claim is available: here is what I made today. For anything bigger I can write unknown, and let the record settle it later. My journal does the job that reconciliation does for the engine.

The stall is the bug. Shrinking so the test never comes is the PENDING state with the key stuck. It looks like safety. It blocks every later attempt.

Where the comparison stops

A person is not a service. Three differences matter.

The engine stops after ten attempts. A person should not cap attempts at ten.

The engine's UNKNOWN resolves against an outside system that holds the answer. A record of my own work does not hold the answer to my worth. It only holds what I did.

The engine decides in milliseconds. This decision takes years, and I may get it wrong in both directions. The feeling that I could do great things is a feeling, not evidence. It may be potential. It may be the weight of an expectation. I cannot tell which from inside.

What I do not know

I do not know how common the shrinking answer is. I am drawing on my own experience, not on data.

I read only createSettlement and finalizeWithRetry for this piece. I am still restudying the rest of the retry path, so I make no claim about how it behaves under load beyond what the code comments say.

What I took from it

I cannot stop what arrives. I can choose the state I write next.
A state that never ends is worse than an honest "unknown".
Make the next claim small enough to be true: what I made today.
Keep the record. It settles questions that a feeling at 4 in the morning cannot.

Originally published on my site.

A Convenience Is Hard to Give Back

Miguel Shinyenyi — Wed, 14 Oct 2026 15:18:31 +0000

The claim, as I understand it

This is my understanding of the "luxury trap" passage in Sapiens. It is not a checked account
of the text.

As I read it, farming was not given up because each change was small. A bit more grain, a bit
more settling, spread over many generations. By the time the cost showed, nobody remembered
another way to live. The extra food had already become the number people lived on.

My own observation, not the book's: the same thing happens now. A convenience arrives, it
becomes normal, and then it brings work of its own. Email saved time. It also made people
answer faster, and many of us now feel behind on it all day.

Three parts, as I see them

Each step is small and it helps. Nobody argues against it.
The step becomes normal, and other things get built on it.
Going back now costs more than staying, so it does not happen.

My earlier piece on the farming chapter asked what a measure counts and what it leaves out. This one
asks something else: why nobody steps back once the cost is visible.

The same shape in code

A payment call sometimes times out. Adding a retry is one small step. Callers stop handling
timeouts and the code gets shorter. This is a toy, run on one machine, with the reply lost
exactly once:

static String withRetry(Supplier<String> call, int attempts) {
    for (int i = 1; ; i++) {
        try {
            return call.get();
        } catch (TimeoutException e) {
            if (i == attempts) throw e;
        }
    }
}

Gateway plain = new Gateway();
String r1 = withRetry(() -> plain.charge(null, 5000), 3);
// Retry, no key:   receipt=R2 charges=2

The first call moved the money and lost the reply. The retry moved it again. The customer was
charged twice, and the convenience caused it.

The fix is a key made once and sent on every attempt. The server remembers it:

Gateway keyed = new Gateway();
String key = UUID.randomUUID().toString();   // made once, reused on every retry
String r2 = withRetry(() -> keyed.charge(key, 5000), 3);
// Retry, with key: receipt=R1 charges=1

That fixes it, and it adds work. The server needs a place to store keys. Something has to
decide how long a key lives. A second request can arrive while the first is still running, so
a key needs a state. In the settlement engine I built, the design has those pieces: keys with
an in-progress and a completed state, and a scheduled sweep for the case where a process
crashed before it finished, which would leave a key stuck in progress. I am describing the design as it stands, not the order I built
it in.

Now take the retry out. Every caller has stopped handling timeouts. Every server is built to
expect repeats. Removing one line is not a one-line change anymore. That is the third part
above, in a place I can read.

Where the comparison stops

The first farmers could not see where it led. I can see this one. I can read the cost in code
before I add the step, and I can choose.

That does not make the step reversible. It only lets me price it first. A habit I want to try
before adding any convenience: write down the work I will have to keep doing because of it,
and who will start depending on it.

What I do not know

I do not know whether AI tools save me hours, or raise what I expect of myself in the same
hours. That is the luxury trap question, asked of my own week. I have a feeling about it, and
a feeling is not a measure. A test I have not run: log hours and output for two weeks, with
and without the tool, and compare.

Other limits. I have not checked the chapter's evidence. The email point is my observation.
The retry example is a toy with one lost reply. The link to the settlement engine describes
its design, not a history of how I got there.

What I took from it

A convenience that helps today can add work you cannot stop doing tomorrow. Look for that work before you add it.
Small steps hide the cost. The first one is the cheapest time to price it.
Seeing the trap does not undo it. Reversing gets harder as other things depend on the step.
A feeling that a tool saves time is not a measure. Count the hours.

Originally published on my site.

More Copies Is Not a Better Life

Miguel Shinyenyi — Fri, 09 Oct 2026 14:06:05 +0000

The claim, as I understand it

This is my understanding of Part 2, Chapter 1 of Sapiens, "History's Biggest Fraud". It is
not a checked account of the text.

As I read it, the chapter says farming did not make people smarter or their lives easier.
The average farmer worked harder than the average forager and ate a narrower diet. The
chapter blames a few plants: wheat, rice, potatoes. They planned nothing. But the people who
tended them left more descendants, so the habit spread, and the plants now cover far more
ground than before. People settled next to the fields and worked to keep them alive. In the
chapter's framing, the plants domesticated us.

I can check one part of this outside the book. Researchers compare skeletons from before and
after farming arrived. I found two sources: a 2011 paper on stature and robusticity across the
agricultural transition, and a review titled "Foraging to Farming Transition: Global Health
Impacts, Trends, and Variation". I could not open either. I only confirmed that they exist.
Both titles point at health costs, and the second points at variation between regions. Treat
that as a lead, not a finding.

Two numbers that moved apart

The chapter's point, as I take it: evolution measures success in copies of DNA. More people
means more copies. By that measure farming won, for the plants and for our species. The same
measure has no column for how any one person's life went.

Farming kept more people alive under worse conditions. Both halves of that sentence are true
at once, because they count different things. One counts how many. The other counts how well.

The chapter also warns against judging thousands of years by how we live now. Today's
security and wealth sit on what farming made possible. A bad deal for the average farmer and
a base for later life can both be true.

Where my company comparison holds

I compared it to a company. A company's goal is to make money, not to keep its employees
happy. Revenue can rise while happiness falls, because happiness is not in the number.

That is the same structure: one measure, and people it does not count.

Where it stops holding

Wheat cannot notice anything. A company has people in it who can notice the gap and close it.
Some leaders treat employee happiness as a way to keep staff and raise output. Some treat it
as a goal. Either way, a person decided. Evolution has nobody to decide.

So the version of the comparison I trust is smaller. A system judged by one number drifts away
from the welfare of the people inside it, unless someone ties the two together on purpose.

A company whose growth and employee welfare both rise today has not shown that anyone tied
them. They may be moving together for now. A test: when the number is threatened, what gets
cut first? If employee welfare goes first, the tie was never there. I have not applied this
test to any company yet.

What I am not sure about

I have not read the chapter's figures on hours worked or diet. "Worked harder, ate worse" is
the chapter's claim to me, not one I checked. The skeleton evidence appears to vary by region,
going by the titles alone.

Farming also gave us things foragers did not have: surplus, people who could stop growing
food, writing, cities. "Fraud" describes one axis, the average person's daily life. I do not
know how to weigh it against the others, and I am not sure the chapter claims to.

What I took from it

A system can win by its own measure while the people inside it lose by theirs. Ask what the number counts and what it leaves out.
More is not better. Copies, revenue and output all count quantity.
A comparison is useful until it stops fitting. Say where it stops. Here: wheat cannot choose, a company can.
A measure lines up with people's welfare on purpose only if something protects the link when protecting it costs. What gets cut first shows which one the system serves.

Originally published on my site.

What the Reconciler Can't See

Miguel Shinyenyi — Fri, 09 Oct 2026 14:05:32 +0000

Explain it cold, then check

Same method as before: explain the subsystem from memory, then check it against the code.

My cold answer was this. Reconciliation is the last resort. When the system cannot settle a
payment on its own, it marks the payment UNKNOWN and pushes it to a place where someone is
alerted. That person looks, finds what went wrong, and either records an answer or edits the
data.

The code disagreed with that in three places. It also showed me a fourth problem I had not
thought to ask about.

Unknown and conflicting are different problems

ReconciliationService.runOnce loads every settlement that has an external reference and
compares each one to the external system's own record. This is what it does with one:

look up the external record
  none, under 5 min   -> wait
  none, over 5 min    -> mismatch
  amount or currency
    differs           -> mismatch
  settlement UNKNOWN  -> adopt external
  both CONFIRMED or
    both FAILED       -> nothing
  anything else       -> mismatch

I had filed UNKNOWN under "needs a person". It does not. If the settlement is UNKNOWN and the
external record agrees on amount and currency, the engine settles it by itself, using the same
method the original request would have used:

SettlementOutcome resolvedOutcome = externalRecord.status() == ExternalStatus.CONFIRMED
        ? SettlementOutcome.CONFIRMED : SettlementOutcome.FAILED;
settlementTransactions.finalizeSettlement(settlement.getId(),
        new GatewayResult(resolvedOutcome, settlement.getExternalRef()));

A person is needed only when the two sides contradict each other. The project's own doc gives
the reason: if a settlement is already CONFIRMED and the external system now disputes it, money
may have moved, and only a person can judge what to do. So the rule is: evidence that fills a
gap gets applied automatically. Evidence that conflicts waits for a person.

It runs on a timer, not on failure

A scheduler calls runOnce every 60 seconds by default, whether or not anything went wrong.
Each pass loads every settlement that has ever had a reference, with no date window and no
paging, and asks the external side about each one. That is fine at this size. At real volume,
the same query is also the cost.

The query decides what it can see

The candidate query is findByExternalRefIsNotNull(). A settlement only gets a reference
inside finalizeSettlement, and only when the gateway returns one:

if (gatewayResult.externalRef() != null) {
    settlement.setExternalRef(gatewayResult.externalRef());
}

Three paths end in UNKNOWN with no reference:

The gateway call throws. SettlementService builds new GatewayResult(UNKNOWN, null).
The finalize retry budget runs out. The fallback uses the same null.
The stale-pending sweep finalizes an orphaned PENDING settlement as UNKNOWN, also with null.

None of these can reach the auto-resolve step, because reconciliation never loads them. They
do not open a mismatch either. They stay UNKNOWN.

The case that worries me most is the second failure type from the project's design doc: the
external system processed the payment, and the response was lost. The money moved. The engine
never learned the reference. The project's own open-questions list names this and two fixes:
ask the external system what it received for this settlement, or match on amount, date and
account. Neither is built. I checked the code instead of trusting the doc, and the doc was
right.

Nobody is told

I said the person gets alerted. In the code, a new mismatch is published as the event
reconciliation.mismatch_found. Nothing consumes it. The Kafka doc says "No consumer yet". The
Prometheus rules cover three things: the backend being down, the ML service being down, and a
spike in failed settlements. None covers mismatches.

So a mismatch waits until someone opens the reconciliation page. A record in a table
guarantees a person can find the problem. It does not guarantee a person will.

Recording a decision is not making it happen

The resolve endpoint takes a reason, marks the mismatch resolved, and writes an audit row and
an event. Its own description says it does not change the settlement or the ledger.

For ledger mismatches the docs say what follows: if the data still disagrees, the next read of
that account opens a new mismatch. Reading flagMismatch, the same should hold for settlement
mismatches, since it only skips a settlement that already has an open row. I read that. I have
not run it.

No endpoint corrects data. REVERSED exists in the state machine and is reachable from
CONFIRMED, but nothing in the code ever moves a settlement there. Today the correction is a
person editing the database by hand. That edit skips the state machine, the ledger entries and
the outbox event, because all three live in application code.

My first answer was "a person edits it". Keeping the decision with a person is right. The
open question is who carries it out. Either the person edits rows directly, or the person
decides and the engine executes it through the same paths as every other money movement. I
have not decided between them.

What I took from it

Evidence that fills a gap can be applied automatically. Evidence that conflicts needs a person.
A checker sees only what its query returns. Ask what the query cannot return.
A record in a table is not an alert. Someone has to be told.
Recording a decision does not carry it out. Decide who does.

Originally published on my site.

What Becomes True When Both Stay

Miguel Shinyenyi — Tue, 29 Sep 2026 22:54:22 +0000

The curtain of silence

Sapiens describes the evidence for prehistoric life as a Rorschach test. A burial site with
ornaments on a skeleton might mean hierarchy. It might mean something else entirely. A
fracture on an ancient bone might mean violence. It might not. The book calls this its
Curtain of Silence, evidence thin enough that even careful scholarship can't get a clean
answer through it. Scholars keep asking anyway, because the past still shaped what came
after it, but the honest version of that inquiry holds its answers loosely.

That's a claim about archaeology. It turned out to be useful somewhere closer to home.

Two truths instead of one

Dialectical thinking, how my mind works?,
it generates a position and its counter-argument at the same time. Useful when pointed
outward. Corrosive when pointed inward, because the counter-argument doesn't sit next to the
first position, it tries to erase it. Grateful for the past, but it cost me something real,
failing a job interview among the costs. Forgiving someone, but not reconciled with them.
The old move was picking one of those as the real feeling and discarding the other as noise.

The question that actually helps isn't which one is true. Both are. The question is what
becomes true if I stop forcing a choice between them.

A system that already does this

The settlement engine I built has a state called
UNKNOWN. It's not a bug, it's a deliberate third option, used when a payment might have
gone through or might not have, and the system genuinely doesn't have enough information yet
to say which.

// Reused across three different situations: a real gateway failure,
// exhausted retries after a database deadlock, and a hard crash that
// left a settlement stuck with no way to confirm what happened.
settlementTransactions.finalizeSettlement(settlementId,
        new GatewayResult(SettlementOutcome.UNKNOWN, null));

The tempting design is to force an answer immediately: retry until you get a clean yes or no,
because "unknown" feels like a failure to decide. The actual design holds UNKNOWN as a
first-class state, moves on, and lets a separate reconciliation process resolve it later,
once there's real information to resolve it with. Forcing a premature CONFIRMED or FAILED
would be lying to the rest of the system, papering over uncertainty that was still genuinely
open. The honest version of "I don't know yet" turns out to be more useful than a fast, fake
certainty. That's the same shape as forgiving without reconciling. Not every open state needs
collapsing into a resolved one on a deadline. Some of them get resolved later, correctly,
once there's enough to resolve them with. Some of them are just allowed to be two things at
once, permanently, the way gratitude and grief can both be true about the same period without
either one cancelling the other.

Where this gets tested, not just described

I've also been carrying a specific belief:
that my anxiety comes from a need to control everything. It's a real, documented idea in
general, intolerance of uncertainty, anxiety tracking unpredictability rather than actual
danger, controlling behavior as an attempt to shrink that unpredictability back down. But a
documented general pattern isn't the same as a verified account of one specific person, and
right now I don't have an instance attached to it, just the label.

There's a code parallel here too, and it's not a flattering one. The same idempotency work
this week turned up two concurrency races, both caused by the system trying to guarantee
too much certainty upfront.

Two requests, same key, same instant.

Option A: lock everything before
either proceeds. Force certainty
immediately.

Option B: let both proceed. The
database's unique constraint decides
who wins. The loser recovers by
reading the winner's result.

The actual code takes option B. Locking everything upfront to force immediate certainty is
exactly what produces a deadlock later, two transactions each holding what the other one
wants, both stuck, because the system tried to control too much of the outcome up front
instead of accepting a controlled version of not-knowing-yet and resolving it after the
fact. The fix that shipped this week for the crash-recovery gap follows the same shape: not
tighter control at the moment of failure, a sweep that runs later, accepts the outcome was
genuinely unknown for a while, and resolves it once enough time has passed to trust the
answer.

I don't know yet whether "addiction to control" is actually what's driving the anxiety, or
just the label that was closest at hand when I needed one. That's not resolved in this
piece, on purpose. If forcing a system to decide before it has enough information produces a
deadlock, forcing a belief about myself into a tidy answer before I've actually tested it
probably isn't any more reliable. The honest state to hold it in, for now, is UNKNOWN.

What I took from it

Two things can both be true about the same period. Stop asking which one is real.
UNKNOWN is an honest state. A fast, fake certainty is worse than a slow, correct answer.
Forcing certainty upfront is what caused the deadlocks. That applies to beliefs about myself too.
"My anxiety comes from control" stays a hypothesis until a real instance tests it.

Originally published on my site.

Knowing and being able to explain are not the same skill

Miguel Shinyenyi — Tue, 29 Sep 2026 22:53:50 +0000

Two different skills

There's a difference between recognizing the shape of a solution and being able to derive it
without the shape already in front of you. The first is pattern matching. The second is
understanding. Most technical education optimizes for the first, because it's faster to teach
and easier to test.

Where the gap shows

The gap only shows up when someone asks "why does it work that way" instead of "does it work."

What I took from it

Recognizing a solution and deriving it are separate skills, and practicing one doesn't train the other.
"Does it work?" tests recognition. "Why does it work that way?" tests understanding. Ask myself the second one.
A gap that only appears under a specific question is worth testing directly, not assuming.

Originally published on my site.

Claim, Argument, and the Discipline of Not Fooling Yourself

Miguel Shinyenyi — Tue, 29 Sep 2026 22:53:19 +0000

The trap

I did a set of exercises this morning to raise my energy, then showered, because I've learned
I need to feel physically settled before my mind is any good to me. Only afterward, reading a
chapter about hunter-gatherer instincts colliding with modern life, did I have language for
why. The insight came after the action, not before it. That ordering turned out to matter
more than the insight itself.

Here's the trap I nearly walked into. I read a compelling argument, that a mismatch between
old wiring and new environments explains a lot of modern unease, and I started treating it as
settled. Not because the evidence demanded it, but because the writing was good. A well-made
argument and a demonstrated fact produce the same feeling of certainty in a reader. That
feeling is not evidence. It's craft.

The distinction that actually matters

A claim is something shown to be true, checked against reality, falsifiable and unfalsified.
An argument is something made to be believed, structured, sequenced, persuasive by design.
Good nonfiction is full of arguments dressed in the confidence of claims, and the better the
writing, the harder that becomes to notice. The specific idea I was reading, that suppressed
movement contributes to depression, is a real argument worth taking seriously. It is not a
settled account of what causes depression, which research treats as a tangle of contributing
factors, not a single mechanism. Losing that distinction wouldn't just cost me an accurate
view of one book. It's the kind of thing that quietly reshapes how you read your own career
decisions, your own dissatisfaction, your own choices, once a persuasive frame is sitting
unexamined in the back of your head.

There's a second version of the same trap, closer in: I'd summarized what I'd read, in my own
words, and then treated my own summary as if it were a faithful account of the source. It's
entirely possible I misread the chapter. My understanding of an idea and a verified account of
that idea are not the same thing, and conflating them means any misreading I make gets
laundered into something that looks like the author's authority instead of my own.

Where I'd already been doing this right

The clearest working example of separating claim from argument that I have access to isn't
philosophical at all. It's a settlement-engine system I built myself, with Claude Code, and
am now restudying deliberately, not to relearn it from scratch, but to make sure I actually
know it well enough to explain it, in passing or in detail, on demand. That gap, between
having built something and being able to explain it under pressure, is one I've caught in
myself before. This restudy is a direct test of it.

What I found worth studying closely is a documentation habit built into the project itself: a
decisions log that doesn't just record what was decided, it records why, one sentence per
decision, stated plainly enough to be checked. "Foreign key validation failures return 400,
not 404, because the bad ID is client input error, not a missing resource being fetched
directly." That's not a conclusion asserted with confidence. That's a reasoned claim, sitting
next to the reasoning that produced it, inviting disagreement rather than discouraging it.

Compare that to a decisions log that just says "we chose 400." Both read as authoritative if
you don't look closely. Only one of them actually lets you check the reasoning instead of
just trusting the confidence of the person who wrote it. That's the whole trick. Confidence is
cheap. Showing your work is not.

What this looks like day to day

In reading: hold two separate questions for anything persuasive. What is being claimed, and
what is being argued for. Notice which one produced your certainty. If it's the second, the
certainty is borrowed from the writing, not earned from the evidence.

In relationships and self-observation: distinguish what happened from what you concluded it
meant. A single explanation that fits doesn't mean it's the only one, or the right one, it
means it's plausible enough to feel finished. Plausible and finished are not the same
condition.

In code, and in any documentation that describes a system: treat a decisions log entry, a
comment, or a design doc the way you'd treat a persuasive book, with the same two questions.
Does this line state a checkable reason, or does it just assert a conclusion with confidence?
A codebase full of confident assertions with no reasoning attached is tunnel vision waiting to
happen, for whoever reads it next, including the person who wrote it, six months later,
having forgotten why. That's precisely the risk I'm restudying my own system to guard
against, having built it doesn't guarantee I could still explain it cold.

The actual discipline isn't suspicion of everything. It's narrower than that: know which of
your beliefs are standing on demonstrated ground and which are standing on good writing, good
confidence, or your own possibly-mistaken summary of something else. Most of the time you can
hold both kinds of belief at once, loosely, without needing to resolve which is which
immediately. The problem only starts when you stop checking.

No tidy ending

I don't have a tidy ending for this one, and I'm treating that as correct rather than
unfinished. The whole point was to stop mistaking a well-argued position for a demonstrated
one. Reaching a neat conclusion here would be exactly the thing I just spent this piece
arguing against doing.

What I took from it

Certainty from good writing feels the same as certainty from evidence. Check which one I'm standing on.
My own summary of a source is not the source. Mark it as mine.
A decision is only checkable when its reason is written next to it, in code and in life.
Not every belief needs resolving now. The problem starts when I stop checking.

Originally published on my site.

A Mind That Lies to You Too

Miguel Shinyenyi — Tue, 29 Sep 2026 22:52:46 +0000

What sapiens traded time for

Every other species that reached a new environment got there the slow way: fins, blubber, a
nose built as a snorkel, millions of years of the body itself changing to fit the place. When
sapiens reached Australia, the body that arrived was the same one that had walked out of
Africa. No gills, no fur suited to a new climate. What made the crossing possible was a mind
good enough to understand the sea and build something that could cross it, in one lifetime,
not a species-length of them.

That's a real trade, not a free upgrade. Every other animal is capped by what its body allows
it to take from an environment. A mind with a plan has no such cap, and it showed almost
immediately. Sapiens gives the number: 23 of 24 Australian animal species over 50kg were gone
within a few thousand years of that crossing. The local megafauna had never seen anything
like this arrival and had no instinct to fear it. In Africa, animals evolved alongside early
humans for close to two million years and learned caution the slow way. Everywhere sapiens
reached quickly instead, that warning never had time to happen.

The same trade explains something closer to us than a wombat the size of a hippo. Before
sapiens spread, several other human species were alive at the same time, Neanderthals,
Homo erectus, others. Sapiens is the only one left. Whatever let a mind out-hunt giant
kangaroos also let it outcompete or eliminate every other kind of human. The mind that solved
the sea didn't stop at animals.

The same tool turns on you

Here's the part that doesn't stay in the past. A mind capable of outthinking millions of
years of evolution is not automatically trustworthy about itself. The same faculty that plans
a boat also generates a justification for whatever you've already decided, and does it just
as fluently. It doesn't announce which mode it's in. A conclusion produced by looking at
evidence and a conclusion produced by protecting a belief you're attached to feel identical
from the inside.

That's not a flaw sapiens escaped by getting smarter. It's the same trade, still running. A
mind good enough to reshape a continent is also good enough to reshape your own account of
yourself, and it will do that for free, without being asked, unless something is actually
checking it.

A short list for checking it

I've been trying to internalize a set of tools for exactly that, and grouping them, not as
ten separate items but as four jobs:

Catching a thought as it forms, and catching the pattern behind thoughts that keep repeating.
One is metacognition, the other is naming your own bias, and they're the same muscle aimed at
two grain sizes.

A test to apply once something's been caught: what evidence would actually prove this wrong?
If the honest answer is nothing, it's stopped being a belief and become something being
protected instead.

The same discipline aimed at feelings instead of thoughts: a feeling is real, but it isn't
automatically information about the world. "This feels like it will fail" is a fact about
right now, not a fact about the plan.

And a decision-making set for anything aimed outward: every choice costs the other choices
you didn't make, small repeated choices compound into large outcomes over time, and a good
decision can still produce a bad outcome, so judge the decision, not just how it turned out.

I don't know yet whether these four groupings hold up as anything more than a convenient way
to remember ten words. That's not settled here on purpose. It's a framework being tried, not
a conclusion being reported.

One place it got tested this week

The clearest test so far wasn't abstract. I explained the ledger in this system's settlement
engine cold, described it as checked double-entry bookkeeping, the kind where a balance is
provable, not just trusted. Then I looked:

public interface LedgerEntryRepository extends JpaRepository<LedgerEntry, UUID> {
}

Nothing in the actual code ever read those entries back. The check I'd described as already
existing didn't exist. That's falsifiability doing its job in real time, not as a concept I
was reading about but as a belief about my own work that had a real, checkable answer, and
the answer was no. The honest move afterward wasn't to feel bad about being wrong. It was to
notice that the explanation had felt just as confident either way, right or wrong, which is
exactly the warning the concept is supposed to be. Confidence was never the signal. Whether
there was a way to be proven wrong was.

What I took from it

A mind good enough to outthink evolution isn't automatically trustworthy about itself. That's the same trade, not a different one.
Confidence in an explanation isn't evidence for it. Whether it could be proven wrong is.
The four groupings are a framework being tested, not a conclusion. One real test this week went the way the framework predicts; that's one data point, not a pattern yet.

Originally published on my site.

The Outbox Held, the Ledger Didn't

Miguel Shinyenyi — Tue, 29 Sep 2026 22:42:43 +0000

Explain it cold, then check

Same method as idempotency: explain the mechanism from memory, then check it against the
actual code, not the other way around. The point isn't to be right on the first try. It's
that a claim about your own system should be falsifiable. If I can't say what I'd expect to
find in the code that would prove my explanation wrong, I'm not really explaining it, I'm
just repeating a shape that sounds right.

Two subsystems, two different results. The outbox explanation held up once checked. The
ledger explanation didn't, not because the mechanism I described was wrong, but because the
part of it I assumed existed simply wasn't built yet.

What the outbox is actually for

The instinct is to describe it as a receipt: proof you sent something, so you can resend it
if it doesn't land. That's close, but backward. The real problem is that a database commit
and a message published to Kafka are two separate systems, and you can't make them one atomic
operation. Publish first, then save to the database, and a crash in between means you told
the world something happened that your own database never recorded. Save first, then
publish, and a crash in between means your database says it happened but nobody downstream
ever hears about it. Either order leaves a gap where the two systems can disagree.

The fix is to make the event write local, so it can share a transaction with the fact it's
describing:

@Transactional
public SettlementExecutionRequest createPendingSettlement(...) {
    ...
    settlementRepository.save(settlement);

    outboxWriter.write(AGGREGATE_TYPE_SETTLEMENT, settlementId, KafkaTopics.SETTLEMENT_REQUESTED,
            new SettlementRequestedEvent(settlementId, source.getId(), destination.getId(),
                    command.amount(), command.currency(), Instant.now()));
    ...
}

Same transaction, same database. The settlement and the event to announce it commit together
or not at all. A separate process then polls for unpublished rows and sends them on, only
marking a row done once the broker actually confirms it:

kafkaTemplate.send(event.getTopic(), event.getAggregateId().toString(), event.getPayload()).get();
event.markPublished();

If that send fails, the row just sits there, unpublished, and gets tried again next poll.
What this buys isn't proof after the fact. It's a guarantee that the event and the thing it
describes can never come apart: if the settlement never happened, no event exists to send. If
it did happen, the event goes out eventually, whatever crashes in between.

What double-entry is supposed to guarantee, and what wasn't there

Double-entry means every transaction writes two sides, a debit somewhere and a credit
somewhere else, and across any transaction the two must net to zero. The point of doing it
this way is that a balance stops being something you just trust. It becomes something you can
prove, by summing the entries that produced it.

The code writes both sides correctly:

source.debit(settlement.getAmount());
destination.credit(settlement.getAmount());
ledgerEntryRepository.save(new LedgerEntry(UUID.randomUUID(), settlementId, source.getId(),
        EntryType.DEBIT, settlement.getAmount()));
ledgerEntryRepository.save(new LedgerEntry(UUID.randomUUID(), settlementId, destination.getId(),
        EntryType.CREDIT, settlement.getAmount()));

But balance is also a stored column, mutated directly by debit()/credit() in that same
method. And this is the whole repository interface for the entries just written:

public interface LedgerEntryRepository extends JpaRepository<LedgerEntry, UUID> {
}

No query beyond what comes free. Nothing anywhere sums those entries back and checks them
against the balance. The entries were being written correctly and never read. A ledger that
nobody checks isn't really double-entry, it's an audit trail sitting next to a number
everyone just trusts.

Deciding what happens when it's wrong

Finding the gap is the easy part. The harder question is what a mismatch should actually do
once the system can see one.

The tempting answer is to auto-correct: recompute the balance from the entries and overwrite
it. That's the wrong incentive to build into the system. A mismatch means you don't yet know
which number is wrong, the stored balance or an entry. Auto-fixing assumes an answer you
don't have, and it means a real bug gets silently absorbed instead of surfaced, which is
exactly how a system keeps producing the same kind of failure: nothing about the mismatch
ever reaches whoever could actually fix the cause. docs/reconciliation.md already states
this as policy for a different case, mismatches with an external gateway: manual review by
default, no silent auto-resolution. That's not a rule specific to talking to a bank. It's the
right rule for any internal disagreement a system can detect but can't safely resolve on its
own, so it's now written that way, a general rule with two implementations rather than one
rule that only happens to apply once.

The fix: every account gets an opening entry so its balance always equals the sum of its own
history, and a check runs on every read. On a disagreement, it records the mismatch, once,
durably, and still refuses to hand back a number it doesn't trust:

public void verify(LedgerAccount account) {
    BigDecimal computed = ledgerEntryRepository.sumNetByAccountId(account.getId());
    BigDecimal stored = account.getBalance();
    if (stored.compareTo(computed) == 0) {
        return;
    }
    if (ledgerMismatchRepository.findByAccountIdAndResolutionStatus(account.getId(), OPEN).isEmpty()) {
        ledgerMismatchRepository.save(new LedgerMismatch(...));
    }
    throw new LedgerInconsistencyException(account.getId(), stored, computed);
}

Resolving that record later never touches the balance either. It records that a human looked
at it and made a decision, the same as the existing gateway-mismatch workflow. Verified
against real data before calling it done: forty-six settlements moved through the actual API,
every untouched account's backfilled history matched its original balance exactly, and one
account whose balance was hand-edited to disagree with its own history was caught,
flagged, and resolved without its balance being touched by the fix itself.

What I took from it

A claim about your own system should be falsifiable. If nothing in the code could prove it wrong, it isn't really an explanation.
The outbox isn't proof you sent something. It's a guarantee that an event and the fact it describes can never come apart.
Entries that are only ever written and never read aren't a checked ledger, whatever they're called.
When a system finds its own mistake, the honest move is to record it and ask, not to quietly fix it and move on.

Originally published on my site.

Idempotency keys, under load

Miguel Shinyenyi — Tue, 29 Sep 2026 22:42:41 +0000

The textbook version

An idempotency key is a client-generated identifier attached to a request, so that if the
same request arrives twice, the server recognizes it and returns the original result instead
of processing it again. That's the textbook version. The real version, the one that shows up
once you put a system under actual concurrent load, has two more layers to it.

The guarantee is "answer it correctly"

The guarantee isn't "block the duplicate," it's "answer it correctly." A retry of an
in-flight request gets told the process is underway. A retry of a finished request gets
back the exact same result the first one got, replayed from a stored snapshot, not a fresh
answer. That second case is the actual point. A caller retrying after a timeout isn't told
"processing," they're told what genuinely happened.

Here's the check, in the real code:

Optional<IdempotencyKey> existing = settlementTransactions.findExisting(idempotencyKey);
if (existing.isPresent()) {
    return handleExisting(existing.get(), requestHash);
}
try {
    executionRequest = settlementTransactions.createPendingSettlement(idempotencyKey, requestHash, command);
} catch (DataAccessException raceLost) {
    IdempotencyKey winner = settlementTransactions.findExisting(idempotencyKey)
            .orElseThrow(() -> raceLost);
    return handleExisting(winner, requestHash);
}

Race one: same key, same instant

Check-then-write always
has a gap. Two requests can both see nothing during findExisting, before either has
written anything. Rather than closing that gap with an upfront lock, the code lets both
proceed, and leans on a database-level unique constraint to guarantee only one insert
survives. The other throws, and gets recovered by re-reading whoever won.

What's not obvious from reading the code: that failure doesn't always look the same. Under
Postgres, the identical race sometimes surfaces as a clean unique-constraint violation, and
sometimes as an outright deadlock between the two competing index insertions, depending on
timing. That's not something you'd predict from the schema. It's something you'd only find
by actually running concurrent load against it and watching what comes back.

t1  A      findExisting(key) -> empty
t1  B      findExisting(key) -> empty
t2  A      INSERT idempotency_keys
t2  B      INSERT idempotency_keys
t3  DB     unique constraint on key:
           only one INSERT wins
t3  loser  fails as EITHER
           - a unique-violation, or
           - a deadlock
             (CannotAcquireLock)
           depending on index timing
t4  loser  catch(DataAccessException)
           -> findExisting(key)
           -> return winner's result

Race two: the winner deadlocks

Race two is a different race, on the winner. The losing threads above, in their own
doomed transactions, are inserting into settlements, which has a foreign key back to
idempotency_keys. Validating that foreign key takes a shared lock on the referenced row.
If those losing transactions haven't rolled back yet when the winner tries to UPDATE that
same row to mark it complete, the update deadlocks against locks held for an unrelated
reason.

1  Winner  createPendingSettlement()
           commits
2  Loser   INSERT INTO settlements
           (FK -> idempotency_keys)
           takes a shared lock on
           that row
3  Loser   rolling back, lock still
           held
4  Winner  finalizeSettlement():
           UPDATE idempotency_keys
           SET status = COMPLETED
           -> blocked on that lock
5  DB      detects deadlock,
           aborts the UPDATE
6  Winner  catches
           TransientDataAccessException
           retries finalizeSettlement
           only (already PENDING;
           rerunning the whole flow
           would see its own key as
           a conflict)
7  Loser   rollback completes,
           lock released
8  Winner  retry succeeds

Five bounded retries, short backoff. If all five are exhausted, still under heavy
contention, the system doesn't leave the settlement stuck. It falls back to a second,
independent retry budget that finalizes the settlement as UNKNOWN, the same "we genuinely
don't know, let reconciliation resolve it later" state used when the external call itself
fails outright. Both concurrency behaviors were found by actually running load and chaos
tests against the system, not by design review.

A third failure mode: the crash

This one was found by studying the first two closely enough to explain them. Neither race above covers what happens if the process hard-crashes between the two
transactions entirely, after the settlement is written as PENDING but before it's ever
finalized. No thread survives to catch anything in that case, and the settlement has no
external reference yet, since that's only set during finalization, which made it invisible
to reconciliation's own query. The idempotency key would stay IN_PROGRESS indefinitely,
409-ing every retry, forever. That was a real, verified gap, confirmed against the actual
repository before writing it down here, not assumed.

It's fixed now. A scheduled sweep, StalePendingSettlementSweepService, runs the same way
reconciliation already does: it finds settlements stuck PENDING with no external reference
past a grace period (300 seconds by default), and finalizes each one as UNKNOWN, reusing
the exact same "we don't know, reconciliation resolves it later" path the two races above
already fall back to. No new state-machine logic, just the same escape hatch applied to a
case that previously had no path to it at all. Covered by its own unit and integration
tests, the way everything else in this system is.

The interesting part was never the key itself. It's every place where "the same request
happening twice" turns out to have more than one way of actually happening, and every place
"finished" turns out to have more than one way of actually finishing.

What I took from it

An idempotency key's job is to answer a retry correctly, not just to block it.
Let the database's unique constraint decide the race instead of locking upfront.
The same race can fail two different ways. Only real concurrent load showed that.
Every path needs an exit to UNKNOWN, including a crash nobody survives to catch.

Originally published on my site.