DEV Community: Mark Flame

Idempotency Is the System Design Concept That Actually Saves You

Mark Flame — Sat, 11 Jul 2026 18:26:54 +0000

Idempotency Is the System Design Concept That Actually Saves You

If you ask backend engineers to name system design concepts, you'll get
"scalability," "load balancing," "caching," "sharding" — the big, glamorous
ones. Idempotency rarely makes the list, and I think that's backwards. It's
one of the few concepts that quietly determines whether your system behaves
correctly the moment anything goes wrong, and things going wrong — networks
dropping, clients retrying, workers crashing mid-job — isn't an edge case in
a real system. It's Tuesday.

What idempotency actually means

An operation is idempotent if running it once has the same effect as running
it multiple times. PUT /users/42 { name: "Chika" } is idempotent — do it
once or five times, the user's name ends up "Chika" either way. POST /orders that creates a new order every time it's called is not — call it
twice and you've got two orders, possibly two charges.

That's the textbook definition. The part that matters in practice is why
you should care: the network doesn't give you exactly-once delivery. A
client sends a request, the server processes it successfully, and the
response gets lost on the way back — timeout, dropped connection, whatever.
From the client's point of view, it looks exactly like the request never
arrived. The only sane thing to do is retry. Which means every operation
your system exposes to retries needs to survive being executed more than
once for the same logical intent.

Where this bites you if you don't design for it

Payment and order creation. The classic example for a reason. A client
submits a payment, the request succeeds server-side, the response times out
before the client sees it, the client retries. Without idempotency
protection, that's a double charge — and it's the kind of bug that doesn't
show up in testing, because your test environment doesn't have a flaky
network. It shows up in production, at scale, as a support ticket queue.

Message queue consumers. Most queue systems (SQS, RabbitMQ, Kafka
consumer groups) give you at-least-once delivery, not exactly-once, as
the default guarantee — a message can be redelivered after a consumer
crashes before acknowledging it, even if the work was actually completed.
If your message handler isn't idempotent, a crash-and-redeliver cycle
silently duplicates whatever side effect that handler causes.

Distributed writes across services. Anytime one service calls another
and the caller can't be certain whether the callee actually completed the
operation before the connection dropped, you have the same problem in
miniature. Retrying blindly assumes non-idempotent operations are safe to
repeat, which is usually the wrong assumption.

How to actually build it in

Idempotency keys. The client generates a unique key per logical
operation (a UUID, typically) and sends it with the request. The server
checks: have I seen this key before? If yes, return the stored result of the
original operation instead of re-executing it. If no, execute it and store
the result against that key. Stripe's API is the reference example most
engineers have actually used as a client — send the same Idempotency-Key
header twice, get the same charge result twice, not two charges.

async function createPayment(req, res) {
  const { idempotencyKey, amount, userId } = req.body;

  const existing = await IdempotencyRecord.findOne({ key: idempotencyKey });
  if (existing) {
    return res.status(existing.statusCode).json(existing.response);
  }

  const payment = await processPayment(userId, amount);

  await IdempotencyRecord.create({
    key: idempotencyKey,
    statusCode: 201,
    response: payment,
  });

  return res.status(201).json(payment);
}

Natural idempotency via upserts. Sometimes you don't need a separate
key at all — the operation itself can be made idempotent by its own
structure. Instead of "insert a new record," do "upsert keyed on a natural
unique identifier" (updateOne({ orderId }, { $set: data }, { upsert: true })). Run it once or five times, you get the same end state, because the
write is keyed on something that already uniquely identifies the logical
entity rather than creating a new one each call.

Set-based state transitions instead of increments. SET status = 'shipped' is idempotent. UPDATE inventory SET quantity = quantity - 1 is
not — run it twice and you've decremented twice. Where possible, prefer
writes that assign an absolute value or a range-conditioned transition over
writes that apply a relative change, precisely because the former survives
duplication and the latter doesn't.

What idempotency doesn't solve

It's not a substitute for genuine distributed transactions when you need
strict atomicity across multiple resources — it solves "this specific
operation is safe to retry," not "these five operations across three
services either all happen or none do." Sagas and compensating transactions
address that broader problem; idempotency is what makes each individual
step in a saga safe to retry when a step fails partway through, which is
usually why the two show up together in the same system.

It also doesn't remove the need to think about concurrency. Two different
clients hitting the same idempotent endpoint at the same time with
different idempotency keys is a race condition idempotency does nothing
for — that's a locking or optimistic-concurrency problem, a separate
concern that idempotency sometimes gets mistaken for solving.

The actual habit worth building

Before writing any endpoint or message handler that causes a side effect —
a write, a charge, a notification, a state change — ask one question: what
happens if this runs twice for the same logical request? If the honest
answer is "something bad," that's the signal to add an idempotency key, an
upsert, or a set-based transition before it ships, not after the first
double-charge ticket comes in. It's a five-minute question at design time
and a much longer one to answer after the fact in production.

Atomic Student Creation Across Microservices: Why I Reached for a Compensating Transaction

Mark Flame — Fri, 10 Jul 2026 15:27:26 +0000

Atomic Student Creation Across Microservices: Why I Reached for a Compensating Transaction

SkillUp Africa, the EdTech platform I'm building, is a NestJS monorepo split
into an auth-service, a school-service, and an API gateway that talks to
both over TCP. Splitting things this way is great for separation of concerns
— until you hit an operation that genuinely needs to touch both services in
one logical unit of work. Creating a student is exactly that operation, and
it's what forced me to actually think about distributed consistency instead
of assuming it for free.

Why this is harder than it looks

In a monolith, creating a student would be one database transaction:
insert the user account, insert the student record, commit. If anything
fails, the whole thing rolls back automatically and you're left with either
nothing or a fully-created student. Postgres/Mongo transactions give you that
guarantee for free within a single database.

The moment you split into services with separate databases, that guarantee
disappears. In SkillUp, creating a student (specifically, a school admin
creating a managed student account) means:

auth-service creates the user record — hashed credentials, role, a server-generated username
school-service creates the student entity — linked to a school, a class, guardian info, etc.

These are two separate databases behind two separate services talking over
TCP via @MessagePattern handlers. There is no single database transaction
that can wrap both. If step 1 succeeds and step 2 fails — school-service is
down, a validation fails, the network hiccups — you're left with a user
account that can log in but has no student record attached to it. That's not
a cosmetic bug. It's an orphaned identity in the system, and depending on
what the gateway does next, it can look to the admin like the creation
"failed" when actually it half-succeeded.

The naive fixes, and why I didn't use them

Two-phase commit (2PC). The classic distributed-transaction answer. I
considered it and ruled it out fast — it requires a transaction coordinator,
both services need to support prepare/commit phases, and it doesn't play
well with service unavailability (a participant going down mid-transaction
can leave things locked). For a two-service, low-throughput operation like
"admin creates a student," that's a lot of infrastructure for the problem
at hand.

Just hope it doesn't fail. Tempting when you're moving fast, genuinely
the wrong call for anything touching account creation. Auth bugs are the
kind that erode trust in a platform fastest.

What I actually built: a compensating transaction (saga pattern, simplified)

The pattern I used is a straightforward saga: perform step one, and if step
two fails, explicitly undo step one. No coordinator, no distributed locks —
just the gateway (or an orchestrating service) being responsible for
cleanup when a later step fails.

// gateway / orchestrating service (simplified)
async createManagedStudent(dto: CreateStudentDto, adminSchoolId: string) {
  // schoolId is resolved from the admin's JWT, never trusted from the request body
  const schoolId = adminSchoolId;

  const user = await this.authClient
    .send('auth.create-managed-user', {
      role: 'student',
      // username is always server-generated — never accepted from the client
      username: generateUsername(dto.firstName, dto.lastName),
    })
    .toPromise();

  try {
    const student = await this.schoolClient
      .send('school.create-student', { ...dto, schoolId, userId: user.id })
      .toPromise();

    return student;
  } catch (err) {
    // school-service failed — compensate by rolling back the user we just created
    await this.authClient
      .send('auth.delete-user', { userId: user.id })
      .toPromise();

    throw new InternalServerErrorException(
      'Student creation failed and was rolled back.',
    );
  }
}

A few details here matter as much as the try/catch structure itself:

schoolId is always resolved from the admin's JWT, never from the request body. If it were client-supplied, any authenticated admin could potentially create a student under a different school just by changing a field in the payload. This isn't strictly part of the compensating transaction pattern, but it lives in the same code path and it's the kind of thing that's easy to get wrong when you're focused on the transaction logic and lose sight of the trust boundary.
The username is server-generated, not client-supplied. This removes an entire class of bugs and abuse (duplicate usernames, impersonation attempts, injection through a username field) at the source, rather than validating it after the fact.
The compensation step (auth.delete-user) has to be idempotent and reliable in its own right. If the delete call itself fails, you now have the orphaned-user problem you were trying to avoid, just one layer deeper. In practice this means the compensation call needs its own retry handling, and worst case, a background reconciliation job that periodically checks for users with no linked student record and flags or cleans them up. I haven't needed the reconciliation job yet at current scale, but I'm not pretending the try/catch alone is a complete guarantee — it's the first line of defense, not the only one.

Why not just make school-service the source of truth and skip auth-service entirely?

I considered this too — collapse the operation into one service so there's
one database and normal ACID transactions apply. It doesn't work here
because auth-service genuinely owns authentication concerns (password
hashing, token issuance, role management) across all user types in the
platform, not just students, and I didn't want to duplicate that logic or
fracture where credentials live. The service boundary is doing real work;
the compensating transaction is the cost of keeping that boundary honest
rather than collapsing it for the sake of one operation's convenience.

The actual lesson

Splitting a system into services doesn't remove the need for transactional
thinking — it just moves the responsibility from the database to your
application code. Postgres and MongoDB give you strong consistency
guarantees for free within their own boundary; the moment an operation spans
two databases, you become the transaction coordinator, whether you
planned to or not. The saga pattern isn't a magic fix, it's an honest
acknowledgment of that: define what "undo" means for every step that can
fail after an earlier step already succeeded, and make sure the undo path
is at least as reliable as the thing it's undoing.

The Mongoose Bug That Taught Me to Stop Trusting findByIdAndUpdate

Mark Flame — Thu, 09 Jul 2026 16:02:04 +0000

The Mongoose Bug That Taught Me to Stop Trusting findByIdAndUpdate

I want to talk about a bug that cost me a few hours, because I think the
reason it happened is more useful than the fix itself. It's the kind of bug
that doesn't throw an error, doesn't crash anything, and just quietly does
the wrong thing — which is the worst kind.

The setup

I was working on the Result module for KDC, a school management system I'm
building. Part of the flow involves updating a student's result document —
recalculating a final score, and by extension, a grade — whenever a score
field changes. I had a Mongoose pre('save') hook on the Result model to
handle exactly that:

resultSchema.pre('save', function (next) {
  this.finalScore = (this.cbtScore + this.writtenScore) / 2;
  this.grade = calculateGrade(this.finalScore);
  next();
});

Straightforward. Whenever a result document gets saved, recompute the score
and grade from whatever the current values are. I tested it against a
document I created and saved directly, and it worked exactly as expected.

Then I wired up the actual update endpoint using the pattern I'd used
everywhere else in the codebase:

const updated = await Result.findByIdAndUpdate(
  resultId,
  { writtenScore: newScore },
  { new: true }
);

And the grade just... didn't update. finalScore stayed stale. No error, no
warning, nothing in the logs. The document saved fine, the writtenScore
field updated fine — just the derived fields silently didn't.

Why this happens

Here's the thing I didn't fully internalize until I hit this: findByIdAndUpdate
(and findOneAndUpdate) talk to MongoDB directly through the driver's update
operators. They don't load a document into memory, mutate it, and call
.save() on it. Which means Mongoose's pre('save') and post('save')
middleware — the hooks tied specifically to the save lifecycle — never
fire. There's a separate pre('findOneAndUpdate') hook if you want middleware
on that path, but it's a genuinely different hook, and if you only defined
pre('save') (like I had), it's simply skipped.

This isn't a Mongoose bug. It's Mongoose behaving exactly as documented. The
bug was mine — I'd built a mental model where "updating a document" always
meant "the save hooks will run," and that assumption was wrong for one of the
two most common ways to update a document in Mongoose.

The fix: fetch, mutate, save

The reliable fix, when you need save-hook logic to actually run, is to stop
using the shortcut and go back to the explicit pattern:

const result = await Result.findById(resultId);
result.writtenScore = newScore;
await result.save(); // pre('save') hook fires here, as expected

Three lines instead of one, and it costs you an extra round trip to the
database. But it means your derived fields, validation, and any other
save-hook logic actually run every time, instead of only running for the
code paths that happen to call .save() directly.

What I actually changed after this

It wasn't just this one endpoint. I went back through the codebase and
audited every place I was using findByIdAndUpdate or findOneAndUpdate on
a model that had pre('save') middleware, because the bug is silent by
nature — it doesn't announce itself, it just produces stale derived data that
looks fine until someone notices the numbers are wrong.

The rule I settled on: if a schema has meaningful pre('save') logic
(derived fields, validation that depends on multiple fields, anything beyond
basic schema-level validation), don't reach for findByIdAndUpdate on it by
default. Use the fetch-mutate-save pattern, and reserve the findAndUpdate
shortcuts for simple field updates where nothing downstream depends on
save-hook side effects.

The actual lesson

The Mongoose docs cover this distinction clearly — it wasn't undocumented
behavior. The real lesson wasn't "read the docs more carefully," though
that's true. It was: when you reach for a shortcut method, know what it
skips, not just what it does. findByIdAndUpdate is genuinely useful and
I still use it constantly — just not on models where I need save-lifecycle
guarantees. The bug taught me to ask that question before writing the update
logic, not after debugging why a grade wasn't recalculating.

"Building Network-Resilient Exam Sessions for a CBT System in Nigeria"

Mark Flame — Thu, 09 Jul 2026 15:55:40 +0000

Building Network-Resilient Exam Sessions for a CBT System in Nigeria

A few months ago I started building KDC, a school management system for an NGO
school here in Abuja. One of the modules is a CBT (computer-based testing)
engine — students sit exams on shared school computers, timed, timetable-gated,
auto-graded.

It sounds like a standard CRUD problem until you remember where it's running.
Nigerian schools deal with unstable power and inconsistent network connections.
A student can be 12 minutes into a 40-minute exam and the connection drops —
NEPA takes the light, the router reboots, whatever. If the exam session just
dies at that point, you've got a genuinely upset student, a teacher who has to
manually intervene, and a system nobody trusts.

So "what happens when the network drops mid-exam" wasn't an edge case I could
shrug off. It had to be a first-class part of the design.

The naive approach (and why it breaks)

The obvious first design is: student starts exam → session created → student
submits answers as they go → session closes on submit or timeout.

The problem is state. If you don't explicitly track where a student is in
an exam — which questions they've seen, what they've answered so far, how
much time is actually left — then a disconnect forces you into one of two bad
options:

Restart the exam. Unacceptable. The student loses their timer, their progress, possibly the trust of the whole system.
Trust the client. Let the frontend hold all the state and just resubmit everything on reconnect. Fragile — browser tabs get closed, local storage gets cleared, and now you have no server-side source of truth for a graded academic exam.

Neither works. The session itself needed to be the source of truth, not the
client.

Designing the ExamSession as a resumable entity

The fix was to make ExamSession a persistent, resumable entity rather than a
transient in-memory thing tied to a socket or a request lifecycle. Every
session tracks:

which exam and student it belongs to
a server-authoritative start timestamp (so remaining time is always calculated server-side, never trusted from the client)
current status (in-progress, submitted, expired)
answers submitted so far, keyed by question ID

When a student reconnects — new tab, refreshed page, different device on the
same network — the client doesn't start a new session. It asks the server:
"does this student have an active session for this exam?" If yes, the
server returns the existing session state: time remaining (calculated from
the stored start time, not a client clock), and everything already answered.
The frontend just re-renders from that state. From the student's point of
view, it looks like nothing happened.

GET /exam-sessions/active?examId=xxxx

// server logic (simplified)
const existingSession = await ExamSession.findOne({
  student: studentId,
  exam: examId,
  status: 'in-progress'
});

if (existingSession) {
  const elapsed = Date.now() - existingSession.startedAt;
  const remaining = exam.durationMs - elapsed;

  if (remaining <= 0) {
    existingSession.status = 'expired';
    await existingSession.save();
    return autoSubmit(existingSession);
  }

  return res.json({ session: existingSession, remainingMs: remaining });
}

The key design decision here: time is never trusted from the client.
Whatever the frontend timer shows is just a UI convenience. The moment a
session is fetched or an answer is submitted, the server recalculates
remaining time from the stored startedAt. That's what makes a disconnect
safe — the clock doesn't stop just because the connection did.

Batch-processing answers with a Map for O(1) lookup

The other piece of this was answer submission. Early on I was updating one
answer at a time — student answers a question, client fires a request,
server updates that one field in the session document. That's a lot of
round trips for a 40-question exam, and it's exactly the kind of pattern
that falls apart on a flaky connection: one failed request out of forty
and now the session state is silently incomplete.

I switched to batching. The client periodically syncs the entire current
answer set, and on the server I process it against a Map keyed by question
ID rather than looping through an array with .find() on every question
(which would be O(n) per lookup, O(n²) for a full batch):

const existingAnswers = new Map(
  session.answers.map(a => [a.questionId.toString(), a])
);

for (const incoming of submittedAnswers) {
  existingAnswers.set(incoming.questionId, {
    questionId: incoming.questionId,
    selectedOption: incoming.selectedOption,
    answeredAt: new Date(),
  });
}

session.answers = Array.from(existingAnswers.values());
await session.save();

This does two things: it makes each sync idempotent (resubmitting the same
batch after a dropped connection doesn't create duplicates or corrupt state),
and it keeps the update fast regardless of how many questions are in the
exam.

What I'd tell someone building this from scratch

Server time, always. Any timed, gradeable system where you let the client be the source of truth for time is a system you don't actually control.
Idempotency isn't optional on flaky networks. If a retry can happen (and on unreliable connections, it will), your write operations need to be safe to repeat.
Design for the network you'll actually be deployed on, not the one on your dev machine. A lot of backend tutorials assume stable, low-latency connections. That assumption doesn't hold for a lot of the world, including where I'm building this.

KDC is still in active development — the exam module is done, results and
analytics are next. If you're building anything for an environment with
unreliable infrastructure, I'd genuinely rather over-engineer for
disconnects up front than field angry calls from a school on exam day.