DEV Community: Taras H

Background Jobs in Production: The Problems Queues Don’t Solve

Taras H — Sun, 08 Mar 2026 11:17:45 +0000

Moving work out of the request path is one of the most common ways to
speed up backend systems.

Emails are sent asynchronously.
Invoices are generated by workers.
Webhooks are delivered through queues.
Image processing and indexing run in background jobs.

Latency improves immediately.

But many teams eventually notice something strange in production:

duplicate emails appear
retries increase system load
dead-letter queues slowly grow
workflows technically "succeed"... but the outcome is wrong

The queue is healthy.
Workers are running.

Yet the system behaves incorrectly.

Moving work to the background changes where failures happen.
It does not remove them.

This post is a shorter version of a deeper engineering write-up
originally published on CodeNotes.

The Assumption Behind Background Jobs

Background job systems are usually introduced with a simple expectation:

If a job fails, the queue will retry it until it succeeds.

Queues also provide useful features:

buffering traffic spikes
independent worker scaling
retry handling
isolation from request latency

Because of this, async processing often feels safer than synchronous
execution.

But that assumption depends on something rarely guaranteed in
production:

that running a job multiple times produces the same result as running
it once.

What "At-Least-Once Delivery" Actually Means

Most queue systems guarantee at-least-once delivery.

That means the system will try hard to deliver a message - even if it
results in duplicate execution.

It does not mean:

the job runs exactly once
side effects happen exactly once
messages are processed in order

In other words, the queue protects against message loss, not
duplicate work.

Once duplicate execution becomes possible, correctness has to come from
somewhere else.

Usually that means:

idempotent handlers
deduplication keys
explicit state transitions
retry boundaries

Without those protections, the infrastructure is reliable while the
workflow is not.

A Classic Failure Scenario

Consider a worker that sends a payment receipt:

await emailClient.send(...)

await db.payment.update({
  receiptSentAt: new Date()
})

If the worker crashes after sending the email but before updating
the database, the job will be retried.

Now the customer receives two receipts.

The queue behaved exactly as designed.

But the business outcome is incorrect.

Why Production Systems Break Here

Background job systems introduce two things that make correctness
harder.

1. Duplicate execution

Workers can crash after performing side effects but before acknowledging
the message.

2. Time separation

Jobs may execute minutes or hours after they were created, when system
state has already changed.

Because of this, retries often interact with partial state or
outdated context.

The Design Rule Most Teams Learn Later

A background job should never be treated as a one-time action.

It should be treated as a replayable command.

Every handler should be safe if it runs:

twice
later than expected
after partial completion
out of order

If those conditions break the workflow, retries will eventually corrupt
system behavior.

The Monitoring Trap

Teams often monitor queue infrastructure:

queue depth
worker throughput
retry counts
dead-letter volume

Those metrics matter - but they don't answer questions like:

Did users receive duplicate emails?
Did a payment create multiple ledger entries?
Did downstream systems receive conflicting updates?

A queue dashboard can look completely healthy while the workflow is
incorrect.

Read the Full Production Breakdown

This post only covers the core failure patterns.

The full article explains:

why retries can make outages worse
how idempotent background jobs are designed
why dead-letter queues silently grow
what production teams monitor beyond queue depth
a practical rollout checklist for new background jobs

👉 Full article:
https://codenotes.tech/blog/background-jobs-in-production

Why AI Code Review Comments Look Right but Miss Real Risks

Taras H — Fri, 27 Feb 2026 16:59:54 +0000

Many teams have added AI code review to their pull request workflow.

The promise is obvious: faster feedback, broader coverage, fewer review bottlenecks. AI scans every diff, flags suspicious code, suggests test cases, and highlights style issues in seconds.

Pull requests move faster. Review queues shrink. Everything looks healthier.

But production incidents don’t disappear.

So the practical question emerges:

If AI reviews every PR, why are high-risk issues still reaching production?

The Reasonable Assumption

It’s natural to assume:

More review coverage + faster feedback = better quality.

AI increases comment volume. It catches missing null checks. It suggests cleaner error handling. It improves surface-level consistency.

At a process level, things look better.

But review activity is not the same thing as risk reduction.

Where the Gap Appears

Most AI code review tools are excellent at:

Pattern matching
Local correctness
Code explanation
Generic best practices

They are much weaker at:

Business logic validation
Authorization boundaries
Implicit architectural constraints
Production failure modes

For example:

export async function updateUserRole(userId: string, role: string) {
  const user = await db.user.findUnique({ where: { id: userId } });

  if (!user) {
    throw new Error('User not found');
  }

  await db.user.update({ where: { id: userId }, data: { role } });
}

An AI reviewer might suggest stronger validation or clearer error handling.

But the real production risk may be completely different:

Who is allowed to change roles?
Is there audit logging?
Does this break cross-service assumptions?
What happens under concurrent updates?

These risks don’t live in the diff. They live in the system.

Why AI Feels More Effective Than It Is

Three patterns show up repeatedly:

1. Plausible Comments Create Confidence

LLMs generate comments that sound correct. That increases perceived rigor — even when the risk profile hasn’t changed.

2. Diffs Hide System Context

Pull requests rarely include architectural history, compliance constraints, or production incident lessons. Humans often carry this context implicitly. AI usually doesn’t.

3. Automation Changes Human Behavior

When AI has already “reviewed” the code, humans subtly shift from critical analysis to verification mode.

The question changes from:

“What could fail in production?”

to:

“Did we resolve the AI comments?”

That shift matters.

The Key Insight

AI expands coverage.

Humans must still own judgment.

AI is strong at local correctness. Production failures usually emerge from system interactions: retries under load, cache drift, authorization boundaries, cross-service contracts.

If the review process optimizes for comment resolution instead of failure thinking, speed improves — but risk stays constant.

If You’re Using AI Review

A useful mental model:

Let AI handle first-pass mechanical checks.
Explicitly reserve human review for system-level risk.
Measure escaped defects — not comment counts.

The real question isn’t whether AI comments are helpful.

It’s whether your review process still forces engineers to think about how systems fail in production.

If this topic resonates, the full breakdown goes deeper into why this happens and how teams misinterpret review signal vs. real risk:

👉 Full article:
https://codenotes.tech/blog/why-ai-code-review-comments-look-right-but-miss-real-risks