DEV Community: Emmanuel Igunnu

Your RabbitMQ Consumer Reconnected. The In-Flight Handler Still Crashed.

Emmanuel Igunnu — Wed, 22 Jul 2026 12:30:06 +0000

In my previous article, I wrote about how a RabbitMQ upgrade exposed assumptions hidden inside our messaging system.

One failure mode deserved a deeper explanation because it reveals an important distinction:

Recovering a consumer does not necessarily recover the work that consumer was already processing.

Our reconnection logic could establish a new connection, create a new channel, redeclare the queue, reapply prefetch, and resume consumption. From the outside, the system looked healthy again. But an older message handler could still be running with a reference to the channel that had just closed. When that handler finished and attempted to acknowledge its message, it failed with:

IllegalOperationError: Channel closed

The consumer had recovered. The in-flight handler had not.

The Failure Timeline

Consider a consumer processing a transfer:

channel.consume("process_transfer", async (msg) => {
  if (!msg) return;

  try {
    await processTransfer(msg);
    channel.ack(msg);
  } catch (error) {
    channel.nack(msg, false, false);
  }
});

Under stable conditions, the flow is straightforward:

RabbitMQ delivers a message.
The handler processes the transfer.
The consumer acknowledges the message.
RabbitMQ removes it from the queue.

Now introduce a broker restart:

RabbitMQ delivers message
        ↓
Handler begins processing
        ↓
RabbitMQ connection closes
        ↓
Consumer starts reconnecting
        ↓
A new connection and channel are created
        ↓
Old handler finishes processing
        ↓
Old handler calls channel.ack(msg)
        ↓
IllegalOperationError: Channel closed

The important detail is that reconnecting does not replace the channel captured by the existing handler. That handler belongs to the lifetime of the old channel.

Recovery Has Two Timelines

It helps to think of recovery as two separate timelines.

The future timeline

This is the part most reconnect implementations handle:

Detect the closed connection.
Open a new connection.
Create a new channel.
Redeclare exchanges and queues.
Reapply prefetch.
Register the consumer again.
Continue receiving messages.

This protects future work.

The current timeline

This contains work that was already running when the connection disappeared:

Database writes may still be executing.
External API requests may still be in progress.
A long-running handler may still complete.
Business logic may succeed after its RabbitMQ channel has died.
The final acknowledgement may target invalid infrastructure.

Reconnection does not automatically resolve any of these cases. A reliable consumer must handle both timelines.

A Delivery Belongs to Its Original Channel

An AMQP channel is not merely a convenient object through which we call ack.

It is part of the identity of the delivery.

RabbitMQ delivery tags are scoped to a channel. A message delivered on one channel must be acknowledged on that same channel. Attempting to acknowledge it through the replacement channel can cause an unknown delivery tag channel error.

This means the following approach is unsafe:

class Consumer {
  private channel: Channel;

  async handle(msg: ConsumeMessage) {
    await processMessage(msg);

    // This may now point to a replacement channel.
    this.channel.ack(msg);
  }
}

If this.channel changes during reconnection, the handler may use the new channel to acknowledge a delivery received through the old one.

The new channel is healthy, but it does not own that delivery.

RabbitMQ’s acknowledgement documentation explicitly requires deliveries to be acknowledged on the channel on which they were received.

A reconnect implementation therefore needs to preserve the relationship between:

The message
The handler
The channel that delivered the message
The lifetime of that channel

Acknowledgements Need Defensive Handling

With amqplib, ack, nack, and reject do not return promises. They send protocol commands without waiting for a response.

This code does not provide useful protection:

await channel.ack(msg);

There is nothing meaningful to await.

If the library already knows that the channel is closed, the operation can throw synchronously. That means settlement operations should be guarded with try/catch.

Here is a simplified pattern:

import { Channel, ConsumeMessage } from "amqplib";

function safelySettle(
  operation: () => void,
  context: {
    action: "ack" | "nack";
    queue: string;
    messageId?: string;
    channelClosed: boolean;
  },
): boolean {
  if (context.channelClosed) {
    console.warn("Skipping message settlement on a closed channel", context);
    return false;
  }

  try {
    operation();
    return true;
  } catch (error) {
    console.error("Failed to settle RabbitMQ delivery", {
      ...context,
      error,
    });

    return false;
  }
}

The consumer can capture the channel that delivered the message and track its state:

async function startConsumer(channel: Channel) {
  let channelClosed = false;

  channel.on("close", () => {
    channelClosed = true;
  });

  channel.on("error", (error) => {
    console.error("RabbitMQ channel error", { error });
  });

  await channel.prefetch(1);

  await channel.consume(
    "process_transfer",
    async (msg: ConsumeMessage | null) => {
      if (!msg) return;

      const messageId = msg.properties.messageId;

      try {
        await processTransfer(msg.content);

        safelySettle(() => channel.ack(msg), {
          action: "ack",
          queue: "process_transfer",
          messageId,
          channelClosed,
        });
      } catch (error) {
        console.error("Transfer processing failed", {
          messageId,
          error,
        });

        safelySettle(() => channel.nack(msg, false, false), {
          action: "nack",
          queue: "process_transfer",
          messageId,
          channelClosed,
        });
      }
    },
    {
      noAck: false,
    },
  );
}

This does not magically acknowledge a message after its channel has closed. That is no longer possible. It prevents a known infrastructure failure from becoming an unhandled application crash and makes the resulting message state visible.

The amqplib documentation also notes that a failed or closed channel is invalidated and that subsequent operations can throw. It recommends handling connection and channel events explicitly. See the amqplib channel API for the underlying behavior.

Skipping the Acknowledgement Is Not Message Loss

It can feel uncomfortable to skip an acknowledgement after business logic completes. However, once the channel has closed, the consumer no longer controls that delivery through the normal acknowledgement path. With manual acknowledgements, RabbitMQ automatically requeues unacknowledged deliveries when their channel or connection closes. The message can then be delivered to the same consumer or another consumer after recovery.

RabbitMQ documents this as part of its automatic requeueing behavior.

This protects the message, but it introduces another problem:

The business operation may already have succeeded.

For example:

The consumer receives transfer-123.
It commits the transfer to the database.
The RabbitMQ connection closes.
The acknowledgement never reaches the broker.
RabbitMQ requeues the message.
Another consumer receives transfer-123.
The transfer is processed again.

The messaging system has behaved correctly. The message was never acknowledged, so RabbitMQ delivered it again.

The duplicate side effect is now the application’s responsibility.

Reconnection Makes Idempotency Non-Optional

RabbitMQ acknowledgements provide at-least-once delivery, not exactly-once business execution. A consumer must assume that any message can be delivered more than once, especially around connection loss. For transaction-sensitive workflows, this usually means giving every operation a stable idempotency key:

async function processTransfer(message: TransferMessage) {
  const existingTransfer = await transferRepository.findByReference(
    message.transferReference,
  );

  if (existingTransfer) {
    return existingTransfer;
  }

  return transferRepository.create({
    reference: message.transferReference,
    amount: message.amount,
    destination: message.destination,
  });
}

A database uniqueness constraint should enforce this guarantee:

CREATE UNIQUE INDEX transfers_reference_unique
ON transfers(reference);

The application-level check is useful, but the database constraint is the real protection against concurrent duplicate deliveries.

For more complicated workflows, the consumer may need to:

Store processed message identifiers.
Make state transitions conditional.
Use database transactions.
Pass idempotency keys to external services.
Record external-operation results before acknowledging.
Design handlers so repeating them produces the same final state.

RabbitMQ sets a redelivery flag when a message is delivered again. That is useful for logging and diagnosis, but it should not be the only duplicate-protection mechanism.

A message can be duplicated in more than one way, and the consumer should be safe regardless of the flag.

RabbitMQ’s reliability guide recommends designing consumers to be idempotent because redelivery is an expected part of connection recovery.

Catching `ack` Errors Is Still Not a Delivery Guarantee

There is a subtle limitation to the defensive wrapper.

If channel.ack(msg) does not throw, that only means the client accepted the operation locally. Consumer acknowledgements do not receive a corresponding promise or confirmation from amqplib. The network could fail immediately after the acknowledgement is written locally but before RabbitMQ processes it.

Therefore:

try {
  channel.ack(msg);
} catch (error) {
  // Handle a channel already known to be unusable.
}

protects the process from a known closed-channel error.

It does not prove that the broker received the acknowledgement.

That ambiguity is another reason idempotency must be part of the business workflow rather than an error-handling afterthought.

What Not to Do

Do not acknowledge through the replacement channel

The new channel does not own the old delivery. Delivery tags are scoped to their original channel.

Do not retry `ack` indefinitely

If the original channel is closed, repeatedly calling ack cannot restore it.

Do not silently swallow the error

Avoiding a crash is useful, but losing all evidence of the event makes the system difficult to operate. Record the queue, message identifier, delivery tag, channel generation, and settlement action.

Do not treat reconnection as successful recovery by itself

A “connected” metric says nothing about handlers that were running during the outage.

Do not assume a successful handler ran only once

The side effect may complete even when its acknowledgement does not.

The Test That Reveals the Problem

This failure mode is straightforward to test, but ordinary integration tests rarely create the required timing.

A useful test should:

Publish a message with a known identifier.
Start processing it with a deliberately slow handler.
Allow the business side effect to complete.
Restart RabbitMQ before the handler acknowledges.
Allow the handler to reach its acknowledgement code.
Wait for the consumer to reconnect.
Observe the message being redelivered.
Verify that the process remains alive.
Verify that the business side effect exists only once.
Verify that the consumer continues processing new messages.

The assertions matter more than whether the reconnect log appears.

A successful test should prove that:

The old handler cannot crash the process.
The closed channel is never reused.
The old delivery is not acknowledged through the new channel.
The message can be redelivered safely.
Duplicate execution does not duplicate the business result.
The recovered consumer continues processing.

This test exercises the boundary between messaging reliability and business reliability.

Observability Must Cover Both Timelines

Connection metrics are necessary, but they are incomplete.

For this failure mode, useful logs include:

queue
messageId
deliveryTag
redelivered
channelGeneration
handlerStartedAt
handlerCompletedAt
channelClosedAt
settlementAction
settlementOutcome

Useful metrics include:

rabbitmq_consumer_reconnect_total
rabbitmq_delivery_redelivered_total
rabbitmq_settlement_skipped_total
rabbitmq_settlement_error_total
consumer_handler_duration
consumer_duplicate_operation_total

A reconnect event tells you the future timeline recovered.

A skipped acknowledgement, redelivered message, or duplicate-operation event tells you what happened to the current timeline.

You need both views to understand the incident.

Final Takeaway

A reconnecting RabbitMQ consumer is better than one that stays disconnected, but reconnection is only half of recovery.

The harder problem is work that crosses the failure boundary:

The message arrived through the old channel.
The handler continued after that channel closed.
The business operation may have completed.
The acknowledgement may not have reached RabbitMQ.
The message may therefore be delivered again.

A reliable consumer must associate every delivery with the channel that owns it, handle settlement operations defensively, expose uncertain outcomes through logs and metrics, and make business processing idempotent.

The goal is not to pretend the failure never happened. The goal is to ensure that when infrastructure dies halfway through the work, the application fails predictably, recovers safely, and produces the correct business result even if the message returns.

That is the difference between a consumer that reconnects and a messaging system that actually recovers.

What Upgrading RabbitMQ Broker Taught Me About Building Reliable Messaging Systems

Emmanuel Igunnu — Sat, 11 Jul 2026 12:40:11 +0000

The RabbitMQ upgrade looked like a straightforward infrastructure task: move from RabbitMQ 3.X to 4.X, provision the new broker, review the client setup, confirm queues still declare correctly, restart consumers, watch the logs, and move on.

But infrastructure upgrades rarely test only infrastructure. They also test the assumptions your application has been making for years. In this case, the upgrade forced a more important question: is our messaging system reliable by design, or has it simply been relying on stable conditions?

That distinction matters because a message queue can appear healthy when the broker is running, the network is stable, consumers are alive, and messages are acknowledged quickly. But production systems are not judged only by how they behave when everything is fine. They are judged by how they behave during restarts, closed channels, slow handlers, bad configuration, deployment windows, and partial failure.

The RabbitMQ upgrade exposed those edges. It revealed assumptions around connection lifecycle, acknowledgements, dead-letter routing, retry behavior, observability, and operational simplicity. The real lesson was not just how to upgrade RabbitMQ. The real lesson was how to build a messaging layer that is easier to operate, easier to reason about, and safer to fail.

Simplicity Is an Operational Feature

One of the first things the upgrade exposed was complexity. Over time, messaging code can quietly become a small internal framework. A connection helper becomes a connection manager. A consumer wrapper becomes a consumer framework. Retry helpers appear, dead-letter helpers appear, failure handlers appear, and monitoring logic gets layered on top.

Each addition may have been reasonable when introduced, but during an incident, complexity has a cost. Every abstraction becomes another place to inspect. Every helper becomes another assumption to validate. Every unused file becomes a possible source of false confidence.

RabbitMQ integration code does not need to be clever. It needs to be understandable under pressure. The better design is usually boring: one clear connection manager, one reusable consumer abstraction, one predictable queue declaration path, one dead-letter queue convention, explicit retry behavior, minimal hidden magic, and no unused reliability code.

This is not about writing less code for its own sake. It is about reducing operational surface area. When production is already under pressure, the questions should be easy to answer: is the app connected, is the channel open, is the consumer active, are messages being acknowledged, are failed messages going to the expected DLQ, and is the alert telling us the real problem?

Simplicity is not just a code-quality preference. In infrastructure-facing code, simplicity is an operational feature.

Queue Declarations Are Infrastructure Contracts

Queue declaration logic deserves more respect than it often gets. It is easy to treat queue declarations as ordinary application setup code, but in RabbitMQ, a declaration is a contract with the broker.

If a queue already exists with one set of arguments and an application later tries to declare it with different arguments, RabbitMQ can reject the declaration and close the channel. That means queue declarations are not harmless. They define durable infrastructure expectations.

Every queue should have clear answers to a few basic questions. Who declares this queue? Is it durable? Does it have a dead-letter queue? What happens when a message is rejected? Are the arguments consistent across all services?

This is where many messaging systems become fragile. Not because RabbitMQ is unreliable, but because different parts of the application slowly develop different ideas of what the same queue should be.

The lesson here is that queue declarations should be treated like schema design or API contracts. They need consistency, ownership, and migration thinking. Changing a queue argument is not just a code change; it may also be an infrastructure migration.

Dead-Letter Queues Should Be Boring

Dead-letter queues are not the place for creativity. When a message fails, the path should be obvious.

A queue named:

process_transfer

can dead-letter into:

process_transfer.dlq

That naming convention is not impressive, and that is the point. The value of a DLQ naming convention is not elegance. It is predictability.

When production is failing, an engineer should not have to search through configuration files to know where a failed message went. The queue name should tell them. The exact convention may differ across systems, but the principle should not: failed messages should have a predictable destination.

That predictability improves debugging, incident response, replay workflows, and operational confidence.

Reconnection Is Required, But It Is Not Enough

RabbitMQ can close connections for valid reasons. A broker restart, upgrade, failover, deployment, maintenance event, or admin action can produce something like:

CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'

This should not surprise the application. A production consumer should assume the broker may disappear and come back. That means the consumer should recover when the initial connection fails, when the connection closes, when the channel closes, when the broker restarts, or when the network temporarily fails.

The consumer should reconnect, recreate the channel, redeclare what it needs, reapply prefetch, and start consuming again. But reconnection alone does not solve everything.

Reconnection protects future work. It does not automatically protect work already in progress. Consider this sequence: a consumer receives a message, the handler starts processing, RabbitMQ closes the connection, the application begins reconnecting, the original handler finishes, and then the handler tries to acknowledge or reject the message using the old channel.

The application may now hit:

IllegalOperationError: Channel closed

The code may simply be doing:

channel.ack(msg)

or:

channel.nack(msg, false, false)

But the channel reference is stale. This is an important distinction because you can have reconnect logic and still crash when an in-flight handler tries to use a closed channel.

That is why acknowledgement operations need defensive handling. If the channel is already closed, the application should log the problem clearly and avoid crashing. This does not mean silently ignoring message state. It means recognizing that once the channel is closed, the application no longer controls that delivery in the normal way. RabbitMQ will decide what happens based on connection loss, acknowledgement state, and queue configuration.

The deeper lesson is that recovery has two timelines. There is the future timeline, where the consumer reconnects and continues. Then there is the current timeline, where existing handlers may still be completing work against dead infrastructure. A reliable consumer has to handle both.

Prefetch Is a Risk Decision, Not Just a Throughput Setting

RabbitMQ prefetch is often discussed as a performance tuning option, but it is more than that. Prefetch controls how many unacknowledged messages a consumer can hold at once, which affects throughput, fairness, failure recovery, duplicate risk, external pressure, and operational behavior during slowdowns.

A conservative default like this is often safer:

prefetch = 1

That means a consumer processes one unacknowledged message at a time. This may be the right default for workflows involving payments, wallet updates, settlements, transfers, inventory mutations, external API calls, and sensitive state transitions.

The trade-off is clear. Lower prefetch may reduce throughput, but it also reduces the number of in-flight messages exposed to failure at any point in time. Higher prefetch can be appropriate for workloads that are idempotent, independent, fast, and safe to parallelize, but that should be an explicit choice.

The better question is not only, “How do we process more messages?” The better question is: what level of concurrency can this workflow safely tolerate? That question turns prefetch from a tuning knob into an engineering decision.

Observability Should Reduce Time to Diagnosis

A vague alert is only slightly better than no alert. An alert like this tells you almost nothing:

Failed to connect to RabbitMQ

It says a failure happened, but not why. During an incident, the first question is always: what failed, and what is the most likely cause?

A better alert includes the actual reason immediately:

Message: `Failed to connect to RabbitMQ: ${err.message}`

This matters because many alerting tools show only the first line or preview text. If the useful error is hidden in expanded fields or logs, the alert slows people down.

Good alerts should make it easier to distinguish between bad credentials, wrong host, DNS failure, connection timeout, TLS issues, broker restarts, missing environment variables, wrong protocol, wrong port, and network reachability issues. The best alert is not the one that simply says something broke. The best alert gives the responder a useful starting point.

The same principle applies to monitoring. Monitoring should reduce uncertainty, not multiply symptoms. If RabbitMQ is unreachable, monitoring should not trigger several duplicate alerts for the same root cause. Observability should improve signal; it should not make an already noisy incident louder.

Reliability Code That Does Not Run Is Not Reliability

Unused code is bad, but unused reliability code is worse. A file named something like this looks reassuring:

consumer-failure.handler.ts

It may include logic to read retry counts, republish messages, route failed messages to a DLQ, attach failure metadata, and decide whether a message should be retried. But if the active consumer does not call it, it is not part of the system’s reliability story. It is just code.

Worse, it can mislead future engineers. Someone may see the file and assume failures are centrally handled. They may trust retry behavior that does not actually run. They may believe the system has a safety net that is not connected to anything.

That is dangerous. Reliability code should be held to a higher standard than ordinary code. If it is not active, tested, observable, and part of the execution path, it should either be wired in properly or removed.

Reliability that only exists in the repository does not exist in production.

Final Takeaway

Upgrading RabbitMQ from 3.X to 4.X was not just a broker upgrade. It was a reliability review.

It exposed the difference between a messaging system that works and a messaging system that can be operated confidently. The biggest lessons were to keep the messaging layer simple, treat queue declarations as infrastructure contracts, make DLQ routing predictable, assume connections and channels will close, reconnect consumers automatically, protect in-flight acknowledgement paths, choose prefetch based on risk, make alerts useful from the first line, and remove unused reliability code.

The best messaging setup is not the most abstract one. It is not the one with the most helper classes, and it is not the one that looks the most sophisticated in the codebase.

The best setup is the one that keeps consuming, fails clearly, recovers predictably, and remains simple enough for engineers to understand when production is already under pressure.

That is the real standard for reliable messaging systems.

DEV Community: Emmanuel Igunnu

Your RabbitMQ Consumer Reconnected. The In-Flight Handler Still Crashed.

The Failure Timeline

Recovery Has Two Timelines

The future timeline

The current timeline

A Delivery Belongs to Its Original Channel

Acknowledgements Need Defensive Handling

Skipping the Acknowledgement Is Not Message Loss

Reconnection Makes Idempotency Non-Optional

Catching ack Errors Is Still Not a Delivery Guarantee

What Not to Do

Do not acknowledge through the replacement channel

Do not retry ack indefinitely

Do not silently swallow the error

Do not treat reconnection as successful recovery by itself

Do not assume a successful handler ran only once

The Test That Reveals the Problem

Observability Must Cover Both Timelines

Final Takeaway

What Upgrading RabbitMQ Broker Taught Me About Building Reliable Messaging Systems

Simplicity Is an Operational Feature

Queue Declarations Are Infrastructure Contracts

Dead-Letter Queues Should Be Boring

Reconnection Is Required, But It Is Not Enough

Prefetch Is a Risk Decision, Not Just a Throughput Setting

Observability Should Reduce Time to Diagnosis

Reliability Code That Does Not Run Is Not Reliability

Final Takeaway

Catching `ack` Errors Is Still Not a Delivery Guarantee

Do not retry `ack` indefinitely