Your RabbitMQ Consumer Reconnected. The In-Flight Handler Still Crashed.

In my previous article, I wrote about how a RabbitMQ upgrade exposed assumptions hidden inside our messaging system.

One failure mode deserved a deeper explanation because it reveals an important distinction:

Recovering a consumer does not necessarily recover the work that consumer was already processing.

Our reconnection logic could establish a new connection, create a new channel, redeclare the queue, reapply prefetch, and resume consumption. From the outside, the system looked healthy again. But an older message handler could still be running with a reference to the channel that had just closed. When that handler finished and attempted to acknowledge its message, it failed with:

IllegalOperationError: Channel closed

The consumer had recovered. The in-flight handler had not.

---

The Failure Timeline

Consider a consumer processing a transfer:

channel.consume("process_transfer", async (msg) => {
  if (!msg) return;

  try {
    await processTransfer(msg);
    channel.ack(msg);
  } catch (error) {
    channel.nack(msg, false, false);
  }
});

Under stable conditions, the flow is straightforward:

1. RabbitMQ delivers a message.
2. The handler processes the transfer.
3. The consumer acknowledges the message.
4. RabbitMQ removes it from the queue.

Now introduce a broker restart:

RabbitMQ delivers message
        ↓
Handler begins processing
        ↓
RabbitMQ connection closes
        ↓
Consumer starts reconnecting
        ↓
A new connection and channel are created
        ↓
Old handler finishes processing
        ↓
Old handler calls channel.ack(msg)
        ↓
IllegalOperationError: Channel closed

The important detail is that reconnecting does not replace the channel captured by the existing handler. That handler belongs to the lifetime of the old channel.

---

Recovery Has Two Timelines

It helps to think of recovery as two separate timelines.

The future timeline

This is the part most reconnect implementations handle:

• Detect the closed connection.
• Open a new connection.
• Create a new channel.
• Redeclare exchanges and queues.
• Reapply prefetch.
• Register the consumer again.
• Continue receiving messages.

This protects future work.

The current timeline

This contains work that was already running when the connection disappeared:

• Database writes may still be executing.
• External API requests may still be in progress.
• A long-running handler may still complete.
• Business logic may succeed after its RabbitMQ channel has died.
• The final acknowledgement may target invalid infrastructure.

Reconnection does not automatically resolve any of these cases. A reliable consumer must handle both timelines.

---

A Delivery Belongs to Its Original Channel

An AMQP channel is not merely a convenient object through which we call ack.

It is part of the identity of the delivery.

RabbitMQ delivery tags are scoped to a channel. A message delivered on one channel must be acknowledged on that same channel. Attempting to acknowledge it through the replacement channel can cause an unknown delivery tag channel error.

This means the following approach is unsafe:

class Consumer {
  private channel: Channel;

  async handle(msg: ConsumeMessage) {
    await processMessage(msg);

    // This may now point to a replacement channel.
    this.channel.ack(msg);
  }
}

If this.channel changes during reconnection, the handler may use the new channel to acknowledge a delivery received through the old one.

The new channel is healthy, but it does not own that delivery.

RabbitMQ’s acknowledgement documentation explicitly requires deliveries to be acknowledged on the channel on which they were received.

A reconnect implementation therefore needs to preserve the relationship between:

• The message
• The handler
• The channel that delivered the message
• The lifetime of that channel

---

Acknowledgements Need Defensive Handling

With amqplib, ack, nack, and reject do not return promises. They send protocol commands without waiting for a response.

This code does not provide useful protection:

await channel.ack(msg);

There is nothing meaningful to await.

If the library already knows that the channel is closed, the operation can throw synchronously. That means settlement operations should be guarded with try/catch.

Here is a simplified pattern:

import { Channel, ConsumeMessage } from "amqplib";

function safelySettle(
  operation: () => void,
  context: {
    action: "ack" | "nack";
    queue: string;
    messageId?: string;
    channelClosed: boolean;
  },
): boolean {
  if (context.channelClosed) {
    console.warn("Skipping message settlement on a closed channel", context);
    return false;
  }

  try {
    operation();
    return true;
  } catch (error) {
    console.error("Failed to settle RabbitMQ delivery", {
      ...context,
      error,
    });

    return false;
  }
}

The consumer can capture the channel that delivered the message and track its state:

async function startConsumer(channel: Channel) {
  let channelClosed = false;

  channel.on("close", () => {
    channelClosed = true;
  });

  channel.on("error", (error) => {
    console.error("RabbitMQ channel error", { error });
  });

  await channel.prefetch(1);

  await channel.consume(
    "process_transfer",
    async (msg: ConsumeMessage | null) => {
      if (!msg) return;

      const messageId = msg.properties.messageId;

      try {
        await processTransfer(msg.content);

        safelySettle(() => channel.ack(msg), {
          action: "ack",
          queue: "process_transfer",
          messageId,
          channelClosed,
        });
      } catch (error) {
        console.error("Transfer processing failed", {
          messageId,
          error,
        });

        safelySettle(() => channel.nack(msg, false, false), {
          action: "nack",
          queue: "process_transfer",
          messageId,
          channelClosed,
        });
      }
    },
    {
      noAck: false,
    },
  );
}

This does not magically acknowledge a message after its channel has closed. That is no longer possible. It prevents a known infrastructure failure from becoming an unhandled application crash and makes the resulting message state visible.

The amqplib documentation also notes that a failed or closed channel is invalidated and that subsequent operations can throw. It recommends handling connection and channel events explicitly. See the amqplib channel API for the underlying behavior.

---

Skipping the Acknowledgement Is Not Message Loss

It can feel uncomfortable to skip an acknowledgement after business logic completes. However, once the channel has closed, the consumer no longer controls that delivery through the normal acknowledgement path. With manual acknowledgements, RabbitMQ automatically requeues unacknowledged deliveries when their channel or connection closes. The message can then be delivered to the same consumer or another consumer after recovery.

RabbitMQ documents this as part of its automatic requeueing behavior.

This protects the message, but it introduces another problem:

The business operation may already have succeeded.

For example:

1. The consumer receives transfer-123.
2. It commits the transfer to the database.
3. The RabbitMQ connection closes.
4. The acknowledgement never reaches the broker.
5. RabbitMQ requeues the message.
6. Another consumer receives transfer-123.
7. The transfer is processed again.

The messaging system has behaved correctly. The message was never acknowledged, so RabbitMQ delivered it again.

The duplicate side effect is now the application’s responsibility.

---

Reconnection Makes Idempotency Non-Optional

RabbitMQ acknowledgements provide at-least-once delivery, not exactly-once business execution. A consumer must assume that any message can be delivered more than once, especially around connection loss. For transaction-sensitive workflows, this usually means giving every operation a stable idempotency key:

async function processTransfer(message: TransferMessage) {
  const existingTransfer = await transferRepository.findByReference(
    message.transferReference,
  );

  if (existingTransfer) {
    return existingTransfer;
  }

  return transferRepository.create({
    reference: message.transferReference,
    amount: message.amount,
    destination: message.destination,
  });
}

A database uniqueness constraint should enforce this guarantee:

CREATE UNIQUE INDEX transfers_reference_unique
ON transfers(reference);

The application-level check is useful, but the database constraint is the real protection against concurrent duplicate deliveries.

For more complicated workflows, the consumer may need to:

• Store processed message identifiers.
• Make state transitions conditional.
• Use database transactions.
• Pass idempotency keys to external services.
• Record external-operation results before acknowledging.
• Design handlers so repeating them produces the same final state.

RabbitMQ sets a redelivery flag when a message is delivered again. That is useful for logging and diagnosis, but it should not be the only duplicate-protection mechanism.

A message can be duplicated in more than one way, and the consumer should be safe regardless of the flag.

RabbitMQ’s reliability guide recommends designing consumers to be idempotent because redelivery is an expected part of connection recovery.

---

Catching ack Errors Is Still Not a Delivery Guarantee

There is a subtle limitation to the defensive wrapper.

If channel.ack(msg) does not throw, that only means the client accepted the operation locally. Consumer acknowledgements do not receive a corresponding promise or confirmation from amqplib. The network could fail immediately after the acknowledgement is written locally but before RabbitMQ processes it.

Therefore:

try {
  channel.ack(msg);
} catch (error) {
  // Handle a channel already known to be unusable.
}

protects the process from a known closed-channel error.

It does not prove that the broker received the acknowledgement.

That ambiguity is another reason idempotency must be part of the business workflow rather than an error-handling afterthought.

---

What Not to Do

Do not acknowledge through the replacement channel

The new channel does not own the old delivery. Delivery tags are scoped to their original channel.

Do not retry ack indefinitely

If the original channel is closed, repeatedly calling ack cannot restore it.

Do not silently swallow the error

Avoiding a crash is useful, but losing all evidence of the event makes the system difficult to operate. Record the queue, message identifier, delivery tag, channel generation, and settlement action.

Do not treat reconnection as successful recovery by itself

A “connected” metric says nothing about handlers that were running during the outage.

Do not assume a successful handler ran only once

The side effect may complete even when its acknowledgement does not.

---

The Test That Reveals the Problem

This failure mode is straightforward to test, but ordinary integration tests rarely create the required timing.

A useful test should:

1. Publish a message with a known identifier.
2. Start processing it with a deliberately slow handler.
3. Allow the business side effect to complete.
4. Restart RabbitMQ before the handler acknowledges.
5. Allow the handler to reach its acknowledgement code.
6. Wait for the consumer to reconnect.
7. Observe the message being redelivered.
8. Verify that the process remains alive.
9. Verify that the business side effect exists only once.
10. Verify that the consumer continues processing new messages.

The assertions matter more than whether the reconnect log appears.

A successful test should prove that:

• The old handler cannot crash the process.
• The closed channel is never reused.
• The old delivery is not acknowledged through the new channel.
• The message can be redelivered safely.
• Duplicate execution does not duplicate the business result.
• The recovered consumer continues processing.

This test exercises the boundary between messaging reliability and business reliability.

---

Observability Must Cover Both Timelines

Connection metrics are necessary, but they are incomplete.

For this failure mode, useful logs include:

queue
messageId
deliveryTag
redelivered
channelGeneration
handlerStartedAt
handlerCompletedAt
channelClosedAt
settlementAction
settlementOutcome

Useful metrics include:

rabbitmq_consumer_reconnect_total
rabbitmq_delivery_redelivered_total
rabbitmq_settlement_skipped_total
rabbitmq_settlement_error_total
consumer_handler_duration
consumer_duplicate_operation_total

A reconnect event tells you the future timeline recovered.

A skipped acknowledgement, redelivered message, or duplicate-operation event tells you what happened to the current timeline.

You need both views to understand the incident.

---

Final Takeaway

A reconnecting RabbitMQ consumer is better than one that stays disconnected, but reconnection is only half of recovery.

The harder problem is work that crosses the failure boundary:

• The message arrived through the old channel.
• The handler continued after that channel closed.
• The business operation may have completed.
• The acknowledgement may not have reached RabbitMQ.
• The message may therefore be delivered again.

A reliable consumer must associate every delivery with the channel that owns it, handle settlement operations defensively, expose uncertain outcomes through logs and metrics, and make business processing idempotent.

The goal is not to pretend the failure never happened. The goal is to ensure that when infrastructure dies halfway through the work, the application fails predictably, recovers safely, and produces the correct business result even if the message returns.

That is the difference between a consumer that reconnects and a messaging system that actually recovers.

---