Kafka Offset Reset vs Message Replay: They Solve Different Problems

Ravitheja — Fri, 17 Jul 2026 11:20:27 +0000

When a Kafka consumer misses records or processes them incorrectly, teams often say:

“We need to replay the messages.”

But that phrase can describe very different operations.

Two approaches are commonly grouped together:

Resetting a consumer group’s offsets
Reading selected records and producing them again

Both can cause records to be processed again, but their operational behavior, risks, and use cases are different.

Understanding that distinction is important before performing recovery in production.

What does a consumer offset reset do?

A Kafka consumer group stores the position from which each partition should continue consuming.

Resetting offsets changes those positions.

For example, a team may move a consumer group from offset 10,000 back to offset 8,000. When the consumers start again, they read those earlier records through the normal consumer application.

The original Kafka records are not copied.

Instead, the existing consumer application reads them again.

This may be appropriate when:

The consumer application failed to process a range of records
A deployment introduced a processing defect
The same consumer logic must run again
The consumer group can be stopped or coordinated safely
Reprocessing the complete selected range is acceptable

However, resetting offsets may cause every downstream side effect in the consumer application to execute again.

That can include:

Database writes
Notifications
External API calls
Payments or ledger actions
Cache updates
Search indexing
Audit events

The offset operation may be simple. The downstream consequences may not be.

What does selective message replay do?

Selective replay is a different workflow.

Instead of moving a consumer group’s position, a replay process reads records from a chosen source range and produces them to a destination topic.

That destination may be:

The original source topic
A retry topic
A recovery topic
A testing topic
A downstream processing topic

The replay operation can independently define:

Source topic
Source partitions
Start and end offsets
Destination topic
Routing behavior
Replay rate
Replay metadata
Execution window

This makes selective replay useful when the team does not want to reposition an entire consumer group.

It also makes the recovery operation separately observable from the normal consumer lifecycle.

Example: failed payment-enrichment events

Assume a consumer reads payment events and enriches them using an external service.

Offsets 50,000 through 50,500 were consumed while the enrichment service was unavailable.

There are two possible recovery approaches.

Option 1: Reset the consumer offsets

The consumer group is moved back to offset 50,000.

The consumer rereads the records and executes its normal logic again.

This may be correct when:

The entire range should be processed again
The consumer is idempotent
Re-running every side effect is safe
The consumer group can be coordinated
No separate recovery route is required

Option 2: Selectively replay the records

The failed records are read from the source or dead-letter topic and produced to a recovery topic.

A recovery consumer processes only those records.

This may be preferable when:

Only specific records require recovery
The normal consumer group should not be repositioned
Recovery traffic needs rate limiting
Additional replay metadata is required
The replay must be separately audited
The destination should differ from the original topic

Neither method is universally better.

The correct choice depends on the recovery objective.

Risks shared by both approaches

Offset resets and selective replay differ, but both create production risk when they are not carefully controlled.

1. Incorrect range selection

A technically valid offset range may still be the wrong business range.

Before execution, the operator should confirm:

Topic
Partition
Start offset
End offset
Expected record count
Event timestamps
Business context

A range can be valid in Kafka and still be unsafe for the business workflow.

2. Duplicate downstream effects

Kafka can deliver records again, but it cannot automatically make every external side effect idempotent.

Consumers may need:

Idempotency keys
Deduplication tables
Unique database constraints
Transactional outbox patterns
Replay-aware business logic
Duplicate-detection rules

A replay that successfully produces records can still create incorrect downstream outcomes.

3. Unexpected load

Replaying records too quickly can overload:

Consumers
Databases
External APIs
Search indexes
Caches
Third-party services

Replay rate should therefore be treated as an operational control, not merely a performance setting.

4. Limited execution visibility

During a replay, teams may need to know:

Which partitions have started
Which offsets have been produced
Which offsets remain
Whether a partition failed
Whether execution is paused
Whether the replay can resume safely
Whether records are still in flight

A process ID and a log file are often not enough during an incident.

5. Weak auditability

After the recovery, teams may need to answer:

Who initiated the replay?
Why was it required?
Which records were selected?
Which destination was used?
What rate was configured?
When was the replay paused or resumed?
Did every partition complete?
Did the job succeed or fail?

These questions become especially important in payments, finance, logistics, healthcare, and other operationally sensitive systems.

A practical decision framework

Use an offset reset when:

The same consumer group should reread the records
The entire selected range should run through the existing consumer logic
The consumer group can be safely stopped and coordinated
Downstream behavior is sufficiently idempotent
Separate replay routing is unnecessary
The consumer’s normal monitoring is sufficient

Use selective replay when:

Only selected records or ranges should be reprocessed
Records need to be sent to a recovery or retry topic
The normal consumer group should remain unchanged
The operation requires separate rate limits
Replay metadata is needed
A dedicated audit trail is required
Pause, resume, or stop controls matter
Partition-level execution progress must be tracked

Use neither approach until:

Downstream side effects have been reviewed
Duplicate-processing behavior is understood
The replay range has been validated
A stop or rollback procedure has been defined

Production replay checklist

Before executing either recovery method, confirm:

Source topic

Is this the correct topic for the recovery scenario?
Partitions

Are only the intended partitions selected?
Offset boundaries

Are the start and end offsets correct and currently available?
Estimated record count

How many records will be processed again?
Event-time boundaries

Do the selected records match the intended incident window?
Destination behavior

Will records return to the source topic, a retry topic, or a recovery topic?
Consumer idempotency

Can downstream consumers safely process duplicates?
External side effects

Could the replay repeat payments, notifications, API calls, or database writes?
Replay rate

What production rate can downstream systems safely absorb?
Monitoring

How will progress and failures be observed for each partition?
Pause or stop procedure

Can execution be interrupted without losing track of progress?
Audit requirements

Are the operator, reason, selected range, and lifecycle actions recorded?

The most dangerous replay is not necessarily the one that fails.

It may be the one that succeeds against the wrong range.

Why we built ReplayGuard

Many Kafka teams can perform a replay technically.

The difficult part is often the operational workflow surrounding it:

Inspecting the correct records
Selecting exact partitions and offsets
Validating the request before execution
Applying replay-rate limits
Monitoring partition-level progress
Pausing, resuming, stopping, or cancelling execution
Preserving the operator, reason, and lifecycle history

We built ReplayGuard as a self-hosted Kafka replay and debugging platform focused on that workflow.

ReplayGuard runs inside the customer’s own infrastructure, allowing Kafka access, credentials, records, and supporting services to remain within the customer-controlled environment.

You can see the workflow here:

https://replayguard.in/

Closing question

How does your team decide between a consumer offset reset and selective message replay?

And which safeguards must be in place before either action is allowed in production?

DEV Community: Ravitheja