When a Kafka consumer misses records or processes them incorrectly, teams often say:
“We need to replay the messages.”
But that phrase can describe very different operations.
Two approaches are commonly grouped together:
- Resetting a consumer group’s offsets
- Reading selected records and producing them again
Both can cause records to be processed again, but their operational behavior, risks, and use cases are different.
Understanding that distinction is important before performing recovery in production.
What does a consumer offset reset do?
A Kafka consumer group stores the position from which each partition should continue consuming.
Resetting offsets changes those positions.
For example, a team may move a consumer group from offset 10,000 back to offset 8,000. When the consumers start again, they read those earlier records through the normal consumer application.
The original Kafka records are not copied.
Instead, the existing consumer application reads them again.
This may be appropriate when:
- The consumer application failed to process a range of records
- A deployment introduced a processing defect
- The same consumer logic must run again
- The consumer group can be stopped or coordinated safely
- Reprocessing the complete selected range is acceptable
However, resetting offsets may cause every downstream side effect in the consumer application to execute again.
That can include:
- Database writes
- Notifications
- External API calls
- Payments or ledger actions
- Cache updates
- Search indexing
- Audit events
The offset operation may be simple. The downstream consequences may not be.
What does selective message replay do?
Selective replay is a different workflow.
Instead of moving a consumer group’s position, a replay process reads records from a chosen source range and produces them to a destination topic.
That destination may be:
- The original source topic
- A retry topic
- A recovery topic
- A testing topic
- A downstream processing topic
The replay operation can independently define:
- Source topic
- Source partitions
- Start and end offsets
- Destination topic
- Routing behavior
- Replay rate
- Replay metadata
- Execution window
This makes selective replay useful when the team does not want to reposition an entire consumer group.
It also makes the recovery operation separately observable from the normal consumer lifecycle.
Example: failed payment-enrichment events
Assume a consumer reads payment events and enriches them using an external service.
Offsets 50,000 through 50,500 were consumed while the enrichment service was unavailable.
There are two possible recovery approaches.
Option 1: Reset the consumer offsets
The consumer group is moved back to offset 50,000.
The consumer rereads the records and executes its normal logic again.
This may be correct when:
- The entire range should be processed again
- The consumer is idempotent
- Re-running every side effect is safe
- The consumer group can be coordinated
- No separate recovery route is required
Option 2: Selectively replay the records
The failed records are read from the source or dead-letter topic and produced to a recovery topic.
A recovery consumer processes only those records.
This may be preferable when:
- Only specific records require recovery
- The normal consumer group should not be repositioned
- Recovery traffic needs rate limiting
- Additional replay metadata is required
- The replay must be separately audited
- The destination should differ from the original topic
Neither method is universally better.
The correct choice depends on the recovery objective.
Risks shared by both approaches
Offset resets and selective replay differ, but both create production risk when they are not carefully controlled.
1. Incorrect range selection
A technically valid offset range may still be the wrong business range.
Before execution, the operator should confirm:
- Topic
- Partition
- Start offset
- End offset
- Expected record count
- Event timestamps
- Business context
A range can be valid in Kafka and still be unsafe for the business workflow.
2. Duplicate downstream effects
Kafka can deliver records again, but it cannot automatically make every external side effect idempotent.
Consumers may need:
- Idempotency keys
- Deduplication tables
- Unique database constraints
- Transactional outbox patterns
- Replay-aware business logic
- Duplicate-detection rules
A replay that successfully produces records can still create incorrect downstream outcomes.
3. Unexpected load
Replaying records too quickly can overload:
- Consumers
- Databases
- External APIs
- Search indexes
- Caches
- Third-party services
Replay rate should therefore be treated as an operational control, not merely a performance setting.
4. Limited execution visibility
During a replay, teams may need to know:
- Which partitions have started
- Which offsets have been produced
- Which offsets remain
- Whether a partition failed
- Whether execution is paused
- Whether the replay can resume safely
- Whether records are still in flight
A process ID and a log file are often not enough during an incident.
5. Weak auditability
After the recovery, teams may need to answer:
- Who initiated the replay?
- Why was it required?
- Which records were selected?
- Which destination was used?
- What rate was configured?
- When was the replay paused or resumed?
- Did every partition complete?
- Did the job succeed or fail?
These questions become especially important in payments, finance, logistics, healthcare, and other operationally sensitive systems.
A practical decision framework
Use an offset reset when:
- The same consumer group should reread the records
- The entire selected range should run through the existing consumer logic
- The consumer group can be safely stopped and coordinated
- Downstream behavior is sufficiently idempotent
- Separate replay routing is unnecessary
- The consumer’s normal monitoring is sufficient
Use selective replay when:
- Only selected records or ranges should be reprocessed
- Records need to be sent to a recovery or retry topic
- The normal consumer group should remain unchanged
- The operation requires separate rate limits
- Replay metadata is needed
- A dedicated audit trail is required
- Pause, resume, or stop controls matter
- Partition-level execution progress must be tracked
Use neither approach until:
- Downstream side effects have been reviewed
- Duplicate-processing behavior is understood
- The replay range has been validated
- A stop or rollback procedure has been defined
Production replay checklist
Before executing either recovery method, confirm:
Source topic
Is this the correct topic for the recovery scenario?Partitions
Are only the intended partitions selected?Offset boundaries
Are the start and end offsets correct and currently available?Estimated record count
How many records will be processed again?Event-time boundaries
Do the selected records match the intended incident window?Destination behavior
Will records return to the source topic, a retry topic, or a recovery topic?Consumer idempotency
Can downstream consumers safely process duplicates?External side effects
Could the replay repeat payments, notifications, API calls, or database writes?Replay rate
What production rate can downstream systems safely absorb?Monitoring
How will progress and failures be observed for each partition?Pause or stop procedure
Can execution be interrupted without losing track of progress?Audit requirements
Are the operator, reason, selected range, and lifecycle actions recorded?
The most dangerous replay is not necessarily the one that fails.
It may be the one that succeeds against the wrong range.
Why we built ReplayGuard
Many Kafka teams can perform a replay technically.
The difficult part is often the operational workflow surrounding it:
- Inspecting the correct records
- Selecting exact partitions and offsets
- Validating the request before execution
- Applying replay-rate limits
- Monitoring partition-level progress
- Pausing, resuming, stopping, or cancelling execution
- Preserving the operator, reason, and lifecycle history
We built ReplayGuard as a self-hosted Kafka replay and debugging platform focused on that workflow.
ReplayGuard runs inside the customer’s own infrastructure, allowing Kafka access, credentials, records, and supporting services to remain within the customer-controlled environment.
You can see the workflow here:
Closing question
How does your team decide between a consumer offset reset and selective message replay?
And which safeguards must be in place before either action is allowed in production?
Top comments (1)
I’m particularly interested in how teams handle replay when downstream systems are not fully idempotent.
Do you prefer consumer-group offset reset, selective replay to a recovery topic, or a custom workflow?