Streaming Oracle changes to Kafka is useful for real-time analytics, search and cache updates, and event-driven services. The engineering choice is how much of the capture, delivery, and recovery logic your team wants to build and operate.
This guide compares two approaches: building a custom Oracle LogMiner pipeline, or using an existing CDC connector or platform. The first gives you control but requires careful transaction and checkpoint handling. The second reduces custom plumbing while leaving you responsible for configuration, deployment, and operational fit.
The Two Approaches at a Glance
| Approach | What you operate | Best fit |
|---|---|---|
| Custom LogMiner pipeline | Oracle snapshot and log reader, event transformation, Kafka producer, offsets, retries, monitoring | A small number of pipelines, specialized event formats, and a team experienced with CDC internals |
| Existing CDC connector or platform | Connector or platform configuration, infrastructure, schema mapping, and downstream validation | Production pipelines where the team wants established capture and recovery components |
Requirements for a Reliable Oracle-to-Kafka Pipeline
A production pipeline has to solve more than reading redo records. It must establish a consistent boundary between the initial load and ongoing changes, preserve transaction behavior, resume safely after failures, and produce events that Kafka consumers can interpret.
A common event envelope includes the source schema and table, operation, event time, primary key, and before/after values. Use the primary key as the Kafka record key when consumers need updates for one row to remain associated with the same entity.
{
"schema": "HR",
"table": "ORDERS",
"op": "UPDATE",
"scn": 1234567900,
"pk": {"ORDER_ID": 1001},
"before": {"STATUS": "PENDING"},
"after": {"STATUS": "PAID"}
}
For LogMiner-based capture, plan for ARCHIVELOG, supplemental logging, the required privileges, and retention of redo logs until the reader has processed them. Requirements vary by Oracle release and deployment, especially in CDB/PDB environments. Review Oracle's LogMiner documentation before applying a setup to production. CONTINUOUS_MINE is desupported in Oracle Database 19c and later; a reader must manage the log ranges it mines.
Method 1: Build a Custom LogMiner Pipeline
A custom implementation typically has five parts:
- Prepare Oracle. Enable the required logging, create a minimally privileged capture user, and confirm the redo logs will be retained long enough for the reader to catch up.
- Take a consistent snapshot. Record a snapshot SCN and export the existing rows using Flashback Query at that SCN.
- Mine changes after the snapshot boundary. Start LogMiner from the corresponding position and read committed changes from the required redo logs.
- Transform and publish events. Convert changes into a stable event format and write them to Kafka with a durable producer configuration.
- Checkpoint only confirmed progress. Persist the source position only after all earlier events in that position range have been acknowledged by Kafka.
The last step is easy to get wrong. Saving the largest offset among successful asynchronous sends can skip an earlier event that failed. Track a source position that is unique enough to resume safely, and advance only through the highest contiguous range of fully acknowledged events. If a process crashes after publishing but before saving its checkpoint, it may publish duplicates after restart; design consumers to be idempotent.
Do not treat SCN alone as a complete checkpoint in a custom reader. The implementation must define how it resumes within a mined position and how it handles transaction commits, rollbacks, and continuation rows. It must also avoid publishing partial transactions. These details are why a short LogMiner polling loop is not a production-ready CDC pipeline.
The custom approach also requires ongoing work for LogMiner session restarts, archived-log gaps, schema changes, retry behavior, monitoring, and throughput limits. It is useful when you need control over the pipeline internals and have engineers to own those responsibilities. For a small prototype, keep the scope narrow and test against failures before connecting production consumers.
Method 2: Use an Existing CDC Connector or Platform
An existing connector or CDC platform can provide the capture, offset, and recovery components that a team would otherwise build and maintain. Two common choices are Kafka-native connectors and self-hosted CDC platforms.
Kafka-native connector
The Debezium Oracle connector uses Oracle LogMiner by default to capture database changes and publish them through Kafka Connect. This can suit teams already operating Kafka Connect that want an open-source connector and are comfortable managing Oracle privileges, connector configuration, offsets, and deployment operations.
A connector still needs careful setup. Validate supported Oracle versions, table selection, snapshot behavior, transaction handling, offset storage, and recovery procedures against the connector documentation and your Kafka Connect environment.
Self-hosted CDC platform
A self-hosted CDC platform can combine source and target configuration, initial load, incremental capture, and job monitoring in one workflow. Disclosure: BladePipe is our product; it is used here as an implementation example. Apply the same evaluation criteria to other platforms.
A typical Oracle-to-Kafka workflow is:
- Prepare Oracle using the platform's LogMiner requirements and privilege guide.
- Install the self-hosted platform. For BladePipe, the on-premises quick start covers installation options.
- Add Oracle as the source and Kafka as the target, then test both connections.
- Configure an initial load with incremental synchronization if Kafka needs both existing rows and ongoing changes.
- Select the tables and columns, review topic and message-format settings, and start the job.
- Monitor initial-load progress, incremental lag, errors, and recovery behavior. Validate representative records in Kafka before routing production consumers to the topics.
This approach reduces the amount of custom CDC plumbing the team must maintain. It does not remove the need to size the deployment, review event formats, test failure recovery, or validate the data with downstream consumers.
Custom Pipeline vs. Existing CDC Components
| Decision area | Custom LogMiner pipeline | Connector or CDC platform |
|---|---|---|
| Control over implementation | Highest | Depends on connector or platform capabilities |
| Time to first working pipeline | Longer | Usually shorter after prerequisites are met |
| Checkpoint and recovery logic | Built and maintained by your team | Provided by the component; still needs configuration and testing |
| Kafka event format | Fully customizable | Uses supported formats and settings |
| Operational responsibility | LogMiner, code, offsets, retries, monitoring | Connector/platform deployment, configuration, monitoring, and upgrades |
| Best fit | Specialized requirements and CDC engineering expertise | Production use where established capture components fit the requirements |
Choose the custom approach when the event flow has requirements that existing tools cannot meet and your team can own the implementation long term. Choose a connector or platform when reducing custom code and operational risk matters more than controlling every part of the capture process. For either approach, test snapshot consistency, transaction ordering, replay behavior, schema changes, and consumer idempotency before production cutover.
Conclusion
Both approaches can stream Oracle changes to Kafka, but they place the engineering work in different places. A custom LogMiner pipeline offers control and requires careful ownership of offsets, transactions, and recovery. An existing connector or CDC platform can shorten implementation, provided its Oracle support, Kafka delivery behavior, and operational model match your workload.
Top comments (0)