Final Project Report 1: Schema Evolution Support on Apache SeaTunnel Flink Engine

Over the past two weeks, we interviewed several outstanding developers from the OSPP program to learn about their project experiences. Today, we’re sharing a detailed technical report from one of these projects — supporting Schema Evolution on the Flink engine — to help the community better understand the latest development progress in Apache SeaTunnel.

I. Completed Work

According to the original plan (link) and schedule, the support for Schema Evolution on the Flink engine has been completed.
On the Sink side, the following components now support dynamic schema changes during streaming:

• JdbcSinkWriter
• JdbcExactlyOnceSinkWriter
• ConsoleSinkWriter

All have passed testing with no significant bugs found during the schema evolution process.

✅ Dynamically inserted operators between source and transform. When a class implementing SupportSchemaEvolution is detected and the configuration is enabled, a SchemaOperator is inserted.
✅ Implemented a SchemaCoordinator to receive flush reports from sink writers and schema change requests from SchemaOperator.
✅ Extended the SchemaChangeEvent subclass to support FlushEvent propagation.
✅ Enhanced SupportSchemaEvolutionSinkWriter to report flush success events and handle FlushEvent.
✅ Implemented SchemaOperator to detect and process schema change events emitted by the source and forward them downstream.
✅ Overrode SupportSchemaEvolutionSinkWriter schema evolution methods, currently supporting JdbcSinkWriter, JdbcExactlyOnceSinkWriter, and ConsoleSinkWriter. All tested and working as expected.
✅ Extended FlinkRowCollector.collect() to support change event collection.
✅ Extended FlinkSinkWriter to detect and process schema change events.
✅ Added detailed error codes and exception handling for schema evolution errors.
✅ Implemented automatic exception throwing and retry mechanism when schema change errors occur.

II. Challenges and Solutions

1. Event Propagation Issue

When inserting an operator between the source and transform, it must distinguish between normal data and event messages. If it encounters an event, it should block, wait for the flush and schema update to complete, then continue propagation.

Two approaches were considered:

• Option 1: Follow the Zeta engine’s approach by creating a new StreamElement class similar to Record. However, this would require modifying the entire data flow from source to sink, introducing high risk and invasiveness.
• Option 2: Add a special marker within SeaTunnelRow, such as an option in options, to indicate events. This approach minimizes intrusion but slightly violates the single-responsibility principle, as SeaTunnelRow ideally should not handle event logic. For now, the focus is on functionality and stability, with future refactoring planned for cleaner design.

This marker allows the SchemaOperator to detect schema change events. However, it also introduced an issue — creating an empty row caused sink write errors. This was resolved by adding detection logic on the sink side.

Through this mechanism, event propagation issues were successfully addressed.

2. Multi-Parallelism Issue

In Flink CDC, during the incremental snapshot phase, multiple tasks run in parallel based on user-defined parallelism to read snapshot data. However, in the incremental phase, only one task continues reading to preserve event order.

Therefore, the coordinator doesn’t need a complex design — it only needs to handle sink responses without worrying about duplicated or out-of-order reports.

Further investigation showed that although the CDC source uses a single task to read binlogs, data is distributed downstream via KeyGroupStreamPartitioner by primary key hash, sending different keys to different sink subtasks.

When sink parallelism differs from input parallelism, Flink performs primary key shuffle to realign partitions.

However, the following challenges were identified:

1. Implementing shuffle logic would be complex.
2. If multiple parallel sinks receive schema change commands simultaneously, non-idempotent databases (e.g., StarRocks) could be affected. A workaround could involve coordinating through acknowledgments, but this would add unnecessary complexity.

For now, schema change detection forces sink parallelism to 1 for simplicity and correctness. Future iterations may reintroduce multi-parallelism support.

3. Flush and Request Execution Order

Originally, the SchemaOperator handled schema changes by flushing first, then requesting schema change from the coordinator.
This caused issues because the coordinator’s SchemaChangeState wasn’t initialized yet, resulting in missed FlushEvent notifications and task timeout failures.

Log analysis showed:

• 12:33:36,597 — FlushEvent processed; sink reported flush success.
• 12:33:36,597 — Coordinator warning: “No schema change state found.”
• 12:33:36,598 — Coordinator created schema change state.

The issue was resolved by reversing the execution order — requesting schema change first, then flushing — ensuring correct synchronization and preventing task blockage.

4. FlushData and Schema Change Separation

Initially, FlushEvent contained SchemaChangeEvent, combining data flushing and schema change in one step.
This mixed responsibilities and caused confusion. The design was refactored so that data flushing only handles data, while schema change is handled separately by the coordinator afterward. This separation clarified logic and improved maintainability.

5. Default Implementation and Interface Responsibilities

To maintain backward compatibility, new methods in SupportSchemaEvolutionSinkWriter are temporarily marked as default.
Once all implementations are complete and stable, the default keyword will be removed in subsequent iterations.

6. Handling Schema Change Failures: Mark Failure vs. Rollback

If a job fails (due to network or other reasons), should it mark the job as failed and let Flink recover from checkpoints, or perform rollback?

Flink CDC handles this by marking the job as failed and recovering from the last checkpoint.
Following the same design, this project marks failure and lets the built-in retry mechanism handle recovery, leveraging Flink’s global retry strategies (no, fixed-delay, failure-rate) and SeaTunnel’s own retry logic.

To improve debuggability, custom exception classes and error codes were introduced instead of using generic RuntimeException, providing clearer diagnostic information for both developers and users.

III. Test Cases and Results

Test Scenario: MySQL CDC → MySQL

• ✅ Add Column Test Report
• ✅ Drop Column Test Report
• ✅ Modify Column Test Report
• ✅ Change Column Test Report

All tests passed successfully, confirming stable schema evolution behavior across multiple DDL event types.

IV. Future Work Plan

• Implement remaining sink types that support SupportSchemaEvolutionSinkWriter.
• Test data flow consistency across different data sources and fix minor bugs if found.
• Conduct stress testing with large datasets to check for potential blocking issues.
• Verify data consistency under high concurrency scenarios.

In Summary

This project marks an important milestone in the evolution of Apache SeaTunnel’s Flink engine, enabling robust Schema Evolution support for streaming pipelines — a foundational step toward more flexible, self-adaptive data integration.