From Four Platforms to One: How Tongcheng Travel Built a Unified Data Integration Platform with Apache SeaTunnel

From four independent data integration platforms to one unified batch-stream architecture—discover how Tongcheng Travel leveraged Apache SeaTunnel's Zeta Engine to migrate tens of thousands of production jobs with zero business impact.

For years, Tongcheng Travel's data integration ecosystem evolved into four separate platforms: Data Transfer, Data Integration, Sqoop, and Apache SeaTunnel. While each platform served specific business needs, overlapping functionality, fragmented execution engines, and increasing operational complexity gradually became major challenges.

To address these issues, Tongcheng Travel adopted Apache SeaTunnel Zeta Engine as the unified foundation for its next-generation data integration platform.

This article shares the complete migration journey—from automatically translating legacy Sqoop and FlinkSQL jobs without requiring code changes, to building an AI-powered Data Copilot capable of creating data pipelines using natural language, to designing a large-scale data consistency validation framework that guarantees migration accuracy.

Ultimately, the project successfully consolidated four independent platforms into a unified batch-stream processing architecture while significantly improving operational efficiency, platform maintainability, and future scalability. Looking ahead, the team is continuing to invest in cloud-native architecture and AI-driven intelligent operations.

Meetup Recording

https://youtu.be/b224ISVIU7A?si=eOxLZ1xyUAfdOUVs

About the Author

Xiaochen Zhou is a Data Platform Engineer at Tongcheng Travel and an Apache SeaTunnel Committer. He has been actively contributing to the SeaTunnel community, focusing on engine evolution, connector development, and large-scale enterprise adoption.

Building a Unified Data Integration Platform

Existing Architecture

Tongcheng Travel previously operated four independent data integration services:

• Data Transfer
• Data Integration
• Sqoop
• Apache SeaTunnel

Although these systems were built for different scenarios over time, they gradually evolved to provide overlapping capabilities, resulting in duplicated functionality, inconsistent architectures, and increasing maintenance costs.

Specifically:

• Data Transfer and Sqoop were built on Flink 1.6 and MapReduce, respectively. They mainly handled offline synchronization between relational databases and big data systems. However, when processing large-scale database synchronization jobs, these engines generated significant pressure on production databases, affecting the stability of online business systems.

• Data Integration, based on Apache Flink, focused on real-time synchronization scenarios, primarily moving operational database data into the company's data lake.

• Meanwhile, the company's healthcare business had already adopted Apache SeaTunnel Zeta Engine. Thanks to its lightweight architecture designed specifically for data integration workloads, SeaTunnel demonstrated superior resource efficiency and synchronization performance.

Overall, maintaining four different data integration platforms had become increasingly unsustainable.

The team decided to consolidate all existing services into a unified data integration platform powered entirely by Apache SeaTunnel, providing a single engine capable of handling both batch and streaming workloads.

Design Goals and Guiding Principles

To ensure the migration could be completed safely and efficiently, Tongcheng Travel established three core principles.

Zero Business Impact

The migration had to be completely transparent to application teams.

Historical jobs should continue running without requiring developers to rewrite existing scripts or learn a new execution model.

Before the final cutover, the platform supports a dual-run validation strategy, allowing both the legacy engine and Apache SeaTunnel to execute the same job simultaneously. Once validation is complete, workloads can be switched seamlessly without any business-side changes.

Guaranteed Data Consistency

Data quality is the foundation of every data integration platform.

A comprehensive validation and fallback mechanism must ensure that migrated jobs produce results identical to those generated by the legacy engines, guaranteeing consistency before, during, and after migration.

Better Performance and Stability

Legacy MapReduce-based pipelines were heavyweight, while Flink 1.6 introduced significant load on production databases and lacked many modern optimizations.

By standardizing on Apache SeaTunnel Zeta Engine, Tongcheng Travel aimed to leverage its lightweight execution model and high-performance architecture to improve synchronization efficiency while reducing infrastructure costs.

Automatic Task Migration

After defining the overall architecture, the team's biggest challenge became clear:

How can tens of thousands of production jobs be migrated from Sqoop and FlinkSQL to Apache SeaTunnel without requiring developers to modify a single line of code?

Migrating Sqoop and FlinkSQL to Apache SeaTunnel

For most organizations, upgrading the underlying data integration engine usually requires significant cooperation from application teams.

Tongcheng Travel took a completely different approach.

Their guiding principle was:

Zero business awareness. Zero code changes.

Existing Sqoop scripts and FlinkSQL jobs continue to be submitted exactly as before.

Behind the scenes, the platform transparently converts them into standard Apache SeaTunnel jobs and executes them on the SeaTunnel engine.

Skill Layer

The first layer of the architecture consists of a collection of intelligent "Skill" modules responsible for understanding different job formats.

Legacy production jobs—including tens of thousands of Sqoop scripts and FlinkSQL statements—remain completely unchanged.

Instead, specialized translation components interpret each job type:

• Sqoop Skill
• FlinkSQL Skill
• SeaTunnel Skill

These Skills recognize syntax, configuration parameters, execution semantics, and connector definitions before mapping them into a standard Apache SeaTunnel configuration.

This abstraction layer isolates application developers from changes in the underlying execution engine.

Execution Layer

After translation is complete, the unified submission service generates a standard SeaTunnel job configuration.

To guarantee migration safety, Tongcheng Travel introduced a dual-run execution mechanism.

Whenever a workflow is triggered, the platform actually launches two independent jobs.

Production Baseline Job

The original Sqoop or FlinkSQL task continues running exactly as before.

This guarantees that existing production data delivery remains completely unaffected throughout the migration process.

Apache SeaTunnel Canary Job

At the same time, the translated Apache SeaTunnel job is submitted to a separate execution cluster.

Instead of writing into production storage, the output is written into a validation environment for comparison.

Intelligent Validation and Seamless Cutover

Once both jobs complete, an automated validation framework compares their outputs.

If every validation rule passes successfully, future executions of that workload are automatically routed to Apache SeaTunnel.

From the application's perspective, nothing changes.

The migration happens entirely behind the scenes.

From Natural Language to Apache SeaTunnel

Migrating historical workloads solves yesterday's problems.

The next challenge is enabling future development to become dramatically easier.

Tongcheng Travel wanted business users—not just experienced data engineers—to create data synchronization tasks without needing to understand connectors, SQL dialects, or execution engines.

Their answer is Data Copilot, an AI-powered pipeline generation framework built on large language models.

Intelligent Information Completion

Real-world user requests are rarely complete.

A request such as:

"Synchronize yesterday's core product sales data into the data warehouse."

contains insufficient information for execution.

It doesn't specify:

• the source database
• the destination table
• partition strategy
• synchronization mode
• connector configuration
• execution parallelism

To bridge this gap, Tongcheng Travel implemented an intelligent information completion framework.

The system first extracts the user's intent.

It then combines:

• historical user behavior,
• enterprise metadata,
• business semantics,
• existing synchronization patterns,

to automatically infer the missing information.

Finally, the system selects the appropriate Source Connector (for example, the MySQL Core Product table) and Sink Connector (such as Hive), while the SeaTunnel Skill automatically generates low-level execution parameters, including partitioning strategy, parallelism, and connector configuration.

The result is a fully executable Apache SeaTunnel job generated entirely from natural language.

Text2SQL: From Natural Language to SQL

Building complete data pipelines from natural language requires more than simply understanding user intent. The platform must also generate accurate, executable SQL statements that can be seamlessly embedded into Apache SeaTunnel jobs.

To achieve this, Tongcheng Travel designed a multi-stage Text2SQL architecture consisting of Schema Linking, Candidate SQL Generation, and Candidate Selection.

Schema Linking

Schema Linking identifies references to database schemas, tables, columns, and filtering conditions within a user's natural language request.

This step significantly improves both cross-domain generalization and the accuracy of complex SQL generation, making it a foundational component of nearly every modern Text2SQL system.

For example, when a user asks, "Synchronize yesterday's hotel orders," the system automatically identifies the corresponding business tables, date fields, and filtering conditions based on enterprise metadata and schema information.

Candidate SQL Generation

Rather than relying on the output of a single large language model, Tongcheng Travel adopts a multi-generator strategy to maximize SQL generation quality and reliability.

Three independent generators work in parallel.

Reasoning Generator

The Reasoning Generator leverages the zero-shot reasoning capabilities of large language models to generate SQL directly from the user's intent and the underlying database schema.

This approach performs well when encountering entirely new business scenarios without requiring historical examples.

In-Context Learning (ICL) Generator

The ICL Generator adopts a Few-shot Learning strategy.

Instead of generating SQL from scratch, it retrieves previously executed high-quality SQL statements that closely resemble the current request and provides them as examples to the language model.

This significantly improves generation accuracy for recurring business scenarios.

Divide-and-Conquer Generator

Some enterprise SQL statements span hundreds or even thousands of lines and include numerous nested subqueries.

Instead of generating one massive SQL statement, the Divide-and-Conquer Generator decomposes the query into multiple Common Table Expressions (CTEs), generates each subquery independently, and finally assembles them into a complete SQL statement.

This approach dramatically improves generation quality for highly complex analytical queries.

Candidate SQL Selection

Once multiple candidate SQL statements have been generated, they undergo an intelligent evaluation process.

Each candidate is scored based on several dimensions, including:

• SQL syntax correctness
• Abstract Syntax Tree (AST) complexity
• Estimated execution cost
• Semantic consistency with the original request

Finally, a dedicated SQL Selector chooses the optimal SQL statement and embeds it directly into the Apache SeaTunnel Transform stage.

This multi-path generation strategy substantially improves both SQL accuracy and execution stability compared to traditional single-model approaches.

Text2ETL: Generating Data Pipelines from Natural Language

Beyond SQL generation, Tongcheng Travel also enables complete ETL workflows to be created through natural language.

Interactive Preview

After processing a user's request, the platform immediately generates a preview of the transformed dataset.

Users can verify whether the generated pipeline matches their expectations before execution, significantly reducing trial-and-error during development.

Intelligent Transform Routing

For standard data transformation requirements, the system automatically maps user intent to Apache SeaTunnel's native Transform plugins, including operations such as:

• Filter
• Replace
• Split
• Additional built-in Transform components

Whenever possible, the platform prioritizes native SeaTunnel capabilities to maximize execution performance and maintainability.

Dynamic Compilation as a Fallback

Some enterprise data cleansing scenarios are too complex to be expressed using combinations of built-in Transform operators.

Examples include sophisticated data quality rules, custom parsing logic, or highly specialized business transformations.

In these situations, the platform automatically generates Java, Scala, or Groovy code implementing a User Defined Function (UDF).

The generated code is dynamically compiled in the background and loaded into the Apache SeaTunnel runtime as a plugin without requiring manual packaging or deployment.

This mechanism allows developers to benefit from AI-generated pipeline creation while preserving the flexibility needed for complex enterprise workloads.

Data Consistency Validation During Migration

Automatically converting jobs is only the first step in a successful migration.

The true challenge lies in ensuring that both the legacy engine and Apache SeaTunnel produce exactly the same results.

Because different execution engines may implement serialization formats, timestamp handling, numeric precision, and connector behavior differently, guaranteeing zero data loss and zero data deviation becomes the most critical requirement of the entire migration project.

Tongcheng Travel therefore designed a comprehensive validation framework capable of verifying large-scale production datasets efficiently.

Validation for File-Based Engines

Unlike relational databases, file systems do not provide built-in query capabilities.

Loading terabytes of files into memory for full comparison would be both time-consuming and extremely memory-intensive, potentially leading to OutOfMemory (OOM) errors.

To address this challenge, Tongcheng Travel designed a highly efficient comparison framework based on side indexes and multi-level filtering.

Step 1. Data Reading and Normalization

The system concurrently scans both output directories:

• Legacy XData/Sqoop output (baseline)
• Apache SeaTunnel Zeta output (candidate)

Each record is read line by line and first normalized to eliminate formatting differences.

After normalization, the entire row is hashed using a deterministic hash algorithm.

Step 2. Building Side Indexes

To quickly locate discrepancies without rescanning original files, the platform builds an ordered external index while computing hash values.

The side index consists of two layers.

File-Level Index

The header records the mapping between a logical file index and its physical storage location.

For example:

fileIndex = 0 → .../part-00000

Record-Level Index

Each normalized record generates an index entry containing:

[hash, fileIndex, lineNumber]

These index entries are globally sorted by hash value using external sorting algorithms.

This preprocessing lays the foundation for highly efficient large-scale comparison.

Step 3. Multi-Level Comparison Algorithm

Instead of comparing every record directly, the validation framework performs comparison in multiple stages.

Level 1: Bloom Filter Screening

Each dataset first generates an independent Bloom Filter.

BloomFilter(XData)

BloomFilter(Zeta)

Because Bloom Filters require very little memory while providing extremely fast lookup performance, they quickly eliminate the vast majority of matching records.

Only potentially inconsistent data proceeds to the next stage.

Level 2: Two-Pointer Comparison

For records that cannot be confirmed through Bloom Filter screening, the system compares the two sorted hash indexes using a classic two-pointer algorithm.

If:

h1 < h2

the baseline dataset contains records missing from Apache SeaTunnel.

The pointer for the baseline dataset advances.

If:

h1 > h2

Apache SeaTunnel contains additional records.

The pointer for the candidate dataset advances.

If:

h1 == h2

the framework performs an additional comparison of the occurrence count for that hash value.

This effectively handles duplicate records while maintaining comparison accuracy.

Step 4. Difference Localization

Whenever inconsistencies are detected, the framework performs a binary search on the side index to rapidly locate the offending hash value.

Using the corresponding:

• fileIndex
• lineNumber

developers can immediately identify the exact row in the original source file where corruption, truncation, or encoding issues occurred.

This reduces troubleshooting time from hours to seconds.

Step 5. Automatic Cutover

Only after every validation rule passes successfully across the complete dataset does the platform perform the final production cutover, routing future executions entirely to Apache SeaTunnel.

Validation for Database Engines

For relational databases and data warehouses, Tongcheng Travel takes advantage of the database engine itself to calculate data fingerprints, eliminating the need to export large datasets for comparison.

Instead of comparing records one by one, the platform computes hash-based feature values for every column and compares the aggregated results between the legacy engine and Apache SeaTunnel.

SELECT
  sum(murmur_hash3_32(coalesce(CAST(order_serial_no AS string), 'NULL'))) AS order_serial_no_hash,
  sum(murmur_hash3_32(coalesce(CAST(platform_code AS string), 'NULL'))) AS platform_code_hash,
  ...
FROM ...

By comparing the hash signatures of each column, the platform can quickly determine whether two datasets are identical with minimal computational overhead.

Compared with traditional full-table comparison, this approach dramatically improves validation efficiency while maintaining high accuracy, making it well suited for large-scale production migrations.

Improving Performance and Stability

Data consistency guarantees migration correctness, but long-term platform success also depends on execution efficiency and operational stability.

To maximize the performance of the unified data integration platform, Tongcheng Travel optimized Apache SeaTunnel across three key areas:

• Connector enhancements
• Intelligent parallelism estimation
• Improved observability

Connector Enhancements

Since connectors serve as the bridge between Apache SeaTunnel and enterprise data systems, improving connector reliability directly improves overall platform stability.

The team contributed a series of enhancements covering multiple ecosystems.

Enhanced Paimon Connector

The Paimon connector received significant improvements, including:

• Full support for the TIME data type
• Predicate pushdown for LIKE and BETWEEN operations, reducing unnecessary data scanning
• Dynamic table option discovery
• Branch writing support for Sink operations
• Fixes for DECIMAL precision loss
• Resolution of missing fields during DataType conversion
• Parallel reading from multiple Paimon Source tables

These enhancements significantly improve both compatibility and execution efficiency for Paimon workloads.

High Availability for OLAP Connectors

The StarRocks and Doris connectors were enhanced to improve availability and reliability.

For StarRocks:

• Added Frontend (FE) High Availability support
• Randomized FE endpoint selection to eliminate single points of failure

For Doris:

• Fixed data loss issues occurring when

request_table_size < BUCKETS

These improvements greatly increase connector resilience in production environments.

Improved Streaming Stability

Several optimizations were introduced for streaming connectors.

For Kafka:

• Added partition filtering in Stream Mode to prevent consumer blocking

For RocketMQ:

• Added support for skipping malformed records instead of terminating the entire synchronization job

For Milvus:

• Fixed missing partition-level Load State validation

These changes improve fault tolerance without sacrificing throughput.

HBase Improvements

The HBase Source connector now supports row-range boundary queries, allowing more efficient incremental extraction and partitioned reads.

HDFS ViewFs Compatibility

Apache SeaTunnel now supports the ViewFs filesystem schema, enabling seamless access to HDFS deployments spanning multiple namespace mount points.

This enhancement improves compatibility with large enterprise Hadoop clusters.

Intelligent Parallelism Estimation

Inspired by Apache Flink's Adaptive Batch Scheduler, Tongcheng Travel implemented an Intelligent Parallelism Estimation mechanism for Apache SeaTunnel Zeta Engine.

Before a job is submitted, the engine performs lightweight metadata analysis to estimate the actual workload.

Depending on the source type, it collects information such as:

• Total table row count
• Directory size
• Kafka partition count
• Paimon bucket distribution
• Physical storage topology

These physical characteristics are then combined with runtime information, including:

• Available cluster resources
• Destination throughput limits
• Current cluster workload
• Resource utilization

Using these inputs, the scheduler dynamically calculates the optimal degree of parallelism and determines the most efficient task partitioning strategy.

Rather than relying on static configuration, computing resources are allocated on demand, allowing Apache SeaTunnel to align task execution with the underlying storage topology.

The result is higher resource utilization, shorter execution times, and improved overall cluster efficiency.

Improved Observability

As the unified platform continued to grow, comprehensive observability became increasingly important.

Tongcheng Travel therefore introduced several monitoring capabilities to improve operational visibility.

Checkpoint Monitoring

Inspired by Apache Flink's mature checkpoint monitoring system, the team built a comprehensive observability framework for Apache SeaTunnel Checkpoints.

Key metrics include:

• End-to-end Checkpoint duration
• State size
• Checkpoint failure frequency
• Checkpoint success rate

These metrics help engineers quickly identify bottlenecks affecting fault tolerance and recovery performance.

Master Election Metrics

Borrowing ideas from Apache Kafka's controller election monitoring, Tongcheng Travel also implemented detailed monitoring for Apache SeaTunnel Master elections.

The platform continuously tracks:

• Election duration
• Master switch frequency
• Active Master availability
• Election success rate

When abnormal conditions occur—such as split-brain scenarios or frequent Master failovers—the monitoring system immediately generates alerts, helping engineers diagnose underlying network issues or cluster resource bottlenecks before they impact production workloads.

Future Roadmap

With the unified data integration platform now running reliably in production, Tongcheng Travel is focusing on the next stage of its evolution.

Over the next one to two years, development will center around two strategic directions:

• Cloud-Native Architecture
• AI-Powered Data Integration

Cloud-Native Architecture

The team plans to fully embrace Kubernetes-native resource scheduling.

Future enhancements include:

• On-demand Worker provisioning
• Automatic Worker lifecycle management
• Elastic scaling based on workload
• Dynamic resource allocation
• Remote Job Log storage following cloud-native logging best practices

These capabilities will further improve infrastructure efficiency while reducing operational overhead.

AI-Powered Data Integration

Artificial intelligence will become a core component of the next-generation data platform.

The long-term vision is to build a closed-loop intelligent operations system capable of:

• Generating complete synchronization jobs from natural language
• Automatically identifying root causes of failed jobs
• Optimizing parallelism based on runtime performance and data latency
• Continuously tuning execution strategies without manual intervention

The goal is to transform data integration from a manually operated platform into a self-healing, self-optimizing system powered by AI.

Final Thoughts

A unified data platform is more than an engine upgrade—it's a foundation for future innovation. With Apache SeaTunnel at its core, Tongcheng Travel is building a cloud-native, AI-powered data integration platform ready for the next generation of enterprise data engineering.