A replication job can report RUNNING while the row you care about is still missing from the target. For an engineer operating a CDC pipeline, the useful questions are concrete: Did the initial data arrive? Did an update change the existing row? Did a delete disappear downstream? Can the job resume and process a new change after a savepoint?
This walkthrough answers those questions with a small, inspectable MySQL CDC → PostgreSQL pipeline. It is based on a real run of the official Apache SeaTunnel 3.1.0 distribution, with source statements and target queries captured at each stage. The exercise covers the initial snapshot, inserts, updates, deletes, one add-column schema change, and controlled savepoint stop-and-restore.
Apache SeaTunnel is an open-source data integration tool. Here, its MySQL CDC connector reads the source, the SeaTunnel engine executes the pipeline, and the JDBC sink writes to PostgreSQL. There is no intermediate transform, Kafka cluster, or custom transfer application in this example.
The commands below are deliberately plain: native SeaTunnel shell scripts, the MySQL client, and psql. The setup is included in the article, so the walkthrough does not depend on a video or an external helper download.
What was actually tested
MySQL 8.0.46 SeaTunnel 3.1.0 PostgreSQL 16.6
integration.customers → MySQL-CDC → Zeta → JDBC → integration.public.customers
one engine node
parallelism = 1
| Component | Tested choice |
|---|---|
| SeaTunnel binary and CDC/JDBC connector artifacts | 3.1.0 |
| Java runtime | Java 11.0.33 |
| MySQL image | mysql:8.0.46 |
| PostgreSQL image | postgres:16.6-alpine |
| MySQL JDBC driver | 8.4.0 |
| PostgreSQL JDBC driver | 42.7.8 |
| Job mode |
STREAMING, asynchronous cluster submission |
| Source table | MySQL database integration, table customers
|
| Target table | PostgreSQL database integration, schema public, table customers
|
The release archive was checked against Apache's SHA512, and the starter JAR reported version=3.1.0. The final run completed all 16 verification steps. Its evidence is functional correctness for these test cases, not a throughput benchmark, a high-availability test, or proof of end-to-end exactly-once delivery. In particular, the configuration below explicitly sets exactly_once = false.
The Java image tag can advance. Check the runtime version in your own run; pin an image digest when you need immutable infrastructure.
1. Prepare a disposable environment
The official 3.1.0 deployment prerequisite is Java 11 or 17 with JAVA_HOME set. Docker is only the isolation mechanism used here, not a SeaTunnel requirement. No Python runtime is needed for this workflow. See the versioned deployment guide.
For this Docker-based route, use Bash on macOS or Linux, a running Docker engine, and curl, tar, unzip, shasum, sed, and openssl. Allow several gigabytes of free space. The Java runtime comes from the engine container, so a host JDK is unnecessary for this route.
Use a new directory and disposable database instances. The commands create and modify test rows, including a delete and an ALTER TABLE. Do not point the configuration at a production database. No database port is published to the host.
mkdir seatunnel-cdc-lab
cd seatunnel-cdc-lab
The lab uses root/postgres accounts for convenience. That is not the recommended production access model.
2. Define the complete job
Create the template below. The quoted heredoc preserves ${table_name} literally; it is a SeaTunnel table mapping, not a shell variable to expand.
cat > customers.conf.template <<'CONF'
env {
job.mode = "STREAMING"
job.name = "customers-cdc"
parallelism = 1
checkpoint.interval = 10000
checkpoint.timeout = 60000
}
source {
MySQL-CDC {
base-url = "jdbc:mysql://st310-tutorial-mysql:3306/integration"
username = "root"
password = "<MYSQL_PASSWORD>"
table-names = ["integration.customers"]
server-id = "6530-6533"
startup.mode = "initial"
schema-changes.enabled = true
exactly_once = false
}
}
sink {
jdbc {
url = "jdbc:postgresql://st310-tutorial-pg:5432/integration"
driver = "org.postgresql.Driver"
user = "postgres"
password = "<POSTGRES_PASSWORD>"
generate_sink_sql = true
database = "integration"
table = "public.${table_name}"
schema_save_mode = "CREATE_SCHEMA_WHEN_NOT_EXIST"
data_save_mode = "APPEND_DATA"
}
}
CONF
A few choices are important:
-
startup.mode = "initial"loads the existing table, then continues with binlog changes. -
integration.customersis selected explicitly. The primary key will give the target a stable row identity for the update and delete checks. -
server-id = "6530-6533"is the CDC reader range for this lab. It must not collide with another replication client. -
schema-changes.enabled = trueenables the later schema-change exercise. The test below establishes ADD COLUMN behavior only. - The environment's checkpoint values are in milliseconds: 10,000 for the interval and 60,000 for the timeout. These are lab settings, not a latency SLA or production tuning recommendation.
The MySQL CDC reference for 3.1.0 is the reference for source prerequisites and options.
On the sink side, generate_sink_sql = true uses the upstream schema and row operation to generate writes. APPEND_DATA is a save-mode setting; it does not turn every CDC update into an extra inserted row. This run verifies that distinction with queries. The configured table mapping resolves to public.customers.
The template retains the tested user field for the JDBC sink; it may produce a deprecation warning recommending username. Review the 3.1.0 JDBC sink reference before changing a working configuration. For Zeta, driver JARs belong in the engine's lib/ on every node and must be available when the processes start.
3. Keep passwords out of database-client command lines
The following setup script downloads the release and matching connectors, checks the archive, generates random credentials, and starts the three isolated containers. It is a lab convenience script, not an Apache-provided installer. Its engine entry point is the native bin/seatunnel-cluster.sh.
The MySQL CLI gets its password from a private option file. This keeps credentials out of the SQL command line. Both that file and the runnable SeaTunnel job configuration contain secrets: do not print, commit, or distribute them. This local setup is not a secret-management system: privileged Docker or host users can inspect container environment variables, and the setup expands credentials while generating the configuration. Use managed secret delivery for production.
The script refuses to reuse existing container names or a pre-existing lab network. If those resources already exist, inspect them rather than deleting them automatically. On a machine where this tutorial has already been run, use a separate environment or consistently change the lab names, network, and configuration hostnames.
cat > setup.sh <<'BASH'
#!/usr/bin/env bash
set -euo pipefail
cd "$(dirname "$0")"
for command in docker curl tar shasum openssl unzip sed; do command -v "$command" >/dev/null || { echo "Missing prerequisite: $command" >&2; exit 1; }; done
for name in st310-tutorial-engine st310-tutorial-mysql st310-tutorial-pg; do
if docker container inspect "$name" >/dev/null 2>&1; then echo "Refusing to reuse existing container: $name" >&2; exit 1; fi
done
if docker network inspect st310-tutorial-net >/dev/null 2>&1; then echo 'Refusing to reuse existing tutorial network' >&2; exit 1; fi
if [ -e .env ] || [ -e runtime ]; then echo 'Use a fresh directory; .env/runtime already exists' >&2; exit 1; fi
if [ -n "${LAB_SUBNET:-}" ]; then
docker network create --subnet "$LAB_SUBNET" st310-tutorial-net
else
docker network create st310-tutorial-net
fi
umask 077
mkdir runtime
BASE=https://downloads.apache.org/seatunnel/3.1.0
curl -fL --retry 3 "$BASE/apache-seatunnel-3.1.0-bin.tar.gz" -o runtime/apache-seatunnel-3.1.0-bin.tar.gz
curl -fL --retry 3 "$BASE/apache-seatunnel-3.1.0-bin.tar.gz.sha512" -o runtime/apache-seatunnel-3.1.0-bin.tar.gz.sha512
(
cd runtime
shasum -a 512 -c apache-seatunnel-3.1.0-bin.tar.gz.sha512
printf '%s %s\n' \
'ca855b7934c375146cdaec9ff0f86e030350210b1391c4fbebea10d472008046b4ca7d05bc4decd5fd2e5f8e57bcde22de79f26ad21bf2d00efbc018c5b613d8' \
'apache-seatunnel-3.1.0-bin.tar.gz' | shasum -a 512 -c -
)
tar -xzf runtime/apache-seatunnel-3.1.0-bin.tar.gz -C runtime
ST="$PWD/runtime/apache-seatunnel-3.1.0"
MAVEN=https://repo.maven.apache.org/maven2
for artifact in connector-cdc-mysql connector-jdbc; do
curl -fL --retry 3 "$MAVEN/org/apache/seatunnel/$artifact/3.1.0/$artifact-3.1.0.jar" -o "$ST/connectors/$artifact-3.1.0.jar"
done
curl -fL --retry 3 "$MAVEN/com/mysql/mysql-connector-j/8.4.0/mysql-connector-j-8.4.0.jar" -o "$ST/lib/mysql-connector-j-8.4.0.jar"
curl -fL --retry 3 "$MAVEN/org/postgresql/postgresql/42.7.8/postgresql-42.7.8.jar" -o "$ST/lib/postgresql-42.7.8.jar"
export MYSQL_ROOT_PASSWORD="$(openssl rand -hex 24)"
export POSTGRES_PASSWORD="$(openssl rand -hex 24)"
printf 'MYSQL_ROOT_PASSWORD=%s\nPOSTGRES_PASSWORD=%s\n' "$MYSQL_ROOT_PASSWORD" "$POSTGRES_PASSWORD" > .env
# Passwords are random hexadecimal strings, so they contain no sed/HOCON metacharacters.
sed -e "s/<MYSQL_PASSWORD>/$MYSQL_ROOT_PASSWORD/g" \
-e "s/<POSTGRES_PASSWORD>/$POSTGRES_PASSWORD/g" \
customers.conf.template > runtime/customers.conf
chmod 600 runtime/customers.conf
# Private native MySQL client option file. Never print or distribute it.
printf '[client]\nuser=root\npassword=%s\n' "$MYSQL_ROOT_PASSWORD" > runtime/mysql-client.cnf
chmod 600 runtime/mysql-client.cnf
docker run -d --name st310-tutorial-mysql --network st310-tutorial-net \
-v "$PWD/runtime/mysql-client.cnf:/tmp/tutorial-client.cnf:ro" \
-e MYSQL_ROOT_PASSWORD -e MYSQL_DATABASE=integration mysql:8.0.46 \
--server-id=3101 --log-bin=mysql-bin --binlog-format=ROW --binlog-row-image=FULL \
--gtid-mode=ON --enforce-gtid-consistency=ON
docker run -d --name st310-tutorial-pg --network st310-tutorial-net \
-e POSTGRES_PASSWORD -e POSTGRES_DB=integration postgres:16.6-alpine
docker run -d --name st310-tutorial-engine --network st310-tutorial-net \
-v "$ST:/opt/seatunnel" -v "$PWD/runtime/customers.conf:/opt/seatunnel/config/customers.conf:ro" \
-w /opt/seatunnel eclipse-temurin:11-jdk bash bin/seatunnel-cluster.sh
printf '\nContainers started. Continue with section 4 of this walkthrough after the databases are ready.\n'
BASH
bash setup.sh
Keep the supplied checksum check. If the release has moved off the downloads mirror, use the corresponding https://archive.apache.org/dist/seatunnel/3.1.0/ location and still verify the archive. A failed download or checksum is a stop condition, not a reason to skip validation.
If Docker has exhausted its predefined address pools, inspect the existing networks and host routes. Choose an unused private CIDR and use LAB_SUBNET=YOUR_UNUSED_CIDR bash setup.sh in a fresh setup. Do not run a global prune to make room. A partially completed setup is not safe to rerun blindly: inspect the resources it created and resume the appropriate command.
Wait for database initialization before continuing:
docker logs --tail 30 st310-tutorial-mysql
docker exec st310-tutorial-pg pg_isready -U postgres -d integration
docker logs --tail 30 st310-tutorial-engine
MySQL should report ready for connections; PostgreSQL should accept connections. Engine startup alone does not mean a job has been submitted.
4. Check versions and the MySQL binlog
docker exec st310-tutorial-engine printenv JAVA_HOME
docker exec st310-tutorial-engine java -version
unzip -p runtime/apache-seatunnel-3.1.0/starter/seatunnel-starter.jar META-INF/maven/org.apache.seatunnel/seatunnel-starter/pom.properties
docker exec st310-tutorial-mysql mysql --defaults-extra-file=/tmp/tutorial-client.cnf --table integration -e "SELECT VERSION(); SHOW VARIABLES WHERE Variable_name IN ('log_bin','binlog_format','binlog_row_image','gtid_mode');"
docker exec st310-tutorial-pg psql -U postgres -d integration -c 'SHOW server_version;'
Check version=3.1.0 in the JAR metadata. The tested MySQL result was:
| Variable | Value |
|---|---|
log_bin |
ON |
binlog_format |
ROW |
binlog_row_image |
FULL |
gtid_mode |
ON |
This lab deliberately enables GTID. Do not infer that GTID is mandatory for every MySQL 8 deployment. For an existing database, check the connector's version-specific preparation guidance and coordinate any global changes with its administrator. The CDC account also needs the appropriate snapshot and replication privileges; using root here avoids turning this small exercise into an account-provisioning tutorial.
Notice that --defaults-extra-file is the first option after mysql. There is no nested sh -c and no password on the command line.
5. Create a source table with a stable key
Run this once in the fresh integration database. A table-exists error means you should inspect your environment, not drop a table automatically.
docker exec -i st310-tutorial-mysql mysql --defaults-extra-file=/tmp/tutorial-client.cnf --table integration <<'SQL'
CREATE TABLE customers (
id BIGINT PRIMARY KEY,
name VARCHAR(100) NOT NULL,
city VARCHAR(100)
);
INSERT INTO customers VALUES
(101, 'Alice', 'Seattle'),
(102, 'Bob', 'Toronto');
SELECT * FROM customers ORDER BY id;
SQL
Expected source rows:
| id | name | city |
|---|---|---|
| 101 | Alice | Seattle |
| 102 | Bob | Toronto |
We will use these exact keys to distinguish row updates from accidental duplicate inserts and to verify deletion. No target table is created manually in this example; the configured sink creates it.
6. Submit the job, then verify the initial snapshot
docker exec st310-tutorial-engine bash bin/seatunnel.sh -c config/customers.conf -m cluster --async -n customers-cdc
Copy the numeric ID from the actual Submit job finished response. The job name is a readable label; the job ID is what subsequent lifecycle commands use.
JOB_ID=REPLACE_WITH_YOUR_NUMERIC_JOB_ID
docker exec st310-tutorial-engine bash bin/seatunnel.sh -j "$JOB_ID" -m cluster
docker exec st310-tutorial-pg psql -U postgres -d integration -c 'SELECT * FROM public.customers ORDER BY id;'
Wait until PostgreSQL contains the same two rows as MySQL. If the target table is not visible yet, initialization may still be in progress. Check the job and engine logs, then query again. Treat persistent errors as errors rather than polling indefinitely.
The acceptance condition is the target data, not just a successful asynchronous submission. Keep the streaming job running throughout the next four checks.
7. Verify INSERT, UPDATE, and DELETE separately
Run each mutation once and check both databases before proceeding. Repeating an INSERT with the same primary key will fail at the source; that is not a CDC recovery test.
INSERT: a new row after the snapshot
docker exec st310-tutorial-mysql mysql --defaults-extra-file=/tmp/tutorial-client.cnf --table integration -e "INSERT INTO customers VALUES (103, 'Carol', 'Austin'); SELECT * FROM customers ORDER BY id;"
docker exec st310-tutorial-pg psql -U postgres -d integration -c 'SELECT * FROM public.customers ORDER BY id;'
The target should now contain Alice, Bob, and Carol. Because the existing job remained running, this check exercises a change following the initial snapshot.
UPDATE: keep the key, change the value
docker exec st310-tutorial-mysql mysql --defaults-extra-file=/tmp/tutorial-client.cnf --table integration -e "UPDATE customers SET city='Vancouver' WHERE id=102; SELECT * FROM customers ORDER BY id;"
docker exec st310-tutorial-pg psql -U postgres -d integration -c 'SELECT * FROM public.customers ORDER BY id;'
Expect one row with ID 102, now in Vancouver. The condition is precise: Bob's existing key must carry the new value, rather than producing a second customer row.
DELETE: remove the disposable test row
docker exec st310-tutorial-mysql mysql --defaults-extra-file=/tmp/tutorial-client.cnf --table integration -e 'DELETE FROM customers WHERE id=101; SELECT * FROM customers ORDER BY id;'
docker exec st310-tutorial-pg psql -U postgres -d integration -c 'SELECT * FROM public.customers ORDER BY id;'
Alice should disappear from both results. Bob and Carol should remain. On a slower machine, query the target again after a short wait; do not assume zero synchronization delay from the edited demonstration.
8. Test one schema change explicitly
Add email, then update Carol's value:
docker exec -i st310-tutorial-mysql mysql --defaults-extra-file=/tmp/tutorial-client.cnf --table integration <<'SQL'
ALTER TABLE customers ADD email VARCHAR(255);
UPDATE customers SET email='carol@example.test' WHERE id=103;
SELECT * FROM customers ORDER BY id;
SQL
docker exec st310-tutorial-pg psql -U postgres -d integration -c 'SELECT * FROM public.customers ORDER BY id;'
docker exec st310-tutorial-pg psql -U postgres -d integration -c "SELECT column_name, data_type FROM information_schema.columns WHERE table_schema='public' AND table_name='customers' ORDER BY ordinal_position;"
Verify both the target structure and data: email exists, Carol has carol@example.test, and Bob's value is NULL. psql displays NULL as an empty field by default; an empty display is not evidence that an empty string was written.
This establishes the tested ADD COLUMN case. A column rename, drop, type conversion, incompatible default, or a different source/sink pair needs its own validation. Do not use one successful DDL test as a blanket schema-evolution guarantee.
9. Stop with a savepoint and restore by job ID
First confirm the target has converged. Then request a savepoint:
docker exec st310-tutorial-engine bash bin/seatunnel.sh -s "$JOB_ID" -m cluster
docker exec st310-tutorial-engine bash bin/seatunnel.sh -j "$JOB_ID" -m cluster
In this tested flow, the savepoint request completed and the job stopped. Check the command result and resulting job state before restoring. Keep the engine container and its checkpoint files intact.
The lab uses the package's file-backed checkpoint location inside the engine container, /tmp/seatunnel/checkpoint_snapshot. Container removal would discard that local state. This is deliberately a controlled job stop-and-resume exercise, not proof of recovery after losing the engine host or its storage.
Restore with the same configuration:
docker exec st310-tutorial-engine bash bin/seatunnel.sh -c config/customers.conf -r "$JOB_ID" -m cluster --async -n customers-cdc-restored
Here, -r takes the historical numeric job ID, not a checkpoint file path. The verified run retained the same ID. Read your submission response and query the returned ID rather than assuming a new identifier was allocated.
A restore response is still only an intermediate check. Write a new row after the restored job is running:
docker exec st310-tutorial-engine bash bin/seatunnel.sh -j "$JOB_ID" -m cluster
docker exec st310-tutorial-mysql mysql --defaults-extra-file=/tmp/tutorial-client.cnf --table integration -e "INSERT INTO customers VALUES (104, 'Dana', 'Boston', 'dana@example.test'); SELECT * FROM customers ORDER BY id;"
docker exec st310-tutorial-pg psql -U postgres -d integration -c 'SELECT * FROM public.customers ORDER BY id;'
The final PostgreSQL result observed in the real run was:
| id | name | city | |
|---|---|---|---|
| 102 | Bob | Vancouver | NULL |
| 103 | Carol | Austin | carol@example.test |
| 104 | Dana | Boston | dana@example.test |
The job status was RUNNING, with errorMessage null. The combination of that status and Dana's new target row provides evidence that this pipeline resumed processing. Alice remained deleted, and the preceding update and schema change remained visible.
Troubleshooting by the failed checkpoint in the walkthrough
| Observation | Start here |
|---|---|
| The engine will not start | Check Java/JAVA_HOME, the mounted release directory, and docker logs st310-tutorial-engine. A missing startup script is a filesystem/setup failure, not a connector failure. |
| MySQL connection refused | Wait for initialization; check the database container and the shared network. Docker-only hostnames will not resolve in an unrelated host process. |
| MySQL access denied | Check the private option file and its mount, then the source credentials in the job config. These are separate consumers of credentials. Do not debug by publishing passwords. |
| Driver class not found | Verify the matching JDBC driver JARs are in the Zeta runtime's lib/ before startup. Restart only the affected engine after correcting its classpath. |
| Submission succeeds but rows do not arrive | Query job status, examine the engine error, verify the selected source table, binlog settings, source privileges, and target permissions. |
| Updates/deletes do not produce the expected target state | Check stable row identity, source and target schemas, the selected JDBC write mode, and the actual row at each endpoint. |
| ADD COLUMN does not converge | Check source schema-change settings, target ALTER permissions, and errors for that exact DDL. Avoid testing another mutation until the failure is understood. |
| Restore cannot find usable state | Verify the historical job ID and that checkpoint storage still exists and is reachable. Do not remove the engine container while testing this local-state recovery path. |
| Names, network, or directory already exist | Inspect the previous lab first. The setup intentionally refuses to overwrite it. Do not delete unrelated Docker resources. |
The 10-second checkpoint interval does not imply a 10-second replication delay. For performance work, measure the source commit, source position progression, pipeline processing, and target visibility separately.
Before moving this pattern into production
Use the walkthrough as a functional acceptance checklist, then extend it for the environment you actually operate:
- Permissions and secrets. Replace root/postgres with dedicated least-privilege accounts. Validate snapshot, binlog, target DML, and required DDL privileges. Add TLS and a deliberate credential-rotation process.
- State durability and retention. Choose checkpoint storage that survives the failures you intend to tolerate. Align MySQL binlog retention with the longest credible outage and recovery window.
- Delivery semantics. Define the contract for retries, duplicate events, source positions, and sink commits. This configuration does not establish end-to-end exactly-once behavior; checkpointing alone is insufficient evidence.
- Schema and type coverage. Test the actual DDL and types your application uses, including precision, timestamps/time zones, NULL behavior, and key changes. This article exercises a deliberately small schema.
- Failure coverage. Test process crashes, network interruptions, database restarts, checkpoint failures, and recovery from durable state. A one-node cluster and a controlled savepoint restore do not prove high availability.
- Capacity and observability. Measure snapshot load, incremental lag, source and sink throughput, checkpoint duration/failures, resource use, and target correctness under realistic traffic. Increase parallelism based on evidence, not on this lab's value of one.
For a non-Docker deployment, install Java 11 or 17, use the same official package and connector/driver layout, and replace Docker DNS names with addresses reachable from the engine. Start bash bin/seatunnel-cluster.sh, then run the same bin/seatunnel.sh job commands from a second terminal. The integration design does not depend on containerization.
Stop the lab without deleting its evidence
Once the final checks are complete, stop only the three containers created for this lab:
docker stop st310-tutorial-engine st310-tutorial-mysql st310-tutorial-pg
This stops the processes without removing the containers. Keep the working directory, database volumes, and engine checkpoint state if you need to inspect or resume the experiment. Removing containers, volumes, or the working directory is a separate destructive operation; no global cleanup command is needed here.
What to take away
The practical value of this exercise is a repeatable verification loop: change one thing in MySQL, inspect PostgreSQL, and make the expected result explicit. Run that loop for the initial snapshot, each mutation type, the schema changes you rely on, and a new change after recovery.
For this SeaTunnel 3.1.0 configuration, the final result is concrete: Bob was updated, Alice was deleted, Carol's new column was populated, and Dana arrived after savepoint restoration. That is a useful baseline for a more demanding production test plan.
Version-pinned references
- Apache SeaTunnel 3.1.0 deployment
- Apache SeaTunnel 3.1.0 MySQL CDC source
- Apache SeaTunnel 3.1.0 JDBC sink
Video walkthrough: Watch the 9:43 hands-on demo.
Top comments (0)