DEV Community

Cover image for Syncing Data from Amazon DynamoDB to Redis with Apache SeaTunnel
Redis profile image Ricardo Ferreira
Ricardo Ferreira for Redis

Posted on

Syncing Data from Amazon DynamoDB to Redis with Apache SeaTunnel

#aws #dynamodb #redis #seatunnel

Let's say you have customer data stored in Amazon DynamoDB, which serves as the single source of truth for your application. However, now you need that same data in Redis for lightning-fast caching, real-time analytics, or perhaps to power a recommendation engine with vector search capabilities. As an example, consider this table named mySourceTable stored at DynamoDB, which currently contains two items.

DynamoDB Source Table

The challenge? How do you keep them in sync without writing a massive pile of code?

Personally, my ideal go-to solution is to use Redis Data Integration (RDI), as I have demonstrated and explained the why in this other post. Unfortunately, at the time of writing this blog, RDI does not support Amazon DynamoDB as a source. That leaves developers with the option to write complex data integration pipelines themselves or to invest in costly (and often overpriced) ETL solutions. None of these options is pleasant.

But there is another way.

Enter Apache SeaTunnel. An open source data integration project that makes this sync as easy as writing a config file. In this blog post, I will delve into this project and explain, step by step, how to synchronize data from Amazon DynamoDB to Redis.

Apache SeaTunnel: Your Open Source Data Pipeline Buddy

Apache SeaTunnel is a multimodal, high-performance, distributed, massive data integration tool that supports both batch and streaming modes. Think of it as the universal translator for your data systems. It speaks DynamoDB, Redis, and over 100 other data systems fluently.

What makes SeaTunnel special for this use case:

  • Zero Code: Define your pipeline in simple configuration files
  • Distributed: Scale syncing horizontally when your data grows
  • Fault Tolerant: Built-in checkpointing ensures no data loss

You can use SeaTunnel with two distinct configurations: standalone mode for one-off data movement, or cluster mode, which enables a data pipeline infrastructure to be up and running, always ready to execute new jobs.

Option 1: The Quick Hit (Standalone Mode)

This option is perfect for one-time migrations, local development, or scheduled batch jobs. One command, and you're done.

Your first step is to create your pipeline configuration (jobs/my.config):

env {
  parallelism = 1
  job.mode = "BATCH"
}

source {
  Amazondynamodb {
    url = "http://dynamodb.us-east-1.amazonaws.com"
    region = "us-east-1"
    access_key_id = "YOUR_ACCESS_KEY"
    secret_access_key = "YOUR_SECRET_KEY"
    table = "mySourceTable"
    schema = {
      fields {
        customerId = int
        customerName = string
        address = string
      }
    }
  }  
}

sink {
  Redis {
    host = host.docker.internal  # or your Redis host
    port = 6379
    support_custom_key = true
    key = "customer:{customerId}"
    data_type = hash
  }
}
Enter fullscreen mode Exit fullscreen mode

Run it with a single Docker command:

docker run --rm -it \
  -v $(pwd)/jobs:/config \
  apache/seatunnel:2.3.12 \
  ./bin/seatunnel.sh -m local -c /config/my.config
Enter fullscreen mode Exit fullscreen mode

This command starts a new container with a standalone instance of SeaTunnel, which executes the job described in the file my.config. You should see an output like this:

2026-01-09 16:57:50,786 INFO  [c.h.i.c.AbstractConfigLocator ] [main] - Loading configuration '/opt/seatunnel/config/seatunnel.yaml' from System property 'seatunnel.config'
2026-01-09 16:57:50,788 INFO  [c.h.i.c.AbstractConfigLocator ] [main] - Using configuration file at /opt/seatunnel/config/seatunnel.yaml
2026-01-09 16:57:50,789 INFO  [o.a.s.e.c.c.SeaTunnelConfig   ] [main] - seatunnel.home is /opt/seatunnel
2026-01-09 16:57:50,830 INFO  [amlSeaTunnelDomConfigProcessor] [main] - Dynamic slot is enabled, the schedule strategy is set to REJECT
2026-01-09 16:57:50,830 INFO  [c.h.i.c.AbstractConfigLocator ] [main] - Loading configuration '/opt/seatunnel/config/hazelcast.yaml' from System property 'hazelcast.config'
2026-01-09 16:57:50,830 INFO  [c.h.i.c.AbstractConfigLocator ] [main] - Using configuration file at /opt/seatunnel/config/hazelcast.yaml
2026-01-09 16:57:50,962 INFO  [c.h.i.c.AbstractConfigLocator ] [main] - Loading configuration '/opt/seatunnel/config/hazelcast-client.yaml' from System property 'hazelcast.client.config'
2026-01-09 16:57:50,962 INFO  [c.h.i.c.AbstractConfigLocator ] [main] - Using configuration file at /opt/seatunnel/config/hazelcast-client.yaml
2026-01-09 16:57:50,985 WARN  [c.h.i.AddressPicker           ] [main] - [LOCAL] [seatunnel-420804] [5.1] You configured your member address as host name. Please be aware of that your dns can be spoofed. Make sure that your dns configurations are correct.
2026-01-09 16:57:50,985 INFO  [c.h.i.AddressPicker           ] [main] - [LOCAL] [seatunnel-420804] [5.1] Resolving domain name 'localhost' to address(es): [127.0.0.1, 0:0:0:0:0:0:0:1]
2026-01-09 16:57:50,986 INFO  [c.h.i.AddressPicker           ] [main] - [LOCAL] [seatunnel-420804] [5.1] Interfaces is disabled, trying to pick one address from TCP-IP config addresses: [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1]
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
2026-01-09 16:57:51,189 INFO  [o.a.s.e.s.SeaTunnelServer     ] [main] - SeaTunnel server start...
2026-01-09 16:57:51,191 INFO  [c.h.system                    ] [main] - [localhost]:5801 [seatunnel-420804] [5.1] Based on Hazelcast IMDG version: 5.1.0 (20220228 - 21f20e7)
2026-01-09 16:57:51,191 INFO  [c.h.system                    ] [main] - [localhost]:5801 [seatunnel-420804] [5.1] Cluster name: seatunnel-420804
2026-01-09 16:57:51,191 INFO  [c.h.system                    ] [main] - [localhost]:5801 [seatunnel-420804] [5.1] 

 _____               _____                             _ 
/  ___|             |_   _|                           | |
\ `--.   ___   __ _   | |   _   _  _ __   _ __    ___ | |
 `--. \ / _ \ / _` |  | |  | | | || '_ \ | '_ \  / _ \| |
/\__/ /|  __/| (_| |  | |  | |_| || | | || | | ||  __/| |
\____/  \___| \__,_|  \_/   \__,_||_| |_||_| |_| \___||_|


2026-01-09 16:57:51,191 INFO  [c.h.system                    ] [main] - [localhost]:5801 [seatunnel-420804] [5.1] Copyright © 2021-2024 The Apache Software Foundation. Apache SeaTunnel, SeaTunnel, and its feather logo are trademarks of The Apache Software Foundation.
2026-01-09 16:57:51,191 INFO  [c.h.system                    ] [main] - [localhost]:5801 [seatunnel-420804] [5.1] Integrity Checker is disabled. Fail-fast on corrupted executables will not be performed.
To enable integrity checker do one of the following: 
  - Change member config using Java API: config.setIntegrityCheckerEnabled(true);
  - Change XML/YAML configuration property: Set hazelcast.integrity-checker.enabled to true
  - Add system property: -Dhz.integritychecker.enabled=true (for Hazelcast embedded, works only when loading config via Config.load)
  - Add environment variable: HZ_INTEGRITYCHECKER_ENABLED=true (recommended when running container image. For Hazelcast embedded, works only when loading config via Config.load)
2026-01-09 16:57:51,193 INFO  [c.h.system                    ] [main] - [localhost]:5801 [seatunnel-420804] [5.1] The Jet engine is disabled.
To enable the Jet engine on the members, do one of the following:
  - Change member config using Java API: config.getJetConfig().setEnabled(true)
  - Change XML/YAML configuration property: Set hazelcast.jet.enabled to true
  - Add system property: -Dhz.jet.enabled=true (for Hazelcast embedded, works only when loading config via Config.load)
  - Add environment variable: HZ_JET_ENABLED=true (recommended when running container image. For Hazelcast embedded, works only when loading config via Config.load)
2026-01-09 16:57:51,331 INFO  [c.h.s.security                ] [main] - [localhost]:5801 [seatunnel-420804] [5.1] Enable DEBUG/FINE log level for log category com.hazelcast.system.security  or use -Dhazelcast.security.recommendations system property to see 🔒 security recommendations and the status of current config.
2026-01-09 16:57:51,392 INFO  [o.a.s.e.s.SeaTunnelNodeContext] [main] - Using LiteNodeDropOutTcpIpJoiner TCP/IP discovery
2026-01-09 16:57:51,393 WARN  [c.h.c.CPSubsystem             ] [main] - [localhost]:5801 [seatunnel-420804] [5.1] CP Subsystem is not enabled. CP data structures will operate in UNSAFE mode! Please note that UNSAFE mode will not provide strong consistency guarantees.
2026-01-09 16:57:51,462 INFO  [.c.c.DefaultClassLoaderService] [main] - start classloader service with cache mode
2026-01-09 16:57:51,464 INFO  [c.h.i.c.AbstractConfigLocator ] [main] - Loading configuration '/opt/seatunnel/config/seatunnel.yaml' from System property 'seatunnel.config'
2026-01-09 16:57:51,464 INFO  [c.h.i.c.AbstractConfigLocator ] [main] - Using configuration file at /opt/seatunnel/config/seatunnel.yaml
2026-01-09 16:57:51,466 INFO  [amlSeaTunnelDomConfigProcessor] [main] - Dynamic slot is enabled, the schedule strategy is set to REJECT
2026-01-09 16:57:51,466 INFO  [c.h.i.c.AbstractConfigLocator ] [main] - Loading configuration '/opt/seatunnel/config/hazelcast.yaml' from System property 'hazelcast.config'
2026-01-09 16:57:51,466 INFO  [c.h.i.c.AbstractConfigLocator ] [main] - Using configuration file at /opt/seatunnel/config/hazelcast.yaml
2026-01-09 16:57:51,471 WARN  [o.a.s.e.s.TaskExecutionService] [pool-3-thread-1] - [localhost]:5801 [seatunnel-420804] [5.1] The Node is not ready yet, Node state STARTING,looking forward to the next scheduling
2026-01-09 16:57:51,471 INFO  [o.a.s.e.s.TaskExecutionService] [main] - [localhost]:5801 [seatunnel-420804] [5.1] Created new BusWork : 1257526338
2026-01-09 16:57:51,474 WARN  [a.s.e.s.s.s.DefaultSlotService] [hz.main.seaTunnel.slotService.thread] - failed send heartbeat to resource manager, will retry later. this address: [localhost]:5801
2026-01-09 16:57:51,475 INFO  [o.a.s.e.s.CoordinatorService  ] [main] - [localhost]:5801 [seatunnel-420804] [5.1] Start pending job schedule thread
2026-01-09 16:57:51,554 WARN  [o.a.h.u.NativeCodeLoader      ] [main] - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2026-01-09 16:57:51,607 INFO  [o.a.s.e.s.CoordinatorService  ] [pool-7-thread-1] - [localhost]:5801 [seatunnel-420804] [5.1] 
***********************************************
     CoordinatorService Thread Pool Status
***********************************************
activeCount               :                   1
corePoolSize              :                  10
maximumPoolSize           :          2147483647
poolSize                  :                   1
completedTaskCount        :                   0
taskCount                 :                   1
***********************************************

2026-01-09 16:57:51,612 INFO  [o.a.s.e.s.JettyService        ] [main] - SeaTunnel REST service will start on port 8080
2026-01-09 16:57:51,617 INFO  [o.a.s.s.o.e.j.u.log           ] [main] - Logging initialized @1069ms to org.apache.seatunnel.shade.org.eclipse.jetty.util.log.Slf4jLog
2026-01-09 16:57:51,643 WARN  [a.s.s.o.e.j.s.h.ContextHandler] [main] - Empty contextPath
2026-01-09 16:57:51,652 INFO  [o.a.s.s.o.e.j.s.Server        ] [main] - jetty-9.4.56.v20240826; built: 2024-08-26T17:15:05.868Z; git: ec6782ff5ead824dabdcf47fa98f90a4aedff401; jvm 1.8.0_342-b07
2026-01-09 16:57:51,665 INFO  [o.a.s.s.o.e.j.s.session       ] [main] - DefaultSessionIdManager workerName=node0
2026-01-09 16:57:51,665 INFO  [o.a.s.s.o.e.j.s.session       ] [main] - No SessionScavenger set, using defaults
2026-01-09 16:57:51,666 INFO  [o.a.s.s.o.e.j.s.session       ] [main] - node0 Scavenging every 660000ms
2026-01-09 16:57:51,727 INFO  [a.s.s.o.e.j.s.h.ContextHandler] [main] - Started o.a.s.s.o.e.j.s.ServletContextHandler@7069f076{/,null,AVAILABLE}
2026-01-09 16:57:51,731 INFO  [.s.s.o.e.j.s.AbstractConnector] [main] - Started ServerConnector@6e31d989{HTTP/1.1, (http/1.1)}{0.0.0.0:8080}
2026-01-09 16:57:51,732 INFO  [o.a.s.s.o.e.j.s.Server        ] [main] - Started @1184ms
2026-01-09 16:57:51,747 INFO  [c.h.i.d.Diagnostics           ] [main] - [localhost]:5801 [seatunnel-420804] [5.1] Diagnostics disabled. To enable add -Dhazelcast.diagnostics.enabled=true to the JVM arguments.
2026-01-09 16:57:51,749 INFO  [c.h.c.LifecycleService        ] [main] - [localhost]:5801 [seatunnel-420804] [5.1] [localhost]:5801 is STARTING
Members {size:1, ver:1} [
        Member [localhost]:5801 - 579e9374-7d47-4d91-80d5-c7a9155d42d0 [master node] [active master] this
]

2026-01-09 16:57:52,775 INFO  [o.a.s.e.s.CoordinatorService  ] [pool-5-thread-1] - [localhost]:5801 [seatunnel-420804] [5.1] This node become a new active master node, begin init coordinator service
2026-01-09 16:57:52,779 INFO  [c.h.c.LifecycleService        ] [main] - [localhost]:5801 [seatunnel-420804] [5.1] [localhost]:5801 is STARTED
2026-01-09 16:57:52,797 INFO  [o.a.s.e.s.CoordinatorService  ] [pool-5-thread-1] - [localhost]:5801 [seatunnel-420804] [5.1] Loaded event handlers: [org.apache.seatunnel.api.event.LoggingEventHandler@1ebbcf65]
2026-01-09 16:57:52,803 INFO  [.c.i.s.ClientInvocationService] [main] - hz.client_1 [seatunnel-420804] [5.1] Running with 2 response threads, dynamic=true
2026-01-09 16:57:52,807 INFO  [.h.i.p.i.PartitionStateManager] [seatunnel-coordinator-service-1] - [localhost]:5801 [seatunnel-420804] [5.1] Initializing cluster partition table arrangement...
2026-01-09 16:57:52,811 INFO  [c.h.c.LifecycleService        ] [main] - hz.client_1 [seatunnel-420804] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is STARTING
2026-01-09 16:57:52,811 INFO  [c.h.c.LifecycleService        ] [main] - hz.client_1 [seatunnel-420804] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is STARTED
2026-01-09 16:57:52,814 INFO  [.c.i.c.ClientConnectionManager] [main] - hz.client_1 [seatunnel-420804] [5.1] Trying to connect to cluster: seatunnel-420804
2026-01-09 16:57:52,815 INFO  [.c.i.c.ClientConnectionManager] [main] - hz.client_1 [seatunnel-420804] [5.1] Trying to connect to [localhost]:5801
2026-01-09 16:57:52,823 INFO  [.p.t.AuthenticationMessageTask] [hz.main.priority-generic-operation.thread-0] - [localhost]:5801 [seatunnel-420804] [5.1] Received auth from Connection[id=1, /127.0.0.1:5801->/127.0.0.1:45255, qualifier=null, endpoint=[127.0.0.1]:45255, remoteUuid=3be41dca-f0e7-4603-ad27-4e5b3aecbb4a, alive=true, connectionType=JVM, planeIndex=-1], successfully authenticated, clientUuid: 3be41dca-f0e7-4603-ad27-4e5b3aecbb4a, client name: hz.client_1, client version: 5.1
2026-01-09 16:57:52,824 INFO  [c.h.c.LifecycleService        ] [main] - hz.client_1 [seatunnel-420804] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is CLIENT_CONNECTED
2026-01-09 16:57:52,824 INFO  [.c.i.c.ClientConnectionManager] [main] - hz.client_1 [seatunnel-420804] [5.1] Authenticated with server [localhost]:5801:579e9374-7d47-4d91-80d5-c7a9155d42d0, server version: 5.1, local address: /127.0.0.1:45255
2026-01-09 16:57:52,825 INFO  [c.h.i.d.Diagnostics           ] [main] - hz.client_1 [seatunnel-420804] [5.1] Diagnostics disabled. To enable add -Dhazelcast.diagnostics.enabled=true to the JVM arguments.
2026-01-09 16:57:52,829 INFO  [c.h.c.i.s.ClientClusterService] [hz.client_1.event-10] - hz.client_1 [seatunnel-420804] [5.1] 

Members [1] {
        Member [localhost]:5801 - 579e9374-7d47-4d91-80d5-c7a9155d42d0 [master node]
}

2026-01-09 16:57:52,841 INFO  [.c.i.s.ClientStatisticsService] [main] - Client statistics is enabled with period 5 seconds.
2026-01-09 16:57:52,858 INFO  [o.a.s.c.s.u.ConfigBuilder     ] [main] - Loading config file from path: /config/my.config
2026-01-09 16:57:52,951 INFO  [o.a.s.c.s.u.ConfigShadeUtils  ] [main] - Load config shade spi: [base64]
2026-01-09 16:57:52,967 INFO  [o.a.s.c.s.u.ConfigBuilder     ] [main] - Parsed config file: 
{
    "env" : {
        "parallelism" : 1,
        "job.mode" : "BATCH"
    },
    "source" : [
        {
            "url" : "http://dynamodb.us-east-1.amazonaws.com",
            "region" : "us-east-1",
            "access_key_id" : "<REDACTED>",
            "secret_access_key" : "<REDACTED>",
            "table" : "mySourceTable",
            "schema" : {
                "fields" : {
                    "customerId" : "int",
                    "customerName" : "string",
                    "address" : "string"
                }
            },
            "plugin_name" : "Amazondynamodb"
        }
    ],
    "sink" : [
        {
            "host" : "host.docker.internal",
            "port" : 6379,
            "support_custom_key" : true,
            "key" : "customer:{customerId}",
            "data_type" : "hash",
            "plugin_name" : "Redis"
        }
    ]
}

2026-01-09 16:57:52,971 INFO  [p.MultipleTableJobConfigParser] [main] - add common jar in plugins :[]
2026-01-09 16:57:52,978 INFO  [.s.p.d.AbstractPluginDiscovery] [main] - Load SeaTunnelSink Plugin from /opt/seatunnel/connectors
2026-01-09 16:57:52,980 INFO  [.s.p.d.AbstractPluginDiscovery] [main] - Discovery plugin jar for: PluginIdentifier{engineType='seatunnel', pluginType='source', pluginName='Amazondynamodb'} at: [file:/opt/seatunnel/connectors/connector-amazondynamodb-2.3.12.jar]
2026-01-09 16:57:52,980 INFO  [.s.p.d.AbstractPluginDiscovery] [main] - find connector jar and dependency for PluginIdentifier{engineType='seatunnel', pluginType='source', pluginName='Amazondynamodb'}: [file:/opt/seatunnel/connectors/connector-amazondynamodb-2.3.12.jar]
2026-01-09 16:57:52,982 INFO  [.s.p.d.AbstractPluginDiscovery] [main] - Load SeaTunnelSink Plugin from /opt/seatunnel/connectors
2026-01-09 16:57:52,985 INFO  [.s.p.d.AbstractPluginDiscovery] [main] - Load SeaTunnelSink Plugin from /opt/seatunnel/connectors
2026-01-09 16:57:52,986 INFO  [.s.p.d.AbstractPluginDiscovery] [main] - Discovery plugin jar for: PluginIdentifier{engineType='seatunnel', pluginType='sink', pluginName='Redis'} at: [file:/opt/seatunnel/connectors/connector-redis-2.3.12.jar]
2026-01-09 16:57:52,986 INFO  [.s.p.d.AbstractPluginDiscovery] [main] - find connector jar and dependency for PluginIdentifier{engineType='seatunnel', pluginType='sink', pluginName='Redis'}: [file:/opt/seatunnel/connectors/connector-redis-2.3.12.jar]
2026-01-09 16:57:52,988 INFO  [p.MultipleTableJobConfigParser] [main] - start generating all sources.
2026-01-09 16:57:53,003 INFO  [o.a.s.a.t.f.FactoryUtil       ] [main] - get the CatalogTable from source AmazonDynamodb: .default.default.default
2026-01-09 16:57:53,008 INFO  [.s.p.d.AbstractPluginDiscovery] [main] - Load SeaTunnelSource Plugin from /opt/seatunnel/connectors
2026-01-09 16:57:53,011 INFO  [.s.p.d.AbstractPluginDiscovery] [main] - Discovery plugin jar for: PluginIdentifier{engineType='seatunnel', pluginType='source', pluginName='Amazondynamodb'} at: [file:/opt/seatunnel/connectors/connector-amazondynamodb-2.3.12.jar]
2026-01-09 16:57:53,011 INFO  [.s.p.d.AbstractPluginDiscovery] [main] - find connector jar and dependency for PluginIdentifier{engineType='seatunnel', pluginType='source', pluginName='Amazondynamodb'}: [file:/opt/seatunnel/connectors/connector-amazondynamodb-2.3.12.jar]
2026-01-09 16:57:53,012 INFO  [p.MultipleTableJobConfigParser] [main] - start generating all transforms.
2026-01-09 16:57:53,012 INFO  [p.MultipleTableJobConfigParser] [main] - start generating all sinks.
2026-01-09 16:57:53,013 INFO  [.s.p.d.AbstractPluginDiscovery] [main] - Load SeaTunnelSink Plugin from /opt/seatunnel/connectors
2026-01-09 16:57:53,014 INFO  [.s.p.d.AbstractPluginDiscovery] [main] - Discovery plugin jar for: PluginIdentifier{engineType='seatunnel', pluginType='sink', pluginName='Redis'} at: [file:/opt/seatunnel/connectors/connector-redis-2.3.12.jar]
2026-01-09 16:57:53,014 INFO  [.s.p.d.AbstractPluginDiscovery] [main] - find connector jar and dependency for PluginIdentifier{engineType='seatunnel', pluginType='sink', pluginName='Redis'}: [file:/opt/seatunnel/connectors/connector-redis-2.3.12.jar]
2026-01-09 16:57:53,024 INFO  [o.a.s.a.t.f.FactoryUtil       ] [main] - Create sink 'Redis' with upstream input catalog-table[database: default, schema: default, table: default]
2026-01-09 16:57:53,042 INFO  [o.a.s.e.c.j.ClientJobProxy    ] [main] - Start submit job, job id: 1062052604328017921, with plugin jar [file:/opt/seatunnel/connectors/connector-amazondynamodb-2.3.12.jar, file:/opt/seatunnel/connectors/connector-redis-2.3.12.jar]
2026-01-09 16:57:53,047 INFO  [.e.s.r.AbstractResourceManager] [hz.main.client.thread-4] - Init ResourceManager
2026-01-09 16:57:53,047 INFO  [.e.s.r.AbstractResourceManager] [hz.main.client.thread-4] - initWorker... 
2026-01-09 16:57:53,047 INFO  [.e.s.r.AbstractResourceManager] [hz.main.client.thread-4] - init live nodes: [[localhost]:5801]
2026-01-09 16:57:53,048 INFO  [.e.s.r.AbstractResourceManager] [SeaTunnel-CompletableFuture-Thread-2] - received new worker register: [localhost]:5801
2026-01-09 16:57:53,152 INFO  [o.a.s.c.s.r.c.RedisParameters ] [hz.main.seaTunnel.task.thread-4] - Try to get redis version information from the jedis.info() method
2026-01-09 16:57:53,152 INFO  [o.a.s.c.s.r.c.RedisParameters ] [hz.main.seaTunnel.task.thread-4] - The version of Redis is :8.4.0
2026-01-09 16:57:53,154 INFO  [a.s.a.s.m.MultiTableSinkWriter] [hz.main.seaTunnel.task.thread-4] - init multi table sink writer, queue size: 1
2026-01-09 16:57:53,233 INFO  [o.a.s.a.e.LoggingEventHandler ] [hz.main.generic-operation.thread-12] - log event: ReaderOpenEvent(createdTime=1767977873233, jobId=1062052604328017921, eventType=LIFECYCLE_READER_OPEN)
2026-01-09 16:57:53,402 INFO  [.s.t.SourceSplitEnumeratorTask] [hz.main.seaTunnel.task.thread-4] - received reader register, readerID: TaskLocation{taskGroupLocation=TaskGroupLocation{jobId=1062052604328017921, pipelineId=1, taskGroupId=2}, taskID=1000200000000, index=0}
2026-01-09 16:57:53,429 INFO  [.s.e.s.c.CheckpointCoordinator] [seatunnel-coordinator-service-9] - checkpoint is disabled, because in batch mode and 'checkpoint.interval' of env is missing.
2026-01-09 16:57:53,506 INFO  [a.s.AmazonDynamoDBSourceReader] [BlockingWorker-TaskGroupLocation{jobId=1062052604328017921, pipelineId=1, taskGroupId=2}] - AmazonDynamoDB Source Reader [0] waiting for splits
2026-01-09 16:57:53,506 INFO  [a.s.AmazonDynamoDBSourceReader] [BlockingWorker-TaskGroupLocation{jobId=1062052604328017921, pipelineId=1, taskGroupId=2}] - AmazonDynamoDB Source Reader [0] waiting for splits
2026-01-09 16:57:53,529 INFO  [.s.t.SourceSplitEnumeratorTask] [BlockingWorker-TaskGroupLocation{jobId=1062052604328017921, pipelineId=1, taskGroupId=1}] - received enough reader, starting enumerator...
2026-01-09 16:57:53,530 INFO  [nDynamoDBSourceSplitEnumerator] [BlockingWorker-TaskGroupLocation{jobId=1062052604328017921, pipelineId=1, taskGroupId=1}] - Assigning org.apache.seatunnel.connectors.seatunnel.amazondynamodb.source.AmazonDynamoDBSourceSplit@36fe5b1d to 0 reader.
2026-01-09 16:57:53,530 INFO  [nDynamoDBSourceSplitEnumerator] [BlockingWorker-TaskGroupLocation{jobId=1062052604328017921, pipelineId=1, taskGroupId=1}] - Assigning org.apache.seatunnel.connectors.seatunnel.amazondynamodb.source.AmazonDynamoDBSourceSplit@81bd425 to 0 reader.
2026-01-09 16:57:53,530 INFO  [nDynamoDBSourceSplitEnumerator] [BlockingWorker-TaskGroupLocation{jobId=1062052604328017921, pipelineId=1, taskGroupId=1}] - Assign splits [org.apache.seatunnel.connectors.seatunnel.amazondynamodb.source.AmazonDynamoDBSourceSplit@36fe5b1d, org.apache.seatunnel.connectors.seatunnel.amazondynamodb.source.AmazonDynamoDBSourceSplit@81bd425] to reader 0
2026-01-09 16:57:53,533 INFO  [a.s.AmazonDynamoDBSourceReader] [hz.main.generic-operation.thread-21] - Reader [0] received noMoreSplit event.
2026-01-09 16:57:53,934 INFO  [a.s.AmazonDynamoDBSourceReader] [BlockingWorker-TaskGroupLocation{jobId=1062052604328017921, pipelineId=1, taskGroupId=2}] - AmazonDynamoDB Source Reader [0] waiting for splits
2026-01-09 16:57:53,934 INFO  [a.s.AmazonDynamoDBSourceReader] [BlockingWorker-TaskGroupLocation{jobId=1062052604328017921, pipelineId=1, taskGroupId=2}] - Closed the bounded amazonDynamodb source
2026-01-09 16:57:53,941 INFO  [.s.e.s.c.CheckpointCoordinator] [hz.main.generic-operation.thread-23] - skip schedule trigger checkpoint because checkpoint type is COMPLETED_POINT_TYPE
2026-01-09 16:57:53,943 INFO  [.s.e.s.c.CheckpointCoordinator] [seatunnel-coordinator-service-8] - wait checkpoint completed: 9223372036854775807
2026-01-09 16:57:55,974 INFO  [.s.e.s.c.CheckpointCoordinator] [seatunnel-coordinator-service-8] - pending checkpoint(9223372036854775807/1@1062052604328017921) notify finished!
2026-01-09 16:57:55,974 INFO  [.s.e.s.c.CheckpointCoordinator] [seatunnel-coordinator-service-8] - start notify checkpoint completed, job id: 1062052604328017921, pipeline id: 1, checkpoint id:9223372036854775807
2026-01-09 16:57:55,979 INFO  [.s.e.s.c.CheckpointCoordinator] [seatunnel-coordinator-service-8] - start clean pending checkpoint cause CheckpointCoordinator completed.
2026-01-09 16:57:55,979 INFO  [.s.e.s.c.CheckpointCoordinator] [seatunnel-coordinator-service-8] - Turn checkpoint_state_1062052604328017921_1 state from null to FINISHED
2026-01-09 16:57:55,987 INFO  [o.a.s.e.s.TaskExecutionService] [BlockingWorker-TaskGroupLocation{jobId=1062052604328017921, pipelineId=1, taskGroupId=1}] - [localhost]:5801 [seatunnel-420804] [5.1] taskDone, taskId = 1000100000000, taskGroup = TaskGroupLocation{jobId=1062052604328017921, pipelineId=1, taskGroupId=1}
2026-01-09 16:57:55,987 INFO  [o.a.s.a.e.LoggingEventHandler ] [hz.main.generic-operation.thread-32] - log event: EnumeratorCloseEvent(createdTime=1767977875987, jobId=1062052604328017921, eventType=LIFECYCLE_ENUMERATOR_CLOSE)
2026-01-09 16:57:55,998 INFO  [o.a.s.e.s.TaskExecutionService] [BlockingWorker-TaskGroupLocation{jobId=1062052604328017921, pipelineId=1, taskGroupId=1}] - [localhost]:5801 [seatunnel-420804] [5.1] taskGroup TaskGroupLocation{jobId=1062052604328017921, pipelineId=1, taskGroupId=1} complete with FINISHED
2026-01-09 16:57:55,998 INFO  [o.a.s.e.s.TaskExecutionService] [hz.main.seaTunnel.task.thread-5] - [localhost]:5801 [seatunnel-420804] [5.1] Task TaskGroupLocation{jobId=1062052604328017921, pipelineId=1, taskGroupId=1} complete with state FINISHED
2026-01-09 16:57:55,998 INFO  [o.a.s.e.s.CoordinatorService  ] [hz.main.seaTunnel.task.thread-5] - [localhost]:5801 [seatunnel-420804] [5.1] Received task end from execution TaskGroupLocation{jobId=1062052604328017921, pipelineId=1, taskGroupId=1}, state FINISHED
2026-01-09 16:57:56,000 INFO  [o.a.s.e.s.d.p.PhysicalVertex  ] [hz.main.seaTunnel.task.thread-5] - Job (1062052604328017921), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-Amazondynamodb]-SplitEnumerator (1/1)], taskGroupLocation: [TaskGroupLocation{jobId=1062052604328017921, pipelineId=1, taskGroupId=1}] turned from state RUNNING to FINISHED.
2026-01-09 16:57:56,000 INFO  [o.a.s.e.s.d.p.PhysicalVertex  ] [hz.main.seaTunnel.task.thread-5] - Job (1062052604328017921), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-Amazondynamodb]-SplitEnumerator (1/1)], taskGroupLocation: [TaskGroupLocation{jobId=1062052604328017921, pipelineId=1, taskGroupId=1}] state process is stopped
2026-01-09 16:57:56,000 INFO  [o.a.s.e.s.d.p.SubPlan         ] [seatunnel-coordinator-service-8] - Job (1062052604328017921), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-Amazondynamodb]-SplitEnumerator (1/1)], taskGroupLocation: [TaskGroupLocation{jobId=1062052604328017921, pipelineId=1, taskGroupId=1}] future complete with state FINISHED
2026-01-09 16:57:56,039 INFO  [o.a.s.a.e.LoggingEventHandler ] [hz.main.generic-operation.thread-33] - log event: ReaderCloseEvent(createdTime=1767977876039, jobId=1062052604328017921, eventType=LIFECYCLE_READER_CLOSE)
2026-01-09 16:57:56,040 INFO  [o.a.s.e.s.TaskExecutionService] [BlockingWorker-TaskGroupLocation{jobId=1062052604328017921, pipelineId=1, taskGroupId=2}] - [localhost]:5801 [seatunnel-420804] [5.1] taskDone, taskId = 1000200000000, taskGroup = TaskGroupLocation{jobId=1062052604328017921, pipelineId=1, taskGroupId=2}
2026-01-09 16:57:56,076 INFO  [o.a.s.e.s.TaskExecutionService] [BlockingWorker-TaskGroupLocation{jobId=1062052604328017921, pipelineId=1, taskGroupId=2}] - [localhost]:5801 [seatunnel-420804] [5.1] taskDone, taskId = 1000200010000, taskGroup = TaskGroupLocation{jobId=1062052604328017921, pipelineId=1, taskGroupId=2}
2026-01-09 16:57:56,077 INFO  [o.a.s.a.e.LoggingEventHandler ] [hz.main.generic-operation.thread-34] - log event: WriterCloseEvent(createdTime=1767977876076, jobId=1062052604328017921, eventType=LIFECYCLE_WRITER_CLOSE)
2026-01-09 16:57:56,079 INFO  [o.a.s.e.s.TaskExecutionService] [BlockingWorker-TaskGroupLocation{jobId=1062052604328017921, pipelineId=1, taskGroupId=2}] - [localhost]:5801 [seatunnel-420804] [5.1] taskGroup TaskGroupLocation{jobId=1062052604328017921, pipelineId=1, taskGroupId=2} complete with FINISHED
2026-01-09 16:57:56,079 INFO  [o.a.s.e.s.TaskExecutionService] [hz.main.seaTunnel.task.thread-2] - [localhost]:5801 [seatunnel-420804] [5.1] Task TaskGroupLocation{jobId=1062052604328017921, pipelineId=1, taskGroupId=2} complete with state FINISHED
2026-01-09 16:57:56,079 INFO  [o.a.s.e.s.CoordinatorService  ] [hz.main.seaTunnel.task.thread-2] - [localhost]:5801 [seatunnel-420804] [5.1] Received task end from execution TaskGroupLocation{jobId=1062052604328017921, pipelineId=1, taskGroupId=2}, state FINISHED
2026-01-09 16:57:56,080 INFO  [o.a.s.e.s.d.p.PhysicalVertex  ] [hz.main.seaTunnel.task.thread-2] - Job (1062052604328017921), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-Amazondynamodb]-SourceTask (1/1)], taskGroupLocation: [TaskGroupLocation{jobId=1062052604328017921, pipelineId=1, taskGroupId=2}] turned from state RUNNING to FINISHED.
2026-01-09 16:57:56,080 INFO  [o.a.s.e.s.d.p.PhysicalVertex  ] [hz.main.seaTunnel.task.thread-2] - Job (1062052604328017921), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-Amazondynamodb]-SourceTask (1/1)], taskGroupLocation: [TaskGroupLocation{jobId=1062052604328017921, pipelineId=1, taskGroupId=2}] state process is stopped
2026-01-09 16:57:56,080 INFO  [o.a.s.e.s.d.p.SubPlan         ] [seatunnel-coordinator-service-8] - Job (1062052604328017921), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-Amazondynamodb]-SourceTask (1/1)], taskGroupLocation: [TaskGroupLocation{jobId=1062052604328017921, pipelineId=1, taskGroupId=2}] future complete with state FINISHED
2026-01-09 16:57:56,080 INFO  [o.a.s.e.s.d.p.SubPlan         ] [seatunnel-coordinator-service-8] - Job SeaTunnel_Job (1062052604328017921), Pipeline: [(1/1)] will end with state FINISHED
2026-01-09 16:57:56,081 INFO  [o.a.s.e.s.m.JobMaster         ] [hz.main.seaTunnel.task.thread-2] - release the task group resource TaskGroupLocation{jobId=1062052604328017921, pipelineId=1, taskGroupId=2}
2026-01-09 16:57:56,081 INFO  [o.a.s.e.s.d.p.SubPlan         ] [seatunnel-coordinator-service-8] - Job SeaTunnel_Job (1062052604328017921), Pipeline: [(1/1)] turned from state RUNNING to FINISHED.
2026-01-09 16:57:56,081 INFO  [a.s.e.s.s.s.DefaultSlotService] [hz.main.generic-operation.thread-35] - received slot release request, jobID: 1062052604328017921, slot: SlotProfile{worker=[localhost]:5801, slotID=2, ownerJobID=1062052604328017921, assigned=true, resourceProfile=ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=0}}, sequence='e992c180-0d29-42db-ac3f-249a1130c0b1'}
2026-01-09 16:57:56,105 INFO  [o.a.s.e.s.m.JobMaster         ] [seatunnel-coordinator-service-8] - release the pipeline Job SeaTunnel_Job (1062052604328017921), Pipeline: [(1/1)] resource
2026-01-09 16:57:56,105 INFO  [a.s.e.s.s.s.DefaultSlotService] [hz.main.generic-operation.thread-39] - received slot release request, jobID: 1062052604328017921, slot: SlotProfile{worker=[localhost]:5801, slotID=1, ownerJobID=1062052604328017921, assigned=true, resourceProfile=ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=0}}, sequence='e992c180-0d29-42db-ac3f-249a1130c0b1'}
2026-01-09 16:57:56,106 INFO  [o.a.s.e.s.d.p.SubPlan         ] [seatunnel-coordinator-service-8] - Job SeaTunnel_Job (1062052604328017921), Pipeline: [(1/1)] state process is stop
2026-01-09 16:57:56,106 INFO  [o.a.s.e.s.d.p.PhysicalPlan    ] [seatunnel-coordinator-service-7] - Job SeaTunnel_Job (1062052604328017921), Pipeline: [(1/1)] future complete with state FINISHED
2026-01-09 16:57:56,107 INFO  [o.a.s.e.s.d.p.PhysicalPlan    ] [seatunnel-coordinator-service-7] - Job SeaTunnel_Job (1062052604328017921) turned from state RUNNING to FINISHED.
2026-01-09 16:57:56,107 INFO  [o.a.s.e.s.d.p.PhysicalPlan    ] [seatunnel-coordinator-service-7] - Job SeaTunnel_Job (1062052604328017921) state process is stop
2026-01-09 16:57:56,116 INFO  [o.a.s.a.e.LoggingEventHandler ] [seatunnel-coordinator-service-7] - log event: JobStateEvent(jobId=1062052604328017921, jobName=SeaTunnel_Job, jobStatus=FINISHED, createdTime=1767977876116)
2026-01-09 16:57:56,117 INFO  [o.a.s.e.c.j.ClientJobProxy    ] [main] - Job (1062052604328017921) end with state FINISHED
2026-01-09 16:57:56,132 INFO  [s.c.s.s.c.ClientExecuteCommand] [main] - 
***********************************************
           Job Statistic Information
***********************************************
Start Time                : 2026-01-09 16:57:52
End Time                  : 2026-01-09 16:57:56
Total Time(s)             :                   3
Total Read Count          :                   2
Total Write Count         :                   2
Total Failed Count        :                   0
***********************************************

2026-01-09 16:57:56,133 INFO  [c.h.c.LifecycleService        ] [main] - hz.client_1 [seatunnel-420804] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTTING_DOWN
2026-01-09 16:57:56,135 INFO  [c.h.i.s.t.TcpServerConnection ] [hz.main.IO.thread-in-1] - [localhost]:5801 [seatunnel-420804] [5.1] Connection[id=1, /127.0.0.1:5801->/127.0.0.1:45255, qualifier=null, endpoint=[127.0.0.1]:45255, remoteUuid=3be41dca-f0e7-4603-ad27-4e5b3aecbb4a, alive=false, connectionType=JVM, planeIndex=-1] closed. Reason: Connection closed by the other side
2026-01-09 16:57:56,135 INFO  [.c.i.c.ClientConnectionManager] [main] - hz.client_1 [seatunnel-420804] [5.1] Removed connection to endpoint: [localhost]:5801:579e9374-7d47-4d91-80d5-c7a9155d42d0, connection: ClientConnection{alive=false, connectionId=1, channel=NioChannel{/127.0.0.1:45255->localhost/127.0.0.1:5801}, remoteAddress=[localhost]:5801, lastReadTime=2026-01-09 16:57:56.131, lastWriteTime=2026-01-09 16:57:56.117, closedTime=2026-01-09 16:57:56.134, connected server version=5.1}
2026-01-09 16:57:56,135 INFO  [c.h.c.LifecycleService        ] [main] - hz.client_1 [seatunnel-420804] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is CLIENT_DISCONNECTED
2026-01-09 16:57:56,136 INFO  [c.h.c.i.ClientEndpointManager ] [hz.main.event-1] - [localhost]:5801 [seatunnel-420804] [5.1] Destroying ClientEndpoint{connection=Connection[id=1, /127.0.0.1:5801->/127.0.0.1:45255, qualifier=null, endpoint=[127.0.0.1]:45255, remoteUuid=3be41dca-f0e7-4603-ad27-4e5b3aecbb4a, alive=false, connectionType=JVM, planeIndex=-1], clientUuid=3be41dca-f0e7-4603-ad27-4e5b3aecbb4a, clientName=hz.client_1, authenticated=true, clientVersion=5.1, creationTime=1767977872821, latest clientAttributes=lastStatisticsCollectionTime=1767977872841,enterprise=false,clientType=JVM,clientVersion=5.1,clusterConnectionTimestamp=1767977872816,clientAddress=127.0.0.1,clientName=hz.client_1,credentials.principal=null,os.committedVirtualMemorySize=9714790400,os.freePhysicalMemorySize=2077376512,os.freeSwapSpaceSize=1073737728,os.maxFileDescriptorCount=1048576,os.openFileDescriptorCount=103,os.processCpuTime=4280000000,os.systemLoadAverage=3.26,os.totalPhysicalMemorySize=8321540096,os.totalSwapSpaceSize=1073737728,runtime.availableProcessors=14,runtime.freeMemory=286196824,runtime.maxMemory=477626368,runtime.totalMemory=376438784,runtime.uptime=2297,runtime.usedMemory=90241960, labels=[]}
2026-01-09 16:57:56,136 INFO  [c.h.c.LifecycleService        ] [main] - hz.client_1 [seatunnel-420804] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTDOWN
2026-01-09 16:57:56,136 INFO  [s.c.s.s.c.ClientExecuteCommand] [main] - Closed SeaTunnel client......
2026-01-09 16:57:56,136 INFO  [c.h.c.LifecycleService        ] [main] - [localhost]:5801 [seatunnel-420804] [5.1] [localhost]:5801 is SHUTTING_DOWN
2026-01-09 16:57:56,138 INFO  [c.h.i.p.i.MigrationManager    ] [hz.main.cached.thread-15] - [localhost]:5801 [seatunnel-420804] [5.1] Shutdown request of Member [localhost]:5801 - 579e9374-7d47-4d91-80d5-c7a9155d42d0 [master node] [active master] this is handled
2026-01-09 16:57:56,141 INFO  [c.h.i.i.Node                  ] [main] - [localhost]:5801 [seatunnel-420804] [5.1] Shutting down connection manager...
2026-01-09 16:57:56,142 INFO  [c.h.i.i.Node                  ] [main] - [localhost]:5801 [seatunnel-420804] [5.1] Shutting down node engine...
2026-01-09 16:57:56,149 INFO  [.s.s.o.e.j.s.AbstractConnector] [main] - Stopped ServerConnector@6e31d989{HTTP/1.1, (http/1.1)}{0.0.0.0:8080}
2026-01-09 16:57:56,149 INFO  [o.a.s.s.o.e.j.s.session       ] [main] - node0 Stopped scavenging
2026-01-09 16:57:56,150 INFO  [a.s.s.o.e.j.s.h.ContextHandler] [main] - Stopped o.a.s.s.o.e.j.s.ServletContextHandler@7069f076{/,null,STOPPED}
2026-01-09 16:57:56,151 INFO  [.c.c.DefaultClassLoaderService] [main] - close classloader service
2026-01-09 16:57:56,152 INFO  [o.a.s.e.s.EventService        ] [event-forwarder-0] - Event forward thread interrupted
2026-01-09 16:57:56,155 INFO  [c.h.i.i.NodeExtension         ] [main] - [localhost]:5801 [seatunnel-420804] [5.1] Destroying node NodeExtension.
2026-01-09 16:57:56,156 INFO  [c.h.i.i.Node                  ] [main] - [localhost]:5801 [seatunnel-420804] [5.1] Hazelcast Shutdown is completed in 18 ms.
2026-01-09 16:57:56,156 INFO  [c.h.c.LifecycleService        ] [main] - [localhost]:5801 [seatunnel-420804] [5.1] [localhost]:5801 is SHUTDOWN
2026-01-09 16:57:56,156 INFO  [s.c.s.s.c.ClientExecuteCommand] [main] - Closed HazelcastInstance ......
2026-01-09 16:57:56,156 INFO  [s.c.s.s.c.ClientExecuteCommand] [main] - Closed metrics executor service ......
2026-01-09 16:57:56,156 INFO  [s.c.s.s.c.ClientExecuteCommand] [SeaTunnel-CompletableFuture-Thread-7] - run shutdown hook because get close signal
Enter fullscreen mode Exit fullscreen mode

Once the job finishes, the Docker container created to execute it will be terminated, and you will be able to see the records written on Redis.

Viewing Data with Redis Insight

As promised, this was as simple as writing a single configuration file. Of course, understanding the numerous configuration options that both the source and sink connectors offer may take some time to grasp. You can check the configuration properties for Amazon DynamoDB here and those for Redis here.

Option 2: The Production Powerhouse (Cluster Mode)

When you need an always-on data pipeline infrastructure that can handle multiple jobs, scale on demand, and provide REST API access, cluster mode is your friend.

Step 1: Set Up Your Infrastructure

Create a docker-compose.yaml that spins up a complete SeaTunnel cluster. It also includes Redis, which, for this use case, will be the sink of the data pipeline.

version: '3.8'

services:

  seatunnel-master:
    image: apache/seatunnel:2.3.12
    container_name: seatunnel-master
    environment:
      - ST_DOCKER_MEMBER_LIST=172.16.0.3,172.16.0.4,172.16.0.5
    volumes:
      - ./config/hazelcast-master.yaml:/opt/seatunnel/config/hazelcast-master.yaml
      - ./config/seatunnel.yaml:/opt/seatunnel/config/seatunnel.yaml
    entrypoint: >
      /bin/sh -c "
      /opt/seatunnel/bin/seatunnel-cluster.sh -r master
      "    
    ports:
      - "5801:5801"
      - "8080:8080"  # REST API endpoint
    networks:
      seatunnel_network:
        ipv4_address: 172.16.0.3
    healthcheck:
      test: [ "CMD-SHELL", "curl -f http://localhost:8080/system-monitoring-information || exit 1" ]
      interval: 30s
      timeout: 10s
      retries: 10
      start_period: 60s

  seatunnel-worker1:
    image: apache/seatunnel:2.3.12
    container_name: seatunnel-worker1
    environment:
      - ST_DOCKER_MEMBER_LIST=172.16.0.3,172.16.0.4,172.16.0.5
    volumes:
      - ./config/hazelcast-worker.yaml:/opt/seatunnel/config/hazelcast-worker.yaml
      - ./config/seatunnel.yaml:/opt/seatunnel/config/seatunnel.yaml
    entrypoint: >
      /bin/sh -c "
      /opt/seatunnel/bin/seatunnel-cluster.sh -r worker
      " 
    depends_on:
      - seatunnel-master
    networks:
      seatunnel_network:
        ipv4_address: 172.16.0.4
    healthcheck:
      test: ["CMD-SHELL", "pgrep -f seatunnel || exit 1"]
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 45s

  seatunnel-worker2:
    image: apache/seatunnel:2.3.12
    container_name: seatunnel-worker2
    environment:
      - ST_DOCKER_MEMBER_LIST=172.16.0.3,172.16.0.4,172.16.0.5
    volumes:
      - ./config/hazelcast-worker.yaml:/opt/seatunnel/config/hazelcast-worker.yaml
      - ./config/seatunnel.yaml:/opt/seatunnel/config/seatunnel.yaml
    entrypoint: >
      /bin/sh -c "
      /opt/seatunnel/bin/seatunnel-cluster.sh -r worker
      " 
    depends_on:
      - seatunnel-master
    networks:
      seatunnel_network:
        ipv4_address: 172.16.0.5
    healthcheck:
      test: ["CMD-SHELL", "pgrep -f seatunnel || exit 1"]
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 45s

  redis-database:
    container_name: redis-database
    hostname: redis-database
    image: redis:8.4.0
    ports:
      - "6379:6379"
    networks:
      seatunnel_network:
        ipv4_address: 172.16.0.2
    healthcheck:
      test: [ "CMD-SHELL", "redis-cli ping | grep PONG" ]
      interval: 10s
      retries: 5
      start_period: 5s
      timeout: 5s

networks:
  seatunnel_network:
    driver: bridge
    ipam:
      config:
        - subnet: 172.16.0.0/24
Enter fullscreen mode Exit fullscreen mode

Step 2: Configure your Cluster

Create config/seatunnel.yaml to enable the REST API:

seatunnel:
  engine:
    http:
      enable-http: true
      port: 8080
Enter fullscreen mode Exit fullscreen mode

Create config/hazelcast-master.yaml and config/hazelcast-worker.yaml:

hazelcast:
  network:
    rest-api:
      enabled: true
      endpoint-groups:
        CLUSTER_READ:
          enabled: true
        CLUSTER_WRITE:
          enabled: true
        HEALTH_CHECK:
          enabled: true
        DATA:
          enabled: true
    port:
      auto-increment: true
      port-count: 100
      port: 5801
    join:
      tcp-ip:
        enabled: true
        member-list:
          - 172.16.0.3
          - 172.16.0.4
          - 172.16.0.5
Enter fullscreen mode Exit fullscreen mode

Step 3: Launch Your Cluster 

docker compose up -d
Enter fullscreen mode Exit fullscreen mode

Your SeaTunnel cluster is now running with:

  • 1 Master node (coordinator + API server)
  • 2 Worker nodes (execute the actual data pipelines)
  • Redis database (your sink and target system)

Step 4: Submit Jobs via REST API

Now the magic happens! Submit jobs using a simple HTTP POST:

curl -X POST http://localhost:8080/submit-job?jobName=dynamodb-to-redis \
  -H "Content-Type: application/json" \
  -d '{
    "env": {
        "parallelism": 2,
        "job.mode": "BATCH"
    },
    "source": [
        {
            "plugin_name": "Amazondynamodb",
            "plugin_output": "dynamodb_source",
            "url": "http://dynamodb.us-east-1.amazonaws.com",
            "region": "us-east-1",
            "access_key_id": "YOUR_ACCESS_KEY",
            "secret_access_key": "YOUR_SECRET_KEY",
            "table": "mySourceTable",
            "schema": {
                "fields": {
                    "customerId": "int",
                    "customerName": "string",
                    "address": "string"
                }
            }
        }
    ],
    "transform": [],
    "sink": [
        {
            "plugin_name": "Redis",
            "host": "redis-database",
            "port": 6379,
            "support_custom_key": true,
            "key": "customer:{customerId}",
            "data_type": "hash"
        }
    ]
}'
Enter fullscreen mode Exit fullscreen mode

Once this job is accepted, it will start executing immediately. You can actually follow up with the execution of this job in the web console that SeaTunnel exposes. Go to the URL http://localhost:8080/#/overview and navigate to the Jobs tab.

SeaTunnel Web Console

Wrapping Up

With Apache SeaTunnel, syncing data from Amazon DynamoDB to Redis is a configuration exercise. Whether you need a one-time migration or an always-on data integration pipeline, Apache SeaTunnel has the tools for your needs.

The beauty of this approach:

  • No code to maintain – just configuration
  • Battle-tested connectors – focus on your business logic
  • Flexible deployment – from laptop to production cluster
  • Scalable by design – scale with your data horiontally

Ready to give it a spin? Start with standalone mode for quick experiments, then graduate to cluster mode when you're ready for production.

Have questions or want to share your specific data integration challenges? Please drop a comment below or find me on social media. Happy data syncing! 🚀

Top comments (0)