What Happens When Apache SeaTunnel Submits a Job?

A SeaTunnel job submission may look like a simple submitJob request from the outside. However, inside the Server, the request goes through multiple stages, including Master node validation, job coordination, JobMaster initialization, physical execution plan construction, Pipeline resource allocation, and TaskGroup deployment.

Based on the submitJob sequence I analyzed, this article focuses on one core path: from the moment a job submission request enters the SeaTunnel Server to the point where TaskExecutionService.deployTask() is finally called to deploy the TaskGroup.

This article does not cover the internal thread model of TaskExecutionService, Task execution details, or data flow processing. Instead, it focuses on the job submission, scheduling, and deployment lifecycle.

Core Components

Before diving into the workflow, let’s first understand the responsibilities of several key components involved in the submitJob execution path.

Role	Responsibility
SubmitJobServlet	Receives external job submission requests and serves as one of the entry points on the Server side.
JobInfoService	Handles job submission logic and determines whether the current node is a Master or a Worker.
MasterNode	When the current node is not the Master, forwards the job submission request to the Master node.
CoordinatorService	Serves as the job coordination entry point, checks whether the job is already running, and creates/manages JobMaster.
JobMaster	Acts as the execution control center for a single job, responsible for initializing the runtime context, classloader, checkpoint configuration, and other settings.
PhysicalPlan	A physical execution plan built from the logical DAG, responsible for driving Job-level state transitions.
SubPlan	A Pipeline-level scheduling unit responsible for resource allocation and Pipeline state transitions.
ResourceUtils	Requests execution resources for the Pipeline.
PhysicalVertex	A finer-grained physical execution node responsible for deploying TaskGroups.
TaskExecutionService	The execution service that ultimately receives and deploys TaskGroups.

Overall Workflow

The following simplified flow diagram provides an overview of the complete lifecycle.

The entire flow can be summarized in one sentence:

SubmitJobServlet
  -> JobInfoService
  -> MasterNode / CoordinatorService
  -> JobMaster
  -> PhysicalPlan
  -> SubPlan
  -> PhysicalVertex
  -> TaskExecutionService

Now, let’s break down each stage.

Stage 1: Request Enters JobInfoService

The job submission entry point first reaches SubmitJobServlet, which then delegates the request to JobInfoService.

The key point here is not to start the job immediately, but to first determine:

Is the node currently receiving the request the Master node?

If the current node is the Master, JobInfoService can continue the submission process locally.

If the current node is a Worker, the request needs to be forwarded to the Master through MasterNode.submitJob().

This design ensures that job submission is always coordinated by the Master node, preventing multiple nodes from independently creating scheduling contexts for the same job.

Stage 2: CoordinatorService Takes Over the Job

After reaching the Master node, the request continues to CoordinatorService.submitJob().

At this stage, CoordinatorService mainly performs two tasks:

1. Check whether the job already exists or is currently running.
2. If it is a new job, create and initialize the corresponding JobMaster.

If the job is already running, SeaTunnel does not need to create another scheduling context and can directly return a successful submission response.

For a new job, the process enters the JobMaster initialization phase.

At this point, submitJob has moved from API request handling into scheduler-level processing.

Stage 3: JobMaster Initialization

JobMaster can be understood as the runtime control center for a job.

After creating JobMaster, SeaTunnel performs several preparation steps required before execution, including:

• Building the classloader required for job execution.
• Initializing checkpoint-related configurations.
• Preparing the context required to build the physical execution plan from the logical DAG.

At this stage, Tasks have not been deployed yet. Instead, the system is preparing the runtime environment required for later scheduling.

Stage 4: From Logical DAG to PhysicalPlan

After JobMaster initialization, SeaTunnel builds a PhysicalPlan based on the logical DAG.

An important concept here is:

SeaTunnel does not start the entire job at once. Instead, it gradually progresses through different states using a state machine.

At the Job level, the core state transition can be simplified as:

CREATED -> SCHEDULED -> startSubPlanStateProcess

PhysicalPlan is responsible for Job-level state transitions, while actual Pipeline scheduling continues further down into SubPlan.

Stage 5: SubPlan Resource Allocation and Deployment

At the SubPlan layer, SeaTunnel shifts its focus from the entire Job to the Pipeline level.

SubPlan.stateProcess() executes different logic based on the current Pipeline state:

The key points at this stage are:

• In the CREATED state, the Pipeline first transitions to SCHEDULED.
• In the SCHEDULED state, SeaTunnel starts requesting resources through ResourceUtils.applyResourceForPipeline().
• After resources are successfully allocated, the Pipeline enters the DEPLOYING state.
• If resource allocation fails, the Pipeline enters makePipelineFailing(e).

Therefore, a Pipeline is not deployed immediately. It must first acquire the required execution resources.

Stage 6: PhysicalVertex Deploys TaskGroup

When the Pipeline enters the DEPLOYING state, the SubPlan starts launching internal PhysicalVertex components.

PhysicalVertex first updates the Task state to DEPLOYING, then performs deployment based on the assigned slotProfile.

During deployment, there is a key decision point:

Is the target Worker local or remote?

If the target Worker is a remote node, SeaTunnel sends a deployment request through DeployTaskOperation. The request is eventually handled on the target Worker through:

TaskExecutionService.deployTask(taskGroupInfo)

After successful deployment, PhysicalVertex updates the Task state to RUNNING.

If deployment fails, the system enters:

makeTaskGroupFailing()

Once all TaskGroups inside the Pipeline are successfully deployed and enter the running state, the SubPlan also transitions to RUNNING.

Failure, Cancellation, and Recovery Paths

In addition to the normal submission and deployment path, the SubPlan state machine also handles failures, cancellations, and recovery scenarios.

The process can be simplified as follows:

This is why the state machine design is important:

• The normal path can continue through deployment and execution.
• Failure paths can transition into failing / failed.
• Cancellation paths can transition into canceling / canceled.
• When recovery conditions are met, resources can be released, requested again, and the Pipeline can be restored.

In other words, the state machine is not designed to make the workflow complicated. It exists to make the entire job lifecycle controllable and reliable.

Complete Sequence Diagram

Finally, the complete sequence diagram connects the entire workflow and provides a clearer view of the execution order.

Summary

After a SeaTunnel job is submitted, the core process is not simply “receive the request and start the task.”

The complete lifecycle roughly follows this path:

SubmitJobServlet
  -> JobInfoService
  -> MasterNode / CoordinatorService
  -> JobMaster
  -> PhysicalPlan
  -> SubPlan
  -> PhysicalVertex
  -> TaskExecutionService

The responsibilities of each component are:

• JobInfoService handles the submission entry point and determines whether the request needs to be forwarded to the Master node.
• CoordinatorService manages job coordination, prevents duplicate submissions, and creates the JobMaster.
• JobMaster initializes the runtime context required by the job.
• PhysicalPlan manages Job-level state transitions.
• SubPlan handles Pipeline-level resource allocation and scheduling.
• PhysicalVertex manages TaskGroup deployment.
• TaskExecutionService is the final entry point responsible for deploying TaskGroups.

Understanding this lifecycle makes it much easier to explore SeaTunnel’s Task execution model, data flow architecture, and checkpoint mechanism, because each module can be placed in its correct position within the overall architecture.