The Task Failed, but Where Did the Logs Go? Troubleshooting an Apache DolphinScheduler Failed Task

A failure with no logs: tracing a task execution issue from metadata to the Master node

One of the most confusing scenarios in production scheduling systems is when a task is clearly marked as "failed" in the UI and an alert has already been generated, but critical execution information such as the task start time, execution host, execution directory, and log path are all empty.

There is no Worker log, no YARN Application ID, and no Spark Driver, Executor, or MapReduce Container logs.

When facing this situation, engineers often start troubleshooting from the Worker node, script paths, YARN environment, or network connectivity. However, this case revealed a different root cause:

The task had never reached the Worker node. The failure occurred while the Master was constructing the task execution context. The actual issue was neither the script nor the virtual IP, but an invalid tenant configuration in the workflow.

Key takeaway:
When a task_instance is already in the FAILED state, but host, start_time, log_path, and execute_path are all NULL, the first place to investigate should be the Master-side task submission and dispatch process, rather than searching for task execution logs that were never generated.

1. Incident Overview: Failed Task, Failed Alert, and Empty Logs

In this case, a batch workflow was scheduled to run every day at 14:15. The scheduling UI showed that the first Shell task failed, and subsequent tasks were not executed.

Meanwhile, the alert table already contained a "scheduler failed" record. However, the alert delivery logs also showed another issue:

no bind plugin instance

The latest failed alert can be retrieved from t_ds_alert:

SELECT id, title, content, alert_status, warning_type,
log, alertgroup_id, create_time,
process_instance_id
FROM t_ds_alert
ORDER BY create_time DESC
LIMIT 1;

The alert content contains the workflow instance ID, task Code, task name, task type, and failure status, making it a useful entry point for reconstructing the entire troubleshooting chain.

However, two issues must be distinguished:

• Task execution failure
• Alert delivery failure

These are independent problems and should not be treated as the same failure.

Problem	Direct Evidence	Impact
Task failure	taskState=FAILURE	The task itself did not execute successfully.
Alert failure	alertgroup_id=0; no bind plugin instance	The failure event was generated, but notifications (e.g., email/DingTalk) were not sent.

Troubleshooting principle:
Do not treat "alert delivery failure" as the cause of task failure. First determine which stage of the task lifecycle failed, then separately troubleshoot alert routing.

2. First Evidence: Metadata Shows the Task Never Reached the Worker

Using the process_instance_id and taskCode obtained from the alert, we can query t_ds_task_instance and locate the exact task execution record.

The result shows that the task instance was created and its state was already 6 (FAILURE). However, except for submit_time, all other runtime-related fields were empty.

Query the task lifecycle fields:

SELECT id AS task_instance_id, name, state, host,
submit_time, start_time, end_time,
log_path, execute_path, app_link
FROM t_ds_task_instance
WHERE process_instance_id = <process_instance_id>
AND task_code = <task_code>
ORDER BY id DESC;

Field	Query Result	Diagnostic Meaning
state	6	The task has been marked as failed by the Master.
submit_time	Has value	The task instance has been created in the metadata database.
start_time	NULL	The Worker never started execution.
host	NULL	No Worker has been assigned yet.
execute_path	NULL	The execution directory has not been created yet.
log_path	NULL	No task execution log will be generated.
app_link	NULL	Not submitted to YARN.

At this point, we can make the first critical conclusion:

The task failed before Worker execution began.

Continuing to SSH into Worker nodes, searching for logs, running yarn logs, or analyzing Spark Executor issues would not provide useful information, because the execution entity itself was never created.

3. A Misleading Clue: Why Does the Master Show a Different IP?

During troubleshooting, the primary IP address of the Master node was found to be inconsistent with the Master address displayed in the scheduling frontend. After SSH-ing into the address shown in the frontend, both the hostname and shell prompt still displayed the primary IP, which initially raised concerns about incorrect routing or an address registration issue.

The SSH connection tuple, hostname, and all network interface addresses were then verified:

echo "$SSH_CONNECTION"
hostname -f
ip -br addr

The output of ip -br addr showed that the primary IP was bound with a /24 network mask, while another address was bound with /32 on the same eth0 interface.

The latter resolved to an EMR virtual hostname, indicating that it was not another machine, but an auxiliary or virtual service IP on the same node.

Testing SSH connectivity to a virtual IP on the Master node only proves that the address is reachable locally.

To verify whether scheduling communication was actually working, the service port needed to be tested from a real Worker node.

Run the following commands from an actual Worker node:

nc -vz -w 3 <master-primary-ip> 5678
nc -vz -w 3 <master-virtual-ip> 5678

Both Master addresses successfully accepted connections on port 5678.

Meanwhile, Master heartbeat logs continued to update successfully under:

/nodes/master/:5678

At this point, the possibility of an unreachable Master registration address could be ruled out.

Lesson learned:
A mismatch between the IP displayed in the UI and the host's primary IP does not necessarily indicate a configuration error. First determine whether it is a secondary IP or VIP on the same machine, then verify service connectivity from the actual communication endpoint. Avoid relying only on local self-connectivity tests.

4. Decisive Evidence: "Tenant does not exists" in Master Logs

Since the task never reached the Worker node, the real evidence had to be found in the Master logs.

By searching the Master logs with multiple stable identifiers, including the workflow instance ID, task instance ID, task Code, and task name, the complete failure chain was reconstructed.

Use multiple identifiers to locate the same scheduling event:

grep -R -nE 'WorkflowInstance-<id>|TaskInstance-<id>|<task_code>|<task_name>' /var/log/.../master-server/

The failure occurred within milliseconds, before the task was actually dispatched to a Worker:

14:15:00.650 Task is ready to dispatch to worker
14:15:00.651 Tenant does not exists
14:15:00.651 Task state changes to FAILURE
14:15:00.652 Get taskExecutionContext fail
14:15:00.653 Dispatch standby task failed
14:15:00.690 Workflow state changes to FAILURE

The log output provided the decisive evidence:

processDefinition.tenantId=-1
processDefinition.tenantCode=null

processInstance.tenantId=-1
processInstance.tenantCode=null
processInstance.queue=null

When constructing the TaskExecutionContext, the Master node must determine the execution tenant information. However, the current workflow did not have a valid tenant configuration.

As a result, DolphinScheduler could not generate the task execution context, and the task was marked as FAILURE before Worker allocation.

Root cause:

The workflow definition and workflow instance both had tenantId=-1, with an empty tenantCode.

The Master failed to construct the TaskExecutionContext, causing the task to fail before it was dispatched to a Worker node.

5. Why Are There No Logs? Understanding the DolphinScheduler Task Lifecycle

In DolphinScheduler, task logs are not generated when a TaskInstance record is created in the database.

Instead, logs become available only after the following steps are completed:

1. The Master successfully constructs the task execution context.
2. The task is assigned to an available Worker.
3. The Worker creates the execution directory and starts running the task.

Understanding this sequence is essential when troubleshooting the situation where a task fails but no logs exist.

Stage	Typical Field Changes	Where to Check
1. Create instance	submit_time has value; state = submit successful	Metadata database, Master logs
2. Build context	Populate tenant, queue, environment, etc.	Master logs
3. Assign Worker	host has value	Master dispatch logs, registration center
4. Worker startup	start_time, execute_path, log_path have values	Worker task logs
5. Submit big-data job	app_link / Application ID has value	YARN and Container logs

These lifecycle fields also provide a practical troubleshooting strategy:

Field status	Investigation direction
host is empty	Check Master-side task creation and dispatch
host exists but start_time is empty	Check Worker assignment and Worker acceptance
log_path exists	Check Worker execution logs
Application ID exists	Proceed with YARN log investigation

In other words, the absence of logs is itself a valuable signal.

6. What Role Does Tenant Play in DolphinScheduler?

In DolphinScheduler, a Tenant is not merely a classification field displayed in the UI.

It is closely related to task execution identity and resource allocation.

When the Master constructs a task execution context, it needs to resolve information such as:

• tenantCode
• execution user
• resource queue

This information tells the Worker:

• Which user identity should execute the task?
• Which resource queue should be used?
• Which permissions should be applied?

Without valid tenant information, the Master cannot safely create an execution environment for the task.

In environments using:

• Linux user switching
• HDFS permission control
• Kerberos authentication
• YARN queue isolation

tenant configuration can further affect:

• Operating system user identity
• HDFS directory permissions
• Authentication tickets
• Resource queue assignment

Therefore, even after repairing the tenant configuration, engineers should still verify the corresponding OS user permissions and big data platform access control.

7. Confirm Tenant Relationships Through Metadata

To verify the relationship between workflows, users, and tenants, query the following metadata:

SELECT pd.id, pd.code, pd.name, pd.version,
pd.user_id, u.user_name,
pd.tenant_id AS workflow_tenant_id,
u.tenant_id AS user_tenant_id,
t.id AS matched_tenant_id,
t.tenant_code, t.queue_id
FROM t_ds_process_definition pd
LEFT JOIN t_ds_user u ON u.id = pd.user_id
LEFT JOIN t_ds_tenant t ON t.id = pd.tenant_id
WHERE pd.code = <process_definition_code>;

Confirm that the tenant exists and check historical workflow versions:

SELECT * FROM t_ds_tenant ORDER BY id;

SELECT * FROM t_ds_user
WHERE id = <user_id>
OR user_name = '<user_name>';

SELECT code, name, version, tenant_id, release_state
FROM t_ds_process_definition_log
WHERE code = <process_definition_code>
ORDER BY version DESC;

The expected abnormal result in this case was:

workflow_tenant_id = -1

and no corresponding record could be found in t_ds_tenant.

If a user already has a tenant but the workflow still shows tenant_id=-1, it indicates that the workflow did not correctly inherit tenant information during creation, import, or version migration.

Simply modifying the user configuration does not necessarily update existing workflows automatically.

8. Fix Procedure: Avoid Direct Updates to Metadata Tables

The recommended approach is to repair the configuration through the DolphinScheduler UI instead of directly modifying metadata tables.

Direct database updates may bypass:

• workflow version management
• cache synchronization
• permission relationships
• audit information

The standard repair process is:

1. Create or select a valid tenant

In Security Center → Tenant Management, create or select a valid tenant and associate the correct YARN Queue.

2. Bind the tenant to the workflow owner

In User Management, assign the tenant to the workflow owner.

3. Update the workflow configuration

Edit the failed workflow, select a valid tenant, save the workflow, and generate a new workflow version.

4. Publish the updated workflow

Release the new workflow version and verify that:

t_ds_process_definition.tenant_id > 0

5. Start a new execution

Trigger a completely new manual run instead of recovering the old failed instance.

6. Verify runtime permissions

After the task reaches the Worker node, verify:

• Linux user permissions
• script execution permissions
• HDFS access
• Kerberos authentication
• YARN queue permissions

Why not rerun the old instance?
A failed instance stores a snapshot of the previous workflow version, where tenantId is still -1. Recovering the old instance may continue using the invalid configuration. After publishing a new workflow version, create a new instance for validation.

9. How to Validate After Fixing the Issue

After the repair is completed, validation should not stop at confirming that the workflow status turns green.

A complete verification process should confirm that the task has successfully passed the original failure point.

It is recommended to verify the workflow from three layers:

Definition layer

• The latest workflow version has a valid tenant_id greater than 0.
• The tenant can be correctly mapped to a record in t_ds_tenant.

Instance layer

• The new workflow instance contains valid tenantCode.
• The resource queue information is no longer empty.

Dispatch layer

• TaskInstance.host is populated.
• Worker nodes receive the task successfully.

Runtime layer

• start_time, execute_path, and log_path are generated correctly.

Big data execution layer

For tasks submitting Spark or MapReduce jobs:

• The Application ID can be retrieved.
• YARN logs can be queried successfully.

Acceptance Field	When Fault Occurs	Expected After Fix
tenant_id	-1	> 0
tenantCode / queue	NULL	Valid value
host	NULL	Worker address
start_time	NULL	Actual start time
execute_path / log_path	NULL	Actual execution directory and log path
app_link	NULL	Application ID or link appears after submission to YARN

Conclusion: No Logs Are Logs

In scheduling systems, "no logs" does not mean there is no evidence.

On the contrary, when host, start_time, execute_path, log_path, and app_link are all empty, these fields already provide a strong indication:

The task never entered the execution layer.

Following this clue back to the Master node, the millisecond-level failure timeline eventually pointed to:

Tenant does not exists

A high-quality troubleshooting process is not about checking every component blindly.

Instead, it requires:

• Using metadata to define the investigation boundary.
• Using logs to reconstruct the execution timeline.
• Using network tests to eliminate unrelated possibilities.
• Separately closing the loop on task failures and alert delivery failures.

This approach transforms troubleshooting from experience-based debugging into a systematic, explainable, and reusable diagnostic methodology.

One-line summary:

When a TaskInstance enters the FAILED state but all runtime fields are NULL, check the Master first. In this case, the direct cause was tenantId=-1, which prevented the Master from constructing the TaskExecutionContext.