This article explains how to find the first real error when a pipeline shows failed, instead of running it again and again. After reading it, you can follow run detail, the failed job, and the log of the failed step, compare the exit code and Failure diagnosis, and tell apart an image pull, a checkout, a missing command, and a timeout. In the YAML, a task is a job: one group of steps that run in order. A runner is the program on a build machine that actually runs those steps.
Pipelines are available on Team and Enterprise. Community has no CI/CD menu, and the steps in this article do not apply there.
The screenshots use the pipeline mcp-sample-fail on the example repository xlliu/test-cicd-example, and its run #3. This run was started by hand, on the runner priv-1, at commit 98d2f2a. The failed job is named fail, and the step is named Fail on purpose. The step name shows this is a step that fails on purpose, so the log only has the checkout record and has no compile error.
1. Find the failed run
- Open the pipeline In the sidebar, open CI/CD → Pipelines and click the pipeline name. The detail page opens on the Runs tab.
- Find the failed run In the run list, filter by status and choose Failed. You can also filter by Trigger, for example only runs whose trigger is Pull request, and you can filter by branch. After you find the failed run, click its number to open the run detail.
The run detail title is Run followed by the run label. Across the top are Refresh, Compare, Re-run, and Re-run Failed Jobs. Re-run runs the jobs in this run again. Re-run Failed Jobs reruns only the failed jobs. While the run has not finished, Cancel run is also there. Under the buttons are three tabs: Jobs, Trigger parameters, and Artifacts.
Compare picks another run and compares status and duration. The description is “Pick another run to compare status and duration differences.” If yesterday succeeded and today failed, compare with the latest successful run and you can see which job changed from success to failure.
2. From the failed job to the failed step
On the Jobs tab, each job is one row: the status first, then the job name. The arrow in front of a job expands its steps. The arrow in front of a step expands that step’s log.
Failed jobs and steps, and ones that are still running, expand on their own. You do not have to open each one. The page also scrolls to the first failed job. The table lists the statuses you usually see, and where to look first:
| What you see in the job list | What it means | Where to look first |
|---|---|---|
| Job Failed | A step in this job failed | Expand the step under it whose status is Failed |
| Job Timeout | The whole job ran longer than the limit (default 3600 seconds) | Read the last few log lines before the timeout, or raise runner.resources.timeout
|
| Job Skipped | An upstream job it depends on failed, so this job did not run | Fix the upstream job that failed |
| Exit code: 1 beside a step | This step’s command exited with a non-zero status (a zero exit code is not shown) | Read the last twenty or thirty lines of that step’s log |
| The step log area says “This step never started. The job failed while preparing the workspace.” | The job failed while preparing the code, and no step started | Look for Git clone failed: … in the first step’s log |
The exit code in the table is the number the command returns to the system when it ends. 0 means success. A non-zero value means failure. If several jobs failed, start with the earliest one whose status is not Skipped. Later failures are often a consequence of that one.
3. Find the error in the log
The step log opens under the step, with line numbers on the left. A step that is still running refreshes live and follows the newest line. When you scroll up, the page stops following, so you can read earlier lines.
A few actions on the log:
-
Search. In Search logs... type
ERROR,FAILED,Exception, or the file name you suspect. Search covers only the log that has already been loaded. If you find nothing, load the earlier log and search again. - See the earlier part. When the log is long, the page first shows only the tail and says “Showing the end of the log. Search applies to the loaded window. Download for the full file.” Click Load earlier lines to move backward. The sample run’s log is only about a dozen lines, so that sentence does not appear.
- Jump to the end. Scroll to bottom, on the right of the log toolbar, returns to the newest line.
- Download. Click Download to save this step’s full log locally and open it in an editor you already use.
Read from the end backward. The cause is usually in the last twenty or thirty lines, because a compile error, a failed test name, or command not found is printed before the command exits. If the last lines are only a summary such as “build failed”, keep going up to the first ERROR.
Secrets in the log are replaced with ***, or shown as password=***MASKED***. The platform masked the secret. The log itself is intact.
On the run-detail screenshot above, the step log has line numbers on the left, and the toolbar is Search logs..., Scroll to bottom, and Download. The last line of this sample log is INFO: git checkout complete. It contains no ERROR.
4. Let Failure diagnosis look first
Next to a failed or timed-out job there is a Failure diagnosis button. The dialog description is “Rule-based diagnosis from failure logs with suggested fixes”. The content includes:
- A category, for example Configuration error, Authentication error, or Container error
- Confidence, for example 85%
- Root cause: one sentence for the cause it settled on
- Suggestions: a few changes you can try
After a job fails, the system runs a diagnosis once on its own. If you open it and see “No diagnosis yet. You can run a log-rule diagnosis”, click Run diagnosis.
Failure diagnosis matches common errors in the log with built-in rules. What it gives you is a suggestion. It does not change your code or your YAML. A high-confidence suggestion is worth trying as written. A low-confidence one is worth confirming in the log yourself. If the diagnosis is off, write the real cause under Feedback at the bottom of the dialog and click Submit feedback.
Take the sample fail job. Its log matches no existing rule, so the summary under the job row is: Unknown · No known pattern matched for job 'fail' step 'Fail on purpose'; last log: INFO: git checkout complete. After you click Failure diagnosis, the dialog title is “Failure diagnosis for job "fail"”. The category badge is Unknown. The summary, root cause, and suggestions follow the console language. With the console in English, the root cause is “Unable to determine root cause from log patterns. Last log line: INFO: git checkout complete”, and the suggestions are “Review the full build log” and “Check for environment issues”.
This step recorded no exit code, so neither the job row nor the diagnosis shows “Exit code”. The other way around: if the last log line is still only INFO, and the step or the job recorded a non-zero exit code, the summary also appends that exit code, as in “; exit code: 1”, and the root cause appends “Step exit code: 1”. When the log matches a rule, the category changes to something such as Configuration error or Authentication error, and the body still follows the current console language. Two exceptions stay as they were saved: notifications and exports use the English snapshot stored with the diagnosis result; an old record with no rule name, and free text produced by a model, keep the sentence that was saved at the time.
5. A few common failures
The table collects how a few common failures look in the log, why they happen, and how to change them:
| What you see in the log | Why | What to change |
|---|---|---|
ERROR: Failed to pull image …, or Timed out pulling image … after 300s
|
The build machine cannot pull the image named by image: the address is wrong, the registry cannot be reached, or the registry requires a login |
On the build machine, try docker pull <image address>. On a private network, push the image to an internal registry and use that address in the YAML |
No step has started, the log area says “This step never started. The job failed while preparing the workspace.”, and the first step’s log has Git clone failed: …
|
The code pull before the job started failed. The usual causes are network or authentication | Read the specific reason after the colon. For an authentication problem, the diagnosis category is usually Authentication error |
command not found, exit code 127 |
The command this step uses does not exist in the execution environment | For a DOCKER job, switch to an image that includes the command, or install it in the step. For a NATIVE job, install it on the build machine |
Step timed out after 1800 seconds (inside a container job, Step timed out after 1800s) |
This step ran longer than the seconds set by timeout in the YAML. The step status is Failed
|
Decide whether it is slow or stuck. If it is slow, raise timeout. If it is stuck, read the last few log lines before the timeout. A common case is waiting for input or for the network |
The job shows Timeout (after automatic retries still time out, it is shown as Failed), and the reason is Job execution exceeded timeout of 3600s
|
The whole job ran longer than the limit. The default is 3600 seconds | Raise runner.resources.timeout on the job (the unit is seconds), or split the job into smaller jobs |
| A compile error or a test failure | A problem in the code itself | Reproduce it locally, fix it, and push a new commit |
DOCKER and NATIVE in the table are the job’s execution modes. A DOCKER job runs in a container the runner starts. A NATIVE job runs directly on the build machine (the host). Both the step-level timeout and the job-level runner.resources.timeout are in seconds. A step timeout can be from 1 to 86400. Today it can be changed only in the YAML. The visual editor has no field for it.
6. After the fix
- A one-off problem (for example a network blip while downloading dependencies): open that run and click Re-run. Re-run uses the same commit again and copies the original run’s trigger parameters. The hint is “Re-run copies dispatch inputs from the original run”.
- A problem in the code or the config: commit the change and push. The push starts a new run according to the trigger rules. You do not also re-run the old one by hand.
- Confirm it is fixed: use Compare to put the new run next to the failed one, and confirm the job that failed is now a success.
7. Hear about the next failure right away
To get a message the next time a run fails, open CI/CD → Settings → Notifications, click New Notification, and under trigger events select Pipeline failed or Job failed. The channel can be Slack, DingTalk, WeCom, Lark, email, or Webhook. Before you save, click Send Test and confirm the channel receives the message.
Common questions
The step log has no time. How do I see which step was slow?
The step log does not show a time on each line. For duration, use Compare on the run detail. It lists each job’s duration and the change. You can also print the time from the step command, for example by running date.
Where is the runner log, and how is it different from the step log?
A runner is the program on the build machine that runs jobs, and it has its own process log. Open CI/CD → Runners, then the Logs tab on the runner detail. The page says “Runner process log: registration, heartbeats, job assignment, image pulls, and step execution. Step output is on the pipeline run page.” For an error in the build itself, read the step log. The runner log is for a runner that cannot take a job, or for an image pull that looks wrong on the runner.
The job went back to Pending while it was running. Did it fail?
That is not a failure. The usual cause is that the runner running it lost the connection in the middle. AngusGit puts the job back on the queue and waits for another runner to take it. How to track that down is in the article “The runner is not taking jobs”, linked at the end.
Summary
When a pipeline fails, go downward in order. Open the failed run in Runs, then on the Jobs tab find the first failed job (one that was skipped comes later) and the failed step under it, then read upward from the last few dozen lines of that step’s log. The log can be searched, earlier lines can be loaded, and the full file can be downloaded. When the cause is unclear, click Failure diagnosis and read Root cause and Suggestions. Image pull, checkout, command not found, and timeout each have a fixed line in the log. Match them to the table in section 5. For a one-off problem, click Re-run. For a code or config problem, push after the fix.
Conclusion: To investigate a failed pipeline, open the Jobs tab on the run detail, find the first Failed job (skip a job that is Skipped) and its failed step, then fix it from the exit code, the tail of the log, and Failure diagnosis. Then re-run or push.
Top comments (0)