How DevOps Engineers Troubleshoot Production Issues: From Symptoms to Root Cause

"The first step in solving a problem is recognizing that it does exist."

Table of Contents

1. Introduction
2. Understanding Production Issues
3. The Troubleshooting Mindset
4. From Symptoms to Root Cause
5. A Step-by-Step Production Troubleshooting Process
6. Common Production Issues and Investigation Techniques
7. Essential DevOps Troubleshooting Tools
8. Interesting Facts & Statistics
9. FAQs
10. Key Takeaways
11. Conclusion

1. Introduction

Production troubleshooting is one of the most important responsibilities of a DevOps Engineer. Applications may work perfectly in development and testing environments but still experience unexpected problems after deployment.

A production issue can appear in many forms:

• Website or API is down
• Application is responding slowly
• Server CPU or memory is high
• Database connections are exhausted
• Docker containers keep restarting
• Kubernetes pods are failing
• Disk space is full
• SSL certificates have expired
• Nginx or Apache returns 502/504 errors
• Users cannot log in
• Background jobs are stuck
• Deployment causes unexpected errors

The visible problem is usually only a symptom. The real responsibility of a DevOps Engineer is to move from that symptom to the root cause, restore the service safely, and prevent the same issue from happening again.

A good troubleshooting process can be summarized as:
Observe → Investigate → Isolate → Identify Root Cause → Fix → Verify → Prevent

2. Understanding Production Issues

A production issue is any unexpected behavior that negatively affects an application, infrastructure, service, or user experience.

For example:
Symptom:
Website returns HTTP 502 Bad Gateway.
This does not automatically mean Nginx is broken.
The actual root cause could be:
Nginx
↓
PHP-FPM
↓
Application
↓
Database

For example:

• PHP-FPM may be stopped.
• PHP-FPM workers may be exhausted.
• The application may be taking too long.
• The database may be unavailable.
• The server may have insufficient memory.

Therefore, troubleshooting should not begin with assumptions.

3. The Troubleshooting Mindset

Effective troubleshooting requires a systematic approach.

3.1 Do Not Guess
Avoid immediately changing configuration files or restarting services without understanding the problem.
Instead of:

systemctl restart nginx

first investigate:

systemctl status nginx
journalctl -u nginx --since "10 minutes ago"

3.2 Start With the Symptoms
Identify exactly what is failing.
For example:
Users report that the website is slow.

Convert this into measurable information:
Response time increased from 300 ms to 8 seconds.
Now the problem can be investigated systematically.

3.3 Change One Thing at a Time
If you change five configurations simultaneously, you may fix the issue but not know which change solved it.

A better approach is:
Observe
↓
Make one controlled change
↓
Test
↓
Observe again

3.4 Preserve Evidence
Logs, metrics, process information, network information, and system statistics can disappear after a restart.

Before restarting a server, collect useful evidence when possible.

4. From Symptoms to Root Cause

The most important troubleshooting skill is understanding the difference between a symptom, cause, and root cause.

Example
Symptom:
The website is returning 504 Gateway Timeout.

Immediate cause:
The backend request did not respond within the configured timeout.

Possible deeper cause:
The application is waiting for a database query.

Root cause:
A missing database index caused a query to scan millions of records.
The troubleshooting journey therefore looks like:
504 Error
↓
Backend Timeout
↓
Slow Application Request
↓
Slow Database Query
↓
Missing Database Index
↓
ROOT CAUSE
This is why simply restarting a service may restore availability temporarily without actually solving the problem.

"A good engineer does not just restore the service; they learn why it failed."

5. A Step-by-Step Production Troubleshooting Process

Step 1: Confirm the Problem
First determine whether the problem is real and reproducible.

Check:

• curl -I https://example.com

Check service status:

• systemctl status nginx

Check running processes:

• ps aux

Check system resources:

• top
• htop

Step 2: Determine the Scope
Ask:

• Is the entire application affected?
• Is only one API affected?
• Is only one server affected?
• Are only certain users affected?
• Did the problem start after a deployment?
• Is the problem intermittent or constant?

For example:
All users affected
↓
Infrastructure/Application issue likely

versus:
One API endpoint affected
↓
Application/database issue likely

Step 3: Check Recent Changes
Many production incidents are related to recent changes.

Investigate:

• Deployment
• Configuration change
• Database migration
• Dependency update
• Infrastructure change
• SSL renewal
• DNS change
• Security rule change

For Git:

• git log --oneline -10

For Docker:

• docker ps
• docker ps -a

For deployments, check CI/CD logs and deployment history.

Step 4: Check Server Health
Important resources include:
CPU

• top
• uptime Memory free -h Disk df -h Disk Inodes df -i Disk Usage du -sh /var/log/*

A full disk can cause surprisingly large numbers of application failures.

Step 5: Check Application Logs
Logs are one of the most valuable sources of evidence.
Examples:

• tail -100 /var/log/nginx/error.log
• tail -100 /var/log/nginx/access.log

For system services:

• journalctl -u nginx

For Docker:

• docker logs

For a Laravel application:

• tail -100 storage/logs/laravel.log

The goal is not simply to find an error message, but to correlate the error with the time and symptoms of the incident.

Step 6: Check Network Connectivity
Test whether services can communicate.
For example:

• curl -I http://127.0.0.1:8080
  Check listening ports:

• ss -lntp
  Check DNS:

• dig example.com
  Check connectivity:

• ping example.com
  For HTTP troubleshooting:

• curl -v https://example.com
  Step 7: Check Dependencies
  Modern applications rarely work independently.
  A typical architecture may look like:
  User
  ↓
  DNS
  ↓
  Load Balancer
  ↓
  Nginx
  ↓
  Application
  ↓
  Redis
  ↓
  Database
  ↓
  External API
  If the application is failing, check each dependency.
  For example:
  Application healthy?
  ↓
  Database healthy?
  ↓
  Redis healthy?
  ↓
  External API healthy?

Step 8: Identify the Root Cause
After collecting evidence, identify the most likely root cause.
A useful technique is the 5 Whys.
Example:

• Why is the website slow? Because the application requests are taking too long.
• Why are requests taking too long? Because database queries are slow.
• Why are database queries slow? Because the query performs a large table scan.
• Why does it perform a table scan? Because the required column is not indexed.
• Why was the index missing? Because the database migration did not include the required index. The root cause is therefore not simply: Website is slow.

It is:
Missing database index introduced by an incomplete database migration.

Step 9: Apply the Fix Safely
Once the root cause is understood, apply the smallest safe change.
Depending on the issue, this could involve:

• Restarting a failed service
• Rolling back a deployment
• Increasing a resource limit
• Fixing application code
• Adding a database index
• Correcting Nginx configuration
• Cleaning unnecessary files
• Fixing permissions
• Renewing an SSL certificate
• Scaling application instances Avoid unnecessary production changes.

Step 10: Verify the Fix
After applying the fix, verify the entire request path.

For example:
curl -I https://example.com

Check logs:
tail -f /var/log/nginx/error.log

Check resources:
top
free -h
df -h

Check application health.
Most importantly, confirm that the original symptom has disappeared.

Step 11: Monitor After Recovery
Do not immediately consider the incident finished.

Monitor:

• CPU
• Memory
• Disk
• Application errors
• HTTP status codes
• Response time
• Database performance
• Container health
• User reports

The objective is to make sure the problem does not return.

Step 12: Document the Incident
A good incident report should contain:

Incident:
Website unavailable

Impact:
Users could not access the application

Start Time:
10:15 AM

Detection:
Monitoring alert

Root Cause:
Database connection pool exhausted

Resolution:
Increased connection pool and fixed connection leak

Prevention:
Added monitoring and application-level connection handling Documentation helps the team respond faster to similar incidents in the future.

6. Common Production Issues and Investigation Techniques

7. Essential DevOps Troubleshooting Tools

Linux
top
htop
ps
free
df
du
ss
lsof
journalctl
systemctl

Networking
curl
wget
ping
dig
nslookup
traceroute
ss

Docker
docker ps
docker ps -a
docker logs
docker inspect
docker stats
docker exec

Kubernetes
kubectl get pods
kubectl describe pod
kubectl logs
kubectl get events
kubectl get nodes

Git
git log
git diff
git show
git status

Log Analysis
grep
awk
sed
cut
sort
uniq
tail
less

Example:
grep "ERROR" application.log | tail -50

8. Interesting Facts & Statistics

• Production incidents are often caused by a combination of factors rather than a single isolated failure.
• A service can be technically "up" while still being unavailable from the user's perspective because of high latency or application errors.
• Restarting a service can sometimes hide the symptom without fixing the root cause.
• Logs provide historical evidence, while monitoring and metrics provide information about system behavior over time.
• Small configuration changes can sometimes create large production impacts.
• Disk-space problems are particularly dangerous because many unrelated services may begin failing when the filesystem becomes full.
• High CPU does not always mean that the server needs more CPU; an inefficient process, infinite loop, or unexpected workload may be the real cause.
• High memory usage does not automatically indicate a memory leak. Linux intentionally uses available memory for caching.

"You cannot fix what you cannot observe."

9. FAQs

Q1. What is the first thing a DevOps Engineer should do during a production incident?
Confirm the problem and understand its scope before making changes.

Q2. Should I restart the server when an application is down?
Not immediately. First collect evidence and determine why the application is failing. Restarting may remove useful diagnostic information.

Q3. What is the difference between a symptom and a root cause?
A symptom is what you observe. The root cause is the underlying reason the problem occurred.
Example:

• Symptom: 502 Bad Gateway
• Root Cause: PHP-FPM workers were exhausted.

Q4. Why are logs important?
Logs provide detailed information about what happened inside applications, services, and infrastructure.

Q5. How do I troubleshoot high CPU usage?
Start with:
top

Then identify the process consuming CPU and investigate why it is consuming excessive resources.

Q6. How do I troubleshoot high memory usage?
free -h
top

Then identify memory-heavy processes and investigate their behavior.

Q7. What should I check when disk space is full?
Start with:
df -h
Then identify large directories:
du -sh /* 2>/dev/null

Q8. What is the 5 Whys technique?
It is a root-cause analysis technique where you repeatedly ask "Why?" until you reach the underlying reason for the problem.

Q9. What should happen after fixing a production issue?
Verify the fix, monitor the system, document the incident, and implement preventive measures.

Q10. Is troubleshooting only about fixing servers?
No. Modern DevOps troubleshooting involves applications, databases, containers, Kubernetes, networking, cloud infrastructure, CI/CD pipelines, monitoring, security, and external dependencies.

10. Key Takeaways

• Do not troubleshoot based on assumptions. Use evidence.
• Understand the difference between symptoms and root causes.
• Check recent deployments and configuration changes.
• Always examine logs and system metrics.
• Check dependencies such as databases, Redis, APIs, and networks.
• Use tools such as top, df, free, curl, journalctl, Docker, and Kubernetes commands.
• Make controlled changes instead of changing everything at once.
• Verify that the original problem is actually resolved.
• Monitor the system after recovery.
• Document incidents and implement preventive measures.

"Symptoms tell you where to look. Evidence tells you what happened."

11. Conclusion

Production troubleshooting is not simply about running commands until an error disappears. It is a structured engineering process that requires observation, logical thinking, evidence collection, system knowledge, and careful decision-making.

A skilled DevOps Engineer does not stop at:
The website is working again.

Instead, they ask:

• Why did it fail?
• What caused the failure?
• Why was the problem not detected earlier?
• How did we fix it?
• How can we prevent it from happening again?

The ultimate goal of production troubleshooting is therefore not only recovery, but also understanding and prevention.

The strongest DevOps engineers turn production incidents into learning opportunities. Every incident can improve monitoring, automation, infrastructure reliability, deployment processes, documentation, and the overall resilience of the system.

Remember the core troubleshooting cycle:
SYMPTOM
↓
OBSERVE
↓
COLLECT EVIDENCE
↓
ISOLATE
↓
FIND ROOT CAUSE
↓
FIX
↓
VERIFY
↓
MONITOR
↓
PREVENT
That mindset transforms production troubleshooting from a stressful emergency activity into a disciplined engineering practice.

About the Author: Nilesh is a Lead DevOps Engineer at AddWebSolution, specializing in automation, CI/CD, and cloud scalability.