DEV Community

Ravi More
Ravi More

Posted on

What Does a Production Support Engineer Actually Do ?

#career #devops #monitoring

A Production Support Engineer ensures that live applications and systems run smoothly without interruptions. They act as the first line of defense when something goes wrong in production. This role involves monitoring, troubleshooting, automation, and communication and all aimed at keeping systems stable and users happy.

Let’s explore the key responsibilities with real-world examples :

1. Monitoring Alerts Using ITRS and Splunk

Production environments generate alerts when something unusual happens — like high CPU usage or failed transactions.

Example:

Using ITRS Geneos, you might receive an alert that a database query is taking too long. You log into the system, check the query logs, and inform the database team.

With Splunk, you can search logs using keywords to find errors like:

ERROR: PaymentService failed to connect to DB

You then investigate the root cause and resolve it.

2. Writing Shell Scripts to Automate Tasks

Manual tasks can be time-consuming. Shell scripting helps automate repetitive actions.

Example:

  • You write a script to:
  • Archive logs older than 7 days
  • Restart a service if it crashes
  • Send email alerts when disk usage crosses 80% 
#!/bin/bash
if [ $(df / | grep -v Filesystem | awk '{print $5}' | sed 's/%//') -gt 80 ]; then
  echo \"Disk usage high\" | mail -s \"Alert\" admin@example.com
fi
Enter fullscreen mode Exit fullscreen mode

Breakdown :

  • df / : Shows disk usage of the root directory.
  • grep -v Filesystem : Removes the header line from the output.
  • awk '{print $5}' : Extracts the percentage of disk used (e.g., 85%).
  • sed 's/%//' : Removes the % symbol to get a pure number.
  • $(...) : Executes the command inside and returns the result.

  • -gt 80 : Compares the result to 80. If greater, the condition is true.

  • echo "Disk usage high" : Creates the message body.

  • mail -s "Alert" : Sends an email with subject “Alert”.

  • admin@example.com : Recipient of the alert.

  • fi : Ends the if block.

3. Monitoring Jobs Using AutoSys

AutoSys is used to schedule and monitor batch jobs like report generation or data sync.

Example:

You check if the EOD job for generating daily sales reports has failed. If it has, you rerun it and notify the business team.

You might use commands like:

autorep -j job_name -q

4. Checking Start-of-Day (SOD) and End-of-Day (EOD) Activities

These checks ensure systems are ready for business operations.

Example:

In the morning (SOD), you verify:

  • All services are running
  • No critical alerts are pending
  • Jobs scheduled overnight completed successfully

At night (EOD), you ensure:

  • Reports are generated
  • Backups are triggered
  • No pending transactions

5. Handling User Tickets via ServiceNow

Users raise issues through ticketing tools like ServiceNow.

Example:

A user reports they can't access a dashboard. You check their permissions, fix the issue, and update the ticket with resolution steps.

You also categorize tickets:

  • Access issues
  • Data mismatches
  • Application errors

6. Troubleshooting Production Issues and Finding Root Cause

When something breaks, you investigate logs, metrics, and configurations.

Example:

An API is returning 500 errors. You:

  • Check logs in Splunk
  • Restart the service
  • Identify a missing config file
  • Fix it and document the RCA (Root Cause Analysis)

7. Using Linux Commands for System Tasks

Linux is widely used in production. You use commands to check system health and perform actions.

Common Commands:

  • tail -f logfile.log → View live logs
  • df -h → Check disk space
  • ps -ef | grep service → Check if a service is running
  • top → Monitor CPU and memory usage

8. Maintaining KT Documents in Confluence

Knowledge Transfer (KT) documents help share information across the team.

Example:

  • You create a Confluence page titled “How to Restart Payment Gateway Service” with: Step-by-step instructions
  • Screenshots
  • Common errors and fixes

This helps new team members learn quickly and ensures consistency.

While Production Support Engineers and DevOps Engineers share some overlapping skills — like automation, monitoring, and troubleshooting — their roles are different in scope.

You can think of a Production Support Engineer as someone who handles real-time operational issues, whereas a DevOps Engineer focuses more on building and maintaining CI/CD pipelines, infrastructure as code, and deployment automation.

The responsibilities of a Production Support Engineer can vary from company to company. The exact tasks often depend on the client’s requirements, the technology stack, and the business domain. While some engineers may focus more on automation and scripting, others might handle more incident management or user support.

Top comments (0)