DEV Community

Janak Shrestha
Janak Shrestha

Posted on

Apache Troubleshooting

#linux

xFusionCorp Industries uses some monitoring tools to check the status of every service, application, etc running on the systems. Recently, the monitoring system identified that Apache service is not running on some of the Nautilus Application Servers in Stratos Datacenter.

  1. Identify the faulty Nautilus Application Server and fix the issue. Also, make sure Apache service is up and running on all Nautilus Application Servers. Do not try to stop any kind of firewall that is already running.
  2. Apache is running on 8089 port on all Nautilus Application Servers and its document root must be /var/www/html on all app servers. o
  3. Finally you can test from jump host using curl command to access Apache on all app servers and it should be reachable and you should get some static page. E.g. curl http://stapp01:8089/.

Environment Details

Server Hostname User Password
App Server 1 stapp01 tony Ir0nM@n
App Server 2 stapp02 steve Am3ric@
App Server 3 stapp03 banner BigGr33n
Jump Host jump-host thor mjolnir123

Step 1: Identify the Faulty Server

Connect to the jump host and test each app server.

ssh thor@jump-host
Password: mjolnir123
Enter fullscreen mode Exit fullscreen mode

Test Apache on each server.

curl -I http://stapp01:8089/
curl -I http://stapp02:8089/
curl -I http://stapp03:8089/
Enter fullscreen mode Exit fullscreen mode

In this scenario, all three servers returned connection refused, indicating Apache was not running on any of them.

curl: (7) Failed to connect to stapp01 port 8089: Connection refused
curl: (7) Failed to connect to stapp02 port 8089: Connection refused
curl: (7) Failed to connect to stapp03 port 8089: Connection refused
Enter fullscreen mode Exit fullscreen mode

Step 2: Troubleshoot App Server 1 (stapp01)

Connect to stapp01 and switch to root.

ssh tony@stapp01
sudo su -
Enter fullscreen mode Exit fullscreen mode

Check the Service Status 

systemctl status httpd
Enter fullscreen mode Exit fullscreen mode

The output showed the service was inactive.

○ httpd.service - The Apache HTTP Server
     Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled)
     Active: inactive (dead)
Enter fullscreen mode Exit fullscreen mode

Check the Configuration for Syntax Errors 

httpd -t
Enter fullscreen mode Exit fullscreen mode

The output revealed the root cause.

AH00526: Syntax error on line 47 of /etc/httpd/conf/httpd.conf:
Invalid command 'Listen 808989', perhaps misspelled or defined by a module not included in the server configuration
Enter fullscreen mode Exit fullscreen mode

The Listen directive had a typo: Listen 808989 instead of Listen 8089.

Examine the Problematic Line 

sed -n '40,55p' /etc/httpd/conf/httpd.conf
Enter fullscreen mode Exit fullscreen mode

The line was:

"Listen 808989"
Enter fullscreen mode Exit fullscreen mode

The line had quotes around it, which prevented standard sed commands from matching it.

Check DocumentRoot 

grep "DocumentRoot" /etc/httpd/conf/httpd.conf
Enter fullscreen mode Exit fullscreen mode

The output showed:

DocumentRoot /var/www/html;
Enter fullscreen mode Exit fullscreen mode

The semicolon at the end made the path invalid.

Fix the Configuration

Remove the problematic line and ensure correct directives.

# Remove the line containing "Listen 808989"
sed -i '/Listen 808989/d' /etc/httpd/conf/httpd.conf

# Remove any other Listen lines with quotes
sed -i '/"Listen/d' /etc/httpd/conf/httpd.conf

# Ensure correct Listen directive
grep -q "^Listen 8089" /etc/httpd/conf/httpd.conf || echo "Listen 8089" >> /etc/httpd/conf/httpd.conf

# Fix DocumentRoot
sed -i 's|^DocumentRoot.*|DocumentRoot "/var/www/html"|' /etc/httpd/conf/httpd.conf
Enter fullscreen mode Exit fullscreen mode

Verify the Fix 

grep -n "^Listen" /etc/httpd/conf/httpd.conf
grep -n "^DocumentRoot" /etc/httpd/conf/httpd.conf
httpd -t
Enter fullscreen mode Exit fullscreen mode

Expected output:

358:Listen 8089
124:DocumentRoot "/var/www/html"
Syntax OK
Enter fullscreen mode Exit fullscreen mode

Start and Enable Apache 

systemctl start httpd
systemctl enable httpd
systemctl status httpd
Enter fullscreen mode Exit fullscreen mode

The service started successfully.

● httpd.service - The Apache HTTP Server
     Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled)
     Active: active (running)
   Main PID: 11989 (httpd)
     Status: "Started, listening on: port 8089"
Enter fullscreen mode Exit fullscreen mode

Step 3: Troubleshoot App Server 2 (stapp02)

Connect to stapp02 and switch to root.

ssh steve@stapp02
sudo su -
Enter fullscreen mode Exit fullscreen mode

Check the Configuration 

grep -n "^Listen" /etc/httpd/conf/httpd.conf
grep -n "^DocumentRoot" /etc/httpd/conf/httpd.conf
grep -n "^ServerRoot" /etc/httpd/conf/httpd.conf
Enter fullscreen mode Exit fullscreen mode

The output showed two issues:

47:Listen 808989
124:DocumentRoot "/var/www/html"
34:ServerRoot "/etc/httpd;"
Enter fullscreen mode Exit fullscreen mode

The Listen directive had a typo, and the ServerRoot had a semicolon making it invalid.

Check the Syntax 

httpd -t
Enter fullscreen mode Exit fullscreen mode

The output revealed the ServerRoot issue.

httpd: Syntax error on line 34 of /etc/httpd/conf/httpd.conf: ServerRoot must be a valid directory
Enter fullscreen mode Exit fullscreen mode

Fix the Configuration 

# Remove the problematic Listen line
sed -i '/Listen 808989/d' /etc/httpd/conf/httpd.conf
sed -i '/"Listen/d' /etc/httpd/conf/httpd.conf

# Ensure correct Listen directive
grep -q "^Listen 8089" /etc/httpd/conf/httpd.conf || echo "Listen 8089" >> /etc/httpd/conf/httpd.conf

# Fix ServerRoot
sed -i 's|^ServerRoot.*|ServerRoot "/etc/httpd"|' /etc/httpd/conf/httpd.conf

# Fix DocumentRoot
sed -i 's|^DocumentRoot.*|DocumentRoot "/var/www/html"|' /etc/httpd/conf/httpd.conf

# Ensure index.html
echo "Welcome to xFusionCorp Industries" > /var/www/html/index.html
Enter fullscreen mode Exit fullscreen mode

Verify and Start Apache 

grep -n "^ServerRoot" /etc/httpd/conf/httpd.conf
grep -n "^Listen" /etc/httpd/conf/httpd.conf
grep -n "^DocumentRoot" /etc/httpd/conf/httpd.conf
httpd -t
systemctl start httpd
systemctl enable httpd
systemctl status httpd
Enter fullscreen mode Exit fullscreen mode

The service started successfully.

● httpd.service - The Apache HTTP Server
     Active: active (running)
     Status: "Started, listening on: port 8089"
Enter fullscreen mode Exit fullscreen mode

Step 4: Troubleshoot App Server 3 (stapp03)

Connect to stapp03 and switch to root.

ssh banner@stapp03
sudo su -
Enter fullscreen mode Exit fullscreen mode

Check the Configuration 

grep -n "^ServerRoot" /etc/httpd/conf/httpd.conf
grep -n "^Listen" /etc/httpd/conf/httpd.conf
grep -n "^DocumentRoot" /etc/httpd/conf/httpd.conf
Enter fullscreen mode Exit fullscreen mode

Apply the Same Fixes 

# Fix ServerRoot
sed -i 's|^ServerRoot.*|ServerRoot "/etc/httpd"|' /etc/httpd/conf/httpd.conf

# Fix Listen
sed -i '/Listen 808989/d' /etc/httpd/conf/httpd.conf
sed -i '/"Listen/d' /etc/httpd/conf/httpd.conf
grep -q "^Listen 8089" /etc/httpd/conf/httpd.conf || echo "Listen 8089" >> /etc/httpd/conf/httpd.conf

# Fix DocumentRoot
sed -i 's|^DocumentRoot.*|DocumentRoot "/var/www/html"|' /etc/httpd/conf/httpd.conf

# Ensure index.html
echo "Welcome to xFusionCorp Industries" > /var/www/html/index.html

# Verify
grep -n "^ServerRoot" /etc/httpd/conf/httpd.conf
grep -n "^Listen" /etc/httpd/conf/httpd.conf
grep -n "^DocumentRoot" /etc/httpd/conf/httpd.conf

# Check syntax
httpd -t

# Start and enable
systemctl start httpd
systemctl enable httpd
systemctl status httpd
Enter fullscreen mode Exit fullscreen mode

The service started successfully.

Step 5: Verify from Jump Host

Return to the jump host and test all servers.

curl http://stapp01:8089/
curl http://stapp02:8089/
curl http://stapp03:8089/
Enter fullscreen mode Exit fullscreen mode

Expected output:

Welcome to xFusionCorp Industries!
Welcome to xFusionCorp Industries
Welcome to xFusionCorp Industries
Enter fullscreen mode Exit fullscreen mode

All three servers returned the static page content successfully.

Summary of Issues and Fixes

Server Issue Fix
stapp01 "Listen 808989" typo with quotes Removed line, added Listen 8089
stapp01 DocumentRoot /var/www/html; Fixed to DocumentRoot "/var/www/html"
stapp02 Listen 808989 typo Removed line, added Listen 8089
stapp02 ServerRoot "/etc/httpd;" Fixed to ServerRoot "/etc/httpd"
stapp03 Same issues as stapp02 Applied same fixes

Verification Checklist

Check stapp01 stapp02 stapp03
Apache running Active Active Active
Port 8089 Listening Listening Listening
DocumentRoot /var/www/html /var/www/html /var/www/html
Index page Present Present Present
curl test 200 OK 200 OK 200 OK

Common Apache Configuration Errors

Error Cause Fix
Invalid command 'Listen ...' Typo in port number Correct the Listen directive
ServerRoot must be a valid directory Semicolon or wrong path Set ServerRoot "/etc/httpd"
DocumentRoot not working Semicolon in path Set DocumentRoot "/var/www/html"
Service fails to start Syntax error Run httpd -t to identify
Port already in use Another service on port Change port or stop conflicting service

Best Practices

  1. Always run httpd -t before restarting Apache to catch syntax errors.
  2. Use grep -n to view configuration lines with line numbers.
  3. Remove problematic lines entirely rather than trying to edit them in place.
  4. Verify the ServerRoot, Listen, and DocumentRoot directives are correct.
  5. Ensure the index.html file exists in the document root.
  6. Check the service status after starting Apache.
  7. Test from a remote host to confirm the service is reachable.

Useful Commands

Task Command
Check syntax httpd -t
Check service systemctl status httpd
Start service systemctl start httpd
Enable service systemctl enable httpd
Check listening ports `ss -tlnp \
View configuration {% raw %}grep -n "^Listen" /etc/httpd/conf/httpd.conf
Remove a line sed -i '/pattern/d' /etc/httpd/conf/httpd.conf
Replace a line `sed -i 's\

Conclusion

This guide covered a complete Apache troubleshooting workflow on three application servers. The root causes were configuration errors: typos in the Listen directive, semicolons in ServerRoot and DocumentRoot paths, and quotes around directive values.

By systematically checking the service status, configuration syntax, and specific directives, the issues were identified and fixed. After correcting the configuration and restarting Apache, all three servers responded correctly on port 8089.

The key takeaway is that Apache configuration errors are often caused by small typos or stray characters. Using {% raw %}httpd -t to check syntax and grep to inspect specific directives makes troubleshooting efficient and reliable.

Top comments (0)