DEV Community

Janak Shrestha
Janak Shrestha

Posted on

Haproxy LBR Troubleshooting

#haproxy #linux #debugging

xFusionCorp Industries has an application running on Nautlitus infrastructure in Stratos Datacenter. The monitoring tool recognised that there is an issue with the haproxy service on the LBR server. That needs to fixed to make the application work properly.

Troubleshoot and fix the issue, and make sure haproxy service is running on the Nautilus LBR server. Once fixed, make sure you are able to access the website by running the command curl http://stlb01:80/ in the terminal.

Understanding the Environment

Component Details
LBR Server stlb01
LBR User loki
LBR Password Mischi3f
App Server 1 stapp01 (Apache on port 8080)
App Server 2 stapp02 (Apache on port 8080)
App Server 3 stapp03 (Apache on port 8080)
HAProxy Port 80

Step 1: Connect to the LBR Server

Access the load balancer server using SSH.

ssh loki@stlb01
Password: Mischi3f
Enter fullscreen mode Exit fullscreen mode

Switch to root to perform administrative tasks.

sudo su -
Password: Mischi3f
Enter fullscreen mode Exit fullscreen mode

Step 2: Check the HAProxy Service Status

The first step is to check whether the HAProxy service is running.

systemctl status haproxy
Enter fullscreen mode Exit fullscreen mode

In this scenario, the output showed the service had failed.

× haproxy.service - HAProxy Load Balancer
     Loaded: loaded (/usr/lib/systemd/system/haproxy.service; enabled; preset: disabled)
    Drop-In: /etc/systemd/system/haproxy.service.d
             └─override.conf
     Active: failed (Result: exit-code)
Enter fullscreen mode Exit fullscreen mode

The service was in a failed state, indicating a configuration or runtime issue.

Step 3: Check HAProxy Logs

Check the system logs for HAProxy to identify the root cause.

journalctl -u haproxy -n 50
Enter fullscreen mode Exit fullscreen mode

The logs showed the service was trying to execute the haproxy binary with a configuration file but failing with an exit code.

haproxy.service: Executing: /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg ...
haproxy.service: Control process exited, code=exited, status=...
haproxy.service: Failed with result 'exit-code'.
Enter fullscreen mode Exit fullscreen mode

Step 4: Verify the Configuration Syntax

Run the HAProxy configuration check to identify syntax errors.

haproxy -c -f /etc/haproxy/haproxy.cfg
Enter fullscreen mode Exit fullscreen mode

The output revealed multiple errors.

[ALERT] (3689) : config : parsing [/etc/haproxy/haproxy.cfg:39] : please use the 'bind' keyword for listening addresses.
[ALERT] (3689) : config : parsing [/etc/haproxy/haproxy.cfg:40] : 'listen' or 'defaults' expected.
[ALERT] (3689) : config : parsing [/etc/haproxy/haproxy.cfg:41] : 'listen' or 'defaults' expected.
[ALERT] (3689) : config : parsing [/etc/haproxy/haproxy.cfg:43] : 'listen' or 'defaults' expected.
[ALERT] (3689) : config : parsing [/etc/haproxy/haproxy.cfg:44] : 'listen' or 'defaults' expected.
[ALERT] (3689) : config : parsing [/etc/haproxy/haproxy.cfg:60]: Missing LF on last line, file might have been truncated at position 36.
[ALERT] (3689) : config : Error(s) found in configuration file : /etc/haproxy/haproxy.cfg
[ALERT] (3689) : config : Fatal errors found in configuration.
Enter fullscreen mode Exit fullscreen mode

Root Cause Identified

  1. Deprecated Syntax: The frontend main *:80 line used old syntax. Modern HAProxy requires frontend main followed by bind *:80.

  2. Commented-Out Backend: The backend app section was commented out, leaving orphaned balance and server lines that caused parsing errors.

  3. Missing Newline: The file was truncated without a newline at the end.

Step 5: Examine the Configuration File

View the current configuration to understand the issues.

cat /etc/haproxy/haproxy.cfg
Enter fullscreen mode Exit fullscreen mode

The problematic sections were identified.

frontend  main *:80
    acl url_static       path_beg       -i /static /images /javascript /stylesheets
    ...

#backend app
    balance     roundrobin
    server  app1 stapp01:3000 check
    server  app2 stapp02:3000 check
    server  app3 stapp03:3000 check
Enter fullscreen mode Exit fullscreen mode

The backend app was commented out, and the server definitions were on port 3000.

Step 6: Check the Drop-In Override

A systemd drop-in override was present at /etc/systemd/system/haproxy.service.d/override.conf.

cat /etc/systemd/system/haproxy.service.d/override.conf
Enter fullscreen mode Exit fullscreen mode 
[Service]
ExecStartPre=
ExecStartPre=/usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -c -q
ExecStart=
ExecStart=/usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -S /run/haproxy-master.sock
Enter fullscreen mode Exit fullscreen mode

This override forces HAProxy to use only the main configuration file, which is fine. The issue was in the main configuration itself.

Step 7: Determine the Correct Backend Port

Before fixing the configuration, verify which port the app servers are listening on.

curl -I http://stapp01:8087
curl -I http://stapp02:8087
curl -I http://stapp03:8087
Enter fullscreen mode Exit fullscreen mode

Result: Connection refused on port 8087.

curl -I http://stapp01:8080
curl -I http://stapp02:8080
curl -I http://stapp03:8080
Enter fullscreen mode Exit fullscreen mode

Result: HTTP 200 OK on port 8080.

The app servers are listening on port 8080, not 3000 or 8087.

Step 8: Replace the Configuration with a Valid Version

Replace the configuration file with a clean, valid version.

cat > /etc/haproxy/haproxy.cfg << 'EOF'
global
    log         127.0.0.1 local2
    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon
    stats socket /var/lib/haproxy/stats

defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 3000

frontend main
    bind *:80
    default_backend app

backend app
    balance roundrobin
    server stapp01 stapp01:8080 check
    server stapp02 stapp02:8080 check
    server stapp03 stapp03:8080 check
EOF
Enter fullscreen mode Exit fullscreen mode

Key Changes

Original Fixed
frontend main *:80 frontend main + bind *:80
#backend app (commented) backend app (uncommented)
Orphaned balance and server lines Properly nested under backend app
Port 3000 Port 8080 (correct port)
Missing newline at end File ends with newline

Step 9: Verify the Configuration

Run the configuration check again.

haproxy -c -f /etc/haproxy/haproxy.cfg
Enter fullscreen mode Exit fullscreen mode

Expected output:

Configuration file is valid
Enter fullscreen mode Exit fullscreen mode

Step 10: Restart HAProxy

Restart the service to apply the changes.

systemctl restart haproxy
systemctl status haproxy
Enter fullscreen mode Exit fullscreen mode

The output should show the service is active and running.

● haproxy.service - HAProxy Load Balancer
     Loaded: loaded (/usr/lib/systemd/system/haproxy.service; enabled; preset: disabled)
    Drop-In: /etc/systemd/system/haproxy.service.d
             └─override.conf
     Active: active (running) since Mon 2026-09-21 11:16:59 UTC; 23ms ago
   Main PID: 3968 (haproxy)
     Status: "Ready."
Enter fullscreen mode Exit fullscreen mode

Step 11: Verify Listening Ports

Confirm HAProxy is listening on port 80.

ss -tlnp | grep :80
Enter fullscreen mode Exit fullscreen mode

Expected output:

LISTEN 0 3000 0.0.0.0:80 0.0.0.0:* users:(("haproxy",pid=3982,fd=9))
Enter fullscreen mode Exit fullscreen mode

Step 12: Test the Website

Test the website using curl.

curl http://stlb01:80/
Enter fullscreen mode Exit fullscreen mode

Expected output:

Welcome to xFusionCorp Industries!
Enter fullscreen mode Exit fullscreen mode

Complete Troubleshooting Workflow 

# 1. Connect to the LBR server
ssh loki@stlb01
sudo su -

# 2. Check service status
systemctl status haproxy

# 3. Check logs
journalctl -u haproxy -n 50

# 4. Check configuration syntax
haproxy -c -f /etc/haproxy/haproxy.cfg

# 5. Examine the configuration file
cat /etc/haproxy/haproxy.cfg

# 6. Check the drop-in override
cat /etc/systemd/system/haproxy.service.d/override.conf

# 7. Determine the correct backend port
curl -I http://stapp01:8080
curl -I http://stapp02:8080
curl -I http://stapp03:8080

# 8. Replace configuration with valid version
cat > /etc/haproxy/haproxy.cfg << 'EOF'
... (valid configuration) ...
EOF

# 9. Verify configuration
haproxy -c -f /etc/haproxy/haproxy.cfg

# 10. Restart HAProxy
systemctl restart haproxy
systemctl status haproxy

# 11. Verify listening ports
ss -tlnp | grep :80

# 12. Test the website
curl http://stlb01:80/
Enter fullscreen mode Exit fullscreen mode

Common HAProxy Issues and Fixes

Issue Symptom Fix
Deprecated syntax please use the 'bind' keyword Use frontend main + bind *:80
Commented backend 'listen' or 'defaults' expected Uncomment backend app
Wrong backend port 503 Service Unavailable Verify port with curl and update config
Missing newline Missing LF on last line Ensure file ends with a newline
Duplicate sections Fatal errors Ensure unique section names
Port conflict Service fails to start Check `ss -tlnp \
Stats socket missing Service fails Ensure {% raw %}stats socket /var/lib/haproxy/stats present

Verification Checklist

Check Command Expected
Configuration valid haproxy -c -f /etc/haproxy/haproxy.cfg Configuration file is valid
Service running systemctl status haproxy active (running)
Listening on port 80 `ss -tlnp \ grep :80`
Website accessible curl http://stlb01:80/ Welcome message
Backends reachable curl -I http://stapp01:8080 HTTP 200 OK
Logs clean journalctl -u haproxy -n 30 No errors

Best Practices

  1. Always verify configuration before restarting: Use haproxy -c -f to check syntax.

  2. Use modern syntax: Use bind instead of the deprecated frontend name *:port syntax.

  3. Verify backend ports: Test each backend with curl before configuring HAProxy.

  4. Keep the stats socket: The stats socket /var/lib/haproxy/stats line is required for monitoring.

  5. Check logs regularly: Use journalctl -u haproxy to identify issues early.

  6. Maintain backups: Back up the configuration before making changes.

  7. Document changes: Keep a record of configuration changes for auditing.

Conclusion

This guide covered a complete HAProxy troubleshooting workflow. The main issues were:

  1. Deprecated frontend syntax that caused parsing errors.
  2. Commented-out backend that left orphaned directives.
  3. Incorrect backend port (3000 instead of 8080) that caused 503 errors.

By systematically checking the service status, logs, and configuration, the root cause was identified and fixed. After replacing the configuration with a valid version, HAProxy started successfully and the website became accessible.

The key takeaways are to always verify configuration syntax before restarting, use modern HAProxy syntax, and confirm backend ports by testing them directly.

Quick Reference

Task Command
Check config haproxy -c -f /etc/haproxy/haproxy.cfg
Check status systemctl status haproxy
Check logs journalctl -u haproxy -n 50
Check ports `ss -tlnp \
Test website {% raw %}curl http://stlb01:80/
Test backend curl -I http://stapp01:8080
Restart service systemctl restart haproxy

This guide was created based on a real-world HAProxy troubleshooting scenario on the Nautilus LBR server in the Stratos Datacenter.

Top comments (0)