loading...

HAProxy watch status when no web browser or syslog!

schollii profile image schollii ・2 min read

The other day I got a challenge: I had to 1) do a basic setup of haproxy so one frontend would round-robin on 3 backends (simple task), but 2) verify that it was actually round-robin'ing, without using a web browser (to access the stats page that haproxy can be
configured to serve, eg see “HAProxy Stats Page” on https://www.haproxy.com/blog/introduction-to-haproxy-logging/) or syslog.

I tried curl which technically is not a web browser, but html was horribly formatted and didn't seem to have the backend status up/down etc; and converting to markdown and plain text with pandoc did not help.

This won't be news to those who work with haproxy all the time, but for those starting:

Turns out the haproxy makes status available through a unix socket that can be configured in the global section of /etc/haproxy/haproxy.cfg. On my centOS it defaults to /var/lib/haproxy/stats, whereas on ubuntu it defaults to /run/haproxy/admin.sock.

After a bit of research I found it was sufficient to send the show stat command to the socket, and then process the output. Even better,

watch 'echo "show stat" | sudo nc -U /var/lib/haproxy/stats | cut -d "," -f 1,2,8-10,18 | column -s, -t'

on the load balancer host (LB_HOST) to get the stats. Then in a separate terminal (same or different host), I ran

while true; do curl LB_HOST:HAPROXY_PORT; sleep 1; done 

to generate http requests on the load balancer, and in a third terminal I could add/remove backend servers and observe the load-balancer status up/down. Example output of the watched command:

$ echo "show stat" | sudo nc -U /run/haproxy/admin.sock | cut -d "," -f 1,2,8-10,18 | column -s, -t
# pxname       svname    stot  bin    bout   status
test_frontend  FRONTEND  275   21725  67755  OPEN
test_backend   server1   158   12482  39026  UP
test_backend   server2   93    6399   19871  DOWN
test_backend   server3   39    2844   8858   DOWN
test_backend   BACKEND   275   21725  67755  UP

Pretty cool!

You can even have a "prompt" where you can do various things other than stats, like take down one of the backend servers for maintenance, debug certificates/SSL, etc.

Eg with only servers 1 and 2 up (ie 3 is down), I disabled 1 and 2 in haproxy for maintenance, by doing this:

$ sudo nc -U /run/haproxy/admin.sock
prompt
> disable server test_backend/server1
> disable server test_backend/server2

then the watch automatically updated to show this:

# pxname       svname    stot  bin    bout    status
test_frontend  FRONTEND  674   53246  164642  OPEN
test_backend   server1   472   37288  116584  MAINT
test_backend   server2   148   11692  36556   MAINT
test_backend   server3   0     0      0       DOWN
test_backend   BACKEND   674   53246  164642  DOWN

Notice now the whole BACKEND is down too, because of server 3. Start server 3 and observe its state going from DOWN to UP, and then that of BACKEND does same.

Enjoy!

A couple of useful refs:

Posted on by:

schollii profile

schollii

@schollii

Apple IIe -> Distributed computing (C/C++, Python, Lua, Java, C#, Javascript) -> DevOps (Docker, Kubernetes, Helm, Terraform, Spinnaker, AWS)

Discussion

markdown guide