How To: Send Email Alert on Different HAProxy Status or Based on HAProxy Stats

HAProxy is a great simple load balancing tool written in Lua. It is extremely efficient as a software load balancer and highly configurable as well. On the contrary, HAProxy lacks programmable automated monitoring tools. It has a directive called ‘mailer’ which has only support above 1.8. Default CentOS 7 repo comes with HAProxy 1.5 and it has no mailer alert support either. Even with 1.8, it doesn’t come with lots of available configuration options neither the tool gets programmable facility.

That is where, I thought to work on to trigger codes from HAProxy stats. This can be done in many ways, in my cases, I did it using per minute crons. If you want it much quicker like every 5 seconds for example, you would have to run this as a daemon, which isn’t like making a rocket, should be easy and short. My entire idea is to allow you understanding how to create programmable 3rd party tools by fetching data from HAProxy socket and trigger monitors.

HAProxy Stats through Unix Socket

First, we need to enable the HAProxy stats that is available through socket. To turn on stats through unix socket, you need put the following line in your global section of haproxy.cfg file:

stats socket /var/lib/haproxy/stats

An example of Global settings section would be like the following:

global
    log         127.0.0.1 local2     #Log configuration

    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     40000
    user        haproxy             #Haproxy running under user and group "haproxy"
    group       haproxy
    daemon

    # turn on stats unix socket
    stats socket /var/lib/haproxy/stats

Check how the stats socket section is placed to fit it for your cfg file.

Once this is done, now you can restart the haproxy to start shooting stats through the socket. The output is basically a csv of the HAProxy stats page. So the values going to be in comma seperated format. To understand HAProxy stats page and exploding the values, you can visit the following:

https://www.haproxy.com/blog/exploring-the-haproxy-stats-page/

Now, how to read the unix-socket using bash? There is a tool called ‘socat’ that can be used to read data from unix socket. ‘socat’ means ‘socket cat’, you may read more details about ‘socat’ here:

https://linux.die.net/man/1/socat

If socat is not available on your CentOS yum, you may get it from epel-release.

yum install epel-release
yum install socat

Once ‘socat’ is available in your system, you can use it to redirect the io and show the output as following:

echo "show stat" | socat unix-connect:/var/lib/haproxy/stats stdio

Now as you can see, you can retrieve the whole HAProxy stats in CSV format, you may easily manipulate and operate data using a shell script. I have created a basic shell script to get the status of the HAProxy backends and send an email alert using ‘ssmtp’. Remember, ssmtp is highly configurable mail tool, you can customize smtp authentication as well with ‘ssmtp’. You may use any other tool like Sendmail for example or ‘Curl’ to any email API like Sendgrid, possibility is infinite here. Remember, as the data is instantly available to socket as soon as the HAProxy generates the event, hence, it can be as efficient as HAProxy built in functions like ‘mailer’ is.

#!/bin/bash

cd /root/
rm -f haproxy_stat.txt
echo "show stat" | socat unix-connect:/var/lib/haproxy/stats stdio|grep app-main > /root/test.txt

SEND_EMAIL=0

while IFS= read -r line
do

APP=`echo $line | cut -d"," -f2`
STAT=`echo $line | cut -d"," -f18`
SESSION=`echo $line | cut -d"," -f34`
if [ "$STAT" != "UP" ]; then
SEND_EMAIL=1
MESSAGE+="$APP $STAT $SESSION
"
fi

done < /root/test.txt

if [ $SEND_EMAIL -eq 1 ]; then
echo -e "Subject: Haproxy Instance Down \n\n$MESSAGE" | sudo ssmtp -vvv [email protected]
echo -e "Subject: Haproxy Instance Down \n\n$MESSAGE" | sudo ssmtp -vvv [email protected]

fi

Data that I am interested in are the status of the backend, name of the backend and the session rate of the backend. So, if the load balancer sees any backend is down, this would trigger the email delivery. You can use this to catch anything in the HAProxy, like a Frontend attack for example, like the delivery optimization of your load balancer etc. As you are now able to retrieve data directly from HAProxy to your own ‘programming’ console, you can program it whatever the way you want to. Hope this helps somebody! For any help, shoot a comment!

Leave a Reply Cancel reply