HAProxy is a great simple load balancing tool written in Lua. It is extremely efficient as a software load balancer and highly configurable as well. On the contrary, HAProxy lacks programmable automated monitoring tools. It has a directive called ‘mailer’ which has only support above 1.8. Default CentOS 7 repo comes with HAProxy 1.5 and it has no mailer alert support either. Even with 1.8, it doesn’t come with lots of available configuration options neither the tool gets programmable facility.
That is where, I thought to work on to trigger codes from HAProxy stats. This can be done in many ways, in my cases, I did it using per minute crons. If you want it much quicker like every 5 seconds for example, you would have to run this as a daemon, which isn’t like making a rocket, should be easy and short. My entire idea is to allow you understanding how to create programmable 3rd party tools by fetching data from HAProxy socket and trigger monitors.
HAProxy Stats through Unix Socket
First, we need to enable the HAProxy stats that is available through socket. To turn on stats through unix socket, you need put the following line in your global section of haproxy.cfg file:
stats socket /var/lib/haproxy/stats
An example of Global settings section would be like the following:
global
log 127.0.0.1 local2 #Log configuration
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 40000
user haproxy #Haproxy running under user and group "haproxy"
group haproxy
daemon
# turn on stats unix socket
stats socket /var/lib/haproxy/stats
Check how the stats socket section is placed to fit it for your cfg file.
Once this is done, now you can restart the haproxy to start shooting stats through the socket. The output is basically a csv of the HAProxy stats page. So the values going to be in comma seperated format. To understand HAProxy stats page and exploding the values, you can visit the following:
https://www.haproxy.com/blog/exploring-the-haproxy-stats-page/
Now, how to read the unix-socket using bash? There is a tool called ‘socat’ that can be used to read data from unix socket. ‘socat’ means ‘socket cat’, you may read more details about ‘socat’ here:
https://linux.die.net/man/1/socat
If socat is not available on your CentOS yum, you may get it from epel-release.
yum install epel-release
yum install socat
Once ‘socat’ is available in your system, you can use it to redirect the io and show the output as following:
echo "show stat" | socat unix-connect:/var/lib/haproxy/stats stdio
Now as you can see, you can retrieve the whole HAProxy stats in CSV format, you may easily manipulate and operate data using a shell script. I have created a basic shell script to get the status of the HAProxy backends and send an email alert using ‘ssmtp’. Remember, ssmtp is highly configurable mail tool, you can customize smtp authentication as well with ‘ssmtp’. You may use any other tool like Sendmail for example or ‘Curl’ to any email API like Sendgrid, possibility is infinite here. Remember, as the data is instantly available to socket as soon as the HAProxy generates the event, hence, it can be as efficient as HAProxy built in functions like ‘mailer’ is.
#!/bin/bash
cd /root/
rm -f haproxy_stat.txt
echo "show stat" | socat unix-connect:/var/lib/haproxy/stats stdio|grep app-main > /root/test.txt
SEND_EMAIL=0
while IFS= read -r line
do
APP=`echo $line | cut -d"," -f2`
STAT=`echo $line | cut -d"," -f18`
SESSION=`echo $line | cut -d"," -f34`
if [ "$STAT" != "UP" ]; then
SEND_EMAIL=1
MESSAGE+="$APP $STAT $SESSION
"
fi
done < /root/test.txt
if [ $SEND_EMAIL -eq 1 ]; then
echo -e "Subject: Haproxy Instance Down \n\n$MESSAGE" | sudo ssmtp -vvv [email protected]
echo -e "Subject: Haproxy Instance Down \n\n$MESSAGE" | sudo ssmtp -vvv [email protected]
fi
Data that I am interested in are the status of the backend, name of the backend and the session rate of the backend. So, if the load balancer sees any backend is down, this would trigger the email delivery. You can use this to catch anything in the HAProxy, like a Frontend attack for example, like the delivery optimization of your load balancer etc. As you are now able to retrieve data directly from HAProxy to your own ‘programming’ console, you can program it whatever the way you want to. Hope this helps somebody! For any help, shoot a comment!
