Can You Cancel/Abort/Rollback a HTTP Request?

There are cases, where Frontend developers wants to cancel the REST Requests that they submit using Ajax. It is one of the popular question among the Frontend developers community. Let’s try to focus on this case today.

To answer the question in short, No, you can not cancel a HTTP Request once it is submitted. There are catches require shear understanding here. Let’s elaborate a bit.

HTTP is Stateless by Design

HTTP is stateless by design, so what happens when there is no state for a request? It means we do not know the destination, what is happening with that specific requests at the back, but only can acknowledge the response. For example, if you have a REST API behind a load balancer. If you put a POST request to the API, you will not know which server going to serve your requests out of many HTTP servers behind the load balancer. For such cases, we create ‘cookies’ or ‘sessions’ on our application end for our full fledged web applications and extend the functionality of HTTP, to store state of a request and following it. For each form request for example, we submit a state along with the app, to let load balancer understand the state and follow me to the destination. Isn’t it? But we have to remember, this is not a HTTP feature, this is done at application level to add statefullness in HTTP requests.

What does this mean? It means, as there is no state for REST, we are not aware of the destination here, but only the response. So how can we cancel a requests, whose actual destination we are not aware of? Now you might think that, how about adding a cookies and cancel it? Yes, we do that on web applications layer, but can REST HTTP Requests be stateful? Not really, that is not by design of HTTP or REST unfortunately.

So, Does That Mean We Can Never Cancel an API Call?

If you are using REST, then it means you can never cancel the API call. If it gets submitted, it will continue processing in the backend even if you stop it in the frontend. But if ‘Cancelling’ is much important for your application through the API call, then you must consider other alternatives, like SOAP or RPC. RPC is stateful API architecture, and it is possible to design a cancel request for this. Please note, RPC doesn’t implement ‘cancel’ by default, but as this is stateful, you are able to design a ‘cancel’ request with the RPC call. Google has a RPC called ‘gRPC’, which is a stateful API architecture. That means, it is possible for you to implement cancel/abort or event rollback/restore a state with gRPC.

A Google application called ‘Firestore’ database has support for gRPC which is basically a stateful version of stateless REST API of Firestore.

How To: Manually Add Support of SSL for WWW on Cyberpanel

hmm, it’s a weird topic to write blog on. Because Cyberpanel comes with a built in Certbot, and can automatically detects www and without www to install SSL for. Then why am I writing this up? All because I found a VPS client today facing the issue. Even though, Cyberpanel was telling me that the SSL is issued, it was only issued for non-www domain, but the www domain left behind. Let’s see how can we resolve this.

First problem

First problem came up when I tried to discover the Cyberpanel certbot binaries.

[root@server-sg /]# find . -name "certbot"
./usr/local/CyberCP/bin/certbot
./usr/local/CyberCP/lib/python3.6/site-packages/certbot
./usr/local/CyberPanel/bin/certbot
./usr/local/CyberPanel/lib/python3.6/site-packages/certbot

[root@server-sg live]# /usr/local/CyberCP/bin/certbot --version
certbot 0.21.1
[root@server-sg live]# /usr/local/CyberPanel/bin/certbot --version
certbot 0.21.1

Both of the certbot I could find from Cyberpanel was very old, Certbot has 1.4 version in the Epel which has support for Acme 2 challenge, while the one that Cyberpanel is using doesn’t. I hence decided to install a certbot for our case:

yum install epel-release
yum install certbot

These should be it for the latest version of certbot to start working in your Cyberpanel host. Once done, you may now generate the SSL using the following:

certbot certonly  --webroot -w /home/yourdomain.com/public_html -d yourdomain.com -d www.yourdomain.com

Remember to replace yourdomain.com with the actual one that is having problem with. Cyberpanel creates the home directory with the primary domain, so the remember to give the correct document root for the value of attribute ‘-w’.

Once this id done, certbot should automatically verify the challenge and get the issued license for you. Lets encrypt license are usually stored at the following directory:

/etc/letsencrypt/live/yourdomain.com/

Files are:
/etc/letsencrypt/live/yourdomain.com/privatekey.pem
/etc/letsencrypt/live/yourdomain.com/fullchain.pem

If you had already created the SSL using Cyberpanel (which you must have done if you viewing this post), then remember, certbot will place the SSLs in /etc/letsencrypt/live/yourdomain.com-001/ folder. The name of the folder would be shown at the time you complete issuing SSL with certbot.

There are couple of ways you may use the SSL now. Either you may replace the old directory with the new, or just change the settings in either the vhost conf or the openlitespeed SSL settings. I find the easiest way is just to replace the old directory with the new. Something like this should work:

mv /etc/letsencrypt/live/yourdomain.com /etc/letsencrypt/live/old_yourdomain.com
mv /etc/letsencrypt/live/yourdomain.com-001 /etc/letsencrypt/live/yourdomain.com

Once this is done, remember to restart your openlitespeed:

service lsws restart

Now your https on the WWW should work without any problem. If not, try clearing your browser cache and retry.

How To: Add Let’s Encrypt SSL in HAProxy – TLS Termination

HAProxy stays in the middle of origin server and the visitors. Hence, You need a SSL for the Visitors to HAProxy. You can use HAProxy is a secure private network to fetch data from backend without any SSL. But the requests between the visitor and HAProxy has to be encrypted. You can use Let’s Encrypt free signed SSL for this purpose.

First, we need to install ‘certbot’, python based client for Let’s Encrypt SSL. It is available in epel repository. In CentOS, you may do the following to install certbot

$ yum install epel-release
$ yum install certbot

Let’s Encrypt uses a Challenge Response technique to verify the host and issue the SSL. While HAProxy is enabled, and used to set to the origin service, this unfortunately, is not possible. certbot comes with an option called ‘standalone’, where it can work as a http server and resolve the Challenge Response issued by Let’s Encrypt. To do this, first we need to stop the haproxy server. You can do this with the following:

# stop haproxy
service haproxy stop

# get the ssl for your domain.com and www.domain.com
certbot certonly --standalone --preferred-challenges http --http-01-port 80 -d www.domain.com -d domain.com

Once this is done, 4 files are saved under /etc/letsencrypt/live/domain.com/

These should be:

cert.pem (Your certificate)
chain.pem
privatekey.pem (Your private key)
fullchain.pem (cert.pem and chain.pem combined)

Now, for haproxy, we need to combine 3 files, cert.pem, chain.pem and privatekey.pem, we can do that by combining fullchain.pem & privatekey.pem. You need to create a directory under /etc/haproxy/certs and then put the file in there. You can do that as following:

# create the directory
mkdir /etc/haproxy/certs

# Combine two files into one in one line
DOMAIN='domain.com' sudo -E bash -c 'cat /etc/letsencrypt/live/$DOMAIN/fullchain.pem /etc/letsencrypt/live/$DOMAIN/privkey.pem > /etc/haproxy/certs/$DOMAIN.pem'

# replace domain.com with each of your domain.

Now, we have the pem file ready to be used on haproxy frontend. To use, you may first edit the haproxy.cfg file, create a new section for frontend https, and use the certificate. An example is given below

frontend main_https
    bind *:443 ssl crt /etc/haproxy/certs/domain.com.pem
    reqadd X-Forwarded-Proto:\ https
    option http-server-close
    option forwardfor
    default_backend app-main

Once the https section is done, you may now want to force the http section to forward to https, you can do as following:

frontend main
    bind *:80
    redirect scheme https code 301 if !{ ssl_fc }
    option http-server-close
    option forwardfor

You should be all set now using Let’s Encrypt with your Haproxy in the frontend.

How to Renew Litespeed Trial Without Going to WHM

Litespeed requires you to download the Trial key to try out their trial or use the WHM to do that. If you want to skip the process and fan of console, here is the easiest way to do this step by step:

$ cd /usr/local/lsws/conf
$ service lsws stop
$ mv trial.key trial.key_backup
$ wget --no-check-certificate https://license2.litespeedtech.com/reseller/trial.key
$ chown root:nobody trial.key
$ service lsws start
$ /usr/local/lsws/bin/lshttpd -t
[OK] Your trial license key will expire in 14 days!

Remember, this can only be done twice, the first month only. So you would definitely like to purchase the license and register it with the purchased key after your trial ends. Kudos!

How To: Send Email Alert on Different HAProxy Status or Based on HAProxy Stats

HAProxy is a great simple load balancing tool written in Lua. It is extremely efficient as a software load balancer and highly configurable as well. On the contrary, HAProxy lacks programmable automated monitoring tools. It has a directive called ‘mailer’ which has only support above 1.8. Default CentOS 7 repo comes with HAProxy 1.5 and it has no mailer alert support either. Even with 1.8, it doesn’t come with lots of available configuration options neither the tool gets programmable facility.

That is where, I thought to work on to trigger codes from HAProxy stats. This can be done in many ways, in my cases, I did it using per minute crons. If you want it much quicker like every 5 seconds for example, you would have to run this as a daemon, which isn’t like making a rocket, should be easy and short. My entire idea is to allow you understanding how to create programmable 3rd party tools by fetching data from HAProxy socket and trigger monitors.

HAProxy Stats through Unix Socket

First, we need to enable the HAProxy stats that is available through socket. To turn on stats through unix socket, you need put the following line in your global section of haproxy.cfg file:

stats socket /var/lib/haproxy/stats

An example of Global settings section would be like the following:

global
    log         127.0.0.1 local2     #Log configuration

    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     40000
    user        haproxy             #Haproxy running under user and group "haproxy"
    group       haproxy
    daemon

    # turn on stats unix socket
    stats socket /var/lib/haproxy/stats

Check how the stats socket section is placed to fit it for your cfg file.

Once this is done, now you can restart the haproxy to start shooting stats through the socket. The output is basically a csv of the HAProxy stats page. So the values going to be in comma seperated format. To understand HAProxy stats page and exploding the values, you can visit the following:

https://www.haproxy.com/blog/exploring-the-haproxy-stats-page/

Now, how to read the unix-socket using bash? There is a tool called ‘socat’ that can be used to read data from unix socket. ‘socat’ means ‘socket cat’, you may read more details about ‘socat’ here:

https://linux.die.net/man/1/socat

If socat is not available on your CentOS yum, you may get it from epel-release.

yum install epel-release
yum install socat

Once ‘socat’ is available in your system, you can use it to redirect the io and show the output as following:

echo "show stat" | socat unix-connect:/var/lib/haproxy/stats stdio

Now as you can see, you can retrieve the whole HAProxy stats in CSV format, you may easily manipulate and operate data using a shell script. I have created a basic shell script to get the status of the HAProxy backends and send an email alert using ‘ssmtp’. Remember, ssmtp is highly configurable mail tool, you can customize smtp authentication as well with ‘ssmtp’. You may use any other tool like Sendmail for example or ‘Curl’ to any email API like Sendgrid, possibility is infinite here. Remember, as the data is instantly available to socket as soon as the HAProxy generates the event, hence, it can be as efficient as HAProxy built in functions like ‘mailer’ is.

#!/bin/bash

cd /root/
rm -f haproxy_stat.txt
echo "show stat" | socat unix-connect:/var/lib/haproxy/stats stdio|grep app-main > /root/test.txt

SEND_EMAIL=0

while IFS= read -r line
do

APP=`echo $line | cut -d"," -f2`
STAT=`echo $line | cut -d"," -f18`
SESSION=`echo $line | cut -d"," -f34`
if [ "$STAT" != "UP" ]; then
SEND_EMAIL=1
MESSAGE+="$APP $STAT $SESSION
"
fi

done < /root/test.txt

if [ $SEND_EMAIL -eq 1 ]; then
echo -e "Subject: Haproxy Instance Down \n\n$MESSAGE" | sudo ssmtp -vvv myownemail@gmail.com
echo -e "Subject: Haproxy Instance Down \n\n$MESSAGE" | sudo ssmtp -vvv otheremail@gmail.com

fi

Data that I am interested in are the status of the backend, name of the backend and the session rate of the backend. So, if the load balancer sees any backend is down, this would trigger the email delivery. You can use this to catch anything in the HAProxy, like a Frontend attack for example, like the delivery optimization of your load balancer etc. As you are now able to retrieve data directly from HAProxy to your own ‘programming’ console, you can program it whatever the way you want to. Hope this helps somebody! For any help, shoot a comment!

How To: Force HTTPS in HAProxy

In Haproxy for frontend, we have to listen to both 80 and 443 port for HTTP and HTTPS. But what if we want to force redirect all requests to https? HAProxy doesn’t support things like htaccess/mod_rewrite. So we have to do it using HAProxy directives and attributes.

HAProxy has a directive called ‘ssl_fc’. This one returns true if the HAProxy frontend is on https. We can use this to force redirect reqeusts to HTTPS as following:

#redirect to HTTPS if ssl_fc is false / off.
redirect scheme https code 301 if !{ ssl_fc }

You can add this code to the section where you have defined the frontend for 80.

Now, you can also redirect reqeusts to https based on the requested domain as following:

redirect scheme https code 301 if { hdr(Host) -i www.yourdomain.com } !{ ssl_fc }

Replace your domain with your expected domain name.

How to Add Openlitespeed Server to Haproxy – Avoid 503 Haproxy Error

For past one month, Openlitespeed has been my favorite piece of web server. Litespeed has always outperformed all the other webservers including Nginx as well in any of my production environment. But I have recently switched to using OLS which is a Opensource version of Litespeed with some limited features. I get LS kind of performance along with no worry for paying. How better could it be?

OLS comes with some weird problem. As OLS is less used, finding a solution for such cases could be difficult. I faced a very similar kind of issue yesterday.

I added a OLS based server to my HAProxy cluster, but the HAProxy can not find the OLS server working. When I try to access the web app hosted under OLS server using local IP masking, I see the website without a problem. That means, OLS is interpreting the Domain with IP relation well. But failing to respond when Haproxy is requesting through IP address.

The problem is, OLS is not configured to respond to ‘default’ requests on ‘127.0.0.1’, ‘localhost’ or the server’s main IP. To find out this, I enabled ‘High’ Debug mode of OLS. To do this, first visit the OLS Webadmin Console, it can be accessed with https://IP:7080

After login, go to Server Configuration >> Log >> Edit Server Log >> Set ‘Debug Level’ to High and Save

Set high debug level in openlitespeed

Once saving is done, you may gracefully restart the OLS

Gracefully restart Openlitespeed

Once this is done, you may now monitor the error.log file located usually under /usr/local/lsws/logs. Now tail the output of error.log while processing requests with Haproxy:

tail -f /usr/local/lsws/logs/error.log

You can see, OLS has returned 404 error for the localhost/ request. That means, Haproxy is requesting the IP with a header ‘localhost/’, and the server should return something with code 200 to make sure the server is in business.

What we need to do, is to make OLS respond to request for basic IP and localhost to 200 with the main site instead of ‘404’ error. To do this, we need to go to Webconsole of OLS again >> Listeners

You will see you have two Listeners, one for Default/Non HTTP and the HTTPS/SSL. In my case, I was using only HAProxy to Origin with no SSL, means 80. I selected the Default.

Open 80 Listener View in Openlitespeed

In the Listener List, you can find your Virtualhost, click on the ‘Edit’ of your Virtualhost

Virtualhost Edit Openlitespeed

Now, you can map the virtualhost. You will see your primary domain as the ‘Virtual Host’, which can’t be changed here. But what you can do is to map this virtualhost to several domains. The trick is to add your server’s IP and the localhost in the ‘domains’ list with comma seperation as following:

localhost mapping to OLS

Once this is saved, restart your OLS and now your HAProxy should be able to read requests and starting forwarding requests to your OLS server.

How to Use Sticky Session for CSRF submission on Highly Scalable Cloud App in Haproxy

HINT: If you are a nginx fan and used it in mass scale, then, you must have done this using ip_hash (Nginx Documentation). It follows the same purpose for Haproxy. Difference and benefits of using Haproxy over Nginx for L7 proxy in a highly scalable and reliable cloud app would be a discussion for another day.

Case Discussion:

Suppose, you have a Cloud app, that is load balanced & scaled between multiple servers using Haproxy, for example:

101.101.101.101
202.202.202.202
303.303.303.303

Now, if your app has a submission form, for example, a poll submission from your users, then, there is an issue in this Haproxy setup.

Let’s say, an User A, requests for the app, and gets the data from the server 101.101.101.101, the CSRF token he gets for the poll submission to his browser, also maintains the app hosted on 101.101.101.101. But when he press the submit button, HAProxy puts him on 202.202.202.202 app, and the app hosted on 202.202.202.202 instantly rejects the token for the session as the session is not registered for that app. For such cases, we need to maintain a ‘Sticky’ session based on the cookie set by the right server. That means, if the cookie is set by 101.101.101.101, HAproxy should obey and give the user 101.101.101.101 until the cookie or the session is reset or regenerated.

How To Do That:

What we need to do, let haproxy write the server id in the cookie, and make the directive ‘server’ to follow the cookie. Please remember, there are couple of other way to achieve this. There is another way of doing this is called ‘IP Affinity’, where you make sticky session based on IP of the user. There is another based on PHP session value. Setting sticky session based on php session should also work. I preferred the cookie based sticky session, just on random selection.

So, to write the server id in the cookie, you need to add the following in the haproxy ‘backend’ directive as following:

backend app-main
balance roundrobin
cookie SERVERID insert indirect nocache

In the cookie directive, you can see, we are taking the HAProxy variable ‘SERVERID’ and inserting that to the cookie attribute. Now, all you need to do, is to configure your balancing IPs to follow the cookie, like the following:

backend app-main
balance roundrobin
cookie SERVERID insert indirect nocache
server nginx1 101.101.101.101 cookie S1
server nginx2 202.202.202.202 cookie S2
server nginx3 303.303.303.303 cookie S3

S1, S2, S3 are just 3 different names of the cookies for the specific servers. After the above is done, you can now restart and see Haproxy is following stickiness based on the session you have.

Just to find out, how to test if you are using laravel, try to regenerate the session based on the session() helper method as following:

session()->regenerate()->csrf_token();

You should be able to see the content loading from different web servers when the session regenerates. But it will persists when the regenerate session method is not called.

How To: Restrict a Folder to Your IP Only

Sometimes, for development purposes, you may want to restrict access to the folder, only to your IP, and deny others from accessing that folder. One way to do that is to use htaccess rules. A common rule, could be denying all the users and allowing your IP. To find out, your IP, you may visit the following:

http://ifconfig.co

Note the IP it has reported. Open the folder that you want to protect. Find the .htaccess file under the folder (If no file available, create one) and add the following:

order deny,allow
deny from all
allow from <your IP goes here>

Replace the <your IP goes here> from the snippets with the IP you have noted from ifconfig.co. Now, your folder should be accessible only from your IP.

Troubleshoot: You must upgrade to Litespeed “5.2.1 build 2 or later”, in order to upgrade to the next version of cPanel & WHM.

Error Message

You must upgrade to Litespeed “5.2.1 build 2 or later”, in order to upgrade to the next version of cPanel & WHM.

Explanation

The error appears because Litespeed below 5.2.1 doesn’t have SSL compatibility with Cpanel 11.68. Although, this goes further, if you uninstall the plugin, and upcp will still fail to update the Cpanel/WHM. A workaround for this, is to install and uninstall the plugin through Litespeed auto installer. Here is how to do that:

Download Litespeed Auto Installer for Cpanel

# wget https://www.litespeedtech.com/packages/cpanel/lsws_whm_autoinstaller.sh
# chmod a+x lsws_whm_autoinstaller.sh

Install Litespeed for Cpanel/WHM using Auto Installer

# ./lsws_whm_autoinstaller.sh TRIAL 1 8080 username testpass1234 s@mellowhost.com 1 0

Uninstall Litespeed

# /usr/local/lsws/admin/misc/uninstall.sh

Uninstall Litespeed WHM Plugin

# /usr/local/cpanel/whostmgr/docroot/cgi/lsws/lsws_whm_plugin_uninstall.sh

Run upcp:

# /scripts/upcp --force