How To Run a Command in All OpenVZ Containers

You can run single command in a container using the following:

vzctl exec 201 service httpd status

How to find out all the VZ containers:

vzlist -a

The other way? Yes, there is. VZ list is stored inside a file /proc/vz/veinfo, and we can use it with the help of shell to run command in each VZ as following:

for i in `cat /proc/vz/veinfo | awk '{print $1}'|egrep -v '^0$'`; \
do echo "Container $i"; vzctl exec $i <your command goes here>; done

An example, can be the following:

for i in `cat /proc/vz/veinfo | awk '{print $1}'|egrep -v '^0$'`; \
do echo "Container $i"; vzctl exec $i service httpd status; done

This should show all the httpd status of the VZ.

How To Send Email From an IP without Authentication – Cpanel/WHM

Since antirelayed is removed by the cpanel team from the latest cpanel, the situation might arise to some people, at least to me. I had a server sending mails without authentication, a trusted IP. Now, how to do this with the latest Cpanel/WHM?

Well, Cpanel still keeps the facility called ‘alwaysrelay’. This one was there when antirelayed was there. Antirelayed used to allow relay for an IP without authentication for a specific period of time, while ‘alwaysrelay’ will allow relaying all the time.

All you need to do, is to add the IP in the following file in a new line:

/etc/alwaysrelay

and restart the Exim:

service exim restart

That should be it. Remember, you might encounter the exim report cpaneleximscanner found your email to be spam. In such cases, go to WHM >> Service Configuration >> Exim Configuration Manager >> Set the following option to ‘Off’ : Scan outgoing messages for spam and reject based on the Apache SpamAssassin™ internal spam_score setting

and Save. Now you may check, it should work.

How To: Restore Zimbra Quarantined Email by Clam AKA Heuristics.Encrypted.PDF Release Point

Zimbra Mail Server automatically quarantines emails that get hit by the Antivirus scan using Clam when the mail is received. While putting the email on the recipient inbox, what it does, instead of giving the original email with the attachment, it sends a virus detected email with the following kind of error message:

Virus (Heuristics.Encrypted.PDF) in mail to YOU

Virus Alert
Our content checker found
virus: Heuristics.Encrypted.PDF

by Zimbra

It actually means, the original mail is now quarantined. Zimbra maintains a virus quarantine email account that is not normally available in the ‘Manage Account’ list of Zimbra Admin panel. You can find it if you search with ‘virus’ in the ‘Search’ box of the admin panel. What zimbra does in quarantine situation, is that, it pushes the mail to the quarantine email instead of original recipient.

Now, to get back the mail delivered to the original recipient, we need to first get the quarantine email account, get the message id, and then we need to inject the mail into the LMTP pipe that bypasses any scanning. Here are the steps on how to do this:

# First get to the zimbra user
$ su - zimbra

# Get the email account that is used to store virus detected mails
$ zmprov gcf zimbraAmavisQuarantineAccount
zimbraAmavisQuarantineAccount: virus-quarantine.mrhmcbg4kz@yourhostname.com

# virus-quarantine.mrhmcbg4kz@yourhostname.com this should be our quarantine email account, now we need to get the quarantine account's mailbox id
$ zmprov gmi virus-quarantine.mrhmcbg4kz@yourhostname.com
mailboxId: 73
quotaUsed: 644183

# Mailbox id here for the quarantine account is 73. Now go to the message storage of this id using the following command: cd /opt/zimbra/store/0/<mailboxId>/msg/0
$ cd /opt/zimbra/store/0/73/msg/0

# list the messages
$ ls *

These are your quarantined emails. Now for example the complainer is ‘sam@yourhostname.com’. To search for the emails designated for this email account, you may use the following:

$ grep -l sam@yourhostname.com *
281-1216.msg
300-1400.msg
301-1476.msg

This should return you all the emails that got quarantined for the above user.

Now the question is, how can we get these emails delivered to the designated user bypassing the antivirus/antispam tools. To do this, you need to inject the mail into LMTP pipe. You may do this using ‘zmlmtpinject’ command as following:

$ zmlmtpinject -r sam@yourhostname.com -s admin@yourhostname.com 281-1216.msg

Remember, to change same@yourhostname.com to the original recipient. admin@yourhostname.com would be the newly rewritten sender for this mail delivery and ‘281-1216.msg’ is the file name of the original email that you found out from the grep command. You can do lmtp injections for one email mail with each command. So, you would require to do this for each emails.

How To: Send Email Alert on Different HAProxy Status or Based on HAProxy Stats

HAProxy is a great simple load balancing tool written in Lua. It is extremely efficient as a software load balancer and highly configurable as well. On the contrary, HAProxy lacks programmable automated monitoring tools. It has a directive called ‘mailer’ which has only support above 1.8. Default CentOS 7 repo comes with HAProxy 1.5 and it has no mailer alert support either. Even with 1.8, it doesn’t come with lots of available configuration options neither the tool gets programmable facility.

That is where, I thought to work on to trigger codes from HAProxy stats. This can be done in many ways, in my cases, I did it using per minute crons. If you want it much quicker like every 5 seconds for example, you would have to run this as a daemon, which isn’t like making a rocket, should be easy and short. My entire idea is to allow you understanding how to create programmable 3rd party tools by fetching data from HAProxy socket and trigger monitors.

HAProxy Stats through Unix Socket

First, we need to enable the HAProxy stats that is available through socket. To turn on stats through unix socket, you need put the following line in your global section of haproxy.cfg file:

stats socket /var/lib/haproxy/stats

An example of Global settings section would be like the following:

global
    log         127.0.0.1 local2     #Log configuration

    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     40000
    user        haproxy             #Haproxy running under user and group "haproxy"
    group       haproxy
    daemon

    # turn on stats unix socket
    stats socket /var/lib/haproxy/stats

Check how the stats socket section is placed to fit it for your cfg file.

Once this is done, now you can restart the haproxy to start shooting stats through the socket. The output is basically a csv of the HAProxy stats page. So the values going to be in comma seperated format. To understand HAProxy stats page and exploding the values, you can visit the following:

https://www.haproxy.com/blog/exploring-the-haproxy-stats-page/

Now, how to read the unix-socket using bash? There is a tool called ‘socat’ that can be used to read data from unix socket. ‘socat’ means ‘socket cat’, you may read more details about ‘socat’ here:

https://linux.die.net/man/1/socat

If socat is not available on your CentOS yum, you may get it from epel-release.

yum install epel-release
yum install socat

Once ‘socat’ is available in your system, you can use it to redirect the io and show the output as following:

echo "show stat" | socat unix-connect:/var/lib/haproxy/stats stdio

Now as you can see, you can retrieve the whole HAProxy stats in CSV format, you may easily manipulate and operate data using a shell script. I have created a basic shell script to get the status of the HAProxy backends and send an email alert using ‘ssmtp’. Remember, ssmtp is highly configurable mail tool, you can customize smtp authentication as well with ‘ssmtp’. You may use any other tool like Sendmail for example or ‘Curl’ to any email API like Sendgrid, possibility is infinite here. Remember, as the data is instantly available to socket as soon as the HAProxy generates the event, hence, it can be as efficient as HAProxy built in functions like ‘mailer’ is.

#!/bin/bash

cd /root/
rm -f haproxy_stat.txt
echo "show stat" | socat unix-connect:/var/lib/haproxy/stats stdio|grep app-main > /root/test.txt

SEND_EMAIL=0

while IFS= read -r line
do

APP=`echo $line | cut -d"," -f2`
STAT=`echo $line | cut -d"," -f18`
SESSION=`echo $line | cut -d"," -f34`
if [ "$STAT" != "UP" ]; then
SEND_EMAIL=1
MESSAGE+="$APP $STAT $SESSION
"
fi

done < /root/test.txt

if [ $SEND_EMAIL -eq 1 ]; then
echo -e "Subject: Haproxy Instance Down \n\n$MESSAGE" | sudo ssmtp -vvv myownemail@gmail.com
echo -e "Subject: Haproxy Instance Down \n\n$MESSAGE" | sudo ssmtp -vvv otheremail@gmail.com

fi

Data that I am interested in are the status of the backend, name of the backend and the session rate of the backend. So, if the load balancer sees any backend is down, this would trigger the email delivery. You can use this to catch anything in the HAProxy, like a Frontend attack for example, like the delivery optimization of your load balancer etc. As you are now able to retrieve data directly from HAProxy to your own ‘programming’ console, you can program it whatever the way you want to. Hope this helps somebody! For any help, shoot a comment!

How to Add Openlitespeed Server to Haproxy – Avoid 503 Haproxy Error

For past one month, Openlitespeed has been my favorite piece of web server. Litespeed has always outperformed all the other webservers including Nginx as well in any of my production environment. But I have recently switched to using OLS which is a Opensource version of Litespeed with some limited features. I get LS kind of performance along with no worry for paying. How better could it be?

OLS comes with some weird problem. As OLS is less used, finding a solution for such cases could be difficult. I faced a very similar kind of issue yesterday.

I added a OLS based server to my HAProxy cluster, but the HAProxy can not find the OLS server working. When I try to access the web app hosted under OLS server using local IP masking, I see the website without a problem. That means, OLS is interpreting the Domain with IP relation well. But failing to respond when Haproxy is requesting through IP address.

The problem is, OLS is not configured to respond to ‘default’ requests on ‘127.0.0.1’, ‘localhost’ or the server’s main IP. To find out this, I enabled ‘High’ Debug mode of OLS. To do this, first visit the OLS Webadmin Console, it can be accessed with https://IP:7080

After login, go to Server Configuration >> Log >> Edit Server Log >> Set ‘Debug Level’ to High and Save

Set high debug level in openlitespeed

Once saving is done, you may gracefully restart the OLS

Gracefully restart Openlitespeed

Once this is done, you may now monitor the error.log file located usually under /usr/local/lsws/logs. Now tail the output of error.log while processing requests with Haproxy:

tail -f /usr/local/lsws/logs/error.log

You can see, OLS has returned 404 error for the localhost/ request. That means, Haproxy is requesting the IP with a header ‘localhost/’, and the server should return something with code 200 to make sure the server is in business.

What we need to do, is to make OLS respond to request for basic IP and localhost to 200 with the main site instead of ‘404’ error. To do this, we need to go to Webconsole of OLS again >> Listeners

You will see you have two Listeners, one for Default/Non HTTP and the HTTPS/SSL. In my case, I was using only HAProxy to Origin with no SSL, means 80. I selected the Default.

Open 80 Listener View in Openlitespeed

In the Listener List, you can find your Virtualhost, click on the ‘Edit’ of your Virtualhost

Virtualhost Edit Openlitespeed

Now, you can map the virtualhost. You will see your primary domain as the ‘Virtual Host’, which can’t be changed here. But what you can do is to map this virtualhost to several domains. The trick is to add your server’s IP and the localhost in the ‘domains’ list with comma seperation as following:

localhost mapping to OLS

Once this is saved, restart your OLS and now your HAProxy should be able to read requests and starting forwarding requests to your OLS server.

Lost connection after starttls from Hostname (IP) – Virtualmin – Postfix

Problem Definition:

I have some VPS clients using Virtualmin as their LAMP/LEMP stack. After some recent updates to Virtualmin, they started seeing some Postfix errors. The error is the following:

lost connection after STARTTLS from unknown[0.0.0.0]

Virtualmin used to configure postfix to allow ‘Non TLS’ connections to the port 587, which they recently stopped configuring. Now, if you connect to 587 port, you have to follow the TLS, no matter what. My clients didn’t bother to use TLS/SSL before, which caused the error.

Virtualmin comes with Let’s Encrypt. That’s make it easy to solve the problem TLS problem.

Solution Summary:

Here is the basic to solve the problem, first you make virtualmin to install Let’s encrypt SSL for the domain you want to use for SMTP. Virtualmin primarily going to install this for your Apache. Once done, Copy the same certificate to your Postfix, Virtualmin allows you to do it with single click.

Detailed Steps:

First, login to your Virtualmin at 10000 port, then select the domain you use for the SMTP. Once done, you can go to Edit Virtual Server and expand the option ‘Enabled Features’. From here check the option says ‘Apache SSL Website Enabled?’

Check Apache SSL Website Enabled

Next, go to Server Configuration >> SSL Certificate, we will get two tabs, ‘Current Certificate’ & ‘Let’s Encrypt’. Both are important. First go to Let’s Encrypt:

Let’s Encrypt Virtualmin

In the Let’s Encrypt tab, select the ‘Domain names listed here‘ and enter the domain that only has valid A Records or loads to the server, otherwise, remember, Let’s Encrypt won’t process for any single exception unlike cpanel or cyberpanel

Let’s Encrypt Virtualmin Add Domains

Once done, request the certificate. After the certificate installation is done, go back to ‘Current Certificate’ tab. On the bottom of the tab, there are couple of Copy To ‘Services’ option available. Here you should see the option says ‘Copy to Postfix’. Use that to copy the certificate to Postfix and use it during TLS/SSL transactions.

Copy SSL to Services (Postfix) Virtualmin.

In my case, I have already copied the SSL to Postfix, which is why it is not showing the option ‘Copy To Postfix’. But the option should be above the ProFTPD.

Once done, you may now recheck and the SMTP should work with TLS and 587 port.

How To: Use SSH Password in a Script

You can obviously use RSA public/private keypair to access servers without password, although, sometimes, it may be desirable to use ssh password on a command line and run a remote command on another server. This can be done using a tool called ‘sshpass’. You can create a simple bash script using sshpass, that can help you monitor and control multiple servers from a single location.

How to use ssh password in script

First install sshpass:

# yum install sshpass -y

Once done, you can use sshpass command as following:

# sshpass -p "SSH_Password" ssh -o StrictHostKeyChecking=no SSH_Username@remote-ssh-server "yourcommand"

Just replace, the password, username, remote-ssh-server and your command with your desired setup, and viola!

Note: If yum says, the following:

No package sshpass* available.

then you would need to install epel repository to install sshpass. To install epel, run the following:

# yum install epel* -y

How To: Start a Screen Session and Run a Command at a Time

Sometimes, you may want to run a screen command in a remote server. That makes it necessary to run the command inside the screen session while starting it.

How to start a screen session and run a command in one line

# screen -d -m sh -c "yourcommand"

From the man page of Screen:

-d -m : Start screen in “detached” mode. This creates a new session but doesn’t attach to it. This is useful for system startup scripts.

sh -c: Starts a shell and runs a command for you.

 

How To: Install AutoSSL for Server Hostname – Cpanel Server

Starting from Cpanel 11.58, Cpanel is offering Free SSL, issued by ‘Cpanel INC’ for free of charge to the valid cpanel license owner. If you are using cpanel, login to your WHM >> Providers >> Enable Cpanel & from Options >> Check Allow AutoSSL to replace invalid or expiring non-AutoSSL certificates.

Now, running upcp should automatically install the free SSL for your cpanel server hostname. If it doesn’t, it is probably because your server IP and the hostname IP are resolving to wrong address. To understand and troubleshoot the problem, run the following script from command line:

# /usr/local/cpanel/bin/checkallsslcerts

This script checks and installs certificate for expired, invalid and self signed certificates for the server services. If you are seeing an error like the following:

[WARN] The system failed to acquire a signed certificate from the cPanel Store because of the following error: (XID 62hp6x) The system queried for a temporary file at “http://server91.mellowhost.com/.well-known/pki-validation/D92868E512FB02354F2498B94E67430B.txt”, but the web server responded with the following error: 404 (Not Found). A DNS (Domain Name System) or web server misconfiguration may exist.

It means, your hostname is resolving to the wrong IP. You would need to check if the hostname is resolving to an IP which has first virtualhost pointed to /var/www/html or not under /etc/apache2/conf/httpd.conf

 

How To: Clear Postfix and Exim Queue

How to clear postfix queue

Postfix queue can be cleared using the postsuper command. The syntax is as following:

# postsuper -d ALL

It will clear all the postfix queue. You can specifically clear the deferred emails for example from the postfix queue as following:

# postsuper -d ALL deferred

If you have a large quantity in postfix queue, there is actually a quicker and easier way to do that. You can simply remove all the folders under /var/spool/postfix and it will clear the queue.

# rm -Rf /var/spool/postfix/*
# mailq | tail -n 1
Mail queue is empty

Note: The folders inside postfix would get created automatically once the queue starts filling up, nothing to worry about.

How to clear Exim Queue

Exim queue can be checked using the following:

# exim -bp

To check the number of mails in queue, you can use:

# exim -bpc

To remove a message from exim queue, you need to use the following:

# exim -Mrm {message-id}

There is no build in command to clear all the mails from exim queue. You can use a pipe command to clear the exim queue as following:

# exim -bp | exiqgrep -i | xargs exim -Mrm

Although, there are even quicker and easier way to clear the exim queue, specially if you have a lot of emails in queue and the server is pretty loaded.

# rm -Rf /var/spool/exim/input

Removing the input directory should clear the exim queue. Note: The directory would automatically create once the exim starts it’s queue again, no need to worry.