How To: Manually Add Support of SSL for WWW on Cyberpanel

hmm, it’s a weird topic to write blog on. Because Cyberpanel comes with a built in Certbot, and can automatically detects www and without www to install SSL for. Then why am I writing this up? All because I found a VPS client today facing the issue. Even though, Cyberpanel was telling me that the SSL is issued, it was only issued for non-www domain, but the www domain left behind. Let’s see how can we resolve this.

First problem

First problem came up when I tried to discover the Cyberpanel certbot binaries.

[root@server-sg /]# find . -name "certbot"
./usr/local/CyberCP/bin/certbot
./usr/local/CyberCP/lib/python3.6/site-packages/certbot
./usr/local/CyberPanel/bin/certbot
./usr/local/CyberPanel/lib/python3.6/site-packages/certbot

[root@server-sg live]# /usr/local/CyberCP/bin/certbot --version
certbot 0.21.1
[root@server-sg live]# /usr/local/CyberPanel/bin/certbot --version
certbot 0.21.1

Both of the certbot I could find from Cyberpanel was very old, Certbot has 1.4 version in the Epel which has support for Acme 2 challenge, while the one that Cyberpanel is using doesn’t. I hence decided to install a certbot for our case:

yum install epel-release
yum install certbot

These should be it for the latest version of certbot to start working in your Cyberpanel host. Once done, you may now generate the SSL using the following:

certbot certonly  --webroot -w /home/yourdomain.com/public_html -d yourdomain.com -d www.yourdomain.com

Remember to replace yourdomain.com with the actual one that is having problem with. Cyberpanel creates the home directory with the primary domain, so the remember to give the correct document root for the value of attribute ‘-w’.

Once this id done, certbot should automatically verify the challenge and get the issued license for you. Lets encrypt license are usually stored at the following directory:

/etc/letsencrypt/live/yourdomain.com/

Files are:
/etc/letsencrypt/live/yourdomain.com/privatekey.pem
/etc/letsencrypt/live/yourdomain.com/fullchain.pem

If you had already created the SSL using Cyberpanel (which you must have done if you viewing this post), then remember, certbot will place the SSLs in /etc/letsencrypt/live/yourdomain.com-001/ folder. The name of the folder would be shown at the time you complete issuing SSL with certbot.

There are couple of ways you may use the SSL now. Either you may replace the old directory with the new, or just change the settings in either the vhost conf or the openlitespeed SSL settings. I find the easiest way is just to replace the old directory with the new. Something like this should work:

mv /etc/letsencrypt/live/yourdomain.com /etc/letsencrypt/live/old_yourdomain.com
mv /etc/letsencrypt/live/yourdomain.com-001 /etc/letsencrypt/live/yourdomain.com

Once this is done, remember to restart your openlitespeed:

service lsws restart

Now your https on the WWW should work without any problem. If not, try clearing your browser cache and retry.

How to Add Openlitespeed Server to Haproxy – Avoid 503 Haproxy Error

For past one month, Openlitespeed has been my favorite piece of web server. Litespeed has always outperformed all the other webservers including Nginx as well in any of my production environment. But I have recently switched to using OLS which is a Opensource version of Litespeed with some limited features. I get LS kind of performance along with no worry for paying. How better could it be?

OLS comes with some weird problem. As OLS is less used, finding a solution for such cases could be difficult. I faced a very similar kind of issue yesterday.

I added a OLS based server to my HAProxy cluster, but the HAProxy can not find the OLS server working. When I try to access the web app hosted under OLS server using local IP masking, I see the website without a problem. That means, OLS is interpreting the Domain with IP relation well. But failing to respond when Haproxy is requesting through IP address.

The problem is, OLS is not configured to respond to ‘default’ requests on ‘127.0.0.1’, ‘localhost’ or the server’s main IP. To find out this, I enabled ‘High’ Debug mode of OLS. To do this, first visit the OLS Webadmin Console, it can be accessed with https://IP:7080

After login, go to Server Configuration >> Log >> Edit Server Log >> Set ‘Debug Level’ to High and Save

Set high debug level in openlitespeed

Once saving is done, you may gracefully restart the OLS

Gracefully restart Openlitespeed

Once this is done, you may now monitor the error.log file located usually under /usr/local/lsws/logs. Now tail the output of error.log while processing requests with Haproxy:

tail -f /usr/local/lsws/logs/error.log

You can see, OLS has returned 404 error for the localhost/ request. That means, Haproxy is requesting the IP with a header ‘localhost/’, and the server should return something with code 200 to make sure the server is in business.

What we need to do, is to make OLS respond to request for basic IP and localhost to 200 with the main site instead of ‘404’ error. To do this, we need to go to Webconsole of OLS again >> Listeners

You will see you have two Listeners, one for Default/Non HTTP and the HTTPS/SSL. In my case, I was using only HAProxy to Origin with no SSL, means 80. I selected the Default.

Open 80 Listener View in Openlitespeed

In the Listener List, you can find your Virtualhost, click on the ‘Edit’ of your Virtualhost

Virtualhost Edit Openlitespeed

Now, you can map the virtualhost. You will see your primary domain as the ‘Virtual Host’, which can’t be changed here. But what you can do is to map this virtualhost to several domains. The trick is to add your server’s IP and the localhost in the ‘domains’ list with comma seperation as following:

localhost mapping to OLS

Once this is saved, restart your OLS and now your HAProxy should be able to read requests and starting forwarding requests to your OLS server.

Lost connection after starttls from Hostname (IP) – Virtualmin – Postfix

Problem Definition:

I have some VPS clients using Virtualmin as their LAMP/LEMP stack. After some recent updates to Virtualmin, they started seeing some Postfix errors. The error is the following:

lost connection after STARTTLS from unknown[0.0.0.0]

Virtualmin used to configure postfix to allow ‘Non TLS’ connections to the port 587, which they recently stopped configuring. Now, if you connect to 587 port, you have to follow the TLS, no matter what. My clients didn’t bother to use TLS/SSL before, which caused the error.

Virtualmin comes with Let’s Encrypt. That’s make it easy to solve the problem TLS problem.

Solution Summary:

Here is the basic to solve the problem, first you make virtualmin to install Let’s encrypt SSL for the domain you want to use for SMTP. Virtualmin primarily going to install this for your Apache. Once done, Copy the same certificate to your Postfix, Virtualmin allows you to do it with single click.

Detailed Steps:

First, login to your Virtualmin at 10000 port, then select the domain you use for the SMTP. Once done, you can go to Edit Virtual Server and expand the option ‘Enabled Features’. From here check the option says ‘Apache SSL Website Enabled?’

Check Apache SSL Website Enabled

Next, go to Server Configuration >> SSL Certificate, we will get two tabs, ‘Current Certificate’ & ‘Let’s Encrypt’. Both are important. First go to Let’s Encrypt:

Let’s Encrypt Virtualmin

In the Let’s Encrypt tab, select the ‘Domain names listed here‘ and enter the domain that only has valid A Records or loads to the server, otherwise, remember, Let’s Encrypt won’t process for any single exception unlike cpanel or cyberpanel

Let’s Encrypt Virtualmin Add Domains

Once done, request the certificate. After the certificate installation is done, go back to ‘Current Certificate’ tab. On the bottom of the tab, there are couple of Copy To ‘Services’ option available. Here you should see the option says ‘Copy to Postfix’. Use that to copy the certificate to Postfix and use it during TLS/SSL transactions.

Copy SSL to Services (Postfix) Virtualmin.

In my case, I have already copied the SSL to Postfix, which is why it is not showing the option ‘Copy To Postfix’. But the option should be above the ProFTPD.

Once done, you may now recheck and the SMTP should work with TLS and 587 port.

Troubleshoot: -bash: fuser: command not found

If you are trying to use a command, fuser, that is used to identify process using lock file or socket file, and having the following error:

-bash: fuser: command not found

that means, your system doesn’t have the psmisc package installed. Usually, killall, fuser type of commands are under this package. CentOS 7 sometimes, do not install the package by default. To install it, run the following:

yum install psmisc -y

Once done, you should be able to use fuser command.

Troubleshoot: You must upgrade to Litespeed “5.2.1 build 2 or later”, in order to upgrade to the next version of cPanel & WHM.

Error Message

You must upgrade to Litespeed “5.2.1 build 2 or later”, in order to upgrade to the next version of cPanel & WHM.

Explanation

The error appears because Litespeed below 5.2.1 doesn’t have SSL compatibility with Cpanel 11.68. Although, this goes further, if you uninstall the plugin, and upcp will still fail to update the Cpanel/WHM. A workaround for this, is to install and uninstall the plugin through Litespeed auto installer. Here is how to do that:

Download Litespeed Auto Installer for Cpanel

# wget https://www.litespeedtech.com/packages/cpanel/lsws_whm_autoinstaller.sh
# chmod a+x lsws_whm_autoinstaller.sh

Install Litespeed for Cpanel/WHM using Auto Installer

# ./lsws_whm_autoinstaller.sh TRIAL 1 8080 username testpass1234 s@mellowhost.com 1 0

Uninstall Litespeed

# /usr/local/lsws/admin/misc/uninstall.sh

Uninstall Litespeed WHM Plugin

# /usr/local/cpanel/whostmgr/docroot/cgi/lsws/lsws_whm_plugin_uninstall.sh

Run upcp:

# /scripts/upcp --force

Troubleshoot: Server IP address could not be found

I had a client ticket today, with the following screenshot:

The error says, ‘server IP address could not be found’. This type of error means there is a DNS resolution error. There could be 3 possibilities:

  1. Client hasn’t updated the dns nameservers for the domain
  2. Host’s DNS server is down.
  3. Client’s DNS resolver isn’t working.

To check if the client has updated the proper dns nameservers, you can use intodns.com. It will also tell you if the host DNS is down or not. If both are ok, you should check if you are able to load other domains using your Internet, if not, it has things to do with the local DNS resolver of your desktop or the ISP. In my case, it was client who failed to update the nameservers of the domain. All that was required to update the nameservers with the server ones.

Troubleshoot: fatal: open lock file /var/lib/postfix/master.lock: unable to set exclusive lock

Error Message & Trace details:

One of my customer came with an error saying the postfix in his server isn’t working. The server was running CentOS 7, and the system postfix status was inactive, means not running. Although, the system queue was running I could see. The error that was returning while restarting/checking status was the following:

# service postfix status
Redirecting to /bin/systemctl status postfix.service
● postfix.service - Postfix Mail Transport Agent
Loaded: loaded (/usr/lib/systemd/system/postfix.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Tue 2018-01-09 04:04:05 UTC; 1s ago
Process: 9201 ExecStart=/usr/sbin/postfix start (code=exited, status=1/FAILURE)
Process: 9197 ExecStartPre=/usr/libexec/postfix/chroot-update (code=exited, status=0/SUCCESS)
Process: 9194 ExecStartPre=/usr/libexec/postfix/aliasesdb (code=exited, status=0/SUCCESS)
Main PID: 1358 (code=killed, signal=TERM)

Jan 09 04:04:03 twin7.hifrank.biz systemd[1]: Starting Postfix Mail Transport Agent...
Jan 09 04:04:03 twin7.hifrank.biz postfix/master[9273]: fatal: open lock file /var/lib/postfix/master.lock: unable to set exclusive lock: Resource tempo...vailable
Jan 09 04:04:04 twin7.hifrank.biz postfix/master[9272]: fatal: daemon initialization failure
Jan 09 04:04:05 twin7.hifrank.biz postfix/postfix-script[9274]: fatal: mail system startup failed
Jan 09 04:04:05 twin7.hifrank.biz systemd[1]: postfix.service: control process exited, code=exited status=1
Jan 09 04:04:05 twin7.hifrank.biz systemd[1]: Failed to start Postfix Mail Transport Agent.
Jan 09 04:04:05 twin7.hifrank.biz systemd[1]: Unit postfix.service entered failed state.
Jan 09 04:04:05 twin7.hifrank.biz systemd[1]: postfix.service failed.

How to fix:

The error to note here is the following:

fatal: open lock file /var/lib/postfix/master.lock

I first killed the smtp and smtpd processes that runs by postfix:

# killall -9 smtp
# killall -9 smtpd

But that didn’t solve the problem. I then used the fuser command to check which process holds the lock file:

# fuser /var/lib/postfix/master.lock
/var/lib/postfix/master.lock: 18698

Then we check the process 18698 and kill the responsible process:

# ps -axwww|grep 18698
9333 pts/0 S+ 0:00 grep --color=auto 18698

18698 ? Ss 4:28 /usr/libexec/postfix/master -w
# killall -9 /usr/libexec/postfix/master
or
# kill -9 18698

Once the process is killed, you can now start the postfix:

# service postfix start
# service postfix status|grep Active
Redirecting to /bin/systemctl status postfix.service
Active: active (running) since Tue 2018-01-09 04:15:50 UTC; 4min 45s ago

Troubleshoot: killall command not found in centos 7

Problem:

If you are using a centos 7 minimal installation, and trying to kill process by name using the command ‘killall’, you are most likely going to see the error:

# killall -9 php
killall: command not found

The error appears because CentOS 7 encourages you to use pkill instead of killall to kill process by name. pkill has versatile application, although, it can be used to kill process by name same as killall.

How to kill process by name in Centos 7

You can use pkill. pkill is a simple command. It’s syntax is as following:

# pkill processname

For example, if you want to kill all the php process, run:

# pkill php

Note: It will kill all the processes that match php. To list the process that pkill going to kill, you can use pgrep as following:

# pgrep -l php

How to use killall in Centos 7

If you do not want to use pkill, and keep using killall commands in centos 7, this is also possible. killall is a part of psmisc yum package. All you have to do, is to install psmisc in your system using yum

# yum install psmisc
# killall -9 php

 

Troubleshooting: Imunify360 database is corrupt. Application cannot run with corrupt database

Error Message:

# service imunify360 start
Starting imunify360: WARNING [+ 3743ms] defence360agent.utils.check_db|DatabaseError detected: database disk image is malformed
WARNING [+ 3766ms] defence360agent.cli.subparsers.common.server|Imunify360 database is corrupt. Application cannot run with corrupt database. Please, contact Imunify360 support team at https://cloudlinux.zendesk.com

Detail Information & Explanation:

If you are using imunify360, an application firewall for linux servers by Cloudlinux team, you might incur an error where it says the database is corrupt. You might first see ‘Imunify360 is not started’ error from the WHM panel and end up getting the above error message as stated. Imunify360 uses a SQL database, located under ‘/var/imunify360/imunify360.db’. This image is checked everytime Imunfi360 tries to start, and if the database is malformed, it would not start. Fortunately, imunify360 comes with tools to handle this database and recover if corrupted.

How to Fix:

First, we start by running database integrity check. This can be done using the following:

imunfiy360-agent checkdb

(From Imunify360 Doc: checkdb  – Check database integrity)

Once done, you can now use ‘migratedb’ to repair and restore if the database is corrupted.

imunify360-agent migratedb

(From Imunify360 Doc: migratedb – Check and repair database if it is corrupted.)

If migratedb fails, the only way to recover this is to reinstall imunify360.

Linux: Assertion failed on job for iptables.service.

If you are using Centos 7 or RHEL 7 or any of it’s variant, you are probably using ‘Firewalld’ by default. Although, if you are a iptables fan like me, who likes it’s simplicity and manipulative nature instead of a full form firewall, then you probably have disabled firewalld from your CentOS 7 instance and using iptables. There are couple of servers, where I use runtime iptables rules for postrouting and masquerading. These rules are dynamically generated by my scripts instead of the sysconfig file under:

/etc/sysconfig/iptables

This file is generated upon running the iptables save command:

service iptables save

which I rarely do so.

Error Details

Which is why, I don’t have a /etc/sysconfig/iptables file in those servers and a common error I see while restarting iptables in those system is the following:

# systemctl restart iptables.service
Assertion failed on job for iptables.service.

How to Fix The Error

The error appears because you don’t have any rule in /etc/sysconfig/iptables or the file doesn’t exist either. You can ignore the error as iptables would still run. To eradicate the error, simply make sure you have some iptables rules loaded on your system using the status command:

iptables -S

And then, run:

service iptables save

Once done, restarting iptables shouldn’t show the error any longer.