How to manually install/renew Let’s Encrypt SSL in Zimbra

If you are having trouble installing Let’s Encrypt SSL with the certbot-zimbra.sh file, then probably you would need to follow this tutorial. To follow this tutorial, we first need to install certbot. certbot has a built in web server to allow you get the certificate without actually installing an extra web server or through Zimbra web server (nginx to be specific).

First, we install certbot with the following:

// install epel-release first
yum install epel-release
// install certbot from epel
yum install certbot

Once done, you may now use the following command to ensure certbot is working:

# certbot --help

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

  certbot [SUBCOMMAND] [options] [-d DOMAIN] [-d DOMAIN] ...

Certbot can obtain and install HTTPS/TLS/SSL certificates.  By default,
it will attempt to use a webserver both for obtaining and installing the
certificate. The most common SUBCOMMANDS and flags are:

obtain, install, and renew certificates:
    (default) run   Obtain & install a certificate in your current webserver
    certonly        Obtain or renew a certificate, but do not install it
    renew           Renew all previously obtained certificates that are near
expiry
    enhance         Add security enhancements to your existing configuration
   -d DOMAINS       Comma-separated list of domains to obtain a certificate for

  (the certbot apache plugin is not installed)
  --standalone      Run a standalone webserver for authentication
  --nginx           Use the Nginx plugin for authentication & installation
  --webroot         Place files in a server's webroot folder for authentication
  --manual          Obtain certificates interactively, or using shell script
hooks

   -n               Run non-interactively
  --test-cert       Obtain a test certificate from a staging server
  --dry-run         Test "renew" or "certonly" without saving any certificates
to disk

manage certificates:
    certificates    Display information about certificates you have from Certbot
    revoke          Revoke a certificate (supply --cert-name or --cert-path)
    delete          Delete a certificate (supply --cert-name)

manage your account:
    register        Create an ACME account
    unregister      Deactivate an ACME account
    update_account  Update an ACME account
  --agree-tos       Agree to the ACME server's Subscriber Agreement
   -m EMAIL         Email address for important account notifications

More detailed help:

  -h, --help [TOPIC]    print this message, or detailed help on a topic;
                        the available TOPICS are:

   all, automation, commands, paths, security, testing, or any of the
   subcommands or plugins (certonly, renew, install, register, nginx,
   apache, standalone, webroot, etc.)
  -h all                print a detailed help page including all topics
  --version             print the version number
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Once you ensure certbot is installed, now you may use certbot to get the certificate, using the certbot –standalone tag. Remember to stop zimbra first, as Zimbra also runs a nginx web server, that would prevent certbot to use standalone or it’s own web server to verify certificate.

// from root, run
[[email protected] ~]# service zimbra stop

// wait until zimbra stops, once done, use the following to get certificate for your domain/hostname in place of mail.domain.com
[[email protected] ~]# certbot certonly --standalone -d mail.domain.com

This would get your certificate and save it in:

/etc/letsencrypt/live/mail.domain.com

Now, that folder would contain 4 files. Something like the following:

]# ls -la /etc/letsencrypt/live/mail.domain.com/
total 16
drwxr-xr-x 2 root root 4096 Apr 16 11:30 .
drwx------ 4 root root 4096 Feb 10  2020 ..
lrwxrwxrwx 1 root root   40 Apr 16 11:30 cert.pem -> ../../archive/mail.domain.com/cert8.pem
lrwxrwxrwx 1 root root   41 Apr 16 11:30 chain.pem -> ../../archive/mail.domain.com/chain8.pem
lrwxrwxrwx 1 root root   45 Apr 16 11:30 fullchain.pem -> ../../archive/mail.domain.com/fullchain8.pem
lrwxrwxrwx 1 root root   43 Apr 16 11:30 privkey.pem -> ../../archive/mail.domain.com/privkey8.pem

As you can see, these files are symbolically linked to another files, depends on how many time you are running certbot. Each time, it generates a number liker cert8.pem, the next one would be cert9.pem and so on. So the orignal files are here:

/etc/letsencrypt/archive/mail.domain.com/cert8.pem
/etc/letsencrypt/archive/mail.domain.com/chain8.pem
/etc/letsencrypt/archive/mail.domain.com/fullchain8.pem
/etc/letsencrypt/archive/mail.domain.com/privkey8.pem

Now, we have our certificates. We need to follow a couple of steps to make sure everything is set correctly.

First, zimbra SSL files are stored here

/etc/zimbra/ssl/letsencrypt

We clean all old pem files

rm -f /etc/zimbra/ssl/letsencrypt/*

Now, copy the pem files we got to this folder with the following:

cp /etc/letsencrypt/archive/mail.domain.com/cert8.pem /opt/zimbra/ssl/letsencrypt/cert.pem
cp /etc/letsencrypt/archive/mail.domain.com/chain8.pem /opt/zimbra/ssl/letsencrypt/chain.pem
cp /etc/letsencrypt/archive/mail.domain.com/fullchain8.pem /opt/zimbra/ssl/letsencrypt/fullchain.pem
cp /etc/letsencrypt/archive/mail.domain.com/privkey8.pem /opt/zimbra/ssl/letsencrypt/privkey.pem

Check, how we are renaming all the files with number to file name without number, like cert8.pem is moved as cert.pem here.

Now, change the ownership of these files to zimbra with the following:

chown -Rf zimbra:zimbra /opt/zimbra/ssl/letsencrypt/*

Now, we are done from root, change your ownership to zimbra

su - zimbra

First job, is to change your directory to the ‘/opt/zimbra/ssl/letsencrypt/’

cd /opt/zimbra/ssl/letsencrypt/

Let’s Encrypt files are very much ready to use, only with one problem. Let’s Encrypt do not add it’s root CA certificate with it’s chain.pem file. We need to do this. First open the certificate with nano editor as following:

nano chain.pem

Now, at the end of the file, add the following section:

-----BEGIN CERTIFICATE-----
MIIDSjCCAjKgAwIBAgIQRK+wgNajJ7qJMDmGLvhAazANBgkqhkiG9w0BAQUFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
DkRTVCBSb290IENBIFgzMB4XDTAwMDkzMDIxMTIxOVoXDTIxMDkzMDE0MDExNVow
PzEkMCIGA1UEChMbRGlnaXRhbCBTaWduYXR1cmUgVHJ1c3QgQ28uMRcwFQYDVQQD
Ew5EU1QgUm9vdCBDQSBYMzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
AN+v6ZdQCINXtMxiZfaQguzH0yxrMMpb7NnDfcdAwRgUi+DoM3ZJKuM/IUmTrE4O
rz5Iy2Xu/NMhD2XSKtkyj4zl93ewEnu1lcCJo6m67XMuegwGMoOifooUMM0RoOEq
OLl5CjH9UL2AZd+3UWODyOKIYepLYYHsUmu5ouJLGiifSKOeDNoJjj4XLh7dIN9b
xiqKqy69cK3FCxolkHRyxXtqqzTWMIn/5WgTe1QLyNau7Fqckh49ZLOMxt+/yUFw
7BZy1SbsOFU5Q9D8/RhcQPGX69Wam40dutolucbY38EVAjqr2m7xPi71XAicPNaD
aeQQmxkqtilX4+U9m5/wAl0CAwEAAaNCMEAwDwYDVR0TAQH/BAUwAwEB/zAOBgNV
HQ8BAf8EBAMCAQYwHQYDVR0OBBYEFMSnsaR7LHH62+FLkHX/xBVghYkQMA0GCSqG
SIb3DQEBBQUAA4IBAQCjGiybFwBcqR7uKGY3Or+Dxz9LwwmglSBd49lZRNI+DT69
ikugdB/OEIKcdBodfpga3csTS7MgROSR6cz8faXbauX+5v3gTt23ADq1cEmv8uXr
AvHRAosZy5Q6XkjEGB5YGV8eAlrwDPGxrancWYaLbumR9YbK+rlmM6pZW87ipxZz
R8srzJmwN0jP41ZL9c8PDHIyh8bwRLtTcm1D9SZImlJnt1ir/md2cXjbDaJWFBM5
JDGFoqgCWjBH4d1QB7wCCZAA62RjYJsWvIjJEubSfZGL+T0yjWW06XyxV3bqxbYo
Ob8VZRzI9neWagqNdwvYkQsEjgfbKbYK7p2CNTUQ
-----END CERTIFICATE-----

After adding the above, your chain.pem file should look like the following

-----BEGIN CERTIFICATE-----
your chain pem encrypted certificate here
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIDSjCCAjKgAwIBAgIQRK+wgNajJ7qJMDmGLvhAazANBgkqhkiG9w0BAQUFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
DkRTVCBSb290IENBIFgzMB4XDTAwMDkzMDIxMTIxOVoXDTIxMDkzMDE0MDExNVow
PzEkMCIGA1UEChMbRGlnaXRhbCBTaWduYXR1cmUgVHJ1c3QgQ28uMRcwFQYDVQQD
Ew5EU1QgUm9vdCBDQSBYMzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
AN+v6ZdQCINXtMxiZfaQguzH0yxrMMpb7NnDfcdAwRgUi+DoM3ZJKuM/IUmTrE4O
rz5Iy2Xu/NMhD2XSKtkyj4zl93ewEnu1lcCJo6m67XMuegwGMoOifooUMM0RoOEq
OLl5CjH9UL2AZd+3UWODyOKIYepLYYHsUmu5ouJLGiifSKOeDNoJjj4XLh7dIN9b
xiqKqy69cK3FCxolkHRyxXtqqzTWMIn/5WgTe1QLyNau7Fqckh49ZLOMxt+/yUFw
7BZy1SbsOFU5Q9D8/RhcQPGX69Wam40dutolucbY38EVAjqr2m7xPi71XAicPNaD
aeQQmxkqtilX4+U9m5/wAl0CAwEAAaNCMEAwDwYDVR0TAQH/BAUwAwEB/zAOBgNV
HQ8BAf8EBAMCAQYwHQYDVR0OBBYEFMSnsaR7LHH62+FLkHX/xBVghYkQMA0GCSqG
SIb3DQEBBQUAA4IBAQCjGiybFwBcqR7uKGY3Or+Dxz9LwwmglSBd49lZRNI+DT69
ikugdB/OEIKcdBodfpga3csTS7MgROSR6cz8faXbauX+5v3gTt23ADq1cEmv8uXr
AvHRAosZy5Q6XkjEGB5YGV8eAlrwDPGxrancWYaLbumR9YbK+rlmM6pZW87ipxZz
R8srzJmwN0jP41ZL9c8PDHIyh8bwRLtTcm1D9SZImlJnt1ir/md2cXjbDaJWFBM5
JDGFoqgCWjBH4d1QB7wCCZAA62RjYJsWvIjJEubSfZGL+T0yjWW06XyxV3bqxbYo
Ob8VZRzI9neWagqNdwvYkQsEjgfbKbYK7p2CNTUQ
-----END CERTIFICATE-----

Now, save the file (CTRL + o) and exit (CTRL + x)

We need to do one more thing before we are ready to verify and deploy the certificate. We need to set the letencrypt private key that we used to generate the certificate as commercial.key of zimbra. You may do this with the following two commands:

rm -f /opt/zimbra/ssl/zimbra/commercial/commercial.key
cp /opt/zimbra/ssl/letsencrypt/privkey.pem /opt/zimbra/ssl/zimbra/commercial/commercial.key

Now, you are ready to complete the job. First verify if everything is alright with the following:

[[email protected] letsencrypt]$ /opt/zimbra/bin/zmcertmgr verifycrt comm privkey.pem cert.pem chain.pem
** Verifying 'cert.pem' against 'privkey.pem'
Certificate 'cert.pem' and private key 'privkey.pem' match.
** Verifying 'cert.pem' against 'chain.pem'
Valid certificate chain: cert.pem: OK

If everything is ok, you may now deploy certificate with the following command:

/opt/zimbra/bin/zmcertmgr deploycrt comm cert.pem chain.pem

Once the certificate is deployed successfully, get out from the zimbra user to root user with the following command

exit

Now, you may start/restart zimbra with the following command:

service zimbra restart

If everything went right, you should now be able to go to your zimbra domain, and under the lock sign on the left of the domain shown in browser, you may click on it to see the extended date of ssl expiry. Sweet!

How to Uninstall Let’s Encrypt from Cpanel / WHM

To uninstall the Cpanel / WHM plugin for Let’s Encrypt, login to your SSH for root and run the following:

/usr/local/cpanel/scripts/uninstall_lets_encrypt_autossl_provider

It might take sometime, once completed, it should remove let’s encrypt as a provider from your AutoSSL plugin.

How to Install Let’s Encrypt in Cpanel

Let’s Encrypt is a popular tool to use free SSL for your website. Cpanel comes with Sectigo free ssl service through requesting and pooling system. Although, you might feel interested in getting the SSL released immediately without a queue based approach, and would prefer to use Let’s Encrypt that’s why.

There are two ways, you may install Let’s Encrypt in Cpanel.

  1. Using Cpanel Plugin

First one would be using the plugin created by Cpanel. Login to your server as root:

ssh [email protected]_ip

Then, run the following to install Let’s Encrypt in your cpanel system

/usr/local/cpanel/scripts/install_lets_encrypt_autossl_provider

It might take a couple of minutes, then it should install Let’s Encrypt as a provider in AutoSSL.

Now, go to WHM >> Manage AutoSSL and select Let’s Encrypt as the provider instead of Sectigo Cpanel default. You need to check the Agreement rules under the Let’s Encrypt selection and you may create the account in Let’s Encrypt using the same tool.

Once done, your new SSLs would be issued using the Let’s Encrypt tool through Cpanel AutoSSL plugin.

2. Using FleetSSL

There is a 3rd party tool, existed before Cpanel provided a plugin for Let’s Encrypt. It’s FleetSSL. One key benefit of using FleetSSL is that, it allows the Cpanel end users to control issuing and renewing the SSL from Cpanel. One key cons of using FleetSSL is that, it is not free of charge, it comes with 30$ one time fees. But mainly hosting provider would not mind to use this as it is a nice addition for the end user feature set in a hosting provider’s point of view.

You may check for details here:

https://letsencrypt-for-cpanel.com/

Now, once you complete installing Let’s Encrypt SSL, you may now use Let’s Encrypt for different cpanel services like webmail/cpanel/whm/calenders/MTA services. You may check the following to know how to:

How To: Manually Add Support of SSL for WWW on Cyberpanel

hmm, it’s a weird topic to write blog on. Because Cyberpanel comes with a built in Certbot, and can automatically detects www and without www to install SSL for. Then why am I writing this up? All because I found a VPS client today facing the issue. Even though, Cyberpanel was telling me that the SSL is issued, it was only issued for non-www domain, but the www domain left behind. Let’s see how can we resolve this.

First problem

First problem came up when I tried to discover the Cyberpanel certbot binaries.

[[email protected] /]# find . -name "certbot"
./usr/local/CyberCP/bin/certbot
./usr/local/CyberCP/lib/python3.6/site-packages/certbot
./usr/local/CyberPanel/bin/certbot
./usr/local/CyberPanel/lib/python3.6/site-packages/certbot

[[email protected] live]# /usr/local/CyberCP/bin/certbot --version
certbot 0.21.1
[[email protected]ver-sg live]# /usr/local/CyberPanel/bin/certbot --version
certbot 0.21.1

Both of the certbot I could find from Cyberpanel was very old, Certbot has 1.4 version in the Epel which has support for Acme 2 challenge, while the one that Cyberpanel is using doesn’t. I hence decided to install a certbot for our case:

yum install epel-release
yum install certbot

These should be it for the latest version of certbot to start working in your Cyberpanel host. Once done, you may now generate the SSL using the following:

certbot certonly  --webroot -w /home/yourdomain.com/public_html -d yourdomain.com -d www.yourdomain.com

Remember to replace yourdomain.com with the actual one that is having problem with. Cyberpanel creates the home directory with the primary domain, so the remember to give the correct document root for the value of attribute ‘-w’.

Once this id done, certbot should automatically verify the challenge and get the issued license for you. Lets encrypt license are usually stored at the following directory:

/etc/letsencrypt/live/yourdomain.com/

Files are:
/etc/letsencrypt/live/yourdomain.com/privatekey.pem
/etc/letsencrypt/live/yourdomain.com/fullchain.pem

If you had already created the SSL using Cyberpanel (which you must have done if you viewing this post), then remember, certbot will place the SSLs in /etc/letsencrypt/live/yourdomain.com-001/ folder. The name of the folder would be shown at the time you complete issuing SSL with certbot.

There are couple of ways you may use the SSL now. Either you may replace the old directory with the new, or just change the settings in either the vhost conf or the openlitespeed SSL settings. I find the easiest way is just to replace the old directory with the new. Something like this should work:

mv /etc/letsencrypt/live/yourdomain.com /etc/letsencrypt/live/old_yourdomain.com
mv /etc/letsencrypt/live/yourdomain.com-001 /etc/letsencrypt/live/yourdomain.com

Once this is done, remember to restart your openlitespeed:

service lsws restart

Now your https on the WWW should work without any problem. If not, try clearing your browser cache and retry.

How To: Add Let’s Encrypt SSL in HAProxy – TLS Termination

HAProxy stays in the middle of origin server and the visitors. Hence, You need a SSL for the Visitors to HAProxy. You can use HAProxy is a secure private network to fetch data from backend without any SSL. But the requests between the visitor and HAProxy has to be encrypted. You can use Let’s Encrypt free signed SSL for this purpose.

First, we need to install ‘certbot’, python based client for Let’s Encrypt SSL. It is available in epel repository. In CentOS, you may do the following to install certbot

$ yum install epel-release
$ yum install certbot

Let’s Encrypt uses a Challenge Response technique to verify the host and issue the SSL. While HAProxy is enabled, and used to set to the origin service, this unfortunately, is not possible. certbot comes with an option called ‘standalone’, where it can work as a http server and resolve the Challenge Response issued by Let’s Encrypt. To do this, first we need to stop the haproxy server. You can do this with the following:

# stop haproxy
service haproxy stop

# get the ssl for your domain.com and www.domain.com
certbot certonly --standalone --preferred-challenges http --http-01-port 80 -d www.domain.com -d domain.com

Once this is done, 4 files are saved under /etc/letsencrypt/live/domain.com/

These should be:

cert.pem (Your certificate)
chain.pem
privatekey.pem (Your private key)
fullchain.pem (cert.pem and chain.pem combined)

Now, for haproxy, we need to combine 3 files, cert.pem, chain.pem and privatekey.pem, we can do that by combining fullchain.pem & privatekey.pem. You need to create a directory under /etc/haproxy/certs and then put the file in there. You can do that as following:

# create the directory
mkdir /etc/haproxy/certs

# Combine two files into one in one line
DOMAIN='domain.com' sudo -E bash -c 'cat /etc/letsencrypt/live/$DOMAIN/fullchain.pem /etc/letsencrypt/live/$DOMAIN/privkey.pem > /etc/haproxy/certs/$DOMAIN.pem'

# replace domain.com with each of your domain.

Now, we have the pem file ready to be used on haproxy frontend. To use, you may first edit the haproxy.cfg file, create a new section for frontend https, and use the certificate. An example is given below

frontend main_https
    bind *:443 ssl crt /etc/haproxy/certs/domain.com.pem
    reqadd X-Forwarded-Proto:\ https
    option http-server-close
    option forwardfor
    default_backend app-main

Once the https section is done, you may now want to force the http section to forward to https, you can do as following:

frontend main
    bind *:80
    redirect scheme https code 301 if !{ ssl_fc }
    option http-server-close
    option forwardfor

You should be all set now using Let’s Encrypt with your Haproxy in the frontend.

How to Make Cloudflare Work with HAProxy for TLS Termination

Remember:
This is a part of dirty hack series. This is not the only way you can achieve what we want to achieve. But this is only used when you can trust the connections between your HAProxy and the Origin servers. Otherwise, you should not use this technique.

One common problem with using HAProxy and Cloudflare is that, the SSL that Cloudflare gives us, it gets terminated at HAProxy on L7 load balancer. For such cases, Cloudflare can not verify the Origin server and drops the connection. For such cases, your HAProxy will not work. What would you do for such cases? There are two ways to do this.

First one is, Cloudflare gives you a origin certificate, that you can install at HAProxy. I won’t dig into deep into this in this blog post.

But if you can trust your connections between HAProxy and backend Origin servers, as well as the connections between Cloudflare and HAproxy, you can choose the second one. For this case, Cloudflare allows you to Encrypt only the connections between the Visitors and Cloudflare. It won’t matter what you are doing behind the Cloudflare. This option is called ‘Flexible’ option, that you can select from your Cloudflare >> SSL/TLS tab.

Fix TLS Termination by HAProxy with Flexible Encryption Mode of Cloudflare

Once you set this to Flexible, this should start working ASAP. Remember, this is not essentially the best way to do this, but the quickest way only if load balancing is more important to you instead the data integrity.