How to manually install/renew Let’s Encrypt SSL in Zimbra

If you are having trouble installing Let’s Encrypt SSL with the certbot-zimbra.sh file, then probably you would need to follow this tutorial. To follow this tutorial, we first need to install certbot. certbot has a built in web server to allow you get the certificate without actually installing an extra web server or through Zimbra web server (nginx to be specific).

First, we install certbot with the following:

// install epel-release first
yum install epel-release
// install certbot from epel
yum install certbot

Once done, you may now use the following command to ensure certbot is working:

# certbot --help

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

  certbot [SUBCOMMAND] [options] [-d DOMAIN] [-d DOMAIN] ...

Certbot can obtain and install HTTPS/TLS/SSL certificates.  By default,
it will attempt to use a webserver both for obtaining and installing the
certificate. The most common SUBCOMMANDS and flags are:

obtain, install, and renew certificates:
    (default) run   Obtain & install a certificate in your current webserver
    certonly        Obtain or renew a certificate, but do not install it
    renew           Renew all previously obtained certificates that are near
expiry
    enhance         Add security enhancements to your existing configuration
   -d DOMAINS       Comma-separated list of domains to obtain a certificate for

  (the certbot apache plugin is not installed)
  --standalone      Run a standalone webserver for authentication
  --nginx           Use the Nginx plugin for authentication & installation
  --webroot         Place files in a server's webroot folder for authentication
  --manual          Obtain certificates interactively, or using shell script
hooks

   -n               Run non-interactively
  --test-cert       Obtain a test certificate from a staging server
  --dry-run         Test "renew" or "certonly" without saving any certificates
to disk

manage certificates:
    certificates    Display information about certificates you have from Certbot
    revoke          Revoke a certificate (supply --cert-name or --cert-path)
    delete          Delete a certificate (supply --cert-name)

manage your account:
    register        Create an ACME account
    unregister      Deactivate an ACME account
    update_account  Update an ACME account
  --agree-tos       Agree to the ACME server's Subscriber Agreement
   -m EMAIL         Email address for important account notifications

More detailed help:

  -h, --help [TOPIC]    print this message, or detailed help on a topic;
                        the available TOPICS are:

   all, automation, commands, paths, security, testing, or any of the
   subcommands or plugins (certonly, renew, install, register, nginx,
   apache, standalone, webroot, etc.)
  -h all                print a detailed help page including all topics
  --version             print the version number
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Once you ensure certbot is installed, now you may use certbot to get the certificate, using the certbot –standalone tag. Remember to stop zimbra first, as Zimbra also runs a nginx web server, that would prevent certbot to use standalone or it’s own web server to verify certificate.

// from root, run
[[email protected] ~]# service zimbra stop

// wait until zimbra stops, once done, use the following to get certificate for your domain/hostname in place of mail.domain.com
[[email protected] ~]# certbot certonly --standalone -d mail.domain.com

This would get your certificate and save it in:

/etc/letsencrypt/live/mail.domain.com

Now, that folder would contain 4 files. Something like the following:

]# ls -la /etc/letsencrypt/live/mail.domain.com/
total 16
drwxr-xr-x 2 root root 4096 Apr 16 11:30 .
drwx------ 4 root root 4096 Feb 10  2020 ..
lrwxrwxrwx 1 root root   40 Apr 16 11:30 cert.pem -> ../../archive/mail.domain.com/cert8.pem
lrwxrwxrwx 1 root root   41 Apr 16 11:30 chain.pem -> ../../archive/mail.domain.com/chain8.pem
lrwxrwxrwx 1 root root   45 Apr 16 11:30 fullchain.pem -> ../../archive/mail.domain.com/fullchain8.pem
lrwxrwxrwx 1 root root   43 Apr 16 11:30 privkey.pem -> ../../archive/mail.domain.com/privkey8.pem

As you can see, these files are symbolically linked to another files, depends on how many time you are running certbot. Each time, it generates a number liker cert8.pem, the next one would be cert9.pem and so on. So the orignal files are here:

/etc/letsencrypt/archive/mail.domain.com/cert8.pem
/etc/letsencrypt/archive/mail.domain.com/chain8.pem
/etc/letsencrypt/archive/mail.domain.com/fullchain8.pem
/etc/letsencrypt/archive/mail.domain.com/privkey8.pem

Now, we have our certificates. We need to follow a couple of steps to make sure everything is set correctly.

First, zimbra SSL files are stored here

/etc/zimbra/ssl/letsencrypt

We clean all old pem files

rm -f /etc/zimbra/ssl/letsencrypt/*

Now, copy the pem files we got to this folder with the following:

cp /etc/letsencrypt/archive/mail.domain.com/cert8.pem /opt/zimbra/ssl/letsencrypt/cert.pem
cp /etc/letsencrypt/archive/mail.domain.com/chain8.pem /opt/zimbra/ssl/letsencrypt/chain.pem
cp /etc/letsencrypt/archive/mail.domain.com/fullchain8.pem /opt/zimbra/ssl/letsencrypt/fullchain.pem
cp /etc/letsencrypt/archive/mail.domain.com/privkey8.pem /opt/zimbra/ssl/letsencrypt/privkey.pem

Check, how we are renaming all the files with number to file name without number, like cert8.pem is moved as cert.pem here.

Now, change the ownership of these files to zimbra with the following:

chown -Rf zimbra:zimbra /opt/zimbra/ssl/letsencrypt/*

Now, we are done from root, change your ownership to zimbra

su - zimbra

First job, is to change your directory to the ‘/opt/zimbra/ssl/letsencrypt/’

cd /opt/zimbra/ssl/letsencrypt/

Let’s Encrypt files are very much ready to use, only with one problem. Let’s Encrypt do not add it’s root CA certificate with it’s chain.pem file. We need to do this. First open the certificate with nano editor as following:

nano chain.pem

Now, at the end of the file, add the following section:

-----BEGIN CERTIFICATE-----
MIIDSjCCAjKgAwIBAgIQRK+wgNajJ7qJMDmGLvhAazANBgkqhkiG9w0BAQUFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
DkRTVCBSb290IENBIFgzMB4XDTAwMDkzMDIxMTIxOVoXDTIxMDkzMDE0MDExNVow
PzEkMCIGA1UEChMbRGlnaXRhbCBTaWduYXR1cmUgVHJ1c3QgQ28uMRcwFQYDVQQD
Ew5EU1QgUm9vdCBDQSBYMzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
AN+v6ZdQCINXtMxiZfaQguzH0yxrMMpb7NnDfcdAwRgUi+DoM3ZJKuM/IUmTrE4O
rz5Iy2Xu/NMhD2XSKtkyj4zl93ewEnu1lcCJo6m67XMuegwGMoOifooUMM0RoOEq
OLl5CjH9UL2AZd+3UWODyOKIYepLYYHsUmu5ouJLGiifSKOeDNoJjj4XLh7dIN9b
xiqKqy69cK3FCxolkHRyxXtqqzTWMIn/5WgTe1QLyNau7Fqckh49ZLOMxt+/yUFw
7BZy1SbsOFU5Q9D8/RhcQPGX69Wam40dutolucbY38EVAjqr2m7xPi71XAicPNaD
aeQQmxkqtilX4+U9m5/wAl0CAwEAAaNCMEAwDwYDVR0TAQH/BAUwAwEB/zAOBgNV
HQ8BAf8EBAMCAQYwHQYDVR0OBBYEFMSnsaR7LHH62+FLkHX/xBVghYkQMA0GCSqG
SIb3DQEBBQUAA4IBAQCjGiybFwBcqR7uKGY3Or+Dxz9LwwmglSBd49lZRNI+DT69
ikugdB/OEIKcdBodfpga3csTS7MgROSR6cz8faXbauX+5v3gTt23ADq1cEmv8uXr
AvHRAosZy5Q6XkjEGB5YGV8eAlrwDPGxrancWYaLbumR9YbK+rlmM6pZW87ipxZz
R8srzJmwN0jP41ZL9c8PDHIyh8bwRLtTcm1D9SZImlJnt1ir/md2cXjbDaJWFBM5
JDGFoqgCWjBH4d1QB7wCCZAA62RjYJsWvIjJEubSfZGL+T0yjWW06XyxV3bqxbYo
Ob8VZRzI9neWagqNdwvYkQsEjgfbKbYK7p2CNTUQ
-----END CERTIFICATE-----

After adding the above, your chain.pem file should look like the following

-----BEGIN CERTIFICATE-----
your chain pem encrypted certificate here
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIDSjCCAjKgAwIBAgIQRK+wgNajJ7qJMDmGLvhAazANBgkqhkiG9w0BAQUFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
DkRTVCBSb290IENBIFgzMB4XDTAwMDkzMDIxMTIxOVoXDTIxMDkzMDE0MDExNVow
PzEkMCIGA1UEChMbRGlnaXRhbCBTaWduYXR1cmUgVHJ1c3QgQ28uMRcwFQYDVQQD
Ew5EU1QgUm9vdCBDQSBYMzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
AN+v6ZdQCINXtMxiZfaQguzH0yxrMMpb7NnDfcdAwRgUi+DoM3ZJKuM/IUmTrE4O
rz5Iy2Xu/NMhD2XSKtkyj4zl93ewEnu1lcCJo6m67XMuegwGMoOifooUMM0RoOEq
OLl5CjH9UL2AZd+3UWODyOKIYepLYYHsUmu5ouJLGiifSKOeDNoJjj4XLh7dIN9b
xiqKqy69cK3FCxolkHRyxXtqqzTWMIn/5WgTe1QLyNau7Fqckh49ZLOMxt+/yUFw
7BZy1SbsOFU5Q9D8/RhcQPGX69Wam40dutolucbY38EVAjqr2m7xPi71XAicPNaD
aeQQmxkqtilX4+U9m5/wAl0CAwEAAaNCMEAwDwYDVR0TAQH/BAUwAwEB/zAOBgNV
HQ8BAf8EBAMCAQYwHQYDVR0OBBYEFMSnsaR7LHH62+FLkHX/xBVghYkQMA0GCSqG
SIb3DQEBBQUAA4IBAQCjGiybFwBcqR7uKGY3Or+Dxz9LwwmglSBd49lZRNI+DT69
ikugdB/OEIKcdBodfpga3csTS7MgROSR6cz8faXbauX+5v3gTt23ADq1cEmv8uXr
AvHRAosZy5Q6XkjEGB5YGV8eAlrwDPGxrancWYaLbumR9YbK+rlmM6pZW87ipxZz
R8srzJmwN0jP41ZL9c8PDHIyh8bwRLtTcm1D9SZImlJnt1ir/md2cXjbDaJWFBM5
JDGFoqgCWjBH4d1QB7wCCZAA62RjYJsWvIjJEubSfZGL+T0yjWW06XyxV3bqxbYo
Ob8VZRzI9neWagqNdwvYkQsEjgfbKbYK7p2CNTUQ
-----END CERTIFICATE-----

Now, save the file (CTRL + o) and exit (CTRL + x)

We need to do one more thing before we are ready to verify and deploy the certificate. We need to set the letencrypt private key that we used to generate the certificate as commercial.key of zimbra. You may do this with the following two commands:

rm -f /opt/zimbra/ssl/zimbra/commercial/commercial.key
cp /opt/zimbra/ssl/letsencrypt/privkey.pem /opt/zimbra/ssl/zimbra/commercial/commercial.key

Now, you are ready to complete the job. First verify if everything is alright with the following:

[[email protected] letsencrypt]$ /opt/zimbra/bin/zmcertmgr verifycrt comm privkey.pem cert.pem chain.pem
** Verifying 'cert.pem' against 'privkey.pem'
Certificate 'cert.pem' and private key 'privkey.pem' match.
** Verifying 'cert.pem' against 'chain.pem'
Valid certificate chain: cert.pem: OK

If everything is ok, you may now deploy certificate with the following command:

/opt/zimbra/bin/zmcertmgr deploycrt comm cert.pem chain.pem

Once the certificate is deployed successfully, get out from the zimbra user to root user with the following command

exit

Now, you may start/restart zimbra with the following command:

service zimbra restart

If everything went right, you should now be able to go to your zimbra domain, and under the lock sign on the left of the domain shown in browser, you may click on it to see the extended date of ssl expiry. Sweet!

How to List members of Distribution List in Zimbra

Problem Definition:

Zimbra provides the ability to use a distribution list, that allows you t add members to the list, and when you send mail to the distribution list, zimbra picks all the members and sends the same mail to all the members. This allows you to easily do group mailing or department wise mailing in a company. Zimbra admin panel does not allow you to list/download all the members in one page and download them. One option available from admin panel is to go to the details of the distribution list email address and from the Member pane, you may download the first page of the list. But the page does not allow you to move on. How can we download all the members of distribution list in Zimbra in one command?

How To List/Download Members of Distribution List in Zimbra

You may do it using the zimprov command given by zimbra. Here are the steps to do so:

~ su - zimbra
~ for i in `zmprov gadl` ; do zmprov gdl $i zimbraMailAlias zimbraMailForwardingAddress ; done

# if you want to store them in a file and download them:
for i in `zmprov gadl` ; do zmprov gdl $i zimbraMailAlias zimbraMailForwardingAddress ; done > /tmp/dd_users.txt

How to Install Zimbra Universal UI Theme on Zimbra 8.8.*

WHAT IS ZIMBRA UNIVERSAL UI

Zimbra universal UI is a theme by Zimbra, that is much more modern, professional looking theme, looks much near to Outlook theme. According to Zimbra, this is still in Beta since 2017, and not available anywhere unfortunately to install/download it from Zimbra until you are using one of the Zimbra Cloud by Zimbra Inc.

HOW DID WE MANAGE TO GET ZIMBRA UNIVERSAL UI?

Zeta Alliance had released an unofficial Zimbra 8.8.5 with the Zimbra Universal UI theme in it, that is downloadable here:

zcs-8.8.5_GA_2000.RHEL7_64.20171222124635.tgz

We installed it and tweaked the ZUU theme to be able to use in any Zimbra 8.8.* version available.

HOW TO INSTALL ZIMBRA UNIVERSAL UI

First keep a copy of zimbra folder:

rsync -vrplogDtH /opt/zimbra/jetty/webapps/zimbra /root/zimbra_backup

Then, download a copy of the zimbra UI from here:

wget https://mellowhost.com/downloads/zimbra-ui.tgz

Untar and migrate to zimbra folder:

tar -xvzf zimbra-ui.tgz
rsync -vrplogDtH zimbra/ /opt/zimbra/jetty/webapps/zimbra/

(Optional) This will install a new theme called ‘clarity’ which is our Zimbra Universal UI theme. You may either use this, and advise your users to choose this as a theme, or replace the default Harmony with this one. To do this, you may do the following:

cd /opt/zimbra/jetty/webapps/zimbra/skins
mv harmony harmony_old
ln -sf clarity harmony

This is done only by creating a softlink for old harmony to clarity theme

Now fix the permissions and restart zimbra to make everything work:

# Fix permission
/opt/zimbra/libexec/zmfixperms -v
service zimbra restart

You may now access Zimbra from your browser to see the new theme loading. If you are having trouble with the http access for your webmail, you may use the following to redirect your http webmail to https webmails

How to Fix zmconfigd failed in Zimbra – Starting zmconfigd…failed.

Sometimes, if you restart Zimbra, you see zmconfigd is not starting or saying it’s failed. You may also see the zmconfigd service is not running in the Zimbra admin panel. There are couple of common reasons why zmconfigd fails to start.

Disable IPv6

One reason of zmconfigd fails to start is IPv6, for some reason, it fails to route the IPv6 and fails to start. A quick solution to this problem is to disable ipv6 and restart zmconfigd. You may do this like the following:

#Edit your sysctl.conf file
nano /etc/sysctl.conf

# paste the following inside the file
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1

# Save the file, and update sysctl in realtime
sysctl -p

# now try to restart zmconfigd
su - zimbra
zmconfigdctl restart

Now you can check the zmconfigd status with the following, to know if it’s running or not:

[[email protected] ~]# cat /opt/zimbra/log/zmconfigd.pid
19722

If it returns an ID, it means the zmconfigd is running.

Netcat is not installed

Another reason of the error could be because nc is not installed in your system. Zimbra zmconfigd has a dependency on netcat package. Netcat is available through nmap-ncat in centos systems. You may run the following to install netcat:

yum install nc
# or 
yum install nmap-netcat

How to Redirect HTTP to HTTPs Zimbra 8.8.*

Zimbra Supports HTTPs by Default:

By default Zimbra will use HTTPs support only and disable HTTP use on the webmail client. But users will always use non http port to access the webclient. Users do not like to type https before the domain each time to get into the webmail client. Zimbra uses Nginx to run the proxy services to access the Javamail Client of Zimbra. Zimbra supports 5 types of proxy services through Nginx:

  1. redirect
  2. both
  3. http
  4. https
  5. mixed

You may check the following for details:

Enabling_Zimbra_Proxy_and_memcached

How to Redirect HTTP to HTTPs automatically in Zimbra 8.8*

The most popular out of 5 options for proxy services, is to redirect. To do this, you can run the following:

zmprov ms `zmhostname` zimbraReverseProxyMailMode redirect

This will redirect your URLs to the zimbra hostname based HTTPs.

Now, restart the proxy services:

su - zimbra
zmproxyctl restart

Hope this helps.

How To: Restore Zimbra Quarantined Email by Clam AKA Heuristics.Encrypted.PDF Release Point

Zimbra Mail Server automatically quarantines emails that get hit by the Antivirus scan using Clam when the mail is received. While putting the email on the recipient inbox, what it does, instead of giving the original email with the attachment, it sends a virus detected email with the following kind of error message:

Virus (Heuristics.Encrypted.PDF) in mail to YOU

Virus Alert
Our content checker found
virus: Heuristics.Encrypted.PDF

by Zimbra

It actually means, the original mail is now quarantined. Zimbra maintains a virus quarantine email account that is not normally available in the ‘Manage Account’ list of Zimbra Admin panel. You can find it if you search with ‘virus’ in the ‘Search’ box of the admin panel. What zimbra does in quarantine situation, is that, it pushes the mail to the quarantine email instead of original recipient.

Now, to get back the mail delivered to the original recipient, we need to first get the quarantine email account, get the message id, and then we need to inject the mail into the LMTP pipe that bypasses any scanning. Here are the steps on how to do this:

# First get to the zimbra user
$ su - zimbra

# Get the email account that is used to store virus detected mails
$ zmprov gcf zimbraAmavisQuarantineAccount
zimbraAmavisQuarantineAccount: [email protected]

# [email protected] this should be our quarantine email account, now we need to get the quarantine account's mailbox id
$ zmprov gmi [email protected]
mailboxId: 73
quotaUsed: 644183

# Mailbox id here for the quarantine account is 73. Now go to the message storage of this id using the following command: cd /opt/zimbra/store/0/<mailboxId>/msg/0
$ cd /opt/zimbra/store/0/73/msg/0

# list the messages
$ ls *

These are your quarantined emails. Now for example the complainer is ‘[email protected]’. To search for the emails designated for this email account, you may use the following:

$ grep -l [email protected] *
281-1216.msg
300-1400.msg
301-1476.msg

This should return you all the emails that got quarantined for the above user.

Now the question is, how can we get these emails delivered to the designated user bypassing the antivirus/antispam tools. To do this, you need to inject the mail into LMTP pipe. You may do this using ‘zmlmtpinject’ command as following:

$ zmlmtpinject -r [email protected] -s [email protected] 281-1216.msg

Remember, to change [email protected] to the original recipient. [email protected] would be the newly rewritten sender for this mail delivery and ‘281-1216.msg’ is the file name of the original email that you found out from the grep command. You can do lmtp injections for one email mail with each command. So, you would require to do this for each emails.

How to Add a Zimbra User to Allow Distribution List Creation

NB: This is going to be another documentation purpose post.

A distribution list allow you to create a mailing list. So for example if you have a CRM with a member of 100, you want to add them to a list, so that you do not need to email each of them distinctively when required, instead, you keep a list with an email like [email protected] and allow somebody to shoot at that email, which would make sure all the CRM members get the email. In Zimbra, there is a feature called ‘Distribution List’. By default this is only permitted to admin user to create. But in case, you want to permit a user to create distribution list, you would need to use ‘zimprov’ command. Here is the reference:

# su - zimbra
# zmprov grantRight domain yourzimbradomain.com usr [email protected] createDistList

Fairly simple!

How To Renew & Deploy Let’s Encrypt SSL on Zimbra Server – 2020

Note: This does not seem to work on 2021. I have written another article on how to do this now: How to manually install/renew let’s encrypt ssl in Zimbra

Ok, there is a reason to put 2020 on the title. Because the process has changed since past. At this moment, I manage a Zimbra server with multiple domains in it, which won’t deploy the ‘other’ domains if not specified. The process is fairly simple, but I am keeping this as a documentation purpose, so that I don’t miss next time.

To renew the certificate for attached domains using certbot is fairly simple, just do:

# certbot renew

Once done, you you want to use the pre-hook and deploy-hook to do the patching and deploying as following using certbot_zimbra.sh

# certbot_zimbra.sh -p
# certbot_zimbra.sh -r -d 'your_domain'

Updated, certbot_zimbra doesn’t take this. ‘-n’ used to be taken as new and ‘-r’ for replacing, now, ‘-r’ is removed. Instead you can use ‘-e’ to specify new domains. So the command for replacement and deployment would be fairly simple as following:

# certbot_zimbra.sh -p
# certbot_zimbra.sh -d -e 'mail.yourdomain.com'
# certbot_zimbra.sh -d -e 'mailapp.yourdomain.com'


… and so on. At this moment, I couldn’t find a way to advise zimbra certbot to follow a list of domains instead of one. But this is probably possible by cracking the certbot.