Author: Surid

Troubleshoot: fatal: open lock file /var/lib/postfix/master.lock: unable to set exclusive lock

Error Message & Trace details:

One of my customer came with an error saying the postfix in his server isn’t working. The server was running CentOS 7, and the system postfix status was inactive, means not running. Although, the system queue was running I could see. The error that was returning while restarting/checking status was the following:

# service postfix status
Redirecting to /bin/systemctl status postfix.service
● postfix.service - Postfix Mail Transport Agent
Loaded: loaded (/usr/lib/systemd/system/postfix.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Tue 2018-01-09 04:04:05 UTC; 1s ago
Process: 9201 ExecStart=/usr/sbin/postfix start (code=exited, status=1/FAILURE)
Process: 9197 ExecStartPre=/usr/libexec/postfix/chroot-update (code=exited, status=0/SUCCESS)
Process: 9194 ExecStartPre=/usr/libexec/postfix/aliasesdb (code=exited, status=0/SUCCESS)
Main PID: 1358 (code=killed, signal=TERM)

Jan 09 04:04:03 twin7.hifrank.biz systemd[1]: Starting Postfix Mail Transport Agent...
Jan 09 04:04:03 twin7.hifrank.biz postfix/master[9273]: fatal: open lock file /var/lib/postfix/master.lock: unable to set exclusive lock: Resource tempo...vailable
Jan 09 04:04:04 twin7.hifrank.biz postfix/master[9272]: fatal: daemon initialization failure
Jan 09 04:04:05 twin7.hifrank.biz postfix/postfix-script[9274]: fatal: mail system startup failed
Jan 09 04:04:05 twin7.hifrank.biz systemd[1]: postfix.service: control process exited, code=exited status=1
Jan 09 04:04:05 twin7.hifrank.biz systemd[1]: Failed to start Postfix Mail Transport Agent.
Jan 09 04:04:05 twin7.hifrank.biz systemd[1]: Unit postfix.service entered failed state.
Jan 09 04:04:05 twin7.hifrank.biz systemd[1]: postfix.service failed.

How to fix:

The error to note here is the following:

fatal: open lock file /var/lib/postfix/master.lock

I first killed the smtp and smtpd processes that runs by postfix:

# killall -9 smtp
# killall -9 smtpd

But that didn’t solve the problem. I then used the fuser command to check which process holds the lock file:

# fuser /var/lib/postfix/master.lock
/var/lib/postfix/master.lock: 18698

Then we check the process 18698 and kill the responsible process:

# ps -axwww|grep 18698
9333 pts/0 S+ 0:00 grep --color=auto 18698
18698 ? Ss 4:28 /usr/libexec/postfix/master -w
# killall -9 /usr/libexec/postfix/master
or
# kill -9 18698

Once the process is killed, you can now start the postfix:

# service postfix start
# service postfix status|grep Active
Redirecting to /bin/systemctl status postfix.service
Active: active (running) since Tue 2018-01-09 04:15:50 UTC; 4min 45s ago

Troubleshoot: killall command not found in centos 7

Problem:

If you are using a centos 7 minimal installation, and trying to kill process by name using the command ‘killall’, you are most likely going to see the error:

# killall -9 php
killall: command not found

The error appears because CentOS 7 encourages you to use pkill instead of killall to kill process by name. pkill has versatile application, although, it can be used to kill process by name same as killall.

How to kill process by name in Centos 7

You can use pkill. pkill is a simple command. It’s syntax is as following:

# pkill processname

For example, if you want to kill all the php process, run:

# pkill php

Note: It will kill all the processes that match php. To list the process that pkill going to kill, you can use pgrep as following:

# pgrep -l php

How to use killall in Centos 7

If you do not want to use pkill, and keep using killall commands in centos 7, this is also possible. killall is a part of psmisc yum package. All you have to do, is to install psmisc in your system using yum

# yum install psmisc
# killall -9 php

 

How to install ifconfig in CentOS 7

CentOS 7 doesn’t come with ifconfig tools. It encourages users to use ‘ip’ tool for network administration. Although, it is still possible to use ifconfig with CentOS 7. ifconfig is a part of net-tools package. All you have to do is to install the net-tools package using yum.

How to install ifconfig in CentOS 7

Run the following command to install net-tools package in CentOS 7, this will install ifconfig as well:

# yum install net-tools -y
# ifconfig

How to: Change Timezone in CentOS / RHEL 6 & CentOS / RHEL 7

How to change Timezone in CentOS 6 / RHEL 6

In CentOS 6, timezone files are located under /usr/share/zoneinfo. So, if your zone is for example, America/Chicago (UTC -6), it would be /usr/share/zoneinfo/America/Chicago and so on.

CentOS 6, uses a file called ‘localtime’ located under /etc to determine it’s currently set timezone.

# ls -la /etc/localtime

This file, is either the actual time zone file moved to this location or a symlink to the timezone under zoneinfo directory. So if you want to change the timezone, first you need to determine which timezone to use and then symlink it to localtime. You can do that using the following:

# rm -f /etc/localtime
# ln -s /usr/share/zoneinfo/Asia/Dhaka /etc/localtime
# date

This would set the current timezone to GMT +6 BDT or Asia/Dhaka timezone, zone I belong to.

How to change timezone in CentOS 7 or RHEL 7

CentOS 7, comes with a tool called ‘timedatectl’. This can be used to find and set the symlink for you instead of doing the work that were required in CentOS 6.

To list available time zones, run:

# timedatectl list-timezones

You can find your desired timezone, as following:

# timedatectl list-timezones | grep Chicago

Now, to set a time zone, use the command set-timezone with timedatectl command. For example, if I want to set the time zone to America/Chicago, I would run the following:

# timedatectl set-timezone America/Chicago
# date

This also should create the symlink of locatime file to the zoneinfo directory. You can see that with the following:

# ls -l /etc/localtime

How To: Get Username from UID & Vice Versa in Linux

Question:

I have an UID, how do I get the username belongs to this UID in Linux?

How to Get UID from USERNAME in Linux

We usually know, there is a dedicated command called ‘id’ in linux given to find UID from username is called ‘id‘

You can use that to get the UID from the username in linux:

# id -u root
0

How to get Username from UID in Linux

Although, there is no built in command get fetch the username from the UID. We can use a pipe and regular expression match on getent to do that.

getent is a unix command that helps a user get entries in a number of important text files called databases. This includes the passwd and group databases which store user information – hence getent is a common way to look up user details on Unix.

You can use the following command to find username of the UID 752 for example in a system:

# getent passwd "752"|cut -d: -f1
texstard

getent can take group database too, although, we have used passwd database as that contains the UID of the respective linux user.

Troubleshooting: Imunify360 database is corrupt. Application cannot run with corrupt database

Error Message:

# service imunify360 start
Starting imunify360: WARNING [+ 3743ms] defence360agent.utils.check_db|DatabaseError detected: database disk image is malformed
WARNING [+ 3766ms] defence360agent.cli.subparsers.common.server|Imunify360 database is corrupt. Application cannot run with corrupt database. Please, contact Imunify360 support team at https://cloudlinux.zendesk.com

Detail Information & Explanation:

If you are using imunify360, an application firewall for linux servers by Cloudlinux team, you might incur an error where it says the database is corrupt. You might first see ‘Imunify360 is not started’ error from the WHM panel and end up getting the above error message as stated. Imunify360 uses a SQL database, located under ‘/var/imunify360/imunify360.db’. This image is checked everytime Imunfi360 tries to start, and if the database is malformed, it would not start. Fortunately, imunify360 comes with tools to handle this database and recover if corrupted.

How to Fix:

First, we start by running database integrity check. This can be done using the following:

imunfiy360-agent checkdb

(From Imunify360 Doc: checkdb  – Check database integrity)

Once done, you can now use ‘migratedb’ to repair and restore if the database is corrupted.

imunify360-agent migratedb

(From Imunify360 Doc: migratedb – Check and repair database if it is corrupted.)

If migratedb fails, the only way to recover this is to reinstall imunify360.

Linux How To: Install IPTABLES in CentOS 7 / RHEL 7 Replacing FirewallD

CentOS 7 / RHEL 7 doesn’t come with iptables by default. It uses a full functional firewall system called ‘firewalld’. I have been a big fan of iptables and it’s capability from the very first, and since I have switched to CentOS 7, I couldn’t stop using it. I had to stop firewalld and install iptables in all of my CentOS 7 installation and start using iptables rules as I was using before. Here is a small How To guide on installing Iptables and disabling firewalld from a CentOS 7 or RHEL 7 or a similar variant distro.

How to Install IPTABLES in CentOS 7

To begin using iptables, you need to download and install iptables-service package from the repo. It isn’t installed automatically on CentOS 7. To do that, run the following command:

# yum install iptables-services -y

How to stop the firewalld service and start the Iptables service

Once the iptables-serivces package is installed, you can now stop the firewalld and start the iptables. Keeping both kind of network filtering too can create conflicts and it is recommended to use any out of two. To do that run the following:

# systemctl stop firewalld
# systemctl start iptables

Now to disable firewalld from the starting after the boot, you need to disable the firewalld:

# systemctl disable firewalld

To disallow starting firewalld manually as well, you can mask it:

# systemctl mask firewalld

Now you can enable iptables to start at the boot time by enabling iptables using systemctl command:

# systemctl enable iptables

How to check status of iptables in centOS 7

In previous distros, iptables status could be fetched using service command, although, the option is no longer available in CentOS 7. To fetch the iptables status, use the following:

# iptables -S

Iptables save command can still be used using service tool:

# service iptables save

This would save your iptables rules to /etc/sysconfig/iptables as it used to do in previous distros.

Linux: Assertion failed on job for iptables.service.

If you are using Centos 7 or RHEL 7 or any of it’s variant, you are probably using ‘Firewalld’ by default. Although, if you are a iptables fan like me, who likes it’s simplicity and manipulative nature instead of a full form firewall, then you probably have disabled firewalld from your CentOS 7 instance and using iptables. There are couple of servers, where I use runtime iptables rules for postrouting and masquerading. These rules are dynamically generated by my scripts instead of the sysconfig file under:

/etc/sysconfig/iptables

This file is generated upon running the iptables save command:

service iptables save

which I rarely do so.

Error Details

Which is why, I don’t have a /etc/sysconfig/iptables file in those servers and a common error I see while restarting iptables in those system is the following:

# systemctl restart iptables.service
Assertion failed on job for iptables.service.

How to Fix The Error

The error appears because you don’t have any rule in /etc/sysconfig/iptables or the file doesn’t exist either. You can ignore the error as iptables would still run. To eradicate the error, simply make sure you have some iptables rules loaded on your system using the status command:

iptables -S

And then, run:

service iptables save

Once done, restarting iptables shouldn’t show the error any longer.

Quick Tip: How to view public IP using SSH terminal/cURL/wget

There are times when you might require to view the IP address your server is using for outgoing connections. If you are in command line/console/mosh/ssh/rsh, you want an one command solution instead of visiting a page like whatismyip.com using lynx browser or so on. Here is a quick tip that I regularly use to perform this:

Using cURL:

curl ifconfig.co

Using wget:

wget -qO- ifconfig.co

ifconfig.co does it simple and easy.

SMTP Error: 550 Please turn on SMTP Authentication in your mail client – IP is not permitted to relay through this server without authentication

We had a customer complaining about a commonly seen error of the following type:

550 Please turn on SMTP Authentication in your mail client. mail-pf0-f172.google.com [209.85.192.172]:38632 is not permitted to relay through this server without authentication.

Diagnostic-Code: smtp; 550-Please turn on SMTP Authentication in your mail client. 550-mail-pf0-f172.google.com [209.85.192.172]:38632 is not permitted to relay 550 through this server without authentication.

reason: 550-Please turn on SMTP Authentication in your mail client.
550-mout.kundenserver.de [212.227.17.24]:49392 is not permitted to relay
550 through this server without authentication.

They were all basically the same error. This is a common error and the solution is pretty simple as it looks like. Enabling ‘SMTP Authentication’ on the outlook or the mail client should solve the problem. But interestingly, the client was smart and he wasn’t doing any mistake with ‘SMTP authentication’. The error was actually showing up when someone was trying to send the mail to him (As a receiver SMTP). We then tried digging the error further.

There is something we need to remember. SMTP is not only authenticated using username and password, it also goes through a dns authentication check too. If your dkim/domainkeys/spf/dmarc do not match as the mail server has advised, the mail will get denied with the same type of error (Error code 550). We then realized the customer account was transfered earlier from a different server and the old domainkeys were still there in it’s DNS zone file. As domainkeys are RSA keys generated per server, it is important to regenerate the keys after the server change. Otherwise, the old key check through the DNS can trigger the 550 error from the receiver relay. We had deleted and generated a new domainkeys for the customer and the error went off.