Category: Tips & Tricks

How to: Find IOPS usage in a Linux Server

Question: How to find iops usage of a linux server?

Answer: Use iostat. Iostat is a tool comes with the ‘sysstat’ package. If you type iostat on your CentOS/Redhat server and it says the command not found, you can install sysstat to avail the iostat command.

yum install -y sysstat

An example iostat usage case could as simple as following:

iostat -x 1

-x tells iostat to give extended statistics which is required to find read/write iops individually. And the 1 tells iostat to repeat the command every 1s.

An example output would be like the following:

If you look at the output, the colum r/s would say the read iops and the colum w/s would say write iops. If you are using simple ‘iostat 1’ then the column tps should show the total iops of the disk in use.

If you are using a spinning disk, and if you are getting anything around 150-200 cumulatively, you are probably hitting the iops limit. With raid, the number would change according to your raid choice. Although, the number can increase in case of using Writeback SSD Cache, Hardware RAID Cache or Pure SSD disks. Most important benefit of using SSD is not essentially the amount of throughput it gives in a practical environment instead the amount of IOPS it can sustain is phenomenal.

Quick How To: Finding IO Abuser in KVM VM

I thought to write a quick how to on finding an abuser in a KVM VM Host. There is a tool shipped with libvirt is called ‘virt-top’. Virt-top  has many usage case. It can be used to detect the IO Abuser. Most of the cases, you would see the abuser is throwing a lot of IO Requests regardless of the amount of IO being written or read. Which is why, it important to first identify if you are hitting the IOPS limit of your disk or not by using iostat. A common tool I regularly use to identify first hand disk problem is iotop as well. The following is the favorite iotop command:

iotop -oaP

-o will only show the threads that are actually doing IO in the server instead of all the sleeping threads, keeping the iotop result clean. ‘P’ will show only the processes instead of every single threads. Each VM can have thousands of threads which will show up on the process ID. ‘a’ is specifically my favorite, that does accumulated output. It will show you the sum of the usage for the time your interactive iotop is running.

Once you are done with the first hand investigation, you may now use virt-top to detect the VM activity further. A most used command for me to detect IO abuser is the following:

virt-top -3 –block-in-bytes -o blockwrrq

-3 tells the virt-top to find block device usage and find them by ‘bytes’ while the -o ‘blockwrrq’ means to sort the output by the write iops of the VM. You can use blockrdrq to sort the result by read iops too.

Once you can mix the output of virt-top and iotop results, you shouldn’t have difficulty to detect the VM that is abusing the IO on the server.

How to Enable TXT Record Addition/Edition in Cpanel

By default, if you are trying to add TXT record from cpanel, you would probably see a screen like the following

It doesn’t show the option to add TXT record. Basically, cpanel won’t allow the cpanel users to add TXT record with Simple DNS Zone Editor permission. It allows adding A, CNAME & MX record with that feature/permission set. To add permission to add TXT record and other, you need to enable Advanced DNS Zone Editor for the user’s feature list. If the user is under default list, go to root WHM >> Feature Manager >> Edit Default Feature list and enable Advanced DNS Zone Editor as following:

What is Kondemand? Why do I see a lot of Kondemand process in my process list?

Question: What is Kondemand? Why do I see a lot of Kondemand process in my process list?

Answer: Kondemand is the process used for automatic CPU scaling on multi core linux system. It automatically reduce/drops the CPU clock speed to power usage when the CPU is not in use. This is done through scaling_governor available on linux. To see if your scaling_governor is set to ‘ondemand’ or not, you may use the following command:

# cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

If your CPU is showing ‘ondemand’ scanling governor then the kondemand kernel process is active and will reduce your CPU clock speed on fly to reduce power usage. You can change this settings to performance on fly using the following small shell code:

for CPUFREQ in /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor; do [ -f $CPUFREQ ] || continue; echo -n performance > $CPUFREQ; done

There is a linux service called CPUSpeed, this can tune your scaling governor back to ondemand after the reboot. You may shut it down:

service cpuspeed stop
chkconfig off cpuspeed

You may check your CPU speed is restored to the original through the proc filesystem:

cat /proc/cpuinfo

Can you run fsck on a disk image or raw file?

Question: Can you run fsck on a disk image or raw file?

Answer: I believe you are referring to KVM disk image/raw files. It could be the disk image created from a dd clone. The answer yes, you can run fsck on a disk image. You can also use other fsck attribute as required as following:

fsck.ext4 -fyC /somefile.raw

Remember to use specific partition fsck binary, here, we have used ext4 binary for a ext4 file system loaded on the image file. Running badblock program on an image file can be bad. You may use -c (small c) to run badblock check with the fsck:

fsck.ext4 -C -fyc /somefile.raw

This sometimes create undesired results and not advisable.

Question: What other option do I have?

Answer: You can mount the image using loop back device and then run the file system check. This can be done using kpartx as described in this post: How to: Mount raw VM disk images (KVM/Xen/VMW)

Identifying File / Inode by Sector / Block Number in Linux

I had an interesting problem earlier today. While running r1soft backup, dmesg was throwing some I/O like the following:

Dec 28 09:28:43 ssd1 kernel: [36701.752626] end_request: I/O error, dev vda, sector 331781632
Dec 28 09:28:43 ssd1 kernel: [36701.755400] end_request: I/O error, dev vda, sector 331781760
Dec 28 09:28:43 ssd1 kernel: [36701.758286] end_request: I/O error, dev vda, sector 331781888
Dec 28 09:28:43 ssd1 kernel: [36701.760992] end_request: I/O error, dev vda, sector 331780864

They didn’t go out after multiple file system checks. That left me no choice other than finding what’s actually in that sector. I could see the sector numbers was increasing by 128 up 10 sequential logs. That makes to understand it could be a specific account causing the errors.

EXT file system comes with an interesting tool called debugfs. This can be used on mounted file system and can be used to track down IO related issues. Although, you require to do some calculation first to convert sector to block number of a specific partition before you can use debugfs.

The lowest sector number in the log was ‘331780864’. First I tracked down the partition where this sector lies. This can be done using fdisk -lu /dev/disk (Make sure to use -u, to ensure fdisk returns the sector numbers instead of cylinder number)

#fdisk -lu /dev/vda

Disk /dev/vda: 1342.2 GB, 1342177280000 bytes
16 heads, 63 sectors/track, 2600634 cylinders, total 2621440000 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0002f013

Device Boot Start End Blocks Id System
/dev/vda1 * 2048 411647 204800 83 Linux
/dev/vda2 411648 205211647 102400000 83 Linux
/dev/vda3 205211648 209307647 2048000 82 Linux swap / Solaris
/dev/vda4 209307648 2621439999 1206066176 83 Linux

Now, find the Start Number < Our Sector number to detect which block contains our desired sector. In our case, it is /dev/vda4. Once done, we need to numeric sector number specifically for this partition, which can be done by subtracting our sector number with start number of the partition. In our case:

331780864 – 209307648 = 122473216

That means, our sector lies in 122473216th sector of /dev/vda4.

Now find the block size by tune2fs:

# tune2fs -l /dev/vda4 | grep Block
Block count: 301516544
Block size: 4096
Blocks per group: 32768

In our case, it is 4096.

Now determine the size of the sectors by bytes. This is shown in fdisk output:

Sector size (logical/physical): 512 bytes / 512 bytes

From the two relations block/bytes and sector/bytes, find sector/block : 512 / 4096 = .125

Now, calculate the block number of 122473216th sector: 122473216 x .125 = 15309152

We can now use debugfs to determine what file we have on that block number as following:

debugfs /dev/vda4

On the debug prompt, type:

debugfs: icheck 15309152

Block   Inode number
15309152   2611435

This will show the inode number of the desired file. Use the inode number to run:

debugfs: ncheck 15309152

Inode   Pathname
15309152 /lost+found/#29938847

This will show you the desired file that is actually causing the issue. In my case, I could find files that were corrupted in some old fsck, were stored in lost+found and they were missing magic number/incomplete files. Once I had deleted all the files from lost+found, my issue was resolved. Viola!

Backing up LVM Cache Volume?

I have been trying to explore what options do we have to use SSD Cache with a HDD driven servers to create faster writes. There are both software and hardware solutions. Hardware solution comes to CacheCade which isn’t really costly at all (roughly costs 250$ extra per license), though I was interested to explore all the software solution that are currently available in the market.

There are bcache, flashcache & lvm cache, that are mostly used in production servers. I firstly discarded bcache because it requires you to format the disk with bcache, that triggers the less flexibility check for a module. I tried flashcache before, and don’t want to go with it in a production server any longer as the module is discontinued (It still works, don’t get me wrong). All it seems, lvm cache is the only one which is stable and going to improve over days.

LVM Cache does work great. With the smq lvm cache policy, writeback cachemode & deadline scheduler, you can reach 220MBps write speed with Intel SSD in RAID 1, which is normally available in a RAID 1 Intel SSD. You can double the speed by putting a RAID 10 SSD array to back the cache. Although, after all the test was done, I realized that lvm cache doesn’t support snapshot unfortunately. At least not yet, at the time I am writing the blog. Without the snapshot facility, the performance benchmark actually goes in vain.

Snapshots of cache type volume vg0/newvz is not supported.

Hardware solutions are always useful as the backend setup goes transparent to the OS, which allows us to use our own tool without worrying about the caching setup. Cachecade is probably the only available solution right now with all facilities for SSD cache in production servers.

How to create /var/cpanel/userdata files using /var/cpanel/users files

While reverting back from a reverse proxy like nginx/varnish installation, you might end up having an inconsistent /var/cpanel/userdata and /var/cpanel/users files. To be noted, /var/cpanel/userdata is used to create the httpd.conf file, while /var/cpanel/users are used to create the dns zone files. If you have an inconsistency between this two, you will have two different IP for named and httpd, which is undesirable. As the reverse proxy plays with /var/cpanel/userdata, which is why, we usually see the userdata folder containing incorrect data. Cpanel comes with a feature to reset userdata directory through it’s tools. Although, the tool uses a valid httpd.conf file or a backup to create userdata directory. Interestingly, if your httpd.conf isn’t valid that was created using an inconsistent userdata, can’t be used backward.

This is when you might require to create a /var/cpanel/userdata directory using the valid /var/cpanel/users directory. We ended up in this situation and written the following bash code to do the job for us:

#!/bin/sh

ls /var/cpanel/users/* > users.txt

while read line; do

UIP=$(grep “IP” /var/cpanel/users/$line|cut -d’=’ -f2)
sed -ie “/^ip: / c ip: $UIP” /var/cpanel/userdata/$line/*

#sed -i ‘/ip:/s/.*/ip: `echo $UIP`/’ /var/cpanel/userdata/$line/*

rm -f /var/cpanel/userdata/$line/*.cache

done < users.txt

 

The code deletes the cache files as well. Cpanel ships a script to rebuild the cache files, but for some reason it didn’t work for us. So we manually deleted the cache using rm. Please remember to backup the /var/cpanel/userdata and /var/cpanel/users directory before running the scripts.

Cpanel given cache rebuild command is:

/scripts/updateuserdatacache

Once the cache removal/build is completed, you may now rebuild your httpd conf file:

/scripts/rebuildhttpd.conf

service httpd restart

Backend log: PHP Warning: File upload error – unable to create a temporary file in Unknown on line 0

It is a common php error appears when PHP handler unable to detect the server temporary folder automatically. A solution to the problem is to define the upload_tmp_dir of the php explicitly. You can do that by editing the php.ini file. Here is an example on how to do it:

To find the running php.ini file, use phpinfo:

root@chicago1 [~]# php -r ‘phpinfo();’|grep php.ini
Configuration File (php.ini) Path => /opt/cpanel/ea-php56/root/etc
Loaded Configuration File => /opt/cpanel/ea-php56/root/etc/php.ini

As the command suggests, your php.ini file is located under the following:

/opt/cpanel/ea-php56/root/etc/php.ini

Now, edit the file using your favorite editor:
nano /opt/cpanel/ea-php56/root/etc/php.ini
Find the location of upload_tmp_dir, which is usually commented out like the following:

;upload_tmp_dir =

Uncomment it by removing the semicolon from front and define the temporary directory location:

upload_tmp_dir = /tmp

Restart your apache.

There is another way you can do it. To do this, go to WHM >> MultiPHP INI Editor >> Editor Mode >> Find upload_tmp_dir and edit it accordingly.

This would create a user.ini file to be included with the original php.ini file.

If none of the above solution works for you, you are probably seeing the error from modsecurity, try following the following steps to see if that overcomes the error:

ModSecurity: Multipart parsing error: Multipart: Failed to create file: /root/tmp/20170526-122120-WSfJYO2KhTvEz5johZF8UQAAAEw-file-9mmG15

How to change WHM reseller password!

After all these years, it never came to my mind that when somebody purchases a reseller, they usually do not change their WHM password for a long period. They keep it ‘as it is’ generated by WHMCS on purchasing the reseller package. The most interesting fact is that they don’t change it, because they fail to find an option to change it in WHM.

WHM doesn’t come with a distinct option saying ‘Change WHM Password’ unfortunately. That makes a percentage of reseller believe that they can not change their WHM password. In recent times, while investigating a couple of reseller hacks, I could determine, one of the primary reason of password leakage is, not changing the WHM password for longer period of time and keeping it ‘saved’ in browser. At a certain point of time, when the browser gets exposed to the hacker, user loose control over their WHM account.

Now the question comes, how to change a WHM password! Your WHM username is basically a cpanel username. It only granted to be able to own multiple cpanel accounts and that is the only difference, that’s all. To change the WHM password, simply login to your cpanel with the WHM details and use the ‘Change Password’ option. So if your WHM url is http://something.com/whm with username: something and password: anything, then you basically login with the same details in http://something.com/cpanel instead of whm. Once logged in, just visit the Change Password to change your WHM/Reseller password.

It is highly recommended for all the users to change the password once they receive their reseller welcome email. You should try changing the reseller password often to prevent any anonymous leakage from unknown attacks. It is also advised not to save the WHM password in your browser. Please keep in mind, your password can leak access to the cpanel accounts under you and cause great threat for their websites & domain reputation. They possibly have no reason to be so.