Category: Linux

Identifying File / Inode by Sector / Block Number in Linux

I had an interesting problem earlier today. While running r1soft backup, dmesg was throwing some I/O like the following:

Dec 28 09:28:43 ssd1 kernel: [36701.752626] end_request: I/O error, dev vda, sector 331781632
Dec 28 09:28:43 ssd1 kernel: [36701.755400] end_request: I/O error, dev vda, sector 331781760
Dec 28 09:28:43 ssd1 kernel: [36701.758286] end_request: I/O error, dev vda, sector 331781888
Dec 28 09:28:43 ssd1 kernel: [36701.760992] end_request: I/O error, dev vda, sector 331780864

They didn’t go out after multiple file system checks. That left me no choice other than finding what’s actually in that sector. I could see the sector numbers was increasing by 128 up 10 sequential logs. That makes to understand it could be a specific account causing the errors.

EXT file system comes with an interesting tool called debugfs. This can be used on mounted file system and can be used to track down IO related issues. Although, you require to do some calculation first to convert sector to block number of a specific partition before you can use debugfs.

The lowest sector number in the log was ‘331780864’. First I tracked down the partition where this sector lies. This can be done using fdisk -lu /dev/disk (Make sure to use -u, to ensure fdisk returns the sector numbers instead of cylinder number)

#fdisk -lu /dev/vda

Disk /dev/vda: 1342.2 GB, 1342177280000 bytes
16 heads, 63 sectors/track, 2600634 cylinders, total 2621440000 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0002f013

Device Boot Start End Blocks Id System
/dev/vda1 * 2048 411647 204800 83 Linux
/dev/vda2 411648 205211647 102400000 83 Linux
/dev/vda3 205211648 209307647 2048000 82 Linux swap / Solaris
/dev/vda4 209307648 2621439999 1206066176 83 Linux

Now, find the Start Number < Our Sector number to detect which block contains our desired sector. In our case, it is /dev/vda4. Once done, we need to numeric sector number specifically for this partition, which can be done by subtracting our sector number with start number of the partition. In our case:

331780864 – 209307648 = 122473216

That means, our sector lies in 122473216th sector of /dev/vda4.

Now find the block size by tune2fs:

# tune2fs -l /dev/vda4 | grep Block
Block count: 301516544
Block size: 4096
Blocks per group: 32768

In our case, it is 4096.

Now determine the size of the sectors by bytes. This is shown in fdisk output:

Sector size (logical/physical): 512 bytes / 512 bytes

From the two relations block/bytes and sector/bytes, find sector/block : 512 / 4096 = .125

Now, calculate the block number of 122473216th sector: 122473216 x .125 = 15309152

We can now use debugfs to determine what file we have on that block number as following:

debugfs /dev/vda4

On the debug prompt, type:

debugfs: icheck 15309152

Block   Inode number
15309152   2611435

This will show the inode number of the desired file. Use the inode number to run:

debugfs: ncheck 15309152

Inode   Pathname
15309152 /lost+found/#29938847

This will show you the desired file that is actually causing the issue. In my case, I could find files that were corrupted in some old fsck, were stored in lost+found and they were missing magic number/incomplete files. Once I had deleted all the files from lost+found, my issue was resolved. Viola!

[Tue Dec 19 20:40:07.097202 2017] [lsapi:error] [pid 532140:tid 139848266454784] [client IP.IP.IP.IP:25021] mod_lsapi: [host domain.com] [req GET / HTTP/1.1] Could not connect to lsphp backend: connect to lsphp refused: 111 (possibly memory limit for LVE ID 1789 too small), referer: domain.com

If you are running Cloudlinux, cagefs and lsapi with cpanel, you are probably familiar with the error. If the error is appearing for one or two sites, then it is probably because the user is hitting the VM/PM limit you have set through Cloudlinux. But if the error is appearing for all the sites, then it is because the cagefs fails with the suexec permission for some reason.

[Tue Dec 19 20:40:07.097202 2017] [lsapi:error] [pid 532140:tid 139848266454784] [client *:25021] mod_lsapi: [host *] [req GET / HTTP/1.1] Could not connect to lsphp backend: connect to lsphp refused: 111 (possibly memory limit for LVE ID 1789 too small), referer: http://*/

One way to solve the problem is to remount all the users. Sometimes, it doesn’t work and you may require to reinitialize cagefs again:

cagefsctl –remount-all
cagefsctl -r

I have seen times, when nothing works, but reinstalling cagefs does the trick. If cagefs doesn’t work, you may try disabling virtual memory from the CloudLinux LVE manager to see if that fix the problem. CloudLinux also has a known Virtual Memory 503 error issue with LSAPI.

error: Unable to create cgroup for vm**: No such file or directory

The error can appear for any type of KVM VM installation like Virtualizor  or SolusVM or Proxmox. You may face the same error if you are simply using virtmanager to manage your KVM installation. The error appear when you try to start/create the VM from the xml file:

[root@vps8 addvs]# virsh create /etc/libvirt/qemu/v1015.xml
error: Failed to create domain from /etc/libvirt/qemu/v1015.xml
error: Unable to create cgroup for v1015: No such file or directory

It appears because for some CentOS starts unmounting the cgroup and breaks libvirt. Easy way to fix this is to restart libvirtd:

service libvirtd restart

The error is more common in CentOS 7 than CentOS 6, as systemd is known to have the bug:

https://bugzilla.redhat.com/show_bug.cgi?id=678555

device-mapper: cache: You have created a cache device with a lot of individual cache blocks

The error would be similar to the following to be exact:

[172593.817178] device-mapper: cache: You have created a cache device with a lot of individual cache blocks (3276800)
[172593.817182] All these mappings can consume a lot of kernel memory, and take some time to read/write.
[172593.817185] Please consider increasing the cache block size to reduce the overall cache block count.

It usually appears because you have created a large cache pool, while using a small chunk size.

Here is what to be said about chunk size in lvmcache2 manual:

The size of data blocks managed by a cache pool can be specified with
the –chunksize option when the cache LV is created. The default
unit is KiB. The value must be a multiple of 32KiB between 32KiB and
1GiB.

Using a chunk size that is too large can result in wasteful use of
the cache, where small reads and writes can cause large sections of
an LV to be mapped into the cache. However, choosing a chunk size
that is too small can result in more overhead trying to manage the
numerous chunks that become mapped into the cache. Overhead can
include both excessive CPU time searching for chunks, and excessive
memory tracking chunks.

Basically chunk size is determining the block size at which your caches are going to be stored. To find your current chunk size run:

lvs -o+chunksize vgname/cachevolumename

Should return something like the following:

#lvs -o+chunksize vg0/newvz
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert Chunk
newvz vg0 Cwi-aoC— 1.50t [cache] 14.14 0.75 5.12 64.00k

A good chunk size for a large cache device of size 200GB+ would be 256k. You may set the chunk size while setting up the device :

lvcreate –type cache-pool –chunksize 256K -L CacheSize -n CachePoolLV VG FastPVs

Backing up LVM Cache Volume?

I have been trying to explore what options do we have to use SSD Cache with a HDD driven servers to create faster writes. There are both software and hardware solutions. Hardware solution comes to CacheCade which isn’t really costly at all (roughly costs 250$ extra per license), though I was interested to explore all the software solution that are currently available in the market.

There are bcache, flashcache & lvm cache, that are mostly used in production servers. I firstly discarded bcache because it requires you to format the disk with bcache, that triggers the less flexibility check for a module. I tried flashcache before, and don’t want to go with it in a production server any longer as the module is discontinued (It still works, don’t get me wrong). All it seems, lvm cache is the only one which is stable and going to improve over days.

LVM Cache does work great. With the smq lvm cache policy, writeback cachemode & deadline scheduler, you can reach 220MBps write speed with Intel SSD in RAID 1, which is normally available in a RAID 1 Intel SSD. You can double the speed by putting a RAID 10 SSD array to back the cache. Although, after all the test was done, I realized that lvm cache doesn’t support snapshot unfortunately. At least not yet, at the time I am writing the blog. Without the snapshot facility, the performance benchmark actually goes in vain.

Snapshots of cache type volume vg0/newvz is not supported.

Hardware solutions are always useful as the backend setup goes transparent to the OS, which allows us to use our own tool without worrying about the caching setup. Cachecade is probably the only available solution right now with all facilities for SSD cache in production servers.

Check File System for Errors with Status/Progress Bar

File system check can be tedious sometimes. User may want to check the progress of the fsck, which is not enabled by default. To do that, add -C (capital C) with the fsck command.

fsck -C /dev/sda1

The original argument is:

fsck -C0 /dev/sda1

Although, it would work without number if you put the -C in front of other arguments, like -f (forcing the file system check) -y (yes to auto repair). A usable fsck command could be the following:

fsck -fy -C0 /dev/sda1

or

fsck -C -fy /dev/sda1

Please note, -c (small C) would result a read only test. This test will try to read all the blocks in the disk and see if it is able to read them or not. It is done through a program called ‘badblock’. If you are running badblock test on a large system, be ready to spend a large amount of time for that.

How to create /var/cpanel/userdata files using /var/cpanel/users files

While reverting back from a reverse proxy like nginx/varnish installation, you might end up having an inconsistent /var/cpanel/userdata and /var/cpanel/users files. To be noted, /var/cpanel/userdata is used to create the httpd.conf file, while /var/cpanel/users are used to create the dns zone files. If you have an inconsistency between this two, you will have two different IP for named and httpd, which is undesirable. As the reverse proxy plays with /var/cpanel/userdata, which is why, we usually see the userdata folder containing incorrect data. Cpanel comes with a feature to reset userdata directory through it’s tools. Although, the tool uses a valid httpd.conf file or a backup to create userdata directory. Interestingly, if your httpd.conf isn’t valid that was created using an inconsistent userdata, can’t be used backward.

This is when you might require to create a /var/cpanel/userdata directory using the valid /var/cpanel/users directory. We ended up in this situation and written the following bash code to do the job for us:

#!/bin/sh

ls /var/cpanel/users/* > users.txt

while read line; do

UIP=$(grep “IP” /var/cpanel/users/$line|cut -d’=’ -f2)
sed -ie “/^ip: / c ip: $UIP” /var/cpanel/userdata/$line/*

#sed -i ‘/ip:/s/.*/ip: `echo $UIP`/’ /var/cpanel/userdata/$line/*

rm -f /var/cpanel/userdata/$line/*.cache

done < users.txt

 

The code deletes the cache files as well. Cpanel ships a script to rebuild the cache files, but for some reason it didn’t work for us. So we manually deleted the cache using rm. Please remember to backup the /var/cpanel/userdata and /var/cpanel/users directory before running the scripts.

Cpanel given cache rebuild command is:

/scripts/updateuserdatacache

Once the cache removal/build is completed, you may now rebuild your httpd conf file:

/scripts/rebuildhttpd.conf

service httpd restart

Backend log: PHP Warning: File upload error – unable to create a temporary file in Unknown on line 0

It is a common php error appears when PHP handler unable to detect the server temporary folder automatically. A solution to the problem is to define the upload_tmp_dir of the php explicitly. You can do that by editing the php.ini file. Here is an example on how to do it:

To find the running php.ini file, use phpinfo:

root@chicago1 [~]# php -r ‘phpinfo();’|grep php.ini
Configuration File (php.ini) Path => /opt/cpanel/ea-php56/root/etc
Loaded Configuration File => /opt/cpanel/ea-php56/root/etc/php.ini

As the command suggests, your php.ini file is located under the following:

/opt/cpanel/ea-php56/root/etc/php.ini

Now, edit the file using your favorite editor:
nano /opt/cpanel/ea-php56/root/etc/php.ini
Find the location of upload_tmp_dir, which is usually commented out like the following:

;upload_tmp_dir =

Uncomment it by removing the semicolon from front and define the temporary directory location:

upload_tmp_dir = /tmp

Restart your apache.

There is another way you can do it. To do this, go to WHM >> MultiPHP INI Editor >> Editor Mode >> Find upload_tmp_dir and edit it accordingly.

This would create a user.ini file to be included with the original php.ini file.

If none of the above solution works for you, you are probably seeing the error from modsecurity, try following the following steps to see if that overcomes the error:

ModSecurity: Multipart parsing error: Multipart: Failed to create file: /root/tmp/20170526-122120-WSfJYO2KhTvEz5johZF8UQAAAEw-file-9mmG15

ModSecurity: Multipart parsing error: Multipart: Failed to create file: /root/tmp/20170526-122120-WSfJYO2KhTvEz5johZF8UQAAAEw-file-9mmG15

If you are using ‘inspectFile’ of modsecurity, you may face an error in cpanel server like following

ModSecurity: Multipart parsing error: Multipart: Failed to create file: /root/tmp/20170526-122120-WSfJYO2KhTvEz5johZF8UQAAAEw-file-9mmG15

This error appears because mod_security fails to detect the tmp folder automatically. To fix the problem, all you need to add the following line in your modsec_user.conf file for cpanel servers:

SecTmpDir /tmp

This file is available under

/etc/apache2/conf.d/modsec

Alternatively, you may add the line from Cpanel >> Modsecurity Tools >> Add Rules.

Why does Your New Site Take Ages to Load?

I was trying to track down a couple of website slow down reports lately. There is an interesting change of slow down behaviour these days in web application. From a conventional standpoint, people firmly believes that their static contents are not going to affect the performance of their websites other than images being heavy.
 
In reality, they are ignoring the fact that they are using jQuery plugins of many kinds from multiple developers. Hence cumulative number & sizes of JS files are pretty large these days comparing with all the plugins were coming from a single developer. Once the number of static file increases and goes beyond 100 per page, a cookie domain can hit some serious performance penalty. Geolocation for these small files and accessing them from single source can also increase the time geometrically. There is undoubtedly a large market of CDN due to the nature of development in web application.
 
I have seen, people these days are more aware about handling large data wisely than before. If you are using a Cloud from any provider, you are possibly using an E5 core or multiple (Mellowhost uses only E5 nodes right at this moment), that usually comes with access to a 16/24/32MB cache. Your static handling going to be more important in performance on these type of resources than your database, as threading is more of a concern than a single process handling in these virtualised resources.