[957833.048873] EXT4-fs warning (device dm-0): ext4_dx_add_entry:2016: Directory index full!

A dedicated server customer came with an error of failed WHM login. When I tried to enter, I received the following 500 error response:

Internal Server Error

500

Cpanel::Exception::IO::FileCreateError/(XID 975n2e) The system failed to create the file “/var/cpanel/sessions/raw/:INPl_XWqgs8SOIL6.lock” (as [asis,EUID]: 0, [asis,EGID]: 0) because of the following error: No space left on device
at /usr/local/cpanel/Cpanel/Exception/CORE.pm line 336, <GEN10> line 2.
Cpanel::Exception::create("IO::FileCreateError", ARRAY(0x36f9a20)) called at /usr/local/cpanel/Cpanel/Exception.pm line 46
Cpanel::Exception::__ANON__(__CPANEL_HIDDEN__, ARRAY(0x36f9a20)) called at /usr/local/cpanel/Cpanel/SafeFile.pm line 363
Cpanel::SafeFile::_safelock("/var/cpanel/sessions/raw/:INPl_XWqgs8SOIL6") called at /usr/local/cpanel/Cpanel/SafeFile.pm line 538
Cpanel::SafeFile::_safe_open(undef, 194, "/var/cpanel/sessions/raw/:INPl_XWqgs8SOIL6", CODE(0x36f98d0), "safesysopen") called at /usr/local/cpanel/Cpanel/SafeFile.pm line 196
Cpanel::SafeFile::safesysopen(undef, "/var/cpanel/sessions/raw/:INPl_XWqgs8SOIL6", 194, 384) called at /usr/local/cpanel/Cpanel/Session.pm line 279
Cpanel::Session::saveSession(":INPl_XWqgs8SOIL6,ce201c7f417e25dee8c76ee8467ff688", HASH(0x36f7fb8), "initial", 1) called at /usr/local/cpanel/Cpanel/Session.pm line 148
Cpanel::Session::create(Cpanel::Session=HASH(0x36f4ab0), "user", undef, "session", HASH(0x144b8b0)) called at /usr/local/cpanel/Cpanel/Server.pm line 1357
Cpanel::Server::newsession(Cpanel::Server=HASH(0x36f4180), "needs_auth", 1, "origin", HASH(0x36f7c90)) called at /usr/local/cpanel/Cpanel/Server.pm line 791
Cpanel::Server::badpass(Cpanel::Server=HASH(0x36f4180), __CPANEL_HIDDEN__, 1) called at cpsrvd.pl line 4784
cpanel::cpsrvd::handle_auth() called at cpsrvd.pl line 1069
cpanel::cpsrvd::handle_one_connection() called at cpsrvd.pl line 806
cpanel::cpsrvd::script() called at cpsrvd.pl line 324


It says the device is out of disk space, but the disk usage was below full quota:

[[email protected] tmp]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg0_481_112717-root 231G 72G 147G 33% /
devtmpfs 24G 0 24G 0% /dev
tmpfs 24G 4.0K 24G 1% /dev/shm
tmpfs 24G 450M 24G 2% /run
tmpfs 24G 0 24G 0% /sys/fs/cgroup
/dev/sda1 922M 156M 703M 19% /boot
tmpfs 4.8G 0 4.8G 0% /run/user/0

After I tried to check the dmesg/syslog, I could see the following error message:

[957833.048873] EXT4-fs warning (device dm-0): ext4_dx_add_entry:2016: Directory index full!

It basically says the directory indexing is out of space. Checking the inode usage, says the root partition is out of inode limit:

[[email protected] tmp]# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/vg0_481_112717-root 15327232 15327232 0 100% /
devtmpfs 6157820 516 6157304 1% /dev
tmpfs 6160532 2 6160530 1% /dev/shm
tmpfs 6160532 770 6159762 1% /run
tmpfs 6160532 16 6160516 1% /sys/fs/cgroup
/dev/sda1 61056 347 60709 1% /boot
tmpfs 6160532 1 6160531 1% /run/user/0

Once you determine which partition your inode limit has overflown, you then need to run the find command to track down your inode usage. You may something like the following:

for dir in `ls -1`; do echo $dir; find ./$dir -type f|wc -l; done



Each time, you see a large number of indoes in a folder, hop into it until you determine the actual folder that contains most of the inodes. I ended up determing these files were in the session folder under client's public_html folder. Client was storing the php sessions under his own folder instead of using /tmp directory. 

To delete a large number of files using rm, you may end up seeing the following error:

[[email protected] sessions]# rm -f ci_session*
-bash: /usr/bin/rm: Argument list too long

To delete them, you would again need to use piping. This time, you may take help from xargs:

[[email protected] sessions]# find . -name "*"|xargs rm -f

  • filesystem, inode, index, centos, linux
  • 0 brukere syntes dette svaret var til hjelp
Var dette svaret til hjelp?

Relaterte artikler

R1soft Backup Error - Failed To Backup LVM/MD Configuration

It looks like R1Soft manages it's own lvm.static file to process lvm backups. But sometimes it...

Does R1Soft work with LVM?

It does. But it is not fully supported as of 28th November, 2017. If you have a solid logical...

Error getting authority: Error initializing authority: Could not connect: Connection refused (g-io-error-quark, 39)

CentOS 7 / CloudLinux 7 / RHEL 7 may show the following error while restarting Apache/Exim or...

lfd on ***.***.***: ModSecurity persistent IP database size alert

LFD was throwing the following error for sometime in a server: ModSecurity persistent IP...

Fatal error: Uncaught exception 'Google_IO_Exception' with message 'Couldn't resolve host 'accounts.google.com''

Sometimes, Google library can return the following error: Fatal error: Uncaught exception...