Internal Server Error
500
Cpanel::Exception::IO::FileCreateError/(XID 975n2e) The system failed to create the file “/var/cpanel/sessions/raw/:INPl_XWqgs8SOIL6.lock” (as [asis,EUID]: 0, [asis,EGID]: 0) because of the following error: No space left on device
at /usr/local/cpanel/Cpanel/Exception/CORE.pm line 336, <GEN10> line 2.
Cpanel::Exception::create("IO::FileCreateError", ARRAY(0x36f9a20)) called at /usr/local/cpanel/Cpanel/Exception.pm line 46
Cpanel::Exception::__ANON__(__CPANEL_HIDDEN__, ARRAY(0x36f9a20)) called at /usr/local/cpanel/Cpanel/SafeFile.pm line 363
Cpanel::SafeFile::_safelock("/var/cpanel/sessions/raw/:INPl_XWqgs8SOIL6") called at /usr/local/cpanel/Cpanel/SafeFile.pm line 538
Cpanel::SafeFile::_safe_open(undef, 194, "/var/cpanel/sessions/raw/:INPl_XWqgs8SOIL6", CODE(0x36f98d0), "safesysopen") called at /usr/local/cpanel/Cpanel/SafeFile.pm line 196
Cpanel::SafeFile::safesysopen(undef, "/var/cpanel/sessions/raw/:INPl_XWqgs8SOIL6", 194, 384) called at /usr/local/cpanel/Cpanel/Session.pm line 279
Cpanel::Session::saveSession(":INPl_XWqgs8SOIL6,ce201c7f417e25dee8c76ee8467ff688", HASH(0x36f7fb8), "initial", 1) called at /usr/local/cpanel/Cpanel/Session.pm line 148
Cpanel::Session::create(Cpanel::Session=HASH(0x36f4ab0), "user", undef, "session", HASH(0x144b8b0)) called at /usr/local/cpanel/Cpanel/Server.pm line 1357
Cpanel::Server::newsession(Cpanel::Server=HASH(0x36f4180), "needs_auth", 1, "origin", HASH(0x36f7c90)) called at /usr/local/cpanel/Cpanel/Server.pm line 791
Cpanel::Server::badpass(Cpanel::Server=HASH(0x36f4180), __CPANEL_HIDDEN__, 1) called at cpsrvd.pl line 4784
cpanel::cpsrvd::handle_auth() called at cpsrvd.pl line 1069
cpanel::cpsrvd::handle_one_connection() called at cpsrvd.pl line 806
cpanel::cpsrvd::script() called at cpsrvd.pl line 324
It says the device is out of disk space, but the disk usage was below full quota:
[[email protected] tmp]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg0_481_112717-root 231G 72G 147G 33% /
devtmpfs 24G 0 24G 0% /dev
tmpfs 24G 4.0K 24G 1% /dev/shm
tmpfs 24G 450M 24G 2% /run
tmpfs 24G 0 24G 0% /sys/fs/cgroup
/dev/sda1 922M 156M 703M 19% /boot
tmpfs 4.8G 0 4.8G 0% /run/user/0
After I tried to check the dmesg/syslog, I could see the following error message:
[957833.048873] EXT4-fs warning (device dm-0): ext4_dx_add_entry:2016: Directory index full!
It basically says the directory indexing is out of space. Checking the inode usage, says the root partition is out of inode limit:
[[email protected] tmp]# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/vg0_481_112717-root 15327232 15327232 0 100% /
devtmpfs 6157820 516 6157304 1% /dev
tmpfs 6160532 2 6160530 1% /dev/shm
tmpfs 6160532 770 6159762 1% /run
tmpfs 6160532 16 6160516 1% /sys/fs/cgroup
/dev/sda1 61056 347 60709 1% /boot
tmpfs 6160532 1 6160531 1% /run/user/0
Once you determine which partition your inode limit has overflown, you then need to run the find command to track down your inode usage. You may something like the following:
for dir in `ls -1`; do echo $dir; find ./$dir -type f|wc -l; done
Each time, you see a large number of indoes in a folder, hop into it until you determine the actual folder that contains most of the inodes. I ended up determing these files were in the session folder under client's public_html folder. Client was storing the php sessions under his own folder instead of using /tmp directory.
To delete a large number of files using rm, you may end up seeing the following error:
[[email protected] sessions]# rm -f ci_session*
-bash: /usr/bin/rm: Argument list too long
To delete them, you would again need to use piping. This time, you may take help from xargs:
[[email protected] sessions]# find . -name "*"|xargs rm -f