mdadm is the software raid tools used in Linux system. One key problem with the software raid, is that it resync is utterly slow comparing with the existing drive speed (SSD or NVMe). The resync speed set by mdadm is default for regardless of whatever the drive type you have. To view the default values, you may run the following:
As you see, the minimum value starts from 1000 and can max upto 200K. Although, it can max upto 200K, but as min value is too low, mdadm always tries to keep the value below average to your speed available. To speed up, we would like to maximize these numbers. To change the numbers, you may run something like the following:
One thing to keep in mind is that, if you try to set the value too high, like we did, this might cause some handsome load on your system. If you see the load is unmanageable, you should focus on decreasing the number to something like 50k-100k for the min value.
Making The Sysctl Value Permanent
As we have established the kernel variable values on runtime, this would go back to default once we restart/reboot the server. If you want to persist the values, you need to put these values to /etc/sysctl.conf file. To make them persist, open the sysctl.conf file:
After you have completed updating your yum, you saw the kernel got updated, and hence restarted the server to take the new kernel. But you find out that the server has never come online. Once you visit the KVM or Serial Console (SOL) of the system, you could see, your system is booted to ‘grub>’ console instead of booting from disk. How can you fix the system now?
This specific issue can appear for any linux server, along with many reasons. Although, if you are running an server from OVH and had faced a similar issue, the boat I am going to show you can navigate to destination. Please note, in many other case of similar situation, you might end up fixing the grub with the same solution.
What and How the Problem Happened
OVH has an interesting strategy of booting. They follow everything through network PXE, even if it is not ‘netboot’, but just the local drives. For this to work out, you need PXE to take the latest grub details pushed once a kernel is updated. This is one reason why, OVH also supplies a custom kernel from a cusstom repo. Although, if you are using the stock kernel, you might come up with a situation, where the latest grub hasn’t been pushed to PXE and your system fails to boot from drives. It then puts you in the ‘grub’ of network.
How to Fix the Problem
Now, one thing is clear, after you completed a kernel update, your grub is broken due to the latest machine code is not available to the booting system. You can go and follow a regular grub repair method for Grub 2, to fix the situation. A couple of things to remember, as your system’s grub is failing to load, you have to use an independent rescue kernel to fix this, this could either be from a personal network repository or a rescue disk available from your datacenter’s location, like ovh has one. Another thing to remember, is that, if you are using CentOS 7 or Ubuntu with UEFI system, using mdadm or linux software raid, it is highly likely, your boot efi is placed in a non raid partition. Preferably in the first drive’s first partition. You can always verify this from your fstab file.
So the first job, is to boot your system into the rescue disk/cd/kernel. I assume you have done that with no difficulty. Once done, first mount your partitions. In OVH cases, it loads the mdadm automatically. In my case, it was /dev/md2.
mount /dev/md2 /mnt
# check what partition is used for /boot/efi
# in my case, it is /dev/nvme0n1p1 (It is a NVMe SSD, and the first partion is used for efi storage
mount /dev/nvme0n1p1 /mnt/boot/efi
Once we have mounted the partitions successfully, you may now chroot the system. Before chrooting, you want the dev, proc and sys to use the /mnt partitions respectively:
mount --bind /dev /mnt/dev
mount --bind /proc /mnt/proc
mount --bind /sys /mnt/sys
If these all goes well, now we can chroot the system:
Now you have successfully changed the root directory of the rescue kernel to the original drive’s root. All you need to do, is to remake the grub config, that will immediately generate the grub.cfg file and sync the machine code:
# we know grub.cfg is available in /boot/grub2/grub.cfg
grub2-mkconfig -o /boot/grub2/grub.cfg
# once this is finished, we have to make sure, grub is also installed for both disks, for my case, these are /dev/nvme0n1 and /dev/nvme1n1
If you see the response is ‘No Error Reported’, then you are good go. You may now reboot your system back to hard disk, and can see your grub is able to load the latest kernel you installed from the original hard disk. Remember, for safety, you should umount all the partition, to avoid any data loss due to OS page cache:
It is trying to look into a sub-sub version of openssl, which is 1h for PHP 7.4, but 1i is installed by default which is not an exact match with 1h, hence the error is appearing. To fix the problem, you can not just run yum update on openssl and fix this as you technically already have the latest version. To work with this, you need to first uninstall the ea-openssl rpm and install the require version. You may do this in the following way:
[[email protected] yum.repos.d]# rpm -qa|grep ea-openssl
[[email protected] yum.repos.d]# yum install ea-openssl11-1.1.1h-1.el7.cloudlinux.x86_64
Loaded plugins: fastestmirror, rhnplugin, universal-hooks
This system is receiving updates from CLN.
Loading mirror speeds from cached hostfile
* cpanel-plugins: 184.108.40.206
* cloudlinux-x86_64-server-7: xmlrpc.cln.cloudlinux.com
Package matching 1:ea-openssl11-1.1.1h-1.el7.cloudlinux.x86_64 already installed. Checking for update.
Nothing to do
A PATH variable is a system variable that stores the information about the binary files location that you may run for commands. When you log in as an user, or use a custom control panel like Plesk/Cyberpanel/Cpanel, you might want to add some custom paths as a user to take binary commands. One of the example, could be to change the default php path, or a laravel command location from vendor folder. To do this, you need to extend/update the PATH variable for a specific user.
PATH variable extends with the “:”. If you type the following, in your shell, you may see the existing paths in the PATH variable:
We have successfully modified the PATH variable, but only for the existing session. If you want to persist the changes, then, you need to add the command in .bashrc/.profile/.bash_profile file depending on your shell type and OS. You can add to either of the file and test with the following command:
First, let’s see why do we need to copy a disk over SSH. One is of course for backup. For example if you have a VM on a LVM partition. You want to keep a copy of the block level backups, you prefer to create a snapshot of the lvm partition and then copy the disk as an image to your backup server. The other being quite the same, but for different purpose. What if you want to migrate a VM that you have created on an LVM partition, and then you want to migrate it as a raw file to another server? Or a LVM partition to another server? For those cases, the technique is pretty awesome.
Copy The Disk to a RAW Image
First, let’s learn how can copy a disk as raw image.
For example, you are sitting in your backup server. And you want to copy a disk or lvm partition or a partition of a disk, from a remote server to your backup server. And you want to keep the copy of the image, then you may run dd command as following from your backup server:
In our case, 10.10.10.10 is the IP of the server, that partition/disk currently reside. We are trying to copy a LVM partition namely: /dev/vg0/v1092-kdkdksjuekksq, just replace this one with the one your desired lvm partition. You may also do this for a disk like /dev/sda, the command would be like the following then:
You may either copy the disk directly to a secondary disk you have on the backup/migrated server or dump the image that you copied to another disk/partition of same size or bigger. To copy directly a partition /dev/sda from another server, to a backup server with the secondary disk /dev/sdb, you may do the following:
Resetting admin password in CentOS 7 is different than of CentOS 6, as CentOS 7 utilized Grub 2 and has a different procedure to access Single User Mode. First, boot your system in Single User Mode to reset the root password by following the below tutorial:
Once done, now, you may first chroot the system:
Now, you may reset the password using the following:
You should be done. If you are using SELinux, then you need to relabel accordingly:
Booting CentOS 7 in single user mode is total different than of CentOS 6 as it uses a different version of Grub. We will go step by step to boot into single user mode in CentOS 7.
First, boot the screen and press an arrow key while the timer comes on this screen to stop the timer
Press ‘e’ on the selected default CentOS version. That will take you to the following screen
These are grub commands that the bootloader is using to boot you to the system. Arrow down in this window to the line, that starts with linux16. Now take the right arrow on the keyboard to take your cursor to the word that says ‘ro’ as shown in the image
Now replace ro with rw init=/sysroot/bin/sh as shown below
Now press Ctrl + x to start Single User Mode
You should be done now, you are on the single user mode console. If you would like to use the main system, you may chroot as the following:
Now, if you are trying to reset the root password, after booting into the single user mode, you may follow the following to reset the root password:
In this server, it has one disk, but you may have multiple disk with different ‘Firemware state’ and ‘Device Id’. To use smartmontools, you need to pick the ‘Device Id’, mentioned here, which is 11. Now you can run the following command to get the device details using smartctl:
smartctl -d megaraid,N -a /dev/sdX
Here, N is the device ID, and X is the device name, you may get the device name using df -h command or fdisk -l. For our case, this command would be like the following:
smartctl -d megaraid,11 -a /dev/sda
This would print a lot of information about your device, but if you are looking to identify the Serial Number only, you may run the following:
~ smartctl -d megaraid,11 -a /dev/sda|grep Serial
Serial Number: 50026B72822A7D3A
One thing to note, we can also get Serial number from the MegaCli tools Inquiry data, you may have already noticed: