How to Mount qcow2 KVM/Xen/VMware VM Image

To mount qcow2, you need to mount this image as a ‘Network Block Device’.

First, enable NBD

modprobe nbd max_part=8

Now, connect the qcow2 as an NBD device

qemu-nbd --connect=/dev/nbd0 /vz/vmprivate/v1002/harddisk.hdd

Now to mount the partition, first detect the partition

fdisk /dev/nbd0 -l

Now, you may mount the partition

mount /dev/nbd0p1 /mnt

Once, all the jobs are done, you may unmount, disconnect, and remove the NBD kernel module

umount /mnt
qemu-nbd --disconnect /dev/nbd0
rmmod nbd

KVM VM Not starting – could not get access to acl tech driver ‘ebiptables’

Issue

KVM VM not starting with the following error when you try to start:

could not get access to acl tech driver 'ebiptables'

There is a nwfilter module for libvirt. If for some reason, it comes up with an issue, the above error would appear. To fix this, you need to update (If any update is available) / reinstall (If no update is available) the following module using Yum:

libvirt-daemon-config-nwfilter

The command would be like the following:

yum update libvirt-daemon-config-nwfilter

That shall fix the issue.

Could not make the Query. Virtualizor Error

Today, when I opened a Virtualizor panel for a VM node, I found an issue like the following:

Could not make the Query.
SELECT tasks.actid, tasks.action, users.uid, users.email FROM `tasks` LEFT JOIN users on tasks.uid = users.uid WHERE action NOT IN ('vpsbackups_plan_email', 'send_background_mail', 'backuply_vpsbackups_plan_email') ORDER BY tasks.actid DESC LIMIT 10
Array
(
    [0] => HY000
    [1] => 144
    [2] => Table './virtualizor/tasks' is marked as crashed and last (automatic?) repair failed

Issue was the tasks table is marked as crashed. To solve this, we need to repair the table tasks.

First, find the password for database ‘virtualizor’ using the following command:

[root@sg40 ~]# grep dbpass /usr/local/virtualizor/universal.php
$globals['dbpass'] = 'gziqr4y989';

Now connect to the mysql using the password:

/usr/local/emps/bin/mysql -u root -p virtualizor

Now repair the table:

use virtualizor;
REPAIR TABLE tasks;

Now the virtualizor shall work.

How to mount raw (.img) images in Linux

You may have raw images in your system coming from some bare metal VM, like KVM/Vmware. You may mount them directly to the host machine without attaching them to a VM. To do so, you need two things:

  1. Partition to mount
  2. Sector size of the partition and the first position of the partition.

Both of these, can be derived using a command called fdisk with attribute l as the following:

[root@vps5 vpsextra]# fdisk -l mhdevelopment.img

Disk mhdevelopment.img: 268.4 GB, 268435456000 bytes, 524288000 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x000f1752

            Device Boot      Start         End      Blocks   Id  System
mhdevelopment.img1            2048   523241471   261619712   83  Linux
mhdevelopment.img2       523241472   524287999      523264   82  Linux swap / Solaris
You have new mail in /var/spool/mail/root
[root@vps5 vpsextra]#

Here, we are trying to mount an old image of mine, called ‘mhdevelopment.img. I would like to mount the partition that is not swap, which is namely ‘mhdevelopment.img2’ here. This partition starts from 2048 and the sector size is 512 bytes.

So to mount the partition, you can now use the following command:

sudo mount -o loop,offset=$((2048 * 512)) mhdevelopment.img /mnt

This will mount the the partition mhdevelopment.img1 to /mnt. If you want to change the mount destination, you may do this at the end in replace of /mnt to wherever you want this to be.

Happy shooting.

How to Add RAW Image into An Existing OpenVZ 7 VM

OpenVZ 7 supports both KVM VM and OpenVZ containers. For a customer, we were trying to import a CentOS 7 based KVM VM to OpenVZ 7 KVM VM. But the problem is, the VM was created in RAW format. But OpenVZ 7 KVM does not support RAW files for KVM, it supports QCOW2. Hence, we had to first convert the RAW image into a QCOW2 image first.

How to convert RAW KVM VM Image to QCOW2 VM Image

To convert raw to qcow2, you may use qemu-img kvm tool. This tool comes with the KVM setup. Let’s say your raw image is ‘harddisk.hdd.raw’, and you would like to produce a qcow2 image, called ‘harddisk.hdd’. To do that, you may run the following:

qemu-img convert -f raw -O qcow2 /root/harddisk.hdd.raw /root/harddisk.hdd.qcow2

Make sure, you are running this script from the place where you have the harddisk.hdd.raw stored.

Replace the disk in OpenVZ 7

In our case, our files are stored under /vz/vmprivate/. Under this folder of OpenVZ, you would find folders with VM id. In our case, it was under the following:

/vz/vmprivate/9d07cfef-42bf-4fbe-ac6a-7af9c85c9475

You can also see the list of VMs UUID, by typing the following command:

prlctl list --all

First, stop the VM

prlctl stop 9d07cfef-42bf-4fbe-ac6a-7af9c85c9475

Now, all you need to do, is to replace the ‘harddisk.hdd’ with the one we had converted

cd /vz/vmprivate/9d07cfef-42bf-4fbe-ac6a-7af9c85c9475
mv harddisk.hdd harddisk.hdd_backup
mv /root/harddisk.hdd.qcow2 harddisk.hdd

Now, start the VM, and you should be good to go

prlctl start 9d07cfef-42bf-4fbe-ac6a-7af9c85c9475

Quick How To: Finding IO Abuser in KVM VM

I thought to write a quick how to on finding an abuser in a KVM VM Host. There is a tool shipped with libvirt is called ‘virt-top’. Virt-topĀ  has many usage case. It can be used to detect the IO Abuser. Most of the cases, you would see the abuser is throwing a lot of IO Requests regardless of the amount of IO being written or read. Which is why, it important to first identify if you are hitting the IOPS limit of your disk or not by using iostat. A common tool I regularly use to identify first hand disk problem is iotop as well. The following is the favorite iotop command:

iotop -oaP

-o will only show the threads that are actually doing IO in the server instead of all the sleeping threads, keeping the iotop result clean. ‘P’ will show only the processes instead of every single threads. Each VM can have thousands of threads which will show up on the process ID. ‘a’ is specifically my favorite, that does accumulated output. It will show you the sum of the usage for the time your interactive iotop is running.

Once you are done with the first hand investigation, you may now use virt-top to detect the VM activity further. A most used command for me to detect IO abuser is the following:

virt-top -3 –block-in-bytes -o blockwrrq

-3 tells the virt-top to find block device usage and find them by ‘bytes’ while the -o ‘blockwrrq’ means to sort the output by the write iops of the VM. You can use blockrdrq to sort the result by read iops too.

Once you can mix the output of virt-top and iotop results, you shouldn’t have difficulty to detect the VM that is abusing the IO on the server.

error: Unable to create cgroup for vm**: No such file or directory

The error can appear for any type of KVM VM installation like VirtualizorĀ  or SolusVM or Proxmox. You may face the same error if you are simply using virtmanager to manage your KVM installation. The error appear when you try to start/create the VM from the xml file:

[root@vps8 addvs]# virsh create /etc/libvirt/qemu/v1015.xml
error: Failed to create domain from /etc/libvirt/qemu/v1015.xml
error: Unable to create cgroup for v1015: No such file or directory

It appears because for some CentOS starts unmounting the cgroup and breaks libvirt. Easy way to fix this is to restart libvirtd:

service libvirtd restart

The error is more common in CentOS 7 than CentOS 6, as systemd is known to have the bug:

https://bugzilla.redhat.com/show_bug.cgi?id=678555