Category: Performance

How to measure the sequential write/read speed of a Hard Disk or SSD?

Question: How to measure the sequential write speed of a hard disk or SSD?

Answer

We may use two ways. The most popular one is to use dd tools. For hard disks, we may not avoid using cache to measure the disk performance. As sequential writes can write the data to the cache, which is basically a DRAM and can perform excessively fast masking the original speed of the HDD. Let’s check how can we measure the HDD speed using dd bypassing HDD write cache:

dd if=/dev/zero of=/hddvz/testfile.img bs=1G count=1 oflag=direct

/hddvz is my HDD mount, oflag=direct instruct the dd to confirm writes to the disk, not just the cache before saying the write is completed.

If it’s a good quality hard disk alone or soft raid 1, you can get speed up to 157-161MBps:

[root@bd3 ~]# dd if=/dev/zero of=/hddvz/testfile.img bs=1G count=1 oflag=direct
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB) copied, 6.82675 s, 157 MB/s
[root@bd3 ~]#

With a good hardware raid controller and raid 10, you may be able to reach rates like 320MBps.

There is something to remember, software raid mdadm is able to read simultaneously from two disks. You may have the evidence, with two dd one by one, both in background as following:

[root@bd3 ~]# dd if=/dev/zero of=/hddvz/testfile.img bs=1G count=1 oflag=direct &
[1] 12039
[root@bd3 ~]# dd if=/dev/zero of=/hddvz/testfile1.img bs=1G count=1 oflag=direct &
[2] 12040
[root@bd3 ~]# 1+0 records in
1+0 records out
1073741824 bytes (1.1 GB) copied, 11.1064 s, 96.7 MB/s
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB) copied, 10.7155 s, 100 MB/s

[1]-  Done                    dd if=/dev/zero of=/hddvz/testfile.img bs=1G count=1 oflag=direct
[2]+  Done                    dd if=/dev/zero of=/hddvz/testfile1.img bs=1G count=1 oflag=direct

As you can see, both the request was able to reach 100MBps roughly while running in parallel.

Testing Read Speed Using hdparm

To check the read speed, you may use hdparm like the following:

[root@bd3 ~]# hdparm -Tt /dev/sda

/dev/sda:
 Timing cached reads:   40444 MB in  1.99 seconds = 20365.12 MB/sec
 Timing buffered disk reads: 396 MB in  3.06 seconds = 129.43 MB/sec
[root@bd3 ~]# hdparm -Tt /dev/sdb

/dev/sdb:
 Timing cached reads:   41006 MB in  1.99 seconds = 20649.43 MB/sec
 Timing buffered disk reads: 562 MB in  3.01 seconds = 186.81 MB/sec
[root@bd3 ~]#

How to Use Sticky Session for CSRF submission on Highly Scalable Cloud App in Haproxy

HINT: If you are a nginx fan and used it in mass scale, then, you must have done this using ip_hash (Nginx Documentation). It follows the same purpose for Haproxy. Difference and benefits of using Haproxy over Nginx for L7 proxy in a highly scalable and reliable cloud app would be a discussion for another day.

Case Discussion:

Suppose, you have a Cloud app, that is load balanced & scaled between multiple servers using Haproxy, for example:

101.101.101.101
202.202.202.202
303.303.303.303

Now, if your app has a submission form, for example, a poll submission from your users, then, there is an issue in this Haproxy setup.

Let’s say, an User A, requests for the app, and gets the data from the server 101.101.101.101, the CSRF token he gets for the poll submission to his browser, also maintains the app hosted on 101.101.101.101. But when he press the submit button, HAProxy puts him on 202.202.202.202 app, and the app hosted on 202.202.202.202 instantly rejects the token for the session as the session is not registered for that app. For such cases, we need to maintain a ‘Sticky’ session based on the cookie set by the right server. That means, if the cookie is set by 101.101.101.101, HAproxy should obey and give the user 101.101.101.101 until the cookie or the session is reset or regenerated.

How To Do That:

What we need to do, let haproxy write the server id in the cookie, and make the directive ‘server’ to follow the cookie. Please remember, there are couple of other way to achieve this. There is another way of doing this is called ‘IP Affinity’, where you make sticky session based on IP of the user. There is another based on PHP session value. Setting sticky session based on php session should also work. I preferred the cookie based sticky session, just on random selection.

So, to write the server id in the cookie, you need to add the following in the haproxy ‘backend’ directive as following:

backend app-main
balance roundrobin
cookie SERVERID insert indirect nocache

In the cookie directive, you can see, we are taking the HAProxy variable ‘SERVERID’ and inserting that to the cookie attribute. Now, all you need to do, is to configure your balancing IPs to follow the cookie, like the following:

backend app-main
balance roundrobin
cookie SERVERID insert indirect nocache
server nginx1 101.101.101.101 cookie S1
server nginx2 202.202.202.202 cookie S2
server nginx3 303.303.303.303 cookie S3

S1, S2, S3 are just 3 different names of the cookies for the specific servers. After the above is done, you can now restart and see Haproxy is following stickiness based on the session you have.

Just to find out, how to test if you are using laravel, try to regenerate the session based on the session() helper method as following:

session()->regenerate()->csrf_token();

You should be able to see the content loading from different web servers when the session regenerates. But it will persists when the regenerate session method is not called.

What is Kondemand? Why do I see a lot of Kondemand process in my process list?

Question: What is Kondemand? Why do I see a lot of Kondemand process in my process list?

Answer: Kondemand is the process used for automatic CPU scaling on multi core linux system. It automatically reduce/drops the CPU clock speed to power usage when the CPU is not in use. This is done through scaling_governor available on linux. To see if your scaling_governor is set to ‘ondemand’ or not, you may use the following command:

# cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

If your CPU is showing ‘ondemand’ scanling governor then the kondemand kernel process is active and will reduce your CPU clock speed on fly to reduce power usage. You can change this settings to performance on fly using the following small shell code:

for CPUFREQ in /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor; do [ -f $CPUFREQ ] || continue; echo -n performance > $CPUFREQ; done

There is a linux service called CPUSpeed, this can tune your scaling governor back to ondemand after the reboot. You may shut it down:

service cpuspeed stop
chkconfig off cpuspeed

You may check your CPU speed is restored to the original through the proc filesystem:

cat /proc/cpuinfo