This content has been marked as final. Show 4 replies
if using ZFS, is it recommended to use zpool iostat over the normal iostat?I like both of them, but I'd recommend 'zpool iostat -v' because it will also give you per-device statistics through zpool.
Basically what is the different between both?iostat displays statistics for all devices, zpool iostat will only give statistics for devices in a zpool. When you have cache and log devices and mirrors and stuff configured, iostat may not be very easy to read. But iostat has information on service time that zpool iostat does not have
In normal iostat, what is the most valuable info we need to look into to measure the performance? a_svc, b%, w%?I usually look at number of operations (read and write), service time (svc_t) and sometimes the throughbut (kread, kwrite). I don't trust %b or %w because I nerver really understood how they are calculated, I also suspect they may be calculated differently on different platforms (linux and solaris) and I have seen cases where they have just been wrong. So I go by rules of thumb for operations and service times:
A single disk should be able to provide ~100 random read operations per second. Maybe 150, maybe 75 but definitely somewhere in that region. If the reported number gets close to that (or bigger), I start to get worried.
The service time of a random read request should be about 4-6ms. Maybe 7 or 8. But if the service time is reported in double-digits, I start to get worried.
Everything else will depend on your application and your threshold for nervousness. Some of my customers feel ok until %b reaches 80%, some may think that service times of 12ms are ok. The best way is to compare these numbers to baselines from when the system is performing OK.
where in zpool iostat, what is the most valuable info we need to look into to measure the performance? and how to determine it is slow or fast? since, there are no a_svc, b%, w%??again, reads and writes is usually what I look for.
in the following info in http://docs.oracle.com/cd/E19253-01/819-5461/gammt/index.html?bandwidth is reported in bytes. In this case, zpool iostat was called without a repeat-interval so only the average since system boot is reported. So in this case 786 bytes have been read per second on average since system start. Quite a useless number.
# zpool iostat
capacity operations bandwidth
pool alloc free read write read write
rpool 6.05G 61.9G 0 0 786 107
tank 31.3G 36.7G 4 1 296K 86.1K
what does 786 and 107 for rpool refer to? kilobyte?bytes? bit?
So can I say, in zpool iostat, :
1. The more number of Read Write operation, the number bandwidth will be smaller?
2. the lesser number of Read Write operation, the number bandwidth will be bigger?
But we can't control the number of read write rite?
Basically, What is the relationship between the Read write and bandwidth?
1. The more number of Read Write operation, the number bandwidth will be smaller?no, that will usually be related to each other in a way that both will get bigger. read/write are the number of operations. the number of times that an application (or the OS) asks the disks for something. Bandwidth is the amount of data transferred between the OS (or app) and the disks. Both number of operations (IOPS) and bandwidth (MB/s) have a limit.
Here is an example:The default oracle block size is 8kB. If a block is not in a buffer in memory (the SGA) it has to be fetched from disk. That will be one IO operation. And 8kB will be transferred from the disk. So if your database fetches 1000 blocks from disks per second, you would see 1000 read operations and a bandwith of 8MB/s in iostat.
Another example is this blog I wrote a few weeks ago: http://portrix-systems.de/blog/brost/asm-vs-zfs-performance/
I benchmarked the IO system and with ASM I maxed out at about 100.000 IOPS. Which translates to about 800MB/s in bandwidth. Now my Server was connected with 2 4Gb/s FC ports. These support a maximum bandwidth of about 400MB/s per port. So it was clear that I was hitting the controllers as a bottleneck. I now know that if I ever hit that number in production on that system, I'd have to upgrade the controllers.
Also, in one of the tests I set the ZFS block size to 128kB. Therefor every Oracle IO request, even for just 8K became a 128k read from disk. Which saturated the 800MB/s with only about 6000 IOPS. I spotted that by looking at the bandwidth in iostat.
But we can't control the number of read write rite?We don't just do these for fun or because we can. IOs usually have a reason. Some app needs access to data that is not in memory. Whenever that happens, we need to read from or write to disk. There is nothing wrong with that. It usually takes a few ms to fulfil each requests. There is also nothing wrong with that. But as the number of requests or bandwidth goes up, this time may increase and that is one of the things we look for and monitor.
If you discover that IO is a problem and that it is caused by an unreasonable amount of requests, we need to investigate further and do something about it. Solutions can be to tune SQL (to use more efficient access paths), create additional indexes, increase caches or rethink what exavtly the application is doing. Or on the other hand, we could also improve the disk subsystem to provide more requests or bandwidth by using faster disks or stripe across more spindles to spread the load.