This discussion is archived
3 Replies Latest reply: Mar 8, 2013 4:54 AM by ekrengel RSS

3.2.1 w NFS PVM slow write performance ~2.7 MB/s

Roddy Rodstein Explorer
Currently Being Moderated
Greetings,

Happy FRIDAY!

Our PVM guests are I/O challenged with ~2.7 MB/s write speeds. To our surprise, our HVM guests do better than PVM with 11 MB/s write speeds. We reckoned the NFS fix was in build 379, we are using 3.2.1. We welcome your comments and suggestions to help explain why PVM writes are so slow compared to HVM, and to help speed up PVM writes.

Thank you in advance for your support!

Environment:
Oracle VM 3.2.1 fully patched on 4 Proliant DL360 G7s each with 4 1G NICs. We setup 2 bonds using mode 4 w LACP & 802.1q. One bond with the SM, CHB, LM and Storage/NFS network channels, and the second bond with public and private tagged vlans. Oracle VM Manager is running on a OL63 VMware VM (2CPU 8G RAM) with MySQL.

We puts a couple days into iperf and sysctl.conf TCP tuning (dom0 and domU) just to get where we are! I/O performance was MUCH worse with the defaults.

Write and read tests:
OL59 PVM w UEK (fully patched via the public repo)
# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 5.9 (Tikanga)
# uname -a
2.6.32-300.39.4.el5uek
# time dd bs=1M count=1024 if=/dev/zero of=test_dd conv=fdatasync
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 399.618 seconds, 2.7 MB/s

real     6m39.843s
user     0m0.004s
sys     0m1.685s

# time dd bs=1M count=1024 if=test_dd of=/dev/null
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 0.129319 seconds, 8.3 GB/s

real     0m0.131s
user     0m0.001s
sys     0m0.130s

OL59 HVM w UEK (fully patched via the public repo)
# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 5.9 (Tikanga)
# uname -a
2.6.32-300.39.4.el5uek
# time dd bs=1M count=1024 if=/dev/zero of=test_dd conv=fdatasync
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 98.2748 s, 10.9 MB/s

real     1m38.422s
user     0m0.000s
sys     0m0.870s

# time dd bs=1M count=1024 if=test_dd of=/dev/null
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 0.115353 s, 9.3 GB/s

real     0m0.117s
user     0m0.000s
sys     0m0.117s

dom0 stats:
# nfsstat -m
/nfsmnt/0077fc1f-9ff6-42ed-a4c3-1dedbc2d8fe0 from 192.168.2.101:/nfs/poofs_01/
Flags:     rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=65535,timeo=600,retrans=2,sec=sys,local_lock=none,addr=192.168.2.101

/OVS/Repositories/0004fb0000030000722fc689b3bd9670 from 192.168.2.101:/nfs/vmfs_01/
Flags:     rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=65535,timeo=600,retrans=2,sec=sys,local_lock=none,addr=192.168.2.101

# cd /OVS/Repositories/0004fb0000030000722fc689b3bd9670
# time dd bs=1M count=1024 if=/dev/zero of=test_dd conv=fdatasync
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 96.6368 seconds, 11.1 MB/s

real     1m36.922s
user     0m0.000s
sys     0m0.573s

# time dd bs=1M count=1024 if=test_dd of=/dev/null
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 91.2823 seconds, 11.8 MB/s

real     1m31.286s
user     0m0.000s
sys     0m0.316s


FYI, on the NAS in the /OVS/Repositories/0004fb0000030000722fc689b3bd9670 export:
# time dd bs=1M count=1024 if=/dev/zero of=test_dd conv=fdatasync
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 2.93967 s, 365 MB/s

real     0m2.955s
user     0m0.003s
sys     0m1.186s

# time dd bs=1M count=1024 if=test_dd of=/dev/null
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 0.277062 s, 3.9 GB/s

real     0m0.278s
user     0m0.000s
sys     0m0.278s


FYI, for fun, a VMware RHEL 59 guest (same CPU, RAM and disk setup as on OVM) on the same server hardware and same NAS:
# time dd bs=1M count=1024 if=/dev/zero of=test_dd conv=fdatasync
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 6.79495 seconds, 158 MB/s

real 0m6.796s
user 0m0.004s
sys 0m1.794s

[root@rpwhqovm01 tmp]# time dd bs=1M count=1024 if=test_dd of=/dev/null
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 0.464443 seconds, 2.3 GB/s

real 0m0.466s
user 0m0.001s
sys 0m0.422s

Thank you for your support!

FYI:
Other NFS posts:
NFS disk performance after upgrade to 3.1.1
Slow disk writes VM 3.1

Regards,
Roddy

Edited by: Roddy Rodstein on Feb 15, 2013 8:55 AM

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points