This discussion is archived
0 Replies Latest reply: Aug 13, 2012 5:45 AM by 955453 RSS

Storage Performance problem

955453 Newbie
Currently Being Moderated
Hello,

Let me ask you for lights troubleshooting this problem ...
Here we have a database server runing on a M5000, SOL10,
with Brocade HBAs to SAN and using 2 IBM DS4700 Storage arrays
(called STOR01 and 02).

Problem : Commit delays, DB slow in creating tables, ....
DB DBP @ SUNGZ Global zone.

Storage info :
DBP on SUNGZ
LUN : 01_B_03 (Tower 01, Ctrl B, Array 03) 275GB.

Testing copies with a test file with File Size 4859109376 Bytes

My Analyse
----------
1)     Poor performance. Suspecting STOR01 to be the guilty.
[root@SUNGZ /DB/DBP/tmp]#time cp temp011.dbf tfd
real 24m30.857s

Moving data away To STOR02...
[root@SUNGZ #]zfs replace ORA-DBP-pool3 OLD NEW)

Info new LUN : 02_A_06 (Tower 02, Ctrl A, Array 06)

[root@SUNGZ /DB/DBP/tmp]#time cp temp011.dbf tfd
real 16m30.857s

Difference 8 MINUTES. Better but not best.

2)     Evacuate freed lun 01_B_03. Testing back 01_B_03 LUN from SUNGZ.
a.     creating pool “zpool create –m none TFDPOOL LUN”
b.     creating volume “zfs create –o mountpoint=/TFD TFDPOOL:TFD”
c.     copy STOR02 -> STOR01
[root@SUNGZ /DB/DBP/tmp]#time cp temp011.dbf /TFD/tfd
real 4m47.3
GREAT ! … But it took 24 mins before zfs replace on this lun …
d.     copy STOR01 -> STOR01
# ls -l /TFD
total 18995430
-rw-r----- 1 root root 4859109376 Aug 13 12:08 tfd2
# time cp temp011.dbf /TFD/tfd
real 4m45.2
GREAT !
e.     copy STOR02 -> STOR02
[root@SUNGZ /DB/DBP/tmp]#time cp temp011.dbf tfd
real 15m59.857s
WHAT A PERF !


3)     Other copy tests :
Copy 01_B_05 (DBD) : 4.8GB in 4m22.
Copy 02_A_04 (DBA) : 1.2GB in 1:04. (4.8GB in 4m16)

Conclusions :

1)     Moving STOR01 -> STOR02, same poor performance. Best but.
Once moved to STOR02, Same poor performance while writing locally.
Other DBs on STOR02 are fine.
Same poor performance behavior with 2 different storage arrays.

2)     After tower swap, From Same Host, Orignal “SLOW LUN 01_B_03” is not the problem
Copy from STOR02 to STOR02 -> 16m30 and 15m59 (Test Analyse 1B & 2E)
Copy from STOR01 to STOR01 -> 4m45
Copy from STOR02 to STOR01 -> 4m47

Perf can change depending of storage array workload (diff 10, 15 or 30 secs)
Slow lun problem moved to STOR02 (who was faster than STOR01)
As writing to STOR01 makes no difference where the IOread comes from,problem is by consequences at WRITE Level.
… WRITE level, yes, but now on STOR02...but we had same behavior on STOR01
Again Same poor performance behavior with 2 different storage arrays.

Defragmentation ? Same IO speed copying to STOR01.
Problem is STORAGE ARRAY INDEPENDENT.

Tested ZFS NO_CACHEFLUSH parameter on Dev and Acc.
No spectaclular wins.

Tests on 2 other Sol10 Global zones (Dev on 01_B_05) and (Acc on 02_A_04) are both ok
SO, it is Hardware platform independent (V490, M4000, M5000)

If storage array independent then problem is HOST SIDE.
(For this case, SUNGZ)

ZFS Problem ?
Zpool upgrade listed pools to release 15. So we ran ZPOOL UPDATE -A
ZFS Upgraded pools to latest level, no spectacular win.

Possible : TFDPOOL is NEWLY created pool while
ORA-DBP-pool3 has been created while ago …

Any idea where that performance difference may come from ?

Thanks

Thierry
Belgium

Edited by: user13397414 on 13-août-2012 5:43

Edited by: user13397414 on 13-août-2012 5:44

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points