0 Replies Latest reply on Aug 13, 2012 12:45 PM by user13397414

    Storage Performance problem


      Let me ask you for lights troubleshooting this problem ...
      Here we have a database server runing on a M5000, SOL10,
      with Brocade HBAs to SAN and using 2 IBM DS4700 Storage arrays
      (called STOR01 and 02).

      Problem : Commit delays, DB slow in creating tables, ....
      DB DBP @ SUNGZ Global zone.

      Storage info :
      DBP on SUNGZ
      LUN : 01_B_03 (Tower 01, Ctrl B, Array 03) 275GB.

      Testing copies with a test file with File Size 4859109376 Bytes

      My Analyse
      1)     Poor performance. Suspecting STOR01 to be the guilty.
      [root@SUNGZ /DB/DBP/tmp]#time cp temp011.dbf tfd
      real 24m30.857s

      Moving data away To STOR02...
      [root@SUNGZ #]zfs replace ORA-DBP-pool3 OLD NEW)

      Info new LUN : 02_A_06 (Tower 02, Ctrl A, Array 06)

      [root@SUNGZ /DB/DBP/tmp]#time cp temp011.dbf tfd
      real 16m30.857s

      Difference 8 MINUTES. Better but not best.

      2)     Evacuate freed lun 01_B_03. Testing back 01_B_03 LUN from SUNGZ.
      a.     creating pool “zpool create –m none TFDPOOL LUN”
      b.     creating volume “zfs create –o mountpoint=/TFD TFDPOOL:TFD”
      c.     copy STOR02 -> STOR01
      [root@SUNGZ /DB/DBP/tmp]#time cp temp011.dbf /TFD/tfd
      real 4m47.3
      GREAT ! … But it took 24 mins before zfs replace on this lun …
      d.     copy STOR01 -> STOR01
      # ls -l /TFD
      total 18995430
      -rw-r----- 1 root root 4859109376 Aug 13 12:08 tfd2
      # time cp temp011.dbf /TFD/tfd
      real 4m45.2
      GREAT !
      e.     copy STOR02 -> STOR02
      [root@SUNGZ /DB/DBP/tmp]#time cp temp011.dbf tfd
      real 15m59.857s
      WHAT A PERF !

      3)     Other copy tests :
      Copy 01_B_05 (DBD) : 4.8GB in 4m22.
      Copy 02_A_04 (DBA) : 1.2GB in 1:04. (4.8GB in 4m16)

      Conclusions :

      1)     Moving STOR01 -> STOR02, same poor performance. Best but.
      Once moved to STOR02, Same poor performance while writing locally.
      Other DBs on STOR02 are fine.
      Same poor performance behavior with 2 different storage arrays.

      2)     After tower swap, From Same Host, Orignal “SLOW LUN 01_B_03” is not the problem
      Copy from STOR02 to STOR02 -> 16m30 and 15m59 (Test Analyse 1B & 2E)
      Copy from STOR01 to STOR01 -> 4m45
      Copy from STOR02 to STOR01 -> 4m47

      Perf can change depending of storage array workload (diff 10, 15 or 30 secs)
      Slow lun problem moved to STOR02 (who was faster than STOR01)
      As writing to STOR01 makes no difference where the IOread comes from,problem is by consequences at WRITE Level.
      … WRITE level, yes, but now on STOR02...but we had same behavior on STOR01
      Again Same poor performance behavior with 2 different storage arrays.

      Defragmentation ? Same IO speed copying to STOR01.

      Tested ZFS NO_CACHEFLUSH parameter on Dev and Acc.
      No spectaclular wins.

      Tests on 2 other Sol10 Global zones (Dev on 01_B_05) and (Acc on 02_A_04) are both ok
      SO, it is Hardware platform independent (V490, M4000, M5000)

      If storage array independent then problem is HOST SIDE.
      (For this case, SUNGZ)

      ZFS Problem ?
      Zpool upgrade listed pools to release 15. So we ran ZPOOL UPDATE -A
      ZFS Upgraded pools to latest level, no spectacular win.

      Possible : TFDPOOL is NEWLY created pool while
      ORA-DBP-pool3 has been created while ago …

      Any idea where that performance difference may come from ?



      Edited by: user13397414 on 13-août-2012 5:43

      Edited by: user13397414 on 13-août-2012 5:44