Forum Stats

  • 3,813,903 Users
  • 2,258,789 Discussions
  • 7,892,437 Comments

Discussions

Question about periodic performance fluctuation (for every 45s) during disk performance test

3214577
3214577 Member Posts: 2
edited Apr 7, 2016 3:38PM in Vdbench

Hello,

I usually evaluate the disk performance using vdbench.

(It's very good for showing the performance result in detail. thanks)

I have some issues about periodic performance fluctuation for every 45s

I set up the reporting interval = 1s, and every 45s, the performance is fluctuated.

First, I changed the disk to other brand, but same phenomenon occured.

Second, I assumed that this issue is made by garbage collection in java.

Thus, using jstat tools for monitoring java GC, there is no relationship between java gc & 45s performance fluctuation.

I also changed the openjdk-7 to oracle-jdk-8, but the phenomenon still remains.

Is there any background work in vdbench for every 45 seconds?

I hope your kind response

Best regards,

sk

Best Answer

  • Henk Vandenbergh-Oracle
    Henk Vandenbergh-Oracle Member Posts: 813
    edited Apr 6, 2016 9:38AM Answer ✓

    Interesting.

    The only '45 second' thing that is going on in Vdbench is the heartbeat monitor (heartbeat timeout = 180 seconds divided by four).

    The heartbeat monitor in the master sends a message to all slaves every 45 seconds and expects the slaves to respond within 180+5 seconds to make sure everything is still alive.

    - What OS are you running?

    - how many iops?

    - how is your cpu usage?

    - please send me a few hundred lines of summary.html, covering a period of maybe 5 minutes.

    thx

    Henk.

Answers

  • Henk Vandenbergh-Oracle
    Henk Vandenbergh-Oracle Member Posts: 813
    edited Apr 6, 2016 9:38AM Answer ✓

    Interesting.

    The only '45 second' thing that is going on in Vdbench is the heartbeat monitor (heartbeat timeout = 180 seconds divided by four).

    The heartbeat monitor in the master sends a message to all slaves every 45 seconds and expects the slaves to respond within 180+5 seconds to make sure everything is still alive.

    - What OS are you running?

    - how many iops?

    - how is your cpu usage?

    - please send me a few hundred lines of summary.html, covering a period of maybe 5 minutes.

    thx

    Henk.

  • 3214577
    3214577 Member Posts: 2
    edited Apr 7, 2016 12:14AM

    Thanks,

    With your answer, I checked whether the performance fluctuation depends on the heartbeat or not.

    I changed the heartbeat period using "-d27" (long timeout: 900s) and "-d28" (short timeout: 10s) and test again.

    The performance fluctuation depends on the heartbeat period (timeout / 4).

    I find out the root cause of the performance fluctuation by your favor. The details are below.

    - HW: Intel Core i7-4770 @ 3.4GHz, 32GB

    - OS: Ubuntu 14.04.2 LTS (kernel 3.16.0)

    - IOPS: avg. 4268.88 (532.11 MB/s, 128KB sequential read in Samsung SSD 840 pro 256GB, Qd=32)

    - CPU: 1.0% (sys+user) & 0.7% (sys)

    - I'll put some lines of summary.html below.

      You can see that the performance is fluctuated for every 45s (+-3~4%)

      I evaluated with the long & short heartbeat period, then this performance fluctuation occured for every 225s & 2.5s.

      I verify that the performance flucuation is due to the heartbeat monitor. (Thank you very much)

    I have another question.

    Could you let me know why the performance is slightly increased & decreased during the heartbeat monitor? (+-3~4%)

    Is there any performance compensation scheme when the vdbench calcuates the performance?

    (The average performance during the heartbeat monitor is similar to the normal performance.)

    Thanks,

    sk.

    ==========================================================================================================

    12:32:53.043    314256.00   532.00  131072 100.007.5217.5210.0007.5610.009  32.0   0.5   0.3
    12:32:54.041    324257.00   532.13  131072 100.007.5167.5160.0007.5740.010  32.0   0.5   0.3
    12:32:55.042    334254.00   531.75  131072 100.007.5177.5170.0007.5440.008  32.0   0.8   0.5
    12:32:56.042    344255.00   531.88  131072 100.007.5217.5210.0007.5480.007  32.0   0.9   0.9
    12:32:57.041    354257.00   532.13  131072 100.007.5187.5180.0007.5840.009  32.0   1.6   0.6
    12:32:58.041    364258.00   532.25  131072 100.007.5177.5170.0007.5520.008  32.0   0.3   0.1
    12:32:59.041    374257.00   532.13  131072 100.007.5217.5210.0007.5600.008  32.0   0.5   0.3
    12:33:00.041    384251.00   531.38  131072 100.007.5197.5190.0007.5460.007  32.0   0.5   0.4
    12:33:01.041    394256.00   532.00  131072 100.007.5227.5220.0007.5500.007  32.0   0.6   0.6
    12:33:02.041    404254.00   531.75  131072 100.007.5217.5210.0007.5660.006  32.0   1.0   0.8
    12:33:03.041    414254.00   531.75  131072 100.007.5247.5240.0007.5670.007  32.0   1.3   0.8
    12:33:04.041    424255.00   531.88  131072 100.007.5227.5220.0007.5930.008  32.0   0.5   0.3
    12:33:05.043    434251.00   531.38  131072 100.007.5217.5210.0007.5630.008  32.0   0.6   0.5
    12:33:06.044    444253.00   531.63  131072 100.007.5247.5240.0007.5820.009  32.0   1.8   0.5
    12:33:07.043    454419.00   552.38  131072 100.007.5227.5220.0008.8900.045  33.2   1.0   0.6
    12:33:08.052    464096.00   512.00  131072 100.007.5227.5220.0007.5910.009  30.8   1.0   0.4
    12:33:09.041    474248.00   531.00  131072 100.007.5217.5210.0007.5500.008  31.9   1.3   0.9
    12:33:10.041    484254.00   531.75  131072 100.007.5237.5230.0007.5740.009  32.0   1.5   0.6
    12:33:11.041    494255.00   531.88  131072 100.007.5227.5220.0007.5680.008  32.0   0.6   0.5
    12:33:12.041    504254.00   531.75  131072 100.007.5227.5220.0007.5550.007  32.0   0.5   0.5
    12:33:13.044    514253.00   531.63  131072 100.007.5237.5230.0007.5490.007  32.0   0.5   0.4
    12:33:14.042    524252.00   531.50  131072 100.007.5247.5240.0007.5460.007  32.0   0.8   0.5
    12:33:15.045    534257.00   532.13  131072 100.007.5247.5240.0007.6360.012  32.0   0.8   0.3
    12:33:16.041    544248.00   531.00  131072 100.007.5227.5220.0007.5890.009  31.9   1.5   0.8
    12:33:17.041    554255.00   531.88  131072 100.007.5217.5210.0007.5780.007  32.0   0.6   0.5
    12:33:18.041    564255.00   531.88  131072 100.007.5237.5230.0007.5700.008  32.0   0.4   0.3
    12:33:19.041    574254.00   531.75  131072 100.007.5237.5230.0007.5530.008  32.0   1.0   0.9
    12:33:20.041    584253.00   531.63  131072 100.007.5257.5250.0007.5520.007  32.0   1.0   0.7
    12:33:21.041    594250.00   531.25  131072 100.007.5237.5230.0007.5530.008  32.0   0.9   0.8
    12:33:22.041    604254.00   531.75  131072 100.007.5257.5250.0007.5970.010  32.0   1.1   0.5

    12:33:23.042    614256.00   532.00  131072 100.007.5207.5200.0007.5500.007  32.0   0.5   0.4
    12:33:24.041    624255.00   531.88  131072 100.007.5217.5210.0007.5470.007  32.0   0.4   0.3
    12:33:25.041    634254.00   531.75  131072 100.007.5247.5240.0007.7370.011  32.0   1.1   0.8
    12:33:26.041    644251.00   531.38  131072 100.007.5217.5210.0007.5460.008  32.0   0.5   0.3
    12:33:27.041    654255.00   531.88  131072 100.007.5247.5240.0007.5600.008  32.0   0.4   0.1
    12:33:28.041    664255.00   531.88  131072 100.007.5217.5210.0007.5720.008  32.0   1.4   0.8
    12:33:29.041    674255.00   531.88  131072 100.007.5217.5210.0007.5520.007  32.0   0.5   0.3
    12:33:30.041    684251.00   531.38  131072 100.007.5217.5210.0007.5450.007  32.0   0.1   0.1
    12:33:31.041    694254.00   531.75  131072 100.007.5277.5270.0007.6570.014  32.0   1.4   0.3
    12:33:32.041    704253.00   531.63  131072 100.007.5227.5220.0007.5500.007  32.0   0.8   0.4
    12:33:33.041    714256.00   532.00  131072 100.007.5207.5200.0007.5540.007  32.0   0.4   0.4
    12:33:34.041    724255.00   531.88  131072 100.007.5227.5220.0007.5620.007  32.0   1.9   1.0
    12:33:35.041    734251.00   531.38  131072 100.007.5217.5210.0007.5460.006  32.0   0.6   0.5
    12:33:36.041    744255.00   531.88  131072 100.007.5237.5230.0007.5520.007  32.0   0.1   0.1
    12:33:37.045    754260.00   532.50  131072 100.007.5217.5210.0007.5560.007  32.0   1.8   1.4
    12:33:38.046    764252.00   531.50  131072 100.007.5217.5210.0007.5690.008  32.0   2.6   2.1
    12:33:39.041    774253.00   531.63  131072 100.007.5227.5220.0007.5860.008  32.0   1.3   1.0
    12:33:40.042    784253.00   531.63  131072 100.007.5227.5220.0007.5670.009  32.0   1.1   0.6
    12:33:41.041    794251.00   531.38  131072 100.007.5217.5210.0007.5480.006  32.0   0.9   0.5
    12:33:42.042    804256.00   532.00  131072 100.007.5227.5220.0007.5440.006  32.0   0.5   0.5
    12:33:43.043    814254.00   531.75  131072 100.007.5247.5240.0007.5610.008  32.0   0.5   0.4
    12:33:44.046    824256.00   532.00  131072 100.007.5217.5210.0007.5590.008  32.0   1.4   1.0
    12:33:45.042    834248.00   531.00  131072 100.007.5237.5230.0007.5540.007  32.0   1.0   0.5
    12:33:46.041    844253.00   531.63  131072 100.007.5247.5240.0007.5890.007  32.0   1.4   0.6
    12:33:47.041    854255.00   531.88  131072 100.007.5237.5230.0007.5700.008  32.0   0.5   0.4
    12:33:48.041    864254.00   531.75  131072 100.007.5247.5240.0007.5550.009  32.0   0.6   0.5
    12:33:49.047    874255.00   531.88  131072 100.007.5247.5240.0007.5490.008  32.0   0.1   0.1
    12:33:50.043    884251.00   531.38  131072 100.007.5227.5220.0007.5840.008  32.0   2.0   1.6
    12:33:51.041    894252.00   531.50  131072 100.007.5237.5230.0007.6000.017  32.0   0.3   0.1
    12:33:52.044    904419.00   552.38 131072 100.007.5257.5250.0007.5940.010  33.2   2.4   1.1

    12:33:53.041    914089.00   511.13 131072 100.007.5227.5220.0007.5570.007  30.8   0.7   0.4
    12:33:54.044    924257.00   532.13  131072 100.007.5257.5250.0007.6070.009  32.0   1.0   0.9
    12:33:55.042    934247.00   530.88  131072 100.007.5247.5240.0007.7200.009  32.0   3.4   0.4
    12:33:56.041    944255.00   531.88  131072 100.007.5247.5240.0007.5550.008  32.0   0.6   0.6
    12:33:57.046    954253.00   531.63  131072 100.007.5257.5250.0007.5520.009  32.0   0.9   0.6
    12:33:58.041    964248.00   531.00  131072 100.007.5257.5250.0007.5650.009  32.0   1.5   0.9
    12:33:59.045    974257.00   532.13  131072 100.007.5267.5260.0007.6450.013  32.0   1.8   0.6
    12:34:00.041    984250.00   531.25  131072 100.007.5247.5240.0007.5650.009  32.0   1.1   0.8
    12:34:01.041    994254.00   531.75  131072 100.007.5247.5240.0007.5520.008  32.0   0.5   0.4
    12:34:02.041   1004248.00   531.00  131072 100.007.5257.5250.0007.5830.008  32.0   0.6   0.4
    12:34:03.042   1014258.00   532.25  131072 100.007.5247.5240.0007.5500.007  32.0   1.4   0.9
    12:34:04.046   1024256.00   532.00  131072 100.007.5257.5250.0007.5620.007  32.0   1.3   1.0
    12:34:05.041   1034245.00   530.63  131072 100.007.5247.5240.0007.5940.010  31.9   2.0   0.9
    12:34:06.041   1044253.00   531.63  131072 100.007.5247.5240.0007.5490.007  32.0   0.6   0.4
    12:34:07.041   1054254.00   531.75  131072 100.007.5267.5260.0007.5520.007  32.0   0.4   0.4
    12:34:08.042   1064254.00   531.75  131072 100.007.5247.5240.0007.5650.008  32.0   3.0   2.4
    12:34:09.044   1074251.00   531.38  131072 100.007.5257.5250.0007.7250.009  32.0   1.4   0.9
    12:34:10.041   1084250.00   531.25  131072 100.007.5257.5250.0007.5620.008  32.0   1.1   0.9
    12:34:11.041   1094254.00   531.75  131072 100.007.5257.5250.0007.5670.008  32.0   1.1   0.6
    12:34:12.041   1104252.00   531.50  131072 100.007.5267.5260.0007.5670.007  32.0   0.9   0.6
    12:34:13.041   1114249.00   531.13  131072 100.007.5267.5260.0007.5570.007  32.0   0.6   0.5
    12:34:14.042   1124255.00   531.88  131072 100.007.5247.5240.0007.5510.007  32.0   0.5   0.4
    12:34:15.041   1134253.00   531.63  131072 100.007.5257.5250.0007.5610.008  32.0   1.3   1.0
    12:34:16.041   1144249.00   531.13  131072 100.007.5327.5320.0008.5580.140  32.0   2.1   0.9
    12:34:17.045   1154260.00   532.50  131072 100.007.5137.5130.0007.5410.007  32.0   0.6   0.6
    12:34:18.041   1164256.00   532.00  131072 100.007.5137.5130.0007.5970.010  32.0   2.8   1.5
    12:34:19.041   1174260.00   532.50  131072 100.007.5137.5130.0007.5490.009  32.0   0.4   0.1
    12:34:20.041   1184256.00   532.00  131072 100.007.5127.5120.0007.5460.007  32.0   0.4   0.3
    12:34:21.042   1194264.00   533.00  131072 100.007.5137.5130.0007.5400.007  32.0   0.6   0.4
    12:34:22.042   1204259.00   532.38  131072 100.007.5147.5140.0007.5600.009  32.0   1.3   0.9

    12:34:23.041   1214258.00   532.25  131072 100.007.5137.5130.0007.5470.007  32.0   1.0   0.8
    12:34:24.041   1224256.00   532.00  131072 100.007.5137.5130.0007.5400.008  32.0   0.4   0.1
    12:34:25.041   1234260.00   532.50  131072 100.007.5127.5120.0007.5450.006  32.0   1.8   1.0
    12:34:26.041   1244258.00   532.25  131072 100.007.5167.5160.0007.5410.007  32.0   0.3   0.3
    12:34:27.041   1254260.00   532.50  131072 100.007.5147.5140.0007.5500.008  32.0   0.5   0.4
    12:34:28.041   1264259.00   532.38  131072 100.007.5147.5140.0007.5570.007  32.0   0.5   0.3
    12:34:29.041   1274257.00   532.13  131072 100.007.5127.5120.0007.5420.007  32.0   0.9   0.6
    12:34:30.041   1284260.00   532.50  131072 100.007.5127.5120.0007.5370.007  32.0   0.6   0.5
    12:34:31.041   1294258.00   532.25  131072 100.007.5167.5160.0007.5810.009  32.0   1.8   0.8
    12:34:32.040   1304260.00   532.50  131072 100.007.5137.5130.0007.5920.009  32.0   0.5   0.3
    12:34:33.040   1314254.00   531.75  131072 100.007.5167.5160.0007.6400.012  32.0   0.4   0.3
    12:34:34.041   1324257.00   532.13  131072 100.007.5197.5190.0007.5530.007  32.0   0.8   0.5
    12:34:35.040   1334259.00   532.38  131072 100.007.5147.5140.0007.5400.007  32.0   0.6   0.3
    12:34:36.040   1344256.00   532.00  131072 100.007.5207.5200.0007.5530.010  32.0   0.5   0.4
    12:34:37.042   1354424.00   553.00 131072 100.007.5137.5130.0007.5480.007  33.2   1.9   0.8
    12:34:38.040   1364092.00   511.50 131072 100.007.5127.5120.0007.5480.007  30.7   0.7   0.4
    12:34:39.041   1374259.00   532.38  131072 100.007.5147.5140.0007.5440.008  32.0   0.9   0.6
    12:34:40.040   1384261.00   532.63  131072 100.007.5147.5140.0007.5420.007  32.0   0.6   0.6
    12:34:41.040   1394257.00   532.13  131072 100.007.5167.5160.0007.5400.007  32.0   0.6   0.4
    12:34:42.040   1404260.00   532.50  131072 100.007.5137.5130.0007.5370.007  32.0   1.4   0.9
    12:34:43.046   1414261.00   532.63  131072 100.007.5117.5110.0007.5410.007  32.0   1.3   0.8
    12:34:44.043   1424258.00   532.25  131072 100.007.5167.5160.0007.5580.011  32.0   1.4   1.0
    12:34:45.045   1434262.00   532.75  131072 100.007.5147.5140.0007.5480.008  32.0   1.5   1.0
    12:34:46.044   1444255.00   531.88  131072 100.007.5137.5130.0007.5690.008  32.0   1.4   1.0
    12:34:47.043   1454258.00   532.25  131072 100.007.5157.5150.0007.5650.012  32.0   1.5   0.6
    12:34:48.040   1464253.00   531.63  131072 100.007.5147.5140.0007.5730.008  32.0   2.1   1.1
    12:34:49.041   1474261.00   532.63  131072 100.007.5147.5140.0007.5450.007  32.0   0.6   0.4
    12:34:50.044   1484263.00   532.88  131072 100.007.5137.5130.0007.5930.009  32.0   1.8   1.5
    12:34:51.045   1494257.00   532.13  131072 100.007.5117.5110.0007.5820.007  32.0   1.6   1.1
    12:34:52.044   1504260.00   532.50  131072 100.007.5127.5120.0007.5690.008  32.0   2.0   1.5
  • Henk Vandenbergh-Oracle
    Henk Vandenbergh-Oracle Member Posts: 813
    edited Apr 7, 2016 3:38PM

    I sent you a 'friend' request allowing us to exchange email addresses and zip files.

    I recognize this pattern, it is a reporting anomaly. Add together each set of two intervals, divide by two and you'll come out with the normal average again.

    However, though I recognize the pattern there is something that does not make sense to me so before I publish 'conclusions' I would like to dig a little deeper.

    Henk.

This discussion has been closed.