3 Replies Latest reply: Jul 7, 2014 3:55 PM by e20f5b36-bf17-40b3-af2a-9bd65e6204ad RSS

    Slave termination error

    e20f5b36-bf17-40b3-af2a-9bd65e6204ad

      Hi Henk,

       

      I apologize for the lack of adequate information.

       

      The following is the message on the logfile.

       

      Jun 25, 2014  interval        i/o   MB/sec   bytes   read     resp     read    write     resp     resp queue  cpu%  cpu%

                                   rate  1024**2     i/o    pct     time     resp     resp      max   stddev depth sys+u   sys

      13:31:42.977        31     135.00    14.68  114056  47.56   24.651   51.483    0.320  371.895   42.754   3.4  94.4  10.3

      13:31:42.990        32     140.00    15.45  115694  47.57   24.563   51.288    0.314  371.895   42.357   3.5  94.4  10.3

      13:31:43.008        33     152.00    16.79  115809  47.89   24.733   51.248    0.360  371.895   42.472   3.9  94.4  10.3

      13:31:43.571 Debug_cmds(); stdout 13:31:43.570 home: /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.9.x86_64/jre

      13:31:43.572 Debug_cmds(); stdout 13:31:43.572 home: /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.9.x86_64

      13:31:43.572 Debug_cmds(); stdout 13:31:43.572 ==> where: Vdb.Jstack.main(Jstack.java:398)

      13:31:43.572 Debug_cmds(); stdout 13:31:43.572 ==> where: Vdb.Jstack.main(Jstack.java:401)

      13:31:43.572 Debug_cmds(); stdout 13:31:43.572 jstack: null

      13:31:43.572 Debug_cmds(); stdout 13:31:43.572 jstack: null

      13:31:43.573 Debug_cmds(); stdout 13:31:43.572 execute(): ps -ef

      13:31:43.609 Debug_cmds(); stdout 13:31:43.609 program: ../jre/bin/java -classpath ../jre/lib/rt.jar:../jre/lib/jsse.jar:../jre/lib/jce.jar:mail.jar:framework.jar -djava.library.path=. framework.frameworkmanager

      13:31:43.609 Debug_cmds(); stdout 13:31:43.609 program: ../jre/bin/java -classpath ../jre/lib/rt.jar:../jre/lib/jsse.jar:../jre/lib/jce.jar:mail.jar:framework.jar -djava.library.path=. framework.frameworkmanager

      13:31:43.609 Debug_cmds(); stdout 13:31:43.609 program: ../jre/bin/java -classpath .:popup.jar:../gui.jar popup.communicator ajsgyqkj=71244

      13:31:43.610 Debug_cmds(); stdout 13:31:43.610 program: java -client -xmx512m -xms64m -cp /opt/vdbench/:/opt/vdbench/classes:/opt/vdbench/vdbench.jar:/opt/vdbench/jaramiko.jar vdb.vdbmain -m 8 -f /var/log/validator-logs/vdbench-config.140625.132608 -o /var/log/validator-logs/vdbench-output.140625.132608 -e 1800

      13:31:43.611 Debug_cmds(); stdout 13:31:43.611 program: java -client -xmx1024m -xms128m -cp /opt/vdbench/:/opt/vdbench/classes:/opt/vdbench/vdbench.jar:/opt/vdbench/jaramiko.jar vdb.slavejvm slavejvm -m localhost -n localhost-11-140625-13.26.18.930 -l localhost-1 -p 5570

      13:31:43.611 Debug_cmds(); stdout 13:31:43.611 program: java -client -xmx1024m -xms128m -cp /opt/vdbench/:/opt/vdbench/classes:/opt/vdbench/vdbench.jar:/opt/vdbench/jaramiko.jar vdb.slavejvm slavejvm -m localhost -n localhost-12-140625-13.26.18.930 -l localhost-2 -p 5570

      13:31:43.611 Debug_cmds(); stdout 13:31:43.611 program: java -client -xmx1024m -xms128m -cp /opt/vdbench/:/opt/vdbench/classes:/opt/vdbench/vdbench.jar:/opt/vdbench/jaramiko.jar vdb.slavejvm slavejvm -m localhost -n localhost-10-140625-13.26.18.930 -l localhost-0 -p 5570

      13:31:43.611 Debug_cmds(); stdout 13:31:43.611 program: java -client -xmx1024m -xms128m -cp /opt/vdbench/:/opt/vdbench/classes:/opt/vdbench/vdbench.jar:/opt/vdbench/jaramiko.jar vdb.slavejvm slavejvm -m localhost -n localhost-13-140625-13.26.18.930 -l localhost-3 -p 5570

      13:31:43.611 Debug_cmds(); stdout 13:31:43.611 program: java -client -xmx1024m -xms128m -cp /opt/vdbench/:/opt/vdbench/classes:/opt/vdbench/vdbench.jar:/opt/vdbench/jaramiko.jar vdb.slavejvm slavejvm -m localhost -n localhost-14-140625-13.26.18.930 -l localhost-4 -p 5570

      13:31:43.612 Debug_cmds(); stdout 13:31:43.612 program: java -client -xmx1024m -xms128m -cp /opt/vdbench/:/opt/vdbench/classes:/opt/vdbench/vdbench.jar:/opt/vdbench/jaramiko.jar vdb.slavejvm slavejvm -m localhost -n localhost-15-140625-13.26.18.930 -l localhost-5 -p 5570

      13:31:43.612 Debug_cmds(); stdout 13:31:43.612 program: java -client -xmx1024m -xms128m -cp /opt/vdbench/:/opt/vdbench/classes:/opt/vdbench/vdbench.jar:/opt/vdbench/jaramiko.jar vdb.slavejvm slavejvm -m localhost -n localhost-16-140625-13.26.18.930 -l localhost-6 -p 5570

      13:31:43.612 Debug_cmds(); stdout 13:31:43.612 program: java -client -xmx1024m -xms128m -cp /opt/vdbench/:/opt/vdbench/classes:/opt/vdbench/vdbench.jar:/opt/vdbench/jaramiko.jar vdb.slavejvm slavejvm -m localhost -n localhost-17-140625-13.26.18.930 -l localhost-7 -p 5570

      13:31:43.612 Debug_cmds(); stdout 13:31:43.612

      13:31:43.612 Debug_cmds(); stdout 13:31:43.612 execute(): null 3268

      13:31:43.615 Debug_cmds(); stdout 13:31:43.614 Jstatus for 'other' 3268 ../jre/bin/java -classpath ../jre/lib/rt.jar:../jre/lib/jsse.jar:../jre/lib/jce.jar:mail.jar:framework.jar -djava.library.path=. framework.frameworkmanager

      13:31:43.615 Debug_cmds(); stdout 13:31:43.615 stderr: null: Command not found.

      13:31:43.615 Debug_cmds(); stdout 13:31:43.615

      13:31:43.615 Debug_cmds(); stdout 13:31:43.615 execute(): null 3523

      13:31:43.617 Debug_cmds(); stdout 13:31:43.617 Jstatus for 'other' 3523 ../jre/bin/java -classpath ../jre/lib/rt.jar:../jre/lib/jsse.jar:../jre/lib/jce.jar:mail.jar:framework.jar -djava.library.path=. framework.frameworkmanager

      13:31:43.617 Debug_cmds(); stdout 13:31:43.617 stderr: null: Command not found.

      13:31:43.617 Debug_cmds(); stdout 13:31:43.617

      13:31:43.617 Debug_cmds(); stdout 13:31:43.617 execute(): null 3527

      13:31:43.619 Debug_cmds(); stdout 13:31:43.619 Jstatus for 'other' 3527 ../jre/bin/java -classpath .:popup.jar:../gui.jar popup.communicator ajsgyqkj=71244

      13:31:43.619 Debug_cmds(); stdout 13:31:43.619 stderr: null: Command not found.

      13:31:43.619 Debug_cmds(); stdout 13:31:43.619

      13:31:43.619 Debug_cmds(); stdout 13:31:43.619 execute(): null 4278

      13:31:43.621 Debug_cmds(); stdout 13:31:43.621 Jstatus for 'other' 4278 java -client -xmx512m -xms64m -cp /opt/vdbench/:/opt/vdbench/classes:/opt/vdbench/vdbench.jar:/opt/vdbench/jaramiko.jar vdb.vdbmain -m 8 -f /var/log/validator-logs/vdbench-config.140625.132608 -o /var/log/validator-logs/vdbench-output.140625.132608 -e 1800

      13:31:43.622 Debug_cmds(); stdout 13:31:43.621 stderr: null: Command not found.

      13:31:43.622 Debug_cmds(); stdout 13:31:43.622

      13:31:43.622 Debug_cmds(); stdout 13:31:43.622 execute(): null 4412

      13:31:43.624 Debug_cmds(); stdout 13:31:43.624 Jstatus for 'other' 4412 java -client -xmx1024m -xms128m -cp /opt/vdbench/:/opt/vdbench/classes:/opt/vdbench/vdbench.jar:/opt/vdbench/jaramiko.jar vdb.slavejvm slavejvm -m localhost -n localhost-11-140625-13.26.18.930 -l localhost-1 -p 5570

      13:31:43.624 Debug_cmds(); stdout 13:31:43.624 stderr: null: Command not found.

      13:31:43.624 Debug_cmds(); stdout 13:31:43.624

      13:31:43.624 Debug_cmds(); stdout 13:31:43.624 execute(): null 4413

      13:31:43.626 Debug_cmds(); stdout 13:31:43.626 Jstatus for 'other' 4413 java -client -xmx1024m -xms128m -cp /opt/vdbench/:/opt/vdbench/classes:/opt/vdbench/vdbench.jar:/opt/vdbench/jaramiko.jar vdb.slavejvm slavejvm -m localhost -n localhost-12-140625-13.26.18.930 -l localhost-2 -p 5570

      13:31:43.627 Debug_cmds(); stdout 13:31:43.626 stderr: null: Command not found.

      13:31:43.627 Debug_cmds(); stdout 13:31:43.627

      13:31:43.627 Debug_cmds(); stdout 13:31:43.627 execute(): null 4414

      13:31:43.629 Debug_cmds(); stdout 13:31:43.629 Jstatus for 'other' 4414 java -client -xmx1024m -xms128m -cp /opt/vdbench/:/opt/vdbench/classes:/opt/vdbench/vdbench.jar:/opt/vdbench/jaramiko.jar vdb.slavejvm slavejvm -m localhost -n localhost-10-140625-13.26.18.930 -l localhost-0 -p 5570

      13:31:43.629 Debug_cmds(); stdout 13:31:43.629 stderr: null: Command not found.

      13:31:43.629 Debug_cmds(); stdout 13:31:43.629

      13:31:43.629 Debug_cmds(); stdout 13:31:43.629 execute(): null 4416

      13:31:44.634 Debug_cmds(); stdout 13:31:44.634 Jstatus for 'other' 4416 java -client -xmx1024m -xms128m -cp /opt/vdbench/:/opt/vdbench/classes:/opt/vdbench/vdbench.jar:/opt/vdbench/jaramiko.jar vdb.slavejvm slavejvm -m localhost -n localhost-13-140625-13.26.18.930 -l localhost-3 -p 5570

      13:31:44.634 Debug_cmds(); stdout 13:31:44.634 stderr: null: Command not found.

      13:31:44.634 Debug_cmds(); stdout 13:31:44.634

      13:31:44.635 Debug_cmds(); stdout 13:31:44.634 execute(): null 4418

      13:31:44.638 Debug_cmds(); stdout 13:31:44.638 Jstatus for 'other' 4418 java -client -xmx1024m -xms128m -cp /opt/vdbench/:/opt/vdbench/classes:/opt/vdbench/vdbench.jar:/opt/vdbench/jaramiko.jar vdb.slavejvm slavejvm -m localhost -n localhost-14-140625-13.26.18.930 -l localhost-4 -p 5570

      13:31:44.638 Debug_cmds(); stdout 13:31:44.638 stderr: null: Command not found.

      13:31:44.638 Debug_cmds(); stdout 13:31:44.638

      13:31:44.639 Debug_cmds(); stdout 13:31:44.638 execute(): null 4419

      13:31:44.642 Debug_cmds(); stdout 13:31:44.641 Jstatus for 'other' 4419 java -client -xmx1024m -xms128m -cp /opt/vdbench/:/opt/vdbench/classes:/opt/vdbench/vdbench.jar:/opt/vdbench/jaramiko.jar vdb.slavejvm slavejvm -m localhost -n localhost-15-140625-13.26.18.930 -l localhost-5 -p 5570

      13:31:44.642 Debug_cmds(); stdout 13:31:44.642 stderr: null: Command not found.

      13:31:44.642 Debug_cmds(); stdout 13:31:44.642

      13:31:44.642 Debug_cmds(); stdout 13:31:44.642 execute(): null 4424

      13:31:44.645 Debug_cmds(); stdout 13:31:44.645 Jstatus for 'other' 4424 java -client -xmx1024m -xms128m -cp /opt/vdbench/:/opt/vdbench/classes:/opt/vdbench/vdbench.jar:/opt/vdbench/jaramiko.jar vdb.slavejvm slavejvm -m localhost -n localhost-16-140625-13.26.18.930 -l localhost-6 -p 5570

      13:31:44.645 Debug_cmds(); stdout 13:31:44.645 stderr: null: Command not found.

      13:31:44.645 Debug_cmds(); stdout 13:31:44.645

      13:31:44.646 Debug_cmds(); stdout 13:31:44.645 execute(): null 4425

      13:31:45.649 Debug_cmds(); stdout 13:31:45.649 Jstatus for 'other' 4425 java -client -xmx1024m -xms128m -cp /opt/vdbench/:/opt/vdbench/classes:/opt/vdbench/vdbench.jar:/opt/vdbench/jaramiko.jar vdb.slavejvm slavejvm -m localhost -n localhost-17-140625-13.26.18.930 -l localhost-7 -p 5570

      13:31:45.649 Debug_cmds(); stdout 13:31:45.649 stderr: null: Command not found.

      13:31:45.649 Debug_cmds(); stdout 13:31:45.649 Jstack completed successfully

      13:31:45.655 Memory total Java heap:   69.562 MB; Free:   52.830 MB; Used:   16.732 MB;

      13:31:45.655 VdbCount statistics for Heartbeat:

      13:31:45.655 Java heap:   69.563 /   52.830 /   16.732

      13:31:45.655 Class                               added   cloned   serial   finals     left

      13:31:45.656 VdbObject.finalize() not active!

      13:31:45.656 tg: java.lang.ThreadGroup[name=main,maxpri=10]

      13:31:45.656 tg.get_name: main

      13:31:45.656 tg.activeCount: 35

      13:31:45.661 tl            [Ljava.lang.Thread;@8a7d8b0

      13:31:45.661 tl[]:         Thread[Vdbmain,5,main]

      13:31:45.661 tl[].isalive: true

      13:31:45.661 Threadname:   Vdbmain

      13:31:45.661 dumpOneStack: java.lang.Thread -2

      13:31:45.661 dumpOneStack: java.lang.Thread 338

      13:31:45.661 dumpOneStack: Vdb.common 376

      13:31:45.661 dumpOneStack: Vdb.common 365

      13:31:45.662 dumpOneStack: Vdb.Vdbmain 800

      13:31:45.662 dumpOneStack: Vdb.Vdbmain 599

      13:31:45.662 tl            [Ljava.lang.Thread;@8a7d8b0

      13:31:45.662 tl[]:         Thread[SlaveStarter localhost-10-140625-13.26.18.930,5,main]

      13:31:45.662 tl[].isalive: true

      13:31:45.662 Threadname:   SlaveStarter localhost-10-140625-13.26.18.930

      13:31:45.662 dumpOneStack: java.lang.Object -2

      13:31:45.662 dumpOneStack: java.lang.Object 503

      13:31:45.662 dumpOneStack: java.lang.UNIXProcess 210

      13:31:45.662 dumpOneStack: Utils.OS_cmd 319

      13:31:45.662 dumpOneStack: Utils.OS_cmd 250

      13:31:45.662 dumpOneStack: Vdb.SlaveStarter 220

      13:31:45.663 dumpOneStack: Vdb.SlaveStarter 169

      13:31:45.663 dumpOneStack: Vdb.SlaveStarter 68

      13:31:45.663 tl            [Ljava.lang.Thread;@8a7d8b0

      13:31:45.663 tl[]:         Thread[SlaveStarter localhost-11-140625-13.26.18.930,5,main]

      13:31:45.663 tl[].isalive: true

      13:31:45.663 Threadname:   SlaveStarter localhost-11-140625-13.26.18.930

      13:31:45.663 dumpOneStack: java.lang.Object -2

      13:31:45.663 dumpOneStack: java.lang.Object 503

      13:31:45.663 dumpOneStack: java.lang.UNIXProcess 210

      13:31:45.663 dumpOneStack: Utils.OS_cmd 319

      13:31:45.663 dumpOneStack: Utils.OS_cmd 250

      13:31:45.664 dumpOneStack: Vdb.SlaveStarter 220

      13:31:45.664 dumpOneStack: Vdb.SlaveStarter 169

      13:31:45.664 dumpOneStack: Vdb.SlaveStarter 68

      13:31:45.664 tl            [Ljava.lang.Thread;@8a7d8b0

      13:31:45.664 tl[]:         Thread[SlaveStarter localhost-12-140625-13.26.18.930,5,main]

      13:31:45.664 tl[].isalive: true

      13:31:45.664 Threadname:   SlaveStarter localhost-12-140625-13.26.18.930

      13:31:45.664 dumpOneStack: java.lang.Object -2

      13:31:45.664 dumpOneStack: java.lang.Object 503

      13:31:45.664 dumpOneStack: java.lang.UNIXProcess 210

      13:31:45.664 dumpOneStack: Utils.OS_cmd 319

      13:31:45.664 dumpOneStack: Utils.OS_cmd 250

      13:31:45.664 dumpOneStack: Vdb.SlaveStarter 220

      13:31:45.665 dumpOneStack: Vdb.SlaveStarter 169

      13:31:45.665 dumpOneStack: Vdb.SlaveStarter 68

      13:31:45.665 tl            [Ljava.lang.Thread;@8a7d8b0

      13:31:45.665 tl[]:         Thread[SlaveStarter localhost-13-140625-13.26.18.930,5,main]

      13:31:45.665 tl[].isalive: true

      13:31:45.665 Threadname:   SlaveStarter localhost-13-140625-13.26.18.930

      13:31:45.665 dumpOneStack: java.lang.Object -2

      13:31:45.665 dumpOneStack: java.lang.Object 503

      13:31:45.665 dumpOneStack: java.lang.UNIXProcess 210

      13:31:45.665 dumpOneStack: Utils.OS_cmd 319

      13:31:45.665 dumpOneStack: Utils.OS_cmd 250

      13:31:45.666 dumpOneStack: Vdb.SlaveStarter 220

      13:31:45.666 dumpOneStack: Vdb.SlaveStarter 169

      13:31:45.666 dumpOneStack: Vdb.SlaveStarter 68

      13:31:45.666 tl            [Ljava.lang.Thread;@8a7d8b0

      13:31:45.666 tl[]:         Thread[SlaveStarter localhost-14-140625-13.26.18.930,5,main]

      13:31:45.666 tl[].isalive: true

      13:31:45.666 Threadname:   SlaveStarter localhost-14-140625-13.26.18.930

      13:31:45.666 dumpOneStack: java.lang.Object -2

      13:31:45.666 dumpOneStack: java.lang.Object 503

      13:31:45.666 dumpOneStack: java.lang.UNIXProcess 210

      13:31:45.666 dumpOneStack: Utils.OS_cmd 319

      13:31:45.666 dumpOneStack: Utils.OS_cmd 250

      13:31:45.667 dumpOneStack: Vdb.SlaveStarter 220

      13:31:45.667 dumpOneStack: Vdb.SlaveStarter 169

      13:31:45.667 dumpOneStack: Vdb.SlaveStarter 68

      13:31:45.667 tl            [Ljava.lang.Thread;@8a7d8b0

      13:31:45.667 tl[]:         Thread[SlaveStarter localhost-15-140625-13.26.18.930,5,main]

      13:31:45.667 tl[].isalive: true

      13:31:45.667 Threadname:   SlaveStarter localhost-15-140625-13.26.18.930

      13:31:45.667 dumpOneStack: java.lang.Object -2

      13:31:45.667 dumpOneStack: java.lang.Object 503

      13:31:45.667 dumpOneStack: java.lang.UNIXProcess 210

      13:31:45.667 dumpOneStack: Utils.OS_cmd 319

      13:31:45.667 dumpOneStack: Utils.OS_cmd 250

      13:31:45.668 dumpOneStack: Vdb.SlaveStarter 220

      13:31:45.668 dumpOneStack: Vdb.SlaveStarter 169

      13:31:45.668 dumpOneStack: Vdb.SlaveStarter 68

      13:31:45.668 tl            [Ljava.lang.Thread;@8a7d8b0

      13:31:45.668 tl[]:         Thread[Get_cmd_stream stderr /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-11-140625-13.26.18.930 -l localhost-1 -p 5570,5,main]

      13:31:45.668 tl[].isalive: true

      13:31:45.668 Threadname:   Get_cmd_stream stderr /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-11-140625-13.26.18.930 -l localhost-1 -p 5570

      13:31:45.668 dumpOneStack: java.io.FileInputStream -2

      13:31:45.668 dumpOneStack: java.io.FileInputStream 242

      13:31:45.668 dumpOneStack: java.io.BufferedInputStream 273

      13:31:45.668 dumpOneStack: java.io.BufferedInputStream 334

      13:31:45.669 dumpOneStack: sun.nio.cs.StreamDecoder 283

      13:31:45.669 dumpOneStack: sun.nio.cs.StreamDecoder 325

      13:31:45.669 dumpOneStack: sun.nio.cs.StreamDecoder 177

      13:31:45.669 dumpOneStack: java.io.InputStreamReader 184

      13:31:45.669 dumpOneStack: java.io.BufferedReader 154

      13:31:45.669 dumpOneStack: java.io.BufferedReader 317

      13:31:45.669 dumpOneStack: java.io.BufferedReader 382

      13:31:45.669 dumpOneStack: Utils.Get_cmd_stream 679

      13:31:45.669 tl            [Ljava.lang.Thread;@8a7d8b0

      13:31:45.669 tl[]:         Thread[Get_cmd_stream stdout /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-11-140625-13.26.18.930 -l localhost-1 -p 5570,5,main]

      13:31:45.669 tl[].isalive: true

      13:31:45.669 Threadname:   Get_cmd_stream stdout /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-11-140625-13.26.18.930 -l localhost-1 -p 5570

      13:31:45.669 dumpOneStack: java.io.FileInputStream -2

      13:31:45.670 dumpOneStack: java.io.FileInputStream 242

      13:31:45.670 dumpOneStack: java.io.BufferedInputStream 273

      13:31:45.670 dumpOneStack: java.io.BufferedInputStream 334

      13:31:45.670 dumpOneStack: sun.nio.cs.StreamDecoder 283

      13:31:45.670 dumpOneStack: sun.nio.cs.StreamDecoder 325

      13:31:45.670 dumpOneStack: sun.nio.cs.StreamDecoder 177

      13:31:45.670 dumpOneStack: java.io.InputStreamReader 184

      13:31:45.670 dumpOneStack: java.io.BufferedReader 154

      13:31:45.670 dumpOneStack: java.io.BufferedReader 317

      13:31:45.670 dumpOneStack: java.io.BufferedReader 382

      13:31:45.670 dumpOneStack: Utils.Get_cmd_stream 679

      13:31:45.670 tl            [Ljava.lang.Thread;@8a7d8b0

      13:31:45.670 tl[]:         Thread[SlaveStarter localhost-16-140625-13.26.18.930,5,main]

      13:31:45.670 tl[].isalive: true

      13:31:45.671 Threadname:   SlaveStarter localhost-16-140625-13.26.18.930

      13:31:45.671 dumpOneStack: java.lang.Object -2

      13:31:45.671 dumpOneStack: java.lang.Object 503

      13:31:45.671 dumpOneStack: java.lang.UNIXProcess 210

      13:31:45.671 dumpOneStack: Utils.OS_cmd 319

      13:31:45.671 dumpOneStack: Utils.OS_cmd 250

      13:31:45.671 dumpOneStack: Vdb.SlaveStarter 220

      13:31:45.671 dumpOneStack: Vdb.SlaveStarter 169

      13:31:45.671 dumpOneStack: Vdb.SlaveStarter 68

      13:31:45.671 tl            [Ljava.lang.Thread;@8a7d8b0

      13:31:45.671 tl[]:         Thread[Get_cmd_stream stderr /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-12-140625-13.26.18.930 -l localhost-2 -p 5570,5,main]

      13:31:45.671 tl[].isalive: true

      13:31:45.672 Threadname:   Get_cmd_stream stderr /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-12-140625-13.26.18.930 -l localhost-2 -p 5570

      13:31:45.672 dumpOneStack: java.io.FileInputStream -2

      13:31:45.672 dumpOneStack: java.io.FileInputStream 242

      13:31:45.672 dumpOneStack: java.io.BufferedInputStream 273

      13:31:45.672 dumpOneStack: java.io.BufferedInputStream 334

      13:31:45.672 dumpOneStack: sun.nio.cs.StreamDecoder 283

      13:31:45.672 dumpOneStack: sun.nio.cs.StreamDecoder 325

      13:31:45.672 dumpOneStack: sun.nio.cs.StreamDecoder 177

      13:31:45.672 dumpOneStack: java.io.InputStreamReader 184

      13:31:45.672 dumpOneStack: java.io.BufferedReader 154

      13:31:45.672 dumpOneStack: java.io.BufferedReader 317

      13:31:45.672 dumpOneStack: java.io.BufferedReader 382

      13:31:45.673 dumpOneStack: Utils.Get_cmd_stream 679

      13:31:45.673 tl            [Ljava.lang.Thread;@8a7d8b0

      13:31:45.673 tl[]:         Thread[Get_cmd_stream stdout /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-12-140625-13.26.18.930 -l localhost-2 -p 5570,5,main]

      13:31:45.673 tl[].isalive: true

      13:31:45.673 Threadname:   Get_cmd_stream stdout /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-12-140625-13.26.18.930 -l localhost-2 -p 5570

      13:31:45.673 dumpOneStack: java.io.FileInputStream -2

      13:31:45.673 dumpOneStack: java.io.FileInputStream 242

      13:31:45.673 dumpOneStack: java.io.BufferedInputStream 273

      13:31:45.673 dumpOneStack: java.io.BufferedInputStream 334

      13:31:45.673 dumpOneStack: sun.nio.cs.StreamDecoder 283

      13:31:45.673 dumpOneStack: sun.nio.cs.StreamDecoder 325

      13:31:45.673 dumpOneStack: sun.nio.cs.StreamDecoder 177

      13:31:45.673 dumpOneStack: java.io.InputStreamReader 184

      13:31:45.673 dumpOneStack: java.io.BufferedReader 154

      13:31:45.674 dumpOneStack: java.io.BufferedReader 317

      13:31:45.674 dumpOneStack: java.io.BufferedReader 382

      13:31:45.674 dumpOneStack: Utils.Get_cmd_stream 679

      13:31:45.674 tl            [Ljava.lang.Thread;@8a7d8b0

      13:31:45.674 tl[]:         Thread[Get_cmd_stream stderr /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-14-140625-13.26.18.930 -l localhost-4 -p 5570,5,main]

      13:31:45.674 tl[].isalive: true

      13:31:45.674 Threadname:   Get_cmd_stream stderr /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-14-140625-13.26.18.930 -l localhost-4 -p 5570

      13:31:45.674 dumpOneStack: java.io.FileInputStream -2

      13:31:45.674 dumpOneStack: java.io.FileInputStream 242

      13:31:45.674 dumpOneStack: java.io.BufferedInputStream 273

      13:31:45.674 dumpOneStack: java.io.BufferedInputStream 334

      13:31:45.675 dumpOneStack: sun.nio.cs.StreamDecoder 283

      13:31:45.675 dumpOneStack: sun.nio.cs.StreamDecoder 325

      13:31:45.675 dumpOneStack: sun.nio.cs.StreamDecoder 177

      13:31:45.675 dumpOneStack: java.io.InputStreamReader 184

      13:31:45.675 dumpOneStack: java.io.BufferedReader 154

      13:31:45.675 dumpOneStack: java.io.BufferedReader 317

      13:31:45.675 dumpOneStack: java.io.BufferedReader 382

      13:31:45.675 dumpOneStack: Utils.Get_cmd_stream 679

      13:31:45.675 tl            [Ljava.lang.Thread;@8a7d8b0

      13:31:45.675 tl[]:         Thread[Get_cmd_stream stdout /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-14-140625-13.26.18.930 -l localhost-4 -p 5570,5,main]

      13:31:45.675 tl[].isalive: true

      13:31:45.675 Threadname:   Get_cmd_stream stdout /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-14-140625-13.26.18.930 -l localhost-4 -p 5570

      13:31:45.675 dumpOneStack: java.io.FileInputStream -2

      13:31:45.675 dumpOneStack: java.io.FileInputStream 242

      13:31:45.675 dumpOneStack: java.io.BufferedInputStream 273

      13:31:45.675 dumpOneStack: java.io.BufferedInputStream 334

      13:31:45.675 dumpOneStack: sun.nio.cs.StreamDecoder 283

      13:31:45.676 dumpOneStack: sun.nio.cs.StreamDecoder 325

      13:31:45.676 dumpOneStack: sun.nio.cs.StreamDecoder 177

      13:31:45.676 dumpOneStack: java.io.InputStreamReader 184

      13:31:45.676 dumpOneStack: java.io.BufferedReader 154

      13:31:45.676 dumpOneStack: java.io.BufferedReader 317

      13:31:45.676 dumpOneStack: java.io.BufferedReader 382

      13:31:45.676 dumpOneStack: Utils.Get_cmd_stream 679

      13:31:45.676 tl            [Ljava.lang.Thread;@8a7d8b0

      13:31:45.676 tl[]:         Thread[SlaveStarter localhost-17-140625-13.26.18.930,5,main]

      13:31:45.676 tl[].isalive: true

      13:31:45.676 Threadname:   SlaveStarter localhost-17-140625-13.26.18.930

      13:31:45.676 dumpOneStack: java.lang.Object -2

      13:31:45.676 dumpOneStack: java.lang.Object 503

      13:31:45.676 dumpOneStack: java.lang.UNIXProcess 210

      13:31:45.676 dumpOneStack: Utils.OS_cmd 319

      13:31:45.676 dumpOneStack: Utils.OS_cmd 250

      13:31:45.676 dumpOneStack: Vdb.SlaveStarter 220

      13:31:45.676 dumpOneStack: Vdb.SlaveStarter 169

      13:31:45.677 dumpOneStack: Vdb.SlaveStarter 68

      13:31:45.677 tl            [Ljava.lang.Thread;@8a7d8b0

      13:31:45.677 tl[]:         Thread[Get_cmd_stream stderr /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-13-140625-13.26.18.930 -l localhost-3 -p 5570,5,main]

      13:31:45.677 tl[].isalive: true

      13:31:45.677 Threadname:   Get_cmd_stream stderr /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-13-140625-13.26.18.930 -l localhost-3 -p 5570

      13:31:45.677 dumpOneStack: java.io.FileInputStream -2

      13:31:45.677 dumpOneStack: java.io.FileInputStream 242

      13:31:45.677 dumpOneStack: java.io.BufferedInputStream 273

      13:31:45.677 dumpOneStack: java.io.BufferedInputStream 334

      13:31:45.677 dumpOneStack: sun.nio.cs.StreamDecoder 283

      13:31:45.677 dumpOneStack: sun.nio.cs.StreamDecoder 325

      13:31:45.677 dumpOneStack: sun.nio.cs.StreamDecoder 177

      13:31:45.677 dumpOneStack: java.io.InputStreamReader 184

      13:31:45.677 dumpOneStack: java.io.BufferedReader 154

      13:31:45.677 dumpOneStack: java.io.BufferedReader 317

      13:31:45.678 dumpOneStack: java.io.BufferedReader 382

      13:31:45.678 dumpOneStack: Utils.Get_cmd_stream 679

      13:31:45.678 tl            [Ljava.lang.Thread;@8a7d8b0

      13:31:45.678 tl[]:         Thread[Get_cmd_stream stderr /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-15-140625-13.26.18.930 -l localhost-5 -p 5570,5,main]

      13:31:45.678 tl[].isalive: true

      13:31:45.678 Threadname:   Get_cmd_stream stderr /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-15-140625-13.26.18.930 -l localhost-5 -p 5570

      13:31:45.678 dumpOneStack: java.io.FileInputStream -2

      13:31:45.678 dumpOneStack: java.io.FileInputStream 242

      13:31:45.678 dumpOneStack: java.io.BufferedInputStream 273

      13:31:45.678 dumpOneStack: java.io.BufferedInputStream 334

      13:31:45.678 dumpOneStack: sun.nio.cs.StreamDecoder 283

      13:31:45.678 dumpOneStack: sun.nio.cs.StreamDecoder 325

      13:31:45.678 dumpOneStack: sun.nio.cs.StreamDecoder 177

      13:31:45.678 dumpOneStack: java.io.InputStreamReader 184

      13:31:45.678 dumpOneStack: java.io.BufferedReader 154

      13:31:45.678 dumpOneStack: java.io.BufferedReader 317

      13:31:45.679 dumpOneStack: java.io.BufferedReader 382

      13:31:45.679 dumpOneStack: Utils.Get_cmd_stream 679

      13:31:45.679 tl            [Ljava.lang.Thread;@8a7d8b0

      13:31:45.679 tl[]:         Thread[Get_cmd_stream stdout /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-15-140625-13.26.18.930 -l localhost-5 -p 5570,5,main]

      13:31:45.679 tl[].isalive: true

      13:31:45.679 Threadname:   Get_cmd_stream stdout /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-15-140625-13.26.18.930 -l localhost-5 -p 5570

      13:31:45.679 dumpOneStack: java.io.FileInputStream -2

      13:31:45.679 dumpOneStack: java.io.FileInputStream 242

      13:31:45.679 dumpOneStack: java.io.BufferedInputStream 273

      13:31:45.679 dumpOneStack: java.io.BufferedInputStream 334

      13:31:45.679 dumpOneStack: sun.nio.cs.StreamDecoder 283

      13:31:45.679 dumpOneStack: sun.nio.cs.StreamDecoder 325

      13:31:45.679 dumpOneStack: sun.nio.cs.StreamDecoder 177

      13:31:45.679 dumpOneStack: java.io.InputStreamReader 184

      13:31:45.679 dumpOneStack: java.io.BufferedReader 154

      13:31:45.679 dumpOneStack: java.io.BufferedReader 317

      13:31:45.680 dumpOneStack: java.io.BufferedReader 382

      13:31:45.680 dumpOneStack: Utils.Get_cmd_stream 679

      13:31:45.680 tl            [Ljava.lang.Thread;@8a7d8b0

      13:31:45.680 tl[]:         Thread[Get_cmd_stream stdout /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-13-140625-13.26.18.930 -l localhost-3 -p 5570,5,main]

      13:31:45.680 tl[].isalive: true

      13:31:45.680 Threadname:   Get_cmd_stream stdout /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-13-140625-13.26.18.930 -l localhost-3 -p 5570

      13:31:45.680 dumpOneStack: java.io.FileInputStream -2

      13:31:45.680 dumpOneStack: java.io.FileInputStream 242

      13:31:45.680 dumpOneStack: java.io.BufferedInputStream 273

      13:31:45.680 dumpOneStack: java.io.BufferedInputStream 334

      13:31:45.680 dumpOneStack: sun.nio.cs.StreamDecoder 283

      13:31:45.680 dumpOneStack: sun.nio.cs.StreamDecoder 325

      13:31:45.680 dumpOneStack: sun.nio.cs.StreamDecoder 177

      13:31:45.680 dumpOneStack: java.io.InputStreamReader 184

      13:31:45.680 dumpOneStack: java.io.BufferedReader 154

      13:31:45.680 dumpOneStack: java.io.BufferedReader 317

      13:31:45.680 dumpOneStack: java.io.BufferedReader 382

      13:31:45.681 dumpOneStack: Utils.Get_cmd_stream 679

      13:31:45.681 tl            [Ljava.lang.Thread;@8a7d8b0

      13:31:45.681 tl[]:         Thread[Get_cmd_stream stderr /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-17-140625-13.26.18.930 -l localhost-7 -p 5570,5,main]

      13:31:45.681 tl[].isalive: true

      13:31:45.681 Threadname:   Get_cmd_stream stderr /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-17-140625-13.26.18.930 -l localhost-7 -p 5570

      13:31:45.681 dumpOneStack: java.io.FileInputStream -2

      13:31:45.681 dumpOneStack: java.io.FileInputStream 242

      13:31:45.681 dumpOneStack: java.io.BufferedInputStream 273

      13:31:45.681 dumpOneStack: java.io.BufferedInputStream 334

      13:31:45.681 dumpOneStack: sun.nio.cs.StreamDecoder 283

      13:31:45.681 dumpOneStack: sun.nio.cs.StreamDecoder 325

      13:31:45.681 dumpOneStack: sun.nio.cs.StreamDecoder 177

      13:31:45.681 dumpOneStack: java.io.InputStreamReader 184

      13:31:45.681 dumpOneStack: java.io.BufferedReader 154

      13:31:45.681 dumpOneStack: java.io.BufferedReader 317

      13:31:45.681 dumpOneStack: java.io.BufferedReader 382

      13:31:45.681 dumpOneStack: Utils.Get_cmd_stream 679

      13:31:45.682 tl            [Ljava.lang.Thread;@8a7d8b0

      13:31:45.682 tl[]:         Thread[Get_cmd_stream stderr /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-16-140625-13.26.18.930 -l localhost-6 -p 5570,5,main]

      13:31:45.682 tl[].isalive: true

      13:31:45.682 Threadname:   Get_cmd_stream stderr /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-16-140625-13.26.18.930 -l localhost-6 -p 5570

      13:31:45.682 dumpOneStack: java.io.FileInputStream -2

      13:31:45.682 dumpOneStack: java.io.FileInputStream 242

      13:31:45.682 dumpOneStack: java.io.BufferedInputStream 273

      13:31:45.682 dumpOneStack: java.io.BufferedInputStream 334

      13:31:45.682 dumpOneStack: sun.nio.cs.StreamDecoder 283

      13:31:45.682 dumpOneStack: sun.nio.cs.StreamDecoder 325

      13:31:45.682 dumpOneStack: sun.nio.cs.StreamDecoder 177

      13:31:45.682 dumpOneStack: java.io.InputStreamReader 184

      13:31:45.682 dumpOneStack: java.io.BufferedReader 154

      13:31:45.682 dumpOneStack: java.io.BufferedReader 317

      13:31:45.682 dumpOneStack: java.io.BufferedReader 382

      13:31:45.682 dumpOneStack: Utils.Get_cmd_stream 679

      13:31:45.683 tl            [Ljava.lang.Thread;@8a7d8b0

      13:31:45.683 tl[]:         Thread[Get_cmd_stream stdout /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-17-140625-13.26.18.930 -l localhost-7 -p 5570,5,main]

      13:31:45.683 tl[].isalive: true

      13:31:45.683 Threadname:   Get_cmd_stream stdout /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-17-140625-13.26.18.930 -l localhost-7 -p 5570

      13:31:45.683 dumpOneStack: java.io.FileInputStream -2

      13:31:45.683 dumpOneStack: java.io.FileInputStream 242

      13:31:45.683 dumpOneStack: java.io.BufferedInputStream 273

      13:31:45.683 dumpOneStack: java.io.BufferedInputStream 334

      13:31:45.683 dumpOneStack: sun.nio.cs.StreamDecoder 283

      13:31:45.683 dumpOneStack: sun.nio.cs.StreamDecoder 325

      13:31:45.683 dumpOneStack: sun.nio.cs.StreamDecoder 177

      13:31:45.683 dumpOneStack: java.io.InputStreamReader 184

      13:31:45.683 dumpOneStack: java.io.BufferedReader 154

      13:31:45.683 dumpOneStack: java.io.BufferedReader 317

      13:31:45.683 dumpOneStack: java.io.BufferedReader 382

      13:31:45.683 dumpOneStack: Utils.Get_cmd_stream 679

      13:31:45.684 tl            [Ljava.lang.Thread;@8a7d8b0

      13:31:45.684 tl[]:         Thread[Get_cmd_stream stderr /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-10-140625-13.26.18.930 -l localhost-0 -p 5570,5,main]

      13:31:45.684 tl[].isalive: true

      13:31:45.684 Threadname:   Get_cmd_stream stderr /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-10-140625-13.26.18.930 -l localhost-0 -p 5570

      13:31:45.684 dumpOneStack: java.io.FileInputStream -2

      13:31:45.684 dumpOneStack: java.io.FileInputStream 242

      13:31:45.684 dumpOneStack: java.io.BufferedInputStream 273

      13:31:45.684 dumpOneStack: java.io.BufferedInputStream 334

      13:31:45.684 dumpOneStack: sun.nio.cs.StreamDecoder 283

      13:31:45.684 dumpOneStack: sun.nio.cs.StreamDecoder 325

      13:31:45.684 dumpOneStack: sun.nio.cs.StreamDecoder 177

      13:31:45.684 dumpOneStack: java.io.InputStreamReader 184

      13:31:45.684 dumpOneStack: java.io.BufferedReader 154

      13:31:45.684 dumpOneStack: java.io.BufferedReader 317

      13:31:45.684 dumpOneStack: java.io.BufferedReader 382

      13:31:45.684 dumpOneStack: Utils.Get_cmd_stream 679

      13:31:45.685 tl            [Ljava.lang.Thread;@8a7d8b0

      13:31:45.685 tl[]:         Thread[Get_cmd_stream stdout /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-10-140625-13.26.18.930 -l localhost-0 -p 5570,5,main]

      13:31:45.685 tl[].isalive: true

      13:31:45.685 Threadname:   Get_cmd_stream stdout /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-10-140625-13.26.18.930 -l localhost-0 -p 5570

      13:31:45.685 dumpOneStack: java.io.FileInputStream -2

      13:31:45.685 dumpOneStack: java.io.FileInputStream 242

      13:31:45.685 dumpOneStack: java.io.BufferedInputStream 273

      13:31:45.685 dumpOneStack: java.io.BufferedInputStream 334

      13:31:45.685 dumpOneStack: sun.nio.cs.StreamDecoder 283

      13:31:45.685 dumpOneStack: sun.nio.cs.StreamDecoder 325

      13:31:45.685 dumpOneStack: sun.nio.cs.StreamDecoder 177

      13:31:45.685 dumpOneStack: java.io.InputStreamReader 184

      13:31:45.685 dumpOneStack: java.io.BufferedReader 154

      13:31:45.685 dumpOneStack: java.io.BufferedReader 317

      13:31:45.685 dumpOneStack: java.io.BufferedReader 382

      13:31:45.686 dumpOneStack: Utils.Get_cmd_stream 679

      13:31:45.686 tl            [Ljava.lang.Thread;@8a7d8b0

      13:31:45.686 tl[]:         Thread[Get_cmd_stream stdout /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-16-140625-13.26.18.930 -l localhost-6 -p 5570,5,main]

      13:31:45.686 tl[].isalive: true

      13:31:45.686 Threadname:   Get_cmd_stream stdout /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-16-140625-13.26.18.930 -l localhost-6 -p 5570

      13:31:45.686 dumpOneStack: java.io.FileInputStream -2

      13:31:45.686 dumpOneStack: java.io.FileInputStream 242

      13:31:45.686 dumpOneStack: java.io.BufferedInputStream 273

      13:31:45.686 dumpOneStack: java.io.BufferedInputStream 334

      13:31:45.686 dumpOneStack: sun.nio.cs.StreamDecoder 283

      13:31:45.686 dumpOneStack: sun.nio.cs.StreamDecoder 325

      13:31:45.686 dumpOneStack: sun.nio.cs.StreamDecoder 177

      13:31:45.686 dumpOneStack: java.io.InputStreamReader 184

      13:31:45.686 dumpOneStack: java.io.BufferedReader 154

      13:31:45.686 dumpOneStack: java.io.BufferedReader 317

      13:31:45.686 dumpOneStack: java.io.BufferedReader 382

      13:31:45.687 dumpOneStack: Utils.Get_cmd_stream 679

      13:31:45.687 tl            [Ljava.lang.Thread;@8a7d8b0

      13:31:45.687 tl[]:         Thread[SlaveOnMaster,10,main]

      13:31:45.687 tl[].isalive: true

      13:31:45.687 Threadname:   SlaveOnMaster

      13:31:45.687 dumpOneStack: java.net.SocketInputStream -2

      13:31:45.687 dumpOneStack: java.net.SocketInputStream 150

      13:31:45.687 dumpOneStack: java.net.SocketInputStream 121

      13:31:45.687 dumpOneStack: java.net.SocketInputStream 203

      13:31:45.687 dumpOneStack: java.io.ObjectInputStream$PeekInputStream 2266

      13:31:45.687 dumpOneStack: java.io.ObjectInputStream$BlockDataInputStream 2559

      13:31:45.687 dumpOneStack: java.io.ObjectInputStream$BlockDataInputStream 2569

      13:31:45.687 dumpOneStack: java.io.ObjectInputStream 1315

      13:31:45.687 dumpOneStack: java.io.ObjectInputStream 369

      13:31:45.687 dumpOneStack: Vdb.SlaveSocket 148

      13:31:45.688 dumpOneStack: Vdb.SlaveOnMaster 118

      13:31:45.688 dumpOneStack: Vdb.SlaveOnMaster 60

      13:31:45.688 tl            [Ljava.lang.Thread;@8a7d8b0

      13:31:45.688 tl[]:         Thread[SlaveOnMaster,10,main]

      13:31:45.688 tl[].isalive: true

      13:31:45.688 Threadname:   SlaveOnMaster

      13:31:45.688 dumpOneStack: java.net.SocketInputStream -2

      13:31:45.688 dumpOneStack: java.net.SocketInputStream 150

      13:31:45.688 dumpOneStack: java.net.SocketInputStream 121

      13:31:45.688 dumpOneStack: java.net.SocketInputStream 203

      13:31:45.688 dumpOneStack: java.io.ObjectInputStream$PeekInputStream 2266

      13:31:45.688 dumpOneStack: java.io.ObjectInputStream$BlockDataInputStream 2559

      13:31:45.688 dumpOneStack: java.io.ObjectInputStream$BlockDataInputStream 2569

      13:31:45.688 dumpOneStack: java.io.ObjectInputStream 1315

      13:31:45.688 dumpOneStack: java.io.ObjectInputStream 369

      13:31:45.688 dumpOneStack: Vdb.SlaveSocket 148

      13:31:45.689 dumpOneStack: Vdb.SlaveOnMaster 118

      13:31:45.689 dumpOneStack: Vdb.SlaveOnMaster 60

      13:31:45.689 tl            [Ljava.lang.Thread;@8a7d8b0

      13:31:45.689 tl[]:         Thread[SlaveOnMaster,10,main]

      13:31:45.689 tl[].isalive: true

      13:31:45.689 Threadname:   SlaveOnMaster

      13:31:45.689 dumpOneStack: java.net.SocketInputStream -2

      13:31:45.689 dumpOneStack: java.net.SocketInputStream 150

      13:31:45.689 dumpOneStack: java.net.SocketInputStream 121

      13:31:45.689 dumpOneStack: java.net.SocketInputStream 203

      13:31:45.689 dumpOneStack: java.io.ObjectInputStream$PeekInputStream 2266

      13:31:45.689 dumpOneStack: java.io.ObjectInputStream$BlockDataInputStream 2559

      13:31:45.689 dumpOneStack: java.io.ObjectInputStream$BlockDataInputStream 2569

      13:31:45.689 dumpOneStack: java.io.ObjectInputStream 1315

      13:31:45.689 dumpOneStack: java.io.ObjectInputStream 369

      13:31:45.689 dumpOneStack: Vdb.SlaveSocket 148

      13:31:45.690 dumpOneStack: Vdb.SlaveOnMaster 118

      13:31:45.690 dumpOneStack: Vdb.SlaveOnMaster 60

      13:31:45.690 tl            [Ljava.lang.Thread;@8a7d8b0

      13:31:45.690 tl[]:         Thread[SlaveOnMaster,10,main]

      13:31:45.690 tl[].isalive: true

      13:31:45.690 Threadname:   SlaveOnMaster

      13:31:45.690 dumpOneStack: java.net.SocketInputStream -2

      13:31:45.690 dumpOneStack: java.net.SocketInputStream 150

      13:31:45.690 dumpOneStack: java.net.SocketInputStream 121

      13:31:45.690 dumpOneStack: java.net.SocketInputStream 203

      13:31:45.690 dumpOneStack: java.io.ObjectInputStream$PeekInputStream 2266

      13:31:45.690 dumpOneStack: java.io.ObjectInputStream$BlockDataInputStream 2559

      13:31:45.690 dumpOneStack: java.io.ObjectInputStream$BlockDataInputStream 2569

      13:31:45.690 dumpOneStack: java.io.ObjectInputStream 1315

      13:31:45.690 dumpOneStack: java.io.ObjectInputStream 369

      13:31:45.690 dumpOneStack: Vdb.SlaveSocket 148

      13:31:45.691 dumpOneStack: Vdb.SlaveOnMaster 118

      13:31:45.691 dumpOneStack: Vdb.SlaveOnMaster 60

      13:31:45.691 tl            [Ljava.lang.Thread;@8a7d8b0

      13:31:45.691 tl[]:         Thread[SlaveOnMaster,10,main]

      13:31:45.691 tl[].isalive: true

      13:31:45.691 Threadname:   SlaveOnMaster

      13:31:45.691 dumpOneStack: java.net.SocketInputStream -2

      13:31:45.691 dumpOneStack: java.net.SocketInputStream 150

      13:31:45.691 dumpOneStack: java.net.SocketInputStream 121

      13:31:45.691 dumpOneStack: java.net.SocketInputStream 203

      13:31:45.691 dumpOneStack: java.io.ObjectInputStream$PeekInputStream 2266

      13:31:45.691 dumpOneStack: java.io.ObjectInputStream$BlockDataInputStream 2559

      13:31:45.691 dumpOneStack: java.io.ObjectInputStream$BlockDataInputStream 2569

      13:31:45.691 dumpOneStack: java.io.ObjectInputStream 1315

      13:31:45.691 dumpOneStack: java.io.ObjectInputStream 369

      13:31:45.692 dumpOneStack: Vdb.SlaveSocket 148

      13:31:45.692 dumpOneStack: Vdb.SlaveOnMaster 118

      13:31:45.692 dumpOneStack: Vdb.SlaveOnMaster 60

      13:31:45.692 tl            [Ljava.lang.Thread;@8a7d8b0

      13:31:45.692 tl[]:         Thread[SlaveOnMaster,10,main]

      13:31:45.692 tl[].isalive: true

      13:31:45.692 Threadname:   SlaveOnMaster

      13:31:45.692 dumpOneStack: java.net.SocketInputStream -2

      13:31:45.692 dumpOneStack: java.net.SocketInputStream 150

      13:31:45.692 dumpOneStack: java.net.SocketInputStream 121

      13:31:45.692 dumpOneStack: java.net.SocketInputStream 203

      13:31:45.692 dumpOneStack: java.io.ObjectInputStream$PeekInputStream 2266

      13:31:45.692 dumpOneStack: java.io.ObjectInputStream$BlockDataInputStream 2559

      13:31:45.692 dumpOneStack: java.io.ObjectInputStream$BlockDataInputStream 2569

      13:31:45.692 dumpOneStack: java.io.ObjectInputStream 1315

      13:31:45.693 dumpOneStack: java.io.ObjectInputStream 369

      13:31:45.693 dumpOneStack: Vdb.SlaveSocket 148

      13:31:45.693 dumpOneStack: Vdb.SlaveOnMaster 118

      13:31:45.693 dumpOneStack: Vdb.SlaveOnMaster 60

      13:31:45.693 tl            [Ljava.lang.Thread;@8a7d8b0

      13:31:45.693 tl[]:         Thread[SlaveOnMaster,10,main]

      13:31:45.693 tl[].isalive: true

      13:31:45.693 Threadname:   SlaveOnMaster

      13:31:45.693 dumpOneStack: java.net.SocketInputStream -2

      13:31:45.693 dumpOneStack: java.net.SocketInputStream 150

      13:31:45.693 dumpOneStack: java.net.SocketInputStream 121

      13:31:45.693 dumpOneStack: java.net.SocketInputStream 203

      13:31:45.693 dumpOneStack: java.io.ObjectInputStream$PeekInputStream 2266

      13:31:45.693 dumpOneStack: java.io.ObjectInputStream$BlockDataInputStream 2559

      13:31:45.693 dumpOneStack: java.io.ObjectInputStream$BlockDataInputStream 2569

      13:31:45.693 dumpOneStack: java.io.ObjectInputStream 1315

      13:31:45.694 dumpOneStack: java.io.ObjectInputStream 369

      13:31:45.694 dumpOneStack: Vdb.SlaveSocket 148

      13:31:45.694 dumpOneStack: Vdb.SlaveOnMaster 118

      13:31:45.694 dumpOneStack: Vdb.SlaveOnMaster 60

      13:31:45.694 tl            [Ljava.lang.Thread;@8a7d8b0

      13:31:45.694 tl[]:         Thread[SlaveOnMaster,10,main]

      13:31:45.694 tl[].isalive: true

      13:31:45.694 Threadname:   SlaveOnMaster

      13:31:45.694 dumpOneStack: java.net.SocketInputStream -2

      13:31:45.694 dumpOneStack: java.net.SocketInputStream 150

      13:31:45.694 dumpOneStack: java.net.SocketInputStream 121

      13:31:45.694 dumpOneStack: java.net.SocketInputStream 203

      13:31:45.694 dumpOneStack: java.io.ObjectInputStream$PeekInputStream 2266

      13:31:45.694 dumpOneStack: java.io.ObjectInputStream$BlockDataInputStream 2559

      13:31:45.694 dumpOneStack: java.io.ObjectInputStream$BlockDataInputStream 2569

      13:31:45.695 dumpOneStack: java.io.ObjectInputStream 1315

      13:31:45.695 dumpOneStack: java.io.ObjectInputStream 369

      13:31:45.695 dumpOneStack: Vdb.SlaveSocket 148

      13:31:45.695 dumpOneStack: Vdb.SlaveOnMaster 118

      13:31:45.695 dumpOneStack: Vdb.SlaveOnMaster 60

      13:31:45.695 tl            [Ljava.lang.Thread;@8a7d8b0

      13:31:45.695 tl[]:         Thread[Slave HeartBeat,5,main]

      13:31:45.695 tl[].isalive: true

      13:31:45.695 Threadname:   Slave HeartBeat

      13:31:45.695 dumpOneStack: java.lang.Thread 1567

      13:31:45.695 dumpOneStack: Vdb.common 1523

      13:31:45.695 dumpOneStack: Vdb.common 1517

      13:31:45.695 dumpOneStack: Vdb.HeartBeat 97

      13:31:45.695 tl            [Ljava.lang.Thread;@8a7d8b0

      13:31:45.695 tl[]:         Thread[Reporter,10,main]

      13:31:45.695 tl[].isalive: true

      13:31:45.695 Threadname:   Reporter

      13:31:45.696 dumpOneStack: java.lang.Thread -2

      13:31:45.696 dumpOneStack: java.lang.Thread 338

      13:31:45.696 dumpOneStack: Vdb.common 376

      13:31:45.696 dumpOneStack: Vdb.common 365

      13:31:45.696 dumpOneStack: Vdb.common 668

      13:31:45.696 dumpOneStack: Vdb.Reporter 122

      13:31:45.696 killCommand: /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-11-140625-13.26.18.930 -l localhost-1 -p 5570

      13:31:45.696 killCommand: /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-10-140625-13.26.18.930 -l localhost-0 -p 5570

      13:31:45.696 killCommand: /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-14-140625-13.26.18.930 -l localhost-4 -p 5570

      13:31:45.696 killCommand: /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-16-140625-13.26.18.930 -l localhost-6 -p 5570

      13:31:45.696

      13:31:45.696 Heartbeat monitor: One or more slaves did not respond

      13:31:45.697

      java.lang.RuntimeException: Heartbeat monitor: One or more slaves did not respond

        at Vdb.common.failure(common.java:307)

        at Vdb.HeartBeat.run(HeartBeat.java:98)

       

      The vdbench parameter file used is as follows:

       

      sd=sd1,lun=/dev/sdl,openflags=o_direct

      sd=sd2,lun=/dev/sdj,openflags=o_direct

      sd=sd3,lun=/dev/sdk,openflags=o_direct

      sd=sd4,lun=/dev/sdh,openflags=o_direct

      sd=sd5,lun=/dev/sdi,openflags=o_direct

      sd=sd6,lun=/dev/sdf,openflags=o_direct

      sd=sd7,lun=/dev/sdg,openflags=o_direct

      sd=sd8,lun=/dev/sdd,openflags=o_direct

      sd=sd9,lun=/dev/sde,openflags=o_direct

      sd=sd10,lun=/dev/sdb,openflags=o_direct

      sd=sd11,lun=/dev/sdc,openflags=o_direct

       

       

      wd=wd1,sd=sd1,xfersize=(16k,25,32k,25,128k,25,256k,25),seekpct=random

      wd=wd2,sd=sd2,xfersize=(16k,25,32k,25,128k,25,256k,25),seekpct=random

      wd=wd3,sd=sd3,xfersize=(16k,25,32k,25,128k,25,256k,25),seekpct=random

      wd=wd4,sd=sd4,xfersize=(16k,25,32k,25,128k,25,256k,25),seekpct=random

      wd=wd5,sd=sd5,xfersize=(16k,25,32k,25,128k,25,256k,25),seekpct=random

      wd=wd6,sd=sd6,xfersize=(16k,25,32k,25,128k,25,256k,25),seekpct=random

      wd=wd7,sd=sd7,xfersize=(16k,25,32k,25,128k,25,256k,25),seekpct=random

      wd=wd8,sd=sd8,xfersize=(16k,25,32k,25,128k,25,256k,25),seekpct=random

      wd=wd9,sd=sd9,xfersize=(16k,25,32k,25,128k,25,256k,25),seekpct=random

      wd=wd10,sd=sd10,xfersize=(16k,25,32k,25,128k,25,256k,25),seekpct=random

      wd=wd11,sd=sd11,xfersize=(16k,25,32k,25,128k,25,256k,25),seekpct=random

       

       

      rd=rd1,wd=wd*,forthreads=16,forrdpct=50,interval=5,iorate=max,pause=0,elapsed=1800

       

      This test would pass if there are no other unconfigured drives in the server other than the ones contained in the sd* defined on this parameter file. The test would fail if there are other unconfigured drives. Could you please share your thoughts?

       

      Thanks,

      Priyanka

        • 1. Re: Slave termination error
          Henk Vandenbergh-Oracle

          Please send me the COMPLETE logfile.html

          • 2. Re: Slave termination error
            e20f5b36-bf17-40b3-af2a-9bd65e6204ad

            <pre>

            13:26:18.958 Vdbench distribution: vdbench503rc11

            13:26:18.959

            13:26:18.973 *

            13:26:18.973 * This beta version 'vdbench503rc11' was built on Fri April 08  9:48:23 MDT 2011.

            13:26:18.973 * which is more than 90 days ago.

            13:26:18.973 * It is recommended that you look for a newer beta or possibly

            13:26:18.974 * a newer GA version.

            13:26:18.974 * Of course, it is preferred you download a newer beta version

            13:26:18.974 * if available to help test the latest code.

            13:26:18.974 *

            13:26:18.976 input argument scanned: '-m8'

            13:26:18.995 input argument scanned: '-f/var/log/validator-logs/vdbench-config.140625.132608'

            13:26:18.996 input argument scanned: '-o/var/log/validator-logs/vdbench-output.140625.132608'

            13:26:18.996 input argument scanned: '-e1800'

            13:26:19.000 java.vendor                   Oracle Corporation

            13:26:19.000 java.home                     /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.9.x86_64/jre

            13:26:19.000 java.vm.specification.version 1.7

            13:26:19.000 java.vm.version               23.2-b09

            13:26:19.000 java.vm.vendor                Oracle Corporation

            13:26:19.001 java.specification.version    1.7

            13:26:19.001 java.class.version            51.0

            13:26:19.001 user.name                     root

            13:26:19.001 user.dir                      /

            13:26:19.001 java.class.path               /opt/vdbench/:/opt/vdbench/classes:/opt/vdbench/vdbench.jar:/opt/vdbench/jaramiko.jar

            13:26:19.001 os.name                       Linux

            13:26:19.001 os.arch                       amd64

            13:26:19.001 os.version                    2.6.32-358.el6.x86_64

            13:26:19.001 sun.arch.data.model           64

            13:26:19.203 will_we_use_waiter: rd1 false; rc=6; req:  999988; iorate= 999988

            13:26:19.204 will_we_use_waiter: rd1 false; rc=6; req:  999988; iorate= 999988

            13:26:19.204 will_we_use_waiter: rd1 false; rc=6; req:  999988; iorate= 999988

            13:26:19.205 will_we_use_waiter: rd1 false; rc=6; req:  999988; iorate= 999988

            13:26:19.205 will_we_use_waiter: rd1 false; rc=6; req:  999988; iorate= 999988

            13:26:19.206 will_we_use_waiter: rd1 false; rc=6; req:  999988; iorate= 999988

            13:26:19.206 will_we_use_waiter: rd1 false; rc=6; req:  999988; iorate= 999988

            13:26:19.207 will_we_use_waiter: rd1 false; rc=6; req:  999988; iorate= 999988

            13:26:19.207 will_we_use_waiter: rd1 false; rc=6; req:  999988; iorate= 999988

            13:26:19.208 will_we_use_waiter: rd1 false; rc=6; req:  999988; iorate= 999988

            13:26:19.208 will_we_use_waiter: rd1 false; rc=6; req:  999988; iorate= 999988

            13:26:19.225 will_we_use_waiter: rd1 false; rc=6; req:  999988; iorate= 999988

            13:26:19.225 will_we_use_waiter: rd1 false; rc=6; req:  999988; iorate= 999988

            13:26:19.226 will_we_use_waiter: rd1 false; rc=6; req:  999988; iorate= 999988

            13:26:19.226 will_we_use_waiter: rd1 false; rc=6; req:  999988; iorate= 999988

            13:26:19.227 will_we_use_waiter: rd1 false; rc=6; req:  999988; iorate= 999988

            13:26:19.227 will_we_use_waiter: rd1 false; rc=6; req:  999988; iorate= 999988

            13:26:19.228 will_we_use_waiter: rd1 false; rc=6; req:  999988; iorate= 999988

            13:26:19.228 will_we_use_waiter: rd1 false; rc=6; req:  999988; iorate= 999988

            13:26:19.229 will_we_use_waiter: rd1 false; rc=6; req:  999988; iorate= 999988

            13:26:19.229 will_we_use_waiter: rd1 false; rc=6; req:  999988; iorate= 999988

            13:26:19.230 will_we_use_waiter: rd1 false; rc=6; req:  999988; iorate= 999988

            13:28:52.947 Starting slave: /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-11-140625-13.26.18.930 -l localhost-1 -p 5570  

            13:28:52.947 Starting slave: /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-12-140625-13.26.18.930 -l localhost-2 -p 5570  

            13:28:52.947 Starting slave: /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-10-140625-13.26.18.930 -l localhost-0 -p 5570  

            13:28:52.947 Starting slave: /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-13-140625-13.26.18.930 -l localhost-3 -p 5570  

            13:28:52.948 Starting slave: /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-14-140625-13.26.18.930 -l localhost-4 -p 5570  

            13:28:52.948 Starting slave: /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-15-140625-13.26.18.930 -l localhost-5 -p 5570  

            13:28:52.949 Starting slave: /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-16-140625-13.26.18.930 -l localhost-6 -p 5570  

            13:28:52.950 Starting slave: /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-17-140625-13.26.18.930 -l localhost-7 -p 5570  

            13:28:53.524 Slave localhost-0 connected

            13:28:53.524 Slave localhost-5 connected

            13:28:53.524 Slave localhost-7 connected

            13:28:53.524 Slave localhost-6 connected

            13:28:53.524 Slave localhost-2 connected

            13:28:53.524 Slave localhost-3 connected

            13:28:53.524 Slave localhost-4 connected

            13:28:53.525 Slave localhost-1 connected

            13:28:53.548 All slaves are now connected

            13:28:54.671 sd=sd1,lun=/dev/sdl lun size: 53687091200 bytes; 50.0000 GB (1024**3); 53.6871 GB (1000**3)

            13:28:54.671 sd=sd2,lun=/dev/sdj lun size: 53687091200 bytes; 50.0000 GB (1024**3); 53.6871 GB (1000**3)

            13:28:54.671 sd=sd3,lun=/dev/sdk lun size: 53687091200 bytes; 50.0000 GB (1024**3); 53.6871 GB (1000**3)

            13:28:54.671 sd=sd4,lun=/dev/sdh lun size: 53687091200 bytes; 50.0000 GB (1024**3); 53.6871 GB (1000**3)

            13:28:54.672 sd=sd5,lun=/dev/sdi lun size: 53687091200 bytes; 50.0000 GB (1024**3); 53.6871 GB (1000**3)

            13:28:54.672 sd=sd6,lun=/dev/sdf lun size: 53687091200 bytes; 50.0000 GB (1024**3); 53.6871 GB (1000**3)

            13:28:54.672 sd=sd7,lun=/dev/sdg lun size: 53687091200 bytes; 50.0000 GB (1024**3); 53.6871 GB (1000**3)

            13:28:54.672 sd=sd8,lun=/dev/sdd lun size: 53687091200 bytes; 50.0000 GB (1024**3); 53.6871 GB (1000**3)

            13:28:54.672 sd=sd9,lun=/dev/sde lun size: 53687091200 bytes; 50.0000 GB (1024**3); 53.6871 GB (1000**3)

            13:28:54.673 sd=sd10,lun=/dev/sdb lun size: 53687091200 bytes; 50.0000 GB (1024**3); 53.6871 GB (1000**3)

            13:28:54.673 sd=sd11,lun=/dev/sdc lun size: 53687091200 bytes; 50.0000 GB (1024**3); 53.6871 GB (1000**3)

            13:28:54.678 ++Interarrival time: wd=wd1 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.678 ++Interarrival time: wd=wd1 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.679 ++Interarrival time: wd=wd1 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.680 ++Interarrival time: wd=wd1 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.681 ++Interarrival time: wd=wd1 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.682 ++Interarrival time: wd=wd1 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.683 ++Interarrival time: wd=wd1 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.684 ++Interarrival time: wd=wd1 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.684 ++Interarrival time: wd=wd2 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.685 ++Interarrival time: wd=wd2 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.686 ++Interarrival time: wd=wd2 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.687 ++Interarrival time: wd=wd2 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.688 ++Interarrival time: wd=wd2 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.689 ++Interarrival time: wd=wd2 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.690 ++Interarrival time: wd=wd2 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.691 ++Interarrival time: wd=wd2 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.692 ++Interarrival time: wd=wd3 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.692 ++Interarrival time: wd=wd3 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.693 ++Interarrival time: wd=wd3 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.693 ++Interarrival time: wd=wd3 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.694 ++Interarrival time: wd=wd3 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.695 ++Interarrival time: wd=wd3 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.696 ++Interarrival time: wd=wd3 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.698 ++Interarrival time: wd=wd3 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.699 ++Interarrival time: wd=wd4 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.700 ++Interarrival time: wd=wd4 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.701 ++Interarrival time: wd=wd4 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.702 ++Interarrival time: wd=wd4 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.702 ++Interarrival time: wd=wd4 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.703 ++Interarrival time: wd=wd4 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.703 ++Interarrival time: wd=wd4 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.704 ++Interarrival time: wd=wd4 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.705 ++Interarrival time: wd=wd5 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.706 ++Interarrival time: wd=wd5 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.707 ++Interarrival time: wd=wd5 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.708 ++Interarrival time: wd=wd5 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.709 ++Interarrival time: wd=wd5 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.710 ++Interarrival time: wd=wd5 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.711 ++Interarrival time: wd=wd5 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.712 ++Interarrival time: wd=wd5 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.712 ++Interarrival time: wd=wd6 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.713 ++Interarrival time: wd=wd6 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.713 ++Interarrival time: wd=wd6 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.714 ++Interarrival time: wd=wd6 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.715 ++Interarrival time: wd=wd6 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.715 ++Interarrival time: wd=wd6 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.716 ++Interarrival time: wd=wd6 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.717 ++Interarrival time: wd=wd6 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.718 ++Interarrival time: wd=wd7 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.719 ++Interarrival time: wd=wd7 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.720 ++Interarrival time: wd=wd7 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.721 ++Interarrival time: wd=wd7 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.721 ++Interarrival time: wd=wd7 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.722 ++Interarrival time: wd=wd7 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.723 ++Interarrival time: wd=wd7 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.724 ++Interarrival time: wd=wd7 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.724 ++Interarrival time: wd=wd8 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.725 ++Interarrival time: wd=wd8 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.725 ++Interarrival time: wd=wd8 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.726 ++Interarrival time: wd=wd8 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.726 ++Interarrival time: wd=wd8 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.727 ++Interarrival time: wd=wd8 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.727 ++Interarrival time: wd=wd8 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.728 ++Interarrival time: wd=wd8 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.728 ++Interarrival time: wd=wd9 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.729 ++Interarrival time: wd=wd9 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.729 ++Interarrival time: wd=wd9 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.730 ++Interarrival time: wd=wd9 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.730 ++Interarrival time: wd=wd9 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.731 ++Interarrival time: wd=wd9 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.731 ++Interarrival time: wd=wd9 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.732 ++Interarrival time: wd=wd9 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.732 ++Interarrival time: wd=wd10 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.733 ++Interarrival time: wd=wd10 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.733 ++Interarrival time: wd=wd10 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.734 ++Interarrival time: wd=wd10 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.735 ++Interarrival time: wd=wd10 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.735 ++Interarrival time: wd=wd10 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.736 ++Interarrival time: wd=wd10 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.736 ++Interarrival time: wd=wd10 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.737 ++Interarrival time: wd=wd11 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.737 ++Interarrival time: wd=wd11 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.738 ++Interarrival time: wd=wd11 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.738 ++Interarrival time: wd=wd11 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.739 ++Interarrival time: wd=wd11 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.740 ++Interarrival time: wd=wd11 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.740 ++Interarrival time: wd=wd11 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.741 ++Interarrival time: wd=wd11 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

             

             

            Link to Run Definitions:         <A HREF="#_215181933">rd1 For loops: threads=16.0 rdpct=50.0</A>

             

             

            13:28:54.772 ++Interarrival time: wd=wd1 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.772 ++Interarrival time: wd=wd1 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.773 ++Interarrival time: wd=wd1 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.773 ++Interarrival time: wd=wd1 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.774 ++Interarrival time: wd=wd1 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.774 ++Interarrival time: wd=wd1 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.775 ++Interarrival time: wd=wd1 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.775 ++Interarrival time: wd=wd1 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.776 ++Interarrival time: wd=wd2 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.776 ++Interarrival time: wd=wd2 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.777 ++Interarrival time: wd=wd2 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.777 ++Interarrival time: wd=wd2 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.778 ++Interarrival time: wd=wd2 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.778 ++Interarrival time: wd=wd2 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.779 ++Interarrival time: wd=wd2 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.779 ++Interarrival time: wd=wd2 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.780 ++Interarrival time: wd=wd3 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.780 ++Interarrival time: wd=wd3 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.781 ++Interarrival time: wd=wd3 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.781 ++Interarrival time: wd=wd3 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.782 ++Interarrival time: wd=wd3 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.782 ++Interarrival time: wd=wd3 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.783 ++Interarrival time: wd=wd3 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.783 ++Interarrival time: wd=wd3 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.784 ++Interarrival time: wd=wd4 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.784 ++Interarrival time: wd=wd4 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.785 ++Interarrival time: wd=wd4 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.785 ++Interarrival time: wd=wd4 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.786 ++Interarrival time: wd=wd4 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.792 ++Interarrival time: wd=wd4 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.792 ++Interarrival time: wd=wd4 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.793 ++Interarrival time: wd=wd4 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.793 ++Interarrival time: wd=wd5 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.794 ++Interarrival time: wd=wd5 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.794 ++Interarrival time: wd=wd5 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.795 ++Interarrival time: wd=wd5 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.795 ++Interarrival time: wd=wd5 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.795 ++Interarrival time: wd=wd5 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.796 ++Interarrival time: wd=wd5 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.796 ++Interarrival time: wd=wd5 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.797 ++Interarrival time: wd=wd6 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.797 ++Interarrival time: wd=wd6 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.798 ++Interarrival time: wd=wd6 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.798 ++Interarrival time: wd=wd6 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.798 ++Interarrival time: wd=wd6 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.799 ++Interarrival time: wd=wd6 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.799 ++Interarrival time: wd=wd6 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.800 ++Interarrival time: wd=wd6 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.800 ++Interarrival time: wd=wd7 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.801 ++Interarrival time: wd=wd7 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.801 ++Interarrival time: wd=wd7 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.802 ++Interarrival time: wd=wd7 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.802 ++Interarrival time: wd=wd7 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.803 ++Interarrival time: wd=wd7 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.803 ++Interarrival time: wd=wd7 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.804 ++Interarrival time: wd=wd7 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.804 ++Interarrival time: wd=wd8 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.805 ++Interarrival time: wd=wd8 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.805 ++Interarrival time: wd=wd8 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.805 ++Interarrival time: wd=wd8 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.806 ++Interarrival time: wd=wd8 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.806 ++Interarrival time: wd=wd8 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.807 ++Interarrival time: wd=wd8 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.807 ++Interarrival time: wd=wd8 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.808 ++Interarrival time: wd=wd9 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.808 ++Interarrival time: wd=wd9 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.808 ++Interarrival time: wd=wd9 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.809 ++Interarrival time: wd=wd9 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.809 ++Interarrival time: wd=wd9 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.810 ++Interarrival time: wd=wd9 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.810 ++Interarrival time: wd=wd9 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.810 ++Interarrival time: wd=wd9 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.811 ++Interarrival time: wd=wd10 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.811 ++Interarrival time: wd=wd10 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.812 ++Interarrival time: wd=wd10 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.812 ++Interarrival time: wd=wd10 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.813 ++Interarrival time: wd=wd10 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.813 ++Interarrival time: wd=wd10 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.813 ++Interarrival time: wd=wd10 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.814 ++Interarrival time: wd=wd10 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.814 ++Interarrival time: wd=wd11 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.814 ++Interarrival time: wd=wd11 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.815 ++Interarrival time: wd=wd11 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.815 ++Interarrival time: wd=wd11 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.815 ++Interarrival time: wd=wd11 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.816 ++Interarrival time: wd=wd11 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.816 ++Interarrival time: wd=wd11 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.816 ++Interarrival time: wd=wd11 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364

            13:28:54.817 Sending to localhost-0  wd=wd1,sd=sd1,lun=/dev/sdl

            13:28:54.817 Sending to localhost-0  wd=wd2,sd=sd2,lun=/dev/sdj

            13:28:54.817 Sending to localhost-0  wd=wd3,sd=sd3,lun=/dev/sdk

            13:28:54.817 Sending to localhost-0  wd=wd4,sd=sd4,lun=/dev/sdh

            13:28:54.817 Sending to localhost-0  wd=wd5,sd=sd5,lun=/dev/sdi

            13:28:54.817 Sending to localhost-0  wd=wd6,sd=sd6,lun=/dev/sdf

            13:28:54.817 Sending to localhost-0  wd=wd7,sd=sd7,lun=/dev/sdg

            13:28:54.817 Sending to localhost-0  wd=wd8,sd=sd8,lun=/dev/sdd

            13:28:54.817 Sending to localhost-0  wd=wd9,sd=sd9,lun=/dev/sde

            13:28:54.817 Sending to localhost-0  wd=wd10,sd=sd10,lun=/dev/sdb

            13:28:54.817 Sending to localhost-0  wd=wd11,sd=sd11,lun=/dev/sdc

            13:28:54.817 Sending to localhost-1  wd=wd1,sd=sd1,lun=/dev/sdl

            13:28:54.817 Sending to localhost-1  wd=wd2,sd=sd2,lun=/dev/sdj

            13:28:54.817 Sending to localhost-1  wd=wd3,sd=sd3,lun=/dev/sdk

            13:28:54.817 Sending to localhost-1  wd=wd4,sd=sd4,lun=/dev/sdh

            13:28:54.818 Sending to localhost-1  wd=wd5,sd=sd5,lun=/dev/sdi

            13:28:54.818 Sending to localhost-1  wd=wd6,sd=sd6,lun=/dev/sdf

            13:28:54.818 Sending to localhost-1  wd=wd7,sd=sd7,lun=/dev/sdg

            13:28:54.818 Sending to localhost-1  wd=wd8,sd=sd8,lun=/dev/sdd

            13:28:54.818 Sending to localhost-1  wd=wd9,sd=sd9,lun=/dev/sde

            13:28:54.818 Sending to localhost-1  wd=wd10,sd=sd10,lun=/dev/sdb

            13:28:54.818 Sending to localhost-1  wd=wd11,sd=sd11,lun=/dev/sdc

            13:28:54.818 Sending to localhost-2  wd=wd1,sd=sd1,lun=/dev/sdl

            13:28:54.818 Sending to localhost-2  wd=wd2,sd=sd2,lun=/dev/sdj

            13:28:54.818 Sending to localhost-2  wd=wd3,sd=sd3,lun=/dev/sdk

            13:28:54.818 Sending to localhost-2  wd=wd4,sd=sd4,lun=/dev/sdh

            13:28:54.818 Sending to localhost-2  wd=wd5,sd=sd5,lun=/dev/sdi

            13:28:54.818 Sending to localhost-2  wd=wd6,sd=sd6,lun=/dev/sdf

            13:28:54.818 Sending to localhost-2  wd=wd7,sd=sd7,lun=/dev/sdg

            13:28:54.818 Sending to localhost-2  wd=wd8,sd=sd8,lun=/dev/sdd

            13:28:54.818 Sending to localhost-2  wd=wd9,sd=sd9,lun=/dev/sde

            13:28:54.819 Sending to localhost-2  wd=wd10,sd=sd10,lun=/dev/sdb

            13:28:54.819 Sending to localhost-2  wd=wd11,sd=sd11,lun=/dev/sdc

            13:28:54.819 Sending to localhost-3  wd=wd1,sd=sd1,lun=/dev/sdl

            13:28:54.819 Sending to localhost-3  wd=wd2,sd=sd2,lun=/dev/sdj

            13:28:54.819 Sending to localhost-3  wd=wd3,sd=sd3,lun=/dev/sdk

            13:28:54.819 Sending to localhost-3  wd=wd4,sd=sd4,lun=/dev/sdh

            13:28:54.819 Sending to localhost-3  wd=wd5,sd=sd5,lun=/dev/sdi

            13:28:54.819 Sending to localhost-3  wd=wd6,sd=sd6,lun=/dev/sdf

            13:28:54.819 Sending to localhost-3  wd=wd7,sd=sd7,lun=/dev/sdg

            13:28:54.819 Sending to localhost-3  wd=wd8,sd=sd8,lun=/dev/sdd

            13:28:54.819 Sending to localhost-3  wd=wd9,sd=sd9,lun=/dev/sde

            13:28:54.819 Sending to localhost-3  wd=wd10,sd=sd10,lun=/dev/sdb

            13:28:54.819 Sending to localhost-3  wd=wd11,sd=sd11,lun=/dev/sdc

            13:28:54.819 Sending to localhost-4  wd=wd1,sd=sd1,lun=/dev/sdl

            13:28:54.819 Sending to localhost-4  wd=wd2,sd=sd2,lun=/dev/sdj

            13:28:54.819 Sending to localhost-4  wd=wd3,sd=sd3,lun=/dev/sdk

            13:28:54.820 Sending to localhost-4  wd=wd4,sd=sd4,lun=/dev/sdh

            13:28:54.820 Sending to localhost-4  wd=wd5,sd=sd5,lun=/dev/sdi

            13:28:54.820 Sending to localhost-4  wd=wd6,sd=sd6,lun=/dev/sdf

            13:28:54.820 Sending to localhost-4  wd=wd7,sd=sd7,lun=/dev/sdg

            13:28:54.820 Sending to localhost-4  wd=wd8,sd=sd8,lun=/dev/sdd

            13:28:54.820 Sending to localhost-4  wd=wd9,sd=sd9,lun=/dev/sde

            13:28:54.820 Sending to localhost-4  wd=wd10,sd=sd10,lun=/dev/sdb

            13:28:54.820 Sending to localhost-4  wd=wd11,sd=sd11,lun=/dev/sdc

            13:28:54.820 Sending to localhost-5  wd=wd1,sd=sd1,lun=/dev/sdl

            13:28:54.820 Sending to localhost-5  wd=wd2,sd=sd2,lun=/dev/sdj

            13:28:54.820 Sending to localhost-5  wd=wd3,sd=sd3,lun=/dev/sdk

            13:28:54.820 Sending to localhost-5  wd=wd4,sd=sd4,lun=/dev/sdh

            13:28:54.820 Sending to localhost-5  wd=wd5,sd=sd5,lun=/dev/sdi

            13:28:54.820 Sending to localhost-5  wd=wd6,sd=sd6,lun=/dev/sdf

            13:28:54.820 Sending to localhost-5  wd=wd7,sd=sd7,lun=/dev/sdg

            13:28:54.820 Sending to localhost-5  wd=wd8,sd=sd8,lun=/dev/sdd

            13:28:54.821 Sending to localhost-5  wd=wd9,sd=sd9,lun=/dev/sde

            13:28:54.821 Sending to localhost-5  wd=wd10,sd=sd10,lun=/dev/sdb

            13:28:54.821 Sending to localhost-5  wd=wd11,sd=sd11,lun=/dev/sdc

            13:28:54.821 Sending to localhost-6  wd=wd1,sd=sd1,lun=/dev/sdl

            13:28:54.821 Sending to localhost-6  wd=wd2,sd=sd2,lun=/dev/sdj

            13:28:54.821 Sending to localhost-6  wd=wd3,sd=sd3,lun=/dev/sdk

            13:28:54.821 Sending to localhost-6  wd=wd4,sd=sd4,lun=/dev/sdh

            13:28:54.821 Sending to localhost-6  wd=wd5,sd=sd5,lun=/dev/sdi

            13:28:54.821 Sending to localhost-6  wd=wd6,sd=sd6,lun=/dev/sdf

            13:28:54.821 Sending to localhost-6  wd=wd7,sd=sd7,lun=/dev/sdg

            13:28:54.821 Sending to localhost-6  wd=wd8,sd=sd8,lun=/dev/sdd

            13:28:54.821 Sending to localhost-6  wd=wd9,sd=sd9,lun=/dev/sde

            13:28:54.821 Sending to localhost-6  wd=wd10,sd=sd10,lun=/dev/sdb

            13:28:54.821 Sending to localhost-6  wd=wd11,sd=sd11,lun=/dev/sdc

            13:28:54.822 Sending to localhost-7  wd=wd1,sd=sd1,lun=/dev/sdl

            13:28:54.822 Sending to localhost-7  wd=wd2,sd=sd2,lun=/dev/sdj

            13:28:54.822 Sending to localhost-7  wd=wd3,sd=sd3,lun=/dev/sdk

            13:28:54.822 Sending to localhost-7  wd=wd4,sd=sd4,lun=/dev/sdh

            13:28:54.822 Sending to localhost-7  wd=wd5,sd=sd5,lun=/dev/sdi

            13:28:54.822 Sending to localhost-7  wd=wd6,sd=sd6,lun=/dev/sdf

            13:28:54.822 Sending to localhost-7  wd=wd7,sd=sd7,lun=/dev/sdg

            13:28:54.822 Sending to localhost-7  wd=wd8,sd=sd8,lun=/dev/sdd

            13:28:54.822 Sending to localhost-7  wd=wd9,sd=sd9,lun=/dev/sde

            13:28:54.822 Sending to localhost-7  wd=wd10,sd=sd10,lun=/dev/sdb

            13:28:54.822 Sending to localhost-7  wd=wd11,sd=sd11,lun=/dev/sdc

            13:28:55.039 SlaveList.printWorkForSlaves() *

            13:28:55.064 slv=localhost-0 rd=rd1    wd=wd1    sd=sd1    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.065 slv=localhost-0 rd=rd1    wd=wd2    sd=sd2    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.065 slv=localhost-0 rd=rd1    wd=wd3    sd=sd3    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.066 slv=localhost-0 rd=rd1    wd=wd4    sd=sd4    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.067 slv=localhost-0 rd=rd1    wd=wd5    sd=sd5    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.067 slv=localhost-0 rd=rd1    wd=wd6    sd=sd6    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.068 slv=localhost-0 rd=rd1    wd=wd7    sd=sd7    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.069 slv=localhost-0 rd=rd1    wd=wd8    sd=sd8    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.070 slv=localhost-0 rd=rd1    wd=wd9    sd=sd9    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.070 slv=localhost-0 rd=rd1    wd=wd10   sd=sd10   rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.071 slv=localhost-0 rd=rd1    wd=wd11   sd=sd11   rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.075 slv=localhost-1 rd=rd1    wd=wd1    sd=sd1    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.076 slv=localhost-1 rd=rd1    wd=wd2    sd=sd2    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.076 slv=localhost-1 rd=rd1    wd=wd3    sd=sd3    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.077 slv=localhost-1 rd=rd1    wd=wd4    sd=sd4    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.078 slv=localhost-1 rd=rd1    wd=wd5    sd=sd5    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.078 slv=localhost-1 rd=rd1    wd=wd6    sd=sd6    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.079 slv=localhost-1 rd=rd1    wd=wd7    sd=sd7    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.080 slv=localhost-1 rd=rd1    wd=wd8    sd=sd8    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.080 slv=localhost-1 rd=rd1    wd=wd9    sd=sd9    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.081 slv=localhost-1 rd=rd1    wd=wd10   sd=sd10   rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.095 slv=localhost-1 rd=rd1    wd=wd11   sd=sd11   rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.096 slv=localhost-2 rd=rd1    wd=wd1    sd=sd1    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.097 slv=localhost-2 rd=rd1    wd=wd2    sd=sd2    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.097 slv=localhost-2 rd=rd1    wd=wd3    sd=sd3    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.098 slv=localhost-2 rd=rd1    wd=wd4    sd=sd4    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.098 slv=localhost-2 rd=rd1    wd=wd5    sd=sd5    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.099 slv=localhost-2 rd=rd1    wd=wd6    sd=sd6    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.100 slv=localhost-2 rd=rd1    wd=wd7    sd=sd7    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.100 slv=localhost-2 rd=rd1    wd=wd8    sd=sd8    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.101 slv=localhost-2 rd=rd1    wd=wd9    sd=sd9    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.102 slv=localhost-2 rd=rd1    wd=wd10   sd=sd10   rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.102 slv=localhost-2 rd=rd1    wd=wd11   sd=sd11   rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.103 slv=localhost-3 rd=rd1    wd=wd1    sd=sd1    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.104 slv=localhost-3 rd=rd1    wd=wd2    sd=sd2    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.105 slv=localhost-3 rd=rd1    wd=wd3    sd=sd3    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.105 slv=localhost-3 rd=rd1    wd=wd4    sd=sd4    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.106 slv=localhost-3 rd=rd1    wd=wd5    sd=sd5    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.107 slv=localhost-3 rd=rd1    wd=wd6    sd=sd6    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.108 slv=localhost-3 rd=rd1    wd=wd7    sd=sd7    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.108 slv=localhost-3 rd=rd1    wd=wd8    sd=sd8    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.109 slv=localhost-3 rd=rd1    wd=wd9    sd=sd9    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.110 slv=localhost-3 rd=rd1    wd=wd10   sd=sd10   rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.110 slv=localhost-3 rd=rd1    wd=wd11   sd=sd11   rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.111 slv=localhost-4 rd=rd1    wd=wd1    sd=sd1    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.112 slv=localhost-4 rd=rd1    wd=wd2    sd=sd2    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.112 slv=localhost-4 rd=rd1    wd=wd3    sd=sd3    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.113 slv=localhost-4 rd=rd1    wd=wd4    sd=sd4    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.114 slv=localhost-4 rd=rd1    wd=wd5    sd=sd5    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.114 slv=localhost-4 rd=rd1    wd=wd6    sd=sd6    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.115 slv=localhost-4 rd=rd1    wd=wd7    sd=sd7    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.116 slv=localhost-4 rd=rd1    wd=wd8    sd=sd8    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.116 slv=localhost-4 rd=rd1    wd=wd9    sd=sd9    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.117 slv=localhost-4 rd=rd1    wd=wd10   sd=sd10   rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.118 slv=localhost-4 rd=rd1    wd=wd11   sd=sd11   rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.118 slv=localhost-5 rd=rd1    wd=wd1    sd=sd1    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.119 slv=localhost-5 rd=rd1    wd=wd2    sd=sd2    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.120 slv=localhost-5 rd=rd1    wd=wd3    sd=sd3    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.120 slv=localhost-5 rd=rd1    wd=wd4    sd=sd4    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.121 slv=localhost-5 rd=rd1    wd=wd5    sd=sd5    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.122 slv=localhost-5 rd=rd1    wd=wd6    sd=sd6    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.122 slv=localhost-5 rd=rd1    wd=wd7    sd=sd7    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.123 slv=localhost-5 rd=rd1    wd=wd8    sd=sd8    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.124 slv=localhost-5 rd=rd1    wd=wd9    sd=sd9    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.124 slv=localhost-5 rd=rd1    wd=wd10   sd=sd10   rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.125 slv=localhost-5 rd=rd1    wd=wd11   sd=sd11   rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.125 slv=localhost-6 rd=rd1    wd=wd1    sd=sd1    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.126 slv=localhost-6 rd=rd1    wd=wd2    sd=sd2    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.127 slv=localhost-6 rd=rd1    wd=wd3    sd=sd3    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.128 slv=localhost-6 rd=rd1    wd=wd4    sd=sd4    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.128 slv=localhost-6 rd=rd1    wd=wd5    sd=sd5    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.129 slv=localhost-6 rd=rd1    wd=wd6    sd=sd6    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.130 slv=localhost-6 rd=rd1    wd=wd7    sd=sd7    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.130 slv=localhost-6 rd=rd1    wd=wd8    sd=sd8    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.131 slv=localhost-6 rd=rd1    wd=wd9    sd=sd9    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.132 slv=localhost-6 rd=rd1    wd=wd10   sd=sd10   rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.132 slv=localhost-6 rd=rd1    wd=wd11   sd=sd11   rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.144 slv=localhost-7 rd=rd1    wd=wd1    sd=sd1    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.145 slv=localhost-7 rd=rd1    wd=wd2    sd=sd2    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.146 slv=localhost-7 rd=rd1    wd=wd3    sd=sd3    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.147 slv=localhost-7 rd=rd1    wd=wd4    sd=sd4    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.147 slv=localhost-7 rd=rd1    wd=wd5    sd=sd5    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.148 slv=localhost-7 rd=rd1    wd=wd6    sd=sd6    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.149 slv=localhost-7 rd=rd1    wd=wd7    sd=sd7    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.149 slv=localhost-7 rd=rd1    wd=wd8    sd=sd8    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.150 slv=localhost-7 rd=rd1    wd=wd9    sd=sd9    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.151 slv=localhost-7 rd=rd1    wd=wd10   sd=sd10   rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.151 slv=localhost-7 rd=rd1    wd=wd11   sd=sd11   rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2

            13:28:55.151

            13:28:55.152 Waiting for synchronization of all slaves

            13:28:55.653 Synchronization of all slaves complete

            13:28:56.003 Starting RD=rd1; I/O rate: Uncontrolled MAX; elapsed=1800; For loops: threads=16.0 rdpct=50.0

             

             

            <a name="_215181933"></a><i><b>13:28:56.013 Starting RD=rd1; I/O rate: Uncontrolled MAX; elapsed=1800; For loops: threads=16.0 rdpct=50.0</b></i>

             

             

             

             

            Jun 25, 2014  interval        i/o   MB/sec   bytes   read     resp     read    write     resp     resp queue  cpu%  cpu%

                                         rate  1024**2     i/o    pct     time     resp     resp      max   stddev depth sys+u   sys

            13:29:01.179         1    3465.60   366.26  110817  49.38   48.609   98.276    0.165 1075.965  100.053 173.8  16.4   2.1

            13:29:05.542

            13:29:05.542 Message from slave localhost-0:

            13:29:05.542 New messages found on /var/adm/messages. Do they belong to you?

            13:29:05.542 /var/log/messages: Jun 25 13:29:04 svt-smx9dri-002 kernel: megasas: Found FW in FAULT state, will reset adapter.

            13:29:05.542 /var/log/messages: Jun 25 13:29:04 svt-smx9dri-002 kernel: megaraid_sas: resetting fusion adapter.

            13:29:05.543

            13:29:06.099         2    1202.80   124.35  108403  49.57   74.091  149.305    0.166 1239.084  136.329 174.8   2.6   0.5

            13:29:11.083         3       0.00     0.00       0   0.00    0.000    0.000    0.000    0.000    0.000 176.0   1.1   0.2

            13:31:23.556 HeartBeat.checkHeartBeat(): slave localhost-0 has not responded for 125 seconds.

            13:31:42.042 HeartBeat.checkHeartBeat(): slave localhost-1 has not responded for 125 seconds.

            13:31:42.042 HeartBeat.checkHeartBeat(): slave localhost-2 has not responded for 125 seconds.

            13:31:42.042 HeartBeat.checkHeartBeat(): slave localhost-3 has not responded for 125 seconds.

            13:31:42.042 HeartBeat.checkHeartBeat(): slave localhost-4 has not responded for 125 seconds.

            13:31:42.042 HeartBeat.checkHeartBeat(): slave localhost-5 has not responded for 125 seconds.

            13:31:42.042 HeartBeat.checkHeartBeat(): slave localhost-6 has not responded for 125 seconds.

            13:31:42.043 HeartBeat.checkHeartBeat(): slave localhost-7 has not responded for 125 seconds.

            13:31:42.043 Start/end command: executing '/opt/vdbench/vdbench jstack'

            13:31:42.043 execute(): /opt/vdbench/vdbench jstack

            13:31:42.066         4       0.00     0.00       0   0.00    0.000    0.000    0.000    0.000    0.000 177.5   1.1   0.2

            13:31:42.099         5       0.00     0.00       0   0.00    0.000    0.000    0.000    0.000    0.000  88.0   0.3   0.1

            13:31:42.154         6       0.00     0.00       0   0.00    0.000    0.000    0.000    0.000    0.000 132.0   0.5   0.3

            13:31:42.277         7       0.00     0.00       0   0.00    0.000    0.000    0.000    0.000    0.000 176.0   0.2   0.1

            13:31:42.362         8       0.00     0.00       0   0.00    0.000    0.000    0.000    0.000    0.000  89.6   0.2   0.0

            13:31:42.456         9       0.00     0.00       0   0.00    0.000    0.000    0.000    0.000    0.000 199.5   0.2   0.0

            13:31:42.520 Debug_cmds(); stderr 13:31:42.519 Jstack positional parameter: 'jstack'

            13:31:42.525        10       0.00     0.00       0   0.00    0.000    0.000    0.000    0.000    0.000  89.6   NaN   NaN

            13:31:42.568        11       0.00     0.00       0   0.00    0.000    0.000    0.000    0.000    0.000 110.0   NaN   NaN

            13:31:42.600        12       0.00     0.00       0   0.00    0.000    0.000    0.000    0.000    0.000 153.9   NaN   NaN

            13:31:42.625        13      48.80     5.15  110592  52.87 37672.016 71255.465    0.146 158989.885 67579.972 1175.5   NaN   NaN

            13:31:42.649        14      35.40     3.68  109041  53.11 21497.768 40479.680    0.181 158826.936 54393.776 443.7   0.2   0.0

            13:31:42.673        15      95.20    10.43  114860  50.21 9012.154 17948.704    0.190 159358.884 36695.871 423.5   0.1   0.0

            13:31:42.698        16       2.00     0.16   81920  80.00   41.933   52.397    0.078  176.845   53.803  22.0   NaN   NaN

            13:31:42.724        17      48.40     5.19  112453  48.35 12460.927 25773.729    0.145 158989.060 42715.660 378.0  81.3  12.5

            13:31:42.744        18      11.40     1.40  128772  49.12 11148.681 22695.195    0.323 158873.193 40875.328   0.3  85.7  14.3

            13:31:42.764        19      28.00     3.32  124284  50.71 2286.839 4509.065    0.200 158448.763 18867.068   0.8  90.0   0.0

            13:31:42.783        20      11.80     1.23  109411  52.54   26.111   49.329    0.406  163.620   39.186   0.4  80.0   0.0

            13:31:42.792

            13:31:42.792 Message from slave localhost-0:

            13:31:42.792 New messages found on /var/adm/messages. Do they belong to you?

            13:31:42.793 /var/log/messages: Jun 25 13:29:10 svt-smx9dri-002 kernel: megasas: Waiting for FW to come to ready state

            13:31:42.793 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: INFO: task jbd2/sda2-8:696 blocked for more than 120 seconds.

            13:31:42.793 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

            13:31:42.793 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: jbd2/sda2-8   D 0000000000000000     0   696      2 0x00000000

            13:31:42.793 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: ffff880276235c20 0000000000000046 ffff880276235bb0 ffffffff8108105d

            13:31:42.793 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: ffff880276235b90 ffffffff81012b69 ffff88027643f938 ffff88027643f848

            13:31:42.793 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: ffff880276681ab8 ffff880276235fd8 000000000000fb88 ffff880276681ab8

            13:31:42.793 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: Call Trace:

            13:31:42.793 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8108105d>] ? del_timer+0x7d/0xe0

            13:31:42.793 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff81012b69>] ? read_tsc+0x9/0x20

            13:31:42.793 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff810a1aa9>] ? ktime_get_ts+0xa9/0xe0

            13:31:42.793 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff811b5f60>] ? sync_buffer+0x0/0x50

            13:31:42.793 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8150de73>] io_schedule+0x73/0xc0

            13:31:42.793 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff811b5fa0>] sync_buffer+0x40/0x50

            13:31:42.793 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8150e82f>] __wait_on_bit+0x5f/0x90

            13:31:42.793 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff811b5f60>] ? sync_buffer+0x0/0x50

            13:31:42.793 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8150e8d8>] out_of_line_wait_on_bit+0x78/0x90

            13:31:42.794 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff81096cc0>] ? wake_bit_function+0x0/0x50

            13:31:42.794 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff811b5f56>] __wait_on_buffer+0x26/0x30

            13:31:42.794 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa02240a6>] jbd2_journal_commit_transaction+0xa76/0x14b0 [jbd2]

            13:31:42.794 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320

            13:31:42.794 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff81081a3b>] ? try_to_del_timer_sync+0x7b/0xe0

            13:31:42.794 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa0229f38>] kjournald2+0xb8/0x220 [jbd2]

            13:31:42.794 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff81096c80>] ? autoremove_wake_function+0x0/0x40

            13:31:42.794 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa0229e80>] ? kjournald2+0x0/0x220 [jbd2]

            13:31:42.794 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff81096916>] kthread+0x96/0xa0

            13:31:42.794 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8100c0ca>] child_rip+0xa/0x20

            13:31:42.794 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff81096880>] ? kthread+0x0/0xa0

            13:31:42.794 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20

            13:31:42.794 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: INFO: task flush-8:0:800 blocked for more than 120 seconds.

            13:31:42.794 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

            13:31:42.794 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: flush-8:0     D 000000000000000a     0   800      2 0x00000080

            13:31:42.794 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: ffff88027a5c1590 0000000000000046 ffff88027a5c1560 ffffffff813791cb

            13:31:42.794 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: 0000000000000003 0000000000000086 ffff88027a5c1540 ffffffff8108105d

            13:31:42.794 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: ffff880273f665f8 ffff88027a5c1fd8 000000000000fb88 ffff880273f665f8

            13:31:42.795 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: Call Trace:

            13:31:42.795 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff813791cb>] ? scsi_request_fn+0xdb/0x7a0

            13:31:42.795 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8108105d>] ? del_timer+0x7d/0xe0

            13:31:42.795 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff81096f6e>] ? prepare_to_wait+0x4e/0x80

            13:31:42.795 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa022305d>] do_get_write_access+0x29d/0x520 [jbd2]

            13:31:42.795 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff81096cc0>] ? wake_bit_function+0x0/0x50

            13:31:42.795 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa0223431>] jbd2_journal_get_write_access+0x31/0x50 [jbd2]

            13:31:42.795 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa026e708>] __ext4_journal_get_write_access+0x38/0x80 [ext4]

            13:31:42.795 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa02705b2>] ext4_mb_mark_diskspace_used+0xf2/0x300 [ext4]

            13:31:42.795 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa02771e1>] ext4_mb_new_blocks+0x311/0x5b0 [ext4]

            13:31:42.795 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa0269feb>] ? ext4_ext_find_extent+0x2ab/0x310 [ext4]

            13:31:42.795 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa026c170>] ext4_ext_get_blocks+0x4f0/0x14b0 [ext4]

            13:31:42.795 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff811b62d0>] ? end_buffer_async_write+0x0/0x190

            13:31:42.795 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa024b270>] ? noalloc_get_block_write+0x0/0x60 [ext4]

            13:31:42.795 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa024b270>] ? noalloc_get_block_write+0x0/0x60 [ext4]

            13:31:42.795 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff811b75d0>] ? block_write_full_page_endio+0xe0/0x120

            13:31:42.795 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa024ab89>] ext4_get_blocks+0xf9/0x2a0 [ext4]

            13:31:42.796 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8112ef45>] ? pagevec_lookup_tag+0x25/0x40

            13:31:42.796 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa024f201>] mpage_da_map_and_submit+0xa1/0x470 [ext4]

            13:31:42.796 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa0222455>] ? jbd2_journal_start+0xb5/0x100 [jbd2]

            13:31:42.796 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa024fe1e>] ext4_da_writepages+0x2ee/0x620 [ext4]

            13:31:42.796 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8112e0f1>] do_writepages+0x21/0x40

            13:31:42.796 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff811ac6ed>] writeback_single_inode+0xdd/0x290

            13:31:42.796 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff811acafe>] writeback_sb_inodes+0xce/0x180

            13:31:42.796 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff811acc5b>] writeback_inodes_wb+0xab/0x1b0

            13:31:42.796 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff811acffb>] wb_writeback+0x29b/0x3f0

            13:31:42.796 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8150d6e0>] ? thread_return+0x4e/0x76e

            13:31:42.796 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff81081ac2>] ? del_timer_sync+0x22/0x30

            13:31:42.796 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff811ad2e9>] wb_do_writeback+0x199/0x240

            13:31:42.796 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff811ad3f3>] bdi_writeback_task+0x63/0x1b0

            13:31:42.796 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff81096b47>] ? bit_waitqueue+0x17/0xd0

            13:31:42.796 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8113ca40>] ? bdi_start_fn+0x0/0x100

            13:31:42.796 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8113cac6>] bdi_start_fn+0x86/0x100

            13:31:42.796 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8113ca40>] ? bdi_start_fn+0x0/0x100

            13:31:42.797 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff81096916>] kthread+0x96/0xa0

            13:31:42.797 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8100c0ca>] child_rip+0xa/0x20

            13:31:42.797 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff81096880>] ? kthread+0x0/0xa0

            13:31:42.797 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20

            13:31:42.797 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: INFO: task jbd2/sda3-8:1948 blocked for more than 120 seconds.

            13:31:42.797 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

            13:31:42.797 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: jbd2/sda3-8   D 0000000000000000     0  1948      2 0x00000080

            13:31:42.797 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: ffff880273605b30 0000000000000046 0000000000000000 ffff880273605af4

            13:31:42.797 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: ffff880200000000 ffff88027fc24180 ffff8800282d6700 0000000000000400

            13:31:42.797 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: ffff8802735465f8 ffff880273605fd8 000000000000fb88 ffff8802735465f8

            13:31:42.797 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: Call Trace:

            13:31:42.797 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff81119cd0>] ? sync_page+0x0/0x50

            13:31:42.797 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8150de73>] io_schedule+0x73/0xc0

            13:31:42.797 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff81119d0d>] sync_page+0x3d/0x50

            13:31:42.797 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8150e82f>] __wait_on_bit+0x5f/0x90

            13:31:42.797 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff81119f43>] wait_on_page_bit+0x73/0x80

            13:31:42.797 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff81096cc0>] ? wake_bit_function+0x0/0x50

            13:31:42.798 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8112ef45>] ? pagevec_lookup_tag+0x25/0x40

            13:31:42.798 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8111a36b>] wait_on_page_writeback_range+0xfb/0x190

            13:31:42.798 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8125d02d>] ? submit_bio+0x8d/0x120

            13:31:42.798 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8111a42f>] filemap_fdatawait+0x2f/0x40

            13:31:42.798 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa0223e19>] jbd2_journal_commit_transaction+0x7e9/0x14b0 [jbd2]

            13:31:42.798 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320

            13:31:42.798 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff81081a3b>] ? try_to_del_timer_sync+0x7b/0xe0

            13:31:42.798 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa0229f38>] kjournald2+0xb8/0x220 [jbd2]

            13:31:42.798 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff81096c80>] ? autoremove_wake_function+0x0/0x40

            13:31:42.798 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa0229e80>] ? kjournald2+0x0/0x220 [jbd2]

            13:31:42.798 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff81096916>] kthread+0x96/0xa0

            13:31:42.798 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8100c0ca>] child_rip+0xa/0x20

            13:31:42.798 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff81096880>] ? kthread+0x0/0xa0

            13:31:42.798 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20

            13:31:42.798 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: INFO: task rs:main Q:Reg:2312 blocked for more than 120 seconds.

            13:31:42.798 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

            13:31:42.798 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: rs:main Q:Reg D 0000000000000005     0  2312      1 0x00000080

            13:31:42.798 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: ffff88026d1ef9a8 0000000000000086 0000000000000000 ffff8802765d4000

            13:31:42.798 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: ffff88026d1efa18 ffff88027396b8c0 ffff88026d1efa18 ffff880272ced480

            13:31:42.798 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: ffff880278488638 ffff88026d1effd8 000000000000fb88 ffff880278488638

            13:31:42.798 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: Call Trace:

            13:31:42.798 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa022305d>] do_get_write_access+0x29d/0x520 [jbd2]

            13:31:42.798 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff81096cc0>] ? wake_bit_function+0x0/0x50

            13:31:42.798 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa0223431>] jbd2_journal_get_write_access+0x31/0x50 [jbd2]

            13:31:42.798 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa026e708>] __ext4_journal_get_write_access+0x38/0x80 [ext4]

            13:31:42.798 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa0249b33>] ext4_reserve_inode_write+0x73/0xa0 [ext4]

            13:31:42.798 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa0249bac>] ext4_mark_inode_dirty+0x4c/0x1d0 [ext4]

            13:31:42.799 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa0222455>] ? jbd2_journal_start+0xb5/0x100 [jbd2]

            13:31:42.799 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa0249ea0>] ext4_dirty_inode+0x40/0x60 [ext4]

            13:31:42.799 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff811ac13b>] __mark_inode_dirty+0x3b/0x160

            13:31:42.799 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8119c352>] file_update_time+0xf2/0x170

            13:31:42.799 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff810a9ad9>] ? futex_wait_queue_me+0xb9/0xf0

            13:31:42.799 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8111c0b0>] __generic_file_aio_write+0x230/0x490

            13:31:42.799 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8111c398>] generic_file_aio_write+0x88/0x100

            13:31:42.799 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa0243fb1>] ext4_file_write+0x61/0x1e0 [ext4]

            13:31:42.799 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff81180c9a>] do_sync_write+0xfa/0x140

            13:31:42.799 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff811925b2>] ? user_path_at+0x62/0xa0

            13:31:42.799 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff81096c80>] ? autoremove_wake_function+0x0/0x40

            13:31:42.799 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff81186794>] ? cp_new_stat+0xe4/0x100

            13:31:42.799 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8100bb8e>] ? apic_timer_interrupt+0xe/0x20

            13:31:42.799 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8121baf6>] ? security_file_permission+0x16/0x20

            13:31:42.799 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff81180f98>] vfs_write+0xb8/0x1a0

            13:31:42.799 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff81181891>] sys_write+0x51/0x90

            13:31:42.799 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff810dc565>] ? __audit_syscall_exit+0x265/0x290

            13:31:42.799 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b

            13:31:42.799 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: INFO: task lsi_mrdsnmpagen:2936 blocked for more than 120 seconds.

            13:31:42.799 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

            13:31:42.799 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: lsi_mrdsnmpag D 0000000000000006     0  2936      1 0x00000080

            13:31:42.799 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: ffff88026a1d3c68 0000000000000086 0000000000000000 0000000000000286

            13:31:42.799 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: ffff88026a1d3bf8 ffffffffa01ca1dc 000000106a1d3c08 00000000000280d4

            13:31:42.799 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: ffff880276719af8 ffff88026a1d3fd8 000000000000fb88 ffff880276719af8

            13:31:42.799 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: Call Trace:

            13:31:42.799 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa01ca1dc>] ? megasas_get_cmd_fusion+0x5c/0x90 [megaraid_sas]

            13:31:42.800 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa01c245d>] megasas_issue_blocked_cmd+0xbd/0x190 [megaraid_sas]

            13:31:42.800 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff81096c80>] ? autoremove_wake_function+0x0/0x40

            13:31:42.800 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa01c870a>] megasas_mgmt_fw_ioctl+0x4aa/0x900 [megaraid_sas]

            13:31:42.800 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8118be35>] ? pipe_read+0x2d5/0x4e0

            13:31:42.800 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa01c8ce2>] megasas_mgmt_ioctl_fw+0x182/0x1f0 [megaraid_sas]

            13:31:42.800 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa01c8fb0>] megasas_mgmt_ioctl+0x30/0x50 [megaraid_sas]

            13:31:42.800 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff81194ed2>] vfs_ioctl+0x22/0xa0

            13:31:42.800 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff81195000>] ? do_vfs_ioctl+0x10/0x580

            13:31:42.800 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff81195074>] do_vfs_ioctl+0x84/0x580

            13:31:42.800 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff811955f1>] sys_ioctl+0x81/0xa0

            13:31:42.800 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b

            13:31:42.800 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: INFO: task java:3524 blocked for more than 120 seconds.

            13:31:42.800 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

            13:31:42.800 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: java          D 0000000000000000     0  3524   3264 0x00000080

            13:31:42.800 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: ffff880258c09c68 0000000000000082 ffff88027745b440 0000000000000286

            13:31:42.800 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: ffff880258c09bf8 ffffffffa01ca1dc 0000001081197250 00000000000280d4

            13:31:42.800 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: ffff880273e405f8 ffff880258c09fd8 000000000000fb88 ffff880273e405f8

            13:31:42.800 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: Call Trace:

            13:31:42.800 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa01ca1dc>] ? megasas_get_cmd_fusion+0x5c/0x90 [megaraid_sas]

            13:31:42.800 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff81096f6e>] ? prepare_to_wait+0x4e/0x80

            13:31:42.800 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa01c245d>] megasas_issue_blocked_cmd+0xbd/0x190 [megaraid_sas]

            13:31:42.800 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff81096c80>] ? autoremove_wake_function+0x0/0x40

            13:31:42.800 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa01c870a>] megasas_mgmt_fw_ioctl+0x4aa/0x900 [megaraid_sas]

            13:31:42.800 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8118be35>] ? pipe_read+0x2d5/0x4e0

            13:31:42.800 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa01c8ce2>] megasas_mgmt_ioctl_fw+0x182/0x1f0 [megaraid_sas]

            13:31:42.800 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa01c8fb0>] megasas_mgmt_ioctl+0x30/0x50 [megaraid_sas]

            13:31:42.800 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff81194ed2>] vfs_ioctl+0x22/0xa0

            13:31:42.801 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff81195074>] do_vfs_ioctl+0x84/0x580

            13:31:42.801 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8118173f>] ? vfs_read+0x12f/0x1a0

            13:31:42.801 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff811955f1>] sys_ioctl+0x81/0xa0

            13:31:42.801 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff810dc565>] ? __audit_syscall_exit+0x265/0x290

            13:31:42.801 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b

            13:31:42.801 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: INFO: task java:4297 blocked for more than 120 seconds.

            13:31:42.801 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

            13:31:42.801 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: java          D 0000000000000000     0  4297   4274 0x00000080

            13:31:42.801 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: ffff88026c5d78c8 0000000000000086 0000000000000000 0000000000003762

            13:31:42.801 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: ffff88026c5d7848 ffffffff81119ade 00000000000135a0 000000000020049f

            13:31:42.801 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: ffff8802748d5098 ffff88026c5d7fd8 000000000000fb88 ffff8802748d5098

            13:31:42.801 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: Call Trace:

            13:31:42.801 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff81119ade>] ? find_get_page+0x1e/0xa0

            13:31:42.801 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa022305d>] do_get_write_access+0x29d/0x520 [jbd2]

            13:31:42.801 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff81096cc0>] ? wake_bit_function+0x0/0x50

            13:31:42.801 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa0223431>] jbd2_journal_get_write_access+0x31/0x50 [jbd2]

            13:31:42.801 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa026e708>] __ext4_journal_get_write_access+0x38/0x80 [ext4]

            13:31:42.801 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa0249b33>] ext4_reserve_inode_write+0x73/0xa0 [ext4]

            13:31:42.801 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa0249bac>] ext4_mark_inode_dirty+0x4c/0x1d0 [ext4]

            13:31:42.801 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa0222455>] ? jbd2_journal_start+0xb5/0x100 [jbd2]

            13:31:42.801 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa0249ea0>] ext4_dirty_inode+0x40/0x60 [ext4]

            13:31:42.801 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff811ac13b>] __mark_inode_dirty+0x3b/0x160

            13:31:42.801 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8119c352>] file_update_time+0xf2/0x170

            13:31:42.801 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff81182ef0>] ? __sb_start_write+0x80/0x120

            13:31:42.801 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa024ad30>] ? ext4_da_get_block_prep+0x0/0x3c0 [ext4]

            13:31:42.801 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff811b810b>] __block_page_mkwrite+0x3b/0x150

            13:31:42.802 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa024ef11>] ext4_page_mkwrite+0x121/0x370 [ext4]

            13:31:42.802 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff81142d80>] do_wp_page+0x640/0x920

            13:31:42.802 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8109bee2>] ? hrtimer_cancel+0x22/0x30

            13:31:42.802 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8114385d>] handle_pte_fault+0x2cd/0xb50

            13:31:42.802 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff810aa3ce>] ? futex_wake+0x10e/0x120

            13:31:42.802 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8114431a>] handle_mm_fault+0x23a/0x310

            13:31:42.802 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff810474c9>] __do_page_fault+0x139/0x480

            13:31:42.802 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8113de80>] ? sys_madvise+0x350/0x7b0

            13:31:42.802 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8150d6e0>] ? thread_return+0x4e/0x76e

            13:31:42.802 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8151311e>] do_page_fault+0x3e/0xa0

            13:31:42.802 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff815104d5>] page_fault+0x25/0x30

            13:31:42.802 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: INFO: task java:4588 blocked for more than 120 seconds.

            13:31:42.802 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

            13:31:42.802        21      11.80     1.34  119408  54.24   25.331   46.424    0.332  167.735   38.274   0.3  90.9   9.1

            13:31:42.802 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: java          D 0000000000000002     0  4588   4274 0x00000080

            13:31:42.802 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: ffff88024628d9a8 0000000000000086 0000000000000000 ffffffff8144c76c

            13:31:42.802 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: 0000000000000000 0000000000000000 ffff88026aff8200 000000000000000e

            13:31:42.802 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: ffff880246289058 ffff88024628dfd8 000000000000fb88 ffff880246289058

            13:31:42.802 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: Call Trace:

            13:31:42.802 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8144c76c>] ? dev_queue_xmit+0x16c/0x550

            13:31:42.802 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa022305d>] do_get_write_access+0x29d/0x520 [jbd2]

            13:31:42.802 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff81096cc0>] ? wake_bit_function+0x0/0x50

            13:31:42.802 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa0223431>] jbd2_journal_get_write_access+0x31/0x50 [jbd2]

            13:31:42.802 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa026e708>] __ext4_journal_get_write_access+0x38/0x80 [ext4]

            13:31:42.802 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa0249b33>] ext4_reserve_inode_write+0x73/0xa0 [ext4]

            13:31:42.802 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa0249bac>] ext4_mark_inode_dirty+0x4c/0x1d0 [ext4]

            13:31:42.802 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa0222455>] ? jbd2_journal_start+0xb5/0x100 [jbd2]

            13:31:42.803 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa0249ea0>] ext4_dirty_inode+0x40/0x60 [ext4]

            13:31:42.803 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff811ac13b>] __mark_inode_dirty+0x3b/0x160

            13:31:42.803 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8119c352>] file_update_time+0xf2/0x170

            13:31:42.803 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8111c0b0>] __generic_file_aio_write+0x230/0x490

            13:31:42.803 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8111c398>] generic_file_aio_write+0x88/0x100

            13:31:42.803 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa0243fb1>] ext4_file_write+0x61/0x1e0 [ext4]

            13:31:42.803 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff81180c9a>] do_sync_write+0xfa/0x140

            13:31:42.803 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff81096c80>] ? autoremove_wake_function+0x0/0x40

            13:31:42.803 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8121baf6>] ? security_file_permission+0x16/0x20

            13:31:42.803 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff81180f98>] vfs_write+0xb8/0x1a0

            13:31:42.803 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff81181891>] sys_write+0x51/0x90

            13:31:42.803 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff810dc565>] ? __audit_syscall_exit+0x265/0x290

            13:31:42.803 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b

            13:31:42.803 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: INFO: task java:4589 blocked for more than 120 seconds.

            13:31:42.803 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

            13:31:42.803 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: java          D 0000000000000008     0  4589   4274 0x00000080

            13:31:42.803 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: ffff88024628f8c8 0000000000000086 0000000000000000 ffffffff81059e47

            13:31:42.803 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: ffff88024628f848 ffffffff81119ade 00000000000135a0 000000000020049f

            13:31:42.803 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: ffff8802462885f8 ffff88024628ffd8 000000000000fb88 ffff8802462885f8

            13:31:42.803 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: Call Trace:

            13:31:42.803 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff81059e47>] ? find_busiest_group+0x907/0x9f0

            13:31:42.803 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff81119ade>] ? find_get_page+0x1e/0xa0

            13:31:42.803 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa022305d>] do_get_write_access+0x29d/0x520 [jbd2]

            13:31:42.803 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff81096cc0>] ? wake_bit_function+0x0/0x50

            13:31:42.803 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa0223431>] jbd2_journal_get_write_access+0x31/0x50 [jbd2]

            13:31:42.803 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa026e708>] __ext4_journal_get_write_access+0x38/0x80 [ext4]

            13:31:42.804 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa0249b33>] ext4_reserve_inode_write+0x73/0xa0 [ext4]

            13:31:42.804 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa0249bac>] ext4_mark_inode_dirty+0x4c/0x1d0 [ext4]

            13:31:42.804 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa0222455>] ? jbd2_journal_start+0xb5/0x100 [jbd2]

            13:31:42.804 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa0249ea0>] ext4_dirty_inode+0x40/0x60 [ext4]

            13:31:42.804 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff811ac13b>] __mark_inode_dirty+0x3b/0x160

            13:31:42.804 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8119c352>] file_update_time+0xf2/0x170

            13:31:42.804 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff81182ef0>] ? __sb_start_write+0x80/0x120

            13:31:42.804 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa024ad30>] ? ext4_da_get_block_prep+0x0/0x3c0 [ext4]

            13:31:42.804 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff811b810b>] __block_page_mkwrite+0x3b/0x150

            13:31:42.804 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa024ef11>] ext4_page_mkwrite+0x121/0x370 [ext4]

            13:31:42.804 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff81142d80>] do_wp_page+0x640/0x920

            13:31:42.804 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8109bee2>] ? hrtimer_cancel+0x22/0x30

            13:31:42.804 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8114385d>] handle_pte_fault+0x2cd/0xb50

            13:31:42.804 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff810aa3ce>] ? futex_wake+0x10e/0x120

            13:31:42.804 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8114431a>] handle_mm_fault+0x23a/0x310

            13:31:42.804 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff810474c9>] __do_page_fault+0x139/0x480

            13:31:42.804 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8143766b>] ? sys_recvfrom+0x16b/0x180

            13:31:42.804 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8100988e>] ? __switch_to+0x26e/0x320

            13:31:42.804 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8100bb8e>] ? apic_timer_interrupt+0xe/0x20

            13:31:42.804 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8151311e>] do_page_fault+0x3e/0xa0

            13:31:42.804 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff815104d5>] page_fault+0x25/0x30

            13:31:42.804 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: INFO: task java:4562 blocked for more than 120 seconds.

            13:31:42.804 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

            13:31:42.804 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: java          D 000000000000000a     0  4562   4378 0x00000080

            13:31:42.804 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: ffff880246053a98 0000000000000086 0000000000000000 ffffffffa024849f

            13:31:42.804 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: ffff88027643f938 ffff88027643f848 ffff880274060378 ffff880274060378

            13:31:42.805 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: ffff880246047058 ffff880246053fd8 000000000000fb88 ffff880246047058

            13:31:42.805 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: Call Trace:

            13:31:42.805 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa024849f>] ? ext4_mark_iloc_dirty+0x35f/0x5a0 [ext4]

            13:31:42.805 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff81119cd0>] ? sync_page+0x0/0x50

            13:31:42.805 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8150de73>] io_schedule+0x73/0xc0

            13:31:42.805 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff81119d0d>] sync_page+0x3d/0x50

            13:31:42.805 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8150e82f>] __wait_on_bit+0x5f/0x90

            13:31:42.805 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff811b5a91>] ? mark_buffer_dirty+0x61/0xa0

            13:31:42.805 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff81119f43>] wait_on_page_bit+0x73/0x80

            13:31:42.805 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff81096cc0>] ? wake_bit_function+0x0/0x50

            13:31:42.805 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff811b81f3>] __block_page_mkwrite+0x123/0x150

            13:31:42.805 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa024ef11>] ext4_page_mkwrite+0x121/0x370 [ext4]

            13:31:42.805 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff81142d80>] do_wp_page+0x640/0x920

            13:31:42.805 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8109bee2>] ? hrtimer_cancel+0x22/0x30

            13:31:42.805 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8114385d>] handle_pte_fault+0x2cd/0xb50

            13:31:42.805 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff810aa3ce>] ? futex_wake+0x10e/0x120

            13:31:42.805 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8114431a>] handle_mm_fault+0x23a/0x310

            13:31:42.805 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff810474c9>] __do_page_fault+0x139/0x480

            13:31:42.805 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8113de80>] ? sys_madvise+0x350/0x7b0

            13:31:42.805 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8100bb8e>] ? apic_timer_interrupt+0xe/0x20

            13:31:42.805 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8151311e>] do_page_fault+0x3e/0xa0

            13:31:42.805 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff815104d5>] page_fault+0x25/0x30

            13:31:42.805 /var/log/messages: Jun 25 13:31:41 svt-smx9dri-002 kernel: megasas: FW now in Ready state

            13:31:42.805 /var/log/messages: Jun 25 13:31:41 svt-smx9dri-002 kernel: megasas:IOC Init cmd success

            13:31:42.805 /var/log/messages: Jun 25 13:31:41 svt-smx9dri-002 kernel: megaraid_sas: Reset successful.

            13:31:42.806 /var/log/messages: Jun 25 13:31:41 svt-smx9dri-002 MR_MONITOR[3268]: <MRMON045> Controller ID:  0   User entered firmware debugger

            13:31:42.806 /var/log/messages: Jun 25 13:31:41 svt-smx9dri-002 MR_MONITOR[3268]: <MRMON000> Controller ID:  0   Firmware initialization started:  #012    ( PCI ID   0x5d/ 0x1000/ 0x9381    / 0x1000)

            13:31:42.806 /var/log/messages: Jun 25 13:31:41 svt-smx9dri-002 MR_MONITOR[3268]: <MRMON001> Controller ID:  0   Image version:   4.230.20-3532

            13:31:42.806 /var/log/messages: Jun 25 13:31:41 svt-smx9dri-002 MR_MONITOR[3268]: <MRMON345> Controller ID:  0  Controller encountered a fatal error and was reset

            13:31:42.806 /var/log/messages: Jun 25 13:31:41 svt-smx9dri-002 MR_MONITOR[3268]: <MRMON141> Controller ID:  0   Battery Present

            13:31:42.806 /var/log/messages: Jun 25 13:31:41 svt-smx9dri-002 MR_MONITOR[3268]: <MRMON261> Controller ID:  0  Package version  #012    24.3.0-0050

            13:31:42.806 /var/log/messages: Jun 25 13:31:41 svt-smx9dri-002 MR_MONITOR[3268]: <MRMON266> Controller ID:  0  Board Revision:   02A

            13:31:42.806 /var/log/messages: Jun 25 13:31:41 svt-smx9dri-002 MR_MONITOR[3268]: <MRMON242> Controller ID:  0   Battery charge complete

            13:31:42.806 /var/log/messages: Jun 25 13:31:41 svt-smx9dri-002 MR_MONITOR[3268]: <MRMON149> Controller ID:  0   Battery temperature is normal

            13:31:42.806 /var/log/messages: Jun 25 13:31:41 svt-smx9dri-002 MR_MONITOR[3268]: <MRMON268> Controller ID:  0  PD Reset:   PD  #012    =   -:-:0,   Critical  #012    =   3,   Path   =#012      0x50014EE3AAB7AC86

            13:31:42.806 /var/log/messages: Jun 25 13:31:41 svt-smx9dri-002 MR_MONITOR[3268]: <MRMON268> Controller ID:  0  PD Reset:   PD  #012    =   -:-:0,   Critical  #012    =   3,   Path   =#012      0x50014EE3AAB7AC86

            13:31:42.806 /var/log/messages: Jun 25 13:31:41 svt-smx9dri-002 MR_MONITOR[3268]: <MRMON268> Controller ID:  0  PD Reset:   PD  #012    =   -:-:0,   Critical  #012    =   3,   Path   =#012      0x50014EE3AAB7AC86

            13:31:42.806 /var/log/messages: Jun 25 13:31:41 svt-smx9dri-002 MR_MONITOR[3268]: <MRMON268> Controller ID:  0  PD Reset:   PD  #012    =   -:-:0,   Critical  #012    =   3,   Path   =#012      0x50014EE3AAB7AC86

            13:31:42.806 /var/log/messages: Jun 25 13:31:41 svt-smx9dri-002 MR_MONITOR[3268]: <MRMON268> Controller ID:  0  PD Reset:   PD  #012    =   -:-:0,   Critical  #012    =   3,   Path   =#012      0x50014EE3AAB7AC86

            13:31:42.806 /var/log/messages: Jun 25 13:31:41 svt-smx9dri-002 MR_MONITOR[3268]: <MRMON164> Controller ID:  0   SES enclosure discovered:  #012    1

            13:31:42.806 /var/log/messages: Jun 25 13:31:41 svt-smx9dri-002 MR_MONITOR[3268]: <MRMON164> Controller ID:  0   SES enclosure discovered:  #012    1

            13:31:42.806 /var/log/messages: Jun 25 13:31:41 svt-smx9dri-002 MR_MONITOR[3268]: <MRMON167> Controller ID:  0   Communication restored on enclosure:  #012    1

            13:31:42.806

            13:31:42.826        22       6.60     0.77  122135  66.67   49.385   73.890    0.375  207.068   59.421   0.2   NaN   NaN

            13:31:42.849        23      20.20     1.99  103494  49.50   26.485   53.031    0.460  245.271   42.993   0.6  81.8   0.0

            13:31:42.868        24      29.00     3.29  119094  44.83   22.083   48.676    0.476  271.109   40.393   0.6   NaN   NaN

            13:31:42.887        25      18.20     2.14  123150  49.45   25.983   52.254    0.283  175.378   39.733   0.6 100.0   8.9

            13:31:42.905        26      68.00     7.11  109628  48.82   26.658   54.320    0.269  349.416   48.685   1.7  98.3   8.6

            13:31:42.923        27      76.00     7.99  110203  48.95   24.312   49.373    0.284  286.698   41.739   1.8  94.4  10.3

            13:31:42.937        28      95.60    10.29  112837  48.12   23.603   48.674    0.351  286.698   40.433   2.3  94.4  10.3

            13:31:42.951        29     118.40    12.78  113221  47.47   24.250   50.727    0.328  371.895   42.791   2.8  94.4  10.3

            13:31:42.964        30     119.20    12.91  113560  47.48   24.160   50.519    0.327  371.895   42.670   2.9  94.4  10.3

             

             

            Jun 25, 2014  interval        i/o   MB/sec   bytes   read     resp     read    write     resp     resp queue  cpu%  cpu%

                                         rate  1024**2     i/o    pct     time     resp     resp      max   stddev depth sys+u   sys

            13:31:42.977        31     135.00    14.68  114056  47.56   24.651   51.483    0.320  371.895   42.754   3.4  94.4  10.3

            13:31:42.990        32     140.00    15.45  115694  47.57   24.563   51.288    0.314  371.895   42.357   3.5  94.4  10.3

            13:31:43.008        33     152.00    16.79  115809  47.89   24.733   51.248    0.360  371.895   42.472   3.9  94.4  10.3

            13:31:43.571 Debug_cmds(); stdout 13:31:43.570 home: /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.9.x86_64/jre

            13:31:43.572 Debug_cmds(); stdout 13:31:43.572 home: /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.9.x86_64

            13:31:43.572 Debug_cmds(); stdout 13:31:43.572 ==> where: Vdb.Jstack.main(Jstack.java:398)

            13:31:43.572 Debug_cmds(); stdout 13:31:43.572 ==> where: Vdb.Jstack.main(Jstack.java:401)

            13:31:43.572 Debug_cmds(); stdout 13:31:43.572 jstack: null

            13:31:43.572 Debug_cmds(); stdout 13:31:43.572 jstack: null

            13:31:43.573 Debug_cmds(); stdout 13:31:43.572 execute(): ps -ef

            13:31:43.609 Debug_cmds(); stdout 13:31:43.609 program: ../jre/bin/java -classpath ../jre/lib/rt.jar:../jre/lib/jsse.jar:../jre/lib/jce.jar:mail.jar:framework.jar -djava.library.path=. framework.frameworkmanager

            13:31:43.609 Debug_cmds(); stdout 13:31:43.609 program: ../jre/bin/java -classpath ../jre/lib/rt.jar:../jre/lib/jsse.jar:../jre/lib/jce.jar:mail.jar:framework.jar -djava.library.path=. framework.frameworkmanager

            13:31:43.609 Debug_cmds(); stdout 13:31:43.609 program: ../jre/bin/java -classpath .:popup.jar:../gui.jar popup.communicator ajsgyqkj=71244

            13:31:43.610 Debug_cmds(); stdout 13:31:43.610 program: java -client -xmx512m -xms64m -cp /opt/vdbench/:/opt/vdbench/classes:/opt/vdbench/vdbench.jar:/opt/vdbench/jaramiko.jar vdb.vdbmain -m 8 -f /var/log/validator-logs/vdbench-config.140625.132608 -o /var/log/validator-logs/vdbench-output.140625.132608 -e 1800

            13:31:43.611 Debug_cmds(); stdout 13:31:43.611 program: java -client -xmx1024m -xms128m -cp /opt/vdbench/:/opt/vdbench/classes:/opt/vdbench/vdbench.jar:/opt/vdbench/jaramiko.jar vdb.slavejvm slavejvm -m localhost -n localhost-11-140625-13.26.18.930 -l localhost-1 -p 5570

            13:31:43.611 Debug_cmds(); stdout 13:31:43.611 program: java -client -xmx1024m -xms128m -cp /opt/vdbench/:/opt/vdbench/classes:/opt/vdbench/vdbench.jar:/opt/vdbench/jaramiko.jar vdb.slavejvm slavejvm -m localhost -n localhost-12-140625-13.26.18.930 -l localhost-2 -p 5570

            13:31:43.611 Debug_cmds(); stdout 13:31:43.611 program: java -client -xmx1024m -xms128m -cp /opt/vdbench/:/opt/vdbench/classes:/opt/vdbench/vdbench.jar:/opt/vdbench/jaramiko.jar vdb.slavejvm slavejvm -m localhost -n localhost-10-140625-13.26.18.930 -l localhost-0 -p 5570

            13:31:43.611 Debug_cmds(); stdout 13:31:43.611 program: java -client -xmx1024m -xms128m -cp /opt/vdbench/:/opt/vdbench/classes:/opt/vdbench/vdbench.jar:/opt/vdbench/jaramiko.jar vdb.slavejvm slavejvm -m localhost -n localhost-13-140625-13.26.18.930 -l localhost-3 -p 5570

            13:31:43.611 Debug_cmds(); stdout 13:31:43.611 program: java -client -xmx1024m -xms128m -cp /opt/vdbench/:/opt/vdbench/classes:/opt/vdbench/vdbench.jar:/opt/vdbench/jaramiko.jar vdb.slavejvm slavejvm -m localhost -n localhost-14-140625-13.26.18.930 -l localhost-4 -p 5570

            13:31:43.612 Debug_cmds(); stdout 13:31:43.612 program: java -client -xmx1024m -xms128m -cp /opt/vdbench/:/opt/vdbench/classes:/opt/vdbench/vdbench.jar:/opt/vdbench/jaramiko.jar vdb.slavejvm slavejvm -m localhost -n localhost-15-140625-13.26.18.930 -l localhost-5 -p 5570

            13:31:43.612 Debug_cmds(); stdout 13:31:43.612 program: java -client -xmx1024m -xms128m -cp /opt/vdbench/:/opt/vdbench/classes:/opt/vdbench/vdbench.jar:/opt/vdbench/jaramiko.jar vdb.slavejvm slavejvm -m localhost -n localhost-16-140625-13.26.18.930 -l localhost-6 -p 5570

            13:31:43.612 Debug_cmds(); stdout 13:31:43.612 program: java -client -xmx1024m -xms128m -cp /opt/vdbench/:/opt/vdbench/classes:/opt/vdbench/vdbench.jar:/opt/vdbench/jaramiko.jar vdb.slavejvm slavejvm -m localhost -n localhost-17-140625-13.26.18.930 -l localhost-7 -p 5570

            13:31:43.612 Debug_cmds(); stdout 13:31:43.612

            13:31:43.612 Debug_cmds(); stdout 13:31:43.612 execute(): null 3268

            13:31:43.615 Debug_cmds(); stdout 13:31:43.614 Jstatus for 'other' 3268 ../jre/bin/java -classpath ../jre/lib/rt.jar:../jre/lib/jsse.jar:../jre/lib/jce.jar:mail.jar:framework.jar -djava.library.path=. framework.frameworkmanager

            13:31:43.615 Debug_cmds(); stdout 13:31:43.615 stderr: null: Command not found.

            13:31:43.615 Debug_cmds(); stdout 13:31:43.615

            13:31:43.615 Debug_cmds(); stdout 13:31:43.615 execute(): null 3523

            13:31:43.617 Debug_cmds(); stdout 13:31:43.617 Jstatus for 'other' 3523 ../jre/bin/java -classpath ../jre/lib/rt.jar:../jre/lib/jsse.jar:../jre/lib/jce.jar:mail.jar:framework.jar -djava.library.path=. framework.frameworkmanager

            13:31:43.617 Debug_cmds(); stdout 13:31:43.617 stderr: null: Command not found.

            13:31:43.617 Debug_cmds(); stdout 13:31:43.617

            13:31:43.617 Debug_cmds(); stdout 13:31:43.617 execute(): null 3527

            13:31:43.619 Debug_cmds(); stdout 13:31:43.619 Jstatus for 'other' 3527 ../jre/bin/java -classpath .:popup.jar:../gui.jar popup.communicator ajsgyqkj=71244

            13:31:43.619 Debug_cmds(); stdout 13:31:43.619 stderr: null: Command not found.

            13:31:43.619 Debug_cmds(); stdout 13:31:43.619

            13:31:43.619 Debug_cmds(); stdout 13:31:43.619 execute(): null 4278

            13:31:43.621 Debug_cmds(); stdout 13:31:43.621 Jstatus for 'other' 4278 java -client -xmx512m -xms64m -cp /opt/vdbench/:/opt/vdbench/classes:/opt/vdbench/vdbench.jar:/opt/vdbench/jaramiko.jar vdb.vdbmain -m 8 -f /var/log/validator-logs/vdbench-config.140625.132608 -o /var/log/validator-logs/vdbench-output.140625.132608 -e 1800

            13:31:43.622 Debug_cmds(); stdout 13:31:43.621 stderr: null: Command not found.

            13:31:43.622 Debug_cmds(); stdout 13:31:43.622

            13:31:43.622 Debug_cmds(); stdout 13:31:43.622 execute(): null 4412

            13:31:43.624 Debug_cmds(); stdout 13:31:43.624 Jstatus for 'other' 4412 java -client -xmx1024m -xms128m -cp /opt/vdbench/:/opt/vdbench/classes:/opt/vdbench/vdbench.jar:/opt/vdbench/jaramiko.jar vdb.slavejvm slavejvm -m localhost -n localhost-11-140625-13.26.18.930 -l localhost-1 -p 5570

            13:31:43.624 Debug_cmds(); stdout 13:31:43.624 stderr: null: Command not found.

            13:31:43.624 Debug_cmds(); stdout 13:31:43.624

            13:31:43.624 Debug_cmds(); stdout 13:31:43.624 execute(): null 4413

            13:31:43.626 Debug_cmds(); stdout 13:31:43.626 Jstatus for 'other' 4413 java -client -xmx1024m -xms128m -cp /opt/vdbench/:/opt/vdbench/classes:/opt/vdbench/vdbench.jar:/opt/vdbench/jaramiko.jar vdb.slavejvm slavejvm -m localhost -n localhost-12-140625-13.26.18.930 -l localhost-2 -p 5570

            13:31:43.627 Debug_cmds(); stdout 13:31:43.626 stderr: null: Command not found.

            13:31:43.627 Debug_cmds(); stdout 13:31:43.627

            13:31:43.627 Debug_cmds(); stdout 13:31:43.627 execute(): null 4414

            13:31:43.629 Debug_cmds(); stdout 13:31:43.629 Jstatus for 'other' 4414 java -client -xmx1024m -xms128m -cp /opt/vdbench/:/opt/vdbench/classes:/opt/vdbench/vdbench.jar:/opt/vdbench/jaramiko.jar vdb.slavejvm slavejvm -m localhost -n localhost-10-140625-13.26.18.930 -l localhost-0 -p 5570

            13:31:43.629 Debug_cmds(); stdout 13:31:43.629 stderr: null: Command not found.

            13:31:43.629 Debug_cmds(); stdout 13:31:43.629

            13:31:43.629 Debug_cmds(); stdout 13:31:43.629 execute(): null 4416

            13:31:44.634 Debug_cmds(); stdout 13:31:44.634 Jstatus for 'other' 4416 java -client -xmx1024m -xms128m -cp /opt/vdbench/:/opt/vdbench/classes:/opt/vdbench/vdbench.jar:/opt/vdbench/jaramiko.jar vdb.slavejvm slavejvm -m localhost -n localhost-13-140625-13.26.18.930 -l localhost-3 -p 5570

            13:31:44.634 Debug_cmds(); stdout 13:31:44.634 stderr: null: Command not found.

            13:31:44.634 Debug_cmds(); stdout 13:31:44.634

            13:31:44.635 Debug_cmds(); stdout 13:31:44.634 execute(): null 4418

            13:31:44.638 Debug_cmds(); stdout 13:31:44.638 Jstatus for 'other' 4418 java -client -xmx1024m -xms128m -cp /opt/vdbench/:/opt/vdbench/classes:/opt/vdbench/vdbench.jar:/opt/vdbench/jaramiko.jar vdb.slavejvm slavejvm -m localhost -n localhost-14-140625-13.26.18.930 -l localhost-4 -p 5570

            13:31:44.638 Debug_cmds(); stdout 13:31:44.638 stderr: null: Command not found.

            13:31:44.638 Debug_cmds(); stdout 13:31:44.638

            13:31:44.639 Debug_cmds(); stdout 13:31:44.638 execute(): null 4419

            13:31:44.642 Debug_cmds(); stdout 13:31:44.641 Jstatus for 'other' 4419 java -client -xmx1024m -xms128m -cp /opt/vdbench/:/opt/vdbench/classes:/opt/vdbench/vdbench.jar:/opt/vdbench/jaramiko.jar vdb.slavejvm slavejvm -m localhost -n localhost-15-140625-13.26.18.930 -l localhost-5 -p 5570

            13:31:44.642 Debug_cmds(); stdout 13:31:44.642 stderr: null: Command not found.

            13:31:44.642 Debug_cmds(); stdout 13:31:44.642

            13:31:44.642 Debug_cmds(); stdout 13:31:44.642 execute(): null 4424

            13:31:44.645 Debug_cmds(); stdout 13:31:44.645 Jstatus for 'other' 4424 java -client -xmx1024m -xms128m -cp /opt/vdbench/:/opt/vdbench/classes:/opt/vdbench/vdbench.jar:/opt/vdbench/jaramiko.jar vdb.slavejvm slavejvm -m localhost -n localhost-16-140625-13.26.18.930 -l localhost-6 -p 5570

            13:31:44.645 Debug_cmds(); stdout 13:31:44.645 stderr: null: Command not found.

            13:31:44.645 Debug_cmds(); stdout 13:31:44.645

            13:31:44.646 Debug_cmds(); stdout 13:31:44.645 execute(): null 4425

            13:31:45.649 Debug_cmds(); stdout 13:31:45.649 Jstatus for 'other' 4425 java -client -xmx1024m -xms128m -cp /opt/vdbench/:/opt/vdbench/classes:/opt/vdbench/vdbench.jar:/opt/vdbench/jaramiko.jar vdb.slavejvm slavejvm -m localhost -n localhost-17-140625-13.26.18.930 -l localhost-7 -p 5570

            13:31:45.649 Debug_cmds(); stdout 13:31:45.649 stderr: null: Command not found.

            13:31:45.649 Debug_cmds(); stdout 13:31:45.649 Jstack completed successfully

            13:31:45.655 Memory total Java heap:   69.562 MB; Free:   52.830 MB; Used:   16.732 MB;

            13:31:45.655 VdbCount statistics for Heartbeat:

            13:31:45.655 Java heap:   69.563 /   52.830 /   16.732

            13:31:45.655 Class                               added   cloned   serial   finals     left

            13:31:45.656 VdbObject.finalize() not active!

            13:31:45.656 tg: java.lang.ThreadGroup[name=main,maxpri=10]

            13:31:45.656 tg.get_name: main

            13:31:45.656 tg.activeCount: 35

            13:31:45.661 tl            [Ljava.lang.Thread;@8a7d8b0

            13:31:45.661 tl[]:         Thread[Vdbmain,5,main]

            13:31:45.661 tl[].isalive: true

            13:31:45.661 Threadname:   Vdbmain

            13:31:45.661 dumpOneStack: java.lang.Thread -2

            13:31:45.661 dumpOneStack: java.lang.Thread 338

            13:31:45.661 dumpOneStack: Vdb.common 376

            13:31:45.661 dumpOneStack: Vdb.common 365

            13:31:45.662 dumpOneStack: Vdb.Vdbmain 800

            13:31:45.662 dumpOneStack: Vdb.Vdbmain 599

            13:31:45.662 tl            [Ljava.lang.Thread;@8a7d8b0

            13:31:45.662 tl[]:         Thread[SlaveStarter localhost-10-140625-13.26.18.930,5,main]

            13:31:45.662 tl[].isalive: true

            13:31:45.662 Threadname:   SlaveStarter localhost-10-140625-13.26.18.930

            13:31:45.662 dumpOneStack: java.lang.Object -2

            13:31:45.662 dumpOneStack: java.lang.Object 503

            13:31:45.662 dumpOneStack: java.lang.UNIXProcess 210

            13:31:45.662 dumpOneStack: Utils.OS_cmd 319

            13:31:45.662 dumpOneStack: Utils.OS_cmd 250

            13:31:45.662 dumpOneStack: Vdb.SlaveStarter 220

            13:31:45.663 dumpOneStack: Vdb.SlaveStarter 169

            13:31:45.663 dumpOneStack: Vdb.SlaveStarter 68

            13:31:45.663 tl            [Ljava.lang.Thread;@8a7d8b0

            13:31:45.663 tl[]:         Thread[SlaveStarter localhost-11-140625-13.26.18.930,5,main]

            13:31:45.663 tl[].isalive: true

            13:31:45.663 Threadname:   SlaveStarter localhost-11-140625-13.26.18.930

            13:31:45.663 dumpOneStack: java.lang.Object -2

            13:31:45.663 dumpOneStack: java.lang.Object 503

            13:31:45.663 dumpOneStack: java.lang.UNIXProcess 210

            13:31:45.663 dumpOneStack: Utils.OS_cmd 319

            13:31:45.663 dumpOneStack: Utils.OS_cmd 250

            13:31:45.664 dumpOneStack: Vdb.SlaveStarter 220

            13:31:45.664 dumpOneStack: Vdb.SlaveStarter 169

            13:31:45.664 dumpOneStack: Vdb.SlaveStarter 68

            13:31:45.664 tl            [Ljava.lang.Thread;@8a7d8b0

            13:31:45.664 tl[]:         Thread[SlaveStarter localhost-12-140625-13.26.18.930,5,main]

            13:31:45.664 tl[].isalive: true

            13:31:45.664 Threadname:   SlaveStarter localhost-12-140625-13.26.18.930

            13:31:45.664 dumpOneStack: java.lang.Object -2

            13:31:45.664 dumpOneStack: java.lang.Object 503

            13:31:45.664 dumpOneStack: java.lang.UNIXProcess 210

            13:31:45.664 dumpOneStack: Utils.OS_cmd 319

            13:31:45.664 dumpOneStack: Utils.OS_cmd 250

            13:31:45.664 dumpOneStack: Vdb.SlaveStarter 220

            13:31:45.665 dumpOneStack: Vdb.SlaveStarter 169

            13:31:45.665 dumpOneStack: Vdb.SlaveStarter 68

            13:31:45.665 tl            [Ljava.lang.Thread;@8a7d8b0

            13:31:45.665 tl[]:         Thread[SlaveStarter localhost-13-140625-13.26.18.930,5,main]

            13:31:45.665 tl[].isalive: true

            13:31:45.665 Threadname:   SlaveStarter localhost-13-140625-13.26.18.930

            13:31:45.665 dumpOneStack: java.lang.Object -2

            13:31:45.665 dumpOneStack: java.lang.Object 503

            13:31:45.665 dumpOneStack: java.lang.UNIXProcess 210

            13:31:45.665 dumpOneStack: Utils.OS_cmd 319

            13:31:45.665 dumpOneStack: Utils.OS_cmd 250

            13:31:45.666 dumpOneStack: Vdb.SlaveStarter 220

            13:31:45.666 dumpOneStack: Vdb.SlaveStarter 169

            13:31:45.666 dumpOneStack: Vdb.SlaveStarter 68

            13:31:45.666 tl            [Ljava.lang.Thread;@8a7d8b0

            13:31:45.666 tl[]:         Thread[SlaveStarter localhost-14-140625-13.26.18.930,5,main]

            13:31:45.666 tl[].isalive: true

            13:31:45.666 Threadname:   SlaveStarter localhost-14-140625-13.26.18.930

            13:31:45.666 dumpOneStack: java.lang.Object -2

            13:31:45.666 dumpOneStack: java.lang.Object 503

            13:31:45.666 dumpOneStack: java.lang.UNIXProcess 210

            13:31:45.666 dumpOneStack: Utils.OS_cmd 319

            13:31:45.666 dumpOneStack: Utils.OS_cmd 250

            13:31:45.667 dumpOneStack: Vdb.SlaveStarter 220

            13:31:45.667 dumpOneStack: Vdb.SlaveStarter 169

            13:31:45.667 dumpOneStack: Vdb.SlaveStarter 68

            13:31:45.667 tl            [Ljava.lang.Thread;@8a7d8b0

            13:31:45.667 tl[]:         Thread[SlaveStarter localhost-15-140625-13.26.18.930,5,main]

            13:31:45.667 tl[].isalive: true

            13:31:45.667 Threadname:   SlaveStarter localhost-15-140625-13.26.18.930

            13:31:45.667 dumpOneStack: java.lang.Object -2

            13:31:45.667 dumpOneStack: java.lang.Object 503

            13:31:45.667 dumpOneStack: java.lang.UNIXProcess 210

            13:31:45.667 dumpOneStack: Utils.OS_cmd 319

            13:31:45.667 dumpOneStack: Utils.OS_cmd 250

            13:31:45.668 dumpOneStack: Vdb.SlaveStarter 220

            13:31:45.668 dumpOneStack: Vdb.SlaveStarter 169

            13:31:45.668 dumpOneStack: Vdb.SlaveStarter 68

            13:31:45.668 tl            [Ljava.lang.Thread;@8a7d8b0

            13:31:45.668 tl[]:         Thread[Get_cmd_stream stderr /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-11-140625-13.26.18.930 -l localhost-1 -p 5570,5,main]

            13:31:45.668 tl[].isalive: true

            13:31:45.668 Threadname:   Get_cmd_stream stderr /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-11-140625-13.26.18.930 -l localhost-1 -p 5570

            13:31:45.668 dumpOneStack: java.io.FileInputStream -2

            13:31:45.668 dumpOneStack: java.io.FileInputStream 242

            13:31:45.668 dumpOneStack: java.io.BufferedInputStream 273

            13:31:45.668 dumpOneStack: java.io.BufferedInputStream 334

            13:31:45.669 dumpOneStack: sun.nio.cs.StreamDecoder 283

            13:31:45.669 dumpOneStack: sun.nio.cs.StreamDecoder 325

            13:31:45.669 dumpOneStack: sun.nio.cs.StreamDecoder 177

            13:31:45.669 dumpOneStack: java.io.InputStreamReader 184

            13:31:45.669 dumpOneStack: java.io.BufferedReader 154

            13:31:45.669 dumpOneStack: java.io.BufferedReader 317

            13:31:45.669 dumpOneStack: java.io.BufferedReader 382

            13:31:45.669 dumpOneStack: Utils.Get_cmd_stream 679

            13:31:45.669 tl            [Ljava.lang.Thread;@8a7d8b0

            13:31:45.669 tl[]:         Thread[Get_cmd_stream stdout /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-11-140625-13.26.18.930 -l localhost-1 -p 5570,5,main]

            13:31:45.669 tl[].isalive: true

            13:31:45.669 Threadname:   Get_cmd_stream stdout /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-11-140625-13.26.18.930 -l localhost-1 -p 5570

            13:31:45.669 dumpOneStack: java.io.FileInputStream -2

            13:31:45.670 dumpOneStack: java.io.FileInputStream 242

            13:31:45.670 dumpOneStack: java.io.BufferedInputStream 273

            13:31:45.670 dumpOneStack: java.io.BufferedInputStream 334

            13:31:45.670 dumpOneStack: sun.nio.cs.StreamDecoder 283

            13:31:45.670 dumpOneStack: sun.nio.cs.StreamDecoder 325

            13:31:45.670 dumpOneStack: sun.nio.cs.StreamDecoder 177

            13:31:45.670 dumpOneStack: java.io.InputStreamReader 184

            13:31:45.670 dumpOneStack: java.io.BufferedReader 154

            13:31:45.670 dumpOneStack: java.io.BufferedReader 317

            13:31:45.670 dumpOneStack: java.io.BufferedReader 382

            13:31:45.670 dumpOneStack: Utils.Get_cmd_stream 679

            13:31:45.670 tl            [Ljava.lang.Thread;@8a7d8b0

            13:31:45.670 tl[]:         Thread[SlaveStarter localhost-16-140625-13.26.18.930,5,main]

            13:31:45.670 tl[].isalive: true

            13:31:45.671 Threadname:   SlaveStarter localhost-16-140625-13.26.18.930

            13:31:45.671 dumpOneStack: java.lang.Object -2

            13:31:45.671 dumpOneStack: java.lang.Object 503

            13:31:45.671 dumpOneStack: java.lang.UNIXProcess 210

            13:31:45.671 dumpOneStack: Utils.OS_cmd 319

            13:31:45.671 dumpOneStack: Utils.OS_cmd 250

            13:31:45.671 dumpOneStack: Vdb.SlaveStarter 220

            13:31:45.671 dumpOneStack: Vdb.SlaveStarter 169

            13:31:45.671 dumpOneStack: Vdb.SlaveStarter 68

            13:31:45.671 tl            [Ljava.lang.Thread;@8a7d8b0

            13:31:45.671 tl[]:         Thread[Get_cmd_stream stderr /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-12-140625-13.26.18.930 -l localhost-2 -p 5570,5,main]

            13:31:45.671 tl[].isalive: true

            13:31:45.672 Threadname:   Get_cmd_stream stderr /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-12-140625-13.26.18.930 -l localhost-2 -p 5570

            13:31:45.672 dumpOneStack: java.io.FileInputStream -2

            13:31:45.672 dumpOneStack: java.io.FileInputStream 242

            13:31:45.672 dumpOneStack: java.io.BufferedInputStream 273

            13:31:45.672 dumpOneStack: java.io.BufferedInputStream 334

            13:31:45.672 dumpOneStack: sun.nio.cs.StreamDecoder 283

            13:31:45.672 dumpOneStack: sun.nio.cs.StreamDecoder 325

            13:31:45.672 dumpOneStack: sun.nio.cs.StreamDecoder 177

            13:31:45.672 dumpOneStack: java.io.InputStreamReader 184

            13:31:45.672 dumpOneStack: java.io.BufferedReader 154

            13:31:45.672 dumpOneStack: java.io.BufferedReader 317

            13:31:45.672 dumpOneStack: java.io.BufferedReader 382

            13:31:45.673 dumpOneStack: Utils.Get_cmd_stream 679

            13:31:45.673 tl            [Ljava.lang.Thread;@8a7d8b0

            13:31:45.673 tl[]:         Thread[Get_cmd_stream stdout /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-12-140625-13.26.18.930 -l localhost-2 -p 5570,5,main]

            13:31:45.673 tl[].isalive: true

            13:31:45.673 Threadname:   Get_cmd_stream stdout /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-12-140625-13.26.18.930 -l localhost-2 -p 5570

            13:31:45.673 dumpOneStack: java.io.FileInputStream -2

            13:31:45.673 dumpOneStack: java.io.FileInputStream 242

            13:31:45.673 dumpOneStack: java.io.BufferedInputStream 273

            13:31:45.673 dumpOneStack: java.io.BufferedInputStream 334

            13:31:45.673 dumpOneStack: sun.nio.cs.StreamDecoder 283

            13:31:45.673 dumpOneStack: sun.nio.cs.StreamDecoder 325

            13:31:45.673 dumpOneStack: sun.nio.cs.StreamDecoder 177

            13:31:45.673 dumpOneStack: java.io.InputStreamReader 184

            13:31:45.673 dumpOneStack: java.io.BufferedReader 154

            13:31:45.674 dumpOneStack: java.io.BufferedReader 317

            13:31:45.674 dumpOneStack: java.io.BufferedReader 382

            13:31:45.674 dumpOneStack: Utils.Get_cmd_stream 679

            13:31:45.674 tl            [Ljava.lang.Thread;@8a7d8b0

            13:31:45.674 tl[]:         Thread[Get_cmd_stream stderr /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-14-140625-13.26.18.930 -l localhost-4 -p 5570,5,main]

            13:31:45.674 tl[].isalive: true

            13:31:45.674 Threadname:   Get_cmd_stream stderr /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-14-140625-13.26.18.930 -l localhost-4 -p 5570

            13:31:45.674 dumpOneStack: java.io.FileInputStream -2

            13:31:45.674 dumpOneStack: java.io.FileInputStream 242

            13:31:45.674 dumpOneStack: java.io.BufferedInputStream 273

            13:31:45.674 dumpOneStack: java.io.BufferedInputStream 334

            13:31:45.675 dumpOneStack: sun.nio.cs.StreamDecoder 283

            13:31:45.675 dumpOneStack: sun.nio.cs.StreamDecoder 325

            13:31:45.675 dumpOneStack: sun.nio.cs.StreamDecoder 177

            13:31:45.675 dumpOneStack: java.io.InputStreamReader 184

            13:31:45.675 dumpOneStack: java.io.BufferedReader 154

            13:31:45.675 dumpOneStack: java.io.BufferedReader 317

            13:31:45.675 dumpOneStack: java.io.BufferedReader 382

            13:31:45.675 dumpOneStack: Utils.Get_cmd_stream 679

            13:31:45.675 tl            [Ljava.lang.Thread;@8a7d8b0

            13:31:45.675 tl[]:         Thread[Get_cmd_stream stdout /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-14-140625-13.26.18.930 -l localhost-4 -p 5570,5,main]

            13:31:45.675 tl[].isalive: true

            13:31:45.675 Threadname:   Get_cmd_stream stdout /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-14-140625-13.26.18.930 -l localhost-4 -p 5570

            13:31:45.675 dumpOneStack: java.io.FileInputStream -2

            13:31:45.675 dumpOneStack: java.io.FileInputStream 242

            13:31:45.675 dumpOneStack: java.io.BufferedInputStream 273

            13:31:45.675 dumpOneStack: java.io.BufferedInputStream 334

            13:31:45.675 dumpOneStack: sun.nio.cs.StreamDecoder 283

            13:31:45.676 dumpOneStack: sun.nio.cs.StreamDecoder 325

            13:31:45.676 dumpOneStack: sun.nio.cs.StreamDecoder 177

            13:31:45.676 dumpOneStack: java.io.InputStreamReader 184

            13:31:45.676 dumpOneStack: java.io.BufferedReader 154

            13:31:45.676 dumpOneStack: java.io.BufferedReader 317

            13:31:45.676 dumpOneStack: java.io.BufferedReader 382

            13:31:45.676 dumpOneStack: Utils.Get_cmd_stream 679

            13:31:45.676 tl            [Ljava.lang.Thread;@8a7d8b0

            13:31:45.676 tl[]:         Thread[SlaveStarter localhost-17-140625-13.26.18.930,5,main]

            13:31:45.676 tl[].isalive: true

            13:31:45.676 Threadname:   SlaveStarter localhost-17-140625-13.26.18.930

            13:31:45.676 dumpOneStack: java.lang.Object -2

            13:31:45.676 dumpOneStack: java.lang.Object 503

            13:31:45.676 dumpOneStack: java.lang.UNIXProcess 210

            13:31:45.676 dumpOneStack: Utils.OS_cmd 319

            13:31:45.676 dumpOneStack: Utils.OS_cmd 250

            13:31:45.676 dumpOneStack: Vdb.SlaveStarter 220

            13:31:45.676 dumpOneStack: Vdb.SlaveStarter 169

            13:31:45.677 dumpOneStack: Vdb.SlaveStarter 68

            13:31:45.677 tl            [Ljava.lang.Thread;@8a7d8b0

            13:31:45.677 tl[]:         Thread[Get_cmd_stream stderr /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-13-140625-13.26.18.930 -l localhost-3 -p 5570,5,main]

            13:31:45.677 tl[].isalive: true

            13:31:45.677 Threadname:   Get_cmd_stream stderr /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-13-140625-13.26.18.930 -l localhost-3 -p 5570

            13:31:45.677 dumpOneStack: java.io.FileInputStream -2

            13:31:45.677 dumpOneStack: java.io.FileInputStream 242

            13:31:45.677 dumpOneStack: java.io.BufferedInputStream 273

            13:31:45.677 dumpOneStack: java.io.BufferedInputStream 334

            13:31:45.677 dumpOneStack: sun.nio.cs.StreamDecoder 283

            13:31:45.677 dumpOneStack: sun.nio.cs.StreamDecoder 325

            13:31:45.677 dumpOneStack: sun.nio.cs.StreamDecoder 177

            13:31:45.677 dumpOneStack: java.io.InputStreamReader 184

            13:31:45.677 dumpOneStack: java.io.BufferedReader 154

            13:31:45.677 dumpOneStack: java.io.BufferedReader 317

            13:31:45.678 dumpOneStack: java.io.BufferedReader 382

            13:31:45.678 dumpOneStack: Utils.Get_cmd_stream 679

            13:31:45.678 tl            [Ljava.lang.Thread;@8a7d8b0

            13:31:45.678 tl[]:         Thread[Get_cmd_stream stderr /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-15-140625-13.26.18.930 -l localhost-5 -p 5570,5,main]

            13:31:45.678 tl[].isalive: true

            13:31:45.678 Threadname:   Get_cmd_stream stderr /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-15-140625-13.26.18.930 -l localhost-5 -p 5570

            13:31:45.678 dumpOneStack: java.io.FileInputStream -2

            13:31:45.678 dumpOneStack: java.io.FileInputStream 242

            13:31:45.678 dumpOneStack: java.io.BufferedInputStream 273

            13:31:45.678 dumpOneStack: java.io.BufferedInputStream 334

            13:31:45.678 dumpOneStack: sun.nio.cs.StreamDecoder 283

            13:31:45.678 dumpOneStack: sun.nio.cs.StreamDecoder 325

            13:31:45.678 dumpOneStack: sun.nio.cs.StreamDecoder 177

            13:31:45.678 dumpOneStack: java.io.InputStreamReader 184

            13:31:45.678 dumpOneStack: java.io.BufferedReader 154

            13:31:45.678 dumpOneStack: java.io.BufferedReader 317

            13:31:45.679 dumpOneStack: java.io.BufferedReader 382

            13:31:45.679 dumpOneStack: Utils.Get_cmd_stream 679

            13:31:45.679 tl            [Ljava.lang.Thread;@8a7d8b0

            13:31:45.679 tl[]:         Thread[Get_cmd_stream stdout /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-15-140625-13.26.18.930 -l localhost-5 -p 5570,5,main]

            13:31:45.679 tl[].isalive: true

            13:31:45.679 Threadname:   Get_cmd_stream stdout /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-15-140625-13.26.18.930 -l localhost-5 -p 5570

            13:31:45.679 dumpOneStack: java.io.FileInputStream -2

            13:31:45.679 dumpOneStack: java.io.FileInputStream 242

            13:31:45.679 dumpOneStack: java.io.BufferedInputStream 273

            13:31:45.679 dumpOneStack: java.io.BufferedInputStream 334

            13:31:45.679 dumpOneStack: sun.nio.cs.StreamDecoder 283

            13:31:45.679 dumpOneStack: sun.nio.cs.StreamDecoder 325

            13:31:45.679 dumpOneStack: sun.nio.cs.StreamDecoder 177

            13:31:45.679 dumpOneStack: java.io.InputStreamReader 184

            13:31:45.679 dumpOneStack: java.io.BufferedReader 154

            13:31:45.679 dumpOneStack: java.io.BufferedReader 317

            13:31:45.680 dumpOneStack: java.io.BufferedReader 382

            13:31:45.680 dumpOneStack: Utils.Get_cmd_stream 679

            13:31:45.680 tl            [Ljava.lang.Thread;@8a7d8b0

            13:31:45.680 tl[]:         Thread[Get_cmd_stream stdout /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-13-140625-13.26.18.930 -l localhost-3 -p 5570,5,main]

            13:31:45.680 tl[].isalive: true

            13:31:45.680 Threadname:   Get_cmd_stream stdout /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-13-140625-13.26.18.930 -l localhost-3 -p 5570

            13:31:45.680 dumpOneStack: java.io.FileInputStream -2

            13:31:45.680 dumpOneStack: java.io.FileInputStream 242

            13:31:45.680 dumpOneStack: java.io.BufferedInputStream 273

            13:31:45.680 dumpOneStack: java.io.BufferedInputStream 334

            13:31:45.680 dumpOneStack: sun.nio.cs.StreamDecoder 283

            13:31:45.680 dumpOneStack: sun.nio.cs.StreamDecoder 325

            13:31:45.680 dumpOneStack: sun.nio.cs.StreamDecoder 177

            13:31:45.680 dumpOneStack: java.io.InputStreamReader 184

            13:31:45.680 dumpOneStack: java.io.BufferedReader 154

            13:31:45.680 dumpOneStack: java.io.BufferedReader 317

            13:31:45.680 dumpOneStack: java.io.BufferedReader 382

            13:31:45.681 dumpOneStack: Utils.Get_cmd_stream 679

            13:31:45.681 tl            [Ljava.lang.Thread;@8a7d8b0

            13:31:45.681 tl[]:         Thread[Get_cmd_stream stderr /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-17-140625-13.26.18.930 -l localhost-7 -p 5570,5,main]

            13:31:45.681 tl[].isalive: true

            13:31:45.681 Threadname:   Get_cmd_stream stderr /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-17-140625-13.26.18.930 -l localhost-7 -p 5570

            13:31:45.681 dumpOneStack: java.io.FileInputStream -2

            13:31:45.681 dumpOneStack: java.io.FileInputStream 242

            13:31:45.681 dumpOneStack: java.io.BufferedInputStream 273

            13:31:45.681 dumpOneStack: java.io.BufferedInputStream 334

            13:31:45.681 dumpOneStack: sun.nio.cs.StreamDecoder 283

            13:31:45.681 dumpOneStack: sun.nio.cs.StreamDecoder 325

            13:31:45.681 dumpOneStack: sun.nio.cs.StreamDecoder 177

            13:31:45.681 dumpOneStack: java.io.InputStreamReader 184

            13:31:45.681 dumpOneStack: java.io.BufferedReader 154

            13:31:45.681 dumpOneStack: java.io.BufferedReader 317

            13:31:45.681 dumpOneStack: java.io.BufferedReader 382

            13:31:45.681 dumpOneStack: Utils.Get_cmd_stream 679

            13:31:45.682 tl            [Ljava.lang.Thread;@8a7d8b0

            13:31:45.682 tl[]:         Thread[Get_cmd_stream stderr /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-16-140625-13.26.18.930 -l localhost-6 -p 5570,5,main]

            13:31:45.682 tl[].isalive: true

            13:31:45.682 Threadname:   Get_cmd_stream stderr /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-16-140625-13.26.18.930 -l localhost-6 -p 5570

            13:31:45.682 dumpOneStack: java.io.FileInputStream -2

            13:31:45.682 dumpOneStack: java.io.FileInputStream 242

            13:31:45.682 dumpOneStack: java.io.BufferedInputStream 273

            13:31:45.682 dumpOneStack: java.io.BufferedInputStream 334

            13:31:45.682 dumpOneStack: sun.nio.cs.StreamDecoder 283

            13:31:45.682 dumpOneStack: sun.nio.cs.StreamDecoder 325

            13:31:45.682 dumpOneStack: sun.nio.cs.StreamDecoder 177

            13:31:45.682 dumpOneStack: java.io.InputStreamReader 184

            13:31:45.682 dumpOneStack: java.io.BufferedReader 154

            13:31:45.682 dumpOneStack: java.io.BufferedReader 317

            13:31:45.682 dumpOneStack: java.io.BufferedReader 382

            13:31:45.682 dumpOneStack: Utils.Get_cmd_stream 679

            13:31:45.683 tl            [Ljava.lang.Thread;@8a7d8b0

            13:31:45.683 tl[]:         Thread[Get_cmd_stream stdout /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-17-140625-13.26.18.930 -l localhost-7 -p 5570,5,main]

            13:31:45.683 tl[].isalive: true

            13:31:45.683 Threadname:   Get_cmd_stream stdout /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-17-140625-13.26.18.930 -l localhost-7 -p 5570

            13:31:45.683 dumpOneStack: java.io.FileInputStream -2

            13:31:45.683 dumpOneStack: java.io.FileInputStream 242

            13:31:45.683 dumpOneStack: java.io.BufferedInputStream 273

            13:31:45.683 dumpOneStack: java.io.BufferedInputStream 334

            13:31:45.683 dumpOneStack: sun.nio.cs.StreamDecoder 283

            13:31:45.683 dumpOneStack: sun.nio.cs.StreamDecoder 325

            13:31:45.683 dumpOneStack: sun.nio.cs.StreamDecoder 177

            13:31:45.683 dumpOneStack: java.io.InputStreamReader 184

            13:31:45.683 dumpOneStack: java.io.BufferedReader 154

            13:31:45.683 dumpOneStack: java.io.BufferedReader 317

            13:31:45.683 dumpOneStack: java.io.BufferedReader 382

            13:31:45.683 dumpOneStack: Utils.Get_cmd_stream 679

            13:31:45.684 tl            [Ljava.lang.Thread;@8a7d8b0

            13:31:45.684 tl[]:         Thread[Get_cmd_stream stderr /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-10-140625-13.26.18.930 -l localhost-0 -p 5570,5,main]

            13:31:45.684 tl[].isalive: true

            13:31:45.684 Threadname:   Get_cmd_stream stderr /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-10-140625-13.26.18.930 -l localhost-0 -p 5570

            13:31:45.684 dumpOneStack: java.io.FileInputStream -2

            13:31:45.684 dumpOneStack: java.io.FileInputStream 242

            13:31:45.684 dumpOneStack: java.io.BufferedInputStream 273

            13:31:45.684 dumpOneStack: java.io.BufferedInputStream 334

            13:31:45.684 dumpOneStack: sun.nio.cs.StreamDecoder 283

            13:31:45.684 dumpOneStack: sun.nio.cs.StreamDecoder 325

            13:31:45.684 dumpOneStack: sun.nio.cs.StreamDecoder 177

            13:31:45.684 dumpOneStack: java.io.InputStreamReader 184

            13:31:45.684 dumpOneStack: java.io.BufferedReader 154

            13:31:45.684 dumpOneStack: java.io.BufferedReader 317

            13:31:45.684 dumpOneStack: java.io.BufferedReader 382

            13:31:45.684 dumpOneStack: Utils.Get_cmd_stream 679

            13:31:45.685 tl            [Ljava.lang.Thread;@8a7d8b0

            13:31:45.685 tl[]:         Thread[Get_cmd_stream stdout /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-10-140625-13.26.18.930 -l localhost-0 -p 5570,5,main]

            13:31:45.685 tl[].isalive: true

            13:31:45.685 Threadname:   Get_cmd_stream stdout /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-10-140625-13.26.18.930 -l localhost-0 -p 5570

            13:31:45.685 dumpOneStack: java.io.FileInputStream -2

            13:31:45.685 dumpOneStack: java.io.FileInputStream 242

            13:31:45.685 dumpOneStack: java.io.BufferedInputStream 273

            13:31:45.685 dumpOneStack: java.io.BufferedInputStream 334

            13:31:45.685 dumpOneStack: sun.nio.cs.StreamDecoder 283

            13:31:45.685 dumpOneStack: sun.nio.cs.StreamDecoder 325

            13:31:45.685 dumpOneStack: sun.nio.cs.StreamDecoder 177

            13:31:45.685 dumpOneStack: java.io.InputStreamReader 184

            13:31:45.685 dumpOneStack: java.io.BufferedReader 154

            13:31:45.685 dumpOneStack: java.io.BufferedReader 317

            13:31:45.685 dumpOneStack: java.io.BufferedReader 382

            13:31:45.686 dumpOneStack: Utils.Get_cmd_stream 679

            13:31:45.686 tl            [Ljava.lang.Thread;@8a7d8b0

            13:31:45.686 tl[]:         Thread[Get_cmd_stream stdout /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-16-140625-13.26.18.930 -l localhost-6 -p 5570,5,main]

            13:31:45.686 tl[].isalive: true

            13:31:45.686 Threadname:   Get_cmd_stream stdout /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-16-140625-13.26.18.930 -l localhost-6 -p 5570

            13:31:45.686 dumpOneStack: java.io.FileInputStream -2

            13:31:45.686 dumpOneStack: java.io.FileInputStream 242

            13:31:45.686 dumpOneStack: java.io.BufferedInputStream 273

            13:31:45.686 dumpOneStack: java.io.BufferedInputStream 334

            13:31:45.686 dumpOneStack: sun.nio.cs.StreamDecoder 283

            13:31:45.686 dumpOneStack: sun.nio.cs.StreamDecoder 325

            13:31:45.686 dumpOneStack: sun.nio.cs.StreamDecoder 177

            13:31:45.686 dumpOneStack: java.io.InputStreamReader 184

            13:31:45.686 dumpOneStack: java.io.BufferedReader 154

            13:31:45.686 dumpOneStack: java.io.BufferedReader 317

            13:31:45.686 dumpOneStack: java.io.BufferedReader 382

            13:31:45.687 dumpOneStack: Utils.Get_cmd_stream 679

            13:31:45.687 tl            [Ljava.lang.Thread;@8a7d8b0

            13:31:45.687 tl[]:         Thread[SlaveOnMaster,10,main]

            13:31:45.687 tl[].isalive: true

            13:31:45.687 Threadname:   SlaveOnMaster

            13:31:45.687 dumpOneStack: java.net.SocketInputStream -2

            13:31:45.687 dumpOneStack: java.net.SocketInputStream 150

            13:31:45.687 dumpOneStack: java.net.SocketInputStream 121

            13:31:45.687 dumpOneStack: java.net.SocketInputStream 203

            13:31:45.687 dumpOneStack: java.io.ObjectInputStream$PeekInputStream 2266

            13:31:45.687 dumpOneStack: java.io.ObjectInputStream$BlockDataInputStream 2559

            13:31:45.687 dumpOneStack: java.io.ObjectInputStream$BlockDataInputStream 2569

            13:31:45.687 dumpOneStack: java.io.ObjectInputStream 1315

            13:31:45.687 dumpOneStack: java.io.ObjectInputStream 369

            13:31:45.687 dumpOneStack: Vdb.SlaveSocket 148

            13:31:45.688 dumpOneStack: Vdb.SlaveOnMaster 118

            13:31:45.688 dumpOneStack: Vdb.SlaveOnMaster 60

            13:31:45.688 tl            [Ljava.lang.Thread;@8a7d8b0

            13:31:45.688 tl[]:         Thread[SlaveOnMaster,10,main]

            13:31:45.688 tl[].isalive: true

            13:31:45.688 Threadname:   SlaveOnMaster

            13:31:45.688 dumpOneStack: java.net.SocketInputStream -2

            13:31:45.688 dumpOneStack: java.net.SocketInputStream 150

            13:31:45.688 dumpOneStack: java.net.SocketInputStream 121

            13:31:45.688 dumpOneStack: java.net.SocketInputStream 203

            13:31:45.688 dumpOneStack: java.io.ObjectInputStream$PeekInputStream 2266

            13:31:45.688 dumpOneStack: java.io.ObjectInputStream$BlockDataInputStream 2559

            13:31:45.688 dumpOneStack: java.io.ObjectInputStream$BlockDataInputStream 2569

            13:31:45.688 dumpOneStack: java.io.ObjectInputStream 1315

            13:31:45.688 dumpOneStack: java.io.ObjectInputStream 369

            13:31:45.688 dumpOneStack: Vdb.SlaveSocket 148

            13:31:45.689 dumpOneStack: Vdb.SlaveOnMaster 118

            13:31:45.689 dumpOneStack: Vdb.SlaveOnMaster 60

            13:31:45.689 tl            [Ljava.lang.Thread;@8a7d8b0

            13:31:45.689 tl[]:         Thread[SlaveOnMaster,10,main]

            13:31:45.689 tl[].isalive: true

            13:31:45.689 Threadname:   SlaveOnMaster

            13:31:45.689 dumpOneStack: java.net.SocketInputStream -2

            13:31:45.689 dumpOneStack: java.net.SocketInputStream 150

            13:31:45.689 dumpOneStack: java.net.SocketInputStream 121

            13:31:45.689 dumpOneStack: java.net.SocketInputStream 203

            13:31:45.689 dumpOneStack: java.io.ObjectInputStream$PeekInputStream 2266

            13:31:45.689 dumpOneStack: java.io.ObjectInputStream$BlockDataInputStream 2559

            13:31:45.689 dumpOneStack: java.io.ObjectInputStream$BlockDataInputStream 2569

            13:31:45.689 dumpOneStack: java.io.ObjectInputStream 1315

            13:31:45.689 dumpOneStack: java.io.ObjectInputStream 369

            13:31:45.689 dumpOneStack: Vdb.SlaveSocket 148

            13:31:45.690 dumpOneStack: Vdb.SlaveOnMaster 118

            13:31:45.690 dumpOneStack: Vdb.SlaveOnMaster 60

            13:31:45.690 tl            [Ljava.lang.Thread;@8a7d8b0

            13:31:45.690 tl[]:         Thread[SlaveOnMaster,10,main]

            13:31:45.690 tl[].isalive: true

            13:31:45.690 Threadname:   SlaveOnMaster

            13:31:45.690 dumpOneStack: java.net.SocketInputStream -2

            13:31:45.690 dumpOneStack: java.net.SocketInputStream 150

            13:31:45.690 dumpOneStack: java.net.SocketInputStream 121

            13:31:45.690 dumpOneStack: java.net.SocketInputStream 203

            13:31:45.690 dumpOneStack: java.io.ObjectInputStream$PeekInputStream 2266

            13:31:45.690 dumpOneStack: java.io.ObjectInputStream$BlockDataInputStream 2559

            13:31:45.690 dumpOneStack: java.io.ObjectInputStream$BlockDataInputStream 2569

            13:31:45.690 dumpOneStack: java.io.ObjectInputStream 1315

            13:31:45.690 dumpOneStack: java.io.ObjectInputStream 369

            13:31:45.690 dumpOneStack: Vdb.SlaveSocket 148

            13:31:45.691 dumpOneStack: Vdb.SlaveOnMaster 118

            13:31:45.691 dumpOneStack: Vdb.SlaveOnMaster 60

            13:31:45.691 tl            [Ljava.lang.Thread;@8a7d8b0

            13:31:45.691 tl[]:         Thread[SlaveOnMaster,10,main]

            13:31:45.691 tl[].isalive: true

            13:31:45.691 Threadname:   SlaveOnMaster

            13:31:45.691 dumpOneStack: java.net.SocketInputStream -2

            13:31:45.691 dumpOneStack: java.net.SocketInputStream 150

            13:31:45.691 dumpOneStack: java.net.SocketInputStream 121

            13:31:45.691 dumpOneStack: java.net.SocketInputStream 203

            13:31:45.691 dumpOneStack: java.io.ObjectInputStream$PeekInputStream 2266

            13:31:45.691 dumpOneStack: java.io.ObjectInputStream$BlockDataInputStream 2559

            13:31:45.691 dumpOneStack: java.io.ObjectInputStream$BlockDataInputStream 2569

            13:31:45.691 dumpOneStack: java.io.ObjectInputStream 1315

            13:31:45.691 dumpOneStack: java.io.ObjectInputStream 369

            13:31:45.692 dumpOneStack: Vdb.SlaveSocket 148

            13:31:45.692 dumpOneStack: Vdb.SlaveOnMaster 118

            13:31:45.692 dumpOneStack: Vdb.SlaveOnMaster 60

            13:31:45.692 tl            [Ljava.lang.Thread;@8a7d8b0

            13:31:45.692 tl[]:         Thread[SlaveOnMaster,10,main]

            13:31:45.692 tl[].isalive: true

            13:31:45.692 Threadname:   SlaveOnMaster

            13:31:45.692 dumpOneStack: java.net.SocketInputStream -2

            13:31:45.692 dumpOneStack: java.net.SocketInputStream 150

            13:31:45.692 dumpOneStack: java.net.SocketInputStream 121

            13:31:45.692 dumpOneStack: java.net.SocketInputStream 203

            13:31:45.692 dumpOneStack: java.io.ObjectInputStream$PeekInputStream 2266

            13:31:45.692 dumpOneStack: java.io.ObjectInputStream$BlockDataInputStream 2559

            13:31:45.692 dumpOneStack: java.io.ObjectInputStream$BlockDataInputStream 2569

            13:31:45.692 dumpOneStack: java.io.ObjectInputStream 1315

            13:31:45.693 dumpOneStack: java.io.ObjectInputStream 369

            13:31:45.693 dumpOneStack: Vdb.SlaveSocket 148

            13:31:45.693 dumpOneStack: Vdb.SlaveOnMaster 118

            13:31:45.693 dumpOneStack: Vdb.SlaveOnMaster 60

            13:31:45.693 tl            [Ljava.lang.Thread;@8a7d8b0

            13:31:45.693 tl[]:         Thread[SlaveOnMaster,10,main]

            13:31:45.693 tl[].isalive: true

            13:31:45.693 Threadname:   SlaveOnMaster

            13:31:45.693 dumpOneStack: java.net.SocketInputStream -2

            13:31:45.693 dumpOneStack: java.net.SocketInputStream 150

            13:31:45.693 dumpOneStack: java.net.SocketInputStream 121

            13:31:45.693 dumpOneStack: java.net.SocketInputStream 203

            13:31:45.693 dumpOneStack: java.io.ObjectInputStream$PeekInputStream 2266

            13:31:45.693 dumpOneStack: java.io.ObjectInputStream$BlockDataInputStream 2559

            13:31:45.693 dumpOneStack: java.io.ObjectInputStream$BlockDataInputStream 2569

            13:31:45.693 dumpOneStack: java.io.ObjectInputStream 1315

            13:31:45.694 dumpOneStack: java.io.ObjectInputStream 369

            13:31:45.694 dumpOneStack: Vdb.SlaveSocket 148

            13:31:45.694 dumpOneStack: Vdb.SlaveOnMaster 118

            13:31:45.694 dumpOneStack: Vdb.SlaveOnMaster 60

            13:31:45.694 tl            [Ljava.lang.Thread;@8a7d8b0

            13:31:45.694 tl[]:         Thread[SlaveOnMaster,10,main]

            13:31:45.694 tl[].isalive: true

            13:31:45.694 Threadname:   SlaveOnMaster

            13:31:45.694 dumpOneStack: java.net.SocketInputStream -2

            13:31:45.694 dumpOneStack: java.net.SocketInputStream 150

            13:31:45.694 dumpOneStack: java.net.SocketInputStream 121

            13:31:45.694 dumpOneStack: java.net.SocketInputStream 203

            13:31:45.694 dumpOneStack: java.io.ObjectInputStream$PeekInputStream 2266

            13:31:45.694 dumpOneStack: java.io.ObjectInputStream$BlockDataInputStream 2559

            13:31:45.694 dumpOneStack: java.io.ObjectInputStream$BlockDataInputStream 2569

            13:31:45.695 dumpOneStack: java.io.ObjectInputStream 1315

            13:31:45.695 dumpOneStack: java.io.ObjectInputStream 369

            13:31:45.695 dumpOneStack: Vdb.SlaveSocket 148

            13:31:45.695 dumpOneStack: Vdb.SlaveOnMaster 118

            13:31:45.695 dumpOneStack: Vdb.SlaveOnMaster 60

            13:31:45.695 tl            [Ljava.lang.Thread;@8a7d8b0

            13:31:45.695 tl[]:         Thread[Slave HeartBeat,5,main]

            13:31:45.695 tl[].isalive: true

            13:31:45.695 Threadname:   Slave HeartBeat

            13:31:45.695 dumpOneStack: java.lang.Thread 1567

            13:31:45.695 dumpOneStack: Vdb.common 1523

            13:31:45.695 dumpOneStack: Vdb.common 1517

            13:31:45.695 dumpOneStack: Vdb.HeartBeat 97

            13:31:45.695 tl            [Ljava.lang.Thread;@8a7d8b0

            13:31:45.695 tl[]:         Thread[Reporter,10,main]

            13:31:45.695 tl[].isalive: true

            13:31:45.695 Threadname:   Reporter

            13:31:45.696 dumpOneStack: java.lang.Thread -2

            13:31:45.696 dumpOneStack: java.lang.Thread 338

            13:31:45.696 dumpOneStack: Vdb.common 376

            13:31:45.696 dumpOneStack: Vdb.common 365

            13:31:45.696 dumpOneStack: Vdb.common 668

            13:31:45.696 dumpOneStack: Vdb.Reporter 122

            13:31:45.696 killCommand: /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-11-140625-13.26.18.930 -l localhost-1 -p 5570

            13:31:45.696 killCommand: /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-10-140625-13.26.18.930 -l localhost-0 -p 5570

            13:31:45.696 killCommand: /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-14-140625-13.26.18.930 -l localhost-4 -p 5570

            13:31:45.696 killCommand: /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-16-140625-13.26.18.930 -l localhost-6 -p 5570

            13:31:45.696

            13:31:45.696 Heartbeat monitor: One or more slaves did not respond

            13:31:45.697

            java.lang.RuntimeException: Heartbeat monitor: One or more slaves did not respond

              at Vdb.common.failure(common.java:307)

              at Vdb.HeartBeat.run(HeartBeat.java:98)

            13:31:46.059        34    2353.40   246.38  109775  50.07   52.286  104.211    0.210 1030.325  100.624 126.7  15.3   3.1

            13:31:46.697 Waiting for command1 '/opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-11-140625-13.26.18.930 -l localhost-1 -p 5570' completion

            13:31:46.697 Waiting for command1 '/opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-14-140625-13.26.18.930 -l localhost-4 -p 5570' completion

            13:31:46.697 Waiting for command1 '/opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-10-140625-13.26.18.930 -l localhost-0 -p 5570' completion

            13:31:46.697 Waiting for command1 '/opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-16-140625-13.26.18.930 -l localhost-6 -p 5570' completion

            13:31:46.698 killCommand: /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-12-140625-13.26.18.930 -l localhost-2 -p 5570

            13:31:46.698 killCommand: /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-15-140625-13.26.18.930 -l localhost-5 -p 5570

            • 3. Re: Slave termination error
              Henk Vandenbergh-Oracle

              13:26:18.958 Vdbench distribution: vdbench503rc11

              13:26:18.959

              13:26:18.973 *

              13:26:18.973 * This beta version 'vdbench503rc11' was built on Fri April 08  9:48:23 MDT 2011.

              13:26:18.973 * which is more than 90 days ago.

               

              This code is too old for me to even go any further.

               

              Please download the latest vdbench50401 from http://www.oracle.com/technetwork/server-storage/vdbench-downloads-1901681.html

               

               

              Henk.