2 Replies Latest reply on Oct 21, 2014 8:41 AM by MichR

    VdBench Undefined error number: 116 ;  Maximum native memory allocation

    MichR

      While in format stage I got:

      I see in the stdout file JVM run out of native memory.

      I was using only 100 threads, and by the looks of it , java heap was more then half free.

      No other VDBench sessions run simultaneously and no other processes in the background.

       

      Any thoughts to what could have been the reason?

      Is there any way to know how much memory VDBench will require?

       

       

      Thank you !

       

      logfile:

      ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

       

       

      18:57:23.896 localhost-0: 18:57:23.884 op: write lun: /mnt/q/node-1/fs14134706851/vdb.1_1.dir/vdb.2_2.dir/vdb.3_2.dir/vdb.4_1.dir/vdb.5_1.dir/vdb.6_2.dir/vdb.7_2.dir/vdb.8_2.dir/vdb.9_2.dir/vdb.10_2.dir/vdb_f0002.file lba: 917504 0x000E0000 xfer: 131072 errno: Undefined error number: 116

      18:57:23.896 localhost-0: 18:57:23.896 op: write lun: /mnt/q/node-1/fs14134706851/vdb.1_1.dir/vdb.2_2.dir/vdb.3_2.dir/vdb.4_1.dir/vdb.5_1.dir/vdb.6_2.dir/vdb.7_2.dir/vdb.8_2.dir/vdb.9_2.dir/vdb.10_2.dir/vdb_f0003.file lba: 262144 0x00040000 xfer: 131072 errno: Undefined error number: 116

      18:57:23.897 localhost-0: 18:57:23.896 op: write lun: /mnt/q/node-1/fs14134706851/vdb.1_1.dir/vdb.2_2.dir/vdb.3_2.dir/vdb.4_1.dir/vdb.5_1.dir/vdb.6_2.dir/vdb.7_2.dir/vdb.8_2.dir/vdb.9_1.dir/vdb.10_2.dir/vdb_f0009.file lba: 655360 0x000A0000 xfer: 131072 errno: Undefined error number: 116

      18:57:23.969

      18:57:23.969 **********************************************************

      18:57:23.969 Slave localhost-0 aborting: open failed for /mnt/q/node-1/fs14134706851/vdb.1_1.dir/vdb.2_2.dir/vdb.3_2.dir/vdb.4_1.dir/vdb.5_1.dir/vdb.6_2.dir/vdb.7_2.dir/vdb.8_2.dir/vdb.9_2.dir/vdb.10_2.dir/vdb_f0006.file

      18:57:23.969 **********************************************************

      18:57:23.969

      18:57:25.516

      18:57:25.517 Slave localhost-0 prematurely terminated.

      18:57:25.517

      18:57:25.517 Slave aborted. Abort message received:

      18:57:25.517 open failed for /mnt/q/node-1/fs14134706851/vdb.1_1.dir/vdb.2_2.dir/vdb.3_2.dir/vdb.4_1.dir/vdb.5_1.dir/vdb.6_2.dir/vdb.7_2.dir/vdb.8_2.dir/vdb.9_2.dir/vdb.10_2.dir/vdb_f0006.file

      18:57:25.517

      18:57:25.517 Look at file localhost-0.stdout.html for more information.

      18:57:25.517 HTML link: localhost-0.stdout.html

      18:57:25.517

      18:57:25.517 Slave localhost-0 prematurely terminated.

      18:57:25.517

      java.lang.RuntimeException: Slave localhost-0 prematurely terminated.

      at Vdb.common.failure(common.java:308)

      at Vdb.SlaveStarter.startSlave(SlaveStarter.java:179)

      at Vdb.SlaveStarter.run(SlaveStarter.java:50)

       

       

      stdout:

      ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

       

       

      18:57:23.883 18:57:23.883 file_open(), open /mnt/q/node-1/fs14134706851/vdb.1_1.dir/vdb.2_2.dir/vdb.3_2.dir/vdb.4_1.dir/vdb.5_1.dir/vdb.6_2.dir/vdb.7_2.dir/vdb.8_2.dir/vdb.9_2.dir/vdb.10_2.dir/vdb_f0005.file failed
      18:57:23.884 18:57:23.883 file_open(), open /mnt/q/node-1/fs14134706851/vdb.1_1.dir/vdb.2_2.dir/vdb.3_2.dir/vdb.4_1.dir/vdb.5_1.dir/vdb.6_2.dir/vdb.7_2.dir/vdb.8_2.dir/vdb.9_2.dir/vdb.10_2.dir/vdb_f0006.file failed
      18:57:23.884 18:57:23.883 error: 116
      18:57:23.884 18:57:23.884 error: 116
      18:57:23.884 18:57:23.884 Memory total Java heap: 126.000 MB; Free: 70.232 MB; Used: 55.768 MB;
      18:57:23.884 18:57:23.884 Memory total Java heap: 126.000 MB; Free: 70.232 MB; Used: 55.768 MB;
      18:57:23.884 18:57:23.884 Maximum native memory allocation: 10,485,760; Current allocation: 10,485,760
      18:57:23.884 18:57:23.884
      18:57:23.884 18:57:23.884 open failed for /mnt/q/node-1/fs14134706851/vdb.1_1.dir/vdb.2_2.dir/vdb.3_2.dir/vdb.4_1.dir/vdb.5_1.dir/vdb.6_2.dir/vdb.7_2.dir/vdb.8_2.dir/vdb.9_2.dir/vdb.10_2.dir/vdb_f0006.file
      18:57:23.884 18:57:23.884
      18:57:23.885 java.lang.RuntimeException: open failed for /mnt/q/node-1/fs14134706851/vdb.1_1.dir/vdb.2_2.dir/vdb.3_2.dir/vdb.4_1.dir/vdb.5_1.dir/vdb.6_2.dir/vdb.7_2.dir/vdb.8_2.dir/vdb.9_2.dir/vdb.10_2.dir/vdb_f0006.file
      18:57:23.885 at Vdb.common.failure(common.java:308)
      18:57:23.885 at Vdb.ActiveFile.openFile(ActiveFile.java:158)
      18:57:23.885 at Vdb.FwgThread.openForWrite(FwgThread.java:375)
      18:57:23.885 at Vdb.OpCreate.doOperation(OpCreate.java:39)
      18:57:23.885 at Vdb.OpFormat.doOperation(OpFormat.java:91)
      18:57:23.885 at Vdb.FwgThread.run(FwgThread.java:157)

       

       

       

       

      paramfile:

      ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

       

      debug=25

      fsd=fsd1,anchor=/mnt/q/node-1/fs14134706851,depth=10,width=2,files=10,size=1M,distribution=all

      fsd=fsd2,anchor=/mnt/q/node-1/fs14134706852,depth=10,width=2,files=10,size=1M,distribution=all

      fsd=fsd3,anchor=/mnt/q/node-1/fs14134706853,depth=10,width=2,files=10,size=1M,distribution=all

      fsd=fsd4,anchor=/mnt/q/node-1/fs14134706854,depth=10,width=2,files=10,size=1M,distribution=all

      fsd=fsd5,anchor=/mnt/q/node-1/fs14134706855,depth=10,width=2,files=10,size=1M,distribution=all

       

      fwd=default,xfersize=4k,fileio=sequential,fileselect=random,threads=100

      fwd=fwd1,fsd=fsd*,operation=read

      #fwd=fwd2,fsd=fsd1,operation=write

       

      rd=rd1,fwd=fwd*,fwdrate=max,format=yes,elapsed=30,interval=1

        • 1. Re: VdBench Undefined error number: 116 ;  Maximum native memory allocation
          Henk Vandenbergh-Oracle

          The 'maximum native memory' message is just some diagnostics data to tell you, in this case, that you are using only about 10mb. No problem here.

          What IS the problem is errno=116.

          Do me a favor and look at your /include/errno.h to find out what 116 really means.

          The errno translation done in Vdbench only has Solaris error codes, something that I need to fix some day.

           

           

          #define ESTALE      116 /* Stale NFS file handle */

           

          This typically means that one system is reading/writing a file, and some other system is deleting it.

          Did you accidentally run two identical copies of a Vdbench parameter file?

           

          Henk.

          • 2. Re: VdBench Undefined error number: 116 ;  Maximum native memory allocation
            MichR

            Hmm,

             

            The 116 erro means the same as you stated : "Stale NFS file handle" ( Operating system is CentOS )

            Now that i know it wasnt a memory problem , I've looked deeper and found another session that was running, and did delete those file handlers.

             

            If i may, this message is very confusing,

            18:57:23.884 18:57:23.884 Maximum native memory allocation: 10,485,760; Current allocation: 10,485,760

            this wasnt clear at all, According to the message there was no more room for allocation. Please consider changing the message to something more understandable. 

             

            Thank you,

            Going to try v50402 now