1 Reply Latest reply: Feb 6, 2012 5:53 PM by 915625 RSS

    OCI JDBC Driver Segmetation Fault using ADR

    915625
      We have a java process that is using the OCI JDBC driver with the libraries from instant client 11.2.0.1.

      When there is a connection error to the database (TNS-12541 TNS:no listener), it results in the VM process crashing with no feedback (no logging, no core dump, etc.). We’ve done a little bit of debugging with gdb and found that the segmentation fault is a result of trying to load data off of a zero address. However, we are unable to determine what line of code is being executed at that point.

      Using strace, we noticed that right before the seg fault, the native OCI code was working with files under /<user.home>/oradiag_<user> directory. This led us to try adding the following parameter to the sqlnet.ora file for the OCI driver to disable ADR:

      DIAG_ADR_ENABLED = OFF

      This solved our immediate issue as the server no longer crashes when there is a connection error. However, we still don’t know why.

      From looking through forums, it appears that the ADR has caused others issues like this, but we have not found one that clearly defines the exact reason why the seg fault occurs.

      Has anybody seen this problem and come across another solution other than setting DIAG_ADR_ENABLED to off?

      Other Info
      1)     We launch the JVM using apache commons-daemon (jsvc). If the parent process is owned by root and the child process, which calls the native OCI code, is owned by another user the issue does not occur. However, if both the parent/child are owned by root or by the user, then the problem occurs. Most likely because the child process in the first setup (parent owned by root) does not have permissions to perform some file activity so it passes over the code path that causes the segmentation fault.

      2)     snip from strace output (parent pid process is 11546)

      -----
      [pid 11657] <... poll resumed> ) = 1 ([{fd=211, revents=POLLERR|POLLHUP}])
      [pid 11657] getsockname(211, {sa_family=AF_INET, sin_port=htons(38819), sin_addr=inet_addr("172.16.10.3")}, [16]) = 0
      [pid 11657] times({tms_utime=3587, tms_stime=113, tms_cutime=0, tms_cstime=0}) = 489278500
      [pid 11657] write(211, "\0\331\0\0\1\0\0\0\1:\1,\fA \0\177\377\177\10\0\0\1\0\0\237\0:\0\0\10\0"..., 217) = -1 ECONNREFUSED (Connection refused)
      [pid 11657] close(211) = 0
      [pid 11657] mkdir("/root/oradiag_root", 0775) = -1 EEXIST (File exists)
      [pid 11657] mkdir("/root/oradiag_root/diag", 0775) = -1 EEXIST (File exists)
      [pid 11657] chmod("/root/oradiag_root/diag", 0775) = 0
      [pid 11657] mkdir("/root/oradiag_root/diag/clients", 0775) = -1 EEXIST (File exists)
      [pid 11657] mkdir("/root/oradiag_root/diag/clients/user_root", 0775) = -1 EEXIST (File exists)
      [pid 11657] stat("/root/oradiag_root/diag/clients/user_root", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
      [pid 11657] mkdir("/root/oradiag_root/diag/clients/user_root/host_18939330_76", 0775) = -1 EEXIST (File exists)
      [pid 11657] stat("/root/oradiag_root/diag/clients/user_root/host_18939330_76", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
      [pid 11657] mkdir("/root/oradiag_root/diag/clients/user_root/host_18939330_76/trace", 0775) = -1 EEXIST (File exists)
      [pid 11657] stat("/root/oradiag_root/diag/clients/user_root/host_18939330_76/trace", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
      [pid 11657] mkdir("/root/oradiag_root/diag/clients/user_root/host_18939330_76/alert", 0775) = -1 EEXIST (File exists)
      [pid 11657] stat("/root/oradiag_root/diag/clients/user_root/host_18939330_76/alert", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
      [pid 11657] mkdir("/root/oradiag_root/diag/clients/user_root/host_18939330_76/incident", 0775) = -1 EEXIST (File exists)
      [pid 11657] stat("/root/oradiag_root/diag/clients/user_root/host_18939330_76/incident", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
      [pid 11657] mkdir("/root/oradiag_root/diag/clients/user_root/host_18939330_76/metadata", 0775) = -1 EEXIST (File exists)
      [pid 11657] stat("/root/oradiag_root/diag/clients/user_root/host_18939330_76/metadata", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
      [pid 11657] mkdir("/root/oradiag_root/diag/clients/user_root/host_18939330_76/incpkg", 0775) = -1 EEXIST (File exists)
      [pid 11657] stat("/root/oradiag_root/diag/clients/user_root/host_18939330_76/incpkg", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
      [pid 11657] mkdir("/root/oradiag_root/diag/clients/user_root/host_18939330_76/sweep", 0775) = -1 EEXIST (File exists)
      [pid 11657] stat("/root/oradiag_root/diag/clients/user_root/host_18939330_76/sweep", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
      [pid 11657] mkdir("/root/oradiag_root/diag/clients/user_root/host_18939330_76/lck", 0775) = -1 EEXIST (File exists)
      [pid 11657] stat("/root/oradiag_root/diag/clients/user_root/host_18939330_76/lck", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
      [pid 11657] mkdir("/root/oradiag_root/diag/clients/user_root/host_18939330_76/cdump", 0775) = -1 EEXIST (File exists)
      [pid 11657] stat("/root/oradiag_root/diag/clients/user_root/host_18939330_76/cdump", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
      [pid 11657] mkdir("/root/oradiag_root/diag/clients/user_root/host_18939330_76/stage", 0775) = -1 EEXIST (File exists)
      [pid 11657] stat("/root/oradiag_root/diag/clients/user_root/host_18939330_76/stage", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
      [pid 11657] mkdir("/root/oradiag_root/diag/clients/user_root/host_18939330_76/metadata", 0775) = -1 EEXIST (File exists)
      [pid 11657] stat("/root/oradiag_root/diag/clients/user_root/host_18939330_76/metadata", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
      [pid 11657] lstat("/root/oradiag_root/diag/clients/user_root/host_18939330_76/metadata/ADR_CONTROL.ams", {st_mode=S_IFREG|0640, st_size=65536, ...}) = 0
      [pid 11657] stat("/root/oradiag_root/diag/clients/user_root/host_18939330_76/metadata/ADR_CONTROL.ams", {st_mode=S_IFREG|0640, st_size=65536, ...}) = 0
      [pid 11657] lstat("/root/oradiag_root/diag/clients/user_root/host_18939330_76/metadata/ADR_INVALIDATION.ams", {st_mode=S_IFREG|0640, st_size=65536, ...}) = 0
      [pid 11657] stat("/root/oradiag_root/diag/clients/user_root/host_18939330_76/metadata/ADR_INVALIDATION.ams", {st_mode=S_IFREG|0640, st_size=65536, ...}) = 0
      [pid 11657] lstat("/root/oradiag_root/diag/clients/user_root/host_18939330_76/metadata/INC_METER_IMPT_DEF.ams", {st_mode=S_IFREG|0640, st_size=65536, ...}) = 0
      [pid 11657] stat("/root/oradiag_root/diag/clients/user_root/host_18939330_76/metadata/INC_METER_IMPT_DEF.ams", {st_mode=S_IFREG|0640, st_size=65536, ...}) = 0
      [pid 11657] lstat("/root/oradiag_root/diag/clients/user_root/host_18939330_76/metadata/INC_METER_PK_IMPTS.ams", {st_mode=S_IFREG|0640, st_size=65536, ...}) = 0
      [pid 11657] stat("/root/oradiag_root/diag/clients/user_root/host_18939330_76/metadata/INC_METER_PK_IMPTS.ams", {st_mode=S_IFREG|0640, st_size=65536, ...}) = 0
      [pid 11657] times({tms_utime=3587, tms_stime=113, tms_cutime=0, tms_cstime=0}) = 489278500
      [pid 11657] lstat("/root/oradiag_root/diag/clients/user_root/host_18939330_76/metadata/ADR_CONTROL.ams", {st_mode=S_IFREG|0640, st_size=65536, ...}) = 0
      [pid 11657] stat("/root/oradiag_root/diag/clients/user_root/host_18939330_76/metadata/ADR_CONTROL.ams", {st_mode=S_IFREG|0640, st_size=65536, ...}) = 0
      [pid 11657] --- SIGSEGV (Segmentation fault) @ 0 (0) ---
      [pid 11657] --- SIGSEGV (Segmentation fault) @ 0 (0) ---

      -----
        • 1. Re: OCI JDBC Driver Segmetation Fault using ADR
          915625
          Some more info from gdb at the point of the segmentation fault:

          (gdb) bt
          #0 0x00002ad2b30a1326 in ?? ()
          #1 0x00002aaadbfcc938 in ?? ()
          #2 0x00002ad2b21dea13 in InterpreterRuntime::resolve_invoke () from /usr/local/java/jre/lib/amd64/server/libjvm.so
          Backtrace stopped: previous frame inner to this frame (corrupt stack?)

          (gdb) info reg
          rax 0x2aaaeac240f8 46913571406072
          rbx 0x2aaaeac240f8 46913571406072
          rcx 0x2aaadbfcc938 46913323583800
          rdx 0x0 0
          rsi 0x2aaadbfcc938 46913323583800
          rdi 0x2aaab1c6dba8 46912615406504
          rbp 0x0 0x0
          rsp 0x42675760 0x42675760
          r8 0x2ad2b27e8000 47084426133504
          r9 0x2aaab1c6d7f8 46912615405560
          r10 0x2ad2b27b9c40 47084425944128
          r11 0x2ad2b321e9b0 47084436842928
          r12 0x2 2
          r13 0x426758b0 1114069168
          r14 0x42675918 1114069272
          r15 0x2aaab1c6d000 46912615403520
          rip 0x2ad2b30a1326 0x2ad2b30a1326
          eflags 0x10206 [ PF IF RF ]
          cs 0x33 51
          ss 0x2b 43
          ds 0x0 0
          es 0x0 0
          fs 0x63 99
          gs 0x0 0
          fctrl 0x27f 639
          fstat 0x20 32
          ftag 0xffff 65535
          fiseg 0x2ad2 10962
          fioff 0xb249282f -1303828433
          foseg 0x0 0
          fooff 0x42676cf8 1114074360
          fop 0x55d 1373
          mxcsr 0x1fa0 [ PE IM DM ZM OM UM PM ]