This discussion is archived
9 Replies Latest reply: Jan 17, 2013 11:17 AM by SunilSharma RSS

DAC server crashes

DilbertsDog Newbie
Currently Being Moderated
Hi,

I have had my DAC server crash on me a number of times during an ETL run, its very confusing as to why it actually crashes. I look in the DAC log there is no indication of anything wrong the logs just stop (see below)


25158 INFO Tue Jan 15 23:12:18 GMT 2013
2013-01-15 23:12:18.814 INFORMATICA TASK:SILOS:SIL_GLCOGSFact_DiffManLoad:(Source : FULL Target : FULL) has started.

25159 INFO Tue Jan 15 23:12:33 GMT 2013
Sessions for workflow SIL_GLCOGSFact_DiffManLoad
SIL_GLCOGSFact_DiffManLoad

25160 INFO Tue Jan 15 23:12:42 GMT 2013
2013-01-15 23:12:42.079 CREATE INDEX:W_AP_INV_DIST_F:W_AP_INV_DIST_F_F21 has started.

25161 SEVERE Tue Jan 15 23:12:42 GMT 2013 Request to start workflow : 'SILOS:SIL_GLCOGSFact_DiffManLoad' has completed with error code 0
25162 INFO Tue Jan 15 23:13:18 GMT 2013
2013-01-15 23:13:18.29 CREATE INDEX:W_AP_INV_DIST_F:W_AP_INV_DIST_F_F22 has started.

25163 INFO Tue Jan 15 23:13:50 GMT 2013 Workflow 'SIL_GLCOGSFact_DiffManLoad' completed executing.
25164 INFO Tue Jan 15 23:13:50 GMT 2013
2013-01-15 23:13:50.818 CREATE INDEX:W_AP_INV_DIST_F:W_AP_INV_DIST_F_F23 has started.



The above line is the last line in my current DAC server log file. I am now unsure where to look with regards to finding the root cause of the error and a resolution.

As it just seems to crash I am leaning towards a java issue and could increase the java heap size, but my question is will this resolve the issue?

$JAVA -server -Xms256m -Xmx800m -cp $DACCLASSPATH com.siebel.etl.net.QServer

I can provide logs or any other information required.
  • 1. Re: DAC server crashes
    Srini VEERAVALLI Guru
    Currently Being Moderated
    To answer to this question need more details about your configuration...

    0) Is it new installation and configuration? whats the latest patch?
    1) Does your DAC has email notifications set? if Yes did you get any email status about DAC server status? if you dont have it then I would suggest to config it.
    2) What is the database server status is it up and running? In general DAC server is down when db is not available
    3) Is it happening given workflow/task running or what event crash is occuring?
    4) Can you let me know whats the maximum idle time you can see on DAC client with no job running?

    Edited by: Srini VEERAVALLI on Jan 16, 2013 6:44 AM
  • 2. Re: DAC server crashes
    DilbertsDog Newbie
    Currently Being Moderated
    0) This is not a clean install it has been running since around July 2012 Dac Version that I am running is - 11.1.1.5.2 that comes with OBIApps

    1) I do not have email notifications setup I was unaware of this (my dac server runs on a linux box) I do have ETL failed/completion notifications - I did not get an ETL failed notification when dac crashed.

    2) Database has not been down and has been up and running for some time.

    3) No specific task is causing it to crash, to make things a little more difficult it was a few months ago was the last time this happened.

    4) Can you let me know whats the maximum idle time you can see on DAC client with no job running? I am unsure what you mean here. If I dont start an ETL the dac server will idle for weeks and weeks.

    Thanks
  • 3. Re: DAC server crashes
    Srini VEERAVALLI Guru
    Currently Being Moderated
    Hmm no clues to narrow down to the issue!.
    Were there any Java updates?

    BTW: DAC doesnt send any notifications during the crash.. bud it does when repository is not available... I was aiming to know that status.

    Curious to know updates on this.

    Edited by: Srini VEERAVALLI on Jan 16, 2013 8:02 AM
  • 4. Re: DAC server crashes
    DilbertsDog Newbie
    Currently Being Moderated
    No Java or Solaris updates have been done on this particular box anytime recently.

    Ive increased the java heap size to 2Gb to mimic the other environment, unfortunately have no way to know or test to see if this resolves my issue.

    I still think its an odd one how the logs just end, dac being based on java it can only point to an error there. Do you know of any java logs that can be looked at to shed some more light on the issue, Ive had a look around other sources but have found nothing worthwhile in relation to the issue.
  • 5. Re: DAC server crashes
    Srini VEERAVALLI Guru
    Currently Being Moderated
    I dont think jdk log anything.
    I see only place to see log is \orahome\10gR3_1\bifoundation\dac\log

    By any chance is there anything in Linux to track event log as in Windows eventviewer?
  • 6. Re: DAC server crashes
    DilbertsDog Newbie
    Currently Being Moderated
    Found this in nohup.out it reveals the issue is with Java, not sure how this would apply to windows to help you out maybe event viewer but I cant be sure



    2013-01-15 23:13:50.818 CREATE INDEX:W_AP_INV_DIST_F:W_AP_INV_DIST_F_F23 has started.

    #
    # A fatal error has been detected by the Java Runtime Environment:
    #
    # SIGSEGV (0xb) at pc=0xffffffff7ed63284, pid=2159, tid=84
    #
    # JRE version: 6.0_26-b03
    # Java VM: Java HotSpot(TM) 64-Bit Server VM (20.1-b02 mixed mode solaris-sparc compressed oops)
    # Problematic frame:
    # C [libc.so.1+0x63284]# [ timer expired, abort... ]

    $ pldd 23497
    23497: /usr/jdk/instances/jdk1.6.0/bin/sparcv9/java -server -Xms256m -Xmx800m
    /usr/jdk/instances/jdk1.6.0/bin/sparcv9/java
    /lib/sparcv9/libthread.so.1
    /usr/jdk/instances/jdk1.6.0/jre/lib/sparcv9/jli/libjli.so
    /lib/sparcv9/libdl.so.1
    /lib/sparcv9/libc.so.1
    /platform/sun4v/lib/sparcv9/libc_psr.so.1
    /usr/jdk/instances/jdk1.6.0/jre/lib/sparcv9/server/libjvm.so
    /lib/sparcv9/libsocket.so.1
    /usr/lib/sparcv9/libsched.so.1
    /lib/sparcv9/libm.so.1
    /usr/lib/sparcv9/libCrun.so.1
    /lib/sparcv9/libdoor.so.1
    /usr/lib/sparcv9/libdemangle.so.1
    /lib/sparcv9/libkstat.so.1
    /lib/sparcv9/libnsl.so.1
    /lib/sparcv9/libm.so.2
    /lib/sparcv9/libscf.so.1
    /lib/sparcv9/libuutil.so.1
    /lib/sparcv9/libgen.so.1
    /lib/sparcv9/libmd.so.1
    /platform/sun4v/lib/sparcv9/libmd_psr.so.1
    /lib/sparcv9/libmp.so.2
    /usr/jdk/instances/jdk1.6.0/jre/lib/sparcv9/libverify.so
    /usr/jdk/instances/jdk1.6.0/jre/lib/sparcv9/libjava.so
    /usr/jdk/instances/jdk1.6.0/jre/lib/sparcv9/libzip.so
    /usr/lib/locale/en_GB.UTF-8/sparcv9/en_GB.UTF-8.so.3
    /usr/lib/locale/common/sparcv9/methods_unicode.so.3
    /usr/jdk/instances/jdk1.6.0/jre/lib/sparcv9/libnet.so
    /usr/jdk/instances/jdk1.6.0/jre/lib/sparcv9/libnio.so
    /lib/sparcv9/librt.so.1
    /lib/sparcv9/libaio.so.1
    /usr/jdk/instances/jdk1.6.0/jre/lib/sparcv9/libj2pkcs11.so
    /usr/lib/sparcv9/libpkcs11.so.1
    /usr/lib/sparcv9/libcryptoutil.so.1
    /usr/lib/security/sparcv9/pkcs11_kernel.so.1
    /usr/lib/security/sparcv9/pkcs11_softtoken.so.1
    /usr/jdk/instances/jdk1.6.0/jre/lib/sparcv9/libmanagement.so
    $
    $ pldd 23495
    23495: ksh startserver.sh
    /usr/bin/ksh
    /lib/libc.so.1
    /platform/sun4v/lib/libc_psr.so.1
    /usr/lib/locale/en_GB.UTF-8/en_GB.UTF-8.so.3
    /usr/lib/locale/common/methods_unicode.so.3
    $

    I suspect the above pldd is now outdated as I have restarted the dac server since

    Edited by: DilbertsDog on 16-Jan-2013 09:18
  • 7. Re: DAC server crashes
    Srini VEERAVALLI Guru
    Currently Being Moderated
    Might be DAC 32bit running on 64bit OS

    Check this link
    http://stackoverflow.com/questions/6448163/a-fatal-error-has-been-detected-by-the-java-runtime-environment-sigsegv-libjvm

    Edited by: Srini VEERAVALLI on Jan 16, 2013 11:27 AM

    Is your DAC using jdk from hits own java? I mean \orahome\10gR3_1\jdk\

    if you use this DAC running on jdk 32bit

    Edited by: Srini VEERAVALLI on Jan 16, 2013 11:32 AM
  • 8. Re: DAC server crashes
    DilbertsDog Newbie
    Currently Being Moderated
    Just to keep you in the loop, I suspect that the crash may have been down to Java garbage collection. I have increased the initial heap size and set the young generation.

    /usr/jdk/instances/jdk1.6.0/bin/sparcv9/java -server -Xmn500m -Xms2048m -Xmx204


    to answer your question on java I use java that is on the box you can see the location above,

    java version "1.6.0_26"
    Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
    Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode)

    Still looking to find an exact answer to why it crashed


    Going to write a script that looks for the DAC java process disappearing and if so restart DAC using startserver.sh
  • 9. Re: DAC server crashes
    SunilSharma Expert
    Currently Being Moderated
    in the DAC properties change the status of server log level from SEVERE to FINEST that will have enough information to see what the root cause for the DAC server crash. And also can you please help me in providing the DAC latest patch which you have in place on server and client?

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points