6 Replies Latest reply: Apr 7, 2014 2:26 PM by Puneeth-Oracle RSS

    Weblogic January PSU appears to cause major performance issues

    BillW

      I am using Weblogic to manage Oracle Internet Directory. I have it installed on two hosts and replication between the two OID installations is configured.

      I downloaded the Weblogic January PSU, p17572726_1036_Generic.zip from oracle support. I used Smart Update to apply the patch. Prior to applying the patch I shut down the weblogic components, the managed server, ODMS and weblogic in that order. I then applied the patch and it appeared to be successful although I don't know where the log files are.

      After applying the patch I attempted to start the weblogic components in the reverse order. The managed server did not start properly. At about 10:25AM the server was rebooted and everything started properly.

      At approximately 1:42PM the server appeared to crash and did not restart properly and we noticed a very high load average. The server was restarted without restarting the Weblogic components. I then started the components manually in the same order noted above and everything came up properly but I monitored the server and noticed the load average climb and hover around 40. At that point I was unable to initiate a new session to the server and response was very slow.

      I then stopped the managed server, it took a very long time. I left the ODMS and Weblogic running. Once the managed server stopped the load average dropped down and hovered around 2, a much more reasonable value and response was again normal.

      It appears that the January PSU is severely impacting the performance of weblogic rendering it unusable and I don't know why. Has anyone else encountered this issue? I need assistance to resolve this problem.

      Thank you.

      Bill Wagman

        • 1. Re: Weblogic January PSU appears to cause major performance issues
          Sharmela-Oracle

          Hi Bill,

           

          You mentioned that the server had crashed .

          Do you have the core dumps ? or any file with the name hs_err_pid.log created ?

          This would help us to know the exact cause of the Crash.

           

          Thanks,

          Sharmela

          • 2. Re: Weblogic January PSU appears to cause major performance issues
            BillW

            Sharmela,

             

            I am unable to find any that fit that description nor am I able to find a core dump, of course I may not be searching for the correct name either. There are all sorts of log files but I am not sure the appropriate ones to look at. It seems like the patch has caused a problem with the managed server as once I stop that the load on the server goes way down. I suspect, although I don't know how to tell for sure, that the increased load caused the crash. I have looked at logs in the directories  Middleware/user_projects/domains/IDMDomain/servers/wls_ods1/logs & Middleware/user_projects/domains/IDMDomain/servers/AdminServer/logs but I am unable to really understand what is being reported there. If you have specific suggestions as to what else to look for plese let me know.


            Thank you.

            Bill Wagman

            • 3. Re: Weblogic January PSU appears to cause major performance issues
              Sharmela-Oracle

              Hi Bill,

               

              From the server logs that you are checking, please post the messages that are logged exactly during the time of issue.

              I can check and let you know.

               

              Thanks,

              Sharmela

              • 4. Re: Weblogic January PSU appears to cause major performance issues
                BillW

                Sharmela, are you referring to the var/log/messages file?

                 

                Thank you.

                • 5. Re: Weblogic January PSU appears to cause major performance issues
                  BillW

                  Sharmela,

                   

                  Here are some entres from the /var/log/messages file. Unfortunately I didn't pinpoint exact times but the problem occurred 2 or 3 times over the course of the morning. Here are some messages from around approximate times I was encountering errors. It shows shutting down at 09:57, a high load average at 10:19, that was the typical symptom I saw when monitoring. There are numerous LDT(4) in loadavg messages as well. If there is anything else I should look for in the messages file please let me know. It is clearly too large to post here.

                   

                   

                   

                  Mar 11 09:41:44 oracle-ldap3 cfenvd[5141]:  LDT(7) in otherprocs chi = 9.27 thresh 9.23

                  Mar 11 09:41:44 oracle-ldap3 cfenvd[5141]:  LDT_BUF (otherprocs): Rot  85.00 88.00 88.00 88.00 88.00 86.00 86.00 89.00 90.00 *91.00*

                  Mar 11 09:41:45 oracle-ldap3 cfenvd[5141]:  LDT(7) in www_out chi = 1.83 thresh 0.20

                  Mar 11 09:41:45 oracle-ldap3 cfenvd[5141]:  LDT_BUF (www_out): Rot  0.00 0.00 *2.00* 0.00 0.00 0.00 0.00 0.00 0.00 *1.00*

                  Mar 11 09:41:45 oracle-ldap3 cfenvd[5141]:  LDT(7) in ssh_in chi = 2.37 thresh 1.15

                  Mar 11 09:41:45 oracle-ldap3 cfenvd[5141]:  LDT_BUF (ssh_in): Rot  1.00 1.00 1.00 1.00 1.00 1.00 1.00 *2.00* *3.00* *3.00*

                   

                  Mar 11 09:42:38 oracle-ldap3 snmpd[14207]: Connection from UDP: [128.120.34.194]:56924

                  Mar 11 09:44:15 oracle-ldap3 cfenvd[5141]:  LDT(8) in otherprocs chi = 9.24 thresh 9.23

                  Mar 11 09:44:15 oracle-ldap3 cfenvd[5141]:  LDT_BUF (otherprocs): Rot  88.00 88.00 88.00 88.00 86.00 86.00 89.00 90.00 *91.00* *91.00*

                  Mar 11 09:44:15 oracle-ldap3 cfenvd[5141]:  LDT(8) in ssh_in chi = 2.24 thresh 1.15

                  Mar 11 09:44:15 oracle-ldap3 cfenvd[5141]:  LDT_BUF (ssh_in): Rot  1.00 1.00 1.00 1.00 1.00 1.00 *2.00* *3.00* *3.00* *3.00*

                  Mar 11 09:46:45 oracle-ldap3 cfenvd[5141]:  LDT(9) in ssh_in chi = 2.12 thresh 1.15

                  Mar 11 09:46:45 oracle-ldap3 cfenvd[5141]:  LDT_BUF (ssh_in): Rot  1.00 1.00 1.00 1.00 1.00 *2.00* *3.00* *3.00* *3.00* *3.00*

                   

                  Mar 11 09:56:37 oracle-ldap3 snmpd[14207]: Connection from UDP: [128.120.34.25]:51846

                  Mar 11 09:56:37 oracle-ldap3 last message repeated 2 times

                  Mar 11 09:56:48 oracle-ldap3 cfenvd[5141]:  LDT(3) in wwws_out chi = 3.81 thresh 0.50

                  Mar 11 09:56:48 oracle-ldap3 cfenvd[5141]:  LDT_BUF (wwws_out): Rot  0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 *7.00* *4.00*

                  Mar 11 09:57:09 oracle-ldap3 gconfd (oracle-1276): GConf server is not in use, shutting down.

                  Mar 11 09:57:09 oracle-ldap3 gconfd (oracle-1276): Exiting

                   

                  Mar 11 10:19:23 oracle-ldap3 cfenvd[5141]:  LDT(2) in loadavg chi = 25.35 thresh 23.54

                  Mar 11 10:19:23 oracle-ldap3 cfenvd[5141]:  LDT_BUF (loadavg): Rot  98.00 20.00 82.00 126.00 173.00 193.00 29.00 218.00 71.00 *300.00*

                  Mar 11 10:19:33 oracle-ldap3 gconfd (oracle-8233): starting (version 2.14.0), pid 8233 user 'oracle'

                  Mar 11 10:19:34 oracle-ldap3 gconfd (oracle-8233): Resolved address "xml:readonly:/etc/gconf/gconf.xml.mandatory" to a read-only configuration source at position 0

                  Mar 11 10:19:34 oracle-ldap3 gconfd (oracle-8233): Resolved address "xml:readwrite:/usr/local/oracle/.gconf" to a writable configuration source at position 1

                  Mar 11 10:19:34 oracle-ldap3 gconfd (oracle-8233): Resolved address "xml:readonly:/etc/gconf/gconf.xml.defaults" to a read-only configuration source at position 2

                  Mar 11 10:19:44 oracle-ldap3 snmpd[14207]: Connection from UDP: [128.120.34.194]:42362

                   

                   

                  Mar 11 10:20:04 oracle-ldap3 gconfd (oracle-8233): Exiting

                  Mar 11 10:20:05 oracle-ldap3 gconfd (oracle-8344): starting (version 2.14.0), pid 8344 user 'oracle'

                  Mar 11 10:20:05 oracle-ldap3 gconfd (oracle-8344): Resolved address "xml:readonly:/etc/gconf/gconf.xml.mandatory" to a read-only configuration source at position 0

                  Mar 11 10:20:05 oracle-ldap3 gconfd (oracle-8344): Resolved address "xml:readwrite:/usr/local/oracle/.gconf" to a writable configuration source at position 1

                  Mar 11 10:20:05 oracle-ldap3 gconfd (oracle-8344): Resolved address "xml:readonly:/etc/gconf/gconf.xml.defaults" to a read-only configuration source at position 2

                  Mar 11 10:21:37 oracle-ldap3 snmpd[14207]: Connection from UDP: [128.120.34.25]:50487

                   

                   

                  Mar 11 10:26:20 oracle-ldap3 shutdown[9012]: shutting down for system reboot

                  Mar 11 10:26:24 oracle-ldap3 init: Switching to runlevel: 6

                  Mar 11 10:26:26 oracle-ldap3 smartd[5240]: smartd received signal 15: Terminated

                  Mar 11 10:26:26 oracle-ldap3 smartd[5240]: smartd is exiting (exit status 0)

                  Mar 11 10:26:27 oracle-ldap3 avahi-daemon[5086]: Got SIGTERM, quitting.

                  Mar 11 10:26:27 oracle-ldap3 avahi-daemon[5086]: Leaving mDNS multicast group on interface eth0.IPv6 with address fe80::250:56ff:febf:131.

                  Mar 11 10:26:27 oracle-ldap3 avahi-daemon[5086]: Leaving mDNS multicast group on interface eth0.IPv4 with address 128.120.32.84.

                  Mar 11 10:26:28 oracle-ldap3 rhnsd[5044]: Exiting

                  Mar 11 10:26:31 oracle-ldap3 kernel: WARM shutting down of: CB... afs... BkG... CTrunc... AFSDB... RxEvent... UnmaskRxkSignals... RxListener...

                  Mar 11 10:26:31 oracle-ldap3 kernel:  ALL allocated tables

                  Mar 11 10:26:32 oracle-ldap3 snmpd[14207]: Received TERM or STOP signal...  shutting down...

                  Mar 11 10:26:33 oracle-ldap3 xinetd[4774]: Exiting...

                  Mar 11 10:27:08 oracle-ldap3 ntpd[4790]: ntpd exiting on signal 15

                  Mar 11 10:27:10 oracle-ldap3 hcid[4211]: Got disconnected from the system message bus

                  Mar 11 10:27:10 oracle-ldap3 auditd[3451]: The audit daemon is exiting.

                  Mar 11 10:27:10 oracle-ldap3 kernel: audit(1394558830.857:501405): audit_pid=0 old=3451 by auid=4294967295

                  Mar 11 10:27:10 oracle-ldap3 pcscd: pcscdaemon.c:572:signal_trap() Preparing for suicide

                  Mar 11 10:27:11 oracle-ldap3 pcscd: hotplug_libusb.c:376:HPRescanUsbBus() Hotplug stopped

                  Mar 11 10:27:11 oracle-ldap3 pcscd: readerfactory.c:1379:RFCleanupReaders() entering cleaning function

                  Mar 11 10:27:11 oracle-ldap3 pcscd: pcscdaemon.c:532:at_exit() cleaning /var/run

                  Mar 11 10:27:12 oracle-ldap3 kernel: Kernel logging (proc) stopped.

                  Mar 11 10:27:12 oracle-ldap3 kernel: Kernel log daemon terminating.

                  Mar 11 10:27:13 oracle-ldap3 exiting on signal 15

                   

                   

                  Mar 11 14:21:54 oracle-ldap3 cfenvd[4360]:  LDT(4) in otherprocs chi = 11.73 thresh 8.41

                  Mar 11 14:21:54 oracle-ldap3 cfenvd[4360]:  LDT_BUF (otherprocs): Rot  44.00 46.00 47.00 48.00 48.00 47.00 49.00 48.00 48.00 *85.00*

                  Mar 11 14:21:58 oracle-ldap3 cfenvd[4360]:  LDT(4) in loadavg chi = 68.37 thresh 23.46

                  Mar 11 14:21:58 oracle-ldap3 cfenvd[4360]:  LDT_BUF (loadavg): Rot  185.00 25.00 289.00 202.00 98.00 138.00 133.00 117.00 105.00 *1051.00*

                  Mar 11 14:24:29 oracle-ldap3 cfenvd[4360]:  LDT(5) in otherprocs chi = 10.17 thresh 8.41

                  Mar 11 14:24:29 oracle-ldap3 cfenvd[4360]:  LDT_BUF (otherprocs): Rot  46.00 47.00 48.00 48.00 47.00 49.00 48.00 48.00 *85.00* *78.00*

                  Mar 11 14:24:29 oracle-ldap3 cfenvd[4360]:  LDT(5) in loadavg chi = 23.76 thresh 23.46

                  Mar 11 14:24:29 oracle-ldap3 cfenvd[4360]:  LDT_BUF (loadavg): Rot  25.00 289.00 202.00 98.00 138.00 133.00 117.00 105.00 *1051.00* *393.00*

                  Mar 11 14:24:44 oracle-ldap3 snmpd[4071]: Connection from UDP: [128.120.34.194]:33490

                  Mar 11 14:24:44 oracle-ldap3 snmpd[4071]: Received SNMP packet(s) from UDP: [128.120.34.194]:33490

                  Mar 11 14:24:44 oracle-ldap3 snmpd[4071]: Connection from UDP: [128.120.34.194]:33490

                  Mar 11 14:24:44 oracle-ldap3 snmpd[4071]: Connection from UDP: [128.120.34.194]:33490

                  Mar 11 14:25:28 oracle-ldap3 gconfd (oracle-8141): starting (version 2.14.0), pid 8141 user 'oracle'

                  Mar 11 14:25:28 oracle-ldap3 gconfd (oracle-8141): Resolved address "xml:readonly:/etc/gconf/gconf.xml.mandatory" to a read-only configuration source at position 0

                  Mar 11 14:25:28 oracle-ldap3 gconfd (oracle-8141): Resolved address "xml:readwrite:/usr/local/oracle/.gconf" to a writable configuration source at position 1

                  Mar 11 14:25:28 oracle-ldap3 gconfd (oracle-8141): Resolved address "xml:readonly:/etc/gconf/gconf.xml.defaults" to a read-only configuration source at position 2

                  Mar 11 14:26:36 oracle-ldap3 snmpd[4071]: Connection from UDP: [128.120.34.25]:34332

                   

                   

                  Thank you.

                  Bill Wagman

                  • 6. Re: Weblogic January PSU appears to cause major performance issues
                    Puneeth-Oracle

                    Hi,

                     

                    Please check the Weblogic server logs and nohup / out file when the issue was seen.

                     

                    Eg : Logs are located in :  " C:\Oracle\Middleware\Oracle_Home\user_projects\domains\base_domain\servers\AdminServer\logs "

                     

                    These logs should have an error trace when the issue happened, please post it here.

                     

                    Regards,

                    Puneeth