Forum Stats

  • 3,851,536 Users
  • 2,263,995 Discussions
  • 7,904,771 Comments

Discussions

BEA - 337 alerts in GC

800104
800104 Member Posts: 86
edited Mar 4, 2011 6:50AM in Enterprise Manager
Hi,

I m constantly getting alerts in grid control related to BEA-337. My GC version is 11.1.0.1.

The alert is:

Incident (BEA-337 [WebLogicServer]) detected in /opt/oracle/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/adr/diag/ofm/GCDomain/EMGC_OMS1/alert/log.xml


The log file shows as follows:

Incident Id: 245
Incident Source: SYSTEM
Create Time: Mon Oct 18 10:31:57 EDT 2010
Problem Key: BEA-337 [WebLogicServer]
Error Message Id: BEA-337

Description
-----------
Incident detected using watch rule "StuckThread":
Watch time: Oct 18, 2010 10:31:57 AM EDT
Watch ServerName: EMGC_OMS1
Watch RuleType: Log
Watch Rule: (SEVERITY = 'Error') AND (MSGID = 'BEA-000337')
Watch DomainName: GCDomain
Watch Data:
DATE : Oct 18, 2010 10:31:57 AM EDT
SERVER : EMGC_OMS1
MESSAGE : [STUCK] ExecuteThread: '13' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "643" seconds working on the request "[email protected][
GET /em/console/database/rec/bkpMgmt?jobDesc=2010-10-15T00%3A01%3A01.536&objType=jobRpt&event=reportByJob&target=adw3p&type=rac_database HTTP/1.1
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, application/x-shockwave-flash, application/x-ms-application, application/x-ms-xbap, application/vnd.ms-xpsdocument, application/xaml+xml, */*
Referer: https://bdln2023.na.pg.com:7799/em/console/database/rec/bkpMgmt?event=startSearchJob&objType=jobs&target=adw3p&type=rac_database
Accept-Language: en-us
UA-CPU: x86
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 2.0.50727; .NET CLR 1.1.4322; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)
Cookie: JSESSIONID=JLTgM8wGjkGLhLkdh9429nyvpQg2phRRBrvTjHdGNyvbcVgtWQng!-2006778521; oracle.uix=0^^GMT+5:30^p
ECID-Context: 1.004^iJZWkim37EvXWXM6Vb0002WE0003Q9;kXjE1ZDLIPJLj5PRj3RRiJVRgLRB^HOTXIPOnLRBmLROj2RQkTJPhKS
Connection: Keep-Alive
Proxy-Client-IP: 137.185.64.199
X-Forwarded-For: 137.185.64.199
X-WebLogic-KeepAliveSecs: 30
X-WebLogic-Force-JVMID: -2006778521

]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(Unknown Source)
oracle.net.ns.Packet.receive(Packet.java:293)
oracle.net.ns.DataPacket.receive(DataPacket.java:92)
oracle.net.ano.CryptoDataPacket.receive(Unknown Source)
oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:174)
oracle.net.ns.NetInputStream.read(NetInputStream.java:119)
oracle.net.ns.NetInputStream.read(NetInputStream.java:94)
oracle.net.ns.NetInputStream.read(NetInputStream.java:79)
oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:122)
oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:78)
oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1040)
oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1016)
oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:589)
oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:194)
oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:791)
oracle.jdbc.driver.T4CPreparedStatement.executeMaybeDescribe(T4CPreparedStatement.java:866)
oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1187)
oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3386)
oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3430)
oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1491)
oracle.sysman.util.jdbc.PreparedStatementWrapper.executeQuery(PreparedStatementWrapper.java:215)
oracle.sysman.util.jdbc.ConnectionWrapper.executeQuery(ConnectionWrapper.java:1275)
oracle.sysman.emo.util.rec.bkpmgt.BkpMgmtReport.doPopulate(BkpMgmtReport.java:699)
oracle.sysman.emo.util.rec.bkpmgt.BkpMgmtBaseBean.populate(BkpMgmtBaseBean.java:217)
oracle.sysman.db.util.rec.BkpMgmtController.populateBean(BkpMgmtController.java:172)
oracle.sysman.db.util.rec.BkpMgmtController.onReportByJob(BkpMgmtController.java:414)
oracle.sysman.db.util.rec.BkpMgmtController.onEvent(BkpMgmtController.java:302)
oracle.sysman.db.util.DBJobController.handleEvent(DBJobController.java:1001)
oracle.sysman.emSDK.svlt.PageHandler.handleRequest(PageHandler.java:578)
oracle.sysman.db.adm.RootController.handleRequest(RootController.java:248)
oracle.sysman.db.util.DBJobController.handleRequest(DBJobController.java:948)
oracle.sysman.db.util.UtilControllerResolver.handleRequest(UtilControllerResolver.java:128)
oracle.sysman.emSDK.svlt.EMServlet.myDoGet(EMServlet.java:738)
oracle.sysman.emSDK.svlt.EMServlet.doGet(EMServlet.java:329)
oracle.sysman.eml.app.Console.doGet(Console.java:352)
javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:292)
weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
oracle.sysman.eml.app.EMRepLoginFilter.doFilter(EMRepLoginFilter.java:311)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
oracle.sysman.core.app.exception.EMRuntimeExceptionFilter.doFilter(EMRuntimeExceptionFilter.java:58)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
oracle.sysman.emas.fwk.MASConnectionFilter.doFilter(MASConnectionFilter.java:41)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
oracle.sysman.eml.app.BrowserVersionFilter.doFilter(BrowserVersionFilter.java:122)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
oracle.sysman.emSDK.svlt.EMRedirectFilter.doFilter(EMRedirectFilter.java:102)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
oracle.sysman.eml.app.ContextInitFilter.doFilter(ContextInitFilter.java:502)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
oracle.adf.library.webapp.LibraryFilter.doFilter(LibraryFilter.java:159)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
oracle.dms.wls.DMSServletFilter.doFilter(DMSServletFilter.java:326)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
weblogic.servlet.internal.RequestEventsFilter.doFilter(RequestEventsFilter.java:27)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3592)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
weblogic.security.service.SecurityManager.runAs(SecurityManager.java:121)
weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2202)
weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2108)
weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1432)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
weblogic.work.ExecuteThread.run(ExecuteThread.java:173)

SUBSYSTEM : WebLogicServer
USERID : <WLS Kernel>
SEVERITY : Error
THREAD : [ACTIVE] ExecuteThread: '53' for queue: 'weblogic.kernel.Default (self-tuning)'
MSGID : BEA-000337
MACHINE : bdln2023.na.pg.com
TXID :
CONTEXTID :
TIMESTAMP : 1287412317766

Stack Trace
-----------
java.lang.Throwable
at oracle.dfw.impl.incident.DiagnosticsDataExtractorImpl.createIncident(DiagnosticsDataExtractorImpl.java:231)
at oracle.dfw.spi.weblogic.JMXWatchNotificationListener.handleNotification(JMXWatchNotificationListener.java:195)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor$ListenerWrapper.handleNotification(Unknown Source)
at javax.management.NotificationBroadcasterSupport.handleNotification(Unknown Source)
at javax.management.NotificationBroadcasterSupport$SendNotifJob.run(Unknown Source)
at javax.management.NotificationBroadcasterSupport$1.execute(Unknown Source)
at javax.management.NotificationBroadcasterSupport.sendNotification(Unknown Source)
at weblogic.management.jmx.modelmbean.WLSModelMBean.sendNotification(WLSModelMBean.java:824)
at weblogic.diagnostics.watch.JMXNotificationProducer.postJMXNotification(JMXNotificationProducer.java:79)
at weblogic.diagnostics.watch.JMXNotificationProducer.sendNotification(JMXNotificationProducer.java:104)
at com.bea.diagnostics.notifications.JMXNotificationService.send(JMXNotificationService.java:122)
at weblogic.diagnostics.watch.JMXNotificationListener.processWatchNotification(JMXNotificationListener.java:103)
at weblogic.diagnostics.watch.Watch.performNotifications(Watch.java:621)
at weblogic.diagnostics.watch.Watch.evaluateLogRuleWatch(Watch.java:546)
at weblogic.diagnostics.watch.WatchManager.evaluateLogEventRulesAsync(WatchManager.java:765)
at weblogic.diagnostics.watch.WatchManager.run(WatchManager.java:525)
at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:516)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)

Diagnostic Dumps
----------------

Name: dms.ecidctx
Dump Files: dms_ecidctx187_i245.dmp

Name: jvm.threads
Dump Files: jvm_threads188_i245.dmp

Name: dms.metrics
Dump Files: dms_metrics189_i245.dmp

Name: odl.logs
Dump Arguments: ecid=0000IizkFt937EvXWXM6Vb1Cj0l800002q
Dump Files: odl_logs190_i245.dmp

Name: odl.logs
Dump Arguments: timestamp=1287412317768
Dump Files: odl_logs191_i245.dmp

Name: wls.image
Dump Files: wls_image192_i245.zip




Please somebody help me on this....I m fed up of this error...


Thanks

Best Answer

  • Ron
    Ron Member Posts: 3
    Answer ✓
    Hi,

    We logged an SR for the with Support and got the reply to add this in de emd.properties of the agent. The string now to ignore BEA-101020 and BEA-337 is:
    adrAlertLogAsErrorCodeExcludeRegex=.*BEA-(101020|337)\D.*

    And yes a restart of the agent is required.

    Regards,
    Ron.

Answers

  • 800104
    800104 Member Posts: 86
    Plz help...I didnt find anything in MOS also
  • EricvdS
    EricvdS Member Posts: 976 Bronze Trophy
    Looks more like a warning than an error. I your environment the default for the StuckThreadMaxTime setting is too narrow. You can change it, see:
    http://soa-bpel-esb.blogspot.com/2010/09/bea-000337-stuck-executethread-1-for.html

    Eric
  • Ron
    Ron Member Posts: 3
    We also see this critical BEA-337 alerts in GC 11G and raised a SR in MOS which is still open. We will check the StuckThreadMaxTime setting which EricvdS suggested.

    Ron.
  • EricvdS
    EricvdS Member Posts: 976 Bronze Trophy
    edited Mar 3, 2011 4:30PM
    This can be avoided according to bug info 10385351:

    WORKAROUND INFORMATION ======================
    To avoid grid control raising this incident: To add the following expression in
    AGENT_HOME/sysman/config/emd.properties of the OMS agent
    adrAlertLogAsErrorCodeExcludeRegex=.*BEA-(337)\D.*

    It does not say it in the note, but a stop and start of the agent is also needed (?)

    Also check patch 9914120:CANNOT CLEAR LOG FILE ALERTS FROM FUSION MIDDLEWARE TARGETS
    to be able to delete the alerts: Incident (BEA-337 [WebLogicServer]) detected in D:\oracle\product\GC11gR1
    The patch should be applied on the agent running on the OMS server.

    Bug 10385351 should be fixed in PSU 2.

    The same workaround can be used for BEA-101020 errors. See note 1139600.1.
    I am wondering what the adrAlertLogAsErrorCodeExcludeRegex should look like if you have the 337 and the 101020 error. Anybody?

    Eric
  • Ron
    Ron Member Posts: 3
    Answer ✓
    Hi,

    We logged an SR for the with Support and got the reply to add this in de emd.properties of the agent. The string now to ignore BEA-101020 and BEA-337 is:
    adrAlertLogAsErrorCodeExcludeRegex=.*BEA-(101020|337)\D.*

    And yes a restart of the agent is required.

    Regards,
    Ron.
  • EricvdS
    EricvdS Member Posts: 976 Bronze Trophy
    Thanks Ron,

    That's what I was looking for.

    Eric
This discussion has been closed.