This site is currently read-only as we are migrating to Oracle Forums for an improved community experience. You will not be able to initiate activity until January 30th, when you will be able to use this site as normal.

    Forum Stats

  • 3,889,699 Users
  • 2,269,776 Discussions
  • 7,916,823 Comments

Discussions

BEA - 337 alerts in GC

800104
800104 Member Posts: 86
edited Mar 4, 2011 6:50AM in Enterprise Manager
Hi,

I m constantly getting alerts in grid control related to BEA-337. My GC version is 11.1.0.1.

The alert is:

Incident (BEA-337 [WebLogicServer]) detected in /opt/oracle/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/adr/diag/ofm/GCDomain/EMGC_OMS1/alert/log.xml


The log file shows as follows:

Incident Id: 245
Incident Source: SYSTEM
Create Time: Mon Oct 18 10:31:57 EDT 2010
Problem Key: BEA-337 [WebLogicServer]
Error Message Id: BEA-337

Description
-----------
Incident detected using watch rule "StuckThread":
Watch time: Oct 18, 2010 10:31:57 AM EDT
Watch ServerName: EMGC_OMS1
Watch RuleType: Log
Watch Rule: (SEVERITY = 'Error') AND (MSGID = 'BEA-000337')
Watch DomainName: GCDomain
Watch Data:
DATE : Oct 18, 2010 10:31:57 AM EDT
SERVER : EMGC_OMS1
MESSAGE : [STUCK] ExecuteThread: '13' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "643" seconds working on the request "[email protected][
GET /em/console/database/rec/bkpMgmt?jobDesc=2010-10-15T00%3A01%3A01.536&objType=jobRpt&event=reportByJob&target=adw3p&type=rac_database HTTP/1.1
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, application/x-shockwave-flash, application/x-ms-application, application/x-ms-xbap, application/vnd.ms-xpsdocument, application/xaml+xml, */*
Referer: https://bdln2023.na.pg.com:7799/em/console/database/rec/bkpMgmt?event=startSearchJob&objType=jobs&target=adw3p&type=rac_database
Accept-Language: en-us
UA-CPU: x86
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 2.0.50727; .NET CLR 1.1.4322; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)
Cookie: JSESSIONID=JLTgM8wGjkGLhLkdh9429nyvpQg2phRRBrvTjHdGNyvbcVgtWQng!-2006778521; oracle.uix=0^^GMT+5:30^p
ECID-Context: 1.004^iJZWkim37EvXWXM6Vb0002WE0003Q9;kXjE1ZDLIPJLj5PRj3RRiJVRgLRB^HOTXIPOnLRBmLROj2RQkTJPhKS
Connection: Keep-Alive
Proxy-Client-IP: 137.185.64.199
X-Forwarded-For: 137.185.64.199
X-WebLogic-KeepAliveSecs: 30
X-WebLogic-Force-JVMID: -2006778521

]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(Unknown Source)
oracle.net.ns.Packet.receive(Packet.java:293)
oracle.net.ns.DataPacket.receive(DataPacket.java:92)
oracle.net.ano.CryptoDataPacket.receive(Unknown Source)
oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:174)
oracle.net.ns.NetInputStream.read(NetInputStream.java:119)
oracle.net.ns.NetInputStream.read(NetInputStream.java:94)
oracle.net.ns.NetInputStream.read(NetInputStream.java:79)
oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:122)
oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:78)
oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1040)
oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1016)
oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:589)
oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:194)
oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:791)
oracle.jdbc.driver.T4CPreparedStatement.executeMaybeDescribe(T4CPreparedStatement.java:866)
oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1187)
oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3386)
oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3430)
oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1491)
oracle.sysman.util.jdbc.PreparedStatementWrapper.executeQuery(PreparedStatementWrapper.java:215)
oracle.sysman.util.jdbc.ConnectionWrapper.executeQuery(ConnectionWrapper.java:1275)
oracle.sysman.emo.util.rec.bkpmgt.BkpMgmtReport.doPopulate(BkpMgmtReport.java:699)
oracle.sysman.emo.util.rec.bkpmgt.BkpMgmtBaseBean.populate(BkpMgmtBaseBean.java:217)
oracle.sysman.db.util.rec.BkpMgmtController.populateBean(BkpMgmtController.java:172)
oracle.sysman.db.util.rec.BkpMgmtController.onReportByJob(BkpMgmtController.java:414)
oracle.sysman.db.util.rec.BkpMgmtController.onEvent(BkpMgmtController.java:302)
oracle.sysman.db.util.DBJobController.handleEvent(DBJobController.java:1001)
oracle.sysman.emSDK.svlt.PageHandler.handleRequest(PageHandler.java:578)
oracle.sysman.db.adm.RootController.handleRequest(RootController.java:248)
oracle.sysman.db.util.DBJobController.handleRequest(DBJobController.java:948)
oracle.sysman.db.util.UtilControllerResolver.handleRequest(UtilControllerResolver.java:128)
oracle.sysman.emSDK.svlt.EMServlet.myDoGet(EMServlet.java:738)
oracle.sysman.emSDK.svlt.EMServlet.doGet(EMServlet.java:329)
oracle.sysman.eml.app.Console.doGet(Console.java:352)
javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:292)
weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
oracle.sysman.eml.app.EMRepLoginFilter.doFilter(EMRepLoginFilter.java:311)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
oracle.sysman.core.app.exception.EMRuntimeExceptionFilter.doFilter(EMRuntimeExceptionFilter.java:58)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
oracle.sysman.emas.fwk.MASConnectionFilter.doFilter(MASConnectionFilter.java:41)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
oracle.sysman.eml.app.BrowserVersionFilter.doFilter(BrowserVersionFilter.java:122)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
oracle.sysman.emSDK.svlt.EMRedirectFilter.doFilter(EMRedirectFilter.java:102)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
oracle.sysman.eml.app.ContextInitFilter.doFilter(ContextInitFilter.java:502)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
oracle.adf.library.webapp.LibraryFilter.doFilter(LibraryFilter.java:159)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
oracle.dms.wls.DMSServletFilter.doFilter(DMSServletFilter.java:326)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
weblogic.servlet.internal.RequestEventsFilter.doFilter(RequestEventsFilter.java:27)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3592)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
weblogic.security.service.SecurityManager.runAs(SecurityManager.java:121)
weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2202)
weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2108)
weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1432)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
weblogic.work.ExecuteThread.run(ExecuteThread.java:173)

SUBSYSTEM : WebLogicServer
USERID : <WLS Kernel>
SEVERITY : Error
THREAD : [ACTIVE] ExecuteThread: '53' for queue: 'weblogic.kernel.Default (self-tuning)'
MSGID : BEA-000337
MACHINE : bdln2023.na.pg.com
TXID :
CONTEXTID :
TIMESTAMP : 1287412317766

Stack Trace
-----------
java.lang.Throwable
at oracle.dfw.impl.incident.DiagnosticsDataExtractorImpl.createIncident(DiagnosticsDataExtractorImpl.java:231)
at oracle.dfw.spi.weblogic.JMXWatchNotificationListener.handleNotification(JMXWatchNotificationListener.java:195)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor$ListenerWrapper.handleNotification(Unknown Source)
at javax.management.NotificationBroadcasterSupport.handleNotification(Unknown Source)
at javax.management.NotificationBroadcasterSupport$SendNotifJob.run(Unknown Source)
at javax.management.NotificationBroadcasterSupport$1.execute(Unknown Source)
at javax.management.NotificationBroadcasterSupport.sendNotification(Unknown Source)
at weblogic.management.jmx.modelmbean.WLSModelMBean.sendNotification(WLSModelMBean.java:824)
at weblogic.diagnostics.watch.JMXNotificationProducer.postJMXNotification(JMXNotificationProducer.java:79)
at weblogic.diagnostics.watch.JMXNotificationProducer.sendNotification(JMXNotificationProducer.java:104)
at com.bea.diagnostics.notifications.JMXNotificationService.send(JMXNotificationService.java:122)
at weblogic.diagnostics.watch.JMXNotificationListener.processWatchNotification(JMXNotificationListener.java:103)
at weblogic.diagnostics.watch.Watch.performNotifications(Watch.java:621)
at weblogic.diagnostics.watch.Watch.evaluateLogRuleWatch(Watch.java:546)
at weblogic.diagnostics.watch.WatchManager.evaluateLogEventRulesAsync(WatchManager.java:765)
at weblogic.diagnostics.watch.WatchManager.run(WatchManager.java:525)
at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:516)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)

Diagnostic Dumps
----------------

Name: dms.ecidctx
Dump Files: dms_ecidctx187_i245.dmp

Name: jvm.threads
Dump Files: jvm_threads188_i245.dmp

Name: dms.metrics
Dump Files: dms_metrics189_i245.dmp

Name: odl.logs
Dump Arguments: ecid=0000IizkFt937EvXWXM6Vb1Cj0l800002q
Dump Files: odl_logs190_i245.dmp

Name: odl.logs
Dump Arguments: timestamp=1287412317768
Dump Files: odl_logs191_i245.dmp

Name: wls.image
Dump Files: wls_image192_i245.zip




Please somebody help me on this....I m fed up of this error...


Thanks

Best Answer

  • Ron
    Ron Member Posts: 3
    Answer ✓
    Hi,

    We logged an SR for the with Support and got the reply to add this in de emd.properties of the agent. The string now to ignore BEA-101020 and BEA-337 is:
    adrAlertLogAsErrorCodeExcludeRegex=.*BEA-(101020|337)\D.*

    And yes a restart of the agent is required.

    Regards,
    Ron.

Answers

  • 800104
    800104 Member Posts: 86
    Plz help...I didnt find anything in MOS also
  • EricvdS
    EricvdS ing. UtrechtMember Posts: 977 Bronze Trophy
    Looks more like a warning than an error. I your environment the default for the StuckThreadMaxTime setting is too narrow. You can change it, see:
    http://soa-bpel-esb.blogspot.com/2010/09/bea-000337-stuck-executethread-1-for.html

    Eric
  • Ron
    Ron Member Posts: 3
    We also see this critical BEA-337 alerts in GC 11G and raised a SR in MOS which is still open. We will check the StuckThreadMaxTime setting which EricvdS suggested.

    Ron.
  • EricvdS
    EricvdS ing. UtrechtMember Posts: 977 Bronze Trophy
    edited Mar 3, 2011 4:30PM
    This can be avoided according to bug info 10385351:

    WORKAROUND INFORMATION ======================
    To avoid grid control raising this incident: To add the following expression in
    AGENT_HOME/sysman/config/emd.properties of the OMS agent
    adrAlertLogAsErrorCodeExcludeRegex=.*BEA-(337)\D.*

    It does not say it in the note, but a stop and start of the agent is also needed (?)

    Also check patch 9914120:CANNOT CLEAR LOG FILE ALERTS FROM FUSION MIDDLEWARE TARGETS
    to be able to delete the alerts: Incident (BEA-337 [WebLogicServer]) detected in D:\oracle\product\GC11gR1
    The patch should be applied on the agent running on the OMS server.

    Bug 10385351 should be fixed in PSU 2.

    The same workaround can be used for BEA-101020 errors. See note 1139600.1.
    I am wondering what the adrAlertLogAsErrorCodeExcludeRegex should look like if you have the 337 and the 101020 error. Anybody?

    Eric
  • Ron
    Ron Member Posts: 3
    Answer ✓
    Hi,

    We logged an SR for the with Support and got the reply to add this in de emd.properties of the agent. The string now to ignore BEA-101020 and BEA-337 is:
    adrAlertLogAsErrorCodeExcludeRegex=.*BEA-(101020|337)\D.*

    And yes a restart of the agent is required.

    Regards,
    Ron.
  • EricvdS
    EricvdS ing. UtrechtMember Posts: 977 Bronze Trophy
    Thanks Ron,

    That's what I was looking for.

    Eric
This discussion has been closed.