Enterprise Manager

Hi,

I m constantly getting alerts in grid control related to BEA-337. My GC version is 11.1.0.1.

The alert is:

Incident (BEA-337 [WebLogicServer]) detected in /opt/oracle/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/adr/diag/ofm/GCDomain/EMGC_OMS1/alert/log.xml


The log file shows as follows:

Incident Id: 245
Incident Source: SYSTEM
Create Time: Mon Oct 18 10:31:57 EDT 2010
Problem Key: BEA-337 [WebLogicServer]
Error Message Id: BEA-337

Description
-----------
Incident detected using watch rule "StuckThread":
Watch time: Oct 18, 2010 10:31:57 AM EDT
Watch ServerName: EMGC_OMS1
Watch RuleType: Log
Watch Rule: (SEVERITY = 'Error') AND (MSGID = 'BEA-000337')
Watch DomainName: GCDomain
Watch Data:
DATE : Oct 18, 2010 10:31:57 AM EDT
SERVER : EMGC_OMS1
MESSAGE : [STUCK] ExecuteThread: '13' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "643" seconds working on the request "weblogic.servlet.internal.ServletRequestImpl@5218d4a[
GET /em/console/database/rec/bkpMgmt?jobDesc=2010-10-15T00%3A01%3A01.536&objType=jobRpt&event=reportByJob&target=adw3p&type=rac_database HTTP/1.1
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, application/x-shockwave-flash, application/x-ms-application, application/x-ms-xbap, application/vnd.ms-xpsdocument, application/xaml+xml, */*
Referer: https://bdln2023.na.pg.com:7799/em/console/database/rec/bkpMgmt?event=startSearchJob&objType=jobs&target=adw3p&type=rac_database
Accept-Language: en-us
UA-CPU: x86
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 2.0.50727; .NET CLR 1.1.4322; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)
Cookie: JSESSIONID=JLTgM8wGjkGLhLkdh9429nyvpQg2phRRBrvTjHdGNyvbcVgtWQng!-2006778521; oracle.uix=0^^GMT+5:30^p
ECID-Context: 1.004^iJZWkim37EvXWXM6Vb0002WE0003Q9;kXjE1ZDLIPJLj5PRj3RRiJVRgLRB^HOTXIPOnLRBmLROj2RQkTJPhKS
Connection: Keep-Alive
Proxy-Client-IP: 137.185.64.199
X-Forwarded-For: 137.185.64.199
X-WebLogic-KeepAliveSecs: 30
X-WebLogic-Force-JVMID: -2006778521

]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(Unknown Source)
oracle.net.ns.Packet.receive(Packet.java:293)
oracle.net.ns.DataPacket.receive(DataPacket.java:92)
oracle.net.ano.CryptoDataPacket.receive(Unknown Source)
oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:174)
oracle.net.ns.NetInputStream.read(NetInputStream.java:119)
oracle.net.ns.NetInputStream.read(NetInputStream.java:94)
oracle.net.ns.NetInputStream.read(NetInputStream.java:79)
oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:122)
oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:78)
oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1040)
oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1016)
oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:589)
oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:194)
oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:791)
oracle.jdbc.driver.T4CPreparedStatement.executeMaybeDescribe(T4CPreparedStatement.java:866)
oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1187)
oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3386)
oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3430)
oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1491)
oracle.sysman.util.jdbc.PreparedStatementWrapper.executeQuery(PreparedStatementWrapper.java:215)
oracle.sysman.util.jdbc.ConnectionWrapper.executeQuery(ConnectionWrapper.java:1275)
oracle.sysman.emo.util.rec.bkpmgt.BkpMgmtReport.doPopulate(BkpMgmtReport.java:699)
oracle.sysman.emo.util.rec.bkpmgt.BkpMgmtBaseBean.populate(BkpMgmtBaseBean.java:217)
oracle.sysman.db.util.rec.BkpMgmtController.populateBean(BkpMgmtController.java:172)
oracle.sysman.db.util.rec.BkpMgmtController.onReportByJob(BkpMgmtController.java:414)
oracle.sysman.db.util.rec.BkpMgmtController.onEvent(BkpMgmtController.java:302)
oracle.sysman.db.util.DBJobController.handleEvent(DBJobController.java:1001)
oracle.sysman.emSDK.svlt.PageHandler.handleRequest(PageHandler.java:578)
oracle.sysman.db.adm.RootController.handleRequest(RootController.java:248)
oracle.sysman.db.util.DBJobController.handleRequest(DBJobController.java:948)
oracle.sysman.db.util.UtilControllerResolver.handleRequest(UtilControllerResolver.java:128)
oracle.sysman.emSDK.svlt.EMServlet.myDoGet(EMServlet.java:738)
oracle.sysman.emSDK.svlt.EMServlet.doGet(EMServlet.java:329)
oracle.sysman.eml.app.Console.doGet(Console.java:352)
javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:292)
weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
oracle.sysman.eml.app.EMRepLoginFilter.doFilter(EMRepLoginFilter.java:311)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
oracle.sysman.core.app.exception.EMRuntimeExceptionFilter.doFilter(EMRuntimeExceptionFilter.java:58)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
oracle.sysman.emas.fwk.MASConnectionFilter.doFilter(MASConnectionFilter.java:41)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
oracle.sysman.eml.app.BrowserVersionFilter.doFilter(BrowserVersionFilter.java:122)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
oracle.sysman.emSDK.svlt.EMRedirectFilter.doFilter(EMRedirectFilter.java:102)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
oracle.sysman.eml.app.ContextInitFilter.doFilter(ContextInitFilter.java:502)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
oracle.adf.library.webapp.LibraryFilter.doFilter(LibraryFilter.java:159)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
oracle.dms.wls.DMSServletFilter.doFilter(DMSServletFilter.java:326)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
weblogic.servlet.internal.RequestEventsFilter.doFilter(RequestEventsFilter.java:27)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3592)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
weblogic.security.service.SecurityManager.runAs(SecurityManager.java:121)
weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2202)
weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2108)
weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1432)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
weblogic.work.ExecuteThread.run(ExecuteThread.java:173)

SUBSYSTEM : WebLogicServer
USERID : <WLS Kernel>
SEVERITY : Error
THREAD : [ACTIVE] ExecuteThread: '53' for queue: 'weblogic.kernel.Default (self-tuning)'
MSGID : BEA-000337
MACHINE : bdln2023.na.pg.com
TXID :
CONTEXTID :
TIMESTAMP : 1287412317766

Stack Trace
-----------
java.lang.Throwable
at oracle.dfw.impl.incident.DiagnosticsDataExtractorImpl.createIncident(DiagnosticsDataExtractorImpl.java:231)
at oracle.dfw.spi.weblogic.JMXWatchNotificationListener.handleNotification(JMXWatchNotificationListener.java:195)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor$ListenerWrapper.handleNotification(Unknown Source)
at javax.management.NotificationBroadcasterSupport.handleNotification(Unknown Source)
at javax.management.NotificationBroadcasterSupport$SendNotifJob.run(Unknown Source)
at javax.management.NotificationBroadcasterSupport$1.execute(Unknown Source)
at javax.management.NotificationBroadcasterSupport.sendNotification(Unknown Source)
at weblogic.management.jmx.modelmbean.WLSModelMBean.sendNotification(WLSModelMBean.java:824)
at weblogic.diagnostics.watch.JMXNotificationProducer.postJMXNotification(JMXNotificationProducer.java:79)
at weblogic.diagnostics.watch.JMXNotificationProducer.sendNotification(JMXNotificationProducer.java:104)
at com.bea.diagnostics.notifications.JMXNotificationService.send(JMXNotificationService.java:122)
at weblogic.diagnostics.watch.JMXNotificationListener.processWatchNotification(JMXNotificationListener.java:103)
at weblogic.diagnostics.watch.Watch.performNotifications(Watch.java:621)
at weblogic.diagnostics.watch.Watch.evaluateLogRuleWatch(Watch.java:546)
at weblogic.diagnostics.watch.WatchManager.evaluateLogEventRulesAsync(WatchManager.java:765)
at weblogic.diagnostics.watch.WatchManager.run(WatchManager.java:525)
at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:516)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)

Diagnostic Dumps
----------------

Name: dms.ecidctx
Dump Files: dms_ecidctx187_i245.dmp

Name: jvm.threads
Dump Files: jvm_threads188_i245.dmp

Name: dms.metrics
Dump Files: dms_metrics189_i245.dmp

Name: odl.logs
Dump Arguments: ecid=0000IizkFt937EvXWXM6Vb1Cj0l800002q
Dump Files: odl_logs190_i245.dmp

Name: odl.logs
Dump Arguments: timestamp=1287412317768
Dump Files: odl_logs191_i245.dmp

Name: wls.image
Dump Files: wls_image192_i245.zip




Please somebody help me on this....I m fed up of this error...


Thanks