Notification Mailers Down

1. Re: Notification Mailers Down

Hussein Sawwan-Oracle Aug 10, 2011 9:23 AM (in response to NewAppsDBA)

I can see status as:

This automatic Service Component has been restarted the maximum of 10 times after stopping with error. Thus, it has been system deactivated -> java.lang.RuntimeException:

This is a generic error -- Please check the workflow log file for the complete error message.

Thanks,
Hussein

2. Re: Notification Mailers Down

NewAppsDBA Aug 10, 2011 10:09 AM (in response to Hussein Sawwan-Oracle)

HI,

I found :

[Aug 9, 2011 3:28:17 PM GST]:1312889297891:-1:-1:erptest.group.com:172.16.10.6:-1:-1:1:20420:SYSADMIN(0):-1:Thread[BES Dispatch Thread,5,main]:172.16.10.6:69020:1312889287701:13:UNEXPECTED:[SVC-GSM-WFMLRSVC-2041214 : oracle.apps.fnd.cp.gsc.SvcComponentContainer.onBusinessEvent(BusinessEvent)]:An unexpected RuntimeException or other Throwable occurred -> java.lang.RuntimeException: Connection refused
[Aug 9, 2011 3:28:17 PM GST]:1312889297891:-1:-1:erptest.group.com:172.16.10.6:-1:-1:1:20420:SYSADMIN(0):-1:Thread[BES Dispatch Thread,5,main]:172.16.10.6:69020:1312889287701:13:EXCEPTION:[SVC-GSM-WFMLRSVC-2041214 : oracle.apps.fnd.cp.gsc.SvcComponentStateMachine.updateComponentStatus(Logger, Connection, int, String, String)]:BEGIN (Logger{mLog=[$Header: AppsLog.java 120.4.12010000.2 2009/06/17 15:14:37 rsantis ship $ @18158190 {oracle.apps.fnd.profiles.ExtendedProfileStore@6d2380}],mUniqueId=SVC-GSM-WFMLRSVC-2041214,mLevel=4}, oracle.jdbc.driver.T4CConnection@9903f4, 10006, STOPPED_ERROR, Could not handle the control event -> java.lang.RuntimeException: Connection refused)
[GC 12883K->9128K(13472K), 0.0010060 secs]
[Aug 9, 2011 3:29:07 PM GST]:1312889347679:-1:-1:erptest.group.com:172.16.10.6:-1:-1:1:20420:SYSADMIN(0):-1:Thread[ComponentMonitor,5,main]:172.16.10.6:69020:1312888747380:1:EXCEPTION:[SVC-GSM-WFMLRSVC-2041214 : oracle.apps.fnd.cp.gsc.SvcComponentMonitor.startAutomaticComponents()]:Starting automatic component 10006
[Aug 9, 2011 3:29:07 PM GST]:1312889347740:-1:-1:erptest.group.com:172.16.10.6:-1:-1:1:20420:SYSADMIN(0):-1:Thread[BES Dispatch Thread,5,main]:172.16.10.6:69020:1312889347740:14:EXCEPTION:[SVC-GSM-WFMLRSVC-2041214 : oracle.apps.fnd.cp.gsc.SvcComponentContainer.onBusinessEvent(BusinessEvent)]:(BusinessEvent{name=oracle.apps.fnd.cp.gsc.SvcComponent.start, key=SVC:09-AUG-2011, priority=50, correlationId=null, sendDate=Tue Aug 09 15:29:07 GST 2011, receiveDate=null, From Agent: , To Agent: , Last Subscription= , Error Message=null, Error Stack=null, CONTAINER_TYPE=GSM, CONTAINER_PROCESS_ID=2041214, COMPONENT_ID=10006, BES_PAYLOAD_OBJECT=false, BES_FROM_AGENT=WF_CONTROL@VIS.GROUP.COM})
[Aug 9, 2011 3:29:17 PM GST]:1312889357744:-1:-1:erptest.group.com:172.16.10.6:-1:-1:1:20420:SYSADMIN(0):-1:Thread[BES Dispatch Thread,5,main]:172.16.10.6:69020:1312889347740:14:EXCEPTION:[SVC-GSM-WFMLRSVC-2041214 : oracle.apps.fnd.cp.gsc.SvcComponentContainer.handleComponentEvent(int, String, String)]:Successfully retrieved component details from the database
[Aug 9, 2011 3:29:17 PM GST]:1312889357837:-1:-1:erptest.group.com:172.16.10.6:-1:-1:1:20420:SYSADMIN(0):-1:Thread[BES Dispatch Thread,5,main]:172.16.10.6:69020:1312889347740:14:ERROR:[SVC-GSM-WFMLRSVC-2041214 : oracle.apps.fnd.cp.gsc.SvcComponentContainer.onBusinessEvent(BusinessEvent)]:Error in handling component event, oracle.apps.fnd.cp.gsc.SvcComponent.start, for component 10006
[Aug 9, 2011 3:29:17 PM GST]:1312889357838:-1:-1:erptest.group.com:172.16.10.6:-1:-1:1:20420:SYSADMIN(0):-1:Thread[BES Dispatch Thread,5,main]:172.16.10.6:69020:1312889347740:14:ERROR:[SVC-GSM-WFMLRSVC-2041214 : oracle.apps.fnd.cp.gsc.SvcComponentContainer.onBusinessEvent(BusinessEvent)]:java.lang.RuntimeException: Connection refused
at oracle.apps.fnd.wf.mailer.MailerUtils.isSocketConnectableNew(MailerUtils.java:300)
at oracle.apps.fnd.wf.mailer.SMTPUtils.isValidOutbound(SMTPUtils.java:522)
at oracle.apps.fnd.wf.mailer.Mailer.validateParameterValues(Mailer.java:1238)
at oracle.apps.fnd.cp.gsc.SvcComponent.performValidateParameterValues(SvcComponent.java:233)
at oracle.apps.fnd.cp.gsc.SvcComponent.start(SvcComponent.java:314)
at oracle.apps.fnd.cp.gsc.SvcComponentContainer.handleComponentEvent(SvcComponentContainer.java:2212)
at oracle.apps.fnd.cp.gsc.SvcComponentContainer.onBusinessEvent(SvcComponentContainer.java:301)
at oracle.apps.fnd.wf.bes.DispatchThread.run(DispatchThread.java:57)

I followed:
Oracle Workflow Release 12 Diagnostics [ID 469822.1]

1.

Ebusiness Suite Version
RELEASE_NAME
12.1.3

Applied ATG/WF Patches
BUG_NUMBER LAST_UPDATE_DATE PATCH ARU_RELEASE_NAME
2728236 31-MAY-2003 23:02:49 OWF.G INCLUDED IN 11.5.9 11i
3031977 29-OCT-2003 10:10:37 POST OWF.G ROLLUP 1 - 11.5.9.1 11i
3061871 29-OCT-2003 10:10:37 POST OWF.G ROLLUP 2 - 11.5.9.2 11i
3124460 29-OCT-2003 10:10:31 POST OWF.G ROLLUP 3 - 11.5.9.3 11i
3140000 01-JUL-2004 14:44:43 ORACLE APPLICATIONS RELEASE 11.5.10 MAINTENANCE PACK 11i
3240000 19-MAR-2005 05:30:20 11.5.10 ORACLE E-BUSINESS SUITE CONSOLIDATED UPDATE 1 11i
3258819 01-JUL-2004 14:49:46 OWF.H INCLUDED IN 11.5.10 11i
3262159 01-JUL-2004 14:50:19 FND.H INCLUDE OWF.H 11i
3262919 01-JUL-2004 14:50:29 FMWK.H 11i
3314376 01-JUL-2004 14:50:29 POST OWF.G ROLLUP 5 - 11.5.9.5 11i
3316333 01-JUL-2004 14:50:29 POST OWF.G ROLLUP 4 - 11.5.9.4.1 11i
3409889 01-JUL-2004 14:50:21 POST OWF.G ROLLUP 5 Consolidated Fixes For OWF.G RUP 5 11i
3438354 18-MAR-2005 02:50:56 11i.ATG_PF.H INCLUDE OWF.H 11i
3460000 27-OCT-2005 08:36:20 11.5.10 ORACLE E-BUSINESS SUITE CONSOLIDATED UPDATE 2 11i
3492743 01-JUL-2004 14:50:03 POST OWF.G ROLLUP 6 - 11.5.9.6 11i
3868138 27-OCT-2005 08:39:51 POST OWF.G ROLLUP 7 - 11.5.9.7 11i
4017300 19-MAR-2005 05:32:57 ATG_PF:11.5.10 Consolidated Update (CU1) for ATG Product Family 11i
4125550 27-OCT-2005 08:39:51 ATG_PF:11.5.10 Consolidated Update (CU2) for ATG Product Family 11i
4334965 24-JUL-2006 23:30:22 11i.ATG_PF.H RUP3 11i
4440000 17-JAN-2007 00:57:05 Oracle Applications Release 12 Maintenance Pack R12
4440000 31-MAR-2009 10:24:26 Oracle Applications Release 12 Maintenance Pack R12
5082400 24-APR-2007 01:39:19 12.0.1 Release Update Pack (RUP1) R12
5484000 18-JUL-2007 07:14:19 12.0.2 Release Update Pack (RUP2) R12
5907545 24-APR-2007 01:39:47 R12.ATG_PF.A.DELTA.1 R12
5907545 31-JAN-2008 04:37:43 R12.ATG_PF.A.DELTA.1 R12
5917344 18-JUL-2007 07:15:34 R12.ATG_PF.A.DELTA.2 R12
5917344 31-JAN-2008 04:37:44 R12.ATG_PF.A.DELTA.2 R12
6077669 26-NOV-2007 00:39:13 R12.ATG_PF.A.DELTA.3 R12
6077669 31-JAN-2008 04:37:43 R12.ATG_PF.A.DELTA.3 R12
6141000 26-NOV-2007 00:37:18 12.0.3 Release Update Pack (RUP3) R12
6272680 31-JAN-2008 04:37:44 R12.ATG_PF.A.DELTA.4 R12
6435000 31-JAN-2008 04:35:21 12.0.4 RELEASE UPDATE PACK (RUP4) R12
7303030 30-MAR-2009 21:08:10 12.1.1 Maintenance Pack R12
7303033 15-JUL-2011 11:23:08 Oracle E-Business Suite 12.1.2 Release Update Pack (RUP2) R12
7651091 15-JUL-2011 11:23:29 Oracle Applications Technology Release Update Pack 2 for 12.1 (R12.ATG_PF.B.DELTA.2) R12
8919491 15-JUL-2011 11:23:11 Oracle Applications Technology 12.1.3 Product Family Release Update Pack R12
9239089 14-JUL-2011 15:38:34 Oracle Applications DBA 12.1.3 Product Release Update Pack R12
9239090 15-JUL-2011 11:19:43 ORACLE E-BUSINESS SUITE 12.1.3 RELEASE UPDATE PACK R12

2

SQL> @$FND_TOP/sql/wfver.sql

PL/SQL procedure successfully completed.

Server Version (last official install)

Version
------------------------------
2.6.0

Local System Name

The local System is ASSIGNED to VIS. GROUP.COM

PL/SQL procedure successfully completed.

Local System Status

Status
----------
ENABLED

Database Instance Details

Name Version Status
------------------------------ ------------------------------ ----------
VIS 11.1.0.7.0 ACTIVE

Database Init.ora Parameter Values

Name Value
------------------------------ -----------------------------------
aq_tm_processes 1
job_queue_processes 2
nls_language american
nls_territory america
utl_file_dir /usr/tmp, /usr/tmp, /testbak/oravis
/VIS/db/tech_st/11.1.0/appsutil/out
bound/VIS_erptest, /usr/tmp

XML Parser version

Version
---------------------------------------------------
Oracle XML Developers Kit 11.1.0.7.0 - Production

XML Parser PL/SQL Package Installation Status

no rows selected

XML Parser Schema Java Class Version

Status Details
----------------------------------------------------------------------
README_xmlparser.txt INSTALLED and VALID in APPS Schema
/379229ed_javaxxmlparsersDocum INSTALLED and VALID in SYS Schema
/bc3b995a_javaxxmlparsersSAXPa INSTALLED and VALID in SYS Schema

Queue Information

Owner Name Status Enqueue Dequeue Retries Retention Delay
------- ------------------- ------- ------- ------- ------- --------- -------
APPLSYS ECX_INBOUND VALID YES YES 5 0 120
APPLSYS ECX_IN_OAG_Q VALID YES YES 5 0 120
APPLSYS ECX_OUTBOUND VALID YES YES 5 0 120
APPS WF_BPEL_Q VALID YES YES 5 86400 0
APPLSYS WF_CONTROL VALID YES YES 5 0 0
APPLSYS WF_DEFERRED VALID YES YES 5 86400 3600
APPLSYS WF_DEFERRED_QUEUE_M VALID YES YES 5 0 0
APPLSYS WF_ERROR VALID YES YES 5 0 3600
APPLSYS WF_IN VALID YES YES 5 604800 0
APPLSYS WF_INBOUND_QUEUE VALID YES YES 5 0 0
APPLSYS WF_JAVA_DEFERRED VALID YES YES 5 86400 3600
APPLSYS WF_JAVA_ERROR VALID YES YES 5 0 3600
APPLSYS WF_JMS_IN VALID YES YES 5 0 0
APPLSYS WF_JMS_JMS_OUT VALID YES YES 5 0 0
APPLSYS WF_JMS_OUT VALID YES YES 5 0 0
APPLSYS WF_NOTIFICATION_IN VALID YES YES 5 86400 3600
APPLSYS WF_NOTIFICATION_OUT VALID YES YES 5 86400 3600
APPLSYS WF_OUT VALID YES YES 5 604800 0
APPLSYS WF_OUTBOUND_QUEUE VALID YES YES 5 0 0
APPLSYS WF_REPLAY_IN VALID YES YES 5 604800 0
APPLSYS WF_REPLAY_OUT VALID YES YES 5 604800 0
APPLSYS WF_SMTP_O_1_QUEUE VALID YES YES 0 0 0
APPLSYS WF_WS_JMS_IN VALID YES YES 5 0 0
APPLSYS WF_WS_JMS_OUT VALID YES YES 5 0 0
APPLSYS WF_WS_SAMPLE VALID YES YES 5 0 0

25 rows selected.

Queue Index List Information

Queue Name Index Name Last Analyzed Column
-------------------- ------------------------- ----------------- ----------
ECX_INBOUND AQ$_ECX_INQUEUE_I 17-JUL-11 Q_NAME
STATE
ENQ_TIME
STEP_NO
CHAIN_NO
LOCAL_ORDE
R_NO

AQ$_ECX_INQUEUE_T 17-JUL-11 TIME_MANAG
ER_INFO

SYS_C00731320 17-JUL-11 MSGID

ECX_IN_OAG_Q AQ$_ECX_IN_OAG_Q_TABLE_I 17-JUL-11 Q_NAME
STATE
ENQ_TIME
STEP_NO
CHAIN_NO
LOCAL_ORDE
R_NO

AQ$_ECX_IN_OAG_Q_TABLE_T 17-JUL-11 TIME_MANAG
ER_INFO

SYS_C00731319 17-JUL-11 MSGID

ECX_OUTBOUND AQ$_ECX_OUTQUEUE_I 17-JUL-11 Q_NAME
STATE
ENQ_TIME
STEP_NO
CHAIN_NO
LOCAL_ORDE
R_NO

AQ$_ECX_OUTQUEUE_T 17-JUL-11 TIME_MANAG
ER_INFO

SYS_C00731323 17-JUL-11 MSGID

WF_BPEL_Q SYS_C001035294 17-JUL-11 MSGID

WF_CONTROL SYS_C00171017 17-JUL-11 MSGID

WF_DEFERRED SYS_C004426 17-JUL-11 MSGID
WF_DEFERRED_N1 17-JUL-11 CORRID
CORRID

WF_DEFERRED_QUEUE_M SYS_C004427 17-JUL-11 MSGID
WF_DEFERRED_TABLE_M_N1 17-JUL-11 CORRID
CORRID

WF_ERROR SYS_C004428 17-JUL-11 MSGID
WF_ERROR_N1 17-JUL-11 CORRID
CORRID

WF_IN SYS_C004445 17-JUL-11 MSGID

WF_INBOUND_QUEUE AQ$_WF_INBOUND_TABLE_I 17-JUL-11 Q_NAME
Q_NAME
STATE
STATE
PRIORITY
ENQ_TIME
STEP_NO
ENQ_TIME
STEP_NO
CHAIN_NO
LOCAL_ORDE
R_NO

CHAIN_NO
LOCAL_ORDE
R_NO

AQ$_WF_INBOUND_TABLE_T 17-JUL-11 TIME_MANAG
ER_INFO

TIME_MANAG
ER_INFO

SYS_C004446 17-JUL-11 MSGID
WF_INBOUND_TABLE_PK 17-JUL-11 DEQUEUE_MS
GID

WF_JAVA_DEFERRED SYS_C00356557 17-JUL-11 MSGID
WF_JAVA_DEFERRED_N1 17-JUL-11 CORRID
CORRID

WF_JAVA_ERROR SYS_C00356578 17-JUL-11 MSGID
WF_JAVA_ERROR_N1 17-JUL-11 CORRID
CORRID

WF_JMS_IN SYS_C00170997 17-JUL-11 MSGID

WF_JMS_JMS_OUT SYS_C00356600 17-JUL-11 MSGID

WF_JMS_OUT SYS_C00171007 17-JUL-11 MSGID

WF_NOTIFICATION_IN SYS_C00171027 17-JUL-11 MSGID
WF_NOTIFICATION_IN_N1 17-JUL-11 CORRID
CORRID

WF_NOTIFICATION_OUT SYS_C00366810 17-JUL-11 MSGID
WF_NOTIFICATION_OUT_N1 17-JUL-11 CORRID

WF_OUT SYS_C004561 17-JUL-11 MSGID

WF_OUTBOUND_QUEUE AQ$_WF_OUTBOUND_TABLE_I 17-JUL-11 Q_NAME
Q_NAME
STATE
STATE
PRIORITY
ENQ_TIME
ENQ_TIME
STEP_NO
CHAIN_NO
STEP_NO
LOCAL_ORDE
R_NO

CHAIN_NO
LOCAL_ORDE
R_NO

AQ$_WF_OUTBOUND_TABLE_T 17-JUL-11 TIME_MANAG
ER_INFO

TIME_MANAG
ER_INFO

SYS_C004562 17-JUL-11 MSGID
WF_OUTBOUND_TABLE_PK 17-JUL-11 DEQUEUE_MS
GID

WF_REPLAY_IN SYS_C004575 17-JUL-11 MSGID

WF_REPLAY_OUT SYS_C004576 17-JUL-11 MSGID

WF_SMTP_O_1_QUEUE AQ$_WF_SMTP_O_1_TABLE_I 17-JUL-11 Q_NAME
Q_NAME
STATE
STATE
PRIORITY
PRIORITY
ENQ_TIME
ENQ_TIME
STEP_NO
STEP_NO
CHAIN_NO
CHAIN_NO
LOCAL_ORDE
R_NO

LOCAL_ORDE
R_NO

AQ$_WF_SMTP_O_1_TABLE_T 17-JUL-11 TIME_MANAG
ER_INFO

TIME_MANAG
ER_INFO

SYS_C001093454 17-JUL-11 MSGID

WF_WS_JMS_IN SYS_C00356561 17-JUL-11 MSGID

WF_WS_JMS_OUT SYS_C00356584 17-JUL-11 MSGID

WF_WS_SAMPLE SYS_C00356610 17-JUL-11 MSGID

107 rows selected.

Queue Subscriber Rule View Status
--------------------------------------------------------------------------
Queue_Name Name Address Protocol
--------------------------------------------------
AQ$_WF_DEFERRED_S Subscriber Information:
--------------------------------------------------
Queue Name : 0, Name : , Address : , Protocol :
Queue Name : WF_DEFERRED, Name : WF_DEFERRED, Address : , Protocol : 0
Queue Name : WF_DEFERRED, Name : , Address : "APPLSYS"."WF_DEFERRED", Protocol :
0
--------------------------------------------------
.
Queue_Name Name Address Protocol
--------------------------------------------------
AQ$_WF_DEFERRED_TABLE_M_S Subscriber Information:
--------------------------------------------------
Queue Name : 0, Name : , Address : , Protocol :
Queue Name : WF_DEFERRED_QUEUE_M, Name : APPS, Address : , Protocol : 0
Queue Name : WF_DEFERRED_QUEUE_M, Name : , Address :
"APPLSYS"."WF_DEFERRED_QUEUE_M", Protocol : 0
--------------------------------------------------
.
Queue_Name Name Address Protocol
--------------------------------------------------
AQ$_WF_ERROR_S Subscriber Information:
--------------------------------------------------
Queue Name : 0, Name : , Address : , Protocol :
Queue Name : WF_ERROR, Name : , Address : "APPLSYS"."WF_ERROR", Protocol : 0
Queue Name : AQ$_WF_ERROR_E, Name : , Address : "APPLSYS"."AQ$_WF_ERROR_E",
Protocol : 0
Queue Name : WF_ERROR, Name : WF_ERROR, Address : , Protocol : 0
--------------------------------------------------
.
Queue_Name Name Address Protocol
--------------------------------------------------
AQ$_WF_IN_S Subscriber Information:
--------------------------------------------------
Queue Name : 0, Name : , Address : , Protocol :
Queue Name : WF_IN, Name : , Address : "APPLSYS"."WF_IN", Protocol : 0
Queue Name : AQ$_WF_IN_E, Name : , Address : "APPLSYS"."AQ$_WF_IN_E", Protocol :
0
Queue Name : WF_IN, Name : WF_IN, Address : , Protocol : 0
--------------------------------------------------
.
Queue_Name Name Address Protocol
--------------------------------------------------
AQ$_WF_JAVA_DE

3. Re: Notification Mailers Down

Helios-GunesEROL Aug 10, 2011 10:52 AM (in response to NewAppsDBA)

Hi;

An unexpected RuntimeException or other Throwable occurred -> java.lang.RuntimeException: Connection refused

Is there any firewall

Regard
Helios

4. Re: Notification Mailers Down

NewAppsDBA Aug 11, 2011 4:00 AM (in response to Helios-GunesEROL)

HI,

Is there any firewall

NO

Thanks

Vishwa

Edited by: Vishwa on Aug 10, 2011 9:00 PM

5. Re: Notification Mailers Down

Hussein Sawwan-Oracle Aug 11, 2011 4:14 AM (in response to NewAppsDBA)

Please see these docs.

Workflow Mailer not Sending Notifications java.net.ConnectException: Connection refused (errno:239) [ID 972161.1]
On E-Business Suite 11.5.10.2, after appying 11i.ATG_PF.H RUP6 Receive The Error: "Connection refused" In OAM [ID 960949.1]
Workflow Service Component is System Deactivated with Error Details : Could not handle the control event -> java.lang.OutOfMemoryError: unable to create new native thread [ID 842968.1]
Unable To Start Workflow Notification Mailer [ID 418329.1]
Java Mailer wont start OPEN_MAIL_FYI=The requested template does not exist [ID 331358.1]

Thanks,
Hussein

6. Re: Notification Mailers Down

NewAppsDBA Aug 11, 2011 4:37 AM (in response to Hussein Sawwan-Oracle)

Hi,

Th first ID looks like my prob .. but we havn't installed LOADBALANCER ?

Thanks
Vishwa

7. Re: Notification Mailers Down

Hussein Sawwan-Oracle Aug 11, 2011 4:49 AM (in response to NewAppsDBA)

Th first ID looks like my prob .. but we havn't installed LOADBALANCER ?

So, it is not applicable. What about the other docs? If none helps, then please go through (Oracle Workflow Release 12 Diagnostics [ID 469822.1]).

Thanks,
Hussein

8. Re: Notification Mailers Down

NewAppsDBA Aug 11, 2011 4:55 AM (in response to Hussein Sawwan-Oracle)

HI,

the 3rd doc.... resembles me ..

i dont have any value under that env..

shell set that .. & check?

Thanks
Vishwa

Edited by: Vishwa on Aug 10, 2011 9:55 PM

9. Re: Notification Mailers Down

Hussein Sawwan-Oracle Aug 11, 2011 5:07 AM (in response to NewAppsDBA)

You are on R12, so ignore that solution.

Please go through the doc referenced in my last update.

Thanks,
Hussein

Join the world’s largest interactive community dedicated to Oracle technologies.