Please ping Database node from application node and see the response whether it is pinging or not.
please review managers log and provide us the contents of logs for all of us to review and update you.
As most of the steps you tried already.
In my previous post I have done all pinging and its good, shutdown/ startup, reboot server. but to no avail
select CONCURRENT_QUEUE_NAME,CONTROL_CODE,TARGET_NODE,NODE_NAME from FND_CONCURRENT_QUEUES where concurrent_queue_name like '%'
CONCURRENT_QUEUE_NAME CONTROL_CODE TARGET_NODE NODE_NAME FNDCRM N ERP FNDICM A6DD6A ERP FNDSCH N ERP FNDTMTST ERP STANDARD N ERP PODAMGR N ERP RCVOLTM N ERP PASMGR N ERP WFMGSMS ERP INVMGR N ERP INVTMRPM N ERP MRPMGR N ERP CRPINQMGR N ERP WFMGSMD ERP WFALSNRSVC N ERP WFMLRSVC N ERP FTE_TXN_MANAGER E ERP WMSTAMGR E ERP XDP_MANAGER N ERP Debug_Service ERP C_AQCT_SVC E ERP IEU_SH_CS N ERP IEU_WL_CS N ERP XDP_CTRL_SVC E ERP XDP_Q_ORDER_SVC E ERP XDP_Q_WI_SVC E ERP XDP_Q_FA_SVC E ERP XDP_Q_FE_READY_SVC E ERP XDP_Q_EVENT_SVC E ERP XDP_Q_IN_MSG_SVC E ERP XDP_Q_TIMER_SVC E ERP XDP_APPL_SVC E ERP XDP_SMIT_SVC E ERP OKCCONCMGR N ERP IEXCONMGR N ERP OAMCOLMGR N ERP CREATE_ACCOUNTING N ERP WFWSSVC N ERP AMSDMIN E ERP FNDCPOPP N ERP FFTM E ERP DownloadProcessorNormalMode E ERP DownloadProcessorMigrationMode E ERP FNDSM_A6DD6A E A6DD6A FNDIM_A6DD6A E A6DD6A OAMGCS_A6DD6A E A6DD6A
I am reviewing this thread. After you ran FND_CONC_CLONE.SETUP_CLEAN, you ran autoconfig on dbTier and appTier(s)? If you do not, you have no fnd_nodes.
Please verify a6dd6a.compute-a467305.oraclecloud.internal is properly configured in /etc/hosts on dbTier/appTier per the 12.1.1 installation/update doc.
Oracle E-Business Suite Installation and Upgrade Notes Release 12 (12.1.1) for Linux x86-64 (Doc ID 761566.1)
Yes after running fnd node cleanup, I run autoconfig on both apps & db tiers.
$ cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
10.106.140.34 a6dd6a.compute-a467305.oraclecloud.internal a6dd6a
Are you sure all FNDLIBR and FNDSM processes are down when running cmclean.sql?
Please reset the target_nodes in fnd_concurrent_queues with the following and restart with adcmctl.sh start apps/apps_pw
create table applsys.fnd_concurrent_queues_bak as select * from fnd_concurrent_queues;
update fnd_concurrent_queues set target_node = NULL where target_node = 'ERP';
update fnd_concurrent_queues set control_code = NULL where control_code <> 'E';
update fnd_concurrent_queues set control_code = NULL where target_node = 'A6DD6A' and CONCURRENT_QUEUE_NAME not like 'FNDIM%';
If the concurrent managers start OK, you can drop the backup table.
Please post the latest ICM log after doing this.
The clue is EBS R12.1.3 from RHEL5/6 cloned to OL6 (this server)
I have similar EBS R12.1.3 for OL5/6 cloned to this server , but it working good.
When changing OS, you need to relink all executables with adadmin. You may also need to install the stub patches to 10.1.2/10.1.3 Oracle Homes if coming from RHEL 5.
Rapid Clone only relinks the Oracle Home, not the EBS executables with the $APPL_TOP.
Yes > I am sure all FNDLIBR and FNDSM processes are down when running cmclean.sql.
SQL> create table applsys.fnd_concurrent_queues_bak as select * from fnd_concurrent_queues;
SQL> update fnd_concurrent_queues set target_node = NULL where target_node = 'ERP';
0 rows updated.
SQL> update fnd_concurrent_queues set control_code = NULL where control_code <> 'E';
21 rows updated.
SQL> update fnd_concurrent_queues set control_code = NULL where target_node = 'A6DD6A' and CONCURRENT_QUEUE_NAME not like 'FNDIM%';
1 row updated.
But still same issue when I startup apps tier
Then try shutting down app tier again, relink all executables with adadmin, should relink ad first manually with
adrelink.sh force=y "ad ALL"
Then adadmin relink all executables
Then run cmclean.sql
Then restart the apps.
I check the log but it did not create for today Jun 23 10AM?
[appmoli@a6dd6a log]$ ls -lrt
-rw-r--r-- 1 appmoli dba 718 Jun 21 06:23 L1991627.log
-rw-r--r-- 1 appmoli dba 718 Jun 21 06:23 L1991628.log
-rw-r--r-- 1 appmoli dba 554233 Jun 22 00:50 MOLI_0621.mgr
-rw-r--r-- 1 appmoli dba 718 Jun 22 01:36 L1991652.log
-rw-r--r-- 1 appmoli dba 718 Jun 22 01:36 L1991653.log
-rw-r--r-- 1 appmoli dba 718 Jun 22 01:43 L1991655.log
-rw-r--r-- 1 appmoli dba 718 Jun 22 01:43 L1991656.log
-rwxrwxrwx 1 appmoli dba 105620 Jun 22 22:11 MOLI_0622.mgr
[appmoli@a6dd6a log]$ pwd
I also recommend relinking 10.1.2/10.1.3 for good measure to ensure it is successful.
R11i / R12 : How to relink an E-Business Suite Installation of Release 11i and Release 12.x ? (Doc ID 356878.1)
Done > adrelink.sh force=y "ad ALL"
Relink all executables .
Still No Manager
Yes. You should verify 10.1.2 relinks OK too from the note.
Wait I check 10.1.2 ...brb
Link is already ok but still no manager
How can I compare this to the another 12.1.3 instance that is working good on the same target server?
How can I compare this to the source instance that is working good at the source server?
I remember my source server has PH timezone, and the target server has US timezone.
Now I changed it to PH timezone, then I have new manager error message:
Starting MOLI_0623@MOLI Internal Concurrent Manager -- shell process ID 7807
Reviver is ENABLED
Found dead process: spid=(14147), cpid=(559393), ORA pid=(24), manager=(0/1)
establish_icm - Unable to insert ICM record
Call to establish_icm failed
The Internal Concurrent Manager has encountered an error.
Review concurrent manager log file for more detailed information. : 23-JUN-2017 13:00:48 -
Shutting down Internal Concurrent Manager : 23-JUN-2017 13:00:48
List of errors encountered:
_ 1 _
Routine AFPCSQ encountered an ORACLE error. .
Review your error messages for the cause of the error. (=<POINTER>)
APP-FND-01564: ORACLE error 12899 in insert_icm_record
Cause: insert_icm_record failed due to ORA-12899: value too large for column "APPLSYS"."FND_CONCURRENT_PROCESSES"."NODE_NAME" (actual: 31, maximum: 30).
The SQL statement being executed at the time of the error was: INSERT INTO FND_CONCURRENT_PROCESSES (CONCURRENT_PROCESS_ID, LAST_UPDATE_DATE, CREATION_DATE, CREATED_BY, LAST_UPDATED_BY, ORACLE_PROCESS_ID, QUEUE_APPLICATION_ID, CONCURRENT_QUEUE_ID, OS_PROCESS_ID, SESSION_ID, PROCESS_START_DATE, PROCESS_STATUS_CODE, MANAGER_TYPE, NODE_NAME, DB_NAME, DB_DOMAIN, SQLNET_STRING, DB_INSTANCE, Instance_Number) VALUES (:cpid, SYSDATE, SYSDATE, :mgrusrid, :mgrusrid2, :opid, :qappid, :qid, :ospid, :osid, SYSDATE, 'A', 0, :node, :dbname, :dbdomain, :dbinst, (Select instance_name from v$instance), (Select instance_number from v$instance)) and was executed from the file &ERRFILE.
List of errors encountered:
_ 1 _
Routine AFPCAL received failure code while parsing or running your
concurrent program CPMGR
Review your concurrent request log file for more detailed information.
Make sure you are passing arguments in the correct format.
The MOLI_0623@MOLI internal concurrent manager has terminated with status 1 - giving up.