adop all phases are failing with errors after addition of external node for iRec

2779587

    adop all phases are failing with errors after addition of external node for iRec. Problem Description --------------------------------------------------- Statement Handler Error Code: 20010 Statement Handler Error Message: ORA-20010: More than one admin node found: osextnp06,osebsnp06. Review and correct your configuration, contacting Oracle Support as needed. ORA-06512: at "APPS.AD_ZD_ADOP", line 3031 ORA-06512: at line 2 (DBD ERROR: OCIStmtExecute) Reference of statement-handler is valid, not attempting database reconnect. Failed to execute SQL statement: begin ad_zd_adop.INSERT_ADD_NODE_RECORDS; end; Error Message: ORA-20010: More than one admin node found: osextnp06,osebsnp06. Review and correct your configuration, contacting Oracle Support as needed. ORA-06512: at "APPS.AD_ZD_ADOP", line 3031 ORA-06512: at line 2 (DBD ERROR: OCIStmtExecute) [UNEXPECTED]Error adding information for new node [UNEXPECTED]Unrecoverable error occurred. Exiting current adop session.

      • 1. Re: adop all phases are failing with errors after addition of external node for iRec
        2779587

        ADOP is failing below errors. Can someone please check. adop phase=apply patches=21351238 hotpatch=yes [UNEXPECTED]Unable to find appltop_id for host osextnp06osebsnp06 in database *******FATAL ERROR******* PROGRAM : (/u01/app/EBS/R122/PRJ/fs1/EBSapps/appl/ad/12.0.0/bin/adzdoptl.pl) TIME : Wed Jun 24 14:10:54 2020 FUNCTION: ADOP::CommonUtil::getAppltopId [ Level 1 ] ERRORMSG: Unable to find appltop_id for host osextnp06osebsnp06 in database

        • 2. Re: adop all phases are failing with errors after addition of external node for iRec
          Magnanimous

          Please find below solution -

           

          Application Tier APPL_TOP name should be same across all the application tier nodes.

          1) Change context variable s_atName on Secondary node to Master Application    Node name .
          2) Executed auto-config and bounced services.

          3) Then adop is running successfully without any error

           

          Regards,

          Prashant Umap

          • 3. Re: adop all phases are failing with errors after addition of external node for iRec
            2779587

            Hi Prashant, I have done the steps. Still same issue. Validating system setup.     Node registry is valid.     [UNEXPECTED]Unable to find appltop_id for host osextnp06osebsnp06 in database *******FATAL ERROR******* PROGRAM : (/u01/app/EBS/R122/PRJ/fs1/EBSapps/appl/ad/12.0.0/bin/adzdoptl.pl) TIME    : Thu Jun 25 08:55:44 2020 FUNCTION: ADOP::CommonUtil::getAppltopId [ Level 1 ] ERRORMSG: Unable to find appltop_id for host osextnp06osebsnp06 in database Thanks, Rajesh B

            • 4. Re: adop all phases are failing with errors after addition of external node for iRec
              2779587

              now i am getting this error: COuld someone check on it Statement Handler Error Code: 20010 Statement Handler Error Message: ORA-20010: More than one admin node found: osebsnp06,osextnp06. Review and correct your configuration, contacting Oracle Support as needed. ORA-06512: at "APPS.AD_ZD_ADOP", line 3031 ORA-06512: at line 2 (DBD ERROR: OCIStmtExecute) Reference of statement-handler is valid, not attempting database reconnect. Failed to execute SQL statement: begin ad_zd_adop.INSERT_ADD_NODE_RECORDS; end; Error Message: ORA-20010: More than one admin node found: osebsnp06,osextnp06. Review and correct your configuration, contacting Oracle Support as needed. ORA-06512: at "APPS.AD_ZD_ADOP", line 3031 ORA-06512: at line 2 (DBD ERROR: OCIStmtExecute) [UNEXPECTED]Error adding information for new node [UNEXPECTED]Unrecoverable error occurred. Exiting current adop session.

              • 5. Re: adop all phases are failing with errors after addition of external node for iRec
                Magnanimous

                Hi,

                Do you see multiple admin node ? is this instance recently refreshed ?

                Select NODE_NAME,SUPPORT_ADMIN,DOMAIN from fnd_nodes;

                 

                If yes, please remove the nodes.

                Also you can try to drop the table and check

                SQLPlus#> truncate table fnd_oam_context_files;

                SQLPlus#> truncate table fnd_nodes;

                  SQLPlus#> truncate table adop_valid_nodes;

                 

                And try to run the adautocfg and re-rerun adop.

                 

                Prashant Umap

                • 6. Re: adop all phases are failing with errors after addition of external node for iRec
                  2779587

                  adop fs_clone is failing on external node. osebsnp06 master FS_CLONE COMPLETED 2020/06/25 20:14:00 2020/06/25 22:07:10 1:11:10 osextnp06 slave FS_CLONE FAILED 2020/06/25 20:14:00 10:64:44 Actual error is: [applmgr@osebsnp06 21351238]$ cat /u01/app/EBS/R122/PRJ/fs_ne/EBSapps/log/adop/7/20200625_201309/fs_clone/validate/remote_execution_result_level2.xml

                  osextnp06osextnp06_osextnp06pool-1-thread-1_osextnp06_osextnp06 [ERROR]: Inconsistency detected in context file Admin Server information between nodes or file systems. [WARNING]: There could be issues while validating the ports used for E-Business Suite instance against ports used in /etc/services. Refer the log file for more details. :failedfailed Verified if adminserver status and web admin status are in sync between RUN and Patch FS.

                  • 7. Re: adop all phases are failing with errors after addition of external node for iRec
                    2779587

                    So i have disabled below on DMZ node by using below, are these should be enabled to fix the inconsistency issues. <oa_service_status oa_var="s_adminserverstatus">disabled <oa_service_group_status oa_var="s_web_admin_status">disabled

                    • 8. Re: adop all phases are failing with errors after addition of external node for iRec
                      Magnanimous

                      After disabling the DMZ node, issue is resolved ?

                      • 9. Re: adop all phases are failing with errors after addition of external node for iRec
                        2779587

                        Yes initial issue (  ORA-20010: More than one admin node found: osextnp06,osebsnp06. ) is resolved after disabling . Now i am getting new issue. [ERROR]: Inconsistency detected in context file Admin Server information between nodes or file systems.

                        • 10. Re: adop all phases are failing with errors after addition of external node for iRec
                          Magnanimous

                          it seems files are not in sync on all nodes, Please check following files -

                          1. Sub-Validation: check the values of s_adminserverstatus between RUN and PATCH file system.

                          Error message:

                          Value of s_adminserverStatus context variable is not consistent between run and patch file systems. Admin Server can be enabled only on the primary node.

                          Corrective Action: Amend value of s_adminserverstatus context variable.

                          2. Sub-Validation: check the values of s_adminserverstatus between RUN file system and database

                          Error message:

                          ERROR: Context file value of s_adminserverstatus on run file system is not consistent between the file system and database context file.

                          Corrective Action: Correct the value of the s_adminserverstatus context variable in the run file system context file, then run AutoConfig.

                          3. Sub-Validation: check the value of s_web_admin_status between RUN file system and database

                          Error message:

                          ERROR: Context file value of s_web_admin_status on run file system is not consistent between the file system and database context file.

                          Corrective Action: Correct the value of the s_web_admin_status context variable in the run file system context file, then run AutoConfig.

                          4. Sub-Validation: check the value of s_web_admin_status between RUN and PATCH file system

                          Error message:

                          Value of s_web_admin_status context variable is not consistent between run and patch file systems.

                          Corrective Action: Amend value of s_web_admin_status context variable

                          • 11. Re: adop all phases are failing with errors after addition of external node for iRec
                            2779587

                            Verified this. RUN & PATCH FS are in sync only. But not sure about checking between RUN FS & Database. check the values of s_adminserverstatus between RUN file system and database. check the value of s_web_admin_status between RUN file system and database

                            • 12. Re: adop all phases are failing with errors after addition of external node for iRec
                              Magnanimous

                              Ok, Is it possible to clone this instance one more time ?

                              Can you send me the adautocfg log ?

                              • 13. Re: adop all phases are failing with errors after addition of external node for iRec
                                Magnanimous

                                Please try the following -

                                - If the run file system context file for a node is missing, run AutoConfig on the run file system of that node to synchronize with the value with the database.

                                - If the patch file system context file of a node is missing, run AutoConfig on the patch file system of that node with the -syncctx option as follows to synchronize with the value with the database.

                                 

                                sh <AD_TOP>/bin/adconfig.sh contextfile=<CONTEXT_FILE> -syncctx