1 2 Previous Next 20 Replies Latest reply on Dec 14, 2018 2:58 PM by Beauty_and_dBest

    Where to find the correct adop error?

    Beauty_and_dBest

      EBS 12.2.4

      OL7

      11gR2

       

       

      Hi ALL,

       

      I am running > adop  phase=prepare

       

      But I got error below:

       

      Completed execution of all tests. There are errors.

      =========================== START OF VALIDATION REPORT ====================

      Validation Results for Node: appclone03

      ---------------------------------------

      [ERROR]: At least one Oracle inventory check has failed.

      [WARNING]: There could be issues while validating the ports used for E-Business Suite instance against ports used in /etc/services. Refer the log file for more details.

      [WARNING]: Either some of the required entries in /etc/hosts file might be missing (e.g. localhost or hostname) OR the file /etc/hosts could not be read.

      Some validations failed, please check /home/appprod/PROD/fs_ne/EBSapps/log/adop/7/prepare_20181120_210302/PROD_appclone03/ADOPValidations_Tue_Nov_20_21_05_16_2018.log

       

      =========================== END OF VALIDATION REPORT ======================

              [UNEXPECTED]Error occurred while executing "perl /home/appprod/PROD/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPValidations.pl  -contextfile=/home/appprod/PROD/fs2/inst/apps/PROD_appclone03/appl/admin/PROD_appclone03.xml -patchctxfile=/home/appprod/PROD/fs1/inst/apps/PROD_appclone03/appl/admin/PROD_appclone03.xml -phase=prepare -logloc=/home/appprod/PROD/fs_ne/EBSapps/log/adop/7/prepare_20181120_210302/PROD_appclone03 -promptmsg=hide"

              [UNEXPECTED]Error 1 occurred while Executing txkADOPValidation script on appclone03

              Log file: /home/appprod/PROD/fs_ne/EBSapps/log/adop/7/adop_20181120_210302.log

       

      [STATEMENT] Please run adopscanlog utility, using the command

       

      "adopscanlog -latest=yes"

       

      to get the list of the log files along with snippet of the error message corresponding to each log file.

       

      adop exiting with status = 1 (Fail)

       

      [appprod@appclone03 ~]$ hostname

      appclone03

       

      [appprod@appclone03 ~]$ cat /etc/hosts

      127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4

      ::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

      192.168.243.191 appclone03.oracle.com.ph appclone03

      192.168.243.192 dbclone03.oracle.com.ph dbclone03

       

       

       

      Please help how to find the cause of the error?

      What "keyword" do I search the metalink or google?

       

       

      Kind regards,

      jc

        • 1. Re: Where to find the correct adop error?
          mdtaylor

          Hi jc,

           

          In the adop window/logs, there are warnings and errors.  I would search on the error:

           

          adop [ERROR]: At least one Oracle inventory check has failed.

           

          It appears the oraInventory is not complete or pointing to the incorrect location.

           

          Regards,

          Michael

          • 2. Re: Where to find the correct adop error?
            Beauty_and_dBest

            Thanks Michael,

             

            I checked the Oracle Inventory and I  got th following:

             

            <?xml version="1.0" standalone="yes" ?>

            <!-- Copyright (c) 2009 Oracle Corporation. All rights Reserved -->

            <!-- Do not modify the contents of this file by hand. -->

            <INVENTORY>

            <COMPOSITEHOME_LIST>

            </COMPOSITEHOME_LIST>

            <VERSION_INFO>

               <SAVED_WITH>10.1.0.6.0</SAVED_WITH>

               <MINIMUM_VER>2.1.0.6.0</MINIMUM_VER>

            </VERSION_INFO>

            <HOME_LIST>

            <HOME NAME="OH423466231" LOC="/home/appprod/PROD/fs1/FMW_Home/oracle_common" TYPE="O" IDX="1">

               <REFHOMELIST>

                  <REFHOME LOC="/home/appprod/PROD/fs1/FMW_Home/webtier"/>

               </REFHOMELIST>

            </HOME>

            <HOME NAME="OH1222067361" LOC="/home/appprod/PROD/fs1/FMW_Home/webtier" TYPE="O" IDX="2">

               <DEPHOMELIST>

                  <DEPHOME LOC="/home/appprod/PROD/fs1/FMW_Home/oracle_common"/>

               </DEPHOMELIST>

            </HOME>

            <HOME NAME="OH604432845" LOC="/home/appprod/PROD/fs1/FMW_Home/Oracle_EBS-app1" TYPE="O" IDX="3"/>

            <HOME NAME="PROD_TOOLS__home_appprod_PROD_fs1_EBSapps_10_1_2" LOC="/home/appprod/PROD/fs1/EBSapps/10.1.2" TYPE="O" IDX="4"/>

            </HOME_LIST>

            </INVENTORY>

             

            It seems that I do not have fs2 and fs_ne in the inventory?

            How do I know what other inventory folders is it looking for? for completeness check?

            Is there a command for Inventory completeness check is appsTier? Just like counterpart OPATCH lsinventory in dbTier?

             

            Please help....

             

             

            Kind regards,

            • 3. Re: Where to find the correct adop error?
              Beauty_and_dBest

              Hi Michael,

               

              I have new ADOP PREPARE error after fixing the inventory or maybe the inventory was not fixed correctly?

               

              [EVENT]The concurrent manager defined to process this request is not active in

              the current work shift. The request will not process within this work

              shift.  Contact your system administrator to verify that the concurrent

              manager work shifts are defined correctly.

              [EVENT][END   2018/11/21 19:56:55] Checking status of the ADZDPATCH concurrent program
              [EVENT][START 2018/11/21 19:56:56] Waiting for ADZDPATCH concurrent program to run
              [ERROR]Failed to execute sql statement :

              select AD_ZD_ADOP.WAIT_FOR_CP_TO_RUN(8358343) from dual

              [ERROR]SQLPLUS error: buffer=

              SQL*Plus: Release 10.1.0.5.0 - Production on Wed Nov 21 19:56:58 2018

              Copyright (c) 1982, 2005, Oracle.  All rights reserved.

              SQL> SQL> Connected.

              SQL> select AD_ZD_ADOP.WAIT_FOR_CP_TO_RUN(8358343) from dual

              *

              ERROR at line 1:

              ORA-20008: No Concurrent Manager is defined that can run concurrent program

              ADZDPATCH

              ORA-06512: at "APPS.AD_ZD_ADOP", line 241

              Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production

              With the Partitioning, OLAP, Data Mining and Real Application Testing options

               

              [UNEXPECTED]Error while runConcurrentProgram sub-routine is called.
              [ERROR]Prepare phase completed with errors or warnings. Please check logfiles.
              Log file: /home/appprod/PROD/fs_ne/EBSapps/log/adop/7/adop_20181121_151044.log

               

               

              [STATEMENT] Please run adopscanlog utility, using the command

               

              "adopscanlog -latest=yes"

               

               

               

              Please help...

               

               

              Kind regards,

              • 4. Re: Where to find the correct adop error?
                mdtaylor

                Hi jc,

                 

                Please navigate to Concurrent->Manager->Administer and see if the Standard Manager is running.  It appears it is not as the prepare phase could not start online patching is progress concurrent request.

                 

                Regards,

                Michael

                • 5. Re: Where to find the correct adop error?
                  Beauty_and_dBest

                  Thanks Michael,

                   

                  I tried to recompile invalid objects, because I got 3,000+.

                  Then rerun ADOP PREPARE again, then I got new error message:

                   

                  Log file located at /home/appprod/PROD/fs2/inst/apps/PROD_appclone03/admin/log/clone/FSCloneStageAppsTier_11222331.log

                  Completed FSCloneStage...

                  Thu Nov 22 23:33:08 2018

                   

                  Beginning application tier FSCloneApply - wlsConfig Thu Nov 22 23:33:10 2018

                   

                  /home/appprod/PROD/fs2/EBSapps/comn/adopclone_appclone03/FMW/t2pjdk/bin/java -Xmx600M -DCONTEXT_VALIDATED=false -Doracle.installer.oui_loc=/oui -classpath /home/appprod/PROD/fs2/EBSapps/comn/adopclone_appclone03/jlib/xmlparserv2.jar:/home/appprod/PROD/fs2/EBSapps/comn/adopclone_appclone03/jlib/ojdbc6.jar:/home/appprod/PROD/fs2/EBSapps/comn/adopclone_appclone03/jlib/java:/home/appprod/PROD/fs2/EBSapps/comn/adopclone_appclone03/jlib/oui/OraInstaller.jar:/home/appprod/PROD/fs2/EBSapps/comn/adopclone_appclone03/jlib/oui/ewt3.jar:/home/appprod/PROD/fs2/EBSapps/comn/adopclone_appclone03/jlib/oui/share.jar:/home/appprod/PROD/fs2/FMW_Home/webtier/../Oracle_EBS-app1/oui/jlib/srvm.jar:/home/appprod/PROD/fs2/EBSapps/comn/adopclone_appclone03/jlib/ojmisc.jar:/home/appprod/PROD/fs2/FMW_Home/wlserver_10.3/server/lib/weblogic.jar:/home/appprod/PROD/fs2/EBSapps/comn/adopclone_appclone03/jlib/obfuscatepassword.jar  oracle.apps.ad.clone.FSCloneApplyAppsTier -e /home/appprod/PROD/fs2/inst/apps/PROD_appclone03/appl/admin/PROD_appclone03.xml -targ /home/appprod/PROD/fs1/inst/apps/PROD_appclone03/appl/admin/PROD_appclone03.xml -stage /home/appprod/PROD/fs2/EBSapps/comn/adopclone_appclone03 -tmp /tmp -nodbac -component wlsConfig -nopromptmsg

                  Log file located at /home/appprod/PROD/fs2/inst/apps/PROD_appclone03/admin/log/clone/FSCloneApplyAppsTier_11222333.log

                   

                  ERROR while running FSCloneApply...

                  Fri Nov 23 01:33:23 2018

                  *******FATAL ERROR*******

                  PROGRAM : (/home/appprod/PROD/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl)

                  TIME    : Fri Nov 23 01:33:24 2018

                  FUNCTION: main::migrateCloneComponentApply [ Level 1 ]

                  ERRORMSG: /home/appprod/PROD/fs2/EBSapps/comn/adopclone_appclone03/bin/adclone.pl did not go through successfully.

                   

                                [UNEXPECTED]Error occurred while executing "perl /home/appprod/PROD/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl -contextfile=/home/appprod/PROD/fs2/inst/apps/PROD_appclone03/appl/admin/PROD_appclone03.xml -patchcontextfile=/home/appprod/PROD/fs1/inst/apps/PROD_appclone03/appl/admin/PROD_appclone03.xml -promptmsg=hide -console=off -mode=migrate -sessionid=7 -timestamp=20181122_224656 -outdir=/home/appprod/PROD/fs_ne/EBSapps/log/adop/7/prepare_20181122_224656/PROD_appclone03"

                                [UNEXPECTED]Error occurred while CONFIG_CLONE Patch File System from Run File System using command: "perl /home/appprod/PROD/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl -contextfile=/home/appprod/PROD/fs2/inst/apps/PROD_appclone03/appl/admin/PROD_appclone03.xml -patchcontextfile=/home/appprod/PROD/fs1/inst/apps/PROD_appclone03/appl/admin/PROD_appclone03.xml -promptmsg=hide -console=off -mode=migrate -sessionid=7 -timestamp=20181122_224656 -outdir=/home/appprod/PROD/fs_ne/EBSapps/log/adop/7/prepare_20181122_224656/PROD_appclone03".

                                [UNEXPECTED]fs_clone phase completed with errors/warnings. Please check logfiles

                                [UNEXPECTED]Error while runPendingConfigClone sub-routine is called

                                  [EVENT]     [START 2018/11/23 01:33:31] Check and Stop Patch FS Admin Server

                   

                  You are running adadminsrvctl.sh version 120.10.12020000.9

                   

                  The AdminServer is already shutdown

                   

                  adadminsrvctl.sh: exiting with status 2

                   

                  adadminsrvctl.sh: check the logfile /home/appprod/PROD/fs1/inst/apps/PROD_appclone03/logs/appl/admin/log/adadminsrvctl.txt for more information ...

                                  [EVENT]     [END   2018/11/23 01:33:40] Check and Stop Patch FS Admin Server

                                [ERROR]     Prepare phase completed with errors or warnings. Please check logfiles.

                                Log file: /home/appprod/PROD/fs_ne/EBSapps/log/adop/7/adop_20181122_224656.log

                   

                  [STATEMENT] Please run adopscanlog utility, using the command

                   

                  "adopscanlog -latest=yes"

                   

                  =================

                  =============================

                  Could not find the pattern...

                  =============================

                   

                  Executing SYSTEM command: perl /home/appprod/PROD/fs2/EBSapps/comn/adopclone_appclone03/bin/adclone.pl java=/home/appprod/PROD/fs2/EBSapps/comn/adopclone_appclone03/FMW/t2pjdk mode=fscloneapply stage=/home/appprod/PROD/fs2/EBSapps/comn/adopclone_appclone03 component=wlsConfig appctx=/home/appprod/PROD/fs2/inst/apps/PROD_appclone03/appl/admin/PROD_appclone03.xml appctxtg=/home/appprod/PROD/fs1/inst/apps/PROD_appclone03/appl/admin/PROD_appclone03.xml

                  EXIT STATUS: 1

                   

                  ======================================

                  Inside copyCloneLogsToFSNE()...

                  ======================================

                   

                  Creating the directory: /home/appprod/PROD/fs_ne/EBSapps/log/adop/7/prepare_20181122_224656/PROD_appclone03/TXK_SYNC_migrate_Thu_Nov_22_23_30_04_2018/wlsConfig_apply

                   

                  Copying the directory

                  ---------------------

                  SOURCE : /home/appprod/PROD/fs2/inst/apps/PROD_appclone03/admin/log/clone

                  TARGET : /home/appprod/PROD/fs_ne/EBSapps/log/adop/7/prepare_20181122_224656/PROD_appclone03/TXK_SYNC_migrate_Thu_Nov_22_23_30_04_2018/wlsConfig_apply

                   

                  /home/appprod/PROD/fs2/EBSapps/comn/adopclone_appclone03/bin/adclone.pl did not go through successfully.

                  LOG DIRECTORY: /home/appprod/PROD/fs_ne/EBSapps/log/adop/7/prepare_20181122_224656/PROD_appclone03/TXK_SYNC_migrate_Thu_Nov_22_23_30_04_2018/wlsConfig_apply.

                  *******FATAL ERROR*******

                  PROGRAM : (/home/appprod/PROD/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl)

                  TIME    : Fri Nov 23 01:33:24 2018

                  FUNCTION: main::migrateCloneComponentApply [ Level 1 ]

                  ERRORMSG: /home/appprod/PROD/fs2/EBSapps/comn/adopclone_appclone03/bin/adclone.pl did not go through successfully.

                  ==========================================

                   

                   

                  Please help which log file do I need to check for hint in what is really the cause?

                   

                   

                  Kind regards,

                  • 6. Re: Where to find the correct adop error?
                    mdtaylor

                    Hi jc,

                     

                    I would start by looking at the first log mentioned:

                     

                    Log file located at /home/appprod/PROD/fs2/inst/apps/PROD_appclone03/admin/log/clone/FSCloneStageAppsTier_11222331.log

                    Completed FSCloneStage...

                    Thu Nov 22 23:33:08 2018

                     

                    Beginning application tier FSCloneApply - wlsConfig Thu Nov 22 23:33:10 2018

                     

                    /home/appprod/PROD/fs2/EBSapps/comn/adopclone_appclone03/FMW/t2pjdk/bin/java -Xmx600M -DCONTEXT_VALIDATED=false -Doracle.installer.oui_loc=/oui -classpath /home/appprod/PROD/fs2/EBSapps/comn/adopclone_appclone03/jlib/xmlparserv2.jar:/home/appprod/PROD/fs2/EBSapps/comn/adopclone_appclone03/jlib/ojdbc6.jar:/home/appprod/PROD/fs2/EBSapps/comn/adopclone_appclone03/jlib/java:/home/appprod/PROD/fs2/EBSapps/comn/adopclone_appclone03/jlib/oui/OraInstaller.jar:/home/appprod/PROD/fs2/EBSapps/comn/adopclone_appclone03/jlib/oui/ewt3.jar:/home/appprod/PROD/fs2/EBSapps/comn/adopclone_appclone03/jlib/oui/share.jar:/home/appprod/PROD/fs2/FMW_Home/webtier/../Oracle_EBS-app1/oui/jlib/srvm.jar:/home/appprod/PROD/fs2/EBSapps/comn/adopclone_appclone03/jlib/ojmisc.jar:/home/appprod/PROD/fs2/FMW_Home/wlserver_10.3/server/lib/weblogic.jar:/home/appprod/PROD/fs2/EBSapps/comn/adopclone_appclone03/jlib/obfuscatepassword.jar  oracle.apps.ad.clone.FSCloneApplyAppsTier -e /home/appprod/PROD/fs2/inst/apps/PROD_appclone03/appl/admin/PROD_appclone03.xml -targ /home/appprod/PROD/fs1/inst/apps/PROD_appclone03/appl/admin/PROD_appclone03.xml -stage /home/appprod/PROD/fs2/EBSapps/comn/adopclone_appclone03 -tmp /tmp -nodbac -component wlsConfig -nopromptmsg

                     

                    Log file located at /home/appprod/PROD/fs2/inst/apps/PROD_appclone03/admin/log/clone/FSCloneApplyAppsTier_11222333.log

                    • 7. Re: Where to find the correct adop error?
                      Beauty_and_dBest

                      Hi Michael,

                       

                      I read some notes that I should run fs_clone first.

                      So I aborted the ADOP PREPARE phase, then run ADOP PHASE=FS_CLONE,

                      but still I got error as well below

                       

                      ERROR while running FSCloneStage...

                      Sat Nov 24 21:08:02 2018

                      *******FATAL ERROR*******

                      PROGRAM : (/home/appprod/PROD/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl)

                      TIME    : Sat Nov 24 21:08:02 2018

                      FUNCTION: main::runFSCloneStage [ Level 1 ]

                      ERRORMSG: /home/appprod/PROD/fs2/EBSapps/appl/ad/12.0.0/bin/adclone.pl did not go through successfully.

                       

                                [UNEXPECTED]Error occurred while executing "perl /home/appprod/PROD/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl -contextfile=/home/appprod/PROD/fs2/inst/apps/PROD_appclone03/appl/admin/PROD_appclone03.xml -patchcontextfile=/home/appprod/PROD/fs1/inst/apps/PROD_appclone03/appl/admin/PROD_appclone03.xml -promptmsg=hide -console=off -mode=create -sessionid=8 -timestamp=20181124_205221 -outdir=/home/appprod/PROD/fs_ne/EBSapps/log/adop/8/fs_clone_20181124_205221/PROD_appclone03"

                                [UNEXPECTED]Error occurred while CLONE Patch File System from Run File System using command: "perl /home/appprod/PROD/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl -contextfile=/home/appprod/PROD/fs2/inst/apps/PROD_appclone03/appl/admin/PROD_appclone03.xml -patchcontextfile=/home/appprod/PROD/fs1/inst/apps/PROD_appclone03/appl/admin/PROD_appclone03.xml -promptmsg=hide -console=off -mode=create -sessionid=8 -timestamp=20181124_205221 -outdir=/home/appprod/PROD/fs_ne/EBSapps/log/adop/8/fs_clone_20181124_205221/PROD_appclone03".

                                [EVENT]     Calling task: "Releasing ports on appclone03"; script: "/home/appprod/PROD/fs2/inst/apps/PROD_appclone03/admin/install/txkCloneAcquirePort.pl"; args: " -option=close -server=appclone03.oracle.amti.com.ph -ports=5577,7022,7622,7422,7222,7822,6121,6221 -log=/home/appprod/PROD/fs_ne/EBSapps/log/adop/8/fs_clone_20181124_205221/PROD_appclone03/txkCloneAcquirePort.log"

                                [EVENT]     [START 2018/11/24 21:08:02] Releasing ports on appclone03

                                  [EVENT]     Calling: /home/appprod/PROD/fs2/inst/apps/PROD_appclone03/admin/install/txkCloneAcquirePort.pl

                                [EVENT]     [END   2018/11/24 21:08:03] Releasing ports on appclone03

                                  [EVENT]     [START 2018/11/24 21:08:04] Check and Stop Patch FS Admin Server

                       

                      You are running adadminsrvctl.sh version 120.10.12020000.9

                       

                      The AdminServer is already shutdown

                       

                      adadminsrvctl.sh: exiting with status 2

                       

                      adadminsrvctl.sh: check the logfile /home/appprod/PROD/fs1/inst/apps/PROD_appclone03/logs/appl/admin/log/adadminsrvctl.txt for more information ...

                       

                                  [EVENT]     [END   2018/11/24 21:08:16] Check and Stop Patch FS Admin Server

                                [UNEXPECTED]fs_clone phase completed with errors/warnings. Please check logfiles

                                Log file: /home/appprod/PROD/fs_ne/EBSapps/log/adop/8/adop_20181124_205221.log

                       

                      [STATEMENT] Please run adopscanlog utility, using the command

                       

                      "adopscanlog -latest=yes"

                       

                      to get the list of the log files along with snippet of the error message corresponding to each log file.

                       

                      adop exiting with status = 2 (Fail)

                      [appprod@appclone03 ~]$

                      =========================

                      ==========================

                      Inside getCloneStatus()...

                      ==========================

                       

                      SQL Command: SELECT clone_status||',' FROM ad_adop_session_patches WHERE status in ('R','F','N') AND bug_number = 'CLONE' AND node_name = 'appclone03' AND adop_session_id = 8

                      clone_status = STARTED

                      clone_status_from_caller = 2

                      clone_status_from_db = 1

                      Executing SYSTEM command: perl /home/appprod/PROD/fs2/EBSapps/appl/ad/12.0.0/bin/adclone.pl java=/home/appprod/PROD/fs2/EBSapps/comn/util/jdk32 mode=fsclonestage stage=/home/appprod/PROD/fs2/EBSapps/comn/adopclone_appclone03 component=appsTier appctx=/home/appprod/PROD/fs2/inst/apps/PROD_appclone03/appl/admin/PROD_appclone03.xml appctxtg=/home/appprod/PROD/fs1/inst/apps/PROD_appclone03/appl/admin/PROD_appclone03.xml

                      EXIT STATUS: 1

                       

                      ======================================

                      Inside copyCloneLogsToFSNE()...

                      ======================================

                       

                      Creating the directory: /home/appprod/PROD/fs_ne/EBSapps/log/adop/8/fs_clone_20181124_205221/PROD_appclone03/TXK_SYNC_create_Sat_Nov_24_20_55_22_2018/fsclone_stage

                       

                      Copying the directory

                      ---------------------

                      SOURCE : /home/appprod/PROD/fs2/inst/apps/PROD_appclone03/admin/log/clone

                      TARGET : /home/appprod/PROD/fs_ne/EBSapps/log/adop/8/fs_clone_20181124_205221/PROD_appclone03/TXK_SYNC_create_Sat_Nov_24_20_55_22_2018/fsclone_stage

                       

                      /home/appprod/PROD/fs2/EBSapps/appl/ad/12.0.0/bin/adclone.pl did not go through successfully.

                      LOG DIRECTORY: /home/appprod/PROD/fs_ne/EBSapps/log/adop/8/fs_clone_20181124_205221/PROD_appclone03/TXK_SYNC_create_Sat_Nov_24_20_55_22_2018/fsclone_stage.

                      *******FATAL ERROR*******

                      PROGRAM : (/home/appprod/PROD/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl)

                      TIME    : Sat Nov 24 21:08:02 2018

                      FUNCTION: main::runFSCloneStage [ Level 1 ]

                      ERRORMSG: /home/appprod/PROD/fs2/EBSapps/appl/ad/12.0.0/bin/adclone.pl did not go through successfully.

                      ==============

                      END: Creating FMW archive.

                       

                      END: Creating FMW Stage Area

                       

                      START: Staging Weblogic Server

                      Evaluated internal cloning stage: /home/appprod/PROD/fs2/EBSapps/comn/clone

                       

                      START: Deploying Shared library ebs-forms

                      Shared library ebs-forms is already deployed.

                       

                      END: Deploying Shared library ebs-forms

                      Obfuscated Password log written to /home/appprod/PROD/fs2/EBSapps/comn/adopclone_appclone03/FMW/obfuscatedPassword_11242107.log

                       

                      START: Creating WLS config archive.

                      Script Executed in 29144 milliseconds, returning status 255

                      ERROR: Script failed, exit code 255

                      ================

                      SEVERE : Nov 24, 2018 9:08:01 PM - CAUSE - CLONE-20606  All managed servers were  down in the provided source environment.  The "copyConfig" operation requires all the managed servers to be up and running.

                      SEVERE : Nov 24, 2018 9:08:01 PM - ACTION - CLONE-20606  Make sure that all managed servers are up and running.

                      oracle.as.t2p.exceptions.FMWT2PCopyConfigException: Error in validating source environment.

                              at oracle.as.clone.cloner.component.j2ee.config.GenericCopyConfigSteps.isFusionAppsDomain(GenericCopyConfigSteps.java:1913)

                              at oracle.as.clone.cloner.component.j2ee.config.GenericCopyConfigSteps.introspectGenericJ2EEComps(GenericCopyConfigSteps.java:216)

                              at oracle.as.clone.cloner.component.J2EEComponentCreateCloner.doClone(J2EEComponentCreateCloner.java:321)

                              at oracle.as.clone.cloner.Cloner.doFinalClone(Cloner.java:61)

                              at oracle.as.clone.request.CreateGenericArchive.doGenericArchive(CreateGenericArchive.java:160)

                              at oracle.as.clone.request.CreateGenericArchive.doGenericArchive(CreateGenericArchive.java:92)

                              at oracle.as.clone.request.CreateCloneRequest._clone(CreateCloneRequest.java:69)

                              at oracle.as.clone.process.CloningExecutionProcess.execute(CloningExecutionProcess.java:131)

                              at oracle.as.clone.process.CloningExecutionProcess.execute(CloningExecutionProcess.java:114)

                              at oracle.as.clone.client.CloningClient.executeT2PCommand(CloningClient.java:238)

                              at oracle.as.clone.client.CloningClient.main(CloningClient.java:115)

                      oracle.as.t2p.exceptions.FMWT2PCopyConfigException: Error in validating source environment.

                              at oracle.as.clone.cloner.component.j2ee.config.GenericCopyConfigSteps.introspectGenericJ2EEComps(GenericCopyConfigSteps.java:375)

                              at oracle.as.clone.cloner.component.J2EEComponentCreateCloner.doClone(J2EEComponentCreateCloner.java:321)

                              at oracle.as.clone.cloner.Cloner.doFinalClone(Cloner.java:61)

                              at oracle.as.clone.request.CreateGenericArchive.doGenericArchive(CreateGenericArchive.java:160)

                              at oracle.as.clone.request.CreateGenericArchive.doGenericArchive(CreateGenericArchive.java:92)

                              at oracle.as.clone.request.CreateCloneRequest._clone(CreateCloneRequest.java:69)

                              at oracle.as.clone.process.CloningExecutionProcess.execute(CloningExecutionProcess.java:131)

                              at oracle.as.clone.process.CloningExecutionProcess.execute(CloningExecutionProcess.java:114)

                              at oracle.as.clone.client.CloningClient.executeT2PCommand(CloningClient.java:238)

                              at oracle.as.clone.client.CloningClient.main(CloningClient.java:115)

                      SEVERE : Nov 24, 2018 9:08:01 PM - ERROR - CLONE-20235   Adding "J2EECOMPONENT@EBS_domain_PROD" to the archive has failed.

                      SEVERE : Nov 24, 2018 9:08:01 PM - CAUSE - CLONE-20235   An internal operation failed.

                      SEVERE : Nov 24, 2018 9:08:01 PM - ACTION - CLONE-20235   Check the clone log and error file for more details.

                      SEVERE : Nov 24, 2018 9:08:01 PM - ERROR - CLONE-20236   Archive creation has failed.

                      SEVERE : Nov 24, 2018 9:08:01 PM - CAUSE - CLONE-20236   An internal operation failed.

                      SEVERE : Nov 24, 2018 9:08:01 PM - ACTION - CLONE-20236   Check the clone log and error file for more details.

                      SEVERE : Nov 24, 2018 9:08:01 PM - ERROR - CLONE-20218   Cloning is not successful.

                      SEVERE : Nov 24, 2018 9:08:01 PM - CAUSE - CLONE-20218   An internal operation failed.

                      SEVERE : Nov 24, 2018 9:08:01 PM - ACTION - CLONE-20218   Provide the clone log and error file for investigation.

                      ==========================

                      FINE : Nov 24, 2018 9:07:59 PM - [GenericCopyConfigSteps:isFusionAppsDomain] oacore_server1 is down

                      FINE : Nov 24, 2018 9:08:00 PM - [GenericCopyConfigSteps:isFusionAppsDomain] forms_server1 is down

                      FINE : Nov 24, 2018 9:08:00 PM - [GenericCopyConfigSteps:isFusionAppsDomain] oafm_server1 is down

                      FINE : Nov 24, 2018 9:08:01 PM - [GenericCopyConfigSteps:isFusionAppsDomain] forms-c4ws_server1 is down

                      FINE : Nov 24, 2018 9:08:01 PM - [GenericCopyConfigSteps:introspectGenericJ2EEComps] Error in validating source environment.

                      INFO : Nov 24, 2018 9:08:01 PM - CLONE-23269  Error in examining J2EE Domain. Check the logs for more details

                      INFO : Nov 24, 2018 9:08:01 PM - CLONE-21096   Process complete in 27723 milli sec.

                      INFO : Nov 24, 2018 9:08:01 PM - CLONE-21031   Time taken to add all the cloners to archive was "27" seconds.

                      INFO : Nov 24, 2018 9:08:01 PM - CLONE-21006   Total time taken by T2P process was 27 seconds.

                       

                       

                      Please help how to debug....

                      Which log above do I check for hint?

                       

                       

                      Kind regards,

                      • 8. Re: Where to find the correct adop error?
                        David Heisler-Oracle

                        Please verify your oraInst.loc between the primary (/etc/oraInst.loc or /var/oracle/oraInst.loc) and the oraInst.loc under each oracle home.  They need to match.

                         

                        What does your current inventory look like now ?

                        • 9. Re: Where to find the correct adop error?
                          Beauty_and_dBest

                          Hi David,

                           

                          Actually I just recycled an old inventory backup which mapped to the existing ORACLE_HOMES.

                          I am not sure if I need to remove all existing oracle homes and recreate everything.

                          But with the recycled backup inventory, I was able to run autoconfig successfully.

                           

                          The current inventory which is referenced by /etc/oraInst.loc  is:

                          inventory_loc=/home/oraInventory

                          inst_group=dba

                           

                          inventory.xml

                          ==================

                          <?xml version="1.0" standalone="yes" ?>

                          <!-- Copyright (c) 2009 Oracle Corporation. All rights Reserved -->

                          <!-- Do not modify the contents of this file by hand. -->

                          <INVENTORY>

                          <COMPOSITEHOME_LIST>

                          </COMPOSITEHOME_LIST>

                          <VERSION_INFO>

                             <SAVED_WITH>10.1.0.6.0</SAVED_WITH>

                             <MINIMUM_VER>2.1.0.6.0</MINIMUM_VER>

                          </VERSION_INFO>

                          <HOME_LIST>

                          <HOME NAME="OH356459478" LOC="/home/appprod/PROD/fs2/FMW_Home/oracle_common" TYPE="O" IDX="1">

                             <REFHOMELIST>

                                <REFHOME LOC="/home/appprod/PROD/fs2/FMW_Home/webtier"/>

                             </REFHOMELIST>

                          </HOME>

                          <HOME NAME="OH224995008" LOC="/home/appprod/PROD/fs2/FMW_Home/webtier" TYPE="O" IDX="2">

                             <DEPHOMELIST>

                                <DEPHOME LOC="/home/appprod/PROD/fs2/FMW_Home/oracle_common"/>

                             </DEPHOMELIST>

                          </HOME>

                          <HOME NAME="OH573413038" LOC="/home/appprod/PROD/fs2/FMW_Home/Oracle_EBS-app1" TYPE="O" IDX="3"/>

                          <HOME NAME="PROD_TOOLS__home_appprod_PROD_fs2_EBSapps_10_1_2" LOC="/home/appprod/PROD/fs2/EBSapps/10.1.2" TYPE="O" IDX="4"/>

                          <HOME NAME="OH423466231" LOC="/home/appprod/PROD/fs1/FMW_Home/oracle_common" TYPE="O" IDX="5">

                             <REFHOMELIST>

                                <REFHOME LOC="/home/appprod/PROD/fs1/FMW_Home/webtier"/>

                             </REFHOMELIST>

                          </HOME>

                          <HOME NAME="OH1222067361" LOC="/home/appprod/PROD/fs1/FMW_Home/webtier" TYPE="O" IDX="6">

                             <DEPHOMELIST>

                                <DEPHOME LOC="/home/appprod/PROD/fs1/FMW_Home/oracle_common"/>

                             </DEPHOMELIST>

                          </HOME>

                          <HOME NAME="OH604432845" LOC="/home/appprod/PROD/fs1/FMW_Home/Oracle_EBS-app1" TYPE="O" IDX="7"/>

                          <HOME NAME="PROD_TOOLS__home_appprod_PROD_fs1_EBSapps_10_1_2" LOC="/home/appprod/PROD/fs1/EBSapps/10.1.2" TYPE="O" IDX="8"/>

                          </HOME_LIST>

                          </INVENTORY>

                          ==============================

                           

                           

                          But the problem is, I can not startup successfully the admin server:

                           

                          [Service Control Execution Report]

                          The report format is:

                            <Service Group>            <Service>                                   <Script>                   <Status>

                           

                            Root Service                                                                                      Enabled

                            Root Service               NodeManager                                 adnodemgrctl.sh            Already Started

                            Web Administration                                                                                Enabled

                            Web Administration         AdminServer                                 adadminsrvctl.sh           Already Started

                            Web Administration         OracleTNSListenerAPPS_PROD_appclone03       adalnctl.sh                Already Started

                            Web Entry Point Services                                                                          Enabled

                            Web Entry Point Services   Oracle HTTP Server PROD_appclone03          adapcctl.sh                Started

                            Web Entry Point Services   Oracle Process Manager for PROD_appclone03  adopmnctl.sh               Started

                            Web Application Services                                                                          Enabled

                            Web Application Services   oacore_server1                              admanagedsrvctl.sh         Failed

                            Web Application Services   forms_server1                               admanagedsrvctl.sh         Failed

                            Web Application Services   oafm_server1                                admanagedsrvctl.sh         Started

                            Web Application Services   forms-c4ws_server1                          admanagedsrvctl.sh         Started

                            Web Application Services   oaea_server1                                admanagedsrvctl.sh         Disabled

                            Batch Processing Services                                                                         Enabled

                            Batch Processing Services  OracleConcMgrPROD_appclone03                adcmctl.sh                 Started

                            Batch Processing Services  Oracle ICSM PROD_appclone03                 ieoicsm.sh                 Disabled

                            Batch Processing Services  Oracle Fulfillment Server PROD_appclone03   jtffmctl.sh                Started

                            Other Services                                                                                    Disabled

                            Other Services             OracleFormsServer-Forms PROD_appclone03     adformsrvctl.sh            Disabled

                            Other Services             Oracle MWA Service PROD_appclone03          mwactlwrpr.sh              Disabled

                           

                          Total time taken for execution is 23 mins 12 seconds.

                           

                          ServiceControl is exiting with status 1

                           

                          11/27/18-16:11:46 :: admanagedsrvctl.sh version 120.14.12020000.9

                           

                          Validated the passed arguments for the option ebs-get-serverstatus

                          AdminServer is currently not running.

                          Checking status of forms_server1 independently

                          Getting connection to t3://appclone03.oracle.com.ph:7421

                          forms_server1 is currently not running.

                           

                          Validated the passed arguments for the option ebs-nmstart-managedsrv

                          ERROR: Unable to obtain connection to the AdminServer.

                          The AdminServer may be down. AdminServer should be started up before starting the managed servers.

                           

                          Managed Server can be started in Independent Mode by executing the control script with option -msimode

                           

                          11/27/18-16:25:09 :: admanagedsrvctl.sh: exiting with status 1

                           

                          ================================================================================

                           

                           

                           

                           

                          Please help....

                           

                          Kind regards,

                          • 10. Re: Where to find the correct adop error?
                            #MVivek

                            cd $FMW_HOME/user_projects/domains/<Domain_NAME>/servers/
                            cd oacore_server1/log
                                                    

                            Same for the other jvm also.

                             

                            share logs.

                            • 11. Re: Where to find the correct adop error?
                              #MVivek

                              Login into weblogic console and check for the servers.
                              There must be pointing correct hostname and port. >> If possible please share the screenshot.

                               

                              Check port pool in the context file to verify with.

                              • 12. Re: Where to find the correct adop error?
                                Beauty_and_dBest

                                Hi Vivek,

                                 

                                I see errors in the logs

                                 

                                 

                                /home/appprod/PROD/fs2/FMW_Home/user_projects/domains/EBS_domain_PROD/servers/forms_server1/logs/forms_server1.out

                                 

                                <11 27, 18 3:06:16 PM PHT> <Notice> <Security> <BEA-090082> <Security initializing using security realm myrealm.>

                                <JTAMigrationHandler> Unexpected exception while getting RemoteMigratableServiceCoordinator : javax.naming.CommunicationException [Root exception is java.net.ConnectException: t3://appclone03.oracle.amti.com.ph:7021: Destination unreachable; nested exception is:

                                        java.net.SocketTimeoutException: Read timed out; No available router to destination]

                                <11 27, 18 3:12:02 PM PHT> <Critical> <WebLogicServer> <BEA-000362> <Server failed. Reason:

                                 

                                There are 1 nested errors:

                                 

                                weblogic.cluster.migration.MigrationException: Could not start JTAMT on local server because it could not be deactivated on the current host.

                                        at weblogic.transaction.internal.TransactionRecoveryService.failbackIfNeeded(TransactionRecoveryService.java:571)

                                        at weblogic.transaction.internal.TransactionRecoveryFailBackService.start(TransactionRecoveryFailBackService.java:23)

                                        at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)

                                        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)

                                        at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)

                                Caused by: weblogic.cluster.migration.MigrationException: Cannot contact the administration server to deactivate JTA.

                                        at weblogic.cluster.migration.JTAMigrationHandler.initializeRemoteMigratableServiceCoordinator(JTAMigrationHandler.java:256)

                                        at weblogic.cluster.migration.JTAMigrationHandler.deactivateJTA(JTAMigrationHandler.java:80)

                                        at weblogic.cluster.migration.JTAMigrationHandler.deactivateJTA(JTAMigrationHandler.java:99)

                                        at weblogic.transaction.internal.TransactionRecoveryService.failbackIfNeeded(TransactionRecoveryService.java:557)

                                        ... 4 more

                                 

                                >

                                <11 27, 18 3:12:04 PM PHT> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FAILED>

                                <11 27, 18 3:12:04 PM PHT> <Error> <WebLogicServer> <BEA-000383> <A critical service failed. The server will shut itself down>

                                <11 27, 18 3:12:04 PM PHT> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FORCE_SHUTTING_DOWN>

                                <11 27, 18 3:12:08 PM> <FINEST> <NodeManager> <Waiting for the process to die: 16532>

                                <11 27, 18 3:12:09 PM> <INFO> <NodeManager> <Server failed during startup so will not be restarted>

                                <11 27, 18 3:12:09 PM> <FINEST> <NodeManager> <runMonitor returned, setting finished=true and notifying waiters>

                                • 13. Re: Where to find the correct adop error?
                                  #MVivek

                                  Hi JC,

                                   

                                  Perform the second check.

                                   

                                  ----------------------------------------------------------------------------------

                                  Login into weblogic console and check for the servers.
                                  There must be pointing correct hostname and port. >> If possible please share the screenshot.

                                   

                                  Check port pool in the context file to verify with.

                                  ----------------------------------------------------------------------------------

                                   

                                  Share the output.

                                   

                                  I am suspecting, host name or port number is missed match.

                                   

                                  Regards

                                  Vivek

                                  • 14. Re: Where to find the correct adop error?
                                    #MVivek

                                    weblogic.cluster.migration.MigrationException: Could not start JTAMT on local server because it could not be deactivated on the current host.

                                            at weblogic.transaction.internal.TransactionRecoveryService.failbackIfNeeded(TransactionRecoveryService.java:571)

                                            at weblogic.transaction.internal.TransactionRecoveryFailBackService.start(TransactionRecoveryFailBackService.java:23)

                                            at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)

                                            at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)

                                            at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)

                                    Caused by: weblogic.cluster.migration.MigrationException: Cannot contact the administration server to deactivate JTA.

                                            at weblogic.cluster.migration.JTAMigrationHandler.initializeRemoteMigratableServiceCoordinator(JTAMigrationHandler.java:256)

                                            at weblogic.cluster.migration.JTAMigrationHandler.deactivateJTA(JTAMigrationHandler.java:80)

                                            at weblogic.cluster.migration.JTAMigrationHandler.deactivateJTA(JTAMigrationHandler.java:99)

                                            at weblogic.transaction.internal.TransactionRecoveryService.failbackIfNeeded(TransactionRecoveryService.java:557)

                                     

                                     

                                    Error is highlighted in bold letters. Normally, this happen because of the error I mentioned above.

                                     

                                    All the best !!

                                    1 2 Previous Next