5 Replies Latest reply on Apr 30, 2019 7:27 PM by Ramaraju

    Urgent FS_CLONE Failed R12.2.7

    Ramaraju

      EBS R12.2.7 & DB: 12.1.0.2

      We do not have appl directory under fs2/EBSapps/

      FS_CLONE failed with following error.

       

      [oracle@dev APPS(fs1)-DEV EBSapps]$ adop phase=fs_clone

       

       

      Enter the APPS password:

      Enter the SYSTEM password:

      Enter the WLSADMIN password:

       

       

      Validating credentials.

       

       

      Initializing.

          Run Edition context  : /u01/app/oracle/apps/fs1/inst/apps/DEV_dev/appl/admin/DEV_dev.xml

          Patch edition context: /u01/app/oracle/apps/fs2/inst/apps/DEV_dev/appl/admin/DEV_dev.xml

          Patch file system free space: 379.42 GB

       

       

      Validating system setup.

          Node registry is valid.

          [WARNING]   SYS.DUAL table has incorrect privileges

                      Shut down the application tier services and then run the following commands in the order shown

                         sqlplus apps/<APPS_Password> @ $AD_TOP/patch/115/sql/ADFIXUSER.sql

                         sqlplus apps/<APPS_Password> @ $AD_TOP/sql/adutlrcmp.sql

       

       

      Checking for existing adop sessions.

          No pending session exists.

          Starting new adop session.

       

       

      ===========================================================================

      ADOP (C.Delta.10)

      Session ID: 90

      Node: dev

      Phase: fs_clone

      Log: /u01/app/oracle/apps/fs_ne/EBSapps/log/adop/90/20190423_050918/adop.log

      ===========================================================================

       

       

      Validating configuration on node: [dev].

          Log: /u01/app/oracle/apps/fs_ne/EBSapps/log/adop/90/20190423_050918/fs_clone/validate/dev

              [WARNING]: There could be issues while validating the ports used for E-Business Suite instance against ports used in /etc/services. Refer the log file for more details.

              [WARNING]: Found invalid cross references in OHS config files.

              [WARNING]: Found invalid cross references in node manager and domain config files.

       

       

      Checking for pending cleanup actions.

          No pending cleanup actions found.

       

       

      Blocking managed server ports.

          Log: /u01/app/oracle/apps/fs_ne/EBSapps/log/adop/90/20190423_050918/fs_clone/dev/txkCloneAcquirePort.log

       

       

      Performing CLONE steps.

          Log: /u01/app/oracle/apps/fs_ne/EBSapps/log/adop/90/20190423_050918/fs_clone/dev

       

       

       

       

       

       

      Beginning application tier FSCloneApply - Tue Apr 23 05:13:53 2019

       

       

      /u01/app/oracle/apps/fs1/EBSapps/comn/adopclone_dev/FMW/t2pjdk/bin/java -Xmx600M -DCONTEXT_VALIDATED=false -Doracle.installer.oui_loc=/oui -classpath /u01/app/oracle/apps/fs1/EBSapps/comn/adopclone_dev/jlib/xmlparserv2.jar:/u01/app/oracle/apps/fs1/EBSapps/comn/adopclone_dev/jlib/ojdbc6.jar:/u01/app/oracle/apps/fs1/EBSapps/comn/adopclone_dev/jlib/java:/u01/app/oracle/apps/fs1/EBSapps/comn/adopclone_dev/jlib/oui/OraInstaller.jar:/u01/app/oracle/apps/fs1/EBSapps/comn/adopclone_dev/jlib/oui/ewt3.jar:/u01/app/oracle/apps/fs1/EBSapps/comn/adopclone_dev/jlib/oui/share.jar:/u01/app/oracle/apps/fs1/FMW_Home/webtier/../Oracle_EBS-app1/oui/jlib/srvm.jar:/u01/app/oracle/apps/fs1/EBSapps/comn/adopclone_dev/jlib/ojmisc.jar:/u01/app/oracle/apps/fs1/FMW_Home/wlserver_10.3/server/lib/weblogic.jar:/u01/app/oracle/apps/fs1/EBSapps/comn/adopclone_dev/jlib/obfuscatepassword.jar  oracle.apps.ad.clone.FSCloneApplyAppsTier -e /u01/app/oracle/apps/fs1/inst/apps/DEV_dev/appl/admin/DEV_dev.xml -targ /u01/app/oracle/apps/fs2/inst/apps/DEV_dev/appl/admin/DEV_dev.xml -stage /u01/app/oracle/apps/fs1/EBSapps/comn/adopclone_dev -tmp /tmp -nodbac -component appsTier -nopromptmsg

      Log file located at /u01/app/oracle/apps/fs1/inst/apps/DEV_dev/admin/log/clone/FSCloneApplyAppsTier_04230513.log

       

       

      Enabling EBS_LOGON trigger

       

       

      ERROR while running FSCloneApply...

      Tue Apr 23 05:14:50 2019

      *******FATAL ERROR*******

      PROGRAM : (/u01/app/oracle/apps/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl)

      TIME    : Tue Apr 23 05:14:50 2019

      FUNCTION: main::runFSCloneApply [ Level 1 ]

      ERRORMSG: /u01/app/oracle/apps/fs1/EBSapps/comn/adopclone_dev/bin/adclone.pl did not go through successfully.

       

       

          [UNEXPECTED]Error occurred running "perl /u01/app/oracle/apps/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl -contextfile=/u01/app/oracle/apps/fs1/inst/apps/DEV_dev/appl/admin/DEV_dev.xml -patchcontextfile=/u01/app/oracle/apps/fs2/inst/apps/DEV_dev/appl/admin/DEV_dev.xml -promptmsg=hide -console=off -mode=create -sync_mode=copy -sessionid=90 -timestamp=20190423_050918 -outdir=/u01/app/oracle/apps/fs_ne/EBSapps/log/adop/90/20190423_050918/fs_clone/dev"

          [UNEXPECTED]occurred during CLONE Patch File System from Run File System, running command: "perl /u01/app/oracle/apps/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl -contextfile=/u01/app/oracle/apps/fs1/inst/apps/DEV_dev/appl/admin/DEV_dev.xml -patchcontextfile=/u01/app/oracle/apps/fs2/inst/apps/DEV_dev/appl/admin/DEV_dev.xml -promptmsg=hide -console=off -mode=create -sync_mode=copy -sessionid=90 -timestamp=20190423_050918 -outdir=/u01/app/oracle/apps/fs_ne/EBSapps/log/adop/90/20190423_050918/fs_clone/dev".

          Releasing managed server ports.

          Log: /u01/app/oracle/apps/fs_ne/EBSapps/log/adop/90/20190423_050918/fs_clone/dev/txkCloneAcquirePort.log

          Stopping services on patch file system.

          Stopping admin server.

       

       

      You are running adadminsrvctl.sh version 120.10.12020000.10

       

       

      Environment file /u01/app/oracle/apps/fs2/EBSapps/appl/APPSDEV_dev.env is not found.

      /u01/app/oracle/apps/fs2/inst/apps/DEV_dev/admin/scripts/adadminsrvctl.sh: line 161: $LOGFILE: ambiguous redirect

          [UNEXPECTED]Error occurred running "sh /u01/app/oracle/apps/fs2/inst/apps/DEV_dev/admin/scripts/adadminsrvctl.sh  stop -nopromptmsg"

          [UNEXPECTED]Error running adadminsrvctl command

       

       

       

       

      [STATEMENT] Please run adopscanlog utility, using the command

       

       

      "adopscanlog -latest=yes"

       

       

      to get the list of the log files along with snippet of the error message corresponding to each log file.

       

       

       

       

      adop exiting with status = 2 (Fail)

       

       

      real    5m33.467s

      user    1m35.822s

      sys     0m12.703s

      [oracle@dev APPS(fs1)-DEV EBSapps]$ adopscanlog -latest=yes

       

       

      Scanning /u01/app/oracle/apps/fs_ne/EBSapps/log/adop/90/ directory ...

       

       

      /u01/app/oracle/apps/fs_ne/EBSapps/log/adop/90/20190423_050918/fs_clone/dev/TXK_SYNC_create/txkADOPPreparePhaseSynchronize.log:

      -------------------------------------------------------------------------------------------------------------------------------------

       

       

      Lines #(312-314):

      TIME    : Tue Apr 23 05:14:50 2019

      FUNCTION: main::runFSCloneApply [ Level 1 ]

      ERRORMSG: /u01/app/oracle/apps/fs1/EBSapps/comn/adopclone_dev/bin/adclone.pl did not go through successfully.

       

       

      /u01/app/oracle/apps/fs_ne/EBSapps/log/adop/90/20190423_050918/fs_clone/dev/TXK_SYNC_create/fsclone_apply/FSCloneApplyAppsTier_04230513.log:

      --------------------------------------------------------------------------------------------------------------------------------------------------

       

       

      Lines #(164-168):

       

       

       

       

      ERROR: RC-50410: Fatal: OUI Registration failed with an Abnormal Termination error during call from ouicli.pl. Please check the log file /u01/app/oracle/apps/fs2/inst/apps/DEV_dev/admin/log/ohclone.log for more details.

      Raised by oracle.apps.ad.clone.ApplyAppsTechStack

       

       

       

       

      Lines #(167-171):

      Raised by oracle.apps.ad.clone.ApplyAppsTechStack

       

       

      WARNING: RC-50208: Warning: Exception  in method deepOUICheck

       

       

      instantiate file:

       

       

      Lines #(214-221):

      AutoConfig is exiting with status 1

       

       

      ERROR: RC-50013: Fatal: Instantiate driver did not complete successfully.

      /u01/app/oracle/apps/fs2/EBSapps/10.1.2/appsutil/driver/regclone.drv

      ERROR: RC-50004: Fatal: Error occurred in ApplyAppsTechStack:

      RC-50013: Fatal: Failed to instantiate driver /u01/app/oracle/apps/fs2/EBSapps/10.1.2/appsutil/driver/regclone.drv

       

       

      START: Inside exitClone....

       

       

      /u01/app/oracle/apps/fs_ne/EBSapps/log/adop/90/20190423_050918/adop.log:

      ------------------------------------------------------------------------

       

       

      Lines #(109-114):

          [PROCEDURE] Calling: /u01/app/oracle/apps/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl

          [EVENT]     Log: /u01/app/oracle/apps/fs_ne/EBSapps/log/adop/90/20190423_050918/fs_clone/dev

          [UNEXPECTED]Error occurred running "perl /u01/app/oracle/apps/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl -contextfile=/u01/app/oracle/apps/fs1/inst/apps/DEV_dev/appl/admin/DEV_dev.xml -patchcontextfile=/u01/app/oracle/apps/fs2/inst/apps/DEV_dev/appl/admin/DEV_dev.xml -promptmsg=hide -console=off -mode=create -sync_mode=copy -sessionid=90 -timestamp=20190423_050918 -outdir=/u01/app/oracle/apps/fs_ne/EBSapps/log/adop/90/20190423_050918/fs_clone/dev"

          [UNEXPECTED]occurred during CLONE Patch File System from Run File System, running command: "perl /u01/app/oracle/apps/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl -contextfile=/u01/app/oracle/apps/fs1/inst/apps/DEV_dev/appl/admin/DEV_dev.xml -patchcontextfile=/u01/app/oracle/apps/fs2/inst/apps/DEV_dev/appl/admin/DEV_dev.xml -promptmsg=hide -console=off -mode=create -sync_mode=copy -sessionid=90 -timestamp=20190423_050918 -outdir=/u01/app/oracle/apps/fs_ne/EBSapps/log/adop/90/20190423_050918/fs_clone/dev".

          [EVENT]     Releasing managed server ports.

              [PROCEDURE] [START 2019/04/23 05:14:50] Releasing ports

       

       

      Lines #(130-135):

                  [PROCEDURE] [START 2019/04/23 05:14:51] Check and stop patch Admin Server

                  [PROCEDURE] [START 2019/04/23 05:14:52] Check and Stop patch FS Admin Server

                  [UNEXPECTED]Error occurred running "sh /u01/app/oracle/apps/fs2/inst/apps/DEV_dev/admin/scripts/adadminsrvctl.sh  stop -nopromptmsg"

                  [UNEXPECTED]Error running adadminsrvctl command

              [PROCEDURE] [START 2019/04/23 05:14:52] Unlocking sessions table

              [STATEMENT] Unlocking ad_adop_sessions table for dev with wait interval of 60 seconds and number of tries 2

       

      Thanks,

      Ramaraju

        • 1. Re: Urgent FS_CLONE Failed R12.2.7
          John_K

          If you have an "Urgent" question then Oracle Support is the correct route.

          • 2. Re: Urgent FS_CLONE Failed R12.2.7
            Kirushna

            Hi,

             

            Have you reviewed the Doc : Fs_Clone fails with "ERROR: RC-50410: Fatal: OUI Registration failed with an Abnormal Termination error" (Doc ID 2095966.1)

             

            Regards,

            Kirushna

            • 3. Re: Urgent FS_CLONE Failed R12.2.7
              lmu

              Hi Have the same issue.  It seems to happen more often than not after a successful patch application.  The last time it failed I could see that it could not write to the jdk directory.

               

              When this happens, we have to rebuild the patch file system because the fs_clone seems to remove the entire appl_top, then rebuild it. If it fails after it's removed it. it will not rebuild it.  This is just my experience, if anyone knows a way to get it to rebuild it I am all ears.

               

              The last time it failed it had errors about not being able to write to $COMMON_TOP/util/jdk32 and $COMMON_TOP/util/jdk64. If you look at that directory, the permissions on some of the files are -r--r--r--. So of course it can't over write them.

               

              -rw-r--r--. 1 applcrp1 dba   109120 Jan 23 21:21 THIRDPARTYLICENSEREADME-JAVAFX.txt

              -r--r--r--. 1 applcrp1 dba   174351 Jan 24 01:26 THIRDPARTYLICENSEREADME.txt

              -r--r--r--. 1 applcrp1 dba      114 Jan 24 01:26 README.html

              -r--r--r--. 1 applcrp1 dba       40 Jan 24 01:26 LICENSE

              -r--r--r--. 1 applcrp1 dba     3339 Jan 24 01:26 COPYRIGHT

               

              The next thing I was going to do it 755 those files after rebuilding the patch fs and see if that keeps it fs_clone from failing. Otherwise, it's back to the drawing board to figure out why it failed.

              I can't wait to hear how you resolved this.

              I opened an SR. The girl working it has not clue how fs_clone works, so they really are no help.  I've had the SR open for months.

               

              I wish this had been the issue, but $ENV is not set.

              [applcrp1@orar12app-crp1 jdk32]$ echo $ENV

              • 4. Re: Urgent FS_CLONE Failed R12.2.7
                lmu

                My fs_clone completed!

                 

                I chmod 755 the jdk32 and jdk64 directories and reran the fs_clone with this command so it would pick up where it left off and it completed successfully all in the time since i last updated this post.

                 

                 

                adop phase=fs_clone force=no

                 

                force=yes makes it start over. 

                • 5. Re: Urgent FS_CLONE Failed R12.2.7
                  Ramaraju

                  Finally was able to get adop phase=fs_clone to run.  It looks like we had memory leak issues that were causing the problem.  We finally had to shutdown the run time apps tier processes to get it to complete.

                   

                  Thanks everyone.