5 Replies Latest reply on Apr 30, 2019 7:27 PM by Ramaraju

Urgent FS_CLONE Failed R12.2.7

Ramaraju Apr 23, 2019 10:21 AM

EBS R12.2.7 & DB: 12.1.0.2

We do not have appl directory under fs2/EBSapps/

FS_CLONE failed with following error.

[oracle@dev APPS(fs1)-DEV EBSapps]$ adop phase=fs_clone

Enter the APPS password:

Enter the SYSTEM password:

Enter the WLSADMIN password:

Validating credentials.

Initializing.

Run Edition context : /u01/app/oracle/apps/fs1/inst/apps/DEV_dev/appl/admin/DEV_dev.xml

Patch edition context: /u01/app/oracle/apps/fs2/inst/apps/DEV_dev/appl/admin/DEV_dev.xml

Patch file system free space: 379.42 GB

Validating system setup.

Node registry is valid.

[WARNING] SYS.DUAL table has incorrect privileges

Shut down the application tier services and then run the following commands in the order shown

sqlplus apps/<APPS_Password> @ $AD_TOP/patch/115/sql/ADFIXUSER.sql

sqlplus apps/<APPS_Password> @ $AD_TOP/sql/adutlrcmp.sql

Checking for existing adop sessions.

No pending session exists.

Starting new adop session.

===========================================================================

ADOP (C.Delta.10)

Session ID: 90

Node: dev

Phase: fs_clone

Log: /u01/app/oracle/apps/fs_ne/EBSapps/log/adop/90/20190423_050918/adop.log

===========================================================================

Validating configuration on node: [dev].

Log: /u01/app/oracle/apps/fs_ne/EBSapps/log/adop/90/20190423_050918/fs_clone/validate/dev

[WARNING]: There could be issues while validating the ports used for E-Business Suite instance against ports used in /etc/services. Refer the log file for more details.

[WARNING]: Found invalid cross references in OHS config files.

[WARNING]: Found invalid cross references in node manager and domain config files.

Checking for pending cleanup actions.

No pending cleanup actions found.

Blocking managed server ports.

Log: /u01/app/oracle/apps/fs_ne/EBSapps/log/adop/90/20190423_050918/fs_clone/dev/txkCloneAcquirePort.log

Performing CLONE steps.

Log: /u01/app/oracle/apps/fs_ne/EBSapps/log/adop/90/20190423_050918/fs_clone/dev

Beginning application tier FSCloneApply - Tue Apr 23 05:13:53 2019

/u01/app/oracle/apps/fs1/EBSapps/comn/adopclone_dev/FMW/t2pjdk/bin/java -Xmx600M -DCONTEXT_VALIDATED=false -Doracle.installer.oui_loc=/oui -classpath /u01/app/oracle/apps/fs1/EBSapps/comn/adopclone_dev/jlib/xmlparserv2.jar:/u01/app/oracle/apps/fs1/EBSapps/comn/adopclone_dev/jlib/ojdbc6.jar:/u01/app/oracle/apps/fs1/EBSapps/comn/adopclone_dev/jlib/java:/u01/app/oracle/apps/fs1/EBSapps/comn/adopclone_dev/jlib/oui/OraInstaller.jar:/u01/app/oracle/apps/fs1/EBSapps/comn/adopclone_dev/jlib/oui/ewt3.jar:/u01/app/oracle/apps/fs1/EBSapps/comn/adopclone_dev/jlib/oui/share.jar:/u01/app/oracle/apps/fs1/FMW_Home/webtier/../Oracle_EBS-app1/oui/jlib/srvm.jar:/u01/app/oracle/apps/fs1/EBSapps/comn/adopclone_dev/jlib/ojmisc.jar:/u01/app/oracle/apps/fs1/FMW_Home/wlserver_10.3/server/lib/weblogic.jar:/u01/app/oracle/apps/fs1/EBSapps/comn/adopclone_dev/jlib/obfuscatepassword.jar oracle.apps.ad.clone.FSCloneApplyAppsTier -e /u01/app/oracle/apps/fs1/inst/apps/DEV_dev/appl/admin/DEV_dev.xml -targ /u01/app/oracle/apps/fs2/inst/apps/DEV_dev/appl/admin/DEV_dev.xml -stage /u01/app/oracle/apps/fs1/EBSapps/comn/adopclone_dev -tmp /tmp -nodbac -component appsTier -nopromptmsg

Log file located at /u01/app/oracle/apps/fs1/inst/apps/DEV_dev/admin/log/clone/FSCloneApplyAppsTier_04230513.log

Enabling EBS_LOGON trigger

ERROR while running FSCloneApply...

Tue Apr 23 05:14:50 2019

*******FATAL ERROR*******

PROGRAM : (/u01/app/oracle/apps/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl)

TIME : Tue Apr 23 05:14:50 2019

FUNCTION: main::runFSCloneApply [ Level 1 ]

ERRORMSG: /u01/app/oracle/apps/fs1/EBSapps/comn/adopclone_dev/bin/adclone.pl did not go through successfully.

[UNEXPECTED]Error occurred running "perl /u01/app/oracle/apps/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl -contextfile=/u01/app/oracle/apps/fs1/inst/apps/DEV_dev/appl/admin/DEV_dev.xml -patchcontextfile=/u01/app/oracle/apps/fs2/inst/apps/DEV_dev/appl/admin/DEV_dev.xml -promptmsg=hide -console=off -mode=create -sync_mode=copy -sessionid=90 -timestamp=20190423_050918 -outdir=/u01/app/oracle/apps/fs_ne/EBSapps/log/adop/90/20190423_050918/fs_clone/dev"

[UNEXPECTED]occurred during CLONE Patch File System from Run File System, running command: "perl /u01/app/oracle/apps/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl -contextfile=/u01/app/oracle/apps/fs1/inst/apps/DEV_dev/appl/admin/DEV_dev.xml -patchcontextfile=/u01/app/oracle/apps/fs2/inst/apps/DEV_dev/appl/admin/DEV_dev.xml -promptmsg=hide -console=off -mode=create -sync_mode=copy -sessionid=90 -timestamp=20190423_050918 -outdir=/u01/app/oracle/apps/fs_ne/EBSapps/log/adop/90/20190423_050918/fs_clone/dev".

Releasing managed server ports.

Log: /u01/app/oracle/apps/fs_ne/EBSapps/log/adop/90/20190423_050918/fs_clone/dev/txkCloneAcquirePort.log

Stopping services on patch file system.

Stopping admin server.

You are running adadminsrvctl.sh version 120.10.12020000.10

Environment file /u01/app/oracle/apps/fs2/EBSapps/appl/APPSDEV_dev.env is not found.

/u01/app/oracle/apps/fs2/inst/apps/DEV_dev/admin/scripts/adadminsrvctl.sh: line 161: $LOGFILE: ambiguous redirect

[UNEXPECTED]Error occurred running "sh /u01/app/oracle/apps/fs2/inst/apps/DEV_dev/admin/scripts/adadminsrvctl.sh stop -nopromptmsg"

[UNEXPECTED]Error running adadminsrvctl command

[STATEMENT] Please run adopscanlog utility, using the command

"adopscanlog -latest=yes"

to get the list of the log files along with snippet of the error message corresponding to each log file.

adop exiting with status = 2 (Fail)

real 5m33.467s

user 1m35.822s

sys 0m12.703s

[oracle@dev APPS(fs1)-DEV EBSapps]$ adopscanlog -latest=yes

Scanning /u01/app/oracle/apps/fs_ne/EBSapps/log/adop/90/ directory ...

/u01/app/oracle/apps/fs_ne/EBSapps/log/adop/90/20190423_050918/fs_clone/dev/TXK_SYNC_create/txkADOPPreparePhaseSynchronize.log:

-------------------------------------------------------------------------------------------------------------------------------------

Lines #(312-314):

TIME : Tue Apr 23 05:14:50 2019

FUNCTION: main::runFSCloneApply [ Level 1 ]

ERRORMSG: /u01/app/oracle/apps/fs1/EBSapps/comn/adopclone_dev/bin/adclone.pl did not go through successfully.

/u01/app/oracle/apps/fs_ne/EBSapps/log/adop/90/20190423_050918/fs_clone/dev/TXK_SYNC_create/fsclone_apply/FSCloneApplyAppsTier_04230513.log:

--------------------------------------------------------------------------------------------------------------------------------------------------

Lines #(164-168):

ERROR: RC-50410: Fatal: OUI Registration failed with an Abnormal Termination error during call from ouicli.pl. Please check the log file /u01/app/oracle/apps/fs2/inst/apps/DEV_dev/admin/log/ohclone.log for more details.

Raised by oracle.apps.ad.clone.ApplyAppsTechStack

Lines #(167-171):

Raised by oracle.apps.ad.clone.ApplyAppsTechStack

WARNING: RC-50208: Warning: Exception in method deepOUICheck

instantiate file:

Lines #(214-221):

AutoConfig is exiting with status 1

ERROR: RC-50013: Fatal: Instantiate driver did not complete successfully.

/u01/app/oracle/apps/fs2/EBSapps/10.1.2/appsutil/driver/regclone.drv

ERROR: RC-50004: Fatal: Error occurred in ApplyAppsTechStack:

RC-50013: Fatal: Failed to instantiate driver /u01/app/oracle/apps/fs2/EBSapps/10.1.2/appsutil/driver/regclone.drv

START: Inside exitClone....

/u01/app/oracle/apps/fs_ne/EBSapps/log/adop/90/20190423_050918/adop.log:

------------------------------------------------------------------------

Lines #(109-114):

[PROCEDURE] Calling: /u01/app/oracle/apps/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl

[EVENT] Log: /u01/app/oracle/apps/fs_ne/EBSapps/log/adop/90/20190423_050918/fs_clone/dev

[EVENT] Releasing managed server ports.

[PROCEDURE] [START 2019/04/23 05:14:50] Releasing ports

Lines #(130-135):

[PROCEDURE] [START 2019/04/23 05:14:51] Check and stop patch Admin Server

[PROCEDURE] [START 2019/04/23 05:14:52] Check and Stop patch FS Admin Server

[UNEXPECTED]Error occurred running "sh /u01/app/oracle/apps/fs2/inst/apps/DEV_dev/admin/scripts/adadminsrvctl.sh stop -nopromptmsg"

[UNEXPECTED]Error running adadminsrvctl command

[PROCEDURE] [START 2019/04/23 05:14:52] Unlocking sessions table

[STATEMENT] Unlocking ad_adop_sessions table for dev with wait interval of 60 seconds and number of tries 2

Thanks,

Ramaraju

1. Re: Urgent FS_CLONE Failed R12.2.7

John_K Apr 24, 2019 9:59 AM (in response to Ramaraju)

If you have an "Urgent" question then Oracle Support is the correct route.
Like Show 0 Likes(0)

Actions
2. Re: Urgent FS_CLONE Failed R12.2.7

Kirushna Apr 24, 2019 10:00 AM (in response to Ramaraju)

Hi,

Have you reviewed the Doc : Fs_Clone fails with "ERROR: RC-50410: Fatal: OUI Registration failed with an Abnormal Termination error" (Doc ID 2095966.1)

Regards,
Kirushna
Like Show 0 Likes(0)

Actions
3. Re: Urgent FS_CLONE Failed R12.2.7

lmu Apr 24, 2019 3:12 PM (in response to Ramaraju)

Hi Have the same issue. It seems to happen more often than not after a successful patch application. The last time it failed I could see that it could not write to the jdk directory.

When this happens, we have to rebuild the patch file system because the fs_clone seems to remove the entire appl_top, then rebuild it. If it fails after it's removed it. it will not rebuild it. This is just my experience, if anyone knows a way to get it to rebuild it I am all ears.

The last time it failed it had errors about not being able to write to $COMMON_TOP/util/jdk32 and $COMMON_TOP/util/jdk64. If you look at that directory, the permissions on some of the files are -r--r--r--. So of course it can't over write them.

-rw-r--r--. 1 applcrp1 dba   109120 Jan 23 21:21 THIRDPARTYLICENSEREADME-JAVAFX.txt
-r--r--r--. 1 applcrp1 dba   174351 Jan 24 01:26 THIRDPARTYLICENSEREADME.txt
-r--r--r--. 1 applcrp1 dba      114 Jan 24 01:26 README.html
-r--r--r--. 1 applcrp1 dba       40 Jan 24 01:26 LICENSE
-r--r--r--. 1 applcrp1 dba     3339 Jan 24 01:26 COPYRIGHT

The next thing I was going to do it 755 those files after rebuilding the patch fs and see if that keeps it fs_clone from failing. Otherwise, it's back to the drawing board to figure out why it failed.
I can't wait to hear how you resolved this.
I opened an SR. The girl working it has not clue how fs_clone works, so they really are no help. I've had the SR open for months.

I wish this had been the issue, but $ENV is not set.
[applcrp1@orar12app-crp1 jdk32]$ echo $ENV
Like Show 0 Likes(0)

Actions
4. Re: Urgent FS_CLONE Failed R12.2.7

lmu Apr 24, 2019 4:23 PM (in response to lmu)

My fs_clone completed!

I chmod 755 the jdk32 and jdk64 directories and reran the fs_clone with this command so it would pick up where it left off and it completed successfully all in the time since i last updated this post.

adop phase=fs_clone force=no

force=yes makes it start over.
Like Show 0 Likes(0)

Actions
5. Re: Urgent FS_CLONE Failed R12.2.7

Ramaraju Apr 30, 2019 7:27 PM (in response to Ramaraju)

Finally was able to get adop phase=fs_clone to run. It looks like we had memory leak issues that were causing the problem. We finally had to shutdown the run time apps tier processes to get it to complete.

Thanks everyone.
Like Show 0 Likes(0)

Actions

Go to original post