1 2 Previous Next 21 Replies Latest reply on Dec 6, 2017 11:52 PM by Beauty_and_dBest

    How to restart adop cutover failed?

    Beauty_and_dBest

      EBS R12.2

      12c

      OL6

       

      I am applying patches=25349496,25914844

      I am already in adop phase=cutover   when my adop patch encountered error

      How to I restart it or abort the patch?

      I read docs that once you entered into cutover phase, you can not do abort anymore?

       

      Running stored cutover DDLs.

          Log: /u03/appv1227/V1227/fs_ne/EBSapps/log/adop/61/20171206_051841/cutover/a6dd6a/log/cutover.log

       

      Loading JAR files into database.

          No JAR files found to load

       

      Performing database cutover.

          Log: @ADZDSHOWLOG.sql "2017/12/06 05:26:05"

       

      Reference of statement-handler is not-valid.

       

      DB-Handle Error Code: 28

      DB-Handle Error Message: ORA-00028: your session has been killed (DBD ERROR: OCIStmtExecute/Describe)

      Reference of statement-handler is not valid, attempting database reconnect

      getConnNExec: Getting db-handler

      getConnNExec: reattempting sql statement

      getConnNExec: Returning 0E0

      Statement Handler Error Code: 20001

      Statement Handler Error Message: ORA-20001: Error while calling ad_zd_log.message(ad.bin.adop,WARNING,).ORA-01400: cannot insert NULL into ("APPLSYS"."AD_ZD_LOGS"."MESSAGE_TEXT")

      ORA-06512: at line 6 (DBD ERROR: OCIStmtExecute)

      Reference of statement-handler is valid, not attempting database reconnect.

      Failed to execute SQL statement:

                       declare

                       begin

                          ad_zd_log.message('ad.bin.adop','WARNING','','a6dd6a');

                       exception

                          when others then

                             raise_application_error(-20001,'Error while calling ad_zd_log.message(ad.bin.adop,WARNING,).' || sqlerrm);

                       end;

       

      Error Message:

      ORA-20001: Error while calling ad_zd_log.message(ad.bin.adop,WARNING,).ORA-01400: cannot insert NULL into ("APPLSYS"."AD_ZD_LOGS"."MESSAGE_TEXT")

      ORA-06512: at line 6 (DBD ERROR: OCIStmtExecute)

      Error calling ad_zd_log.message to log messages to the database.

      Please check if database is up

       

      [STATEMENT] Please run adopscanlog utility, using the command

       

      "adopscanlog -latest=yes"

       

      to get the list of the log files along with snippet of the error message corresponding to each log file.

       

       

      adop exiting with status = 1 (Fail)

      =============

       

      When I check the adop status, it is still running?

       

      Capture.PNG

       

      [appv1227@a6dd6a ~]$ adop --status

       

      Enter the APPS password:

      Connected.

      ==============================================================

      ADOP (C.Delta.10)

      Session Id: 61

      Command: status

      Output: /u03/appv1227/V1227/fs_ne/EBSapps/log/adop/61/20171206_143703/adzdshowstatus.out

      ===============================================================

      Node Name       Node Type  Phase           Status          Started              Finished             Elapsed

      --------------- ---------- --------------- --------------- -------------------- -------------------- ------------

      a6dd6a        master       PREPARE         COMPLETED       2017/12/05 21:59:11    2017/12/05 22:29:00  0:29:49

                                 APPLY           COMPLETED       2017/12/05 23:33:05    2017/12/05 23:38:01  0:04:56

                                 FINALIZE        COMPLETED       2017/12/05 23:47:05    2017/12/06 00:38:58  0:51:53

                                 CUTOVER         RUNNING         2017/12/06 05:19:40                         9:17:29

                                 CLEANUP         NOT STARTED

       

       

       

      File System Synchronization Type: None

       

       

      Please help..

       

      Kind regards,

      jc

        • 1. Re: How to restart adop cutover failed?
          mdtaylor

          Hi jc,

           

          i too am confused as to how adop -status could connect to the DB but cutover could not.

           

          Please just execute the adop phase=cutover command again and it should restart where it left off.

           

          Just retrying the command is the first thing I do when adop fails.  During apply phase, you get a message on retry that you after to add restart=yes to the command.

           

          Regards,

          Michael

          • 2. Re: How to restart adop cutover failed?
            Beauty_and_dBest

            Thanks Micheal,

             

            I will try that brb....

            • 3. Re: How to restart adop cutover failed?
              Beauty_and_dBest

              Hi Michael,

               

              I repeat the adop phase=cutover, but it failed again

               

              [appv1227@a6dd6a ~]$ adop phase=cutover

               

              Enter the APPS password:

              Enter the SYSTEM password:

              Enter the WLSADMIN password:

               

              Validating credentials.

               

              Initializing.

                  Run Edition context  : /u03/appv1227/V1227/fs1/inst/apps/V1227_a6dd6a/appl/admin/V1227_a6dd6a.xml

                  Patch edition context: /u03/appv1227/V1227/fs2/inst/apps/V1227_a6dd6a/appl/admin/V1227_a6dd6a.xml

                  Patch file system free space: 136.57 GB

               

              Validating system setup.

                  Node registry is valid.

                  [WARNING]   ETCC: The following required database fixes have not been applied to node a6dd6a:

                                24589081

                                25099339

                                25919622

                              Refer to My Oracle Support Knowledge Document 1594274.1 for instructions.

                  [WARNING]   MTCC has not been run. Apply Patch 17537119 and run checkMTpatch.sh.

               

              Checking for existing adop sessions.

                  Continuing with existing session [Session ID: 61].

                      Session Id            :   61

                      Prepare phase status  :   COMPLETED

                      Apply phase status    :   COMPLETED

                      Cutover  phase status :   RUNNING

                      Abort phase status    :   NOT COMPLETED

                      Session status        :   RUNNING

               

              ===========================================================================

              ADOP (C.Delta.10)

              Session ID: 61

              Node: a6dd6a

              Phase: cutover

              Log: /u03/appv1227/V1227/fs_ne/EBSapps/log/adop/61/20171206_182803/adop.log

              ===========================================================================

               

              Validating configuration on node: [a6dd6a].

                  Log: /u03/appv1227/V1227/fs_ne/EBSapps/log/adop/61/20171206_182803/cutover/validate/a6dd6a

               

              You are running adadminsrvctl.sh version 120.10.12020000.10

               

              Stopping WLS Admin Server...

              Refer /u03/appv1227/V1227/fs1/inst/apps/V1227_a6dd6a/logs/appl/admin/log/adadminsrvctl.txt for details

               

              AdminServer logs are located at /u03/appv1227/V1227/fs1/FMW_Home/user_projects/domains/EBS_domain_V1227/servers/AdminServer/logs

               

              adadminsrvctl.sh: exiting with status 0

               

              adadminsrvctl.sh: check the logfile /u03/appv1227/V1227/fs1/inst/apps/V1227_a6dd6a/logs/appl/admin/log/adadminsrvctl.txt for more information ...

               

               

              You are running adnodemgrctl.sh version 120.11.12020000.12

               

               

              NodeManager log is located at /u03/appv1227/V1227/fs1/FMW_Home/wlserver_10.3/common/nodemanager/nmHome1

               

              adnodemgrctl.sh: exiting with status 0

               

              adnodemgrctl.sh: check the logfile /u03/appv1227/V1227/fs1/inst/apps/V1227_a6dd6a/logs/appl/admin/log/adnodemgrctl.txt for more information ...

               

               

              Performing database cutover.

                  Log: @ADZDSHOWLOG.sql "2017/12/06 18:30:53"

               

              Flipping snapshots in run and patch editions.

               

              Performing file system cutover.

               

              Checking if file system cutover completed.

               

               

              Broadcast message from appv1227@a6dd6a.oraclecloud.com (Wed Dec  6 18:31:59 2017):

               

              V1227 environment has changed.

              All users must re-source the environment.

              Starting run edition Admin Server.

               

              Starting application tier services.

              Executing service control script:

              /u03/appv1227/V1227/fs2/inst/apps/V1227_a6dd6a/admin/scripts/adapcctl.sh start

              Timeout specified in context file: 100 second(s)

               

              script returned:

              ****************************************************

               

              You are running adapcctl.sh version 120.0.12020000.6

               

              Starting OPMN managed Oracle HTTP Server (OHS) instance ...

               

              adapcctl.sh: exiting with status 204

               

              adapcctl.sh: check the logfile /u03/appv1227/V1227/fs2/inst/apps/V1227_a6dd6a/logs/appl/admin/log/adapcctl.txt for more information ...

               

               

              .end std out.

               

              .end err out.

               

              ****************************************************

               

              *******FATAL ERROR*******

              PROGRAM : (/u03/appv1227/V1227/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPCutOverPhaseCtrlScript.pl)

              TIME    : Wed Dec  6 18:54:21 2017

              FUNCTION: main::forceStartupServices [ Level 1 ]

              ERRORMSG: 1

                  [UNEXPECTED]Error occurred running "perl /u03/appv1227/V1227/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPCutOverPhaseCtrlScript.pl -contextfile=/u03/appv1227/V1227/fs1/inst/apps/V1227_a6dd6a/appl/admin/V1227_a6dd6a.xml -patchcontextfile=/u03/appv1227/V1227/fs2/inst/apps/V1227_a6dd6a/appl/admin/V1227_a6dd6a.xml -promptmsg=hide -console=off -sessionid=61 -timestamp=20171206_182803 -outdir=/u03/appv1227/V1227/fs_ne/EBSapps/log/adop/61/20171206_182803/cutover/a6dd6a -action=forcestartup"

                  [UNEXPECTED]Cutover phase has failed.

               

               

              [STATEMENT] Please run adopscanlog utility, using the command

               

              "adopscanlog -latest=yes"

               

              to get the list of the log files along with snippet of the error message corresponding to each log file.

               

               

              adop exiting with status = 1 (Fail)

              You have new mail in /var/spool/mail/appv1227

              [appv1227@a6dd6a ~]$ adop --status

               

              Enter the APPS password:

              Connected.

              ==============================================================

              ADOP (C.Delta.10)

              Session Id: 61

              Command: status

              Output: /u03/appv1227/V1227/fs_ne/EBSapps/log/adop/61/20171206_190512/adzdshowstatus.out

              ===============================================================

              Node Name       Node Type  Phase           Status          Started              Finished             Elapsed

              --------------- ---------- --------------- --------------- -------------------- -------------------- ------------

              a6dd6a          master     PREPARE         COMPLETED       2017/12/05 21:59:11  2017/12/05 22:29:00  0:29:49

                                         APPLY           COMPLETED       2017/12/05 23:33:05  2017/12/05 23:38:01  0:04:56

                                         FINALIZE        COMPLETED       2017/12/05 23:47:05  2017/12/06 00:38:58  0:51:53

                                         CUTOVER         FAILED          2017/12/06 05:19:40  2017/12/06 18:54:21  13:34:41

                                         CLEANUP         NOT STARTED

               

              File System Synchronization Type: None

               

               

               

               

              Kind regards,

              • 4. Re: How to restart adop cutover failed?
                Magnanimous

                Please upload the full cutover log before abort the session.

                • 5. Re: How to restart adop cutover failed?
                  Beauty_and_dBest

                  Hi Magna,

                   

                  I already closed the patch window

                  Where can I get the log location?

                   

                   

                  Kind regards,

                  • 6. Re: How to restart adop cutover failed?
                    Magnanimous

                    You can find all the logs located -

                    e.g. /u01/PROD/fs_ne/EBSapps/log/adop/9/apply_20121011_024437

                    • 7. Re: How to restart adop cutover failed?
                      Beauty_and_dBest

                      Hi Magna,

                       

                      How do I upload the log? Do I need to cut and paste? Its quite long

                       

                      I just copy the last page:

                      [EVENT] Performing file system cutover.
                      [PROCEDURE] [START 2017/12/06 18:30:57] Performing File System Cutover
                      [PROCEDURE] [END   2017/12/06 18:31:58] Performing File System Cutover

                      [STATEMENT] Sourcing Patch edition environment /u03/appv1227/V1227/fs2/EBSapps/appl/APPSV1227_a6dd6a.env

                       

                      [EVENT] Checking if file system cutover completed.
                      [PROCEDURE] [START 2017/12/06 18:31:58] Checking if File System Cutover completed on all nodes
                      [PROCEDURE] [START 2017/12/06 18:31:58] Identifying non-abandoned nodes
                      [PROCEDURE] [START 2017/12/06 18:31:58] Checking if a6dd6a was abandoned
                      [PROCEDURE] [END   2017/12/06 18:31:58] Checking if a6dd6a was abandoned
                      [PROCEDURE] [END   2017/12/06 18:31:58] Identifying non-abandoned nodes
                      [STATEMENT] Count : "0"
                      [PROCEDURE] [END   2017/12/06 18:31:59] Checking if File System Cutover completed on all nodes

                      [PROCEDURE] [START 2017/12/06 18:31:59] Compiling invalid objects

                      [PROCEDURE] [END   2017/12/06 18:31:59] Compiling invalid objects

                       

                      [EVENT] Starting run edition Admin Server.
                      [PROCEDURE] [START 2017/12/06 18:31:59] Start patch Admin Server
                      [PROCEDURE] [START 2017/12/06 18:31:59] Start Admin Server
                      [PROCEDURE] [END   2017/12/06 18:35:00] Start Admin Server
                      [PROCEDURE] [END   2017/12/06 18:35:00] Start patch Admin Server

                      [PROCEDURE] [START 2017/12/06 18:35:00] Updating session status

                      [STATEMENT] Updating cutover_status=5 for session ID : 61

                      [PROCEDURE] [END   2017/12/06 18:35:00] Updating session status

                      [STATEMENT] Sourcing Patch edition environment /u03/appv1227/V1227/fs2/EBSapps/appl/APPSV1227_a6dd6a.env

                       

                      [EVENT] Starting application tier services.
                      [PROCEDURE] [START 2017/12/06 18:35:00] Starting Middle Tier Services
                      [UNEXPECTED]Error occurred running "perl /u03/appv1227/V1227/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPCutOverPhaseCtrlScript.pl -contextfile=/u03/appv1227/V1227/fs1/inst/apps/V1227_a6dd6a/appl/admin/V1227_a6dd6a.xml -patchcontextfile=/u03/appv1227/V1227/fs2/inst/apps/V1227_a6dd6a/appl/admin/V1227_a6dd6a.xml -promptmsg=hide -console=off -sessionid=61 -timestamp=20171206_182803 -outdir=/u03/appv1227/V1227/fs_ne/EBSapps/log/adop/61/20171206_182803/cutover/a6dd6a -action=forcestartup"
                      [UNEXPECTED]Cutover phase has failed.
                      [PROCEDURE] [START 2017/12/06 18:54:21] Updating session timestamps
                      [STATEMENT] CUTOVER Phase END TIME: 06-12-2017 18:54:21
                      [PROCEDURE] [END   2017/12/06 18:54:21] Updating session timestamps
                      [PROCEDURE] [START 2017/12/06 18:54:22] Unlocking sessions table
                      [STATEMENT] Unlocking ad_adop_sessions table for a6dd6a with wait interval of 60 seconds and number of tries 2

                       

                      [PROCEDURE] [END   2017/12/06 18:54:22] Unlocking sessions table
                      [PROCEDURE] [START 2017/12/06 18:54:22] Updating adop session status
                      [STATEMENT] SQL statement : "  update ad_adop_sessions

                        set status='F'

                         where adop_session_id = 61 and appltop_id = 1047 and node_name='a6dd6a'

                      "

                      [PROCEDURE] [END   2017/12/06 18:54:22] Updating adop session status

                       

                       

                       

                       

                      Kind regards,

                      • 8. Re: How to restart adop cutover failed?
                        Magnanimous

                        Please check all cutover logs and see the exact error and paste it here.

                        • 9. Re: How to restart adop cutover failed?
                          Beauty_and_dBest

                          Hi Magna,

                           

                          Do you mean the entire file? What error keyword do I look for?

                           

                          From the above logs, I see error:

                          [UNEXPECTED]Error occurred running "perl /u03/appv1227/V1227/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPCutOverPhaseCtrlScript.pl -contextfile=/u03/appv1227/V1227/fs1/inst/apps/V1227_a6dd6a/appl/admin/V1227_a6dd6a.xml -patchcontextfile=/u03/appv1227/V1227/fs2/inst/apps/V1227_a6dd6a/appl/admin/V1227_a6dd6a.xml -promptmsg=hide -console=off -sessionid=61 -timestamp=20171206_182803 -outdir=/u03/appv1227/V1227/fs_ne/EBSapps/log/adop/61/20171206_182803/cutover/a6dd6a -action=forcestartup"
                          [UNEXPECTED]Cutover phase has failed.

                           

                           

                          Kind regards,

                          • 10. Re: How to restart adop cutover failed?
                            Magnanimous

                            Please look for error as a keyword and paste it here.

                            • 11. Re: How to restart adop cutover failed?
                              Beauty_and_dBest

                              I also see :

                               

                              ================================

                              Inside forceStartupServices()...

                              ================================

                               

                              Executing SYSTEM command: /u03/appv1227/V1227/fs2/inst/apps/V1227_a6dd6a/admin/scripts/adstrtal.sh -nopromptmsg -console=off

                              EXIT STATUS: 1

                              Copying the file

                              ----------------

                              SOURCE : /u03/appv1227/V1227/fs2/inst/apps/V1227_a6dd6a/logs/appl/admin/log/adstrtal.log

                              TARGET : /u03/appv1227/V1227/fs_ne/EBSapps/log/adop/61/20171206_182803/cutover/a6dd6a/TXK_CTRL_forcestartup/adstrtal.log

                               

                              /u03/appv1227/V1227/fs2/inst/apps/V1227_a6dd6a/admin/scripts/adstrtal.sh did not go through successfully.

                              LOG FILE: /u03/appv1227/V1227/fs_ne/EBSapps/log/adop/61/20171206_182803/cutover/a6dd6a/TXK_CTRL_forcestartup/adstrtal.log.

                              *******FATAL ERROR*******

                              PROGRAM : (/u03/appv1227/V1227/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPCutOverPhaseCtrlScript.pl)

                              TIME    : Wed Dec  6 18:54:21 2017

                              FUNCTION: main::forceStartupServices [ Level 1 ]

                              ERRORMSG: 1

                               

                               

                               

                              Kind regards,

                              • 12. Re: How to restart adop cutover failed?
                                Magnanimous

                                Is the patch applied successfully ?

                                • 13. Re: How to restart adop cutover failed?
                                  Beauty_and_dBest

                                  Yes, but the cutover failed

                                   

                                  All the following adop are successful:

                                   

                                  $ adop phase=prepare

                                  $ adop phase=apply patches=24761003 patchtop=/home/PATCH

                                  $ adop phase=finalize

                                  • 14. Re: How to restart adop cutover failed?
                                    Magnanimous

                                    Ok Do you see any other error, except adstartal.sh ? Please check /u03/appv1227/V1227/fs_ne/EBSapps/log/adop/61/20171206_182803/cutover/a6dd6a/TXK_CTRL_forcestartup/adstrtal.log

                                    1 2 Previous Next