1 2 Previous Next 15 Replies Latest reply on Oct 24, 2016 10:11 AM by Bashar.

    Do we need cold backups -before patching using online patching (using ADOP)

    3265587

      I have installed oracle application 12.2.5  with Database 12.1.2 On oracle linux 7.2 , I have used startCD51

       

      My installation was successful and it was confirmed by oracle support.

       

      Then i have move to the post installation Task on 1320300.1. And i am on following task.

       

      2.3 Apply the latest AD and TXK Release Update Pack for Release 12.2

       

      While applying the patch  using following command

      adop phase=apply patches=21841299 hotpatch=yes

       

      Unfortunately patch was hang fore long time, this is my first 12.2 installation. So i have check the worker log files.

      then i saw two workers are having following error

       

      Exception in thread "main" java.sql.SQLSyntaxErrorException: ORA-01031: insufficient privileges

      ORA-06512: at "SYS.DBMS_SESSION", line 122

      ORA-06512: at "APPS.AD_ZD_CTX", line 13

      ORA-06512: at line 1

       

      at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:462)

      at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:405)

      at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:931)

      at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:481)

      at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:205)

      at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:548)

      at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:217)

      at oracle.jdbc.driver.T4CPreparedStatement.executeForRows(T4CPreparedStatement.java:1115)

      at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1488)

      at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3769)

      at oracle.jdbc.driver.OraclePreparedStatement.execute(OraclePreparedStatement.java:3954)

      at oracle.jdbc.driver.OraclePreparedStatementWrapper.execute(OraclePreparedStatementWrapper.java:1539)

      at oracle.apps.ad.worker.AdJavaWorker.getAppsConnection(AdJavaWorker.java:1063)

      at oracle.apps.ad.worker.AdJavaWorker.main(AdJavaWorker.java:278)

       

       

      Since patch was  hanging for long time i have used adctrl to exit the workers

       

      then i have raise SR with support.

       

      we try to restart the patching process and it's not working

       

      After Analyzing the   issue  Support eng has explained  that there's data corruption  and  need to restore the system through backups .

       

      Since i am new to 12,2 , i thought  we should not need to take backups before online patching ??

       

      Does Any one face this kind of issue

       

      If we need backup before online patching  !!!  what is the concept of online patching all about

       

      Still support is checking the possibility of sorting the issue with out Any backups.

        • 1. Re: Do we need cold backups -before patching using online patching (using ADOP)
          3265587

          Hello Dear,

           

          Why did you manually interrupt the ADOP utility using adctrl , followed which note? It is online patching , not adpatch.

           

          Now the error/issue is clear there is a data corruption in the adop tables, the newly copied files are having some references to the DB object which is not yet created by last failed HOT patch, it will always failed with same error

           

          As a support engineer and base on my research I would like to say it's just common sense to keep backups of your system and there is no supported way to fix the issue:

           

          This has been discussed in Bug 21802095 - qrep1225.12:.ader:adop does not work as expected on rerun of ad delta on hotpatch.

           

          This is what  Support people tells about the issue

           

          Eranga

          • 2. Re: Do we need cold backups -before patching using online patching (using ADOP)
            Pravin Takpire

            It is always better to take backup before any big patching.

            regards

            Pravin

            • 3. Re: Do we need cold backups -before patching using online patching (using ADOP)
              Narsi M-Oracle

              Hello,

               

              Thanks for the post.

              There are multiple scenarios here:

               

              a) When applying a patch using hotpatch mode.

              This shouldn't be done until told otherwise by Oracle Support.

              It is always a good idea to take a clean backup before you use hotpatch as you are directly applying patch on Run FS.

               

              b) If applying a patch in hotpatch when not advised, then face any irrecoverable issues, backup is the only resort.

              Refer below:

              12.2 - How To Use Online Patching Related To "Hotpatch" Mode, And Is It Possible and/or Supported To Apply Or Abort A Patch Applied To The Run Filesystem Via "Hotpatch" (Instead Of Using ADOP) (Doc ID 2054074.1)

               

              c) Taking a backup prior to running cutover. If in any event cutover fails and taking too much time to resolve the issue. You could revert the backup.

              Refer below:

              Oracle E-Business Suite Release 12.2: Backup and Recovery Guidelines For Online Patching (adop) Cutover (Doc ID 1584097.1)

               

               

              Hope this helps!!!

               

               

              Regards,

              Narsi

              • 4. Re: Do we need cold backups -before patching using online patching (using ADOP)
                Kanda-Oracle

                Cutover fails, and you need to go back to the state of the system , then you require backup (DB+Filesystem). Please read this for more detail.

                 

                Assume uou are running an Online Patching cycle

                 

                $ adop phase=prepare

                ...

                $ adop phase=apply patches=11111111,22222222

                ...

                $ adop phase=finalize

                ...

                $ adop phase=cutover

                 

                 

                Now you are running into issues. i.e. Cutover fails, and you need to go back to the state of the system before you ran the cutover phase.

                 

                Note: If you had not run the cutover phase, you would have been able to roll back the patch application process by running the adop abort phase. However, this is not possible once cutover has been run.

                 

                There are two main parts to the restore procedure:

                 

                1. You will at least need to restore the database using the Flashback feature (described in Section 4)
                2. Depending on when cutover failed, you may also need to restore the application tier file systems (described in Section 5).

                 

                 

                Hope this helps!

                 

                Thanks

                 

                N Kandasamy

                 

                 

                 

                Oracle E-Business Suite Release 12.2: Backup and Recovery Guidelines For Online Patching (adop) Cutover (Doc ID 1584097.1)

                • 5. Re: Do we need cold backups -before patching using online patching (using ADOP)
                  3265587

                  This is what oracle support explains about backup

                   

                  BACKUPS -

                   

                  NO - You do NOT have to perform a backup prior to having a adop cycle. This is why the abort phase is there. If you patch session fails, you execute the abort and then a cleanup and a fs_clone and start over with a new adop cycle.

                   

                  However, this does not mean NEVER backup your server. You should backup your server at regular intervals using whatever methods you used in the past. It just means that backups before patching are not necessary.

                   

                  While examining your logfiles I noticed this error in the fsclone log (error below) You need more space on your operating system for you applications to work with adop and fsclone phase.

                  ***************************************************************************************************************************

                  So it's clear according to oracle support, any patching error is recoverable with out any backups.

                   

                  Oracle application 12.2 Online patching concept  is all about patching online with out taking backups. That's how oracle is market the product.

                   

                  But i have a case where i restore the system while applying a online patch using hotpatch mode 

                   

                   

                  • 6. Re: Do we need cold backups -before patching using online patching (using ADOP)
                    Kanda-Oracle

                    Kindly refer  (Doc ID 1584097.1) for the clarity

                     

                    NOTE If you had not run the "cutover" phase, you would have been able to roll back the patch application process by running the adop abort phase. However, this is not possible once cutover has been run.

                     

                    My understanding is Backup required ; if Cutover fails, and you need to go back to the state of the system before you ran the cutover phase.

                     

                    Thanks

                     

                    N Kandasmay

                    • 7. Re: Do we need cold backups -before patching using online patching (using ADOP)
                      Narsi M-Oracle

                      Hello,

                       

                      I second Kanda in this case.

                       

                      NO - You do NOT have to perform a backup prior to having a adop cycle. This is why the abort phase is there. If you patch session fails, you execute the abort and then a cleanup and a fs_clone and start over with a new adop cycle.

                       

                      The above is true if & only if "adop phase=cutover" is not yet run.

                       

                       

                      Regards,

                      Narsi

                      • 8. Re: Do we need cold backups -before patching using online patching (using ADOP)
                        3265587

                        According to the Oracle they are saying no backup is needed.

                         

                        Means if cut over is fail due to error, oracle can recover the process.

                         

                        But my problem is not that cut over.

                         

                        Before  going to my problem I will explain what I have done

                         

                        I have install oracle application 12.2.5 with the database of 12.1.2.0 on oracle linux 7.2

                         

                         

                        My installation was successful and it’s confirm by SR - SR 3-13429885971
                        I have downloaded the software’s according to note Doc ID 1596433.1 - SR 3-12973208041
                        After this installation I was thinking that version is application version is 12.2.5 . since I have downloaded 12.2.5 binaries ( Doc ID 1596433.1 )
                        After the installation immediately I have check two things
                        1.log in to the application and check forms are working
                        2.EM database express working for access the database

                         

                        I found that forms are working only from server and EM Database express is not working
                        So I have logged two SR’s
                        1.log in to the application and check forms are working  SR 3-13456472361 with Tony
                        2.EM database express working for access the database -> SR 3-13446890711
                        Mean While I was doing the post installation steps on 1320300.1 (applying ETCC patches)

                         

                        Meanwhile through SR 3-13456472361 ask me to apply patch
                        Patch 20007138:R12.TXK.C IE11: OJSPCOMPILE SHOULD REFER TO EBSUIX.JAR IN THE CLASSPATH --> Not applied
                        Patch 20518047:R12.FWK.C IE11: ADDITIONAL CERTIFICATION FIXES -- > Not Applied
                        Patch 19863340:R12.FND.C BROWSERDETECTOR RETURNS WRONG INFORMATION FOR IE11 STANDARD MODE -> Not applied
                        Then while start applying the patch 20007138 , it’s fails , new SR has been created for this issue : SR 3-13467671871

                         

                        Meanwhile I have finished applying ETCC patches 1320300.1. then I got to know I have to upgrade my EBS installation to 12.2.5 or 12.2.6 from SR 3-13467671871 & 3-13456472361
                        Then I was proceeding the other steps on 1320300.1.
                        2.2 Apply Consolidated Seed Table Upgrade Patch 17204589:12.2.0 -- > Done- SR 3-13490190261
                        2.3 Apply the latest AD and TXK Release Update Pack for Release 12.2  this is the point where I am having the issue.--> SR 3-13495670491

                         


                        SO I have got two conclusions
                        1. My patch fails because I have exit the patching process using adctrl, because of that there is a data corruption So I have to restore the system from backup or redo the installation agagin
                        Similar cases are
                        SR 3-13245947751 : ADOP_VALID_NODES table doesn't exist
                        SR 3-13185769571 : adop_valid_nodes -O RA-00942: table or view does not exist
                        SR 3-12822195681 : Can not restart the patch saying table ADOP_VALID_NODES
                        SR 3-12655726741 Can not restart the patch saying table ADOP_VALID_NODES
                        SR 3-12572647841 Issues when applying AD and TXK patches of Delta 7 as apart of application upgrade 12.1.3 to 12.2.4
                        SR 3-12332921161 Errors applying patch 20745242
                        SR 3-11910206681 ORA-01400: cannot insert NULL into ("APPLSYS"."ADOP_VALID_NODES"."ZD_EDITION_NAME") ORA-06512: at li
                        2. Patch was falling because Have run the adop phase=prepare where it should not run until the system is upgrade to 12.2.5 or 12.2.6 where I was completely blind was thinking I have installed 12.2.5 which is not the case it’s 12.2.0

                         

                         

                        • 9. Re: Do we need cold backups -before patching using online patching (using ADOP)
                          3265587

                          Ok let assume in cutover we need backups

                          In my case even before the cutover, I had restore the system with backups

                           

                          I never backup my system before patching thinking that online patching does not need backups to recovery.

                          While I am applying the patch  21841299 through following command

                          adop phase=apply patches=21841299 hotpatch=yes Failed

                          It was hang for long time, then I have check the worker log files,

                          I have seen there’s a error

                          Exception in thread "main" java.sql.SQLSyntaxErrorException: ORA-01031: insufficient privileges
                          ORA-06512: at "SYS.DBMS_SESSION", line 122
                          ORA-06512: at "APPS.AD_ZD_CTX", line 13
                          ORA-06512: at line 1

                          Then I have check the worker status through  adctrl. It’s shows as running

                          Then I exit the workers through  adctrl  (I am new to online patching on 12.2 and it’s my mistake )

                          Then I raise a SR

                          We never able to, restart the patch again.  So according to support I have restore the system.

                           

                          I have start to apply the patch again

                          Only different I have done

                          In the first run I have run adgrants.sql to APPLSYS echema  (this is mention in readme of the patch)

                          After the restoration I have run  adgrants.sql to APPS sechema   (this is mention in readme of the patch)

                          Second time (After the restoration) patch was applied successfully

                           

                          I am assuming

                          If we exit the patch using adctrl,

                          Then we need backups to recover the situation.

                          • 10. Re: Do we need cold backups -before patching using online patching (using ADOP)
                            Narsi M-Oracle

                            Hello,

                             

                            In your 1st iteration, the patch might have failed since you are running adgrants.sql to "APPLSYS" schema.

                            As per the Patch Readme, you need to run it adgrants.sql to "APPS" schema.

                             

                            It is not mandatory to restore from backup when hotpatch fails.

                             

                            For example, you are applying patch using hotpatch mode as advised by some MOS Notes and the patch fails with some space issue.

                            In above case, you can fix the space issue and restart the patch using below:

                            adop phase=apply patches=xxxxxx hotpatch=yes abandon=yes restart=no

                             

                            Hope this helps!!!

                             

                             

                            Regards,

                            Narsi

                            • 11. Re: Do we need cold backups -before patching using online patching (using ADOP)
                              Narsi M-Oracle

                              Hello,

                               

                              As per what I understand, you may have done too much at this point to recover this instance and it may be a quicker resolution if you start your installation attempt over and chalk this up to experience.

                               

                              So, to keep everything simple, please follow below: (Same Action Provided in MOS Notes "Oracle E-Business Suite Release 12.2.5 Readme (Doc ID 1983050.1)")

                               

                              a) Complete the 12.2.0 i.e., base installation.

                               

                              b) Run the ETCC Reports for DB & Middle Tier.

                               

                              c) Stop all the services -- DB + Application and apply all the ETCC Patches for both DB Tier & Application Tier.

                               

                              d) Apply Consolidated Seed Table Upgrade Patch. (Follow the relevant MOS Notes)

                               

                              e) Apply the AD/TXK Patches. (Follow the relevant MOS Notes)

                               

                              f) Apply the 12.2.5 RUP

                               

                               

                              NOTE:

                              =====

                               

                              a) For a fresh installation of 12.2, until you upgrade to 12.2.5/12.2.6,  all the application patches (using adop) should be applied on the Run FS i.e. adop should be used in downtime mode or hotpatch mode as per the MOS Notes.

                               

                              b) It is not recommended to run adop phase=prepare until you complete the 12.2.5 as per MOS Notes "Oracle E-Business Suite Release 12.2.5 Readme (Doc ID 1983050.1)"

                               

                               

                               

                              Regards,

                              Narsi

                              • 12. Re: Do we need cold backups -before patching using online patching (using ADOP)
                                3265587

                                I have kept every thing very simple in my 12.2 installation

                                 

                                After the patch has failed  we never able to start the patch

                                 

                                But adop phase=apply patches=xxxxxx hotpatch=yes abandon=yes restart=no

                                 

                                Following is what oracle support saying about it

                                 

                                SQL> SQL> Connected.

                                SQL> from fnd_nodes fn, adop_valid_nodes avn

                                *

                                ERROR at line 3:

                                ORA-00942: table or view does not exist

                                 

                                It's a well known problem and the discription of the problem is -

                                .

                                Suppose a patch is having file system and DB actions both, and after

                                performing the file system actions, patch application failed. and suppose the

                                newly copied files are having some references to the DB object which is not

                                yet created by last failed patch.

                                .

                                So, now if you will restart the same patch in hotpatch, it will use the newly

                                copied files and will try to access the DB objects, which don't exist

                                in the DB and it will fail.

                                 

                                It's a well known problem and the discription of the problem is -

                                Suppose a patch is having ADOP related file system and DB actions both, and

                                after performing the file system actions (ADOP related files), patch

                                application failed. and suppose the newly copied perl files are having some

                                references to the DB object which is not yet created by last failed patch.

                                So, now if you will restart the same patch in hotpatch, it will use the newly

                                copied perl files and will try to access the DB objects, which don't exist

                                in the DB and it will fail.

                                 

                                In Parallel to it, we were discussing the solution for this kind of situation

                                internally, and some solution approach came, which can discuss one more time

                                and target this bug for 12.2.8 to implement the finalized solution approach.

                                ========================================================================

                                 

                                The Bug is saying that if you had any failed AD/TXK patching session, then would get this error while re-applying the patch.

                                 

                                From support point of view, the only working and supported solution I have found in our knowledge database regarding similar issues is to restore your environment prior to this patch application, apply the missing technology patches and reapply the patch.

                                 

                                SR 3-13245947751 : ADOP_VALID_NODES table doesn't exist

                                SR 3-13185769571 : adop_valid_nodes -O RA-00942: table or view does not exist

                                SR 3-12822195681 : Can not restart the patch saying table ADOP_VALID_NODES

                                SR 3-12655726741 Can not restart the patch saying table ADOP_VALID_NODES

                                SR 3-12572647841 Issues when applying AD and TXK patches of Delta 7 as apart of application upgrade 12.1.3 to 12.2.4

                                SR 3-12332921161 Errors applying patch 20745242

                                SR 3-11910206681 ORA-01400: cannot insert NULL into ("APPLSYS"."ADOP_VALID_NODES"."ZD_EDITION_NAME") ORA-06512: at li

                                 

                                Could you please perform the upgrade again and reapply the AD/TXK patch , please make sure DO NOT interrupt it manually while ADOP is running.

                                 

                                3. To make your upgrade to Oracle E-Business Suite Release 12.2 as smooth as possible , please check following:

                                 

                                Oracle E-Business Suite Release 12.2: Technical Planning, Getting Started, and Go-Live Checklist ( Doc ID 1585857.1 )

                                 

                                And make sure you have followed every steps outlined in our official note , if HOTPATCH(especially AD/TXK patch ) still hanging there please log an SR to fix it instead of interrupting it manually.

                                 

                                4. If you are not happy with the solution , please let me know then I will create a pre-defect for this issue.

                                 

                                As I said this has been already discussed in Bug 21802095 ,it's faster to perform the upgrade again, than trying to debug a known issue with DEV.

                                • 13. Re: Do we need cold backups -before patching using online patching (using ADOP)
                                  3265587

                                  Most Impotent thing i want to find out is

                                   

                                  Whether we need to backup Our system before using adop for patching

                                   

                                  I am looking forward  to have a starting answer for that

                                   

                                  Oracle Support says

                                  ****************

                                  NO - You do NOT have to perform a backup prior to having a adop cycle. This is why the abort phase is there. If you patch session fails, you execute the abort and then a cleanup and a fs_clone and start over with a new adop cycle.

                                   

                                  However, this does not mean NEVER backup your server. You should backup your server at regular intervals using whatever methods you used in the past. It just means that backups before patching are not necessary.

                                  *****************

                                   

                                  As community experts  what procedure's  are you following ??

                                  • 14. Re: Do we need cold backups -before patching using online patching (using ADOP)
                                    3265587

                                    It is always better to take backup before any big patching.

                                    regards

                                    Pravin

                                     

                                     

                                     

                                    a) When applying a patch using hotpatch mode.

                                    This shouldn't be done until told otherwise by Oracle Support.

                                    It is always a good idea to take a clean backup before you use hotpatch as you are directly applying patch on Run FS.

                                     

                                     

                                     

                                    b) If applying a patch in hotpatch when not advised, then face any irrecoverable issues, backup is the only resort.

                                    Refer below:

                                    12.2 - How To Use Online Patching Related To "Hotpatch" Mode, And Is It Possible and/or Supported To Apply Or Abort A Patch Applied To The Run Filesystem Via "Hotpatch" (Instead Of Using ADOP) (Doc ID 2054074.1)

                                     

                                     

                                     

                                     

                                    c) Taking a backup prior to running cutover. If in any event cutover fails and taking too much time to resolve the issue. You could revert the backup.

                                    Refer below:

                                    Oracle E-Business Suite Release 12.2: Backup and Recovery Guidelines For Online Patching (adop) Cutover (Doc ID 1584097.1)

                                     

                                     

                                     

                                    So above are the recommendation before patching 

                                    1 2 Previous Next