11 Replies Latest reply on Nov 29, 2013 2:40 AM by Kabeer Khan

    Upgrade from 12.2.0 to 12.2.2 - Patch 16207672 taking too long

    Kabeer Khan

      Hi

      Issue = Patch 16207672 taking too long and its already close to 15 hours that adop is running. No errors were thrown so far. I followed instructions in MOS Note 1506669.1.

       

      Backgrouud

      1. Environment - Oracle Linux x86-64 on a server with 128 GB RAM and dual Intel Xeon Processors. Host on Windows Server.

      2. Installed R12.2.0 from scratch on Virtualbox VM.

      3. Completed all prerequisite tasks and applied patches as noted in MOS Docs 1320300.1 and 1560906.1. No errors were thrown and always ended up in completing normal.

      4. Followed MOS Note 1506669.1 to apply mega patch 16207672 for 12.2.2 upgrade with patch 16207673. Its currently over 15 hours that its running on the above machine after we issued following adop command:

       

      adop phase=apply patches 16207672, 16207673

       

      I understand that patch 16207672 for 12.2.2 upgrade is a mega patch, but given the configuration of our server I was prepared for a couple of hours. But it seems to be taking too long now. Any ideas on how long will it take on application and/or is there a way I can find out if everything is going alright?


      Any pointers will be greatly appreciated. Thanks.

       

      Regards

        • 1. Re: Upgrade from 12.2.0 to 12.2.2 - Patch 16207672 taking too long
          Kabeer Khan

          Some other details:

          1. DB up and running.

          2. Listener is up and running.

          3. Apps Tier up and running (including WebLogic Admin Server)

          4. So far received following messages on command prompt (in last 15 hours) from running patches 16207672, 16207673.

           

          [applvis@ebs122 1222]$ adop phase=apply patches=16207672,16207673

          Enter the APPS password:

          Enter the SYSTEM password:

          Enter the WLSADMIN password:

           

          Please wait. Validating credentials...

          RUN file system context file: /u01/oracle/VIS/fs2/inst/apps/VIS_ebs122/appl/admin/VIS_ebs122.xml

          PATCH file system context file: /u01/oracle/VIS/fs1/inst/apps/VIS_ebs122/appl/admin/VIS_ebs122.xml

          ************* Start of  session *************

          version: 12.2.0

          started at: Wed Nov 27 2013 21:27:47

           

          APPL_TOP is set to /u01/oracle/VIS/fs2/EBSapps/appl

            Using ADOP Session ID from currently incomplete patching cycle

          [START 2013/11/27 21:28:21] adzdoptl.pl run

            ADOP Session ID: 4

            Phase: apply

            Log file: /u01/oracle/VIS/fs_ne/EBSapps/log/adop/4/adop_20131127_212707.log

            [START 2013/11/27 21:28:38] apply phase

              Calling: adpatch  workers=24      flags=autoskip   console=no interactive=no  defaultsfile=/u01/oracle/VIS/fs1/EBSapps/appl/admin/VIS_patch/adalldefaults.txt patchtop=/u01/oracle/VIS/fs_ne/EBSapps/patch/16207672 driver=u16207672.drv logfile=u16207672.log

              ADPATCH Log directory: /u01/oracle/VIS/fs_ne/EBSapps/log/adop/4/apply_20131127_212707/VIS_ebs122/16207672/log

          • 2. Re: Upgrade from 12.2.0 to 12.2.2 - Patch 16207672 taking too long
            Hussein Sawwan-Oracle

            It took less than 5 hours last time I did this upgrade.

             

            What is the status of the workers? What script(s) are running right now?

             

            Can you find any errors in the database log file?

             

            Did you review (Best Practices for Minimizing Oracle E-Business Suite Release 12 Upgrade Downtime (Doc ID 1581549.1))?

             

            Thanks,

            Hussein

            • 3. Re: Upgrade from 12.2.0 to 12.2.2 - Patch 16207672 taking too long
              Kabeer Khan

              Hi

              Thanks for replying. I am looking at (Doc ID 1581549.1), which provides some good information.

               

              Just looked at u16207672.log and below given is the latest status. There are a couple of errors and number of warnings in the log, which I have pasted below. Unlike adpatch, this adop utility is running in the background and not printing messages on the console. I wonder if there's any way to see the latest patch messages OR looking at the log is the only way?

               

              1. Latest Status

              There are now 3728 jobs remaining (current phase=A64):

                  16 running, 2110 ready to run and 1602 waiting.

               

              2. Errors (pasted all errors) and Warnings (pasted selected warnings but there are many many more):

              ---gmf----------------------------------------

              Extracting object modules for product gmf...

               

               

              adlibout: error: The following object modules were not found in libgmf.a.

               

               

               

              ---mso----------------------------------------

              Extracting object modules for product mso...

               

               

              adlibout: error: The following object modules were not found in libmso.a.

               

               

               

              ---msr----------------------------------------

              Extracting object modules for product msr...

               

               

              adlibout: error: The following object modules were not found in libmsr.a.

               

               

              msrtest.o

               

               

              Warning: oracle/apps/xdo/excel/calcmodel/ErrorToken - Invalid entry. No resource unit matching this pattern found.

               

               

               

              WARNING: Unable to load metadata entry from zip file /u01/oracle/VIS/fs1/EBSapps/10.1.2/forms/java/frmall.jar

              Cause: Error while trying to read metadata from file /u01/oracle/VIS/fs1/EBSapps/10.1.2/forms/java/frmall.jar: No metadata entry META-INF/JRIMETA.DAT found

               

               

               

               

               

              WARNING: Unable to load metadata entry from zip file /u01/oracle/VIS/fs1/EBSapps/10.1.2/jlib/ewt3.jar

              Cause: Error while trying to read metadata from file /u01/oracle/VIS/fs1/EBSapps/10.1.2/jlib/ewt3.jar: No metadata entry META-INF/JRIMETA.DAT found

               

               

               

               

               

              WARNING: Unable to load metadata entry from zip file /u01/oracle/VIS/fs1/EBSapps/10.1.2/jlib/ewt3-nls.jar

              Cause: Error while trying to read metadata from file /u01/oracle/VIS/fs1/EBSapps/10.1.2/jlib/ewt3-nls.jar: No metadata entry META-INF/JRIMETA.DAT found

               

               

               

              WARNING: Unable to load metadata entry from zip file /u01/oracle/VIS/fs1/FMW_Home/oracle_common/modules/oracle.jdbc_11.1.1/ojdbc6dms.jar

              Cause: Error while trying to read metadata from file /u01/oracle/VIS/fs1/FMW_Home/oracle_common/modules/oracle.jdbc_11.1.1/ojdbc6dms.jar: No metadata entry META-INF/JRIMETA.DAT found

               

               

               

               

               

              WARNING: Unable to load metadata entry from zip file /u01/oracle/VIS/fs1/FMW_Home/oracle_common/modules/oracle.xdk_11.1.0/xmlparserv2.jar

              Cause: Error while trying to read metadata from file /u01/oracle/VIS/fs1/FMW_Home/oracle_common/modules/oracle.xdk_11.1.0/xmlparserv2.jar: No metadata entry META-INF/JRIMETA.DAT found

               

               

               

               

               

              WARNING: Unable to load metadata entry from zip file /u01/oracle/VIS/fs1/FMW_Home/oracle_common/modules/oracle.help_5.0/ohw-uix.jar

              Cause: Error while trying to read metadata from file /u01/oracle/VIS/fs1/FMW_Home/oracle_common/modules/oracle.help_5.0/ohw-uix.jar: No metadata entry META-INF/JRIMETA.DAT found

               

               

               

              WARNING: Unable to load metadata entry from zip file /u01/oracle/VIS/fs1/FMW_Home/oracle_common/jlib/share.jar

              Cause: Error while trying to read metadata from file /u01/oracle/VIS/fs1/FMW_Home/oracle_common/jlib/share.jar: No metadata entry META-INF/JRIMETA.DAT found

               

               

               

              WARNING: Unable to load metadata entry from zip file /u01/oracle/VIS/fs1/FMW_Home/jrockit32/jre/lib/rt.jar

              Cause: Error while trying to read metadata from file /u01/oracle/VIS/fs1/FMW_Home/jrockit32/jre/lib/rt.jar: No metadata entry META-INF/JRIMETA.DAT found

              • 5. Re: Upgrade from 12.2.0 to 12.2.2 - Patch 16207672 taking too long
                Srini Chavali-Oracle

                How big (processors / memory) is your VM ? What is the SGA size ? What kind of storage are you using ?

                 

                In my home VM ( 2 processors, 12G RAM, with USB3 storage ) this patch process took about 24 hours

                 

                HTH
                Srini

                • 6. Re: Upgrade from 12.2.0 to 12.2.2 - Patch 16207672 taking too long
                  Hussein Sawwan-Oracle

                  You can safely ignore the above warning messages.

                   

                  I wonder if there's any way to see the latest patch messages OR looking at the log is the only way?

                   

                  Please check the worker log files for details about each worker script/process.

                   

                  Thanks,

                  Hussein

                  • 7. Re: Upgrade from 12.2.0 to 12.2.2 - Patch 16207672 taking too long
                    Kabeer Khan

                    It finally completed after 24 hours. I realized that the messages was not printing on the console because adop was run with console=no, interactive=no, and instead the messages were printing in the background.

                     

                    I still wonder why its taken so long since the actual 12.2.0 installation was complete in around 12 hours.

                     

                    Anyways I will proceed now to run rest of the steps.

                     

                    Thanks guys for your help.

                     

                    Regards

                    Kabeer

                    • 8. Re: Upgrade from 12.2.0 to 12.2.2 - Patch 16207672 taking too long
                      Hussein Sawwan-Oracle
                      I still wonder why its taken so long since the actual 12.2.0 installation was complete in around 12 hours.

                       

                      There are many factors, like how much memory you have, the number of CPUs, the number of workers, ..etc.

                       

                      I wouldn't recommended using more than (2 x # CPUs) for the number of workers. And, I would suggest you follow the guidelines in the doc referenced in one of my previous replies.

                       

                      Thanks,

                      Hussein

                      • 9. Re: Upgrade from 12.2.0 to 12.2.2 - Patch 16207672 taking too long
                        Kabeer Khan
                        I wouldn't recommended using more than (2 x # CPUs) for the number of workers.

                        Are you referring to physical CPU's or the threads within a CPU? Is there a way to impose this restriction while running adop?


                        And, I would suggest you follow the guidelines in the doc referenced in one of my previous replies.

                        Sure, will do. Thanks.

                        • 10. Re: Upgrade from 12.2.0 to 12.2.2 - Patch 16207672 taking too long
                          Hussein Sawwan-Oracle

                          Please see the following docs.

                           

                          How To Select Number of Workers Based on Number of CPUs When Running ADPATCH (Doc ID 226191.1)

                          How Does Adpatch Determine The Number Of Workers To Recommend? (Doc ID 800024.1)

                           

                          Thanks,

                          Hussein

                          1 person found this helpful
                          • 11. Re: Upgrade from 12.2.0 to 12.2.2 - Patch 16207672 taking too long
                            Kabeer Khan

                            Thank you and we could finally manage to install 12.2.2 successfully.