13 Replies Latest reply on Dec 21, 2015 7:32 PM by Timo Hahn

    Cloning appTier taking too long?

    Beauty_and_dBest

      Hi ALL,

       

      EBS R12.2.4

      OEL6

       

      I am running cloning again (appsTier only) to transfer it to another server.

      But the partner dbTier will just be on the same server.

      I run node clean first on the dbTier and then run autoconfig.

      Then I start cloning the appsTier, but it is take too long like 2 hrs staying on %50.

      I monitor  "top" activity but there no much activity

      Capture.PNG

       

      Please help....

       

      Thanks a lot,

      jc

        • 1. Re: Cloning appTier taking too long?
          Kj - Kiran Jadhav

          Please check the cloning log file(ApplyAppsTier_12201226.log) as shown in screenshot and see which script is currently running.

           

          Regards,

          Kiran

          1 person found this helpful
          • 2. Re: Cloning appTier taking too long?
            Hussein Sawwan-Oracle

            In addition to the log file (please upload it for us to review) check the database alert log and make sure no errors are there.

             

            Also, please check if there are any locks in the DB. If you find any, bounce the DB and restart postclone on the apps tier node.

             

            Thanks,

            Hussein

            1 person found this helpful
            • 3. Re: Cloning appTier taking too long?
              Beauty_and_dBest

              Thanks hussein and kj,

               

              I will check and rerun the process....brb

              • 4. Re: Cloning appTier taking too long?
                Beauty_and_dBest

                Hi hussein and ALL,

                 

                This is the log of thang hung process:

                 

                +++++++++++++++++++++++++

                configGroup Type name = MACHINE_CONFIG

                configProperty id = Machine1

                Count for NodeIterator nextNode = 3

                        Updating Machine Name to dbs

                        Updating dbs Listen Address to dbs.oracle.amti.com.ph

                        Updating dbs Listen Port to 5576

                +++++++++++++++++++++++++

                configGroup Type name = CLUSTER_CONFIG

                configGroup Type name = DATASOURCE

                Setting Url for DataSource1 to jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS_LIST=(LOAD_BALANCE=YES)(FAILOVER=YES)(ADDRESS=(PROTOCOL=tcp)(HOST=dbp.oracle.amti.com.ph)(PORT=1541)))(CONNECT_DATA=(SERVICE_NAME=PROD)))

                Setting Password file for DataSource1 to /home/appprod/PROD/fs1/EBSapps/comn/clone/FMW/EBSDataSource

                +++++++++++++++++++++++++

                configGroup Type name = OPSS_SECURITY

                configGroup Type name = DEPLOYMENT_PLAN_CONFIG

                Writing moveplan xml file

                Updating the deployment plan /home/appprod/PROD/fs1/EBSapps/comn/clone/FMW/WLS/plan/deployment_plans/forms#1.0.0_plan.xml

                Updating config-root element with val =/home/appprod/PROD/fs1/FMW_Home/Oracle_EBS-app1/deployment_plans/forms

                Updating the deployment plan /home/appprod/PROD/fs1/EBSapps/comn/clone/FMW/WLS/plan/deployment_plans/forms-c4ws#1.0.0_plan.xml

                Updating config-root element with val =/home/appprod/PROD/fs1/FMW_Home/Oracle_EBS-app1/deployment_plans/forms-c4ws

                Updating the deployment plan /home/appprod/PROD/fs1/EBSapps/comn/clone/FMW/WLS/plan/deployment_plans/oacore#1.0.0_plan.xml

                Refreshing the content of help_InitParam_ohwConfigFileURL to file:/home/appprod/PROD/fs1/FMW_Home/Oracle_EBS-app1/common/config/ohwconfig.xml

                Refreshing the content of CGIServlet_InitParam_cgiDir to /home/appprod/PROD/fs1/FMW_Home/Oracle_EBS-app1/common/scripts

                Refreshing the content of CGIServlet_InitParam_*.pl to /home/appprod/PROD/fs1/FMW_Home/webtier/perl/bin/perl

                Refreshing the content of oowa_InitParam_log_main_file to /usr/tmp/oowa.log

                Refreshing the content of oowa_InitParam_log_spl2_file to /usr/tmp/oowa.log

                Refreshing the content of LoopProcServlet_InitParam_ARCHIVE to /usr/tmp

                Refreshing the content of TransmitServlet_InitParam_ARCHIVE to /usr/tmp

                Updating config-root element with val =/home/appprod/PROD/fs1/FMW_Home/Oracle_EBS-app1/deployment_plans/oacore

                Updating the deployment plan /home/appprod/PROD/fs1/EBSapps/comn/clone/FMW/WLS/plan/deployment_plans/oafm#1.0.0_plan.xml

                Updating config-root element with val =/home/appprod/PROD/fs1/FMW_Home/Oracle_EBS-app1/deployment_plans/oafm

                Backup domain directory if exists

                Backup applications domain directory if exists

                 

                START: Creating new WLS domain.

                Running /home/appprod/PROD/fs1/FMW_Home/oracle_common/bin/pasteConfig.sh -javaHome /home/appprod/PROD/fs1/EBSapps/comn/util/jdk64 -al /home/appprod/PROD/fs1/EBSapps/comn/clone/FMW/WLS/EBSdomain.jar -tdl /home/appprod/PROD/fs1/FMW_Home/user_projects/domains/EBS_domain_PROD -tmw /home/appprod/PROD/fs1/FMW_Home -mpl /home/appprod/PROD/fs1/EBSapps/comn/clone/FMW/WLS/plan/moveplan.xml -ldl /home/appprod/PROD/fs1/inst/apps/PROD_dbs/admin/log/clone/wlsT2PApply -silent true -debug true -domainAdminPassword /home/appprod/PROD/fs1/EBSapps/comn/clone/FMW/tempinfo.txt

                Script Executed in 7200100 milliseconds, returning status -1

                ERROR: Script timed out.

                 

                ================

                 

                Initializing WebLogic Scripting Tool (WLST) ...

                Welcome to WebLogic Server Administration Scripting Shell

                Type help() for help on available commands

                 

                Error: cd() failed. Do dumpStack() to see details.

                SSL setting failed

                Error: cd() failed. Do dumpStack() to see details.

                SSL setting failed

                Error: cd() failed. Do dumpStack() to see details.

                SSL setting failed

                Error: cd() failed. Do dumpStack() to see details.

                SSL setting failed

                Configuring the Server -  AdminServer

                Configuring the Server -  oacore_server1

                Configuring the Server -  forms_server1

                Configuring the Server -  oafm_server1

                Configuring the Server -  forms-c4ws_server1

                Configuring the Machine -  dbs

                Configuring the Cluster -  oafm_cluster1

                Configuring the Cluster -  forms-c4ws_cluster1

                Configuring the Cluster -  forms_cluster1

                Configuring the Cluster -  oacore_cluster1

                Configuring the Datasource -  EBSDataSource

                 

                Exiting WebLogic Scripting Tool.

                 

                INFO : Dec 20, 2015 5:54:50 PM - CLONE-23283  Unpack Domain.

                FINE : Dec 20, 2015 5:54:50 PM - [J2EEComponentApplyCloner:unpackDomain] Domain Target Location:/home/appprod/PROD/fs1/FMW_Home/user_projects/domains/EBS_domain_PROD

                FINE : Dec 20, 2015 5:54:50 PM - [J2EEComponentApplyCloner:unpackDomain] Unpack Command:/home/appprod/PROD/fs1/FMW_Home/oracle_common/common/bin/unpack.sh -domain=/home/appprod/PROD/fs1/FMW_Home/user_projects/domains/EBS_domain_PROD -template=/tmp/CLONINGCLIENT349527964440992550/packed_template.jar -server_start_mode=prod

                << read template from "/tmp/CLONINGCLIENT349527964440992550/packed_template.jar"

                >>  succeed: read template from "/tmp/CLONINGCLIENT349527964440992550/packed_template.jar"

                << set config option ServerStartMode to "prod"

                >>  succeed: set config option ServerStartMode to "prod"

                << write Domain to "/home/appprod/PROD/fs1/FMW_Home/user_projects/domains/EBS_domain_PROD"

                Exception in thread "Thread-1" java.lang.OutOfMemoryError: Java heap space

                        at java.util.Arrays.copyOfRange(Arrays.java:2694)

                        at java.lang.String.<init>(String.java:203)

                        at java.io.BufferedReader.readLine(BufferedReader.java:349)

                        at java.io.BufferedReader.readLine(BufferedReader.java:382)

                        at com.oracle.cie.common.util.CRLF.readData(CRLF.java:129)

                        at com.oracle.cie.common.util.CRLF.processFile(CRLF.java:67)

                        at com.oracle.cie.common.util.CRLF.process(CRLF.java:57)

                        at com.oracle.cie.domain.util.stringsub.SubsScriptHelper.processCRLF(SubsScriptHelper.java:167)

                        at com.oracle.cie.domain.DomainGenerator.generate(DomainGenerator.java:478)

                        at com.oracle.cie.domain.script.ScriptExecutor$2.run(ScriptExecutor.java:2992)

                 

                =============

                 

                Please help...

                 

                Thanks,

                • 5. Re: Cloning appTier taking too long?
                  Chiranjeevi M A

                  Hello JC,

                   

                  Please refer to the following MOS: ADOP Fails with OutOfMemoryError: Java Heap Space Error (Doc ID 1679761.1)

                   

                  -Chiranjeevi M A

                  1 person found this helpful
                  • 7. Re: Cloning appTier taking too long?
                    Beauty_and_dBest

                    I tried running 2nd time to see if the same error will occur, but I do not know if I am reading the same error logs or new ones?

                    Are cloning logs overwritten? or appeded? Thanks

                     

                     

                    START: Creating new WLS domain.

                    Running /home/appprod/PROD/fs1/FMW_Home/oracle_common/bin/pasteConfig.sh -javaHome /home/appprod/PROD/fs1/EBSapps/comn/util/jdk64 -al /home/appprod/PROD/fs1/EBSapps/comn/clone/FMW/WLS/EBSdomain.jar -tdl /home/appprod/PROD/fs1/FMW_Home/user_projects/domains/EBS_domain_PROD -tmw /home/appprod/PROD/fs1/FMW_Home -mpl /home/appprod/PROD/fs1/EBSapps/comn/clone/FMW/WLS/plan/moveplan.xml -ldl /home/appprod/PROD/fs1/inst/apps/PROD_dbs/admin/log/clone/wlsT2PApply -silent true -debug true -domainAdminPassword /home/appprod/PROD/fs1/EBSapps/comn/clone/FMW/tempinfo.txt

                    Script Executed in 7200049 milliseconds, returning status -1

                    ERROR: Script timed out.

                     

                    ================

                    Exiting WebLogic Scripting Tool.

                     

                    INFO : Dec 21, 2015 12:52:34 PM - CLONE-23283  Unpack Domain.

                    FINE : Dec 21, 2015 12:52:34 PM - [J2EEComponentApplyCloner:unpackDomain] Domain Target Location:/home/appprod/PROD/fs1/FMW_Home/user_projects/domains/EBS_domain_PROD

                    FINE : Dec 21, 2015 12:52:34 PM - [J2EEComponentApplyCloner:unpackDomain] Unpack Command:/home/appprod/PROD/fs1/FMW_Home/oracle_common/common/bin/unpack.sh -domain=/home/appprod/PROD/fs1/FMW_Home/user_projects/domains/EBS_domain_PROD -template=/tmp/CLONINGCLIENT2525752224566029745/packed_template.jar -server_start_mode=prod

                    << read template from "/tmp/CLONINGCLIENT2525752224566029745/packed_template.jar"

                    >>  succeed: read template from "/tmp/CLONINGCLIENT2525752224566029745/packed_template.jar"

                    << set config option ServerStartMode to "prod"

                    >>  succeed: set config option ServerStartMode to "prod"

                    << write Domain to "/home/appprod/PROD/fs1/FMW_Home/user_projects/domains/EBS_domain_PROD"

                    Exception in thread "Thread-1" java.lang.OutOfMemoryError: Java heap space

                            at java.util.Arrays.copyOfRange(Arrays.java:2694)

                            at java.lang.String.<init>(String.java:203)

                            at java.io.BufferedReader.readLine(BufferedReader.java:349)

                            at java.io.BufferedReader.readLine(BufferedReader.java:382)

                            at com.oracle.cie.common.util.CRLF.readData(CRLF.java:129)

                            at com.oracle.cie.common.util.CRLF.processFile(CRLF.java:67)

                            at com.oracle.cie.common.util.CRLF.process(CRLF.java:57)

                            at com.oracle.cie.domain.util.stringsub.SubsScriptHelper.processCRLF(SubsScriptHelper.java:167)

                            at com.oracle.cie.domain.DomainGenerator.generate(DomainGenerator.java:478)

                            at com.oracle.cie.domain.script.ScriptExecutor$2.run(ScriptExecutor.java:2992)

                    • 8. Re: Cloning appTier taking too long?
                      Bashar.

                      Check out this document:

                       

                      ADOP Fails with OutOfMemoryError: Java Heap Space Error (Doc ID 1679761.1)

                       

                      Regards,

                      Bashar

                      1 person found this helpful
                      • 9. Re: Cloning appTier taking too long?
                        Beauty_and_dBest

                        Hi ALL,

                         

                        I just confused why this happened first time

                        I been cloning on the same or similar servers many times but this is the first time I encountered this error.

                        There must me parameters I have not set correctly? related to this error?

                        Could this be one of these parameters?

                        1.

                        # vi /etc/sysctl.conf (half the size of Physical Mem)

                        net.ipv4.ip_forward = 0

                        net.ipv4.conf.default.rp_filter = 1

                        net.ipv4.conf.default.accept_source_route = 0

                        kernel.sysrq = 0

                        kernel.core_uses_pid = 1

                        net.ipv4.tcp_syncookies = 1

                        net.bridge.bridge-nf-call-ip6tables = 0

                        net.bridge.bridge-nf-call-iptables = 0

                        net.bridge.bridge-nf-call-arptables = 0

                        kernel.msgmnb = 65536

                        kernel.msgmax = 8192

                        kernel.shmall = 2097152

                        net.ipv4.conf.all.arp_notify = 1

                        kernel.msgmni = 2878

                        fs.file-max = 6815744

                        kernel.sem = 256 32000 100 142

                        kernel.shmmni = 4096

                        kernel.shmmax = 6000000000   

                        kernel.panic_on_oops = 1

                        net.core.rmem_default = 262144

                        net.core.rmem_max = 4194304

                        net.core.wmem_default = 262144

                        net.core.wmem_max = 1048576

                        fs.aio-max-nr = 1048576

                        net.ipv4.ip_local_port_range = 9000 65500

                        net.ipv4.tcp_tw_recycle = 0

                         

                        2.

                        # vi /etc/security/limits.conf

                        * hard nofile 65536

                        * soft nofile 4096

                        * hard nproc 16384

                        * soft nproc 2047

                        * hard stack 16384

                        * soft stack 10240

                        • 10. Re: Cloning appTier taking too long?
                          Hussein Sawwan-Oracle

                          Jenna_C wrote:

                           

                          Hi ALL,

                           

                          The MOS noted also mentioned:

                           

                          The log files under /EBS_domain_<SID>/servers/oacore_server1/logs are too big and cause the error.

                          Removing these log files can be a valid workaround as well.

                           

                          What is the actaul name of the file? I tried searching "oacore_server1" but there is no such folder name

                           

                          Thanks.

                           

                          Jc,

                           

                          We already shared with you the doc to find the logs and the command to use in your other thread -- https://community.oracle.com/thread/3872948

                           

                          You marked one of the replies in this thread as Correct so we believe your question was already answered so why keep updating same thread?

                           

                          Thanks,

                          Hussein

                          • 11. Re: Cloning appTier taking too long?
                            Beauty_and_dBest

                            I was looking for 2nd option as what the MOS notes suggested.

                             

                             

                            Thanks,

                            • 12. Re: Cloning appTier taking too long?
                              Hussein Sawwan-Oracle

                              Jenna_C wrote:

                               

                              I was looking for 2nd option as what the MOS notes suggested.

                               

                               

                              Thanks,

                               

                              And you asked a question that we already answered many times before

                              • 13. Re: Cloning appTier taking too long?
                                Timo Hahn

                                **************************Moderator action (Timo)

                                Locked this thread as it is answered.

                                The information given should be enough to solve the problem.

                                **************************