1 2 Previous Next 15 Replies Latest reply on Apr 12, 2017 6:06 PM by 966638

    Add node in R12.2

    966638

      Hi,

       

      I have added apps node into R12.2 instance and all went fine.  I have started the apps and some script exit with status 0.

       

      When I check in Weblogic console, I can see below screen.

       

      Name        Cluster  Machine     State  Health     Listen Port
          AdminServer(admin)    oracleerpdev    RUNNING      OK    7002
          forms-c4ws_server1    forms-c4ws_cluster1    APP1 SHUTDOWN 7802
          forms-c4ws_server2    forms-c4ws_cluster1    APP2 SHUTDOWN 7801
          forms_server1 forms_cluster1 APP1 RUNNING      OK    7402
          forms_server2 forms_cluster1 APP2 FAILED_NOT_RESTARTABLE 7401
          oacore_server1 oacore_cluster1 APP1 RUNNING      OK    7202
          oacore_server2 oacore_cluster1 APP2 FAILED_NOT_RESTARTABLE 7201
          oafm_server1 oafm_cluster1 APP1 RUNNING      OK    7602
          oafm_server2 oafm_cluster1 APP2 FAILED_NOT_RESTARTABLE 7601

       

      I followed the note :  1383621.1 ->  5.3 Adding a New Application Tier Node to an Existing System. In this step, skipped step (4. Register the new topology from the newly added application tier node.) .  Do i need to execute this step also.

       

      CM script completed " adcmctl.sh: exiting with status 0" but I can't find the FND process on this server

       

      In front end concurrent managers screen, service manager for this APP2 node - Actual 1  and target is 0.

       

      My requirement is to add node with a complete environment ( forms/web/CM).

        • 1. Re: Add node in R12.2
          Shaik

          Hi

           

          Are you going to load balance the application?

           

          If yes please go through the below note

          Using Load-Balancers with Oracle E-Business Suite Release 12.2 (Doc ID 1375686.1)

           

          Regards

          Shaik

          • 2. Re: Add node in R12.2
            966638

            Thanks . I don't have load balancer

            I want to add and try high availability and PCP. How I can achieve without HW loadbalancer.

             

             

            Please confirm below step (4) need to perform?

             

            1383621.1 ->  5.3 Adding a New Application Tier Node to an Existing System. In this step, skipped step (4. Register the new topology from the newly added application tier node.) .  Do i need to execute this step also.

            • 3. Re: Add node in R12.2
              Prashant Umap

              Hello,

              Thanks for the post !

               

              Yes, You need to complete this step in order to achieve PCP.

               

              Regards

              Prashant Umap

              • 4. Re: Add node in R12.2
                966638

                Thanks a lot Prasanth,

                 

                1)   After completing the step 4, PCP is working. But when I stop listerner / cmctl stop on second node, it is stopping CMs on first node also.

                      How I can fix?

                 

                2) Still my Weblogic console is showing "FAILED_NOT_RESTARTABLE" for oacore/forms/oafm.

                    How I can fix?

                 

                 

                Thanks,

                Sal

                • 5. Re: Add node in R12.2
                  Shaik

                  Hi Sai

                   

                  For point 1.

                  Its Normal behaviour, if you stop CM on second node , node 1 Cm services also will go down

                   

                  PCP Failover is the process of migrating all of the defined concurrent managers to a secondary node upon listener or node failure.

                   

                  You can see below thread which i raised previously for the same

                  PCP Configuration in Two Node RAC

                   

                  Point 2:

                  I hope its setup failure , please doble check the steps

                   

                  Is any ADOP session failed?

                   

                   

                  Regards

                  Shaik

                  • 6. Re: Add node in R12.2
                    966638

                    Thanks for the update.

                     

                    Point 1:

                    --------

                    I have created custom manager assigned APP2 as primary node and APP1 as secondary node.Assigned Active users prgms.

                    I have shutdown the node2 (APP2), but the custom manager took more than 10 min to migrarte to APP1 and to complete the requests

                    What is normal time and where I can set/fix to reduce the time

                     

                    Point 2:

                    --------

                    No failed adop session, what could be the possibilty

                    • 7. Re: Add node in R12.2
                      966638

                      For clarity:

                      ---------------

                       

                      In primary node ( APP1)

                           The run file system port pool is "1"

                                    <PORT_POOL oa_var="s_port_pool">1</PORT_POOL>

                           The patch file system port pool is "0"

                                    <PORT_POOL oa_var="s_port_pool">0</PORT_POOL>           

                       

                      In Secondary node (APP2)

                           The run file system port pool is "0"

                                   <PORT_POOL oa_var="s_port_pool">0</PORT_POOL>

                           The patch file system port pool is "1"

                                   <PORT_POOL oa_var="s_port_pool">1</PORT_POOL>

                       

                      Could be a reason for the above issue?

                      • 8. Re: Add node in R12.2
                        966638

                        $admanagedsrvctl.sh  start forms_server2

                         

                        And when I start manager server individually, get below error:

                        -------------------------------------------------------------------------------

                         

                        ####<Mar 16, 2017 3:07:17 PM AST> <Critical> <WebLogicServer> <APP2> <forms_server2> <main> <<WLS Kernel>> <> <> <1489666037601> <BEA-000386> <Server subsystem failed. Reason: weblogic.security.SecurityInitializationException: The loading of OPSS java security policy provider failed due to exception, see the exception stack trace or the server log file for root cause. If still see no obvious cause, enable the debug flag -Djava.security.debug=jpspolicy to get more information. Error message: JPS-06514: Opening of file based keystore failed.

                        weblogic.security.SecurityInitializationException: The loading of OPSS java security policy provider failed due to exception, see the exception stack trace or the server log file for root cause. If still see no obvious cause, enable the debug flag -Djava.security.debug=jpspolicy to get more information. Error message: JPS-06514: Opening of file based keystore failed.

                                at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.loadOPSSPolicy(CommonSecurityServiceManagerDelegateImpl.java:1402)

                                at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initialize(CommonSecurityServiceManagerDelegateImpl.java:1022)

                                at weblogic.security.service.SecurityServiceManager.initialize(SecurityServiceManager.java:888)

                                at weblogic.security.SecurityService.start(SecurityService.java:141)

                                at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)

                                at weblogic.work.ExecuteThread.execute(ExecuteThread.java:263)

                                at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)

                        Caused By: oracle.security.jps.JpsRuntimeException: JPS-06514: Opening of file based keystore failed.

                                at oracle.security.jps.internal.policystore.PolicyDelegationController.<init>(PolicyDelegationController.java:190)

                                at oracle.security.jps.internal.policystore.TenantJavaPolicyProvider.<init>(TenantJavaPolicyProvider.java:161)

                                at oracle.security.jps.internal.policystore.JavaPolicyProvider.<init>(JavaPolicyProvider.java:306)

                                at oracle.security.jps.internal.policystore.JavaPolicyProvider.<init>(JavaPolicyProvider.java:279)

                                at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

                                at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)

                                at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

                                at java.lang.reflect.Constructor.newInstance(Constructor.java:526)

                                at java.lang.Class.newInstance(Class.java:383)

                                at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.loadOPSSPolicy(CommonSecurityServiceManagerDelegateImpl.java:1343)

                                at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initialize(CommonSecurityServiceManagerDelegateImpl.java:1022)

                                at weblogic.security.service.SecurityServiceManager.initialize(SecurityServiceManager.java:888)

                        at weblogic.security.SecurityService.start(SecurityService.java:141)

                                at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)

                                at weblogic.work.ExecuteThread.execute(ExecuteThread.java:263)

                                at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)

                        Caused By: oracle.security.jps.JpsException: JPS-06514: Opening of file based keystore failed.

                                at oracle.security.jps.internal.policystore.PolicyUtil.getDefaultPDPService(PolicyUtil.java:3180)

                                at oracle.security.jps.internal.policystore.PolicyUtil.getPDPService(PolicyUtil.java:3480)

                                at oracle.security.jps.internal.policystore.PolicyUtil.getPDPService(PolicyUtil.java:3466)

                                at oracle.security.jps.internal.policystore.PolicyDelegationController.<init>(PolicyDelegationController.java:188)

                                at weblogic.security.SecurityService.start(SecurityService.java:141)

                                at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)

                                at weblogic.work.ExecuteThread.execute(ExecuteThread.java:263)

                                at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)

                        Caused By: oracle.security.jps.service.keystore.KeyStoreServiceException: JPS-06514: Opening of file based keystore failed.

                                at oracle.security.jps.internal.keystore.file.FileKeyStoreManager.openKeyStore(FileKeyStoreManager.java:458)

                                at oracle.security.jps.internal.keystore.file.FileKeyStoreManager.openKeyStore(FileKeyStoreManager.java:392)

                                at oracle.security.jps.internal.keystore.file.FileKeyStoreServiceImpl.doInit(FileKeyStoreServiceImpl.java:128)

                                at oracle.security.jps.internal.keystore.file.FileKeyStoreServiceImpl.start(FileKeyStoreServiceImpl.java:901)

                                at oracle.security.jps.internal.core.runtime.ContextFactoryImpl.findServiceInstance(ContextFactoryImpl.java:139)

                                at oracle.security.jps.internal.core.runtime.ContextFactoryImpl.getContext(ContextFactoryImpl.java:170)

                                at oracle.security.jps.internal.core.runtime.ContextFactoryImpl.getContext(ContextFactoryImpl.java:191)

                                at oracle.security.jps.internal.core.runtime.JpsContextFactoryImpl.getContext(JpsContextFactoryImpl.java:132)

                                at oracle.security.jps.internal.core.runtime.JpsContextFactoryImpl.getContext(JpsContextFactoryImpl.java:159)

                                at oracle.security.jps.internal.core.runtime.JpsContextFactoryImpl.getContext(JpsContextFactoryImpl.java:165)

                                at oracle.security.jps.internal.policystore.PolicyUtil$3.run(PolicyUtil.java:3152)

                                at oracle.security.jps.internal.policystore.PolicyUtil$3.run(PolicyUtil.java:3149)

                             at java.security.AccessController.doPrivileged(Native Method)

                                at oracle.security.jps.internal.policystore.PolicyUtil.getDefaultPDPService(PolicyUtil.java:3149)

                                at oracle.security.jps.internal.policystore.PolicyUtil.getPDPService(PolicyUtil.java:3480)

                                at oracle.security.jps.internal.policystore.PolicyUtil.getPDPService(PolicyUtil.java:3466)

                                at oracle.security.jps.internal.policystore.PolicyDelegationController.<init>(PolicyDelegationController.java:188)

                                at oracle.security.jps.internal.policystore.TenantJavaPolicyProvider.<init>(TenantJavaPolicyProvider.java:161)

                                at oracle.security.jps.internal.policystore.JavaPolicyProvider.<init>(JavaPolicyProvider.java:306)

                                at oracle.security.jps.internal.policystore.JavaPolicyProvider.<init>(JavaPolicyProvider.java:279)

                                at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

                                at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)

                                at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

                                at java.lang.reflect.Constructor.newInstance(Constructor.java:526)

                                at java.lang.Class.newInstance(Class.java:383)

                                at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.loadOPSSPolicy(CommonSecurityServiceManagerDelegateImpl.java:1343)

                                at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initialize(CommonSecurityServiceManagerDelegateImpl.java:1022)

                                at weblogic.security.service.SecurityServiceManager.initialize(SecurityServiceManager.java:888)

                                at weblogic.security.SecurityService.start(SecurityService.java:141)

                                at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)

                                at weblogic.work.ExecuteThread.execute(ExecuteThread.java:263)

                                at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)

                        Caused By: oracle.security.jps.JpsRuntimeException: java.io.FileNotFoundException: /tmp/keystores1320796146xml.lck (Permission denied)

                                at oracle.security.jps.internal.core.datastore.xml.FileLocker.lock(FileLocker.java:82)

                                at oracle.security.jps.internal.keystore.file.FileKeyStoreManager.openKeyStore(FileKeyStoreManager.java:431)

                                at oracle.security.jps.internal.keystore.file.FileKeyStoreManager.openKeyStore(FileKeyStoreManager.java:392)

                                at oracle.security.jps.internal.keystore.KeyStoreProvider.getInstance(KeyStoreProvider.java:77)

                                at oracle.security.jps.internal.core.runtime.ContextFactoryImpl.findServiceInstance(ContextFactoryImpl.java:139)

                                at oracle.security.jps.internal.core.runtime.ContextFactoryImpl.getContext(ContextFactoryImpl.java:170)

                                at oracle.security.jps.internal.core.runtime.ContextFactoryImpl.getContext(ContextFactoryImpl.java:191)

                                at oracle.security.jps.internal.core.runtime.JpsContextFactoryImpl.getContext(JpsContextFactoryImpl.java:132)

                                at oracle.security.jps.internal.core.runtime.JpsContextFactoryImpl.getContext(JpsContextFactoryImpl.java:159)

                                at oracle.security.jps.internal.core.runtime.JpsContextFactoryImpl.getContext(JpsContextFactoryImpl.java:165)

                                at oracle.security.jps.internal.policystore.PolicyUtil$3.run(PolicyUtil.java:3152)

                                at oracle.security.jps.internal.policystore.PolicyUtil$3.run(PolicyUtil.java:3149)

                                at java.security.AccessController.doPrivileged(Native Method)

                                at oracle.security.jps.internal.policystore.PolicyUtil.getDefaultPDPService(PolicyUtil.java:3149)

                                at oracle.security.jps.internal.policystore.PolicyUtil.getPDPService(PolicyUtil.java:3480)

                        at oracle.security.jps.internal.policystore.PolicyDelegationController.<init>(PolicyDelegationController.java:188)

                                at oracle.security.jps.internal.policystore.TenantJavaPolicyProvider.<init>(TenantJavaPolicyProvider.java:161)

                                at oracle.security.jps.internal.policystore.JavaPolicyProvider.<init>(JavaPolicyProvider.java:306)

                                at oracle.security.jps.internal.policystore.JavaPolicyProvider.<init>(JavaPolicyProvider.java:279)

                                at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

                                at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)

                                at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

                                at java.lang.reflect.Constructor.newInstance(Constructor.java:526)

                                at java.lang.Class.newInstance(Class.java:383)

                                at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.loadOPSSPolicy(CommonSecurityServiceManagerDelegateImpl.java:1343)

                                at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initialize(CommonSecurityServiceManagerDelegateImpl.java:1022)

                                at weblogic.security.service.SecurityServiceManager.initialize(SecurityServiceManager.java:888)

                                at weblogic.security.SecurityService.start(SecurityService.java:141)

                                at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)

                                at weblogic.work.ExecuteThread.execute(ExecuteThread.java:263)

                                at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)

                        Caused By: java.io.FileNotFoundException: /tmp/keystores1320796146xml.lck (Permission denied)

                                at java.io.RandomAccessFile.open(Native Method)

                                at java.io.RandomAccessFile.<init>(RandomAccessFile.java:241)

                                at oracle.security.jps.internal.core.datastore.xml.FileLocker$1.run(FileLocker.java:143)

                                at oracle.security.jps.internal.core.datastore.xml.FileLocker$1.run(FileLocker.java:141)

                                at oracle.security.jps.internal.core.datastore.xml.FileLocker.newRandomAccessFile(FileLocker.java:148)

                                at oracle.security.jps.internal.core.datastore.xml.FileLocker.lock(FileLocker.java:75)

                                at oracle.security.jps.internal.keystore.file.FileKeyStoreManager.openKeyStore(FileKeyStoreManager.java:431)

                                at oracle.security.jps.internal.keystore.file.FileKeyStoreManager.openKeyStore(FileKeyStoreManager.java:392)

                                at oracle.security.jps.internal.keystore.file.FileKeyStoreServiceImpl.doInit(FileKeyStoreServiceImpl.java:128)

                                at oracle.security.jps.internal.keystore.KeyStoreProvider.getInstance(KeyStoreProvider.java:77)

                        • 9. Re: Add node in R12.2
                          Prashant Umap

                          Hello,

                          Please check the following setup -

                          1)    In DB, run these two commands as sysdba:

                          a.    alter system set “_lm_global_posts”=TRUE scope=spfile sid=’*’;

                          b.    alter system set “_immediate_commit_propagation”=TRUE scope=spfile sid=’*’;

                          2)    For each internal node, set autoconfig parameter APPLDCP to ON.

                          3)    For each internal node, verify autoconfig parameters APPLCSF and APPLPTMP match.  (typically APPLPTMP will not be correct on secondary nodes…make it match the primary node).

                          4)    Shut down application tier services and run autoconfig to implement changes.

                          5)    While application is down, bounce the DB for initialization parameters to take effect.

                          a.    srvctl stop database –d <SID>

                          b.    srvctl start database –d <SID>

                          6)    Log into SQL*Plus and confirm parameters have been set:

                          a.    show parameter lm

                          b.    show parameter immediate

                          7)    On application tier, set your environment again and make sure $APPLDCP environment variable is set:

                          a.    echo $APPLDCP (it should return ON…this variable is turned on after setting APPLDCP autoconfig parameter to ON)

                          8)    If $APPLDCP is set to ON, start the environment.  This will trigger the concurrent managers to start in PCP mode.

                          9)    Don’t forget to check $APPLDCP on the other internal node before starting the environment there.  If it’s not set on the second node after resetting your environment, then you may have forgotten to run autoconfig to enable that parameter.  But, if it is set, then also start services on the other internal node.

                          10)    Log in to the application and make sure the following profiles are set:

                          a.    Concurrent: PCP Instance Check set to ON at site level

                          b.    Concurrent: TM Transport Type set to QUEUE at site level

                          11)    Navigate to Concurrent|Manager|Define and make the following changes:

                          a.    Query for “Internal Monitor” and you’ll see there is an internal monitor for each node (there is a node mentioned in the name – i.e. “Internal Monitor: node1”).  Make sure the primary node field matches the node in the name.  Click “Work Shifts” button and select the Standard work shift and set Processes to 1.  Leave Secondary node field blank.

                          b.    Next query the “Internal Concurrent Manager”.  Set both primary and secondary nodes and make sure the Standard Work Shift is in place with 1 process.

                          c.    Next go through the rest of the enabled managers and set both primary and secondary nodes and make sure Standard Work Shift is in place with 1 process.

                          12)    Navigate to Concurrent|Manager|Administer and perform the following:

                          a.    Find the “Internal Monitor: <Node 1>” and click the “Activate” button.

                          b.    Find the “Internal Monitor: <Node 2>” and click the “Activate” button.

                          c.    Click “Refresh” button and wait for these two managers to start with 1 process each.

                          13)    Bounce concurrent managers on all nodes for changeds to take effect.

                          14)    You should see several FND* processes on Node 1.  And you should see FNDSM and FNDIMON started on node 2.

                           

                           

                          Regards

                          Prashant Umap

                          • 10. Re: Add node in R12.2
                            966638

                            Hi Prashant,

                             

                              PCP is working fine and no issue with CM level.

                             

                              Only issue with forms/web level and got the above error when try to start with forms managed server.

                            • 11. Re: Add node in R12.2
                              Prashant Umap

                              Hello,

                              Thanks for the post !

                               

                              /tmp/keystores1320796146xml.lck (Permission denied) at oracle.security.jps.internal.core.datastore.xml.FileLocker.lock(FileLocker.java:82)

                              Can you check if the user has permission to update the lock file.

                               

                              Regards

                              Prashant Umap

                              • 12. Re: Add node in R12.2
                                966638

                                Yes, I have given full permission

                                • 13. Re: Add node in R12.2
                                  Prashant Umap

                                  Hello,

                                  Sorry for late reply, have you got any resolution or still same error.

                                   

                                  Regards

                                  Prashant Umap

                                  • 14. Re: Add node in R12.2
                                    Prashant Umap

                                    Hello,

                                     

                                    Can you please try the following  -

                                    As per the MOS - Error Message: JPS-06514 Opening Of File Based Keystore Failed (Doc ID 1560188.1)

                                     

                                    1. Backup the current ${domain.home}/config/fmwconfig/keystores.xml and ${domain.home}/config/fmwconfig/bootstrap/cwallet.sso files.

                                    2. Delete or rename the ${domain.home}/config/fmwconfig/keystores.xml and ${domain.home}/config/fmwconfig/bootstrap/cwallet.sso files.

                                    3. Restart the AdminServer.

                                     

                                    MOS - WLS Admin Server Fails To Start With weblogic.security.SecurityInitializationException Error And BEA-090403 Authentication For User Weblogic Denied (Doc ID 2241778.1)

                                    1. Create a backup of the following folders:

                                    cp /applw010/admin/oam/user_projects/domains/oam_domain/security /applw010/admin/oam/user_projects/domains/oam_domain/security.20170306

                                    cp /applw010/admin/oam/user_projects/domains/oam_domain/servers/AdminServer/data /applw010/admin/oam/user_projects/domains/oam_domain/servers/AdminServer/data.20170306

                                     

                                    2. Creat the folder and file:

                                    /applw010/admin/oam/user_projects/domains/oam_domain/servers/AdminServer/security/boot.properties

                                    With the following content:

                                    username=weblogic

                                    password=XXXXXXX

                                     

                                    3. Starting Weblogic again gives us the error that the credentials in the boot.properties were incorrect.

                                     

                                    4. We can change the Weblogic admin password using the commands:

                                    cd /applw010/admin/oam/user_projects/domains/oam_domain/security

                                    . ../bin/setDomainEnv.sh

                                    java weblogic.security.utils.AdminAccount weblogic <password>

                                     

                                    5. Start the AdminServer again and this time it comes up without errors.

                                     

                                    Please share the result.

                                     

                                    Regards

                                    Prashant Umap

                                    1 2 Previous Next