1 2 Previous Next 24 Replies Latest reply on Nov 21, 2019 10:07 AM by Martien van den Akker

    Setting up 2 Weblogic servers with 1 admin

    user11319095
      • New install on both servers
      • OS: Windows 2016
      • Weblogic Version: 12.2.1.3.0
      • Servers: 2, "app" server has weblogic admin installed, "batch" server is to be administered by app weblogic admin
      • Machines In weblogic Admin : 2, app and batch
      • Connectivity: The admin shows the batch machine as "reachable". There are no firewall issues and it connects and triggers log events over on the batch server's managed server. They are communicating.
      • Admin Running as a Windows service and runs as LocalHost
      • Nodemanagers running as a windows service on both app and batch.

       

      Issue: When I try to start or stop the managed server on the batch server from the admin installed on the app server I get security errors (see below). Seems I have 2 issues. The managed server credentials in boot.properties cannot be properly de-crypted and the batch server is unable to connect back to the admin. What step(s) am I missing on the configuration of the batch server so that I can administer it from the weblogic admin on the app server?

       

      What I've tried: Changed the encrypted attribute to its clear text value. This did not fix the issues.

       

      Logs (redacted) in the batch server after trying to START the managed server:

      <Sep 18, 2019 3:01:41 PM PDT> <Info> <Security> <BEA-090905> <Disabling the CryptoJ JCE Provider self-integrity check for better startup performance. To enable this check, specify -Dweblogic.security.allowCryptoJDefaultJCEVerification=true.>

      <Sep 18, 2019 3:01:41 PM PDT> <Info> <Security> <BEA-090906> <Changing the default Random Number Generator in RSA CryptoJ from redacted to redacted. To disable this change, specify -Dweblogic.security.allowCryptoJDefaultPRNG=true.>

      <Sep 18, 2019 3:01:42 PM PDT> <Info> <WebLogicServer> <BEA-000377> <Starting WebLogic Server with Java HotSpot(TM) 64-Bit Server VM Version 25.201-b09 from Oracle Corporation.>

      <Sep 18, 2019 3:01:42 PM PDT> <Info> <RCM> <BEA-2165021> <"ResourceManagement" is not enabled in this JVM. Enable "ResourceManagement" to use the WebLogic Server "Resource Consumption Management" feature. To enable "ResourceManagement", you must specify the following JVM options in the WebLogic Server instance in which the JVM runs: -XX:+UnlockCommercialFeatures -XX:+ResourceManagement.>

      <Sep 18, 2019 3:01:42 PM PDT> <Critical> <Security> <BEA-090518> <Could not decrypt the username attribute value of {AES}Ogs+yyKo2MNDpRW7redacted from the file D:\oracle\middleware\user_projects\domains\s4devenvironment\servers\s4dev-batch01\data\nodemanager\boot.properties. If an encrypted attribute was copied from boot.properties from another domain into D:\oracle\middleware\user_projects\domains\s4devenvironment\servers\s4dev-batch01\data\nodemanager\boot.properties, change the encrypted attribute to its clear text value, and then restart the server. The attribute will be encrypted again. Otherwise, change all encrypted attributes to their clear text values, then restart the server. All encryptable attributes will be encrypted again. The decryption failed with the exception weblogic.security.internal.encryption.EncryptionServiceException: com.rsa.jsafe.JSAFE_PaddingException: Invalid padding..>

      <Sep 18, 2019 3:01:42 PM PDT> <Critical> <Security> <BEA-090518> <Could not decrypt the password attribute value of {AES}5I2ZwkJ+o31KrZ8dredacted from the file D:\oracle\middleware\user_projects\domains\s4devenvironment\servers\s4dev-batch01\data\nodemanager\boot.properties. If an encrypted attribute was copied from boot.properties from another domain into D:\oracle\middleware\user_projects\domains\s4devenvironment\servers\s4dev-batch01\data\nodemanager\boot.properties, change the encrypted attribute to its clear text value, and then restart the server. The attribute will be encrypted again. Otherwise, change all encrypted attributes to their clear text values, then restart the server. All encryptable attributes will be encrypted again. The decryption failed with the exception weblogic.security.internal.encryption.EncryptionServiceException: com.rsa.jsafe.JSAFE_PaddingException: Invalid padding..>

      Enter username to boot WebLogic server:<Sep 18, 2019 3:01:43 PM PDT> <Info> <Management> <BEA-141307> <Unable to connect to the Administration Server. Waiting 5 second(s) to retry (attempt number 1 of 3).>

      <Sep 18, 2019 3:01:48 PM PDT> <Info> <Management> <BEA-141307> <Unable to connect to the Administration Server. Waiting 5 second(s) to retry (attempt number 2 of 3).>

      <Sep 18, 2019 3:01:53 PM PDT> <Info> <Management> <BEA-141307> <Unable to connect to the Administration Server. Waiting 5 second(s) to retry (attempt number 3 of 3).>

      <Sep 18, 2019 3:01:58 PM PDT> <Info> <Management> <BEA-141298> <Could not register with the Administration Server: java.rmi.RemoteException: [Deployer:149150]An IOException occurred while reading the input. : with response code '401' : with response message 'Unauthorized'>

      <Sep 18, 2019 3:01:58 PM PDT> <Info> <Management> <BEA-141107> <Version: WebLogic Server 12.2.1.3.0 Thu Aug 17 13:39:49 PDT 2017 1882952>

      <Sep 18, 2019 3:01:58 PM PDT> <Error> <Configuration Management> <BEA-150021> <The Administration Server failed to authenticate the identity of the user  starting the Managed Server. The reason for the error is .>

      <Sep 18, 2019 3:01:58 PM PDT> <Alert> <Management> <BEA-141151> <The Administration Server could not be reached at http://redacted:7001.>

      <Sep 18, 2019 3:01:58 PM PDT> <Info> <Configuration Management> <BEA-150018> <This server is being started in Managed Server independence mode in the absence of the Administration Server.>

      <Sep 18, 2019 3:01:58 PM PDT> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING.>

      <Sep 18, 2019 3:01:58 PM PDT> <Info> <WorkManager> <BEA-002900> <Initializing self-tuning thread pool.>

      <Sep 18, 2019 3:01:58 PM PDT> <Info> <WorkManager> <BEA-002942> <CMM memory level becomes 0. Setting standby thread pool size to 256.>

      <Sep 18, 2019 3:01:59,474 PM PDT> <Notice> <Log Management> <BEA-170019> <The server log file weblogic.logging.FileStreamHandler instance=958052832

      Current log file=D:\oracle\middleware\user_projects\domains\s4devenvironment\servers\s4dev-batch01\logs\s4dev-batch01.log

      Rotation dir=D:\oracle\middleware\user_projects\domains\s4devenvironment\servers\s4dev-batch01\logs

      is opened. All server side log events will be written to this file.>

      <Sep 18, 2019 3:01:59,627 PM PDT> <Notice> <Security> <BEA-090946> <Security pre-initializing using security realm: myrealm>

      <Sep 18, 2019 3:02:00,102 PM PDT> <Notice> <Security> <BEA-090947> <Security post-initializing using security realm: myrealm>

      <Sep 18, 2019 3:02:00,717 PM PDT> <Notice> <Security> <BEA-090082> <Security initialized using administrative security realm: myrealm>

      <Sep 18, 2019 3:02:00,734 PM PDT> <Critical> <Security> <BEA-090402> <Authentication denied: Boot identity not valid. The user name or password or both from the boot identity file (boot.properties) is not valid. The boot identity may have been changed since the boot identity file was created. Please edit and update the boot identity file with the proper values of username and password. The first time the updated boot identity file is used to start the server, these new values are encrypted.>

      <Sep 1<Sep 18, 2019 3:02:02 PM PDT> <INFO> <NodeManager> <The server 's4dev-batch01' with process id null is no longer alive; waiting for the process to die.>

        • 1. Re: Setting up 2 Weblogic servers with 1 admin
          Martien van den Akker

          Hi,

           

          In the domain folder from where the particular server is starting up, there is a servers folder. In that there is a folder with the name of the server, and that contains a security folder. In short $DOMAIN_HOME/servers/$SERVER_NAME/security. In that folder you may find a boot.properties file. It probably contains faulty credentials.

          Edit it and set the admin username password in plain text, like:

           

          username=weblogic

          password=Welcome1

          At first startup the properties will be encrypted.

           

          Regards,
          Martien

          • 2. Re: Setting up 2 Weblogic servers with 1 admin
            user11319095

            Thank you for the recommendation. Per what I wrote in the section "What I've tried" above - I've already tried that and it did not change anything.

            • 3. Re: Setting up 2 Weblogic servers with 1 admin
              Martien van den Akker

              Oh, excuse me: I've overlooked that. I think I've seen that, but it did not land.

               

              How did you create/clone the domain on the batch-server-side?

              If you remove the boot.properties file, and start the batchserver using the startWeblogic script, it will ask for a username password. If you use the username password that you also use to logon to the admin-console. Will it then start?

               

              Regards,
              Martien

              • 4. Re: Setting up 2 Weblogic servers with 1 admin
                Henk van Dorp

                Hi,

                I do have exact the same problem. I'm using WebLogic 12.2.1.3 on redhat 7.6 I've created with WLST an empty Weblogic Domain on node_A (AdminServer, nodemanager, cluster, ManagedServers). The servers on node_A all start and are manageble in the webconsole.

                Now I've pack/unpack the domain to node_B. The nodemanager starts and communicates with the AdminServer. But when I start the ManagedServer i'm getting the same errors as shown above.

                 

                <14-nov-2019 12:52:40 uur UTC> <Critical> <Security> <BEA-090518> <Could not decrypt the username attribute value of {AES}yatZ2aFCUCb8MPZ536Wy1TKuda7gDMQglmcWNseyBKU= from the file /opt/wl/domains/HVD_DEV_Domain/servers/HVD_DEV_appl01_Managed04_Server/data/nodemanager/boot.properties. If an encrypted attribute was copied from boot.properties from another domain into /opt/wl/domains/HVD_DEV_Domain/servers/HVD_DEV_appl01_Managed04_Server/data/nodemanager/boot.properties, change the encrypted attribute to its clear text value, and then restart the server. The attribute will be encrypted again. Otherwise, change all encrypted attributes to their clear text values, then restart the server. All encryptable attributes will be encrypted again. The decryption failed with the exception weblogic.security.internal.encryption.EncryptionServiceException: com.rsa.jsafe.JSAFE_PaddingException: Invalid padding..>

                 

                Since it can't decrypt the username and password, the next error is understandable:

                 

                 

                <14-nov-2019 12:52:55 uur UTC> <Info> <Management> <BEA-141298> <Could not register with the Administration Server: java.rmi.RemoteException: [Deployer:149150]An IOException occurred while reading the input. : with response code '401' : with response message 'Unauthorized'>

                 

                 

                Also I retried by editing the boot.properties on Node_B and restart the managed server, but there is no change in the kind of error except for a different encryption of username and password.
                And I've succesfully connected to the AdminServer from Node_B using WLST interactively with the connect() command. So connectivity is not the problem.

                 

                Does anybody have a clue?

                Help is very appreciated

                 

                Thanks in advance

                 

                Henk

                • 5. Re: Setting up 2 Weblogic servers with 1 admin
                  Martien van den Akker

                  Again. If you remove the boot.properties, and start the weblogic server on NodeB with the startWeblogic.sh script (so not through the nodemanager), it will ask for the boot credentials. If you supply the admin user + password known by you, will it then start?

                  If so, create a new boot.properties file with that ussername password. Restart the server again with the script (so again not through nodemanager), will it then start without asking for credentials?

                   

                  Kind regards,

                  Martien

                  • 6. Re: Setting up 2 Weblogic servers with 1 admin
                    Martien van den Akker

                    Oh, and you could also remove the boot.properties at all and then try to start it using the nodemanager. I think the nodemanager does not need a boot.properties to start the managed server.

                     

                    Kind regards,
                    Martien

                    • 7. Re: Setting up 2 Weblogic servers with 1 admin
                      Henk van Dorp

                      Hi Martien,

                       

                      Thank you for your respons.

                      I did not see any command it did work after your recommendations, while the issue had status "Not answerd" so I assumed it was not solved yet.

                       

                      I've renamed the boot.properties file and restart it using the nodemanager. The issue stays the same. It created a new boot.properties with different encryption data for username and password.

                       

                      If I use the startWeblogic script, it starts thej adminserver on node_B which is not what I want

                       

                      I used on node_B startManagedWebLogic.sh now (I guess you ment that script). After starting it on port 7002, it prompted for username and password, the server seems to hang.

                      If I start it on port 7001 the server crashes on trying to download on an empty file list.

                       

                      <Nov 14, 2019 2:25:02 PM UTC> <Info> <Security> <BEA-090065> <Getting boot identity from user.>

                      Enter username to boot WebLogic server:system

                      Enter password to boot WebLogic server:

                      java.io.IOException: [DeploymentService:290066]Error occurred while downloading files from Administration Server for deployment request "9,609,795,23179,795". Underlying error is: "[DeploymentService:290065]Deployment service servlet encountered an Exception while handling the deployment datatransfemessage for request id "9,609,795,231,379,795" from server "HVD_DEV_appl01_Managed04_Server". Exception is: "files list is empty"."

                              at weblogic.deploy.service.datatransferhandlers.HttpDataTransferHandler.getDataAsStream(HttpDataTransferHandler.java:92)

                              at weblogic.deploy.service.datatransferhandlers.DataHandlerManager$RemoteDataTransferHandler.getDataAsStream(DataHandlerManager.java:175)

                              at weblogic.deploy.internal.targetserver.datamanagement.ConfigDataUpdate.doDownload(ConfigDataUpdate.java:153)

                       

                       

                       

                       

                       

                       

                      with regards

                      Henk

                      • 8. Re: Setting up 2 Weblogic servers with 1 admin
                        Martien van den Akker

                        Hi Henk,

                        'I did not see any command it did work after your recommendations, while the issue had status "Not answerd" so I assumed it was not solved yet.' I don't understand what this means.

                         

                        I indeed meant that script, although if you use startWeblogic.sh from the bin folder (not the one in the root of the Domain home), then it will call the startManagedWebLogic.sh .

                        For the record, you should call it like startManagedWebLogic.sh adminServerHost:adminServerPort nameOfYourManagedServer. (You probably did)

                         

                        I don't recognize this error. It's quite strange. But it is interesting, because it seems that it succeeds in connecting and authenticating against the AdminServer. But it cannot download the deployment files. You could check by deliberately provide a faulty-password. If that gives an unauthorized error then we're closing in.

                         

                        By the way, when you did an unpack of the domain: did it contain already a server/managed-server-B folder? If so, try again after removing the complete folder.

                         

                        Kind regards,
                        Martien

                         

                        .

                        • 9. Re: Setting up 2 Weblogic servers with 1 admin
                          Henk van Dorp

                          Hi Martien,

                           

                          What I did found on Oracle support was https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=35293330270602&id=2589468.1&displayIndex=7&_afrWindowMod…

                          it refers to WebLogic 12.2.1.3, although I found issues with it back to V10.3.6.

                          Next week I will dive in to this further and will let you know the results.

                          For now the managedserver on Node_B gets it's connection to the AdminServer on node_A. I started startManagedWebLogic.sh <ManagedServername> https://<Adminserver_nodename>:7002 as it is mentioned this way in the script itself.

                           

                          I regularly remove the entire domain and build it up by my scripts after hacking and trying, just to be sure i've fixed things
                          I believe that the directories of ManagedServer on node_B are created as soon as you start it and before it tries to connect to the adminserver, but I'm not 100% sure of that. i'll take a peek next time

                           

                          I'll keep you updated. Have a nice weekend

                           

                          with regards

                           

                          Henk

                          • 10. Re: Setting up 2 Weblogic servers with 1 admin
                            Martien van den Akker

                            Hi Henk,

                             

                            I believe I saw that one too. This is specifically around the SerializedSystemIni.dat file. I did not get the impression that it is the same as in this thread.

                             

                            Kind regards,
                            Martien

                            • 11. Re: Setting up 2 Weblogic servers with 1 admin
                              Henk van Dorp

                              Hi Martien,

                               

                              indeed it did not work. Here is the summary I did:

                              • Create a domain/adminserver, added 2 nodemanagers, one cluster and 2 managedServers. Configuration is over two (virtual) nodes
                              • Added -Dweblogic.data.canTransferAnyFile=true in the arguments of the managedservers;
                              • pack / unpack the domain (no weblogic processes running)
                              • start adminserver and nodemanagers

                               

                              While the local ManagedServer starts without any problems, the remote ManagedServer is stumbling over the decryption of the username and its password:

                               

                              <18-nov-2019 10:13:00 uur UTC> <Critical> <Security> <BEA-090518> <Could not decrypt the username attribute value of {AES}gz+jOXr/A5mtVMoLHj25UYbRzFGj2ph+ssksTXky9lU= from the file /opt/wl/domains/HVD_DEV_Domain/servers/HVD_DEV_appl01_Managed04_Server/data/nodemanager/boot.properties. If an encrypted attribute was copied from boot.properties from another domain into /opt/wl/domains/HVD_DEV_Domain/servers/HVD_DEV_appl01_Managed04_Server/data/nodemanager/boot.properties, change the encrypted attribute to its clear text value, and then restart the server. The attribute will be encrypted again. Otherwise, change all encrypted attributes to their clear text values, then restart the server. All encryptable attributes will be encrypted again. The decryption failed with the exception weblogic.security.internal.encryption.EncryptionServiceException: com.rsa.jsafe.JSAFE_PaddingException: Invalid padding..>

                               

                              <18-nov-2019 10:13:00 uur UTC> <Critical> <Security> <BEA-090518> <Could not decrypt the password attribute value of {AES}h8YKmNVSn7++S0//v0bl3HAwyFX3dDKL/VroStD9p4PK7aLrVt0WLfc4h3saO714 from the file /opt/wl/domains/HVD_DEV_Domain/servers/HVD_DEV_appl01_Managed04_Server/data/nodemanager/boot.properties. If an encrypted attribute was copied from boot.properties from another domain into /opt/wl/domains/HVD_DEV_Domain/servers/HVD_DEV_appl01_Managed04_Server/data/nodemanager/boot.properties, change the encrypted attribute to its clear text value, and then restart the server. The attribute will be encrypted again. Otherwise, change all encrypted attributes to their clear text values, then restart the server. All encryptable attributes will be encrypted again. The decryption failed with the exception weblogic.security.internal.encryption.EncryptionServiceException: com.rsa.jsafe.JSAFE_PaddingException: Invalid padding..>

                               

                               

                               

                              Enter username to boot WebLogic server:<18-nov-2019 8:46:11 uur UTC> <Info> <Management> <BEA-141307> <Unable to connect to the Administration Server. Waiting 5 second(s) to retry (attempt number 1 of 3).>

                               

                              On the adminserver you see the failed login attempts:

                               

                              ####<Nov 18, 2019 10:12:57,595 AM UTC> <Warning> <NodeManager> <dev002wlsl03.keylanehosting.local> <HVD_DEV_AdminServer> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <system> <> <4c627bae-3838-4296-914b-b3a5b4dd1d14-0000007c> <1574071977595> <[severity-value: 16] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-300057> <Starting server HVD_DEV_appl01_Managed04_Server on machine dev002wlsl04Machine at dev002wlsl04:7005>

                               

                              ####<Nov 18, 2019 10:13:00,458 AM UTC> <Warning> <DeploymentService> <dev002wlsl03.keylanehosting.local> <HVD_DEV_AdminServer> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <> <4c627bae-3838-4296-914b-b3a5b4dd1d14-0000007d> <1574071980458> <[severity-value: 16] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-290014> <Invalid user name or password.>

                              ####<Nov 18, 2019 10:13:05,141 AM UTC> <Warning> <Socket> <dev002wlsl03.keylanehosting.local> <HVD_DEV_AdminServer> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <4c627bae-3838-4296-914b-b3a5b4dd1d14-0000007e> <1574071985141> <[severity-value: 16] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-000449> <Closing the socket, as no data read from it on 10.53.0.42:55,045 during the configured idle timeout of 5 seconds.>

                              ####<Nov 18, 2019 10:13:06,194 AM UTC> <Warning> <DeploymentService> <dev002wlsl03.keylanehosting.local> <HVD_DEV_AdminServer> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <> <4c627bae-3838-4296-914b-b3a5b4dd1d14-0000007f> <1574071986194> <[severity-value: 16] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-290014> <Invalid user name or password.>

                              ####<Nov 18, 2019 10:13:11,199 AM UTC> <Warning> <DeploymentService> <dev002wlsl03.keylanehosting.local> <HVD_DEV_AdminServer> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <> <4c627bae-3838-4296-914b-b3a5b4dd1d14-00000080> <1574071991199> <[severity-value: 16] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-290014> <Invalid user name or password.>

                              ####<Nov 18, 2019 10:13:16,205 AM UTC> <Warning> <DeploymentService> <dev002wlsl03.keylanehosting.local> <HVD_DEV_AdminServer> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <> <4c627bae-3838-4296-914b-b3a5b4dd1d14-00000081> <1574071996205> <[severity-value: 16] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-290014> <Invalid user name or password.>

                              ####<Nov 18, 2019 10:13:16,261 AM UTC> <Error> <Configuration Management> <dev002wlsl03.keylanehosting.local> <HVD_DEV_AdminServer> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <4c627bae-3838-4296-914b-b3a5b4dd1d14-00000082> <1574071996261> <[severity-value: 8] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-150035> <An attempt was made to download the configuration for the server HVD_DEV_appl01_Managed04_Server by the user  with an invalid password.>

                               

                              So there is a connection between the remote managedserver and the adminserver. But due to failing to decrypt username and password, leaves probably the vaiables void/empty/uninitialised but are used to make the connection to the adminserver. It is logical that the login attempts on the adminserver fails.

                              -------------------------------------------------------------------------------------------------------------------------------------

                               

                              If I start the managedServer with the command $./startManagedWebLogic.sh HVD_DEV_appl01_Managed04_Server  https://dev002wlsl03.keylanehosting.local:7002 it seems to login to the adminserver, but runs into other issues:

                               

                              Managedserver log:

                              ================

                              Enter username to boot WebLogic server:system

                              Enter password to boot WebLogic server:

                              <Nov 18, 2019 10:24:04 AM UTC> <Warning> <Security> <BEA-090960> <The servers SSL configuration is not available. There will potentially be SSL handshake failures.>

                              <Nov 18, 2019 10:24:04 AM UTC> <Warning> <Security> <BEA-090924> <JSSE has been selected by default, since the SSLMBean is not available.>

                              <Nov 18, 2019 10:24:04 AM UTC> <Info> <Security> <BEA-090908> <Using the default WebLogic SSL Hostname Verifier implementation.>

                              <Nov 18, 2019 10:24:05 AM UTC> <Notice> <Security> <BEA-090169> <Loading trusted certificates from the jks keystore file /opt/wl/wl12213/wlserver/server/lib/DemoTrust.jks.>

                              <Nov 18, 2019 10:24:05 AM UTC> <Notice> <Security> <BEA-090169> <Loading trusted certificates from the jks keystore file /opt/jdk/jdk1.8.0_212/jre/lib/security/cacerts.>

                              <Nov 18, 2019 10:24:05 AM UTC> <Info> <Management> <BEA-141107> <Version: WebLogic Server 12.2.1.3.0 Thu Aug 17 13:39:49 PDT 2017 1882952>

                              <Nov 18, 2019 10:24:05 AM UTC> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING.>

                              <Nov 18, 2019 10:24:05 AM UTC> <Info> <WorkManager> <BEA-002900> <Initializing self-tuning thread pool.>

                              <Nov 18, 2019 10:24:05 AM UTC> <Info> <WorkManager> <BEA-002942> <CMM memory level becomes 0. Setting standby thread pool size to 256.>

                              <Nov 18, 2019 10:24:07,039 AM UTC> <Notice> <Security> <BEA-090171> <Loading the identity certificate and private key stored under the alias DemoIdentity from the jks keystore file /opt/wl/domains/HVD_DEV_Domain/security/DemoIdentity.jks.>

                              <Nov 18, 2019 10:24:07,042 AM UTC> <Notice> <Security> <BEA-090169> <Loading trusted certificates from the jks keystore file /opt/wl/wl12213/wlserver/server/lib/DemoTrust.jks.>

                              <Nov 18, 2019 10:24:07,042 AM UTC> <Notice> <Security> <BEA-090169> <Loading trusted certificates from the jks keystore file /opt/jdk/jdk1.8.0_212/jre/lib/security/cacerts.>

                              <Nov 18, 2019 10:24:07,055 AM UTC> <Notice> <Security> <BEA-090169> <Loading trusted certificates from the jks keystore file /opt/wl/wl12213/wlserver/server/lib/DemoTrust.jks.>

                              <Nov 18, 2019 10:24:07,055 AM UTC> <Notice> <Security> <BEA-090169> <Loading trusted certificates from the jks keystore file /opt/jdk/jdk1.8.0_212/jre/lib/security/cacerts.>

                              <Nov 18, 2019 10:24:07,087 AM UTC> <Notice> <Log Management> <BEA-170019> <The server log file weblogic.logging.FileStreamHandler instance=575575380

                              Current log file=/var/log/weblogic/HVD_DEV_appl01_Cluster/HVD_DEV_appl01_Managed04_Server.log

                              Rotation dir=/var/log/weblogic/HVD_DEV_appl01_Cluster

                              is opened. All server side log events will be written to this file.>

                              <Nov 18, 2019 10:24:07,209 AM UTC> <Notice> <Security> <BEA-090169> <Loading trusted certificates from the jks keystore file /opt/wl/wl12213/wlserver/server/lib/DemoTrust.jks.>

                              <Nov 18, 2019 10:24:07,276 AM UTC> <Notice> <Security> <BEA-090169> <Loading trusted certificates from the jks keystore file /opt/jdk/jdk1.8.0_212/jre/lib/security/cacerts.>

                              <Nov 18, 2019 10:24:07,343 AM UTC> <Notice> <Security> <BEA-090169> <Loading trusted certificates from the jks keystore file /opt/wl/wl12213/wlserver/server/lib/DemoTrust.jks.>

                              <Nov 18, 2019 10:24:07,349 AM UTC> <Notice> <Security> <BEA-090169> <Loading trusted certificates from the jks keystore file /opt/jdk/jdk1.8.0_212/jre/lib/security/cacerts.>

                              <Nov 18, 2019 10:24:07,384 AM UTC> <Warning> <EmbeddedLDAP> <BEA-171529> <Unable to download the initial replica of security data from the AdminServer. Verify that clock skew has not occurred between the servers in the domain. The underlying exception was java.io.IOException: Multiple failures to read VDE replica, network issues suspected>

                              <Nov 18, 2019 10:24:07,506 AM UTC> <Notice> <Security> <BEA-090946> <Security pre-initializing using security realm: myrealm>

                              <Nov 18, 2019 10:24:07,873 AM UTC> <Notice> <Security> <BEA-090947> <Security post-initializing using security realm: myrealm>

                              <Nov 18, 2019 10:24:08,296 AM UTC> <Warning> <Security> <BEA-090076> <A failure occurred attempting to load LDIF for provider Authorizer from file /opt/wl/wl12213/wlserver/server/lib/XACMLAuthorizerInit.ldift.>

                              <Nov 18, 2019 10:24:08,297 AM UTC> <Error> <Security> <BEA-090870> <The realm "myrealm" failed to be loaded: weblogic.security.service.SecurityServiceException: com.bea.common.engine.ServiceInitializationException: weblogic.security.spi.ProviderInitializationException: A failure occurred attempting to load LDIF for provider Authorizer from file /opt/wl/wl12213/wlserver/server/lib/XACMLAuthorizerInit.ldift..

                              weblogic.security.service.SecurityServiceException: com.bea.common.engine.ServiceInitializationException: weblogic.security.spi.ProviderInitializationException: A failure occurred attempting to load LDIF for provider Authorizer from file /opt/wl/wl12213/wlserver/server/lib/XACMLAuthorizerInit.ldift.

                               

                              Whereafter it crashes and stops.

                               

                               

                               

                               

                              In the Adminserver log:

                              ==================

                               

                              ####<Nov 18, 2019 10:24:05,461 AM UTC> <Warning> <DeploymentService> <dev002wlsl03.keylanehosting.local> <HVD_DEV_AdminServer> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <system> <> <4c627bae-3838-4296-914b-b3a5b4dd1d14-0000008e> <1574072645461> <[severity-value: 16] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-290074> <Deployment service servlet received file download request for file "security/SerializedSystemIni.dat". The file may exist, but automatic download of this file is not allowed; it must be manually transferred.>

                              ####<Nov 18, 2019 10:24:07,169 AM UTC> <Error> <Management> <dev002wlsl03.keylanehosting.local> <HVD_DEV_AdminServer> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <> <4c627bae-3838-4296-914b-b3a5b4dd1d14-00000090> <1574072647169> <[severity-value: 8] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-141149> <An invalid attempt was made to connect to the Administration Server with a salt of /LJ7YAH5jM58hQAaHg9xxA== and a signature of ge5j0IcHhn4pIHq9azL7MQ92yzAyUpZjX8+nNINRZ7Y=, likely due to private key mismatch or clock skew between servers.>

                              ####<Nov 18, 2019 10:24:07,313 AM UTC> <Error> <Management> <dev002wlsl03.keylanehosting.local> <HVD_DEV_AdminServer> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <> <4c627bae-3838-4296-914b-b3a5b4dd1d14-00000091> <1574072647313> <[severity-value: 8] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-141149> <An invalid attempt was made to connect to the Administration Server with a salt of DmiytJuQiqGysDJ5SommeA== and a signature of LfMsemj2Ai/oYI8mZ622agTg64UKwf1hhKxUcqeDdVM=, likely due to private key mismatch or clock skew between servers.>

                              ####<Nov 18, 2019 10:24:07,372 AM UTC> <Error> <Management> <dev002wlsl03.keylanehosting.local> <HVD_DEV_AdminServer> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <> <065348b9-bad5-4244-8b4f-30415e8b3381-00000001> <1574072647372> <[severity-value: 8] [rid: 0:1] [partition-id: 0] [partition-name: DOMAIN] > <BEA-141149> <An invalid attempt was made to connect to the Administration Server with a salt of GGBwi8dtazNLk/fnNnx/jQ== and a signature of J1Ps3lttTtemRsDQG+tM4u8iIP06bVBXEHr8dJzrSNo=, likely due to private key mismatch or clock skew between servers.>

                              ####<Nov 18, 2019 10:24:08,458 AM UTC> <Error> <Security> <dev002wlsl03.keylanehosting.local> <HVD_DEV_AdminServer> <ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'> <<WLS Kernel>> <> <4c627bae-3838-4296-914b-b3a5b4dd1d14-00000092> <1574072648458> <[severity-value: 8] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-090513> <ServerIdentity failed validation, downgrading to anonymous.>

                              ####<Nov 18, 2019 10:24:08,458 AM UTC> <Error> <Security> <dev002wlsl03.keylanehosting.local> <HVD_DEV_AdminServer> <ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'> <<WLS Kernel>> <> <4c627bae-3838-4296-914b-b3a5b4dd1d14-00000092> <1574072648458> <[severity-value: 8] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-090513> <ServerIdentity failed validation, downgrading to anonymous.>

                              ####<Nov 18, 2019 10:24:08,458 AM UTC> <Error> <Security> <dev002wlsl03.keylanehosting.local> <HVD_DEV_AdminServer> <ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'> <<WLS Kernel>> <> <4c627bae-3838-4296-914b-b3a5b4dd1d14-00000092> <1574072648458> <[severity-value: 8] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-090513> <ServerIdentity failed validation, downgrading to anonymous.>

                              • 12. Re: Setting up 2 Weblogic servers with 1 admin
                                Martien van den Akker

                                Hi,

                                 

                                I also found a hint here https://www.databasedevelop.com/article/12464306/Error+occurred+while+downloading+files+from+Administration+Server... : that one or the other may be due to a Host Intruder Prevention System (HIPS) agent, or any other agent that prevents downloading.

                                 

                                However, now this error is interesting: '<BEA-171529> <Unable to download the initial replica of security data from the AdminServer. Verify that clock skew has not occurred between the servers in the domain. The underlying exception was java.io.IOException: Multiple failures to read VDE replica, network issues suspected>'.

                                Found this by looking up the BEA error in support: https://support.oracle.com/epmos/faces/DocumentDisplay?id=2400988.1

                                This seems due (as in the error) to the difference in clock between the two hosts. Could you check that and maybe implement network clock synchronization? Also you could try to manually copy the SerializedSystemIni.dat file.

                                 

                                Kind regards,

                                Martien

                                • 13. Re: Setting up 2 Weblogic servers with 1 admin
                                  Henk van Dorp

                                  Martien van den Akker

                                  Hi Martien,

                                   

                                  i've checked the timedifference between the machines and it is less than 1 ms. I cant imagine that this would be an issue.

                                   

                                  I've done a md5sum on the SerializedSystemIni.dat and it was different. (it might be updated in time??) but it is delivered with the Weblogic domain.jar with packing

                                  However I copied the file to the remote system and started the nodemanager again. But then the nodemanager does not seem to be able to communicate with the AdminServer anymore (you can check that in the machines - Monitor tab)

                                  After unpacking the jar-file again, placing the original SerializedSystemIni.dat back, the nodemanager communicates again with the AdminServer, but the start of the ManagedServer still failed by decrypting username and password.

                                   

                                  kind regards

                                   

                                  Henk

                                  • 14. Re: Setting up 2 Weblogic servers with 1 admin
                                    Martien van den Akker

                                    Hi,

                                     

                                    I guess you should create a Service Request. I haven't seen anything like this in the past.

                                     

                                    I see by the way that the message I mentioned is actually a warning. The actual error is:

                                    <BEA-090870> <The realm "myrealm" failed to be loaded: weblogic.security.service.SecurityServiceException: com.bea.common.engine.ServiceInitializationException: weblogic.security.spi.ProviderInitializationException: A failure occurred attempting to load LDIF for provider Authorizer from file /opt/wl/wl12213/wlserver/server/lib/XACMLAuthorizerInit.ldift..

                                    weblogic.security.service.SecurityServiceException: com.bea.common.engine.ServiceInitializationException: weblogic.security.spi.ProviderInitializationException: A failure occurred attempting to load LDIF for provider Authorizer from file /opt/wl/wl12213/wlserver/server/lib/XACMLAuthorizerInit.ldift.

                                    Based on BEA-090870 I found this one: https://support.oracle.com/epmos/faces/DocumentDisplay?id=1327167.1

                                     

                                    Do you perhaps use a RDBMS based realm?

                                     

                                    Kind regards,
                                    Martien

                                    1 2 Previous Next