7 Replies Latest reply: Jul 15, 2014 9:01 AM by Luz Mestre-Oracle RSS

    javax.net.ssl.SSLException: Inbound closed before receiving peer's close_notify: possible truncation attack?: Failed@starting 2nd Managed Server:

    ssine

      Hi, I' trying to complete of SSL Cert. implementation on Weblogic Servers as mentioned in this ticket :  Re: <Security> <BEA-090482> <BAD_CERTIFICATE - Managed Servers' NodeMngr is unreachable

       

      I've got 1 AdminServer with 2 Managed Servers. AdminServer and Mngd Server#1 are in the same server as normally and Managed Server#2 is on the another server. I tried to make as much as identical Mngd Server#1 with Mngd Server#2 in terms of nodemanager.properties, Ident/TrustKeystore files but it is failing while trying to start Managed Server#2.

       

      Managed Server#2/nodemanager.properties

      Mngd Server#2/nodemanager.properties file details are shown as below: (below details are same as with (working) Mngd Server#1)

      CrashRecoveryEnabled=true

      PropertiesVersion=10.3

       

      KeyStores=CustomIdentityAndCustomTrust

      CustomIdentityAlias=sslkey

      CustomIdentityKeyStoreFileName=/xxx/oracle/middleware/wlserver_10.3/server/lib/IdentKeystore.jks

      CustomIdentityKeyStoreType=JKS

      CustomIdentityKeyStorePassPhrase={3DES}xxx

      CustomIdentityPrivateKeyPassPhrase={3DES}xxx

       

      AuthenticationEnabled=true

      ListenAddress=

      ListenPort=5556

      SecureListener=true

      StartScriptEnabled=true

      CipherSuite=SSL_RSA_EXPORT_WITH_RC4_40_MD5  >> This parameter is added because our CSR is generated with "-sigalg MD5withRSA" parameter.

       

      Managed Server#2/IdentKeystore.jks, TrustKeystore.jks and JAVA/cacerts

      Also, I've copied and pasted working/correct IdentKeystore.jks, TrustKeystore.jks and JAVA/cacerts files from Mngd Server#1 to Mngd Server#2.

       

      Managed Server#2/startNodeManager.sh

      I added below lines into "/xxx/oracle/middleware/wlserver_10.3/server/bin/startNodeManager.sh"

      ...

      export PATH

      JAVA_OPTIONS="${JAVA_OPTIONS} -Dweblogic.nodemanager.sslHostNameVerificationEnabled=false -Dweblogic.security.SSL.enableJSSE=true -Dweblogic.security.SSL.enforceConstraints=off"

      export JAVA_OPTIONS

      cd "${NODEMGR_HOME}"

      ...

      Note: Weblogic GUI: Home >Summary of Machines > MngdServer2 > Monitoring Tab > Node Manager Status Tab > Status: Reachable / Version: 10.3

       

      Then, I've started Nodemanager from Mngd Server#2/CLI

       

      [oracle@t1a-smp-2 init.d]$ ps -ef | grep java

      oracle   25235 25203  1 14:15 ?        00:00:02 /usr/java/jdk160_24/bin/java -client -Xms32m -Xmx200m -XX:MaxPermSize=128m -Dcoherence.home=/opt/oracle/middleware/coherence_3.6 -Dbea.home=/xxx/oracle/middleware -Xverify:none -Dweblogic.nodemanager.sslHostNameVerificationEnabled=false -Dweblogic.security.SSL.enableJSSE=true -Dweblogic.security.SSL.enforceConstraints=off -Djava.security.policy=/xxx/oracle/middleware/wlserver_10.3/server/lib/weblogic.policy -Dweblogic.nodemanager.javaHome=/usr/java/jdk160_24 weblogic.NodeManager -v

      oracle   25736 25143  0 14:17 pts/2    00:00:00 grep java

       

      and after that, on Weblogic GUI, I pressed on "start" button of Managed Server#2 but it is Failed (FAILED_NOT_RESTARTABLE) with having below notification on "/opt/oracle/middleware/wlserver_10.3/common/nodemanager/nodemanager.log"

       

      <Jul 7, 2014 12:31:47 PM> <WARNING> <Uncaught exception in server handlerjavax.net.ssl.SSLException: Inbound closed before receiving peer's close_notify: possible truncation attack?>

      javax.net.ssl.SSLException: Inbound closed before receiving peer's close_notify: possible truncation attack?

             at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:190)

             at com.sun.net.ssl.internal.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1429)

             at com.sun.net.ssl.internal.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1397)

             at com.sun.net.ssl.internal.ssl.SSLEngineImpl.closeInbound(SSLEngineImpl.java:1336)

             at weblogic.security.SSL.jsseadapter.JaSSLEngine$7.run(JaSSLEngine.java:174)

             at weblogic.security.SSL.jsseadapter.JaSSLEngine.doAction(JaSSLEngine.java:732)

             at weblogic.security.SSL.jsseadapter.JaSSLEngine.closeInbound(JaSSLEngine.java:172)

             at weblogic.security.SSL.jsseadapter.JaSSLEngineRunner$Context.fillBufferNetIn(JaSSLEngineRunner.java:337)

             at weblogic.security.SSL.jsseadapter.JaSSLEngineRunner$Transition_NeedUnwrap.getNextState(JaSSLEngineRunner.java:822)

             at weblogic.security.SSL.jsseadapter.JaSSLEngineRunner.doTransitions(JaSSLEngineRunner.java:763)

             at weblogic.security.SSL.jsseadapter.JaSSLEngineRunner.unwrap(JaSSLEngineRunner.java:1122)

             at weblogic.security.SSL.jsseadapter.JaApplicationReadableByteChannel.read(JaApplicationReadableByteChannel.java:40)

             at weblogic.security.SSL.jsseadapter.JaChannelInputStream.read(JaChannelInputStream.java:71)

             at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:264)

             at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:306)

             at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158)

             at java.io.InputStreamReader.read(InputStreamReader.java:167)

             at java.io.BufferedReader.fill(BufferedReader.java:136)

             at java.io.BufferedReader.readLine(BufferedReader.java:299)

             at java.io.BufferedReader.readLine(BufferedReader.java:362)

             at weblogic.nodemanager.server.Handler.run(Handler.java:71)

             at java.lang.Thread.run(Thread.java:662)

       

      And lastly, I found below topic, is that the only way?

      SSL Exception within the Node Manager logs

      This is a known issue.

      Apply patch for BUG 13351178.

      https://updates.oracle.com/download/13351178.html

      Patches are available for WLS 1035 and 1036.

      Fixed Version : 12.1.2

       

      PS: Communication from AdminServer to Mngd Server#2 is with "SSL" not "PLAIN" but even trying with PLAIN mode in Mngd Server#2 is complaining about non-existence "of /opt/oracle/middleware/user_projects/domains/MotiveSMP/bin/startWebLogic.sh" shown as below:

      java.io.IOException: Executable /opt/oracle/middleware/user_projects/domains/MotiveSMP/bin/startWebLogic.sh does not exist

       

      Does anybody know how to resolve this issue? (without applying patches, just tweaking)

        • 1. Re: javax.net.ssl.SSLException: Inbound closed before receiving peer's close_notify: possible truncation attack?: Failed@starting 2nd Managed Server:
          Luz Mestre-Oracle

          Unfortunately, this issue does not have a workaround. You need to have the patch applied

          References:

          "javax.net.ssl.SSLException" or "java.io.IOException: Read channel closed" in Node Manager Logs (Doc ID 1644036.1)

          Best Regards

          Luz

          • 2. Re: javax.net.ssl.SSLException: Inbound closed before receiving peer's close_notify: possible truncation attack?: Failed@starting 2nd Managed Server:
            ssine

            Hi Mestre,

            thanks indeed for your reply.

            Questions:

            1. I clicked on the link that you provided but it is asking for "Support Identifier" / Request Access. How to find that Support Access contact in order to grant access or how/where to download mentioned Patch?

            2. After applying this patch, Mngd Server#2 will start without any issue, right? Or any other configuration (On Weblogic GUI or some *.properties file) is needed after applying this patch?

            3. Root Cause was generating and requesting CSR with using "-sigalg MD5withRSA" parameter? [mentioned as in:  Re: <Security> <BEA-090482> <BAD_CERTIFICATE - Managed Servers' NodeMngr is unreachable]

            • 3. Re: javax.net.ssl.SSLException: Inbound closed before receiving peer's close_notify: possible truncation attack?: Failed@starting 2nd Managed Server:
              Luz Mestre-Oracle

              You need to hava a active support contract to have access to the patches. If so, you just need to be logged to My Oracle Support. As per the document "These are harmless messages observed in Node Manager logs, because the remote server/client had already closed the connection before Node Manager closes it. The behavior has been corrected via unpublished defect 13351178.". Are you able to start the managed server using startManagedWebLogic script? (Thi will help to make sure there are no other issues around)

              Best Regards

              Luz


              • 4. Re: javax.net.ssl.SSLException: Inbound closed before receiving peer's close_notify: possible truncation attack?: Failed@starting 2nd Managed Server:
                ssine

                Hi Mestre,

                Are you able to start the managed server using startManagedWebLogic script? (Thi will help to make sure there are no other issues around)

                Here is the output from MngdServer#2, what do you think?


                [oracle@MngdServer#2 bin]$ ps -ef | grep java

                oracle    6093  5828  0 14:20 pts/2    00:00:00 grep java

                oracle   25235 25203  0 Jul10 ?        00:00:02 /usr/java/jdk160_24/bin/java -client -Xms32m -Xmx200m -XX:MaxPermSize=128m -Dcoherence.home=/xxx/oracle/middleware/coherence_3.6 -Dbea.home=/xxx/oracle/middleware -Xverify:none -Dweblogic.nodemanager.sslHostNameVerificationEnabled=false -Dweblogic.security.SSL.enableJSSE=true -Dweblogic.security.SSL.enforceConstraints=off -Djava.security.policy=/xxx/oracle/middleware/wlserver_10.3/server/lib/weblogic.policy -Dweblogic.nodemanager.javaHome=/usr/java/jdk160_24 weblogic.NodeManager -v

                 

                [oracle@MngdServer#2 bin]$ ./startManagedWebLogic.sh MngdServer#2 http://[IP]:[Port]

                CLASSPATH=/xxx/oracle/middleware/patch_wls1035/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/usr/java/jdk160_24/lib/tools.jar:/xxx/oracle/middleware/wlserver_10.3/server/lib/weblogic_sp.jar:/xxx/oracle/middleware/wlserver_10.3/server/lib/weblogic.jar:/xxx/oracle/middleware/modules/features/weblogic.server.modules_10.3.5.0.jar:/xxx/oracle/middleware/wlserver_10.3/server/lib/webservices.jar:/xxx/oracle/middleware/modules/org.apache.ant_1.7.1/lib/ant-all.jar:/xxx/oracle/middleware/modules/net.sf.antcontrib_1.1.0.0_1-0b2/lib/ant-contrib.jar::/xxx/oracle/middleware/wlserver_10.3/common/derby/lib/derbynet.jar:/xxx/oracle/middleware/wlserver_10.3/common/derby/lib/derbyclient.jar::/usr/java/jdk160_24/jre/lib/rt.jar:/xxx/oracle/middleware/wlserver_10.3/server/lib/webservices.jar:

                 

                PATH=/xxx/oracle/middleware/wlserver_10.3/server/bin:/xxx/oracle/middleware/modules/org.apache.ant_1.7.1/bin:/usr/java/jdk160_24/jre/bin:/usr/java/jdk160_24/bin:/usr/java/jdk160_24/bin:/usr/lib/oracle/11.2/client64/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/home/oracle/bin

                ***************************************************

                *  To start WebLogic Server, use a username and   *

                *  password assigned to an admin-level user.  For *

                *  server administration, use the WebLogic Server *

                *  console at http://<hostname>:<port>/console    *

                ***************************************************

                <Jul 11, 2014 2:07:51 PM UTC> <Info> <Security> <BEA-090905> <Disabling CryptoJ JCE Provider self-integrity check for better startup performance. To enable this check, specify -Dweblogic.security.allowCryptoJDefaultJCEVerification=true>

                <Jul 11, 2014 2:07:51 PM UTC> <Info> <Security> <BEA-090906> <Changing the default Random Number Generator in RSA CryptoJ from ECDRBG to FIPS186PRNG. To disable this change, specify -Dweblogic.security.allowCryptoJDefaultPRNG=true>

                <Jul 11, 2014 2:07:52 PM UTC> <Info> <WebLogicServer> <BEA-000377> <Starting WebLogic Server with Java HotSpot(TM) Client VM Version 19.1-b02 from Sun Microsystems Inc.>

                <Jul 11, 2014 2:07:52 PM UTC> <Info> <Security> <BEA-090065> <Getting boot identity from user.>

                Enter username to boot WebLogic server:

                Enter password to boot WebLogic server:

                <Jul 11, 2014 2:08:08 PM UTC> <Info> <Management> <BEA-141107> <Version: WebLogic Server 10.3.5.0  Fri Apr 1 20:20:06 PDT 2011 1398638 >

                <Jul 11, 2014 2:08:08 PM UTC> <Critical> <WebLogicServer> <BEA-000362> <Server failed. Reason:

                 

                There are 1 nested errors:

                weblogic.management.ManagementException: [Management:141247]The configuration directory /xxx/oracle/middleware/wlserver_10.3/common/bin/config does not exist and the admin server is not available.

                        at weblogic.management.provider.internal.RuntimeAccessImpl.parseNewStyleConfig(RuntimeAccessImpl.java:200)

                        at weblogic.management.provider.internal.RuntimeAccessImpl.<init>(RuntimeAccessImpl.java:115)

                        at weblogic.management.provider.internal.RuntimeAccessService.start(RuntimeAccessService.java:41)

                        at weblogic.t3.srvr.ServerServicesManager.startService(ServerServicesManager.java:461)

                        at weblogic.t3.srvr.ServerServicesManager.startInStandbyState(ServerServicesManager.java:166)

                        at weblogic.t3.srvr.T3Srvr.initializeStandby(T3Srvr.java:881)

                        at weblogic.t3.srvr.T3Srvr.startup(T3Srvr.java:568)

                        at weblogic.t3.srvr.T3Srvr.run(T3Srvr.java:469)

                        at weblogic.Server.main(Server.java:71)>

                <Jul 11, 2014 2:08:08 PM UTC> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FAILED>

                <Jul 11, 2014 2:08:08 PM UTC> <Error> <WebLogicServer> <BEA-000383> <A critical service failed. The server will shut itself down>

                <Jul 11, 2014 2:08:08 PM UTC> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FORCE_SHUTTING_DOWN>

                 

                 

                Is this sth related with boot.properties file? When comparing MngdServer#1 boot.pro file with MngdServer#2's... they are different? No idea, does it work, if I copy and past context of MngdServer#1's boot.pro file into MngdServer#2

                Or,

                Is this sth related with <listen-address></listen-address> parameter in config.xml file? (MngdServer#2 : /xxx/oracle/middleware/user_projects/domains/xxx/config/config.xml)


                Note: There is no AdminServer.lok file is existing in MngdServer#2

                • 5. Re: javax.net.ssl.SSLException: Inbound closed before receiving peer's close_notify: possible truncation attack?: Failed@starting 2nd Managed Server:
                  Luz Mestre-Oracle

                  Some Questions

                  Are you starting managed server in  <domain>/bin folder?

                  Did you use pack and unpack to move the domain from the first box?

                  Creating and Starting a Managed Server on a Remote Machine

                  Thanks

                  Luz

                  • 6. Re: javax.net.ssl.SSLException: Inbound closed before receiving peer's close_notify: possible truncation attack?: Failed@starting 2nd Managed Server:
                    ssine

                    Hi,

                    Are you starting managed server in  <domain>/bin folder? > Yes

                    Did you use pack and unpack to move the domain from the first box? > No (To be honest, I didnt do installation, so I don't know, but both Weblogic installations should be fresh installation)

                     

                    I've 2 servers: 1st one has Admin+MngdSrvr#1 and 2nd one has only Managed Server #2

                    • 7. Re: javax.net.ssl.SSLException: Inbound closed before receiving peer's close_notify: possible truncation attack?: Failed@starting 2nd Managed Server:
                      Luz Mestre-Oracle

                      Please confirm if pack and unpack was used, if it was not used this is won't work.

                      The error suggest that the second managed server i being started using

                      ./wlserver_10.3/common/bin/startManagedWebLogic.sh <managedServerName> <protocol>://<adminServerIPorHostame>:<adminServerPort>


                      After pack and unpack, managed server should be started as follows:


                      ./$DOMAIN_HOME/bin/startManagedWebLogic.sh <managedServerName> <protocol>://<adminServerIPorHostame>:<adminServerPort>


                      References:

                      weblogic.management.ManagementException: [Management:141247]The configuration directory /Middleware/config does not exist and the admin server is not available (Doc ID 1465312.1)weblogic.management.ManagementException: [Management:141247]The configuration directory /Middleware/config does not exist and the admin server is not available (Doc ID 1465312.1)

                       

                       

                      Best Regards

                      Luz