12 Replies Latest reply: Mar 20, 2011 11:51 PM by radiatejava RSS

    RMI over SSL - handshake failure

    radiatejava
      I am having a hard time in making my RMI over SSL work. Certificates etc are all fine. I enabled ssl debug but I am not able to get any conclusive reason. What I see is this line :

      RMI TCP Connection(5)-172.16.2.2, received EOFException: error.

      I am posting the SSL trace (unable to post the full trace since there is a limit for 300000 chars while posting a topic):
      Any help is appreciated.

      -------------------------------------
      RMI TCP Connection(5)-172.16.2.2, setSoTimeout(30000) called
      RMI TCP Connection(5)-172.16.2.2, received EOFException: error
      RMI TCP Connection(5)-172.16.2.2, handling exception: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
      RMI TCP Connection(5)-172.16.2.2, SEND TLSv1 ALERT: fatal, description = handshake_failure
      RMI TCP Connection(5)-172.16.2.2, WRITE: TLSv1 Alert, length = 2
      [Raw write]: length = 7
      0000: 15 03 01 00 02 02 28 ......(
      RMI TCP Connection(5)-172.16.2.2, called closeSocket()
      RMI TCP Connection(5)-172.16.2.2, called close()
      RMI TCP Connection(5)-172.16.2.2, called closeInternal(true)
      RMI ConnectionExpiration-[10.103.202.10:1099], called close()
      RMI ConnectionExpiration-[10.103.202.10:1099], called closeInternal(true)
      RMI ConnectionExpiration-[10.103.202.10:1099], SEND TLSv1 ALERT: warning, description = close_notify
      Padded plaintext before ENCRYPTION: len = 18
      0000: 01 00 B8 48 4B D1 C6 52 F4 F2 48 CF 08 59 F4 C7 ...HK..R..H..Y..
      0010: 1F 27 .'
      RMI ConnectionExpiration-[10.103.202.10:1099], WRITE: TLSv1 Alert, length = 18
      [Raw write]: length = 23
      0000: 15 03 01 00 12 C1 58 22 E0 BF 47 70 2F 09 C6 48 ......X"..Gp/..H
      0010: 0E 0C 86 5C DC BD A0 ...\...
      RMI ConnectionExpiration-[10.103.202.10:1099], called close()
      RMI ConnectionExpiration-[10.103.202.10:1099], called closeInternal(true)
      RMI ConnectionExpiration-[10.103.202.10:1099], SEND TLSv1 ALERT: warning, description = close_notify
      Padded plaintext before ENCRYPTION: len = 18
      0000: 01 00 CC 3A 24 15 EC F6 63 DB 09 14 33 44 58 CE ...:$...c...3DX.
      0010: 1C 14 ..
      RMI ConnectionExpiration-[10.103.202.10:1099], WRITE: TLSv1 Alert, length = 18
      [Raw write]: length = 23
      [Raw read]: length = 5
      0000: 15 03 01 00 12 .....
      [Raw read]: length = 18
      0000: C1 58 22 E0 BF 47 70 2F 09 C6 48 0E 0C 86 5C DC .X"..Gp/..H...\.
      0010: BD A0 ..
      RMI TCP Connection(2)-10.103.202.10, READ: TLSv1 Alert, length = 18
      Padded plaintext after DECRYPTION: len = 18
      0000: 01 00 B8 48 4B D1 C6 52 F4 F2 48 CF 08 59 F4 C7 ...HK..R..H..Y..
      0010: 1F 27 .'
      RMI TCP Connection(2)-10.103.202.10, RECV TLSv1 ALERT: warning, close_notify
      RMI TCP Connection(2)-10.103.202.10, called closeInternal(false)
      RMI TCP Connection(2)-10.103.202.10, SEND TLSv1 ALERT: warning, description = close_notify
      Padded plaintext before ENCRYPTION: len = 18
      0000: 01 00 93 22 96 5A 3E F8 76 32 9E B0 9C 34 5A 9E ...".Z>.v2...4Z.
      0010: EB 77 .w
      RMI TCP Connection(2)-10.103.202.10, WRITE: TLSv1 Alert, length = 18
      [Raw write]: length = 23
      0000: 15 03 01 00 12 52 AA 32 CD B1 C3 75 A3 9E 51 72 .....R.2...u..Qr
      0010: 5F 22 E0 DF 90 C9 41 _"....A
      RMI TCP Connection(2)-10.103.202.10, called close()
      RMI TCP Connection(2)-10.103.202.10, called closeInternal(true)
      RMI TCP Connection(2)-10.103.202.10, called close()
      RMI TCP Connection(2)-10.103.202.10, called closeInternal(true)
      [Raw read]: length = 5
      0000: 15 03 01 00 12 .....
      [Raw read]: length = 18
      0000: 1F 38 19 B4 07 46 93 F7 06 71 C9 06 7A D7 69 32 .8...F...q..z.i2
      0010: EF 10 ..
      RMI TCP Connection(3)-10.103.202.10, READ: TLSv1 Alert, length = 18
      Padded plaintext after DECRYPTION: len = 18
      0000: 01 00 CC 3A 24 15 EC F6 63 DB 09 14 33 44 58 CE ...:$...c...3DX.
      0010: 1C 14 ..
      RMI TCP Connection(3)-10.103.202.10, RECV TLSv1 ALERT: warning, close_notify
      RMI TCP Connection(3)-10.103.202.10, called closeInternal(false)
      RMI TCP Connection(3)-10.103.202.10, SEND TLSv1 ALERT: warning, description = close_notify
      Padded plaintext before ENCRYPTION: len = 18
      0000: 01 00 49 65 0E 4D 2B 84 18 D3 D9 91 1E 6E D8 36 ..Ie.M+......n.6
      0010: AC 68 .h
      0000: 15 03 01 00 12 1F 38 19 B4 07 46 93 F7 06 71 C9 ......8...F...q.
      0010: 06 7A D7 69 32 EF 10 .z.i2..
      RMI TCP Connection(3)-10.103.202.10, WRITE: TLSv1 Alert, length = 18
      [Raw write]: length = 23
      0000: 15 03 01 00 12 B9 FA F2 20 4B A3 28 A6 D4 19 B3 ........ K.(....
      0010: 77 17 DC 69 61 44 50 w..iaDP
      RMI TCP Connection(3)-10.103.202.10, called close()
      RMI TCP Connection(3)-10.103.202.10, called closeInternal(true)
      RMI TCP Connection(3)-10.103.202.10, called close()
      RMI TCP Connection(3)-10.103.202.10, called closeInternal(true)
      RMI TCP Connection(4)-172.16.2.2, handling exception: java.net.SocketTimeoutException: Read timed out
      RMI TCP Connection(4)-172.16.2.2, called close()
      RMI TCP Connection(4)-172.16.2.2, called closeInternal(true)
      RMI TCP Connection(4)-172.16.2.2, SEND TLSv1 ALERT: warning, description = close_notify
      RMI TCP Connection(4)-172.16.2.2, WRITE: TLSv1 Alert, length = 2
        • 1. Re: RMI over SSL - handshake failure
          EJP
          You would have to post the same trace from the server.
          • 2. Re: RMI over SSL - handshake failure
            radiatejava
            Above trace was on the client side. On server side, I see:

            RMI ConnectionExpiration-[10.63.4.10:1099], WRITE: TLSv1 Alert, length = 18
            [Raw write]: length = 23
            0000: 15 03 01 00 12 52 51 BE 18 04 4D 9D 0D 18 6A 87 .....RQ...M...j.
            0010: 42 DA 82 30 AD 4D 34 B..0.M4
            Finalizer, called close()
            Finalizer, called closeInternal(true)
            Finalizer, called close()
            Finalizer, called closeInternal(true)
            Finalizer, called close()
            Finalizer, called closeInternal(true)
            Finalizer, called close()
            Finalizer, called closeInternal(true)
            [Raw read]: length = 5
            0000: 15 03 01 00 02 .....
            [Raw read]: length = 2
            0000: 01 00 ..
            Timer-10, READ: TLSv1 Alert, length = 2
            Timer-10, RECV TLSv1 ALERT: warning, close_notify
            Timer-10, SEND TLSv1 ALERT: fatal, description = unexpected_message
            Padded plaintext before ENCRYPTION: len = 18
            0000: 02 0A 13 5F 25 7B 85 BC 5B 50 35 59 B8 FF C5 42 ..._%...[P5Y...B
            0010: 3A D7 :.
            Timer-10, WRITE: TLSv1 Alert, length = 18
            [Raw write]: length = 23
            0000: 15 03 01 00 12 42 BE 25 8B 16 0D 73 B2 AE 92 6A .....B.%...s...j
            0010: E2 D6 49 47 90 9D 06 ..IG...
            Timer-10, called closeSocket()
            Timer-10, handling exception: javax.net.ssl.SSLException: Received close_notify during handshake
            Timer-10, called close()
            Timer-10, called closeInternal(true)
            • 3. Re: RMI over SSL - handshake failure
              radiatejava
              Further to add, on looking carefully, I see some exception in ssl trace (RemoteException, no such object in table).. is that the reason here ?

              RMI TCP Connection(1)-10.103.202.10, READ: TLSv1 Application Data, length = 2904
              Padded plaintext after DECRYPTION: len = 2904
              0000: 50 AC ED 00 05 77 22 00 00 00 00 00 00 00 00 00 P....w".........
              0010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
              0020: 00 44 15 4D C9 D4 E6 3B DF 74 00 0B 6C 6F 63 61 .D.M...;.t..loca
              0030: 6C 68 6F 73 74 56 32 73 72 00 35 63 6F 6D 2E 73 lhostV2sr.5com.s
              0040: 75 6E 2E 6A 64 6D 6B 2E 63 6F 6D 6D 2E 52 6D 69 un.jdmk.comm.Rmi
              0050: 43 6F 6E 6E 65 63 74 6F 72 53 65 72 76 65 72 4F ConnectorServerO
              0060: 62 6A 65 63 74 49 6D 70 6C 56 32 5F 53 74 75 62 bjectImplV2_Stub
              0070: A5 8D D4 AD 97 BD 11 11 02 00 00 74 0A 39 66 69 ...........t.9fi
              0080: 6C 65 3A 2F 70 65 72 66 69 67 6F 2F 61 63 63 65 le:/perfigo/acce
              0090: 73 73 2F 74 6F 6D 63 61 74 2F 73 68 61 72 65 64 ss/tomcat/shared
              00A0: 2F 63 6C 61 73 73 65 73 2F 20 66 69 6C 65 3A 2F /classes/ file:/
              00B0: 70 65 72 66 69 67 6F 2F 61 63 63 65 73 73 2F 74 perfigo/access/t
              00C0: 6F 6D 63 61 74 2F 73 68 61 72 65 64 2F 6C 69 62 omcat/shared/lib
              00D0: 2F 61 78 6C 72 61 64 69 75 73 2D 63 6C 69 65 6E /axlradius-clien
              00E0: 74 2D 33 2E 32 36 2E 6A 61 72 20 66 69 6C 65 3A t-3.26.jar file:
              00F0: 2F 70 65 72 66 69 67 6F 2F 61 63 63 65 73 73 2F /perfigo/access/
              0100: 74 6F 6D 63 61 74 2F 73 68 61 72 65 64 2F 6C 69 tomcat/shared/li
              0110: 62 2F 45 63 63 70 72 65 73 73 6F 41 6C 6C 2D 78 b/EccpressoAll-x
              0120: 2E 6A 61 72 20 66 69 6C 65 3A 2F 70 65 72 66 69 .jar file:/perfi
              0130: 67 6F 2F 61 63 63 65 73 73 2F 74 6F 6D 63 61 74 go/access/tomcat
              0140: 2F 73 68 61 72 65 64 2F 6C 69 62 2F 61 63 74 69 /shared/lib/acti
              0150: 76 61 74 69 6F 6E 2D 31 2E 31 2E 6A 61 72 20 66 vation-1.1.jar f
              0160: 69 6C 65 3A 2F 70 65 72 66 69 67 6F 2F 61 63 63 ile:/perfigo/acc
              0170: 65 73 73 2F 74 6F 6D 63 61 74 2F 73 68 61 72 65 ess/tomcat/share
              0180: 64 2F 6C 69 62 2F 63 6F 6D 6D 6F 6E 73 2D 62 65 d/lib/commons-be
              0190: 61 6E 75 74 69 6C 73 2D 31 2E 37 2E 30 2E 6A 61 anutils-1.7.0.ja
              01A0: 72 20 66 69 6C 65 3A 2F 70 65 72 66 69 67 6F 2F r file:/perfigo/
              01B0: 61 63 63 65 73 73 2F 74 6F 6D 63 61 74 2F 73 68 access/tomcat/sh
              01C0: 61 72 65 64 2F 6C 69 62 2F 62 63 70 72 6F 76 2D ared/lib/bcprov-
              01D0: 6A 64 6B 31 34 2D 31 2E 33 39 2E 6A 61 72 20 66 jdk14-1.39.jar f
              01E0: 69 6C 65 3A 2F 70 65 72 66 69 67 6F 2F 61 63 63 ile:/perfigo/acc
              01F0: 65 73 73 2F 74 6F 6D 63 61 74 2F 73 68 61 72 65 ess/tomcat/share
              0200: 64 2F 6C 69 62 2F 63 6F 6D 6D 6F 6E 73 2D 63 6F d/lib/commons-co
              0210: 6C 6C 65 63 74 69 6F 6E 73 2D 33 2E 31 2E 6A 61 llections-3.1.ja
              0220: 72 20 66 69 6C 65 3A 2F 70 65 72 66 69 67 6F 2F r file:/perfigo/
              0230: 61 63 63 65 73 73 2F 74 6F 6D 63 61 74 2F 73 68 access/tomcat/sh
              0240: 61 72 65 64 2F 6C 69 62 2F 63 6F 6D 6D 6F 6E 73 ared/lib/commons
              0250: 2D 63 6F 64 65 63 2D 31 2E 32 2E 6A 61 72 20 66 -codec-1.2.jar f
              0260: 69 6C 65 3A 2F 70 65 72 66 69 67 6F 2F 61 63 63 ile:/perfigo/acc
              0270: 65 73 73 2F 74 6F 6D 63 61 74 2F 73 68 61 72 65 ess/tomcat/share
              0280: 64 2F 6C 69 62 2F 63 6F 6D 6D 6F 6E 73 2D 63 6F d/lib/commons-co
              0290: 64 65 63 2D 31 2E 33 2E 6A 61 72 20 66 69 6C 65 dec-1.3.jar file
              02A0: 3A 2F 70 65 72 66 69 67 6F 2F 61 63 63 65 73 73 :/perfigo/access
              02B0: 2F 74 6F 6D 63 61 74 2F 73 68 61 72 65 64 2F 6C /tomcat/shared/l
              02C0: 69 62 2F 63 6F 6D 6D 6F 6E 73 2D 66 69 6C 65 75 ib/commons-fileu
              02D0: 70 6C 6F 61 64 2D 31 2E 32 2E 31 2E 6A 61 72 20 pload-1.2.1.jar
              02E0: 66 69 6C 65 3A 2F 70 65 72 66 69 67 6F 2F 61 63 file:/perfigo/ac
              02F0: 63 65 73 73 2F 74 6F 6D 63 61 74 2F 73 68 61 72 cess/tomcat/shar
              0300: 65 64 2F 6C 69 62 2F 63 6F 6D 6D 6F 6E 73 2D 64 ed/lib/commons-d
              0310: 69 67 65 73 74 65 72 2D 31 2E 36 2E 6A 61 72 20 igester-1.6.jar
              0320: 66 69 6C 65 3A 2F 70 65 72 66 69 67 6F 2F 61 63 file:/perfigo/ac
              0330: 63 65 73 73 2F 74 6F 6D 63 61 74 2F 73 68 61 72 cess/tomcat/shar
              0340: 65 64 2F 6C 69 62 2F 63 6F 6D 6D 6F 6E 73 2D 65 ed/lib/commons-e
              0350: 6C 2D 31 2E 30 2E 6A 61 72 20 66 69 6C 65 3A 2F l-1.0.jar file:/
              0360: 70 65 72 66 69 67 6F 2F 61 63 63 65 73 73 2F 74 perfigo/access/t
              0370: 6F 6D 63 61 74 2F 73 68 61 72 65 64 2F 6C 69 62 omcat/shared/lib
              0380: 2F 6E 6F 74 2D 79 65 74 2D 63 6F 6D 6D 6F 6E 73 /not-yet-commons
              0390: 2D 73 73 6C 2D 30 2E 33 2E 39 2E 6A 61 72 20 66 -ssl-0.3.9.jar f
              03A0: 69 6C 65 3A 2F 70 65 72 66 69 67 6F 2F 61 63 63 ile:/perfigo/acc
              03B0: 65 73 73 2F 74 6F 6D 63 61 74 2F 73 68 61 72 65 ess/tomcat/share
              03C0: 64 2F 6C 69 62 2F 63 6F 6D 6D 6F 6E 73 2D 68 74 d/lib/commons-ht
              03D0: 74 70 63 6C 69 65 6E 74 2D 33 2E 31 2E 6A 61 72 tpclient-3.1.jar
              03E0: 20 66 69 6C 65 3A 2F 70 65 72 66 69 67 6F 2F 61 file:/perfigo/a
              03F0: 63 63 65 73 73 2F 74 6F 6D 63 61 74 2F 73 68 61 ccess/tomcat/sha
              0400: 72 65 64 2F 6C 69 62 2F 63 6F 6D 6D 6F 6E 73 2D red/lib/commons-
              0410: 69 6F 2D 31 2E 34 2E 6A 61 72 20 66 69 6C 65 3A io-1.4.jar file:
              0420: 2F 70 65 72 66 69 67 6F 2F 61 63 63 65 73 73 2F /perfigo/access/
              0430: 74 6F 6D 63 61 74 2F 73 68 61 72 65 64 2F 6C 69 tomcat/shared/li
              0440: 62 2F 63 6F 6D 6D 6F 6E 73 2D 6C 61 6E 67 2D 32 b/commons-lang-2
              0450: 2E 31 2E 6A 61 72 20 66 69 6C 65 3A 2F 70 65 72 .1.jar file:/per
              0460: 66 69 67 6F 2F 61 63 63 65 73 73 2F 74 6F 6D 63 figo/access/tomc
              0470: 61 74 2F 73 68 61 72 65 64 2F 6C 69 62 2F 63 6F at/shared/lib/co
              0480: 6D 6D 6F 6E 73 2D 76 61 6C 69 64 61 74 6F 72 2D mmons-validator-
              0490: 31 2E 33 2E 31 2E 6A 61 72 20 66 69 6C 65 3A 2F 1.3.1.jar file:/
              04A0: 70 65 72 66 69 67 6F 2F 61 63 63 65 73 73 2F 74 perfigo/access/t
              04B0: 6F 6D 63 61 74 2F 73 68 61 72 65 64 2F 6C 69 62 omcat/shared/lib
              04C0: 2F 66 6C 65 78 6C 6D 2D 38 2E 34 2E 6A 61 72 20 /flexlm-8.4.jar
              04D0: 66 69 6C 65 3A 2F 70 65 72 66 69 67 6F 2F 61 63 file:/perfigo/ac
              04E0: 63 65 73 73 2F 74 6F 6D 63 61 74 2F 73 68 61 72 cess/tomcat/shar
              04F0: 65 64 2F 6C 69 62 2F 6A 61 63 6C 2D 31 2E 32 2E ed/lib/jacl-1.2.
              0500: 36 2E 6A 61 72 20 66 69 6C 65 3A 2F 70 65 72 66 6.jar file:/perf
              0510: 69 67 6F 2F 61 63 63 65 73 73 2F 74 6F 6D 63 61 igo/access/tomca
              0520: 74 2F 73 68 61 72 65 64 2F 6C 69 62 2F 6A 61 72 t/shared/lib/jar
              0530: 67 73 2D 31 2E 30 2E 6A 61 72 20 66 69 6C 65 3A gs-1.0.jar file:
              0540: 2F 70 65 72 66 69 67 6F 2F 61 63 63 65 73 73 2F /perfigo/access/
              0550: 74 6F 6D 63 61 74 2F 73 68 61 72 65 64 2F 6C 69 tomcat/shared/li
              0560: 62 2F 6A 61 76 61 63 73 76 2D 32 2E 30 2E 6A 61 b/javacsv-2.0.ja
              0570: 72 20 66 69 6C 65 3A 2F 70 65 72 66 69 67 6F 2F r file:/perfigo/
              0580: 61 63 63 65 73 73 2F 74 6F 6D 63 61 74 2F 73 68 access/tomcat/sh
              0590: 61 72 65 64 2F 6C 69 62 2F 6A 63 69 66 73 2D 31 ared/lib/jcifs-1
              05A0: 2E 32 2E 36 2E 6A 61 72 20 66 69 6C 65 3A 2F 70 .2.6.jar file:/p
              05B0: 65 72 66 69 67 6F 2F 61 63 63 65 73 73 2F 74 6F erfigo/access/to
              05C0: 6D 63 61 74 2F 73 68 61 72 65 64 2F 6C 69 62 2F mcat/shared/lib/
              05D0: 6A 64 6D 6B 2D 78 2E 6A 61 72 20 66 69 6C 65 3A jdmk-x.jar file:
              05E0: 2F 70 65 72 66 69 67 6F 2F 61 63 63 65 73 73 2F /perfigo/access/
              05F0: 74 6F 6D 63 61 74 2F 73 68 61 72 65 64 2F 6C 69 tomcat/shared/li
              0600: 62 2F 6A 6F 64 61 2D 74 69 6D 65 2D 31 2E 35 2E b/joda-time-1.5.
              0610: 32 2E 6A 61 72 20 66 69 6C 65 3A 2F 70 65 72 66 2.jar file:/perf
              0620: 69 67 6F 2F 61 63 63 65 73 73 2F 74 6F 6D 63 61 igo/access/tomca
              0630: 74 2F 73 68 61 72 65 64 2F 6C 69 62 2F 6A 73 6E t/shared/lib/jsn
              0640: 6D 70 61 70 69 2D 78 2E 6A 61 72 20 66 69 6C 65 mpapi-x.jar file
              0650: 3A 2F 70 65 72 66 69 67 6F 2F 61 63 63 65 73 73 :/perfigo/access
              0660: 2F 74 6F 6D 63 61 74 2F 73 68 61 72 65 64 2F 6C /tomcat/shared/l
              0670: 69 62 2F 6D 61 67 65 6C 6C 61 6E 2D 78 2E 6A 61 ib/magellan-x.ja
              0680: 72 20 66 69 6C 65 3A 2F 70 65 72 66 69 67 6F 2F r file:/perfigo/
              0690: 61 63 63 65 73 73 2F 74 6F 6D 63 61 74 2F 73 68 access/tomcat/sh
              06A0: 61 72 65 64 2F 6C 69 62 2F 6D 61 69 6C 2D 31 2E ared/lib/mail-1.
              06B0: 34 2E 6A 61 72 20 66 69 6C 65 3A 2F 70 65 72 66 4.jar file:/perf
              06C0: 69 67 6F 2F 61 63 63 65 73 73 2F 74 6F 6D 63 61 igo/access/tomca
              06D0: 74 2F 73 68 61 72 65 64 2F 6C 69 62 2F 6D 79 66 t/shared/lib/myf
              06E0: 61 63 65 73 2D 61 70 69 2D 31 2E 31 2E 35 2E 6A aces-api-1.1.5.j
              06F0: 61 72 20 66 69 6C 65 3A 2F 70 65 72 66 69 67 6F ar file:/perfigo
              0700: 2F 61 63 63 65 73 73 2F 74 6F 6D 63 61 74 2F 73 /access/tomcat/s
              0710: 68 61 72 65 64 2F 6C 69 62 2F 6D 79 66 61 63 65 hared/lib/myface
              0720: 73 2D 69 6D 70 6C 2D 31 2E 31 2E 35 2E 6A 61 72 s-impl-1.1.5.jar
              0730: 20 66 69 6C 65 3A 2F 70 65 72 66 69 67 6F 2F 61 file:/perfigo/a
              0740: 63 63 65 73 73 2F 74 6F 6D 63 61 74 2F 73 68 61 ccess/tomcat/sha
              0750: 72 65 64 2F 6C 69 62 2F 73 6E 6D 70 34 6A 2D 31 red/lib/snmp4j-1
              0760: 2E 39 2E 33 64 2E 6A 61 72 20 66 69 6C 65 3A 2F .9.3d.jar file:/
              0770: 70 65 72 66 69 67 6F 2F 61 63 63 65 73 73 2F 74 perfigo/access/t
              0780: 6F 6D 63 61 74 2F 73 68 61 72 65 64 2F 6C 69 62 omcat/shared/lib
              0790: 2F 6F 72 6F 2D 32 2E 30 2E 38 2E 6A 61 72 20 66 /oro-2.0.8.jar f
              07A0: 69 6C 65 3A 2F 70 65 72 66 69 67 6F 2F 61 63 63 ile:/perfigo/acc
              07B0: 65 73 73 2F 74 6F 6D 63 61 74 2F 73 68 61 72 65 ess/tomcat/share
              07C0: 64 2F 6C 69 62 2F 6E 61 63 2D 77 65 62 61 70 70 d/lib/nac-webapp
              07D0: 73 2D 73 68 61 72 65 64 2D 34 2E 36 2E 31 2D 53 s-shared-4.6.1-S
              07E0: 4E 41 50 53 48 4F 54 2E 6A 61 72 20 66 69 6C 65 NAPSHOT.jar file
              07F0: 3A 2F 70 65 72 66 69 67 6F 2F 61 63 63 65 73 73 :/perfigo/access
              0800: 2F 74 6F 6D 63 61 74 2F 73 68 61 72 65 64 2F 6C /tomcat/shared/l
              0810: 69 62 2F 6F 70 65 6E 73 61 6D 6C 2D 32 2E 31 2E ib/opensaml-2.1.
              0820: 30 2E 6A 61 72 20 66 69 6C 65 3A 2F 70 65 72 66 0.jar file:/perf
              0830: 69 67 6F 2F 61 63 63 65 73 73 2F 74 6F 6D 63 61 igo/access/tomca
              0840: 74 2F 73 68 61 72 65 64 2F 6C 69 62 2F 6F 70 65 t/shared/lib/ope
              0850: 6E 77 73 2D 31 2E 31 2E 30 2E 6A 61 72 20 66 69 nws-1.1.0.jar fi
              0860: 6C 65 3A 2F 70 65 72 66 69 67 6F 2F 61 63 63 65 le:/perfigo/acce
              0870: 73 73 2F 74 6F 6D 63 61 74 2F 73 68 61 72 65 64 ss/tomcat/shared
              0880: 2F 6C 69 62 2F 70 65 72 66 69 67 6F 2E 6A 61 72 /lib/perfigo.jar
              0890: 20 66 69 6C 65 3A 2F 70 65 72 66 69 67 6F 2F 61 file:/perfigo/a
              08A0: 63 63 65 73 73 2F 74 6F 6D 63 61 74 2F 73 68 61 ccess/tomcat/sha
              08B0: 72 65 64 2F 6C 69 62 2F 70 6F 69 2D 32 2E 35 2E red/lib/poi-2.5.
              08C0: 31 2D 66 69 6E 61 6C 2D 32 30 30 34 30 38 30 34 1-final-20040804
              08D0: 2E 6A 61 72 20 66 69 6C 65 3A 2F 70 65 72 66 69 .jar file:/perfi
              08E0: 67 6F 2F 61 63 63 65 73 73 2F 74 6F 6D 63 61 74 go/access/tomcat
              08F0: 2F 73 68 61 72 65 64 2F 6C 69 62 2F 70 6F 73 74 /shared/lib/post
              0900: 67 72 65 73 71 6C 2D 38 2E 31 2D 34 30 38 2E 6A gresql-8.1-408.j
              0910: 64 62 63 33 2E 6A 61 72 20 66 69 6C 65 3A 2F 70 dbc3.jar file:/p
              0920: 65 72 66 69 67 6F 2F 61 63 63 65 73 73 2F 74 6F erfigo/access/to
              0930: 6D 63 61 74 2F 73 68 61 72 65 64 2F 6C 69 62 2F mcat/shared/lib/
              0940: 73 70 72 69 6E 67 2D 32 2E 30 2E 31 2E 6A 61 72 spring-2.0.1.jar
              0950: 20 66 69 6C 65 3A 2F 70 65 72 66 69 67 6F 2F 61 file:/perfigo/a
              0960: 63 63 65 73 73 2F 74 6F 6D 63 61 74 2F 73 68 61 ccess/tomcat/sha
              0970: 72 65 64 2F 6C 69 62 2F 74 63 6C 6A 61 76 61 2D red/lib/tcljava-
              0980: 31 2E 32 2E 36 2E 6A 61 72 20 66 69 6C 65 3A 2F 1.2.6.jar file:/
              0990: 70 65 72 66 69 67 6F 2F 61 63 63 65 73 73 2F 74 perfigo/access/t
              09A0: 6F 6D 63 61 74 2F 73 68 61 72 65 64 2F 6C 69 62 omcat/shared/lib
              09B0: 2F 74 6F 6D 61 68 61 77 6B 2D 31 2E 31 2E 36 2E /tomahawk-1.1.6.
              09C0: 6A 61 72 20 66 69 6C 65 3A 2F 70 65 72 66 69 67 jar file:/perfig
              09D0: 6F 2F 61 63 63 65 73 73 2F 74 6F 6D 63 61 74 2F o/access/tomcat/
              09E0: 73 68 61 72 65 64 2F 6C 69 62 2F 74 6F 6D 61 68 shared/lib/tomah
              09F0: 61 77 6B 2D 73 61 6E 64 62 6F 78 2D 31 2E 31 2E awk-sandbox-1.1.
              0A00: 36 2E 6A 61 72 20 66 69 6C 65 3A 2F 70 65 72 66 6.jar file:/perf
              0A10: 69 67 6F 2F 61 63 63 65 73 73 2F 74 6F 6D 63 61 igo/access/tomca
              0A20: 74 2F 73 68 61 72 65 64 2F 6C 69 62 2F 76 65 6C t/shared/lib/vel
              0A30: 6F 63 69 74 79 2D 31 2E 35 2E 6A 61 72 20 66 69 ocity-1.5.jar fi
              0A40: 6C 65 3A 2F 70 65 72 66 69 67 6F 2F 61 63 63 65 le:/perfigo/acce
              0A50: 73 73 2F 74 6F 6D 63 61 74 2F 73 68 61 72 65 64 ss/tomcat/shared
              0A60: 2F 6C 69 62 2F 78 6D 6C 2D 73 65 63 75 72 69 74 /lib/xml-securit
              0A70: 79 2D 31 2E 34 2E 31 2E 6A 61 72 20 66 69 6C 65 y-1.4.1.jar file
              0A80: 3A 2F 70 65 72 66 69 67 6F 2F 61 63 63 65 73 73 :/perfigo/access
              0A90: 2F 74 6F 6D 63 61 74 2F 73 68 61 72 65 64 2F 6C /tomcat/shared/l
              0AA0: 69 62 2F 78 6D 6C 74 6F 6F 6C 69 6E 67 2D 31 2E ib/xmltooling-1.
              0AB0: 30 2E 31 2E 6A 61 72 78 72 00 1A 6A 61 76 61 2E 0.1.jarxr..java.
              0AC0: 72 6D 69 2E 73 65 72 76 65 72 2E 52 65 6D 6F 74 rmi.server.Remot
              0AD0: 65 53 74 75 62 E9 FE DC C9 8B E1 65 1A 02 00 00 eStub......e....
              0AE0: 70 78 72 00 1C 6A 61 76 61 2E 72 6D 69 2E 73 65 pxr..java.rmi.se
              0AF0: 72 76 65 72 2E 52 65 6D 6F 74 65 4F 62 6A 65 63 rver.RemoteObjec
              0B00: 74 D3 61 B4 91 0C 61 33 1E 03 00 00 70 78 70 77 t.a...a3....pxpw
              0B10: 36 00 0A 55 6E 69 63 61 73 74 52 65 66 00 0D 31 6..UnicastRef..1
              0B20: 30 2E 31 30 33 2E 32 30 32 2E 31 30 00 00 04 4B 0.103.202.10...K
              0B30: 00 00 00 00 00 00 00 00 7B 4C 40 BF 00 00 01 2E .........L@.....
              0B40: 52 54 43 BC 80 00 00 78 D0 79 62 A8 20 0C A9 B2 RTC....x.yb. ...
              0B50: 4E C4 BE F1 21 C4 0A 82 N...!...
              Padded plaintext before ENCRYPTION: len = 441
              0000: 51 AC ED 00 05 77 0F 02 7B 4C 40 BF 00 00 01 2E Q....w...L@.....
              0010: 52 54 43 BC 80 02 73 72 00 1E 6A 61 76 61 2E 72 RTC...sr..java.r
              0020: 6D 69 2E 4E 6F 53 75 63 68 4F 62 6A 65 63 74 45 mi.NoSuchObjectE
              0030: 78 63 65 70 74 69 6F 6E 5B DC D1 8C 01 04 50 19 xception[.....P.
              0040: 02 00 00 70 78 72 00 18 6A 61 76 61 2E 72 6D 69 ...pxr..java.rmi
              0050: 2E 52 65 6D 6F 74 65 45 78 63 65 70 74 69 6F 6E .RemoteException
              0060: B8 8C 9D 4E DE E4 7A 22 02 00 01 4C 00 06 64 65 ...N..z"...L..de
              0070: 74 61 69 6C 74 00 15 4C 6A 61 76 61 2F 6C 61 6E tailt..Ljava/lan
              0080: 67 2F 54 68 72 6F 77 61 62 6C 65 3B 70 78 72 00 g/Throwable;pxr.
              0090: 13 6A 61 76 61 2E 69 6F 2E 49 4F 45 78 63 65 70 .java.io.IOExcep
              00A0: 74 69 6F 6E 6C 80 73 64 65 25 F0 AB 02 00 00 70 tionl.sde%.....p
              00B0: 78 72 00 13 6A 61 76 61 2E 6C 61 6E 67 2E 45 78 xr..java.lang.Ex
              00C0: 63 65 70 74 69 6F 6E D0 FD 1F 3E 1A 3B 1C C4 02 ception...>.;...
              00D0: 00 00 70 78 72 00 13 6A 61 76 61 2E 6C 61 6E 67 ..pxr..java.lang
              00E0: 2E 54 68 72 6F 77 61 62 6C 65 D5 C6 35 27 39 77 .Throwable..5'9w
              00F0: B8 CB 03 00 03 4C 00 05 63 61 75 73 65 71 00 7E .....L..causeq..
              0100: 00 02 4C 00 0D 64 65 74 61 69 6C 4D 65 73 73 61 ..L..detailMessa
              0110: 67 65 74 00 12 4C 6A 61 76 61 2F 6C 61 6E 67 2F get..Ljava/lang/
              0120: 53 74 72 69 6E 67 3B 5B 00 0A 73 74 61 63 6B 54 String;[..stackT
              0130: 72 61 63 65 74 00 1E 5B 4C 6A 61 76 61 2F 6C 61 racet..[Ljava/la
              0140: 6E 67 2F 53 74 61 63 6B 54 72 61 63 65 45 6C 65 ng/StackTraceEle
              0150: 6D 65 6E 74 3B 70 78 70 70 74 00 17 6E 6F 20 73 ment;pxppt..no s
              0160: 75 63 68 20 6F 62 6A 65 63 74 20 69 6E 20 74 61 uch object in ta
              0170: 62 6C 65 75 72 00 1E 5B 4C 6A 61 76 61 2E 6C 61 bleur..[Ljava.la
              0180: 6E 67 2E 53 74 61 63 6B 54 72 61 63 65 45 6C 65 ng.StackTraceEle
              0190: 6D 65 6E 74 3B 02 46 2A 3C 3C FD 22 39 02 00 00 ment;.F*<<."9...
              01A0: 70 78 70 00 00 00 00 78 70 29 23 E9 DD FF BE 19 pxp....xp)#.....
              01B0: 1E D7 8F 2E 9A 45 2D 16 36 .....E-.6
              RMI TCP Connection(1)-10.103.202.10, WRITE: TLSv1 Application Data, length = 441
              [Raw read]: length = 5
              0000: 17 03 01 01 B9 .....
              [Raw read]: length = 441
              0000: 39 7E 36 25 1F 9C C4 90 9A FD E9 C4 A5 DF 64 1A 9.6%..........d.
              0010: AE 76 CB 4C 67 26 C5 CA 67 94 C6 BA 9F 4C F9 B2 .v.Lg&..g....L..
              0020: 99 96 A7 FC E2 06 0C D1 57 15 B5 DE 5C AD 04 AB ........W...\...
              0030: EB 43 FB 8E C9 EC 0F DB 7E BF CE 50 54 80 8F BF .C.........PT...
              0040: 74 03 4B 08 AA 5B E7 E2 AF 18 2C D3 CC AD E1 8A t.K..[....,.....
              0050: BE 91 DD 7E C6 4B DE 8C A3 3B F0 9D 19 38 7C 13 .....K...;...8..
              0060: 8C 58 E2 C5 F6 C8 7B 74 42 D6 4B 9B 6B C5 5C EB .X.....tB.K.k.\.
              0070: EA 13 12 C0 73 8E 35 E3 72 CD 44 36 17 45 CC 51 ....s.5.r.D6.E.Q
              0080: 7D F2 CD 4E B7 65 4F CE F2 56 1F B7 49 B2 45 7B ...N.eO..V..I.E.
              0090: DB DA 40 FE 1B 61 30 DC 08 C1 91 81 8C 31 31 3F ..@..a0......11?
              00A0: 89 2C DF 00 56 AF E5 9D 81 6F 84 E5 97 D6 93 68 .,..V....o.....h
              00B0: 43 82 78 BD 57 C4 3F 36 08 7F 7B CD 4B B2 C4 E6 C.x.W.?6....K...
              00C0: B8 E9 D3 5C A1 E5 1B F5 7B 7A B3 E0 AA 06 09 82 ...\.....z......
              00D0: B4 13 33 FC 63 45 19 52 9B 35 6B 7D 36 B7 24 55 ..3.cE.R.5k.6.$U
              00E0: 31 42 D1 08 CF D0 32 FB BC E5 A0 CF DC 71 D5 AC 1B....2......q..
              00F0: 55 94 31 50 D9 52 70 8F 2F 26 E9 0E EA 14 3F BD U.1P.Rp./&....?.
              0100: EA D5 5B 86 D4 CF D9 D7 12 F9 28 2E A0 A6 8D EE ..[.......(.....
              0110: 5B 15 FD 01 30 65 37 52 5F 93 5F 46 3A 5A 80 5D [...0e7R_._F:Z.]
              0120: D2 3E C5 A4 74 0B 93 D1 C9 70 CA B0 BE AB 19 0A .>..t....p......
              0130: 0F B4 DF 7C 17 2C 29 B1 D1 A2 F9 C9 BC AF 96 3E .....,)........>
              0140: 2D 92 70 9D CF 62 5E BF 9B 0E 70 36 8D 6C C4 D0 -.p..b^...p6.l..
              0150: F0 F9 60 47 2E 5B A6 47 B6 74 5B C9 56 37 09 11 ..`G.[.G.t[.V7..
              0160: F8 D7 68 0C A2 6F 5B 9A 6D AE A8 FF 85 FE B4 23 ..h..o[.m......#
              0170: E3 3E 12 63 F2 DD 2C 78 94 81 13 79 4B 87 ED 96 .>.c..,x...yK...
              0180: 74 13 34 07 E8 20 D2 ED EC B8 CF FF 1D AC A1 F0 t.4.. ..........
              0190: AC 7A A6 D2 57 D9 E0 DC 35 7D F2 A9 63 EE 24 48 .z..W...5...c.$H
              01A0: FB C7 B9 93 6B 0E 23 CA F5 73 18 D5 51 E2 44 5F ....k.#..s..Q.D_
              01B0: A1 CC 89 31 1C 00 75 B5 D6 ...1..u..
              DefaultDomain:name=RmiConnectorServer, READ: TLSv1 Application Data, length = 441
              Padded plaintext after DECRYPTION: len = 441
              0000: 51 AC ED 00 05 77 0F 02 7B 4C 40 BF 00 00 01 2E Q....w...L@.....
              0010: 52 54 43 BC 80 02 73 72 00 1E 6A 61 76 61 2E 72 RTC...sr..java.r
              0020: 6D 69 2E 4E 6F 53 75 63 68 4F 62 6A 65 63 74 45 mi.NoSuchObjectE
              0030: 78 63 65 70 74 69 6F 6E 5B DC D1 8C 01 04 50 19 xception[.....P.
              0040: 02 00 00 70 78 72 00 18 6A 61 76 61 2E 72 6D 69 ...pxr..java.rmi
              0050: 2E 52 65 6D 6F 74 65 45 78 63 65 70 74 69 6F 6E .RemoteException
              0060: B8 8C 9D 4E DE E4 7A 22 02 00 01 4C 00 06 64 65 ...N..z"...L..de
              0070: 74 61 69 6C 74 00 15 4C 6A 61 76 61 2F 6C 61 6E tailt..Ljava/lan
              0080: 67 2F 54 68 72 6F 77 61 62 6C 65 3B 70 78 72 00 g/Throwable;pxr.
              0090: 13 6A 61 76 61 2E 69 6F 2E 49 4F 45 78 63 65 70 .java.io.IOExcep
              00A0: 74 69 6F 6E 6C 80 73 64 65 25 F0 AB 02 00 00 70 tionl.sde%.....p
              00B0: 78 72 00 13 6A 61 76 61 2E 6C 61 6E 67 2E 45 78 xr..java.lang.Ex
              00C0: 63 65 70 74 69 6F 6E D0 FD 1F 3E 1A 3B 1C C4 02 ception...>.;...
              00D0: 00 00 70 78 72 00 13 6A 61 76 61 2E 6C 61 6E 67 ..pxr..java.lang
              00E0: 2E 54 68 72 6F 77 61 62 6C 65 D5 C6 35 27 39 77 .Throwable..5'9w
              00F0: B8 CB 03 00 03 4C 00 05 63 61 75 73 65 71 00 7E .....L..causeq..
              0100: 00 02 4C 00 0D 64 65 74 61 69 6C 4D 65 73 73 61 ..L..detailMessa
              0110: 67 65 74 00 12 4C 6A 61 76 61 2F 6C 61 6E 67 2F get..Ljava/lang/
              0120: 53 74 72 69 6E 67 3B 5B 00 0A 73 74 61 63 6B 54 String;[..stackT
              0130: 72 61 63 65 74 00 1E 5B 4C 6A 61 76 61 2F 6C 61 racet..[Ljava/la
              0140: 6E 67 2F 53 74 61 63 6B 54 72 61 63 65 45 6C 65 ng/StackTraceEle
              0150: 6D 65 6E 74 3B 70 78 70 70 74 00 17 6E 6F 20 73 ment;pxppt..no s
              0160: 75 63 68 20 6F 62 6A 65 63 74 20 69 6E 20 74 61 uch object in ta
              0170: 62 6C 65 75 72 00 1E 5B 4C 6A 61 76 61 2E 6C 61 bleur..[Ljava.la
              0180: 6E 67 2E 53 74 61 63 6B 54 72 61 63 65 45 6C 65 ng.StackTraceEle
              0190: 6D 65 6E 74 3B 02 46 2A 3C 3C FD 22 39 02 00 00 ment;.F*<<."9...
              01A0: 70 78 70 00 00 00 00 78 70 29 23 E9 DD FF BE 19 pxp....xp)#.....
              01B0: 1E D7 8F 2E 9A 45 2D 16 36 .....E-.6
              DefaultDomain:name=RmiConnectorServer, called close()
              DefaultDomain:name=RmiConnectorServer, called closeInternal(true)
              DefaultDomain:name=RmiConnectorServer, SEND TLSv1 ALERT: warning, description = close_notify
              Padded plaintext before ENCRYPTION: len = 18
              0000: 01 00 FE 19 02 8D 7A 8B 3B 6D 22 8E D8 79 F6 AA ......z.;m"..y..
              0010: BB 45 .E
              DefaultDomain:name=RmiConnectorServer, WRITE: TLSv1 Alert, length = 18
              [Raw write]: length = 23
              0000: 15 03 01 00 12 67 39 63 5F E5 4E 45 B5 18 86 3C .....g9c_.NE...<
              0010: E4 40 58 ED 04 BD 3F .@X...?
              DefaultDomain:name=RmiConnectorServer, setSoTimeout(30000) called
              • 4. Re: RMI over SSL - handshake failure
                EJP
                NoSuchObjectInTable means you are using a 'stale' stub: the remote object that exported it has been unexported, possibly via DGC or local GC. When you get that exception in the client you should renew the stub, e.g. do the Naming.lookup() again, and be prepared for whatever exceptions may arise out of that.

                However none of that would cause an SSL problem. Can you post the missing part of the server trace, from the beginning?
                • 5. Re: RMI over SSL - handshake failure
                  radiatejava
                  I am unable to post the full trace due to the 30k char limit. Anyway I could do that ?
                  • 6. Re: RMI over SSL - handshake failure
                    EJP
                    We only need whatever led to the server's decision to close the socket. All the successfull stuff before that can be deleted. Also all the plaintext, not interested in that.
                    • 7. Re: RMI over SSL - handshake failure
                      radiatejava
                      I have uploaded the ssl (as client) trace here. Once you open the link, you would have an option to download that as text. This is a production trace. Server IP is 10.63.4.10 for which we have the RMI/SSL failure.

                      https://docs.google.com/document/d/1RgqVfDpH_t1lhqCb7L9A-zDgNBJUGeEhfv7zXDc6grQ/edit?hl=en&authkey=CPvd97YK

                      Thanks for your co-operation.

                      Edited by: radiatejava on Mar 7, 2011 5:04 PM
                      • 8. Re: RMI over SSL - handshake failure
                        EJP
                        Your server is engaging in SSL as a client with another server (SSLv2ClientHello line), and that conversation appears to be failing. SSLSockets appear to be getting garbage-collected (Finalizer lines). There are several successful connections in this trace. There is too much data. Can you try it with just javax.net.debug=ssl,handshake.
                        • 9. Re: RMI over SSL - handshake failure
                          radiatejava
                          Thanks for the prompt reply. You are right, the trace is for ssl as a client. Hence I renamed the existing file in google docs to ss-client.txt.

                          Actual server ssl trace is now at : https://docs.google.com/document/d/1z4VBbxUqCeapWMxXlAb8B2RQtBlJUBCLUlpwUo1pA1w/edit?hl=en&authkey=CI36vvEP

                          Would you be able to take a look into this now ?
                          • 10. Re: RMI over SSL - handshake failure
                            EJP
                            There are no errors there at all, it all looks like success apart from the Finalizer lines, which of course are unexpected closes from the client's point of view. I don't see how they are even possible with pure RMI. Is the server maybe doing some SSLSocket-level stuff of it's own other than the RMI?
                            • 11. Re: RMI over SSL - handshake failure
                              radiatejava
                              We are into troubleshooting this. The 2 hosts (server and client) are sitting in 2 different cities and we are trying to see if anything in between (like firewall etc) is eating up some TCP packets of the SSL connection.
                              • 12. Re: RMI over SSL - handshake failure
                                radiatejava
                                We were finally able to do some network troubleshooting. Somewhere in the path network MTU was not 1500 that is normally used and this was causing lots of TCP fragmentation and re-transmissions of packets, leading to these SSL failures.