This discussion is archived
1 2 Previous Next 17 Replies Latest reply: Nov 16, 2012 12:37 PM by Mark Malakanov (user11181920) RSS

Connection reset from JDBC to 11g

721832 Newbie
Currently Being Moderated
Hello,

I'm running prod db on server A (SPARC), Solaris10, DB 11gR2 (11.2.0.3). Connecting by JDBC my web apps. Everything is ok.

Now - I've set up other server - DEVEL (SPARC), Solaris11, DB11gR2 (11.2.0.3). I've got two instances (two solaris zones), and I'm receiving messages from my JDBC clients, that connection was rapidly reset.

So, the same arch, the same JDBC client, the same network segment, the same DB engine. Only difference is base OS (Sol11 and Sol10) and Sparc version (SPARC VII vs SPARC T2). How to verify what is causing such problems? Both servers are connected to the same switch, so even the network physical layer is the same.

Regards
Rafal
  • 1. Re: Connection reset from JDBC to 11g
    Osama_Mustafa Oracle ACE
    Currently Being Moderated
    Mark this thread as answered and open it in


    Oracle Discussion Forums » Fusion Middleware
  • 2. Re: Connection reset from JDBC to 11g
    721832 Newbie
    Currently Being Moderated
    But I don't think that problem is in middleware - middleware works fine with prod db. I think the problem is in listener and network handling on db side.
  • 3. Re: Connection reset from JDBC to 11g
    Osama_Mustafa Oracle ACE
    Currently Being Moderated
    Well lets see that

    Post lsnrctl status
    and
    sqlnet.ora
  • 4. Re: Connection reset from JDBC to 11g
    Osama_Mustafa Oracle ACE
    Currently Being Moderated
    zawierta wrote:
    But I don't think that problem is in middleware - middleware works fine with prod db. I think the problem is in listener and network handling on db side.
    For Quick process you have to check With your network administrator Firewall , Switch , ... to prove everything is Ok
  • 5. Re: Connection reset from JDBC to 11g
    Mark Malakanov (user11181920) Expert
    Currently Being Moderated
    I'm receiving messages from my JDBC clients, that connection was rapidly reset.
    post the original error message (and stack if possible).
  • 6. Re: Connection reset from JDBC to 11g
    721832 Newbie
    Currently Being Moderated
    Thanks for response.



    I'm also network admin - trust me, it is no network problem. It is complex environment, but to simplify let's assume that:
    PROD_DB - 10.0.0.11
    DEV1_DB - 10.0.0.21
    DEV2_DB - 10.0.0.22
    DEV3_DB - 10.0.0.23
    PROD_AppSrv - 10.0.0.14
    DEV_AppSrv - 10.0.0.8


    DEV_AppSrv is hosting three deploys of the same application, each connected to different DEVx_DB.
    Everything is on the same subnet, there is no firewall, everything is on the same switch (enterprise-level cisco).

    Prod_DB - Solaris10(M4000), 11.2.0.3
    DEV1_DB - Solaris10(old sparc machine) - db 10.2.0.5 (this one was frozen with upgrade due to problem with resetting connection)
    DEV2_DB - Solaris11(T5220), 11.2.0.3
    DEV3_DB - Solaris11(T5220), 11.2.0.3

    Now, how it works:

    PROD_APP + PROD_DB - fine
    DEV_APP + DEV1_DB - fine
    DEV_APP + DEV2_DB - reset
    DEV_APP + DEV3_DB - reset

    As you can see, middleware seems to be ok, when dev2 and dev3 was 10g on old machine (like dev1 does now) - all three instances were fine.
    Also upgrading from 10 to 11g2 causes no problem, because I'm running my prod instance on 11g on extremely high load (100 times bigger than dev) and it works.

    Only difference between prod_db and dev2/3_db is, that base os is Solaris11 on dev systems. Of course I can install 11gR2 on "old machine" (where DEV1_DB is working fine), but I guess it is not a good way. Downgrading Solaris11 to Solaris10... well, if everything else fails :)

    That's all about my env.
    In next post I'll give you more info.
  • 7. Re: Connection reset from JDBC to 11g
    721832 Newbie
    Currently Being Moderated
    DEV2_DB:

    oracle@test:/opt/oracle
    $ lsnrctl status

    LSNRCTL for Solaris: Version 11.2.0.3.0 - Production on 15-NOV-2012 21:56:04

    Copyright (c) 1991, 2011, Oracle. All rights reserved.

    Connecting to (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521))
    STATUS of the LISTENER
    ------------------------
    Alias LISTENER
    Version TNSLSNR for Solaris: Version 11.2.0.3.0 - Production
    Start Date 12-NOV-2012 14:22:07
    Uptime 3 days 7 hr. 33 min. 56 sec
    Trace Level off
    Security ON: Local OS Authentication
    SNMP OFF
    Listener Parameter File /opt/oracle/112/network/admin/listener.ora
    Listener Log File /opt/oracle/diag/tnslsnr/dbtest/listener/alert/log.xml
    Listening Endpoints Summary...
    (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=dbtest)(PORT=1521)))
    Services Summary...
    Service "dbtest" has 1 instance(s).
    Instance "dbtest", status READY, has 1 handler(s) for this service...
    Service "dbtestXDB" has 1 instance(s).
    Instance "dbtest", status READY, has 1 handler(s) for this service...
    The command completed successfully


    No sqlnet.ora on DEV2_DB host.
  • 8. Re: Connection reset from JDBC to 11g
    sb92075 Guru
    Currently Being Moderated
    Oracle depends upon the OS for all network activity.
    Oracle RDBMS can & does run fine WITHOUT any network connection.
    If/when a problem exists with any packet exchange, the root cause & solution are EXTERNAL to Oracle.
    Oracle is the victim; not the culprit!
  • 9. Re: Connection reset from JDBC to 11g
    721832 Newbie
    Currently Being Moderated
    Ok, maybe Oracle DB is victim... I'm just asking you to help me trace the problem. Now I guess, that it can be different OS. If so, I'll reinstall solaris tomorrow and try to solve that, but that is whole day of dirty job - SAN/FC network, multipath, zone creation and so on, so reinstalling Solaris is on the last place in my TODO list.

    Can I somehow force listener to generate a lot of debug logs?

    R
  • 10. Re: Connection reset from JDBC to 11g
    Mark Malakanov (user11181920) Expert
    Currently Being Moderated
    >
    PROD_APP + PROD_DB - fine
    DEV_APP + DEV1_DB - fine
    DEV_APP + DEV2_DB - reset
    DEV_APP + DEV3_DB - reset
    >

    It looks to me like Solaris 11 TCP/IP stack breaks network connection for some reason.
    Is it JDBC connection cannot be established or it is established and then broken after some period?

    Again. post the original App error message and its stack.
  • 11. Re: Connection reset from JDBC to 11g
    sb92075 Guru
    Currently Being Moderated
    zawierta wrote:
    Ok, maybe Oracle DB is victim... I'm just asking you to help me trace the problem. Now I guess, that it can be different OS. If so, I'll reinstall solaris tomorrow and try to solve that, but that is whole day of dirty job - SAN/FC network, multipath, zone creation and so on, so reinstalling Solaris is on the last place in my TODO list.

    Can I somehow force listener to generate a lot of debug logs?
    Once the client has established a session within the DB,
    the listener has NO involvement with the ongoing packet exchange.
    You can stop the listener & existing sessions continue without any disruption!
    Listener is only involved establishing the session after getting original request from the client.
  • 12. Re: Connection reset from JDBC to 11g
    Justin Cave Oracle ACE
    Currently Being Moderated
    If you're a network admin, presumbably you've got plenty of tools to be able to trace the exchange of packets between the client and server machine and identify exactly what is resetting the connection.

    You ask about doing additional logging in the listener. Does that imply that you have traced the network connections and identified that the problem is occurring while the connection is being established and the listener is still involved in the process? The listener simply accepts the initial connection request and hands off the client to a particular database process-- once the connection is established, the listener isn't involved any longer-- it can go down and existing connections will continue to work just fine. When you said initially that the connection was getting "reset", that implied that it was being established successfully. Perhaps that was not the implication that you intended.

    What, exactly, is the error that you are getting? Are you getting the behavior only when the connection is opened via JDBC? Or do you get the same behavior with a SQL*Plus connection? Are you getting any errors in the alert log?

    Justin
  • 13. Re: Connection reset from JDBC to 11g
    jgarry Guru
    Currently Being Moderated
    zawierta wrote:
    Thanks for response.



    I'm also network admin - trust me, it is no network problem. It is complex environment, but to simplify let's assume that:
    You know what I do whenever anyone says "trust me?" You know what happens to my opinion of the ability of netadmins to diagnose a database issue when they can't even show the error message? You know what they say about assume? Why should we simplify, when that could leave the actual error out of the problem domain? [url http://bit.ly/XK1HSB]BAAG that!
    PROD_DB - 10.0.0.11
    DEV1_DB - 10.0.0.21
    DEV2_DB - 10.0.0.22
    DEV3_DB - 10.0.0.23
    PROD_AppSrv - 10.0.0.14
    DEV_AppSrv - 10.0.0.8


    DEV_AppSrv is hosting three deploys of the same application, each connected to different DEVx_DB.
    Everything is on the same subnet, there is no firewall, everything is on the same switch (enterprise-level cisco).

    Prod_DB - Solaris10(M4000), 11.2.0.3
    DEV1_DB - Solaris10(old sparc machine) - db 10.2.0.5 (this one was frozen with upgrade due to problem with resetting connection)
    Well hells's bell's, it sounds like you have a jar file jdk version mismatch or classpath problem. Of course "sounds like" is probably one of the less reliable tracing methodologies. Why don't you post the actual error given back? Are there no log files on your middleware? Do you not have support? Have you searched the support site for the connection reset error?
    DEV2_DB - Solaris11(T5220), 11.2.0.3
    DEV3_DB - Solaris11(T5220), 11.2.0.3

    Now, how it works:

    PROD_APP + PROD_DB - fine
    DEV_APP + DEV1_DB - fine
    DEV_APP + DEV2_DB - reset
    DEV_APP + DEV3_DB - reset

    As you can see, middleware seems to be ok, when dev2 and dev3 was 10g on old machine (like dev1 does now) - all three instances were fine.
    Also upgrading from 10 to 11g2 causes no problem, because I'm running my prod instance on 11g on extremely high load (100 times bigger than dev) and it works.

    Only difference between prod_db and dev2/3_db is, that base os is Solaris11 on dev systems. Of course I can install 11gR2 on "old machine" (where DEV1_DB is working fine), but I guess it is not a good way. Downgrading Solaris11 to Solaris10... well, if everything else fails :)

    That's all about my env.
    In next post I'll give you more info.
  • 14. Re: Connection reset from JDBC to 11g
    721832 Newbie
    Currently Being Moderated
    Hi again!
    Thanks for all responses. Now it's time for some explains:

    1) In fact there are two types of connection, which are causing problem.
    1.1) Our own "wrapper", whatever it is. We are running third party system, which is based on Oracle iAS WebApp (Portal) and java desktop client. If someone have to use desktop client, to avoid direct connection to db, connection is established between client and my "wrapper". Wrapper a jar, it is started on app server, it has port 12345 opened and at startup it connects to DB (it has user/pass embedded in its conf file). What it does exactly - no idea. I'm waiting for response from that company which has written such wrapper. Wrapper version is very stable, as far as I remember it hasn't changed since 2009. Also, as far as I know, it is very sensible for network anomalies (someone has pointed -> tcp/ip on solaris11), if you have lost 1 packet, connection is broken. Forget about timeouts, retransmission and so on.

    The only message from wrapper's log:
    Thu Nov 15 15:06:40 CET 2012 Client_to_DB: IOException in loop PipedSocketStream Connection reset
    Thu Nov 15 15:06:40 CET 2012 Tunnels = 3
    Thu Nov 15 15:13:56 CET 2012 DB_to_Client: IOException in loop PipedSocketStream DB_to_Client: End of file in communication channel
    Thu Nov 15 15:13:56 CET 2012 Client_to_DB: IOException in loop PipedSocketStream Client_to_DB: End of file in communication channel
    Thu Nov 15 15:13:56 CET 2012 Tunnels = 2
    Thu Nov 15 15:16:33 CET 2012 Client_to_DB: IOException in loop PipedSocketStream Connection reset
    Thu Nov 15 15:16:34 CET 2012 Tunnels = 1
    Thu Nov 15 15:27:29 CET 2012 Client_to_DB: IOException in loop PipedSocketStream Connection reset
    Thu Nov 15 15:27:29 CET 2012 Tunnels = 0

    1.2) Our new app, based on API. API server is an jboss app (also, it is third party code). WebPortal connects to API, API connects to DB. I was unable to reproduce error, but I was noticed, that in random order sometimes it works like it should, but sometimes it won't start (sorry, I'll get more details on that case as soon as it's possible).

    2) @user11181920 So, connection always is established, but then it is broken.

    3) @sb92075 I know how listener works. Sorry for misspelling. I'm just wondering is it any chance to get some more info from DB side why connection was broken, not only such small message from wrapper (see above).

    4) @Justin Cave: Yes, I'm net admin. I can do tcpdump, SPAN, RSPAN or whatever, but unfortunately our problem is unpredictable. Fighting with 6-hours-network-capture-file is really ugly thing :) But I have no choice - tcpdump started, I'm waiting for an error.
    I'll try to establish connection from sqldeveloper from my desktop, but, like I said before, only wrapper is too sensible for tcp/ip anomalies and probably I'll get no error from sqldev.

    Rafal
1 2 Previous Next

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points