8 Replies Latest reply on May 27, 2011 2:30 AM by 863323

    server hang for waitting test sql response

    863323
      hi all

      I have deploy ear to weblogic and use jndi to lookup weblogic datasource pool.

      sometimes the server will hang about 10 miniute.

      the thread dump show that a thread wait weblogic test sql to response.

      I had ever try to change jdbc driver ,but the problem is still exist.

      any suggestion?

      ===weblogic datasource config===
      url : jdbc:jtds:sybase://192.168.1.1:5100/NIAIMM2
      driver : net.sourceforge.jtds.jdbc.Driver
      Initial Capacity: 5
      Maximum Capacity: 150
      Capacity Increment: 1
      Test Connections On Reserve : unchecked
      Test Frequency:120
      Test Table Name:SQL SELECT 1
      Seconds to Trust an Idle Pool Connection:10
      Shrink Frequency:900
      Init SQL:
      Connection Creation Retry Frequency:60
      Login Delay:0
      Inactive Connection Timeout:0
      Maximum Waiting for Connection:2147483647
      Connection Reserve Timeout:10
      Statement Timeout:3
      Ignore In-Use Connections:checked
      Pinned-To-Thread:unchecked
      Remove Infected Connections Enabled:checked

      ===environment===
      Weblogic 10.3.3
      Sybase ASE 15.5
      Linux ubuntu

      ===thread dump===

      "[STUCK] ExecuteThread: '4' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x0000000042b7d000 nid=0x9a2 runnable [0x00007ff979c16000]
      java.lang.Thread.State: RUNNABLE
           at java.net.SocketInputStream.socketRead0(Native Method)
           at java.net.SocketInputStream.read(SocketInputStream.java:129)
           at java.io.DataInputStream.readFully(DataInputStream.java:178)
           at java.io.DataInputStream.readFully(DataInputStream.java:152)
           at net.sourceforge.jtds.jdbc.SharedSocket.readPacket(SharedSocket.java:846)
           at net.sourceforge.jtds.jdbc.SharedSocket.getNetPacket(SharedSocket.java:727)
           - locked <0x00007ffa0732c2a0> (a java.util.ArrayList)
           at net.sourceforge.jtds.jdbc.ResponseStream.getPacket(ResponseStream.java:466)
           at net.sourceforge.jtds.jdbc.ResponseStream.read(ResponseStream.java:103)
           at net.sourceforge.jtds.jdbc.ResponseStream.peek(ResponseStream.java:88)
           at net.sourceforge.jtds.jdbc.TdsCore.wait(TdsCore.java:3932)
           at net.sourceforge.jtds.jdbc.TdsCore.executeSQL(TdsCore.java:1046)
           - locked <0x00007ffa07337830> (a net.sourceforge.jtds.jdbc.TdsCore)
           at net.sourceforge.jtds.jdbc.JtdsStatement.executeSQL(JtdsStatement.java:537)
           at net.sourceforge.jtds.jdbc.JtdsPreparedStatement.execute(JtdsPreparedStatement.java:558)
           - locked <0x00007ffa0732bb90> (a net.sourceforge.jtds.jdbc.ConnectionJDBC3)
           at weblogic.jdbc.common.internal.ConnectionEnv.testInternal(ConnectionEnv.java:821)
           at weblogic.jdbc.common.internal.ConnectionEnv.test(ConnectionEnv.java:756)
           at weblogic.jdbc.common.internal.ConnectionEnv.test(ConnectionEnv.java:495)
           at weblogic.common.resourcepool.ResourcePoolImpl.testResource(ResourcePoolImpl.java:2345)
           at weblogic.common.resourcepool.ResourcePoolImpl.checkResource(ResourcePoolImpl.java:1657)
           at weblogic.common.resourcepool.ResourcePoolImpl.checkAndReturnResource(ResourcePoolImpl.java:1564)
           at weblogic.common.resourcepool.ResourcePoolImpl.checkAndReturnResource(ResourcePoolImpl.java:1553)
           at weblogic.common.resourcepool.ResourcePoolImpl.testUnusedResources(ResourcePoolImpl.java:2001)
           at weblogic.common.resourcepool.ResourcePoolImpl.access$900(ResourcePoolImpl.java:40)
           at weblogic.common.resourcepool.ResourcePoolImpl$ResourcePoolMaintanenceTask.timerExpired(ResourcePoolImpl.java:2433)
           at weblogic.timers.internal.TimerImpl.run(TimerImpl.java:273)
           at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:528)
           at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
           at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)

      Edited by: 860320 on May 20, 2011 3:56 AM
        • 1. Re: server hang for waitting test sql response
          Joe Weinstein-Oracle
          You need to check your DBMS. there is something seriously wrong if anyone has to wait for a "SELECT 1" query to return...
          • 2. Re: server hang for waitting test sql response
            dvohra21
            Could be a bug.
            http://sourceforge.net/tracker/?func=detail&aid=3020730&group_id=33291&atid=407762

            Which other driver than jtds generates the error?
            • 3. Re: server hang for waitting test sql response
              863323
              I have ever change driver from sybase jconn3.jar(jconnect 6.0.5) to jtds-1.2.5.jar
              The problem is the same.
              • 4. Re: server hang for waitting test sql response
                Joe Weinstein-Oracle
                Yeah, it's never going to have been the driver. There aren't many ways
                a driver can send SQL to the DBMS and wait for a response. You should
                definitely get a DBA to determine what the DBMS thinks is going on for
                that session. However, you might want to look into an OS problem with
                socket processing. One simple test is to do the same client code from
                some other sort of box, and/or get a network sniffer to compare the
                packets to-and-from the client running on the two types of box...
                • 5. Re: server hang for waitting test sql response
                  863323
                  thanks for your help.

                  I will try to check if the network problem .

                  And I will check the db log from sybase DBA.
                  • 6. Re: server hang for waitting test sql response
                    863323
                    hi all
                    I have encounter the problem again at 09:26 AM
                    and the DBA give me the log as follow:

                    does anyone know what RepAgent(6) is?
                    is it "ct_results()" cause the db to wait sql response.
                    .

                    01:09:00000:01089:2011/05/25 08:37:31.08 server A client process exited abnormally, or a network error was enco
                    untered. Unless other errors occurred, continue processing normally.
                    01:09:00000:01089:2011/05/25 08:37:31.08 kernel extended error information: hostname: login: utfadm
                    01:12:00000:00000:2011/05/25 08:40:18.16 kernel nrpacket: recv, Connection timed out, spid: 2457, suid: 3
                    01:12:00000:00000:2011/05/25 08:40:18.16 kernel extended error information: hostname: n-chk-app5 login: utfadm
                    01:05:00000:00000:2011/05/25 08:40:18.77 kernel nrpacket: recv, Connection timed out, spid: 2593, suid: 3
                    01:05:00000:00000:2011/05/25 08:40:18.77 kernel extended error information: hostname: n-chk-app5 login: utfadm
                    01:09:00000:02329:2011/05/25 09:13:31.38 kernel Cannot send, host process disconnected: 0000016a suid: 3
                    01:09:00000:02329:2011/05/25 09:13:31.38 server Error: 1608, Severity: 18, State: 3
                    01:09:00000:02329:2011/05/25 09:13:31.38 server A client process exited abnormally, or a network error was enco
                    untered. Unless other errors occurred, continue processing normally.
                    01:09:00000:02329:2011/05/25 09:13:31.38 kernel extended error information: hostname: login: utfadm
                    01:00:00000:02161:2011/05/25 09:29:19.80 server RepAgent(6): Received the following communications error messag
                    e: Msg 05030506: ct_results(): network packet layer: internal net library error: Net-Lib protocol driver call to
                    read data failed.
                    01:08:00000:00369:2011/05/25 09:44:10.01 kernel Cannot send, host process disconnected: 0000016a suid: 3
                    01:08:00000:00369:2011/05/25 09:44:10.01 server Error: 1608, Severity: 18, State: 3
                    01:08:00000:00369:2011/05/25 09:44:10.01 server A client process exited abnormally, or a network error was enco
                    untered. Unless other errors occurred, continue processing normally.
                    01:08:00000:00369:2011/05/25 09:44:10.01 kernel extended error information: hostname: login: utfadm
                    • 7. Re: server hang for waitting test sql response
                      dvohra21
                      Though not a JDBC error, does the Net-Lib protocol driver call to read data failed error occur periodically? The network seems to be timing out. Is the network timeout set low?
                      • 8. Re: server hang for waitting test sql response
                        863323
                        Could you tell me how to check the network timeout value? Is it depends on OS?

                        Our AP Server is run on Ubuntu system.

                        thank you.