14 Replies Latest reply: Oct 21, 2012 2:02 PM by 896779 RSS

    Getting error when tried to create new user role from the application

    user1583259
      Hi,

      We have tried to create new user role from application, but got data base error "ROLESCRT: ERROR - CM90000-Database error." and after verified the weblogic manage server logs, we got the following exception while creating new user role.

      ####<Aug 31, 2012 4:29:51 PM IST> <Warning> <JDBC> <fss-pgs-app-01> <PGMS> <[ACTIVE] ExecuteThread: '78' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1346410791212> <BEA-001153> <Forcibly releasing inactive connection "weblogic.jdbc.wrapper.PoolConnection_oracle_jdbc_driver_T4CConnection@847f12" back into the connection pool "jdbcdb1", currently reserved by: java.lang.Exception
      at weblogic.jdbc.common.internal.ConnectionEnv.setup(ConnectionEnv.java:308)
      at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:314)
      at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:292)
      at weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:425)
      at weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:316)
      at weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:93)
      at weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:106)
      at weblogic.jdbc.pool.Driver.connect(Driver.java:149)
      at weblogic.jdbc.jts.Driver.getNonTxConnection(Driver.java:642)
      at weblogic.jdbc.jts.Driver.connect(Driver.java:124)
      at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:338)
      at com.aciworldwide.framework.database.DBConnectionManager$DBConnectionPool.newConnection(DBConnectionManager.java:181)
      at com.aciworldwide.framework.database.DBConnectionManager$DBConnectionPool.getConnection(DBConnectionManager.java:101)
      at com.aciworldwide.framework.database.DBConnectionManager.getConnection(DBConnectionManager.java:398)
      at com.aciworldwide.framework.database.DBProcess.<init>(DBProcess.java:31)
      at com.aciworldwide.application.UserSecurityAdmin.addRole(UserSecurityAdmin.java:51)
      at jsp_servlet.__procrole._jspService(__procrole.java:1007)
      at com.aciworldwide.framework.JSPPageBase.service(Unknown Source)
      at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
      at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
      at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:292)
      at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:175)
      at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3498)
      at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
      at weblogic.security.service.SecurityManager.runAs(Unknown Source)
      at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2180)
      at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2086)
      at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1406)
      at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
      at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
      .>

      Database: Oracle 11g

      we are using weblogic 10.3.2 and the jdbc datasource configured in weblogic itself. We have below setting in our environment.

      Initial capacity - 400
      Maximum capacity - 400
      Capacity Increment - 25
      Test Connections On Reserve - true
      Test Frequency - 120
      Seconds to Trust an Idle Pool Connection - 10
      Shrink Frequency - 900
      Connection Creation Retry Frequency - 60
      Inactive Connection Timeout - 250
      Connection Reserve Timeout - 10
      Ignore In-Use Connections - true
      Remove Infected Connections Enabled - true


      Please anyone help us to resolve the issue.

      Thanks in advance


      Thanks & Regards
      Tamil
        • 1. Re: Getting error when tried to create new user role from the application
          Kalyan Pasupuleti-Oracle
          Hi,

          This Warning can be ignored.

          <BEA-001153> <Forcibly releasing inactive connection "weblogic.jdbc.wrapper.PoolConnection_oracle_jdbc_driver


          this warning only appears when you have Inactive Connection timeout enabled.

          To avoid such warning make it Inactive Connection timeout as zero.

          But for Performance factor and avoid Connection pool leak this parameter is required.


          Check more specific to following role set error with in log.

          Regards,
          Kal
          • 2. Re: Getting error when tried to create new user role from the application
            Joe Weinstein-Oracle
            Please ignore Kalyan's post. He is incorrect.

            If you have a concern for pool connections being leaked, you need the 'inactive connection timeout'
            set so the pool does retract abandoned connections. However, the exception below is special,
            not because of the message, but because of the stack trace. This is a known bug and already fixed
            in a later version of WLS, where this exception is being thrown needlessly in a place where a connection
            has just been reserved and has not even yet been given to an application.

            So in general, this warning is an important sign that an application is either leaking connections
            or still has them but is not using them for longer than your configured limit.

            HTH,
            Joe
            • 3. Re: Getting error when tried to create new user role from the application
              user1583259
              Hi Joe,

              Thanks for your reply. Could you please let me know is it possible to resolve this warning by setting up any parameter in weblogic level.

              Waiting for your response.

              Thanks
              Tamil
              • 4. Re: Getting error when tried to create new user role from the application
                Joe Weinstein-Oracle
                To get rid of that particular error, for that reason, you need to contact Oracle support and
                get a patch for that bug, or upgrade to the latest WLS version.
                • 5. Re: Getting error when tried to create new user role from the application
                  user1583259
                  Hi Joe,

                  Once again thanks for your response. Could you please let me know how we can find if there is a connection leak, is there any parameter we can set in weblogic level to find the connection leak. If it is confirmed there is a connection leak, how we can resolve the same. Is there any possible settings we can do from weblogic level for further analysis.

                  We assumed that we are getting this warnings because of we have not enabled the "Remove Infected Connections" in weblogic data source. If we enable this parameter, the infected connections supposed to be closed. Since the weblogic throws warning while forcefully releasing the inactive connections back into the connection pool, we think that application is leaking the connection. Please correct me if I am wrong and kindly provide your suggestions.


                  Thanks,
                  Tamil
                  • 6. Re: Getting error when tried to create new user role from the application
                    Joe Weinstein-Oracle
                    Actually, I must correct myself. This message is probably showing you specifically where the
                    leak is. This code reserved the connection:

                    at com.aciworldwide.framework.database.DBConnectionManager$DBConnectionPool.newConnection(DBConnectionManager.java:181)
                    at com.aciworldwide.framework.database.DBConnectionManager$DBConnectionPool.getConnection(DBConnectionManager.java:101)
                    at com.aciworldwide.framework.database.DBConnectionManager.getConnection(DBConnectionManager.java:398)
                    at com.aciworldwide.framework.database.DBProcess.<init>(DBProcess.java:31)
                    at com.aciworldwide.application.UserSecurityAdmin.addRole(UserSecurityAdmin.java:51)

                    And is responsible for closing it when the connection is no longer needed. Instead, it did not close the
                    connection, and had either lost the reference or simply kept it without using it for your inactive connection
                    timeout setting.

                    That code is what you need to debug. Look for non-thread-safety, such as the connection being a class/instance
                    variable instead of a method-level variable. It should be a method-level variable, closed without fail by the end of a
                    method.
                    • 7. Re: Getting error when tried to create new user role from the application
                      user1583259
                      Hi Joe,

                      Kindly suggest is there any settings to do in weblogic level to find the error message stating that there is a connection leak and can clearly say's where exactly it's leaking.

                      Thanks,
                      Tamil
                      • 8. Re: Getting error when tried to create new user role from the application
                        user1583259
                        Hi Joe,

                        Even couple of days back also they have tried the same thing from application. But this time trace log has been enabled by application and we are getting below exception.

                        2012-09-04 17:38:24,940 trace.database.connection.com.aciworldwide.framework.database.DBProcess com.aciworldwide.application.UserSecurityAdmin.addRole(UserSecurityAdmin.java:154) [[ACTIVE] ExecuteThread: '51' for queue: 'weblogic.kernel.Default (self-tuning)'] - SQLException occurred: java.sql.SQLException: Connection has already been closed. SQLState: null ErrorCode: 0



                        But from weblogic managed server logs we got below warning 2 minutes before when the above exception occured.


                        ####<Sep 4, 2012 5:36:03 PM IST> <Warning> <JDBC> <fss-pgs-app-01> <PGMS> <[ACTIVE] ExecuteThread: '42' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1346760363923> <BEA-001153> <Forcibly releasing inactive connection "weblogic.jdbc.wrapper.PoolConnection_oracle_jdbc_driver_T4CConnection@a40de3" back into the connection pool "jdbcdb1", currently reserved by: java.lang.Exception
                             at weblogic.jdbc.common.internal.ConnectionEnv.setup(ConnectionEnv.java:308)
                             at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:314)
                             at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:292)
                             at weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:425)
                             at weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:316)
                             at weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:93)
                             at weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:106)
                             at weblogic.jdbc.pool.Driver.connect(Driver.java:149)
                             at weblogic.jdbc.jts.Driver.getNonTxConnection(Driver.java:642)
                             at weblogic.jdbc.jts.Driver.connect(Driver.java:124)
                             at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:338)
                             at com.aciworldwide.framework.database.DBConnectionManager$DBConnectionPool.newConnection(DBConnectionManager.java:181)
                             at com.aciworldwide.framework.database.DBConnectionManager$DBConnectionPool.getConnection(DBConnectionManager.java:101)
                             at com.aciworldwide.framework.database.DBConnectionManager.getConnection(DBConnectionManager.java:398)
                             at com.aciworldwide.framework.database.DBProcess.<init>(DBProcess.java:31)
                             at com.aciworldwide.application.UserSecurityAdmin.addRole(UserSecurityAdmin.java:51)
                             at jsp_servlet.__procrole._jspService(__procrole.java:1007)
                             at com.aciworldwide.framework.JSPPageBase.service(Unknown Source)
                             at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
                             at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
                             at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:292)
                             at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:175)
                             at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3498)
                             at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
                             at weblogic.security.service.SecurityManager.runAs(Unknown Source)
                             at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2180)
                             at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2086)
                             at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1406)
                             at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
                             at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
                        .>


                        Not sure this time how the connection has been closed, at which point the connection has closed and where (whether it is closed by application or DB).


                        Thanks,
                        Tamil

                        Edited by: user1583259 on Sep 6, 2012 7:07 AM

                        Edited by: user1583259 on Sep 8, 2012 2:34 AM

                        Edited by: user1583259 on Sep 8, 2012 2:35 AM
                        • 9. Re: Getting error when tried to create new user role from the application
                          Joe Weinstein-Oracle
                          that last exception is a separate issue. The connection was closed from within the driver, for internal failure reasons.
                          • 10. Re: Getting error when tried to create new user role from the application
                            Joe Weinstein-Oracle
                            And I cannot be more clear about the leak. I told you exactly where to look and what to do:

                            This message is probably showing you specifically where the
                            leak is. This code reserved the connection:

                            at com.aciworldwide.framework.database.DBConnectionManager$DBConnectionPool.newConnection(DBConnectionManager.java:181)
                            at com.aciworldwide.framework.database.DBConnectionManager$DBConnectionPool.getConnection(DBConnectionManager.java:101)
                            at com.aciworldwide.framework.database.DBConnectionManager.getConnection(DBConnectionManager.java:398)
                            at com.aciworldwide.framework.database.DBProcess.<init>(DBProcess.java:31)
                            at com.aciworldwide.application.UserSecurityAdmin.addRole(UserSecurityAdmin.java:51)

                            And is responsible for closing it when the connection is no longer needed. Instead, it did not close the
                            connection, and had either lost the reference or simply kept it without using it for your inactive connection
                            timeout setting.

                            That code is what you need to debug. Look for non-thread-safety, such as the connection being a class/instance
                            variable instead of a method-level variable. It should be a method-level variable, closed without fail by the end of a
                            method.
                            • 11. Re: Getting error when tried to create new user role from the application
                              user1583259
                              Hi Joe,

                              I guess we can find where the connection leak is happening by enabling the parameter "Profile Connection Leak" on data source configuration.
                              I have seen this parameter from the weblogic console Data sources-> "Name of the Data Source" -> Configuration -> Diagnostics

                              This parameter's description was "Collect profile information about threads that have reserved a connection from the data source and the connection leaked (was not properly returned to the pool of connections)".

                              Can you please suggest me whether we can enable this parameter to check and also what exactly this parameter will do. If we enable this parameter, what kind of output we can get.

                              There will be any impact if we enable this parameter...


                              Thanks,
                              Tamil

                              Edited by: user1583259 on Sep 8, 2012 2:33 AM
                              • 12. Re: Getting error when tried to create new user role from the application
                                Joe Weinstein-Oracle
                                Don't guess. I am telling you (and that stack trace is telling you) where the leak is. That code
                                reserved a pool connection and that connection was never closed. You have to examine that
                                application code to figure out why.
                                • 13. Re: Getting error when tried to create new user role from the application
                                  795854
                                  Just mentioned as a known bug, could you advise the bug number?
                                  • 14. Re: Getting error when tried to create new user role from the application
                                    896779
                                    Please take a look at the Oracle Support site, ID 1329416.1 (WebLogic Server JDBC Connections Being Forcibly Released with Error BEA-001153: Forcibly releasing inactive connection )

                                    It has reference to BUG:9020369 and the solution. Good luck.

                                    Lawrence Manickam
                                    http://toyork.blogspot.ca