1 2 3 4 Previous Next 45 Replies Latest reply on Jun 5, 2020 12:07 PM by thatJeffSmith-Oracle

    ORDS 19.4 is breaking

    Test2

      Hello,

       

      Kris Rice-Oracle thatJeffSmith-Oracle,

      After upgrading ORDS from 18.4 to 19.4, ORDS is breaking every now and then, and we started getting this error

      Screen Shot 2020-01-05 at 6.40.44 PM.png

       

      Env: TomCat9. Windows server 2016, used mainly for APEX 19.2 on Oracle 18.3 SE

       

      05-Jan-2020 11:56:03.542 INFO [Catalina-utility-1] . Oracle REST Data Services initialized

      Oracle REST Data Services version : 19.4.0.r3521226

      Oracle REST Data Services server info: Apache Tomcat/9.0.14

       

       

      05-Jan-2020 11:56:04.299 INFO [Catalina-utility-1] org.apache.catalina.core.StandardContext.reload Reloading Context with name [] is completed

      05-Jan-2020 11:56:05.499 SEVERE [http-nio-8080-exec-40] oracle.dbtools.common.di.JDKLogSink.log No provider found for:  requires oracle.dbtools.http.filters.Filters

      oracle.dbtools.common.di.MissingDependencyException: No provider found for:  requires oracle.dbtools.http.filters.Filters

      at oracle.dbtools.common.di.DependencyInjectionException.convert(DependencyInjectionException.java:56)

      at oracle.dbtools.common.di.Services.acquire(Services.java:160)

      at oracle.dbtools.http.entrypoint.EntryPoint.service(EntryPoint.java:79)

      at oracle.dbtools.http.entrypoint.EntryPointServlet.service(EntryPointServlet.java:117)

      at oracle.dbtools.entrypoint.WebApplicationRequestEntryPoint.service(WebApplicationRequestEntryPoint.java:50)

      at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)

      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)

      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

      at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)

      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)

      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

      at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:239)

      at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:215)

      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)

      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

      at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)

      at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)

      at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:490)

      at org.apache.catalina.valves.rewrite.RewriteValve.invoke(RewriteValve.java:542)

      at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)

      at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)

      at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:668)

      at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:668)

      at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:679)

      at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)

      at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)

      at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408)

      at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)

      at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:834)

      at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1417)

      at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)

      at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

      at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

      at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

      at java.lang.Thread.run(Unknown Source)

      Caused by: oracle.dbtools.injector.impl.MissingDependencyException: No provider found for:  requires oracle.dbtools.http.filters.Filters

      at oracle.dbtools.injector.impl.MissingDependencyException.noProvider(MissingDependencyException.java:75)

      at oracle.dbtools.injector.impl.MissingDependencyException.noProvider(MissingDependencyException.java:62)

      at oracle.dbtools.injector.impl.InstanceProviderAdapter.get(InstanceProviderAdapter.java:30)

      at oracle.dbtools.common.di.Services.acquire(Services.java:158)

      ... 33 more

      Caused by: oracle.dbtools.plugin.api.types.TypeDependencyNotAvailableException: The type: (provides oracle.dbtools.http.filters.Filters) could not be instantiated due to the following error: No provider found for:  requires oracle.dbtools.rest.resource.modules.xml.StaticModules

      at oracle.dbtools.plugin.api.types.TypeDependencyNotAvailableException.from(TypeDependencyNotAvailableException.java:92)

      at oracle.dbtools.plugin.api.di.ResolvedInstances.get(ResolvedInstances.java:44)

      at oracle.dbtools.injector.impl.InstanceProviderAdapter.get(InstanceProviderAdapter.java:28)

      ... 34 more

      Caused by: oracle.dbtools.injector.impl.MissingDependencyException: No provider found for:  requires oracle.dbtools.rest.resource.modules.xml.StaticModules

      at oracle.dbtools.injector.impl.MissingDependencyException.noProvider(MissingDependencyException.java:75)

      at oracle.dbtools.injector.impl.MissingDependencyException.noProvider(MissingDependencyException.java:62)

      at oracle.dbtools.injector.impl.DependencyInjectionException.instantationError(DependencyInjectionException.java:107)

      at oracle.dbtools.injector.impl.ServiceFactory.instantiate(ServiceFactory.java:122)

      at oracle.dbtools.injector.impl.ServiceFactory.access$300(ServiceFactory.java:29)

      at oracle.dbtools.injector.impl.ServiceFactory$ResolvedDependenciesFactory.newInstance(ServiceFactory.java:366)

      at oracle.dbtools.injector.impl.ServiceFactory.instantiate(ServiceFactory.java:102)

      at oracle.dbtools.injector.impl.ServiceFactory.access$200(ServiceFactory.java:29)

      at oracle.dbtools.injector.impl.ServiceFactory$InstanceFactory.newInstance(ServiceFactory.java:348)

      at oracle.dbtools.injector.impl.Singletons.newInstance(Singletons.java:121)

      at oracle.dbtools.injector.impl.SingletonsBase.get(SingletonsBase.java:35)

      at oracle.dbtools.injector.impl.SingletonsBase.get(SingletonsBase.java:49)

      at oracle.dbtools.injector.impl.ServiceFactory.resolveDependencies(ServiceFactory.java:244)

      at oracle.dbtools.injector.impl.ServiceFactory.instantiate(ServiceFactory.java:97)

      at oracle.dbtools.injector.impl.ServiceFactory.access$200(ServiceFactory.java:29)

      at oracle.dbtools.injector.impl.ServiceFactory$InstanceFactory.newInstance(ServiceFactory.java:348)

      at oracle.dbtools.injector.impl.Singletons.newInstance(Singletons.java:121)

      at oracle.dbtools.injector.impl.SingletonsBase.get(SingletonsBase.java:35)

      at oracle.dbtools.injector.impl.SingletonsBase.get(SingletonsBase.java:49)

      at oracle.dbtools.injector.impl.ServiceFactory.newInstance(ServiceFactory.java:55)

      at oracle.dbtools.injector.impl.InjectorImpl.select(InjectorImpl.java:183)

      at oracle.dbtools.plugin.api.di.InstanceLocator.select(InstanceLocator.java:42)

      ... 34 more

      Caused by: oracle.dbtools.plugin.api.types.TypeDependencyNotAvailableException: The type: (provides oracle.dbtools.rest.resource.modules.xml.StaticModules) could not be instantiated due to the following error: javax.xml.transform.TransformerException: java.lang.IllegalArgumentException: type: -1

      at oracle.dbtools.plugin.api.types.TypeDependencyNotAvailableException.from(TypeDependencyNotAvailableException.java:92)

      at oracle.dbtools.plugin.api.types.TypeDependencyNotAvailableException.from(TypeDependencyNotAvailableException.java:40)

      at oracle.dbtools.plugin.api.types.TypeDependencies.newInstance(TypeDependencies.java:154)

      at oracle.dbtools.plugin.api.types.TypeReflections$ReflectiveInstantiator.load(TypeReflections.java:568)

      at oracle.dbtools.injector.impl.ServiceFactory.instantiate(ServiceFactory.java:119)

      ... 52 more

      Caused by: java.lang.IllegalArgumentException: javax.xml.transform.TransformerException: java.lang.IllegalArgumentException: type: -1

      at oracle.dbtools.rest.resource.modules.xml.Content.clone(Content.java:75)

      at oracle.dbtools.rest.resource.modules.xml.Content.access$100(Content.java:18)

      at oracle.dbtools.rest.resource.modules.xml.Content$Builder.xml(Content.java:143)

      at oracle.dbtools.rest.resource.modules.xml.Handler.content(Handler.java:85)

      at oracle.dbtools.rest.resource.modules.xml.Handler.content(Handler.java:22)

      at oracle.dbtools.rest.resource.modules.xml.XMLResourceModuleSet$Builder.buildPathTemplates(XMLResourceModuleSet.java:150)

      at oracle.dbtools.rest.resource.modules.xml.XMLResourceModuleSet$Builder.read(XMLResourceModuleSet.java:213)

      at oracle.dbtools.rest.resource.modules.xml.XMLResourceModuleSet$Builder.add(XMLResourceModuleSet.java:94)

      at oracle.dbtools.rest.resource.modules.xml.StaticModules.add(StaticModules.java:77)

      at oracle.dbtools.rest.resource.modules.xml.StaticModules.<init>(StaticModules.java:49)

      at java.lang.invoke.MethodHandle.invokeWithArguments(Unknown Source)

      at oracle.dbtools.plugin.api.types.TypeDependencies.newInstance(TypeDependencies.java:151)

      ... 54 more

      Caused by: javax.xml.transform.TransformerException: java.lang.IllegalArgumentException: type: -1

      at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform(Unknown Source)

      at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform(Unknown Source)

      at oracle.dbtools.common.xml.JAXPBase.clone(JAXPBase.java:61)

      at oracle.dbtools.rest.resource.modules.xml.Content.clone(Content.java:73)

      ... 65 more

      Caused by: java.lang.IllegalArgumentException: type: -1

      at com.sun.org.apache.xerces.internal.dom.DeferredDocumentImpl.getNodeObject(Unknown Source)

      at com.sun.org.apache.xerces.internal.dom.DeferredElementNSImpl.synchronizeData(Unknown Source)

      at com.sun.org.apache.xerces.internal.dom.ElementImpl.getNodeName(Unknown Source)

      at com.sun.org.apache.xalan.internal.xsltc.trax.DOM2TO.parse(Unknown Source)

      at com.sun.org.apache.xalan.internal.xsltc.trax.DOM2TO.parse(Unknown Source)

      at com.sun.org.apache.xalan.internal.xsltc.trax.DOM2TO.parse(Unknown Source)

      at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transformIdentity(Unknown Source)

      ... 69 more

        • 1. Re: ORDS 19.4 is breaking
          thatJeffSmith-Oracle

          it would be useful to know:

          + JDK version.

          + if the issue is reported in the logs at startup or only after a request is received.

          + if they are adding any plugins to ords.war ( unlikely but it would be good to rule it out )

           

          Also, what url is being requested when this occurs?

           

          And finally, is it possible this isn't the entire stack/error message?

          • 2. Re: ORDS 19.4 is breaking
            EJ-Egyed

            I have received the error as well.  Answers to your questions below:

             

            1. Java version:

            > java -version

            java version "13" 2019-09-17

            Java(TM) SE Runtime Environment (build 13+33)

            Java HotSpot(TM) 64-Bit Server VM (build 13+33, mixed mode, sharing)

             

            2. Nothing reported on startup.  Only when a request is received.  I was able to navigate a few pages before running into the error for the first time. Then refreshing the page seemed to resolve the issue.

             

            3. No plugins being added to ords.war

             

            I received this error when attempting to navigate to the APEX page designer page (Application 4000 Page 4500).  Here are a few lines from my Tomcat localhost_access_log file:

             

            (IP Obsured) - - [06/Jan/2020:09:10:21 -0500] "POST /ords/wwv_flow.ajax HTTP/1.1" 500 16163

            (IP Obsured) - - [06/Jan/2020:09:10:21 -0500] "POST /ords/wwv_flow.ajax HTTP/1.1" 500 16163

            (IP Obsured) - - [06/Jan/2020:09:10:23 -0500] "GET /ords/f?p=4000:4500:12748700618835::NO:1,4150:FB_FLOW_ID,FB_FLOW_PAGE_ID,F4000_P1_FLOW,F4000_P4150_GOTO_PAGE,F4000_P1_PAGE:106,1,106,1,1 HTTP/1.1" 500 16163

            (IP Obsured) - - [06/Jan/2020:09:10:25 -0500] "GET /ords/f?p=4000:4500:12748700618835::NO:1,4150:FB_FLOW_ID,FB_FLOW_PAGE_ID,F4000_P1_FLOW,F4000_P4150_GOTO_PAGE,F4000_P1_PAGE:106,1,106,1,1 HTTP/1.1" 500 16163

             

            Today was my first time running into the error, but I have now turned on debug.PrintDebugToScreen to hopefully capture more information in the future if it happens again.

            • 3. Re: ORDS 19.4 is breaking
              m_yasirali

              Hi thatJeffSmith-Oracle,

              Following is the information you requested.

              1. The + JDK version.

              Java version "1.8.0_201"

              Java(TM) SE Runtime Environment (build 1.8.0_201-b09)

              Java HotSpot(TM) 64-Bit Server VM (build 25.201-b09, mixed mode)

               

              2. The error arises anytime during the day, so we restart the tomcat to solve the error.  Here, is a the tomcat log file attached:

              http://famproperties.com/assets/tomcat.log

               

              3. No plugins being added to ords.war.

               

              Thanks advance for a prompt reply.

              • 4. Re: ORDS 19.4 is breaking
                thatJeffSmith-Oracle

                your error log looks completely different than the one user test2 opened the thread with

                • 5. Re: ORDS 19.4 is breaking
                  thatJeffSmith-Oracle

                  everyone on this thread - please include your specific error stack messages

                   

                  and, are these intermittent (sounds like it is) issues? If so, any idea of what the load is like, number of users sending concurrent requests?

                  • 6. Re: ORDS 19.4 is breaking
                    thatJeffSmith-Oracle

                    We're now able to reproduce this issue and are evaluating fixes...thanks for the report folks.

                    • 7. Re: ORDS 19.4 is breaking
                      EJ-Egyed

                      Glad to hear! Are there any configuration changes we can make to prevent this error until a fix is released?

                      • 8. Re: ORDS 19.4 is breaking
                        thatJeffSmith-Oracle

                        No, it's a boring Java library loading issue. It's also benign from what we can tell.

                        • 9. Re: ORDS 19.4 is breaking
                          m_yasirali

                          Hi thatJeffSmith-Oracle,

                          Hope you are fine. This is today's log when the error occurred. Kindly have a look into this, as this is intermittently happening and we have to restart Apache and Tomcat to fix it.

                          https://famproperties.com/assets/catalina.2020-01-12.log

                          • 10. Re: ORDS 19.4 is breaking
                            thatJeffSmith-Oracle

                            This is a completely different error stack than the issue orginally being discussed.

                             

                            Please start a new thread, and include all of the relevant version info
                              Version of ORDS
                                               Java
                                               Tomcat

                                              APEX

                                              Oracle Database

                             

                            The logs are saying the DB is killing the session, so I need you to check your Oracle DB Alert logs as well.

                             

                            I'm not familiar with the ORA being called for apex and the ajax bits, but we'll dig into it first thing tomorrow.

                            • 11. Re: ORDS 19.4 is breaking
                              thatJeffSmith-Oracle

                              something is going on with apex...could be a bug  on the apex side, recommend you open a SR

                              • 12. Re: ORDS 19.4 is breaking
                                Olafur T

                                This is happening also to me after upgrade to 19.4.  I need to simply stop tomcat again, wait 1-2 seconds and start again. (with frontend down)

                                 

                                Affects ORDS services. That is restart tomcat, no using APEX. All ORDS services will have that error, sometimes Tomcat needs 2 restarts. In the end it works fine and will run until Tomcat is restarted again.

                                 

                                Biggest problem is that all restarts need to be done manually (no automatic restarts since it will not get better until Tomcat is restarted)

                                 

                                This issue plus Performance issues under constant high load  makes it a drag since the DB needs to be manually restarted once a day with supervision.

                                 

                                Oli

                                • 13. Re: ORDS 19.4 is breaking
                                  SveinOle

                                  Have you had any progress on these, now, two issues?

                                   

                                  We are still on 19.2, waiting to upgrade to 19.4 (was planned last weekend, but postponed when new issues with 19.4 was discussed here).

                                   

                                  We restart manually each 48 hours now, time consuming and should be unneccesary.

                                  • 14. Re: ORDS 19.4 is breaking
                                    SveinOle

                                    Or did the 19.4 at least resolve the first issue with the high load on that specific query that ORDS is executing?

                                    1 2 3 4 Previous Next