Forum Stats

  • 3,740,795 Users
  • 2,248,322 Discussions
  • 7,861,444 Comments

Discussions

ORDS 19.4 is breaking

Test2
Test2 Member Posts: 181 Blue Ribbon

Hello,

@Kris Rice-Oracle @thatJeffSmith-Oracle,

After upgrading ORDS from 18.4 to 19.4, ORDS is breaking every now and then, and we started getting this error

Screen Shot 2020-01-05 at 6.40.44 PM.png

Env: TomCat9. Windows server 2016, used mainly for APEX 19.2 on Oracle 18.3 SE

05-Jan-2020 11:56:03.542 INFO [Catalina-utility-1] . Oracle REST Data Services initialized

Oracle REST Data Services version : 19.4.0.r3521226

Oracle REST Data Services server info: Apache Tomcat/9.0.14

05-Jan-2020 11:56:04.299 INFO [Catalina-utility-1] org.apache.catalina.core.StandardContext.reload Reloading Context with name [] is completed

05-Jan-2020 11:56:05.499 SEVERE [http-nio-8080-exec-40] oracle.dbtools.common.di.JDKLogSink.log No provider found for:  requires oracle.dbtools.http.filters.Filters

oracle.dbtools.common.di.MissingDependencyException: No provider found for:  requires oracle.dbtools.http.filters.Filters

at oracle.dbtools.common.di.DependencyInjectionException.convert(DependencyInjectionException.java:56)

at oracle.dbtools.common.di.Services.acquire(Services.java:160)

at oracle.dbtools.http.entrypoint.EntryPoint.service(EntryPoint.java:79)

at oracle.dbtools.http.entrypoint.EntryPointServlet.service(EntryPointServlet.java:117)

at oracle.dbtools.entrypoint.WebApplicationRequestEntryPoint.service(WebApplicationRequestEntryPoint.java:50)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:239)

at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:215)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)

at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)

at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:490)

at org.apache.catalina.valves.rewrite.RewriteValve.invoke(RewriteValve.java:542)

at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)

at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)

at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:668)

at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:668)

at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:679)

at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)

at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)

at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408)

at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)

at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:834)

at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1417)

at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)

at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

at java.lang.Thread.run(Unknown Source)

Caused by: oracle.dbtools.injector.impl.MissingDependencyException: No provider found for:  requires oracle.dbtools.http.filters.Filters

at oracle.dbtools.injector.impl.MissingDependencyException.noProvider(MissingDependencyException.java:75)

at oracle.dbtools.injector.impl.MissingDependencyException.noProvider(MissingDependencyException.java:62)

at oracle.dbtools.injector.impl.InstanceProviderAdapter.get(InstanceProviderAdapter.java:30)

at oracle.dbtools.common.di.Services.acquire(Services.java:158)

... 33 more

Caused by: oracle.dbtools.plugin.api.types.TypeDependencyNotAvailableException: The type: (provides oracle.dbtools.http.filters.Filters) could not be instantiated due to the following error: No provider found for:  requires oracle.dbtools.rest.resource.modules.xml.StaticModules

at oracle.dbtools.plugin.api.types.TypeDependencyNotAvailableException.from(TypeDependencyNotAvailableException.java:92)

at oracle.dbtools.plugin.api.di.ResolvedInstances.get(ResolvedInstances.java:44)

at oracle.dbtools.injector.impl.InstanceProviderAdapter.get(InstanceProviderAdapter.java:28)

... 34 more

Caused by: oracle.dbtools.injector.impl.MissingDependencyException: No provider found for:  requires oracle.dbtools.rest.resource.modules.xml.StaticModules

at oracle.dbtools.injector.impl.MissingDependencyException.noProvider(MissingDependencyException.java:75)

at oracle.dbtools.injector.impl.MissingDependencyException.noProvider(MissingDependencyException.java:62)

at oracle.dbtools.injector.impl.DependencyInjectionException.instantationError(DependencyInjectionException.java:107)

at oracle.dbtools.injector.impl.ServiceFactory.instantiate(ServiceFactory.java:122)

at oracle.dbtools.injector.impl.ServiceFactory.access$300(ServiceFactory.java:29)

at oracle.dbtools.injector.impl.ServiceFactory$ResolvedDependenciesFactory.newInstance(ServiceFactory.java:366)

at oracle.dbtools.injector.impl.ServiceFactory.instantiate(ServiceFactory.java:102)

at oracle.dbtools.injector.impl.ServiceFactory.access$200(ServiceFactory.java:29)

at oracle.dbtools.injector.impl.ServiceFactory$InstanceFactory.newInstance(ServiceFactory.java:348)

at oracle.dbtools.injector.impl.Singletons.newInstance(Singletons.java:121)

at oracle.dbtools.injector.impl.SingletonsBase.get(SingletonsBase.java:35)

at oracle.dbtools.injector.impl.SingletonsBase.get(SingletonsBase.java:49)

at oracle.dbtools.injector.impl.ServiceFactory.resolveDependencies(ServiceFactory.java:244)

at oracle.dbtools.injector.impl.ServiceFactory.instantiate(ServiceFactory.java:97)

at oracle.dbtools.injector.impl.ServiceFactory.access$200(ServiceFactory.java:29)

at oracle.dbtools.injector.impl.ServiceFactory$InstanceFactory.newInstance(ServiceFactory.java:348)

at oracle.dbtools.injector.impl.Singletons.newInstance(Singletons.java:121)

at oracle.dbtools.injector.impl.SingletonsBase.get(SingletonsBase.java:35)

at oracle.dbtools.injector.impl.SingletonsBase.get(SingletonsBase.java:49)

at oracle.dbtools.injector.impl.ServiceFactory.newInstance(ServiceFactory.java:55)

at oracle.dbtools.injector.impl.InjectorImpl.select(InjectorImpl.java:183)

at oracle.dbtools.plugin.api.di.InstanceLocator.select(InstanceLocator.java:42)

... 34 more

Caused by: oracle.dbtools.plugin.api.types.TypeDependencyNotAvailableException: The type: (provides oracle.dbtools.rest.resource.modules.xml.StaticModules) could not be instantiated due to the following error: javax.xml.transform.TransformerException: java.lang.IllegalArgumentException: type: -1

at oracle.dbtools.plugin.api.types.TypeDependencyNotAvailableException.from(TypeDependencyNotAvailableException.java:92)

at oracle.dbtools.plugin.api.types.TypeDependencyNotAvailableException.from(TypeDependencyNotAvailableException.java:40)

at oracle.dbtools.plugin.api.types.TypeDependencies.newInstance(TypeDependencies.java:154)

at oracle.dbtools.plugin.api.types.TypeReflections$ReflectiveInstantiator.load(TypeReflections.java:568)

at oracle.dbtools.injector.impl.ServiceFactory.instantiate(ServiceFactory.java:119)

... 52 more

Caused by: java.lang.IllegalArgumentException: javax.xml.transform.TransformerException: java.lang.IllegalArgumentException: type: -1

at oracle.dbtools.rest.resource.modules.xml.Content.clone(Content.java:75)

at oracle.dbtools.rest.resource.modules.xml.Content.access$100(Content.java:18)

at oracle.dbtools.rest.resource.modules.xml.Content$Builder.xml(Content.java:143)

at oracle.dbtools.rest.resource.modules.xml.Handler.content(Handler.java:85)

at oracle.dbtools.rest.resource.modules.xml.Handler.content(Handler.java:22)

at oracle.dbtools.rest.resource.modules.xml.XMLResourceModuleSet$Builder.buildPathTemplates(XMLResourceModuleSet.java:150)

at oracle.dbtools.rest.resource.modules.xml.XMLResourceModuleSet$Builder.read(XMLResourceModuleSet.java:213)

at oracle.dbtools.rest.resource.modules.xml.XMLResourceModuleSet$Builder.add(XMLResourceModuleSet.java:94)

at oracle.dbtools.rest.resource.modules.xml.StaticModules.add(StaticModules.java:77)

at oracle.dbtools.rest.resource.modules.xml.StaticModules.<init>(StaticModules.java:49)

at java.lang.invoke.MethodHandle.invokeWithArguments(Unknown Source)

at oracle.dbtools.plugin.api.types.TypeDependencies.newInstance(TypeDependencies.java:151)

... 54 more

Caused by: javax.xml.transform.TransformerException: java.lang.IllegalArgumentException: type: -1

at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform(Unknown Source)

at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform(Unknown Source)

at oracle.dbtools.common.xml.JAXPBase.clone(JAXPBase.java:61)

at oracle.dbtools.rest.resource.modules.xml.Content.clone(Content.java:73)

... 65 more

Caused by: java.lang.IllegalArgumentException: type: -1

at com.sun.org.apache.xerces.internal.dom.DeferredDocumentImpl.getNodeObject(Unknown Source)

at com.sun.org.apache.xerces.internal.dom.DeferredElementNSImpl.synchronizeData(Unknown Source)

at com.sun.org.apache.xerces.internal.dom.ElementImpl.getNodeName(Unknown Source)

at com.sun.org.apache.xalan.internal.xsltc.trax.DOM2TO.parse(Unknown Source)

at com.sun.org.apache.xalan.internal.xsltc.trax.DOM2TO.parse(Unknown Source)

at com.sun.org.apache.xalan.internal.xsltc.trax.DOM2TO.parse(Unknown Source)

at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transformIdentity(Unknown Source)

... 69 more

thatJeffSmith-OracleR HachemOlafur TzagirovArnost KlaskasocratesgrahamcreeRonald van der Fange
«1

Answers

  • thatJeffSmith-Oracle
    thatJeffSmith-Oracle Distinguished Product Manager Posts: 7,770 Employee
    edited Jan 6, 2020 6:13AM

    it would be useful to know:

    + JDK version.

    + if the issue is reported in the logs at startup or only after a request is received.

    + if they are adding any plugins to ords.war ( unlikely but it would be good to rule it out )

    Also, what url is being requested when this occurs?

    And finally, is it possible this isn't the entire stack/error message?

  • EJ-Egyed
    EJ-Egyed Member Posts: 125 Blue Ribbon
    edited Jan 6, 2020 9:37AM

    I have received the error as well.  Answers to your questions below:

    1. Java version:

    > java -version

    java version "13" 2019-09-17

    Java(TM) SE Runtime Environment (build 13+33)

    Java HotSpot(TM) 64-Bit Server VM (build 13+33, mixed mode, sharing)

    2. Nothing reported on startup.  Only when a request is received.  I was able to navigate a few pages before running into the error for the first time. Then refreshing the page seemed to resolve the issue.

    3. No plugins being added to ords.war

    I received this error when attempting to navigate to the APEX page designer page (Application 4000 Page 4500).  Here are a few lines from my Tomcat localhost_access_log file:

    (IP Obsured) - - [06/Jan/2020:09:10:21 -0500] "POST /ords/wwv_flow.ajax HTTP/1.1" 500 16163

    (IP Obsured) - - [06/Jan/2020:09:10:21 -0500] "POST /ords/wwv_flow.ajax HTTP/1.1" 500 16163

    (IP Obsured) - - [06/Jan/2020:09:10:23 -0500] "GET /ords/f?p=4000:4500:12748700618835::NO:1,4150:FB_FLOW_ID,FB_FLOW_PAGE_ID,F4000_P1_FLOW,F4000_P4150_GOTO_PAGE,F4000_P1_PAGE:106,1,106,1,1 HTTP/1.1" 500 16163

    (IP Obsured) - - [06/Jan/2020:09:10:25 -0500] "GET /ords/f?p=4000:4500:12748700618835::NO:1,4150:FB_FLOW_ID,FB_FLOW_PAGE_ID,F4000_P1_FLOW,F4000_P4150_GOTO_PAGE,F4000_P1_PAGE:106,1,106,1,1 HTTP/1.1" 500 16163

    Today was my first time running into the error, but I have now turned on debug.PrintDebugToScreen to hopefully capture more information in the future if it happens again.

  • m_yasirali
    m_yasirali Member Posts: 15
    edited Jan 7, 2020 9:20AM

    Hi @thatJeffSmith-Oracle,

    Following is the information you requested.

    1. The + JDK version.

    Java version "1.8.0_201"

    Java(TM) SE Runtime Environment (build 1.8.0_201-b09)

    Java HotSpot(TM) 64-Bit Server VM (build 25.201-b09, mixed mode)

    2. The error arises anytime during the day, so we restart the tomcat to solve the error.  Here, is a the tomcat log file attached:

    http://famproperties.com/assets/tomcat.log

    3. No plugins being added to ords.war.

    Thanks advance for a prompt reply.

  • thatJeffSmith-Oracle
    thatJeffSmith-Oracle Distinguished Product Manager Posts: 7,770 Employee
    edited Jan 7, 2020 10:34AM

    your error log looks completely different than the one user test2 opened the thread with

  • thatJeffSmith-Oracle
    thatJeffSmith-Oracle Distinguished Product Manager Posts: 7,770 Employee
    edited Jan 7, 2020 10:43AM

    everyone on this thread - please include your specific error stack messages

    and, are these intermittent (sounds like it is) issues? If so, any idea of what the load is like, number of users sending concurrent requests?

  • thatJeffSmith-Oracle
    thatJeffSmith-Oracle Distinguished Product Manager Posts: 7,770 Employee
    edited Jan 10, 2020 6:15AM

    We're now able to reproduce this issue and are evaluating fixes...thanks for the report folks.

  • EJ-Egyed
    EJ-Egyed Member Posts: 125 Blue Ribbon
    edited Jan 10, 2020 4:24PM

    Glad to hear! Are there any configuration changes we can make to prevent this error until a fix is released?

  • thatJeffSmith-Oracle
    thatJeffSmith-Oracle Distinguished Product Manager Posts: 7,770 Employee
    edited Jan 10, 2020 8:29PM

    No, it's a boring Java library loading issue. It's also benign from what we can tell.

  • m_yasirali
    m_yasirali Member Posts: 15
    edited Jan 12, 2020 1:21AM

    Hi @thatJeffSmith-Oracle,

    Hope you are fine. This is today's log when the error occurred. Kindly have a look into this, as this is intermittently happening and we have to restart Apache and Tomcat to fix it.

    https://famproperties.com/assets/catalina.2020-01-12.log

  • thatJeffSmith-Oracle
    thatJeffSmith-Oracle Distinguished Product Manager Posts: 7,770 Employee
    edited Jan 12, 2020 9:43AM

    This is a completely different error stack than the issue orginally being discussed.

    Please start a new thread, and include all of the relevant version info
      Version of ORDS
                       Java
                       Tomcat

                      APEX

                      Oracle Database

    The logs are saying the DB is killing the session, so I need you to check your Oracle DB Alert logs as well.

    I'm not familiar with the ORA being called for apex and the ajax bits, but we'll dig into it first thing tomorrow.

  • thatJeffSmith-Oracle
    thatJeffSmith-Oracle Distinguished Product Manager Posts: 7,770 Employee
    edited Jan 13, 2020 1:17PM

    something is going on with apex...could be a bug  on the apex side, recommend you open a SR

  • Olafur T
    Olafur T Member Posts: 237 Bronze Badge
    edited Jan 13, 2020 1:28PM

    This is happening also to me after upgrade to 19.4.  I need to simply stop tomcat again, wait 1-2 seconds and start again. (with frontend down)

    Affects ORDS services. That is restart tomcat, no using APEX. All ORDS services will have that error, sometimes Tomcat needs 2 restarts. In the end it works fine and will run until Tomcat is restarted again.

    Biggest problem is that all restarts need to be done manually (no automatic restarts since it will not get better until Tomcat is restarted)

    This issue plus Performance issues under constant high load  makes it a drag since the DB needs to be manually restarted once a day with supervision.

    Oli

  • SveinOle
    SveinOle Member Posts: 13 Green Ribbon
    edited Jan 16, 2020 4:44AM

    Have you had any progress on these, now, two issues?

    We are still on 19.2, waiting to upgrade to 19.4 (was planned last weekend, but postponed when new issues with 19.4 was discussed here).

    We restart manually each 48 hours now, time consuming and should be unneccesary.

  • SveinOle
    SveinOle Member Posts: 13 Green Ribbon
    edited Jan 16, 2020 4:46AM

    Or did the 19.4 at least resolve the first issue with the high load on that specific query that ORDS is executing?

    zagirov
  • thatJeffSmith-Oracle
    thatJeffSmith-Oracle Distinguished Product Manager Posts: 7,770 Employee
    edited Jan 16, 2020 8:26AM

    We have a fix, you need to create an index. I can get you that and you can run it today in 19.2

    We're working on a possible 19.4.1 to address this issue with the static xml resources causing the 500's

  • SveinOle
    SveinOle Member Posts: 13 Green Ribbon
    edited Jan 16, 2020 1:37PM

    Hi @thatJeffSmith-Oracle!

    That would be excellent if a index would amend the 19.2 issue. How do we proceed to get this?

    kind regards, Svein Ole

  • thatJeffSmith-Oracle
    thatJeffSmith-Oracle Distinguished Product Manager Posts: 7,770 Employee
    edited Jan 16, 2020 2:02PM

    Don't change the name - we may remove it later as the fix in the ORDS product may ultimately be something other than this INDEX. Internal testing saw this would resolve/fix your issue however.

    CREATE INDEX ORDS_METADATA.ORDS_OBJECTS_UOS_IDX ON ORDS_METADATA.ORDS_OBJECTS

        (

          SCHEMA_ID ASC ,

          PARSING_OBJECT ASC ,

          TYPE ASC ,

          STATUS ASC

        ) ;

  • SveinOle
    SveinOle Member Posts: 13 Green Ribbon
    edited Jan 17, 2020 7:27AM

    Thank you!

    Index is created with supplied name and we have been running a few hours with it now and operations so far is smooth and fine.

    We'll know much about how this improves over the weekend, we estimate high traffic on our solution this weekend.

    With kind regards

    Svein Ole

    thatJeffSmith-OraclethatJeffSmith-Oracle
  • SveinOle
    SveinOle Member Posts: 13 Green Ribbon
    edited Jan 20, 2020 4:05PM

    Experiences from a busy weekend on our systems with the new index in place for ORDS 19.2 is that it seems to have mitigated or solved the issues with the specific query being executed to often that it leads to some sort of CPU starvation that again leads to timeouts and multiple new connections beeing startet (from web/rest-endpoint users externally). We have increased traffic and ORDS pools increases number of active connections, but after traffic reduces, the pools are reduced and load/CPU on main DB server is normal.

    So thanks @thatJeffSmith-Oracle for this patch so far, we are also ready for 19.4 as soon as we know that the java loading issues as addressed

    Kind regards,

    Svein Ole

  • Jonas de Graaff (CACI)
    Jonas de Graaff (CACI) Member Posts: 10 Blue Ribbon
    edited Apr 22, 2020 8:57AM

    Is there any news on the initial problem of this issue, the  'boring Java library loading issue'. We encounter the same 'No provider found for: requires oracle.dbtools.http.filters.Filters' error after upgrading to ORDS 19.4. Is there a BUG reported at Oracle Support?

    Kind regards,

    Jonas de Graaff

  • R Hachem
    R Hachem Member Posts: 15 Blue Ribbon
    edited Apr 29, 2020 8:35AM

    Have you guys created a version 19.4.1 to address this issue? Can this be downloaded anywhere?

  • thatJeffSmith-Oracle
    thatJeffSmith-Oracle Distinguished Product Manager Posts: 7,770 Employee
    edited Apr 29, 2020 9:26AM

    We're working on version 20.2, which will have a fix for this issue.

    How often are you experiencing this issue? It was identified as intermittent, and not necessarily new for 19.4, so it had been in the product for awhile...

  • Olafur T
    Olafur T Member Posts: 237 Bronze Badge
    edited Apr 29, 2020 3:14PM

    In my case (my projects), 4 ORDS instances running on Tomcat.  I always manually restart the Tomcat service and check if ORDS is running.

    My guessimate (from memory) is that one in every 3 times I need to restart again. This is way higher than previous instances, since I never saw this problem until 19.4.

    Perhaps some component is now much bigger making some libraries loading too late now and then?

    Best regards

    Oli

  • R Hachem
    R Hachem Member Posts: 15 Blue Ribbon
    edited Apr 30, 2020 7:53AM

    I haven't upgraded to 19.4 because of the various reports from people saying that once they do so they have to restart Tomcat/ORDS every day or every 2 days or whenever they start getting the error. Right now my 18.3 is working fine and I don't want to make things worse by upgrading to 19.4. I might go with 19.2 because I haven't heard of any such problems with it. The reason I need to upgrade ORDS is because we need to uptake a newer version of APEX and this newer version requires ORDS 19.x or higher.

  • zagirov
    zagirov Member Posts: 52 Blue Ribbon
    edited May 1, 2020 5:55AM

    It is sad that there will be no ORDS 20.1 - we were waiting for it so much.

    19.2 with the created index ORDS_METADATA.ORDS_OBJECTS_UOS_IDX makes the server more stable?

    What features of APEX 20.1 will be integrated with the new version of ORDS?

Sign In or Register to comment.