6 Replies Latest reply: Jun 11, 2009 11:10 AM by tbeets RSS

    JBO-26000 and JBO-29000 errors testing WL deployed ADF BC SDO Web Services

    585360
      Is anyone else having problems testing / using ADF BC SDO Web Services that they deploy to WebLogic?

      More often than not I get into a position where, while trying to test ADF BC Projects in Enterprise Manager (EM) that have been deployed to a WebLogic server from the JDeveloper Menu Application / Deploy, I am getting the same errors on multiple projects and simple cannot exercise any methods (and therefore use the Web Service). I get the same errors when trying to use the Service by calling with via a Web Service Proxy.

      The same problem occurs if I make a change to an Application that is working and then redeploy.

      Or, if I make a change in WebLogic (e.g. change parameters for a Database connection to promote to a different environment), I'll run into the same set of errors.

      Yet at the same time - if a I have a project that was deployed and working previously and I don't make any changes to it, it will continue to work as it did before.

      The other thing I see is that if I stress test some of my Applications, they run into about a 10% failure rate - with the same errors I get when I'm completely hosed up.

      Here is a listing of the errors I've seen. Yes the last three look like there is something wrong with the Application, but these occurred in projects that would succeed 90% of the time during a stress test.

      h4. Errors Summary
      The most notorious error is (and I'll post the lengthy out-put below).
      1. JBO-26000: A Generic exception occurred during loading Customizations.: MDS-00168: MDS object oracle.mds.core.MDSInstance@8e2815 is being used after it or its MDSInstance or PManager has been released.

      Other errors are:
      2.JBO-29000: Unexpected exception caught: java.lang.NoClassDefFoundError, msg=com/qualcomm/util/EDFApplicationModuleDefImpl : com/qualcomm/util/QCApplicationModuleDefImpl at oracle.sysman.emas.model.wsmgt.WSTestModel.invokeOperation(WSTestModel.java:566)

      3. JBO-29000: Unexpected exception caught: java.lang.ClassCastException, msg=oracle.jbo.common.ampool.PoolMgr: oracle.jbo.common.ampool.PoolMgr at oracle.sysman.emas.model.wsmgt.WSTestModel.invokeOperation(WSTestModel.java:566) at

      4. JBO-29000: Unexpected exception caught: java.lang.NullPointerException, msg=null: null at oracle.sysman.emas.model.wsmgt.WSTestModel.invokeOperation(WSTestModel.java:566)

      h4. Restart WL helps for short-while
      Now, we've just seen that I restarting both the admin and managed server makes these errors go away for a time. Though we have not had enough time to figure out a pattern as to when the errors will start coming back - we have seen them come back in a day or two with not a lot of testing going on - just a bit of POC Developer work.

      h4. Steps to reproduce
      Before we saw this behavior (reboot and all is good for awhile), I documented the steps I go through to build, deploy and test an ADF SDO Web Service to see if these errors were User errors. So I'll include the following anyways, though I suspect this is a buggy WebLogic Candidate Release Version issue and hope like crazy Oracle can confirm it has already been addressed in the next GA Release:

      Following some basic OTN ADF BC tutorials and the ‘e2e-105-POProcessing-G-SDO.doc’ from the SOA11gFoundationTutorial docs, here are the most basic steps for me to re-produce the error condition:

      1. Create an Application:
      1a. File / New / General / Generic Application / Application Name (in my case Device_2)
      1b. Name the Application Package Prefix: com.companyb.device
      1c. Project Name: Device
      1d. Project Technologies: Selected ADF Business Components and Java
      1e. Default Package: com.companyb.device - leave defaults for Java Source Path and Output Directory. Finish.

      2. Create BC Objects in Package
      2a. Select New / Business Tier / Business Components from Tables.
      2b. Select an existing Connection (we use a EDF_USER connection and that User has read / write privileges to an EDF schema). You could easily do the same in the HR schema with the Employee table. Copy connection into Project. Select OK.
      2c. Step 1. Enter Package Name: com.companyb.device.entity
      2d. Select EDF Schema (owns the tables).
      2e. Select just one table (Device) and select Next.
      2f. Step 2 Make View Objects package name: com.companyb.device.view, Select the Device Entity to create a selected DeviceView.
      2g. Step 3 Configure nothing. Select Next.
      2h. Step 4 Set Package name com.companyb.device.service. Leave Name AppModule as is. Select Next Next and Finish.

      3. Create SDO
      3a. Double-click the AppModule in the service package.
      3b. Select the Service Interface tab and the green + sign to launch wizard.
      3c. Step 1. Make Web Service Name: DeviceService. Leave target namespace unchanged and select Next.
      3d. Step 2. Select Next.
      3e. Step 3. Make DeviceView1 selected and enable Operations Create and GetByKey. Select Next and Select Finish.

      4 Configure SDO
      4a. Select the Configuration tab.
      4b. Select DeviceService and the edit icon.
      4c. Change the Datasource name to match what is normally configured in EM (e.g. jdbc/EDF_USERDS).
      4d. From the Default Configuration select the just created SDO (e.g. DeviceService).

      5. Configure deployment
      5a. Select the Project, right-click and select Project Properties.
      5b. Select Deployment and button New.
      5c. Select Business Components Service Interface from dropdown.
      5d. Change Name to: DeviceServiceInterface.
      5e. Drill into DeviceServiceInterface and select MiddleTier - change name to DeviceServiceInterface. Select OK, OK and save project.

      6. View what App name will be.
      6a. Select Application / Application Profile / Deployment. Select button Edit and view what Application name will be (Device_DeviceServiceInterface). Select OK.
      6b. Select Application / Deploy / Device_DevicsServiceInterface / Deploy / to eavlab56 (this is not my WebLogic Developer instance on my machine - it is a dedicated WebLogic 11gR1 Server.
      6c. Select a non-AdminServer Instance and watch the deploy succeed.

      7. Test the Service in EM (Enterprise Manager)
      7a. Open a browser session and select the Application in the left hand list of Applications.
      7b. Select the Web Services / Test Icon.
      7c. Select a Get transaction (in my case - getDeviceView1).
      7d. Enter a valid key (in my case deviceKey) and select Test Web Service.
      7e. For a variation, on 7d. Select Additional Test Options / Enable Stress Test and use defaults
      7f. It is at 7d and 7e where the fun begins and either the errors start flying or things work like they should for a while before the errors get you in a position where nothing works:

      h4. Output for JB0-26000 MDSInstance or PManager error
      Here is the complete output for the MDSInstance or PManager error.

      java.lang.Exception: oracle.sysman.emSDK.webservices.wsdlapi.SoapTestException: JBO-26000: A Generic exception occurred during loading Customizations.: MDS-00168: MDS object oracle.mds.core.MDSInstance@8e2815 is being used after it or its MDSInstance or PManager has been released. at oracle.sysman.emas.model.wsmgt.WSTestModel.invokeOperation(WSTestModel.java:566) at oracle.sysman.emas.view.wsmgt.WSView.invokeOperation(WSView.java:381) at oracle.sysman.emas.view.wsmgt.WSView.invokeOperation(WSView.java:298) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at com.sun.el.parser.AstValue.invoke(AstValue.java:157) at com.sun.el.MethodExpressionImpl.invoke(MethodExpressionImpl.java:283) at org.apache.myfaces.trinidadinternal.taglib.util.MethodExpressionMethodBinding.invoke(MethodExpressionMethodBinding.java:53) at org.apache.myfaces.trinidad.component.UIXComponentBase.broadcastToMethodBinding(UIXComponentBase.java:1245) at org.apache.myfaces.trinidad.component.UIXCommand.broadcast(UIXCommand.java:183) at oracle.adf.view.rich.component.fragment.ContextSwitchingComponent$1.run(ContextSwitchingComponent.java:87) at oracle.adf.view.rich.component.fragment.ContextSwitchingComponent._processPhase(ContextSwitchingComponent.java:298) at oracle.adf.view.rich.component.fragment.ContextSwitchingComponent.broadcast(ContextSwitchingComponent.java:91) at oracle.adf.view.rich.component.fragment.UIXInclude.broadcast(UIXInclude.java:87) at oracle.adf.view.rich.component.fragment.ContextSwitchingComponent$1.run(ContextSwitchingComponent.java:87) at oracle.adf.view.rich.component.fragment.ContextSwitchingComponent._processPhase(ContextSwitchingComponent.java:298) at oracle.adf.view.rich.component.fragment.ContextSwitchingComponent.broadcast(ContextSwitchingComponent.java:91) at oracle.adf.view.rich.component.fragment.UIXInclude.broadcast(UIXInclude.java:81) at javax.faces.component.UIViewRoot.broadcastEvents(UIViewRoot.java:475) at javax.faces.component.UIViewRoot.processApplication(UIViewRoot.java:756) at oracle.adfinternal.view.faces.lifecycle.LifecycleImpl._invokeApplication(LifecycleImpl.java:673) at oracle.adfinternal.view.faces.lifecycle.LifecycleImpl._executePhase(LifecycleImpl.java:273) at oracle.adfinternal.view.faces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:165) at javax.faces.webapp.FacesServlet.service(FacesServlet.java:265) at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227) at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125) at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:292) at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56) at oracle.adfinternal.view.faces.webapp.rich.RegistrationFilter.doFilter(RegistrationFilter.java:85) at org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl$FilterListChain.doFilter(TrinidadFilterImpl.java:420) at oracle.adfinternal.view.faces.activedata.AdsFilter.doFilter(AdsFilter.java:54) at org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl$FilterListChain.doFilter(TrinidadFilterImpl.java:420) at org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl._doFilterImpl(TrinidadFilterImpl.java:247) at org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl.doFilter(TrinidadFilterImpl.java:157) at org.apache.myfaces.trinidad.webapp.TrinidadFilter.doFilter(TrinidadFilter.java:92) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56) at oracle.help.web.rich.OHWFilter.doFilter(Unknown Source) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56) at oracle.sysman.emSDK.license.LicenseFilter.doFilter(LicenseFilter.java:101) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56) at oracle.adf.model.servlet.ADFBindingFilter.doFilter(ADFBindingFilter.java:189) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56) at oracle.security.wls.filter.SSOSessionSynchronizationFilter.doFilter(SSOSessionSynchronizationFilter.java:224) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56) at oracle.sysman.emas.fwk.MASConnectionFilter.doFilter(MASConnectionFilter.java:41) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56) at oracle.adf.library.webapp.LibraryFilter.doFilter(LibraryFilter.java:159) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56) at oracle.sysman.eml.app.AuditServletFilter.doFilter(AuditServletFilter.java:179) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56) at oracle.sysman.eml.app.EMRepLoginFilter.doFilter(EMRepLoginFilter.java:203) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56) at oracle.sysman.core.app.perf.PerfFilter.doFilter(PerfFilter.java:139) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56) at oracle.sysman.eml.app.ContextInitFilter.doFilter(ContextInitFilter.java:527) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56) at oracle.dms.wls.DMSServletFilter.doFilter(DMSServletFilter.java:202) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56) at weblogic.servlet.internal.RequestEventsFilter.doFilter(RequestEventsFilter.java:27) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56) at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3588) at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321) at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:121) at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2200) at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2106) at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1428) at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201) at weblogic.work.ExecuteThread.run(ExecuteThread.java:173) Caused by: oracle.sysman.emSDK.webservices.wsdlapi.SoapTestException: JBO-26000: A Generic exception occurred during loading Customizations.: MDS-00168: MDS object oracle.mds.core.MDSInstance@8e2815 is being used after it or its MDSInstance or PManager has been released. at oracle.sysman.emas.model.wsmgt.PortName.invokeOperation(PortName.java:710) at oracle.sysman.emas.model.wsmgt.WSTestModel.invokeOperation(WSTestModel.java:560) ... 70 more Caused by: oracle.sysman.emSDK.webservices.wsdlapi.SoapTestException: JBO-26000: A Generic exception occurred during loading Customizations.: MDS-00168: MDS object oracle.mds.core.MDSInstance@8e2815 is being used after it or its MDSInstance or PManager has been released. at oracle.sysman.emSDK.webservices.wsdlapi.dispatch.DispatchUtil.invoke(DispatchUtil.java:257) at oracle.sysman.emSDK.webservices.wsdlparser.OperationInfoImpl.invokeWithDispatch(OperationInfoImpl.java:843) at oracle.sysman.emas.model.wsmgt.PortName.invokeOperation(PortName.java:662) ... 71 more Caused by: javax.xml.ws.soap.SOAPFaultException: JBO-26000: A Generic exception occurred during loading Customizations.: MDS-00168: MDS object oracle.mds.core.MDSInstance@8e2815 is being used after it or its MDSInstance or PManager has been released. at oracle.j2ee.ws.client.jaxws.DispatchImpl.throwJAXWSSoapFaultException(DispatchImpl.java:874) at oracle.j2ee.ws.client.jaxws.DispatchImpl.invoke(DispatchImpl.java:707) at oracle.j2ee.ws.client.jaxws.OracleDispatchImpl.synchronousInvocationWithRetry(OracleDispatchImpl.java:226) at oracle.j2ee.ws.client.jaxws.OracleDispatchImpl.invoke(OracleDispatchImpl.java:97) at oracle.sysman.emSDK.webservices.wsdlapi.dispatch.DispatchUtil.invoke(DispatchUtil.java:253) ... 73 more

      Edited by: user582357 on Jun 9, 2009 7:19 AM
        • 1. Re: JBO-26000 and JBO-29000 errors testing WL deployed ADF BC SDO Web Services
          tbeets
          The intermittent JBO-26000 runtime error is the most perplexing.

          I have a simple ADF BC Service -- we're talking about the OFM 11gR1 (11.1.1.1.0) EAP RC2 version -- when I invoke it in a loop (say 10 iterations), sometimes it works every time and every iteration, sometimes every invocation fails with a JBO-26000, and sometime a percentage of the invocations fail with a JBO-26000.

          The service logic is a trivial -- hello world so there are no other service dependencies.

          Todd
          • 2. Re: JBO-26000 and JBO-29000 errors testing WL deployed ADF BC SDO Web Services
            SteveMuench
            The first issue is bug# 8235654 which development mentioned should be documented in the release notes (I haven't been able to verify that claim myself). The issue is fixed in later 11g R1 release candidate builds. The only workaround at the moment is to restart the WLS server.

            For the other items, I'm still waiting for a reply from development and will update this thread when it comes.
            • 3. Re: JBO-26000 and JBO-29000 errors testing WL deployed ADF BC SDO Web Services
              SteveMuench
              Development clarified that "as stated in the release note, the customer has to add this line in the generated service impl class file:"
              @javax.interceptor.Interceptors(oracle.jbo.server.svc.ServiceContextInterceptor.class)
              • 4. Re: JBO-26000 and JBO-29000 errors testing WL deployed ADF BC SDO Web Services
                tbeets
                I'm not able to find this in the release notes for RC2 (now a couple months old unfortunately)...

                In any case, I did try to add that annotation manually to the generated service implementation class. JDeveloper though is signaling to me in the editor that class oracle.jbo.server.svc.ServiceContextInterceptor does not exist.

                Is there an additional library that needs to be added to the project (that is not automatically added when creating the AppModule in the project)?
                package com.qualcomm.ea.ws.server.serviceinterface;
                
                import com.qualcomm.ea.ws.HRService1AppModuleImpl;
                import com.qualcomm.ea.ws.common.EmployeesViewSDO;
                import com.qualcomm.ea.ws.common.serviceinterface.HRService1EmployeeSvc;
                
                import java.lang.reflect.Method;
                
                import java.util.HashMap;
                import java.util.List;
                
                import java.util.Map;
                
                import javax.ejb.Remote;
                import javax.ejb.Stateless;
                
                import oracle.jbo.common.Diagnostic;
                import oracle.jbo.common.sdo.SDOHelper;
                import oracle.jbo.common.service.types.FindControl;
                import oracle.jbo.common.service.types.FindCriteria;
                import oracle.jbo.server.svc.ServiceImpl;
                import oracle.jbo.service.errors.ServiceException;
                
                import oracle.webservices.annotations.ManagementPolicy;
                import oracle.webservices.annotations.PortableWebService;
                
                import weblogic.javaee.CallByReference;
                // ---------------------------------------------------------------------
                // ---    File generated by Oracle ADF Business Components Design Time.
                // ---    Thu May 21 13:55:37 PDT 2009
                // ---------------------------------------------------------------------
                
                // *** Manually added per Re: JBO-26000 and JBO-29000 errors testing WL deployed ADF BC SDO Web Services ***
                @javax.interceptor.Interceptors(oracle.jbo.server.svc.ServiceContextInterceptor.class)
                // **********************
                
                @Stateless(name="com.qualcomm.ea.ws.common.HRService1EmployeeSvcBean",
                    mappedName="HRService1EmployeeSvcBean")
                @Remote(HRService1EmployeeSvc.class)
                @PortableWebService(targetNamespace="/com/qualcomm/ea/ws/common/",
                    serviceName="HRService1EmployeeSvc", portName="HRService1EmployeeSvcSoapHttpPort",
                    endpointInterface="com.qualcomm.ea.ws.common.serviceinterface.HRService1EmployeeSvc")
                @CallByReference
                public class HRService1EmployeeSvcImpl extends ServiceImpl implements HRService1EmployeeSvc {
                    private static boolean _isInited = false;
                
                    private static final Map _map = new HashMap();
                
                    /**
                     * This is the default constructor (do not remove).
                     */
                    public HRService1EmployeeSvcImpl() {
                        init();
                        setApplicationModuleDefName("com.qualcomm.ea.ws.HRService1AppModule");
                        setConfigurationName("HRService1EmployeeSvc");
                    }
                
                
                    /**
                     * Generated method. Do not modify. Do initialization in the constructor
                     */
                    protected void init() {
                        if (_isInited) {
                            return;
                        }
                        synchronized (HRService1EmployeeSvcImpl.class) {
                            if (_isInited) {
                                return;
                            }
                            try {
                                SDOHelper.INSTANCE.defineSchema("com/qualcomm/ea/ws/common/serviceinterface/", "HRService1EmployeeSvc.xsd");
                                _map.put("sayHello",
                                         HRService1AppModuleImpl.class.getMethod("sayHello",
                                                                                 new Class[] { String.class }));
                                _map.put("strLength",
                                         HRService1AppModuleImpl.class.getMethod("strLength",
                                                                                 new Class[] { String.class }));
                                _isInited = true;
                            } catch (Exception ex) {
                                Diagnostic.printStackTrace(ex);
                            }
                        }
                    }
                
                    /**
                     * getEmployeesView1: generated method. Do not modify.
                     */
                    public EmployeesViewSDO getEmployeesView1(Integer employeeId) throws ServiceException {
                        return (EmployeesViewSDO)get(new Object[] { employeeId },
                                                     "EmployeesView1", EmployeesViewSDO.class);
                    }
                
                    /**
                     * createEmployeesView1: generated method. Do not modify.
                     */
                    public EmployeesViewSDO createEmployeesView1(EmployeesViewSDO employeesView1) throws ServiceException {
                        return (EmployeesViewSDO)create(employeesView1, "EmployeesView1");
                    }
                
                    /**
                     * updateEmployeesView1: generated method. Do not modify.
                     */
                    public EmployeesViewSDO updateEmployeesView1(EmployeesViewSDO employeesView1) throws ServiceException {
                        return (EmployeesViewSDO)update(employeesView1, "EmployeesView1");
                    }
                
                    /**
                     * findEmployeesView1: generated method. Do not modify.
                     */
                    public List<EmployeesViewSDO> findEmployeesView1(FindCriteria findCriteria,
                                                                     FindControl findControl) throws ServiceException {
                        return (List<EmployeesViewSDO>)find(findCriteria, findControl,
                                                            "EmployeesView1",
                                                            EmployeesViewSDO.class);
                    }
                
                    /**
                     * sayHello: generated method. Do not modify.
                     */
                    public String sayHello(String name) throws ServiceException {
                        return (String)invokeCustom((Method)_map.get("sayHello"),
                                                    new Object[] { name },
                                                    new String[] { null }, false);
                    }
                
                    /**
                     * strLength: generated method. Do not modify.
                     */
                    public int strLength(String word) throws ServiceException {
                        return (Integer) invokeCustom((Method) _map.get("strLength"), new Object[] { word }, new String[] { null }, false);
                    }
                }
                • 5. Re: JBO-26000 and JBO-29000 errors testing WL deployed ADF BC SDO Web Services
                  SteveMuench
                  Is the "signalling" a warning or a fatal compilation error?

                  If you use [Ctrl]-[Minus] to file the java type of the ServiceContextInterceptor, does the name show up in the list of classes that match the name?
                  • 6. Re: JBO-26000 and JBO-29000 errors testing WL deployed ADF BC SDO Web Services
                    tbeets
                    It's a fatal compilation error. ServiceContextInterceptor does not show up in a class match (Ctrl-Minus).

                    Here's the JDev Studio version that I'm using:

                    Build JDEVADF_11.1.1.1.0_GENERIC_090401.2036.5341

                    ADF Business Components     11.1.1.53.41


                    This is the latest posting to the EAP download site (April 10) which I've been told by Oracle is referred to as RC2.

                    -Todd