10 Replies Latest reply: Nov 30, 2005 10:01 PM by 420008 RSS

    Multiple, simultaneous users of Java CSDK-based service?

    420008
      What suggestions may be available regarding permitting multiple, simultaneous users of a service based on the Java CSDK?

      We've been building a simple web services interface to selected CSDK functions - to add, delete, and retrieve calendar events - using version 9.0.4.2.10 of the Java CSDK API. This service - which is hosted in a standalone servlet container using webMethods Glue 6.0sp1, highly akin to Tomcat and other similar servlet containers - works fine when a single client is accessing the service. However, when two or more users are attempting to simultaneously access the service, either the service quits or each client's last access attempt fails, depending on the configuration of the service. We've experimented with various coding techniques and activation methods for the web service, so far without success.

      Are there any tips to avoid this problem, or sample code that may help, based on Oracle's experience with its own Web Services interface or any readers' own experiences? For instance, what objects need to be completely un-duplicated in each instance of the service that is offered to clients? And does this usually involve creating unique new sessions, each containing their own copies of such CSDK objects as Session, Handle, and RequestResult. Finally, what role do native resources associated with those objects play? We're aware of the injunction in the Javadocs about repeated calls to Api.init(), but are there specific coding techniques involving that method or others to help ensure that services built from the CSDK are safe for multiple, simultaneous accesses?
        • 1. Re: Multiple, simultaneous users of Java CSDK-based service?
          414326
          Hi Aron,

          First, which version of the sdk are you using?
          What suggestions may be available regarding
          permitting multiple, simultaneous users of a service
          based on the Java CSDK?
          "No two threads should concurrently use a session or handle object, even if the threads
          are performing operations on the same user's agenda or handle: The SDK does not
          support the concurrent use of sessions and handles."

          This is from the 10g Applications Developers Guide which I have mentioned in a previous post, that can be found at http://st-doc.us.oracle.com/cs/1012/calendar.1012/b25489.pdf

          The comment above is valid for RequestResults also. One important thing to note is that a Handle is only valid during the lifetime of the Session used to obtain the Handle. This is enforced in the 10g version of the calendar sdk but not in 9.0.4.2.X, so problems can occur if a Handle is reused after the lifetime of a Session.
          Finally, what
          role do native resources associated with those
          objects play? We're aware of the injunction in the
          Javadocs about repeated calls to Api.init(), but are
          there specific coding techniques involving that
          method or others to help ensure that services built
          from the CSDK are safe for multiple, simultaneous
          accesses?
          ex: Don't get Handle1 from Session1 and use Handle1 with Session2 + don't use the same Calendar Sdk Objects concurrently.

          Please let me know if you need further clarifications.

          Regards,
          Jean-Philippe
          • 2. Re: Multiple, simultaneous users of Java CSDK-based service?
            420008
            Many thanks for your response, Jean-Philippe!

            You asked:
            First, which version of the sdk are you using?
            Version 9.4.0.2.10 of the Java CSDK on both Mac OS X and Solaris. And now 10.1.1 on Mac OS X (so far).

            We've identified and resolved one issue that was preventing multiple, simultaneous accesses: Api.init() was being called repeatedly. This was resulting in CAPI_STAT_LIBRARY_INTERNAL errors.

            Now, simultaneous multiple accesses to the web service, with the Mac OS X 10.1.1 CSDK 'under the hood', appear to be working fine. However, after between 500 and 2000 mixed operations - specifically getCapabilities(), fetchEventsByUID(), fetchEventsByRange(), storeEvents(), and deleteEvents() - the service has each time quit with an error identical to the following (with the exception of the variable numeric/hex components below):

            *** malloc[3819]: Deallocation of a pointer not malloced: 0x1d46200; This could be a double free(), or free() called with the middle of an allocated block; Try setting environment variable MallocHelp to see tools to help debug

            Several more data points which may - or may not - be relevant:

            1. We're calling dispose() after we've completed using each Session, Handle, and RequestResult object.

            2. We're using the 'session' and/or 'request' activation modes in the webMethods Glue servlet container, both of which that vendor states "creates a new object to handle each request and destroys the object at the end of the request."

            3. We're repeatedly seeing the following errors at system startup, even when we run test scripts against our Java implementation class (which indirectly invokes the relevant CSDK methods), entirely outside of the webMethods servlet container:

            /Users/macbuild/builds/ade/view_storage/kmacdona_ocal_main_mac/ocal/src/lib/core/prod/shlib/capi/machocapi.cpp:1927: Failed to load function pointer for function "CAPI_Capabilities".
            /Users/macbuild/builds/ade/view_storage/kmacdona_ocal_main_mac/ocal/src/lib/core/prod/shlib/capi/machocapi.cpp:1927: Failed to load function pointer for function "CAPI_CreateFileStream".

            4. Our stress test scripts include simultaneous (or nearly so) authentications as the same users. In other words, to date we've always had at least two scripts running simultaneously which authenticate as the same calendar user. (So far, we haven't tested designate access.) This appears to work - at least, we can perform many simultaneous - or nearly so - operations on the same user's agenda from two separate scripted web services clients without error, before encountering the malloc deallocation problem listed above.

            Any further troubleshooting suggestions you can provide would be greatly appreciated!
            • 3. Re: Multiple, simultaneous users of Java CSDK-based service?
              420008
              A couple of clarifications to the previous post:

              - Two error messages listed in that post:
              /Users/macbuild/builds/ade/view_storage/kmacdona_ocal_main_mac/ocal/src/lib/core/prod/shlib/capi/machocapi.cpp:1927: Failed to load function pointer for function "CAPI_Capabilities".
              /Users/macbuild/builds/ade/view_storage/kmacdona_ocal_main_mac/ocal/src/lib/core/prod/shlib/capi/machocapi.cpp:1927: Failed to load function pointer for function "CAPI_CreateFileStream".
              are appearing on the console upon the first invocation of Api.init( configfile, logfile ) in the 10.1.1 Mac OS X CSDK.

              - We're calling Handle.dispose() as soon as we've finished using the Handle object(s) in fetchEventsByUID() or fetchEventsByRange(). We're calling the dispose() methods on our RequestResult and Session objects soon after we've we've completed the single operation - getCapabilities(), fetchEventsByUID(), fetchEventsByRange(), storeEvents(), or deleteEvents() - which relies on the presence of those objects.

              - We're not doing our own session or thread management. Rather, we're relying on the capabilities of the webMethods Glue servlet container to give us a new set of objects with each new web services request, and are not doing any explicit session management, nor explicitly creating any Java threads, on the server side.
              • 4. Re: Multiple, simultaneous users of Java CSDK-based service?
                414326
                *** malloc[3819]: Deallocation of a pointer not
                malloced: 0x1d46200; This could be a double free(),
                or free() called with the middle of an allocated
                block; Try setting environment variable MallocHelp to
                see tools to help debug
                Who reports that error? Does the OS report that error? Do you have a core dump? A stack trace of that core dump? Does your calendar sdk log contain any errors before you get that core? What are your calendar sdk configurations?
                1. We're calling dispose() after we've completed
                using each Session, Handle, and RequestResult
                object.
                That is good practice.
                2. We're using the 'session' and/or 'request'
                activation modes in the webMethods Glue servlet
                container, both of which that vendor states "creates
                a new object to handle each request and destroys the
                object at the end of the request."
                If I understand correctly, each request will get its own objects(Session, Handle, RequestResult, ....). No Calendar Sdk Object can be shared between 2 requests?
                3. We're repeatedly seeing the following errors at
                system startup, even when we run test scripts against
                our Java implementation class (which indirectly
                invokes the relevant CSDK methods), entirely outside
                of the webMethods servlet container:

                /Users/macbuild/builds/ade/view_storage/kmacdona_ocal_
                main_mac/ocal/src/lib/core/prod/shlib/capi/machocapi.c
                pp:1927: Failed to load function pointer for function
                "CAPI_Capabilities".
                /Users/macbuild/builds/ade/view_storage/kmacdona_ocal_
                main_mac/ocal/src/lib/core/prod/shlib/capi/machocapi.c
                pp:1927: Failed to load function pointer for function
                "CAPI_CreateFileStream".
                Thats ok, that issue as been resolved (the fix will be available in a further mac build) and it doesn't impact your application. One bug that could impact your mac app, is a bug with function Api.getStatusCode(), so please don't use that function (problem only on Mac, the function is not safe to use till you get a patch with a fix for bug 4547364 )
                • 5. Re: Multiple, simultaneous users of Java CSDK-based service?
                  420008
                  *** malloc[3819]: Deallocation of a pointer not malloced:
                  Who reports that error? Does the OS report that error?
                  This is displayed on the console (presumably stdout or stderr) in the terminal window in which the output from the webapp is occurring. The reporter isn't obvious; my naive guess would be the Java interpreter or webMethods Glue, but the message itself doesn't indicate that - the text posted in a previous entry in this thread constitutes the complete error message. We'll check on whether there are any other traces of this error in OS X logs.
                  Do you have a core dump? A stack trace of that core dump?
                  No core dump files are evident in the webapps filesystem tree, which is where these would usually appear.
                  If I understand correctly, each request will get its own objects(Session, Handle, RequestResult, ....). No Calendar Sdk Object can be shared between 2 requests?
                  Yes, that's our understanding as well. Note that we're able to perform many thousands of CSDK operations from up to four - the maximum we've tried - simultaneous web services clients before this problem crops up. Is that by itself sufficient external evidence that we're not - at least routinely - sharing objects between requests?
                  Does your calendar sdk log contain any errors before you get that core?
                  None whatsoever. However, we're logging only at the activity level, not the trace or debug level. We'll generate more detailed (and voluminous) logs to see if we can spot the issue within the CSDK's own logging.
                  What are your calendar sdk configurations?
                  What information would be useful regarding this? We'll be glad to provide it. calendar.ini, DYLD_LIBRARY_PATH, CLASSPATH, others?
                  One bug that could impact your mac app, is a bug with function Api.getStatusCode()
                  We're not calling that function, but we are calling Api.getStatusString() and Api.StatusException.getStatus(). Are those both safe in the current Mac OS X CSDK?

                  Thanks again!
                  Aron
                  • 6. Re: Multiple, simultaneous users of Java CSDK-based service?
                    420008
                    *** malloc[3819]: Deallocation of a pointer not malloced:
                    Who reports that error? Does the OS report that error?
                    This is displayed on the console (presumably stdout or stderr) in the terminal window in which the >output from the webapp is occurring. The reporter isn't obvious ...
                    For whatever this may be worth, a quick Google search suggests that error message may, at its lowest level, be associated with C or C++ code compiled by the gcc compiler ...
                    • 7. Re: Multiple, simultaneous users of Java CSDK-based service?
                      420008
                      We'll check on whether there are any other traces of this error in OS X logs.
                      In the two most recent entries in the Mac OS X ~/Library/logs/Java.crash.log file, for the user account which is launching the webMethods Glue webapp which is invoking CSDK functions, both entries have a "Thread 14 Crashed:". One representative example is shown below; we can provide this log file and any additional details you may require under separate cover.

                      ...
                      OS Version: 10.3.9 (Build 7W98)
                      Report Version: 2

                      Command: java
                      Path: /usr/bin/java
                      Version: ??? (???)
                      PID: 3238
                      Thread: 14

                      Exception: EXC_BAD_ACCESS (0x0001)
                      Codes: KERN_INVALID_ADDRESS (0x0001) at 0x3b69df70

                      ...

                      Thread 14 Crashed:
                      0 <<00000000>>      0x3b69df70 0 + 0x3b69df70
                      1 ...ple.CoreServices.CarbonCore      0x96335e50 InitializeConnections + 0x88
                      2 ...ple.CoreServices.CarbonCore      0x9631e89c PrepareClosure + 0x16c
                      3 ...ple.CoreServices.CarbonCore      0x96335a14 PCFragPrepareClosureByName + 0x12c
                      4 ...ple.CoreServices.CarbonCore      0x9634465c GetSharedLibrary + 0xa4
                      5 ctcore      0x024c6988 SLOsLoadLibrary + 0xe4
                      6 ctcore      0x024c5030 SL_LoadLibrary + 0x84
                      7 ctcore      0x024c4a38 SL_LoadLibraryAndFunctions + 0x94
                      8 ctcalcli      0x1c420708 AuthLoadLib + 0x3b4
                      9 ctcalcli      0x1c420b14 AuthInitLib + 0x340
                      10 ctcalcli      0x1c42184c AuthGetFunctionPtr + 0x368
                      11 ctcalcli      0x1c4272b4 Auth_QueryLib + 0x1d0
                      12 ctcalcli      0x1c428550 AuthGetTypeInfo + 0x114
                      13 ctcalcli      0x1c428880 Auth_ClientGetInfo + 0x278
                      14 ctcalcli      0x1c3de92c do_AuthGetInfo + 0x90
                      15 ctcalcli      0x1c3dc6c8 do_AuthRequestContext + 0x29c
                      16 ctcalcli      0x1c3dc308 SRV_AuthRequestContext + 0xa8
                      17 ctcalcli      0x1c3dc3d8 CAL_AuthRequestContext + 0x70
                      18 ctcalcli      0x1c3d97a8 ClientOpen + 0xac
                      19 ctcalcli      0x1c3dc1b4 ConnectToService + 0x970
                      20 OracleCalendarSDK      0x023a1754 NewConnection__19CcapiConnectionPoolFQ219CcapiConnectionPool7NodeUidRP10BaseAccess + 0x144
                      21 OracleCalendarSDK      0x023a0a38 RequestConnection__19CcapiConnectionPoolFRCQ23std59basic_string<c,Q23std14char_traits<c>,Q23std12allocator<c>>RQ219CcapiConnectionPool9ConnToken + 0x414
                      22 OracleCalendarSDK      0x02371798 ConnectToServer__12CcapiSessionFPCcUl + 0x64
                      23 OracleCalendarSDK      0x02371a5c connect__12CcapiSessionFPCcUlRP12CcapiSession + 0xec
                      24 OracleCalendarSDK      0x0232fd7c OCAP_Connect + 0x18c
                      25 OracleCalendarSDK      0x023ab57c CSDK_Connect + 0x58
                      26 libcsdkjni.jnilib      0x017eb180 Java_oracle_calendar_sdk_Session_jniConnect + 0x8c
                      27 <<00000000>>      0x03f365a0 0 + 0x3f365a0
                      [... multiple log entry lines in highly similar format omitted here for brevity ...]
                      59 <<00000000>>      0x03f2d16c 0 + 0x3f2d16c
                      60 libclient.dylib      0x96f3afe8 JVM_GetCPMethodClassNameUTF + 0xb38
                      61 libclient.dylib      0x96f5c438 JVM_GetCPClassNameUTF + 0x998
                      62 libclient.dylib      0x96f99098 JVM_Close + 0x4b8
                      63 libclient.dylib      0x96fa9960 JVM_Interrupt + 0x2e0
                      64 libclient.dylib      0x9704e6fc JVM_UnloadLibrary + 0xcdec
                      65 libclient.dylib      0x96f94df4 JVM_FindClassFromClass + 0xc14
                      66 libclient.dylib      0x970f290c JVM_UnloadLibrary + 0xb0ffc
                      67 libSystem.B.dylib      0x94747990 pthreadbody + 0x28
                      • 8. Re: Multiple, simultaneous users of Java CSDK-based service?
                        420008
                        A progress update ... after making two minor code changes: checking for null values before calling dispose() on the three relevant CSDK objects and not explicitly setting the CSDK session object to null after calling dispose() on it, we so far haven't yet encountered the malloc deallocation error.

                        At the moment, the web service has been running for over 14 hours with over 4,200 CSDK operations performed, generated from four test scripts which are authenticating as two different calendar users (including one designate access to another users' calendar). We'll post an update when this continues to work much longer without incident, or if the problem reoccurs.
                        • 9. Re: Multiple, simultaneous users of Java CSDK-based service?
                          414326
                          Hi Aron,
                          Yes, that's our understanding as well. Note that
                          we're able to perform many thousands of CSDK
                          operations from up to four - the maximum we've tried
                          - simultaneous web services clients before this
                          problem crops up. Is that by itself sufficient
                          external evidence that we're not - at least routinely
                          - sharing objects between requests?
                          No, that is not sufficient external evidence that your not sharing objects between request. I am not saying that you are sharing these objects between threads, just that multi-threaded issues can be tricky and non-deterministic.
                          What are your calendar sdk configurations?
                          What information would be useful regarding this?
                          We'll be glad to provide it. calendar.ini,
                          , DYLD_LIBRARY_PATH, CLASSPATH, others?
                          mostly calendar.ini
                          Also ho many user concurrently authenticate to the servers, how many nodes are there, what are the ACE plug-ins used?
                          Api.getStatusCode()
                          We're not calling that function, but we are calling
                          ng Api.getStatusString() and
                          Api.StatusException.getStatus(). Are those both safe
                          in the current Mac OS X CSDK?
                          Yes they are.

                          Regards,
                          Jean-Philippe
                          • 10. Re: Multiple, simultaneous users of Java CSDK-based service?
                            420008
                            mostly calendar.ini
                            Also ho many user concurrently authenticate to the servers,
                            how many nodes are there, what are the ACE plug-ins used?
                            calendar.ini appears below.

                            At peak, we've had four test scripts generating server events (capability, store, get, get by range, delete):

                            Script 1 authenticated as user 1
                            Script 3 authenticated as user 2
                            Script 2 authenticated as user 1, then acted with designate access on user 2's calendar
                            Script 4 authenticated as user 2

                            To the best of my knowledge, there are two nodes on this test Oracle Calendar server instance. All of the accesses above are to the same node (ND=1 in the login string).

                            I don't know how to answer the ACE plug-ins question; perhaps you can help clarify. A capabilities query against the server returns "cs-standard,cs-basic" authentication methods. On the Macintosh on which these scripts are running, the full set of five authentication plug-ins provided with the CSDK, such as "OCALStdAuthLib", are included in ~/Library/CFMSupport.

                            --

                            ; calendar.ini
                            ;
                            ; See the the "Oracle Calendar SDK Configuration Settings" chapter
                            ; of the Oracle Calendar Application Developers Guide 10g Release 1 (10.1.1)

                            [CAPI]
                            client_name = Calendaraccess
                            client_version = 10.1.1

                            [LOG]
                            ; Turn on "activity" level logging (the default level)
                            log_activity = true

                            ; Optionally turn on more detailed logging
                            ; log_trace = true
                            ; log_debug = true
                            ; log_state = true

                            ; Identify the modules to be logged
                            ; e.g. { CAPI } or { CAPI, ICAL }
                            log_modulesinclude = { CAPI }

                            [CONNPOOL]
                            ; max_user = 5