6 Replies Latest reply: Jun 13, 2013 7:10 AM by JimKlimov RSS

    CLI tools to import Calendar and AddressBook entries into multiple users?

      We've exported data from an older 3rd-party messaging system being replaced with OCUCS 7u2, with calendar and task data as ICS files (one file with dozens of entries), and address-book data in a similar format (carddav?), one file of each type per user (who had non-empty calendars or address-books).

      Now the quest is to get this data into OCUCS :)

      Unfortunately, davadmin did not work for us (see post davadmin does not work with a GF instance without admin server for more details), and there seems to be no similar tool for PAB to import/export entries.

      We expected to find or write some scripts that would either add data into OCUCS backends (PAB LDAP, davadmin or even direct mysql for calendar+tasks), but it seems more proper to use standard tools (if available) instead of inventing our own wheels - that is, caldav/carddav or WCAP/WABP HTTP clients, or other protocol tools that can do import/export like GUI clients but invoked from command line and with proxy-auth capability.

      To review, we need to be able to import some file-based representation of calendar and addressbook data into OCUCS users' accounts from Unix command-line/script. If using protocol tools, this should work using only one admin account with the known password (and proxy-authenticating into other users' addressbooks and calendars for import tasks). For additional imports (i.e. synchronization while the old system is primary) no duplicate entries should appear, at least as long as they are identical; preferably UID and timestamp (if available) should be used to update older entries.

      Surprisingly, there are tons of GUI and mobile clients and plugins, but we found nearly no command-line ones. Seemingly, our best automation option might be to impose scripting onto clients like Thunderbird/Lightning or Convergence to do the jobs, or possibly fool an OutLook with the Connector into doing the job. Automation like CURL proxy-logins into Convergence and calling import/export URLs from there are being explored now...

      What do other people use for such quests? What can you suggest? What do you know we should look out for? ;)

        • 1. Re: CLI tools to import Calendar and AddressBook entries into multiple users?
          For importing calendar data, curl + [import.wcap|https://wikis.oracle.com/display/CommSuite/import.wcap] should be relatively straightfoward with the following hints:
          * instead of passing an id session parameter, pass httpauth=1 in your url. That way, you can use standard http authentication instead of having first to do a login.wcap to retrieve a sessionid.
          * for proxy auth, use the admin uid followed by ';' followed by the end user id (e.g. "calmaster;johndoe") and use the admin password

          For addressbook, you should be able to derive the same type of method by looking at what the convergence is sending while issueing an import, except that you will probably need to get first some kind of JSESSION stuff first.
          • 2. Re: CLI tools to import Calendar and AddressBook entries into multiple users?
            Thanks a lot for the hint, it does seem the way to go, except that I can't get it working so far :(
            One thing to look out for is the line-ending character (should be DOS-style CRLF), so I have to "dos2unix" and "unix2dos" my prepared POST data to get it cleaned up properly (comes from various sources, so both endings are present in original blob of text).

            However, now that I got the appserver to accept the post data, calendar says "missing file param", or in more detail:
            PRODID:-//Oracle Corporation/CS 7u2-4.19//EN
            X-NSCP-WCAP-ERRNO-MESSAGE:Invalid arguments such as invalid email address
            , invalid date.
            X-NSCP-WCAP-ERRNO-DETAILED-MESSAGE:missing file param
            I've tried with "id" and proper login session (via login.wcap) instead of simple http auth, with the same result.

            Am I hitting something stupid simple? ;)

            Next I'd have to try it with a real web-browser...

            Edited by: JimKlimov on Apr 24, 2013 3:22 AM

            UPDATE: Tried with a browser, same result when I login with either a proxy-user and its pass, or the end-user and his pass. Logins with bad credentials do fail.
            Maybe the ICS data format is unexpected by Calendar Server (too many fields, some Microsoft-centric), or there are problems with UTF-8 text pieces in the entries? (Produced with DavMail, an open-source tool recommended by an Oracle engineer)
            Alas, same result for the sample ICS markup in that Wiki post...

            Thanks in advance for any more ideas :)
            • 3. Re: CLI tools to import Calendar and AddressBook entries into multiple users?
              Could you put the following import.html in your app server hosting the Calendar Server and use it in a browser to test your import? Adjust the form ACTION and calid as appropriate for your testing.

              <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
              <title>WCAP Import Page</title>
              <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
              <FORM METHOD=POST ENCTYPE="multipart/form-data"
              <li>file to import<input type="file" accept="text" name="file"></li>
              <li>Press Import Now:<input type="submit" value="Import Now"></li>
              • 4. Re: CLI tools to import Calendar and AddressBook entries into multiple users?
                I believe, this is a variation on the sample form in https://wikis.oracle.com/display/CommSuite/import.wcap wiki post, and I've unsuccessfully tried a few of these.
                Thanks for your suggestion, it does differ in some details, so I'll try it - but don't hold my breath now that it would really work.

                Here's what I last tried:

                <FORM METHOD=POST ENCTYPE="multipart/form-data"
                     &lt;li>file to import:<input type="file" accept="text" name="Upload">&lt;/li>
                     &lt;li>Press Import Now:<input type="submit" value="Import Now">&lt;/li>
                • 5. Re: CLI tools to import Calendar and AddressBook entries into multiple users?
                  The name parameter of the <input> has to be "file" and that's what the import.wcap handler is looking for.
                  • 6. Re: CLI tools to import Calendar and AddressBook entries into multiple users?

                    Doh, that was an out-of-attention minor point with vast consequences.
                    Thanks man, you rock! You know more than the Wiki, which I'll now ask to fix

                    At least for small test ICS files, the import succeeded, from my script also. Larger ones (one user's export is a 2Mbyte text ICS file) did not pass well, but I think in the worst case I can split them into smaller files by date-ranges, line amount, kilobytes or such.

                    Do you by chance know any similar secret paths to AddressBook imports?

                    Thanks again,
                    //Jim Klimov

                    Edited by: JimKlimov on May 4, 2013 4:53 AM

                    UPDATE: The big files' problem does not seem to be a server problem, but rather one of my data provider (exporter from another system) which does inded build problematic markup (chunked encoding interspersed with chunk size tags, which can happen in middles of data lines).


                    UPDATE2: With a proper script to parse the un-chunking of ICS text received from another system, we were able to verify the validity of the stream (exporter had its hiccups and aborted replies as well), and applying some format workarounds for WCAP parser in OUCS Caledar Server, ultimately ensuring higher format validity of ICS data for OUCS. In the end run, all calendar data got imported, even for a large account with 3Mb worth of markup in an ICS file.