1 2 Previous Next 20 Replies Latest reply: Jul 13, 2011 8:11 AM by 775682 RSS

    Issues using Perl with BDB XML 2.5.16

    775682
      Hi,

      I wrote a perl script to insert a big XML (with empty values). When I try to do it (the script creates the container when it starts), the insertion is not done and the process does not end, being blocked on the statement "err = stash.updateIndex(oc, this)" in Container::reindex function in dbxml/Container.cpp file
      The surprising thing is that if I insert a short XML (by deleting the env files and restarting it from scratch), the insertion of the big XML always work after that. I tried to load the big document with dbxml binary and it works fine.

      The perl code is available at http://pastebin.fr/8790 (the xml is included), but you can access the xml too at http://pastebin.fr/8791

      The last error is when I try to not import simple from Sleepycat::DbXml. The perl script segfaults.

      Regards
        • 1. Re: Issues using Perl with BDB XML 2.5.16
          Laurenfoutz-Oracle
          It seems that dbxml is deadlocking itself while automatically creating indexes for that document when you insert it. I will look into this and get back to you when I find a solution to the bug, but until then you can get around the problem by turning off auto indexing in the container. The Perl API does not have the command to do this yet, but you can do it in the dbxml shell as follows:
          dbxml-2.5.16\DATA>dbxml.exe -tc
          
          dbxml> createContainer test.dbxml
          Creating node storage container
          dbxml> setAutoIndexing off
          Set auto-indexing state to off, was on
          Note that you will want to manually add some indexes after turning off auto indexing in order to speed up querying your document.

          Lauren Foutz
          • 2. Re: Issues using Perl with BDB XML 2.5.16
            Laurenfoutz-Oracle
            It turns out the bug is due to a mistake in your program. You do not properly configure the container for transactions, and that is what results in the deadlock. You need to add $xcc->setTransactional(1); to your program as follows.
              $xcc = new XmlContainerConfig();
              $xcc->setAllowCreate(1);
              $xcc->setTransactional(1);
              $container = $theMgr->openContainer($theContainer,$xcc);
            Lauren Foutz
            • 3. Re: Issues using Perl with BDB XML 2.5.16
              775682
              It turns out the bug is due to a mistake in your program. You do not properly configure the container for transactions, and that is what results in the deadlock. You need to add $xcc->setTransactional(1); to your program as follows.
              do you have any explanation for the reason it segfaults when I don't import 'simple' from the module ?
              • 4. Re: Issues using Perl with BDB XML 2.5.16
                Gmfeinberg-Oracle
                XML::Simple has nothing to do with BDB XML. Is that what you are referring to? With the change Lauren mentioned I can run your program as well.

                Regards,
                George
                • 5. Re: Issues using Perl with BDB XML 2.5.16
                  775682
                  XML::Simple has nothing to do with BDB XML. Is that what you are referring to? With the change Lauren mentioned I can run your program as well.
                  No, I was talking about Sleepycat::DbXml <b>'simple'</b>
                  • 6. Re: Issues using Perl with BDB XML 2.5.16
                    Gmfeinberg-Oracle
                    Oh, I see. That looks to be a bug... Perl is maintained partially externally. It's not clear when this might get investigated. Is it blocking you?

                    Regards,
                    George
                    • 7. Re: Issues using Perl with BDB XML 2.5.16
                      775682
                      Is it blocking you?
                      No, just curious :)
                      • 8. Re: Issues using Perl with BDB XML 2.5.16
                        775682
                        It turns out the bug is due to a mistake in your program. You do not properly configure the container for transactions, and that is what results in the deadlock. You need to add $xcc->setTransactional(1); to your program as follows.
                        Ok, but why does it work if I first insert a small document ? shouldn't it yield if we do something wrong (like using transactions with a container opened without the support set) ?
                        • 9. Re: Issues using Perl with BDB XML 2.5.16
                          Gmfeinberg-Oracle
                          Berkeley DB itself is extremely flexible in how it can be configured and used. The example you use "using transactions when not configured for them" will actually generate an error because DB checks for that. However it's perfectly OK to not use transactions when they are configured. This is usually where people get into trouble with locking, because locks are taken regardless of transactions.

                          Regards,
                          George
                          • 10. Re: Issues using Perl with BDB XML 2.5.16
                            775682
                            The example you use "using transactions when not configured for them" will actually generate an error because DB checks for that.
                            The fact is that I got no error and the process is blocked. That's why I say it should complain and throw an exception.
                            • 11. Re: Issues using Perl with BDB XML 2.5.16
                              Gmfeinberg-Oracle
                              BDB XML and Berkeley DB attempt to detect and fail for illegal configurations. The problem is that the one you ended up with is legal, it just does not work properly if you have more than one process involved. It works fine with just one. It's not possible to detect that you really wanted a transactional configuration for that particular container.

                              That said, there's a valid argument for simplifying the array of allowable Berkeley DB into a small set of allowable Berkeley DB XML configurations. That is, adding a policy layer that effectively protects applications from themselves. That sort of thing has been on the "list" for a while because of situations like this one. Also, it could be that the default configuration for a container in an environment with transactions would be "transactional" (right now that's not the case).

                              For example, let's say we created a hypothetical XmlManagerConfig object passed to the XmlManager constructor with options like:
                              o transactional or not
                              o transactional with MVCC
                              o containers default to transactional (or not)
                              o cache size
                              o a few other useful DB_ENV items like deadlock detection policy,

                              These would create the DB_ENV for you with the correct flags so you don't have to worry about things like the log and locks (which are required for transactional).

                              Do you think that would help? I'm not saying this will happen, just brainstorming a little. Note that it's not hard for an application to do this itself in some self-contained, well-tested classes. It's better done internally but it could be done externally as well.

                              Regards,
                              George
                              • 12. Re: Issues using Perl with BDB XML 2.5.16
                                775682
                                BDB XML and Berkeley DB attempt to detect and fail for illegal configurations. The problem is that the one you ended up with is legal, it just does not work properly if you have more than one process involved.
                                You missed something : there's only one process doing only one insert
                                For example, let's say we created a hypothetical XmlManagerConfig object passed to the XmlManager constructor with options like:
                                o transactional or not
                                o transactional with MVCC
                                o containers default to transactional (or not)
                                o cache size
                                o a few other useful DB_ENV items like deadlock detection policy,
                                These would create the DB_ENV for you with the correct flags so you don't have to worry about things like the log and locks (which are required for transactional).
                                Do you think that would help?
                                I think only dying (or throwing an exception, or activating it if possible) if we try to start a transaction while the transactional mode or support is not enabled, should be enough at first.
                                • 13. Re: Issues using Perl with BDB XML 2.5.16
                                  Gmfeinberg-Oracle
                                  Sorry, I was juggling too many things... Yes, it is possible to detect an attempt to pass a transaction to a non-transactional container and throw an exception. We will look at adding this but it may not be a patch for the general public. It'd involve a number of small changes at various entry points.

                                  Regards,
                                  George
                                  • 14. Re: Issues using Perl with BDB XML 2.5.16
                                    775682
                                    We will look at adding this but it may not be a patch for the general public.
                                    Why don't you give access to a git repository with multiple branch with patches for example ? It would be great ! I don't understand why you only send patch by email :(

                                    Regards
                                    1 2 Previous Next