11 Replies Latest reply: Apr 8, 2010 11:59 PM by 765576 RSS

    Python segfault

    761557
      Hello,

      Follows is a test program I have written. I am attempting to do multi-process inserts. Sometimes the program appears to deadlock and every other time, when it completes, it segfaults at the end complaining that a database handle is still in use. I am opening and deleting the container in the parent process and using the multiprocess module to handle the forking and IPC for me.

      from bsddb3.db import *
      from dbxml import *
      import time

      from multiprocessing import Process, Pool, Queue

      numberOfItems = 100000

      xml = """<item><type/></item>"""

      def strAsDocument(mgr, str):
      doc = mgr.createDocument()
      doc.setContent(str)
      return doc

      def insertDoc(container, environment, mgr, number):
      xtxn = mgr.createTransaction()
      uc = mgr.createUpdateContext()
      names = [];
      print "inserting " + str(number) + " records"
      for i in xrange(number):
      name = container.putDocument(xtxn, 'item', xml, uc, DBXML_GEN_NAME)
      names.append(name)
      xtxn.commit()
      print "done";
      del uc
      del xtxn

      def go():
      environment = DBEnv()
      environment.set_cachesize(0, 25 * 1024 * 1024)
      environment.open("env", DB_CREATE|DB_INIT_LOCK|DB_INIT_LOG|DB_INIT_MPOOL|DB_INIT_TXN|DB_RECOVER|DB_THREAD, 0)
      try:
      config = XmlContainerConfig()
      config.setAllowCreate(True)
      config.setTransactional(True)
      mgr = XmlManager(environment, 0)
      uc = mgr.createUpdateContext()

      try:
      mgr.removeContainer("test.dbxml")
      except:
      pass
      container = mgr.openContainer("test.dbxml", config)
      container.setAutoIndexing(False, uc)

      before = time.time()
      ps = []
      for i in range(5):
      p = Process(target=insertDoc, args=(container, environment, mgr, 2000))
      p.start()
      ps.append(p)

      for p in ps:
      p.join()

      print time.time() - before

      del container
      del mgr
      del uc

      except XmlException, inst:
      print "XmlException (", inst.exceptionCode,"): ", inst.what
      if inst.exceptionCode == DATABASE_ERROR:
      print "Database error code:",inst.dbError

      environment.close(0)

      for i in range(5):
      go()



      gives me:

      [root@vladivar python]# python test.py
      inserting 2000 records
      inserting 2000 records
      inserting 2000 records
      inserting 2000 records
      inserting 2000 records
      done
      done
      done
      done
      done
      2.8827149868
      Traceback (most recent call last):
      File "test.py", line 72, in <module>
      go()
      File "test.py", line 69, in go
      environment.close(0)
      bsddb3.db.DBInvalidArgError: (22, 'Invalid argument -- Open database handle: test.dbxml/secondary_configuration')
      Segmentation fault

      Thanks.
        • 1. Re: Python segfault
          637288
          I don't have any experience with Python bindings for DB XML, but the error looks general to me. It seems that you don't close/delete some resources, like XmlResults or XmlQueryExpresssion (I don't see such objects in your code, but maybe you simplified the program for the forum?). Also there was a nice FAQ entry explaining which resources should be close/deleted explicitly. But AFAIK it's slightly out-dated now

          Vyacheslav
          • 2. Re: Python segfault
            671148
            please post this code in paste or format - it's just about impossible to read

            Because the python bindings are using swig - you have to exercise the "thisown" attribute of some objects. I can't tell with your code where it is but in most cases you have to set manager.thisown = False

            hth,

            eleddy
            • 3. Re: Python segfault
              761557
              Apologies for the formatting. I read the FAQ so here goes:
              from bsddb3.db import *
              from dbxml import *
              import time
              
              from multiprocessing import Process, Pool, Queue
              
              numberOfItems = 100000
              
              xml = """<item><type/></item>"""
              
              def strAsDocument(mgr, str):
                  doc = mgr.createDocument()
                  doc.setContent(str)
                  return doc
              
              def insertDoc(container, environment, mgr, number):
                  xtxn = mgr.createTransaction()
                  uc = mgr.createUpdateContext()
                  names = [];
                  print "inserting " + str(number) + " records"
                  for i in xrange(number):
                      name = container.putDocument(xtxn, 'item', xml, uc, DBXML_GEN_NAME)
                      names.append(name)
                  xtxn.commit()
                  print "done";
                  del uc
                  del xtxn
                  del container
              
              def go():
                  environment = DBEnv()
                  environment.set_cachesize(0, 25 * 1024 * 1024)
                  environment.open("env", DB_CREATE|DB_INIT_LOCK|DB_INIT_LOG|DB_INIT_MPOOL|DB_INIT_TXN|DB_RECOVER|DB_THREAD, 0)
                  try:
                      config = XmlContainerConfig()
                      config.setAllowCreate(True)
                      config.setTransactional(True)
                      mgr = XmlManager(environment, 0)
                      uc = mgr.createUpdateContext()
              
                      try:
                          mgr.removeContainer("test.dbxml")
                      except:
                          pass
                      container = mgr.openContainer("test.dbxml", config)
                      container.setAutoIndexing(False, uc)
              
                      before = time.time()
                      ps = []
                      for i in range(5):
                          p = Process(target=insertDoc, args=(container, environment, mgr, 2000))
                          p.start()
                          ps.append(p)
              
                      for p in ps:
                          p.join()
              
                      print time.time() - before
                      
                      del container
                      del mgr
                      del uc
              
                  except XmlException, inst:
                      print "XmlException (", inst.exceptionCode,"): ", inst.what
                      if inst.exceptionCode == DATABASE_ERROR:
                          print "Database error code:",inst.dbError
              
                  environment.close(0)
              
              for i in range(5):
                  go()
              gives me:
              [root@vladivar python]# python test.py
              inserting 2000 records
              inserting 2000 records
              inserting 2000 records
              inserting 2000 records
              inserting 2000 records
              done
              done
              done
              done
              done
              2.8827149868
              Traceback (most recent call last):
              File "test.py", line 72, in <module>
              go()
              File "test.py", line 69, in go
              environment.close(0)
              bsddb3.db.DBInvalidArgError: (22, 'Invalid argument -- Open database handle: test.dbxml/secondary_configuration')
              Segmentation fault
              I didn't simplify the code for the forum, I made this test case for it and ran it.

              If I remove the multiprocess logic, and single thread it I get no warnings about open database handles. I'll read the SWIG documentation and see what I can figure out.

              Out of interest, does anyone know how to configure whether the logs and database are flushed to disk with Python? Doing it with Java is easy - but I couldn't find a flag in the documentation for Python.

              I also found that Python performance is much faster than Java which I believe might be because Python has different defaults for bdb. The performance in Python seems similar to what I get in Java if I set setLogInMemory(true).
              • 4. Re: Python segfault
                671148
                just looking at your code there are a couple things that stick out. First, in insertDoc() you are deleting a container which was created outside and passed in. I suspect there is your rub - especially since you delete it later in go(). As a rule when working with swig and this lib, always del in the exact opposite order its created, and never in another function. messy!

                as far as multiprocessing goes, I'm not sure why you would be using it here, and especially without some sort of locking implemented. not all of these operations are safe to be run in a multiprocess environment - there is a C++ manual on that and all those rules apply since this is a swig wrapper. simple semaphores should work though and this is the one place where the GIL will seem to help you out with deadlocks and serialization. Curious as to your choice of the multiprocessing lib - any particular reason why?

                by default logs+data are flushed to disk. You have to use flags to get that to not happen. theoretically you can set up memp_trickle but this is my notes on the code

                # XXX: This causes mega env corruption. hmmmm...
                #environment.memp_trickle(15)

                hth,
                eleddy
                • 5. Re: Python segfault
                  761557
                  Hi Eleddy,

                  Thanks for your response.

                  The reason I am using multiprocessing is that I am attempting to prototype a design that allows multiple processes to service requests that make unrelated database calls. In order to increase efficiency I do not want to reopen containers/the environment on every request so I will have a pool of workers that maintain their environment and container handles between requests.

                  The reason I am deleting the objects in insertDoc() is because this is only called by the child processes and I was getting errors that a database handle was still open. Therefore I was experimenting to see whether the child process was holding the handles open because of these objects. The child execution path does not return to go() therefore those del calls will not happen.

                  I previously read in the C++ documentation you refer to that environment handles (if DB_THREAD is given) and XmlContainer handles are free-threaded and that is why I am attempting to use them cross-process. AFAIK I am not trying to use any objects that the documentation says are not free-threaded.

                  The reason I am using the multiprocessing library rather than threads is because I did not like what I read about GIL contention in threaded environments.
                  • 6. Re: Python segfault
                    671148
                    Hey -

                    I attached gdb to your thread and its crashing because you are deleting the manager and env in the process that finished first while the others are still working. You need to implement something like join until you delete all the stuff.

                    Also, you at the very minimum need to have your environment creation and open and container opening protected with some sort of synchronization primitive. Otherwise your environment WILL corrupt with any sort of concurrency at all. I've been there. It sucks.

                    An aside/recommendation, running the multiprocess module is not going to give you what you describe. It's main purpose is to better utilize multi-threading on multiple core machines. Additionally, the bdbxml framework handles multiple processes through those environment files so if you are sharing an environment in code you aren't going fully be testing that. As to performance and the GIL, in most cases its better just to use the threading model because of the expense of forking, ESPECIALLY on windows. (David Beasley did a great study/talk on this that you can easily find.)This is especially true for db work since its disk bound, not CPU bound, and multiprocess is meant to free up the CPU. Furthermore, sharing state between processes is highly discouraged. I use threads in a highly concurrent environment and have no issues. If you truly run into performance hits I'd be glad to help out, or explain these modules better.

                    hth,
                    eleddy
                    • 7. Re: Python segfault
                      761557
                      This is very helpful, thanks.

                      w.r.t the GIL it was this posting that put me off: http://www.dabeaz.com/blog/2010/01/python-gil-visualized.html. But on reflection I think you're right; it's probably not an issue because the application will most likely be IO-bound. I will try the threading module on your recommendation (I much prefer threading anyway). I wasn't worried about the expense of forking as I would do this rarely (when a worker was started, which would service multiple requests) and it'll never be running on Windows. It appears though that controlling the database handles through a fork() is harder than I expected.

                      Considering I am going to now try the threading module this might seem a mute point but I did rewrite my test case to open the environment in each process instead of before fork()ing and I had deadlock issues, I'd like to understand why. Do I understand you correctly that I need to serialize opening of the environment and containers? I understand that creation needs to be serialized but opening too? In case you are interested here is my deadlocking (non-segfaulting) test case:
                      from bsddb3.db import *
                      from dbxml import *
                      import time
                      
                      from multiprocessing import Process
                      
                      xml = """<item><type/></item>"""
                      
                      class DBTest:
                          def insertDoc(self, number):
                              uc = self.mgr.createUpdateContext()
                              try:
                                  names = [];
                                  print "inserting " + str(number) + " records"
                                  for i in xrange(number):
                                      name = self.container.putDocument('item', xml, uc, DBXML_GEN_NAME)
                                      names.append(name)
                                  print "done";
                              finally:
                                  del uc
                              
                          def joinEnvironment(self):
                              self.environment = DBEnv()
                              self.environment.open("env", DB_JOINENV|DB_THREAD)
                              
                          @staticmethod
                          def createEnvironment():
                              environment = DBEnv()
                              environment.set_cachesize(0, 25 * 1024 * 1024)
                              environment.open("env", DB_CREATE|
                                               DB_INIT_LOCK|
                                               DB_INIT_MPOOL|
                                               DB_THREAD, 0)
                              environment.close(0)
                      
                          def createContainers(self):
                              mgr = XmlManager(self.environment, 0)
                              uc = mgr.createUpdateContext()
                              config = XmlContainerConfig()
                              config.setAllowCreate(True)
                              config.setThreaded(True)
                      
                              try:
                                  mgr.removeContainer("test.dbxml")
                              except:
                                  pass
                              
                              container = mgr.openContainer("test.dbxml", config)
                              container.setAutoIndexing(False, uc)
                              del container
                              
                          def openContainers(self):
                              config = XmlContainerConfig()
                              config.setAllowCreate(False)
                              config.setThreaded(True)
                      
                              self.mgr = XmlManager(self.environment, 0)
                      
                              self.container = self.mgr.openContainer("test.dbxml", config)
                              
                          def cleanup(self):
                              if hasattr(self, 'container'):
                                  del self.container
                              if hasattr(self, 'mgr'):
                                  del self.mgr
                              if hasattr(self, 'environment'):
                                  self.environment.close(0)
                                  del self.environment
                      
                      
                      # called by fork()ed process
                      def doProcess(num):
                          test = DBTest()
                          try:
                              test.joinEnvironment()
                              test.openContainers()
                              test.insertDoc(num)
                          except XmlException, inst:
                              print "XmlException (", inst.exceptionCode,"): ", inst.what
                              if inst.exceptionCode == DATABASE_ERROR:
                                  print "Database error code:",inst.dbError
                          finally:
                              test.cleanup()
                      
                      # main
                      DBTest.createEnvironment()
                      
                      test = DBTest()
                      test.joinEnvironment()
                      test.createContainers()
                      test.cleanup()
                      
                      ps = []
                      for i in range(3):
                          p = Process(target=doProcess, args=(5000,))
                          p.start()
                          ps.append(p)
                      
                      for p in ps:
                         p.join()
                      • 8. Re: Python segfault
                        671148
                        sigh yeah, the opening needs to be serialized and done by one process only. the reason is that it needs to be atomic in creating the environment. That's because the environment is on disk - those __db files. If more than one process creates or recovers and environment at the same time - kaboom!

                        It gets more interesting. If a transaction fails or something gross happens, the environment needs to be deleted and recovered. This means that if you have multiple processes using the same environment, they need to communicate to eachother when this scenario happens, all except one have to stop and drop all operations, wait for a single process to recover, then retry all transactions. If you are just messing around don't worry about this but if you are building a scalable app this should play into your immediate design. If you are considering this route I'll spend some time finding a "application design manual" that I happenstanced across a while ago. I can also spend some time talking about my implementation but if you aren't looking there I'll save a million words.

                        Your deadlock is most likely occurring from the transactions munging eachother. When I was running your other code I could invoke deadlocks occasionally as well so I wouldn't spend too much time pondering it.

                        Let me know if you need that info - I have a great library for python that is unfortunately closed up in IP blah but I have been meaning to write up a python+xmldb survivor manual for someone who needs it.

                        eleddy
                        • 9. Re: Python segfault
                          761557
                          What about if I open the env and container without specifying create?

                          I switched my first example (minus del calls in insert) to threading instead of multiprocessing and all my deadlock issues went away. I am prototyping for a scalable app so I would be very interested in any documentation you could give me.
                          • 10. Re: Python segfault
                            671148
                            Theoretically you can open without create with no issues, but when you get high concurrency this will break and fast. I have test cases that show it (and therefore make sure my code doesn't do that). If you have concurrency, the best solution is to have one process per environment multi threaded. The good thing about that is that xml data shards really well. We have 3 environments per client * number of clients. It's worked out rather well :)

                            In the multi threaded environment, I have put locks on environment open/create, creating transactions (seems weird but has to be protected), and creating (NOT opening) containers. For the container thing to work, you have to check if it exists first then implement the locking if you need to create it.

                            Let me know if that doesn't answer some questions. I think I can strip out some code from our crap that may help - I'll try and do that.
                            • 11. Re: Python segfault
                              765576
                              Only One word to characterize such a great post “WOW” that was a very interesting read
                              such a wonderful information for me..i am really impress it.
                              Force Factor
                              Force Factor

                              Edited by: user12962801 on Apr 8, 2010 9:59 PM