2 Replies Latest reply: Jul 7, 2012 4:15 PM by 909806 RSS

    INCompressor thread lagging

    909806
      I have a tool which I run when I end up with a large cleaner backlog. It's based on some sample code I fished out of the forum, and it basically just opens the environment, optionally deletes a couple of expendable databases, manually runs the cleaner in several threads until the backlog is gone, and closes the environment. (Both je.env.runCleaner and je.env.runCheckpointer are set to false.)

      I've noticed that in some cases the INCompressor thread keeps running after the final checkpoint has finished, leading to a very large recovery time when the environment is next opened (because a lot ends up being written to the log after the last checkpoint). I presume this is due to the large number of cleaner threads.

      (I see that early in the environment close code, the daemon threads are notified to exit, but the INCompressor code doesn't check this flag until after it has finished a set of work items, and I'm guessing it's working through a very long list.)

      As a test, I set je.env.runINCompressor to false for a run of my tool, and in this case the environment closes quickly and also subsequently opens quickly. But I don't know what the consequences are, of not running the INCompressor thread at all during this operation. Is this safe to do? Does it cause bloat in the log files (because it's skipping some sort of cleanup)?

      I didn't see any way to manually wait for the INCompressor thread to terminate before issuing a final checkpoint, so disabling the INCompressor thread was the only workaround I could think of.

      (This is under JE 4.1.17.)
        • 1. Re: INCompressor thread lagging
          Greybird-Oracle
          (I see that early in the environment close code, the daemon threads are notified to exit, but the INCompressor code doesn't check this flag until after it has finished a set of work items, and I'm guessing it's working through a very long list.)
          Correct. We may want to improve this by exiting the compression loop sooner when the environment is being shut down, but that may have negative effects as well. I don't think we've done an analysis of the trade-offs. Thanks for pointing it out.
          As a test, I set je.env.runINCompressor to false for a run of my tool, and in this case the environment closes quickly and also subsequently opens quickly. But I don't know what the consequences are, of not running the INCompressor thread at all during this operation. Is this safe to do? Does it cause bloat in the log files (because it's skipping some sort of cleanup)?
          There is a drawback to disabling the compressor thread, which is that deleted branches of the Btree (if you've deleted large ranges of records) or individual record slots (if you haven't) will not be removed from the Btree before the final checkpoint. Although these will be removed eventually, after opening the environment later, it is more efficient for them to be removed before the checkpoint, since there will be less total writing of the log.
          I didn't see any way to manually wait for the INCompressor thread to terminate before issuing a final checkpoint, so disabling the INCompressor thread was the only workaround I could think of.
          There is another way to force the compressor to finish its work: After doing your log cleaning, call Environment.compress(). This method is synchronized, so it will block if compression is running in the background thread, and then try compressing again. In any case, when it returns, compression will be complete. Then close the environment.

          Please let us know if this helps.

          One thing I'm confused about is that you said delete a couple databases and then run the log cleaner. Bu if you're calling Environment.removeDatabase or truncateDatabase, compression will not be necessary, so I'm not sure why the compressor is doing so much work. Are you deleting individual records? If so, that explains it.

          --mark                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               
          • 2. Re: INCompressor thread lagging
            909806
            Thank you, that's very helpful, and exactly what I needed to know. I'll add the call to Environment.compress() before closing the environment.
            One thing I'm confused about is that you said delete a couple databases and then run the log cleaner. But if you're calling Environment.removeDatabase or truncateDatabase, compression will not be necessary, so I'm not sure why the compressor is doing so much work. Are you deleting individual records? If so, that explains it.
            Sorry, that was probably an irrelevant detail, but I included it in case it was significant. I'm running my tool because I already have a cleaner backlog beforehand -- I assume it's the cleaning of this backlog that's generating the work for the compressor. I am in fact using Environment.removeDatabase(), and doing it at this point just to free up additional space, since I'm cleaning anyway, but this isn't the origin of all of the cleaner work.
            We may want to improve this by exiting the compression loop sooner when the environment is being shut down, but that may have negative effects as well. I don't think we've done an analysis of the trade-offs. Thanks for pointing it out.
            Also as-is I think you are sort of in a middle-ground, because it keeps running until the current batch is done, but doesn't consume before exiting any additional work which has accumulated. So there could be negative effects of already stopping somewhat early.