4 Replies Latest reply: Feb 19, 2013 4:28 PM by vinothchandar RSS

    Rationale behind the default cleaning interval

    vinothchandar
      Hi,

      [We are using 4.1.17]

      I have a question around the rationale behind setting the default cleaner interval to 25% of log file size, while the "je.cleaner.minUtilization"=50 by default.

      As I understand it, the equilibrium is when you write X bytes and you have to read a single log file 2X in size. That way, for every X bytes written, the cleaner writes back X bytes of data and gets rid of X bytes of stale data. So, based on this I set the log file to 60MB and adjusted the cleaner byte interval to 30MB. But, we find cleaner falling behind or reading a lot more, when we do this, than when setting cleaner bytes interval to 15MB.

      So, if someone can clarify why 25% minUtilization as default, could help us a lot.

      Thanks
      Vinoth
        • 1. Re: Rationale behind the default cleaning interval
          Greybird-Oracle
          The cleaner byte interval is just a wakeup interval. When it wakes up, it determines whether cleaning is needed (based on current log utilization) and doesn't do anything if utilization is above the configured threshold. The cleaner should be woken up often enough ensure that cleaning happens, but there's very little cost to waking it up more often than necessary.

          Does that answer your question?

          --mark                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           
          • 2. Re: Rationale behind the default cleaning interval
            vinothchandar
            Hi Mark,

            Thanks for the quick answer.

            I still need some clarification I think. I think 30MB cleaner interval would do less actual cleaning writes/reads than 15MB (60MB log size, 50% util and assume each file is at 50% util for simplicity), for the following reason

            Assume we start at 50% util,
            With 30MB cleaner interval : We read one file and log half of it (60MB read, 30MB written) , for 30MB worth of actual writes
            With 15MB cleaner interval : We read one file and log half of it (60MB read, 30MB written) , for 15MB worth of actual writes.

            We would need two cleaner runs with 15MB cleaner interval for 30MB worth of actual writes, doubling the IO. The idea is basically adjust the cleaner interval to write/read least amount of data during the cleaner run.

            I am not asking for an optimal config. just making sure my understanding is right?

            [In both cases, of course the next cleaner run will be a cheap no-op as you mentioned. Thanks for clarifying this]

            Thanks
            Vinoth
            • 3. Re: Rationale behind the default cleaning interval
              Greybird-Oracle
              Maybe, I don't really know. When the cleaner is woken, it often queues multiple files for cleaning. We've never had a problem with waking up the cleaner using the default setting, so we've never investigated further. If you're in doubt, simply shorten the interval.

              --mark                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           
              • 4. Re: Rationale behind the default cleaning interval
                vinothchandar
                Makes sense.. I will leave it at 15MB, default. Thanks Mark!

                Thanks
                Vinoth