2 Replies Latest reply: Apr 19, 2009 9:15 PM by 807557 RSS

    Need your help. File system Jitter.

    807557
      I just srarted working on RTSJ. The real time GC is great and behaves as expected. Now I'm having a problem.
      I realized that my file system is not real time and it creates a lot jitters. Basically this is how my application works: It has to persist every message it received to a file. For performance reason we don't want to force sync on every write. And the write is always sequential. Each Considering the file system should provide cached buffer there shouldn't be any problem. However it is not the case.

      I wrote the following program to perform a test: It opens 5 files. Each time it may perform burst sequential writes to the file. Each write is 100-200 bytes. Each burst is 1000-2000 writes. Then the program measures the worst case latency.

      To my surprise, the worst case latency in my case is huge. May be over 1 second.

      I understand it may not be a java issue. It may be OS issue. But I'm wondering if there is any way to tune the file system to avoid the jitter (use larger buffer maybe?)

      File IO on /tmp with tmpfs creates no jitter at all.

      Thanks a lot.

      public class TestMain {

      public class TestMain {

      /**
      * @param args
      */
      public static void main(String[] args) {
      PriorityScheduler sched = PriorityScheduler.instance();
      new RealtimeThread(
      new PriorityParameters(sched.getMaxPriority()),
      new SporadicParameters(new RelativeTime(0L, 50000), new RelativeTime(0L, 5000000), new RelativeTime(0L, 500000), null, null),
      null,
      null,
      null,
      task).start();
      try {
      Thread.sleep(Long.MAX_VALUE);
      } catch (InterruptedException e) {
      e.printStackTrace();
      }
      }

      private static void statistics(double[] intervals, int length) {
      double total = 0;
      double min = Double.MAX_VALUE;
      double max = Double.MIN_VALUE;
      for (int i=0; i<length; i++) {
      total += intervals;
      if (intervals[i] < min)
      min = intervals[i];
      if (intervals[i] > max)
      max = intervals[i];
      }
      System.out.println(String.format("Count=%d; Average=%.2f; Min=%.2f; Max=%.2f",
      length, total/length, min, max));
      }

      private static final int BATCH_MIN = 1000;
      private static final int BATCH_MAX = 2000;
      private static final int FILES = 5;
      private static void run() {
      try {
      FileOutputStream[] files = new FileOutputStream[FILES];
      for (int i=0; i<FILES; i++) {
      files[i] = new FileOutputStream("/home/weiqj/temp/FIXLog/Test.bin" + i);
      }
      BufferedReader input = new BufferedReader(new InputStreamReader(System.in));
      double[] intervals = new double[BATCH_MAX];
      byte[] buffer = new byte[BATCH_MAX];
      Random r = new Random();
      Clock clock = Clock.getRealtimeClock();
      while (true) {
      int curBatch = r.nextInt(BATCH_MAX - BATCH_MIN) + BATCH_MIN;
      System.out.println(String.format("Press Enter to send %d messages...", curBatch));
      String command = input.readLine().toLowerCase();
      if (command == "exit" || command == "quit" || command == "bye")
      break;
      for (int i = 0; i < curBatch; i++) {
      int size = r.nextInt(100) + 100;
      AbsoluteTime startTime = clock.getTime();
      r.nextBytes(buffer);
      for (int j=0; j<FILES; j++) {
      files[j].write(buffer, 0, size);
      files[j].getChannel().position();
      }
      AbsoluteTime endTime = clock.getTime();
      RelativeTime elapsed = endTime.subtract(startTime);
      intervals[i] = (double)(elapsed.getMilliseconds() * 1000000 + elapsed.getNanoseconds()) * 1e-6;
      }
      statistics(intervals, curBatch);
      }
      } catch (IOException e) {
      }
      }

      private static Runnable task = new Runnable() {
      public void run() {
      TestMain.run();
      }
      };
      }

      Edited by: weiqj on Apr 18, 2009 10:05 PM
        • 1. Re: Need your help. File system Jitter.
          807557
          By the way, I'm using Suse with Ext3 file system.
          • 2. Re: Need your help. File system Jitter.
            807557
            ext3 is a journaling filesystem so I would not be surprised to see occasional very long latencies - indeed google for "Ext3 latency fixes" and you'll find current work in this area. But for details on how ext3 operates and how it might be tuned for improved latency you are best of contacting a Linux/ext3 forum or newsgroup somewhere.

            From a real-time perspective I'd suggest using your own buffering scheme so that the main application real-time threads don't write the files to disk - use a background thread for that.

            David Holmes