6 Replies Latest reply: Jan 25, 2011 8:49 PM by user13712203 RSS

    Orion on RHEL 5.4 - weird error

    user13712203
      Hey everyone. I am receiving the below error every time I try to run Orion. This is on RHEL5.4 x64. The error always appears during the first operation dealing with both a small and large I/O request.
      I have verified that libaio (libaio-devel as well) is installed. I have added /usr/lib and /usr/lib64 to the $LD_LIBRARY_PATH variable just to be safe.
      I am using a multipath SAN over FC. This is just for testing so I ran the following commands:
      /bin/raw /dev/raw/raw1 /dev/mpath/mpath0
      /bin/raw /dev/raw/raw2 /dev/mpath/mpath1
      /bin/raw /dev/raw/raw3 /dev/mpath/mpath2

      I have verified the raw volumes work correctly with dd.

      This is the command I run to execute orion along with the output:


      [root@baspeadbs01 orion]# ./orion_linux_x86-64 -run advanced -testname provisioning -num_disks 24 -write 57 -matrix detailed -duration 60 -cache_size 2048 -simu
      late concat -verbose
      ORION: ORacle IO Numbers -- Version 11.1.0.7.0
      provisioning_20110125_1441
      Test will take approximately 417 minutes
      Larger caches may take longer

      Name: /dev/raw/raw1 Size: 586608541696
      Name: /dev/raw/raw2 Size: 586608541696
      Name: /dev/raw/raw3 Size: 586608541696
      3 FILEs found.


      TEST START

      ----------------------------------------------


      Running point: Small=0, Large=0
      Point 1 out of 416
      Valid



      Running point: Small=1, Large=0
      Point 2 out of 416
      Valid



      Running point: Small=2, Large=0
      Point 3 out of 416
      Valid



      Running point: Small=3, Large=0
      Point 4 out of 416
      Valid



      Running point: Small=4, Large=0
      Point 5 out of 416
      Valid



      Running point: Small=5, Large=0
      Point 6 out of 416
      Valid



      Running point: Small=6, Large=0
      Point 7 out of 416
      Valid



      Running point: Small=12, Large=0
      Point 8 out of 416
      Valid



      Running point: Small=18, Large=0
      Point 9 out of 416
      Valid



      Running point: Small=24, Large=0
      Point 10 out of 416
      Valid



      Running point: Small=30, Large=0
      Point 11 out of 416
      Valid



      Running point: Small=36, Large=0
      Point 12 out of 416
      Valid



      Running point: Small=42, Large=0
      Point 13 out of 416
      Valid



      Running point: Small=48, Large=0
      Point 14 out of 416
      Valid



      Running point: Small=54, Large=0
      Point 15 out of 416
      Valid



      Running point: Small=60, Large=0
      Point 16 out of 416
      Valid



      Running point: Small=66, Large=0
      Point 17 out of 416
      Valid



      Running point: Small=72, Large=0
      Point 18 out of 416
      Valid



      Running point: Small=78, Large=0
      Point 19 out of 416
      Valid



      Running point: Small=84, Large=0
      Point 20 out of 416
      Valid



      Running point: Small=90, Large=0
      Point 21 out of 416
      Valid



      Running point: Small=96, Large=0
      Point 22 out of 416
      Valid



      Running point: Small=102, Large=0
      Point 23 out of 416
      Valid



      Running point: Small=108, Large=0
      Point 24 out of 416
      Valid



      Running point: Small=114, Large=0
      Point 25 out of 416
      Valid



      Running point: Small=120, Large=0
      Point 26 out of 416
      Valid


      ----------------------------------------------


      Running point: Small=0, Large=1
      Point 27 out of 416
      Valid



      Running point: Small=1, Large=1
      Point 28 out of 416
      storax_aiowait: IO returned an error 27061
      OER 27061: waiting for async I/Os failed
      Linux-x86_64 Error: 14: Bad address
      Additional information: -1
      Additional information: 1048576
      lun_aiowait: storax_aiowait failed.
      rwbase_run_test: rwbase_reap_req failed
      rwbase_run_process: rwbase_run_test failed
      rwbase_rwluns: rwbase_run_process failed
      orion_thread_main: rw_luns failed
      Test error occurred
      Orion exiting



      If anyone has any ideas I would greatly appreciate the help. As it stands I am getting some information about the small I/O's.
      Thank you!
        • 2. Re: Orion on RHEL 5.4 - weird error
          user13712203
          Thank you for the response. I did see the link, however I'm not sure it applies here. First, there's no way to set the parameters as suggested since we're not running VxFS, or Oracle (yet). We're running Orion on raw filesystems. I don't know how to set the max IO rate on a raw filesystem in linux when there's no Oracle installed.

          Any suggestions would really be appreciated.

          I've also looked at these articles:
          Asysnc IO Error: 14: Bad address
          http://seilerwerks.wordpress.com/2007/12/
          • 3. Re: Orion on RHEL 5.4 - weird error
            Dude!
            How about the following:

            Running orion – to avoid Bug 9104898: ORION FAILS WITH ORA-27061: WAITING FOR ASYNC I/OS FAILED

            BUG 9104898 mentions that version 11.1.0.0 is working

            http://blogs.netapp.com/databases/2010/12/running-orion-to-avoid-bug-9104898-orion-fails-with-ora-27061-waiting-for-async-ios-failed.html

            (Bug 9104898: ORION FAILS WITH ORA-27061: WAITING FOR ASYNC I/OS FAILED).

            Orion runs just fine until it comes to random large i/o:

            ran (large): VLun = 0 Size = 2914705735680
            Error completing IO
            (storax_aiowait)
            ORA-27061: waiting for async I/Os failed
            Linux-x86_64 Error: 14: Bad address
            Additional information: -1
            Additional information: 1048576
            lun_aiowait: storax_aiowait failed.
            rwbase_run_test: rwbase_reap_req failed
            rwbase_run_process: rwbase_run_test failed
            rwbase_rwluns: rwbase_run_process failed
            orion_thread_main:rwbase_rwluns failed

            Orion 11.1.0.0.0 works, Orion 11.1.0.7.0 shows the same error.
            • 4. Re: Orion on RHEL 5.4 - weird error
              user13712203
              Thank you for the pointer, I hadn't seen that article. Do you happen to know where an older version of Orion is available? There are no options listed to download prior versions on the download page here:
              http://www.oracle.com/technetwork/topics/index-089595.html
              • 5. Re: Orion on RHEL 5.4 - weird error
                Dude!
                Sorry, no idea where to download it. I found some info that until 11.1.0.7 Orion wasn’t supported but in 11.2.0.1 Orion is part of the RDBMS installation. Orion 11.1.0.0.0 is the last known working version, Orion 11.1.0.7.0 and 11.2.0.1 both fail with the above error (since Nov 2009).
                • 6. Re: Orion on RHEL 5.4 - weird error
                  user13712203
                  Drat. Well, thank you very much for the heads up, I really appreciate it. The wayback machine doesn't seem to have any links and there's nothing on oldversion. I'll see if I can't find a workaround. Cheers!