After updating the kernel to 2.6.39-200.1.9 (this is done automatically by the server manager) then performing a reboot on all 3 OVS 3.1.1 servers they are stuck on the boot screen with the following error:
fsck.ext3: Device or resource busy while trying to open /dev/sda2
Filesystem mounted or opened exclusively by another program?
*** An error occurred during the file system check.
*** Dropping you to a shell; the system will reboot
*** when you leave the shell.
Give root password for maintenance
(or type Control-D to continue):
This only happens when I boot using the new kernel 2.6.39-200.1.9. If I boot the old kernel, 2.6.39-200.1.4 the servers come up clean. I have had an SR open for about 10 days without resolution. I just want to know if anyone else is experiencing this odd issue. These servers are all Dell Poweredge R815's and they all boot off of local storage. Some things Oracle has had me try:
Remove kernel 2.6.39-200.1.9, then reinstall and reboot -- same error
Remove kernel 2.6.39-200.1.9 firmware, then reinstall and reboot -- same error
Remove both the kernel 2.6.39-200.1.9 and the firmware, then reinstall both and reboot -- same error
Any one have anything else I can try ? If so, please advise.
You can see the list of changes in the kernel update at the dom0 shell on any of the servers just by querying it like any other linux RPM as per
+[root@server01 ~]# rpm -q --changelog kernel-uek-2.6.39-200.1.9.el5uek | more+ * Tue Oct 09 2012 Joe Jin <firstname.lastname@example.org> [2.6.39-200.1.9.el5uek] - Fix bug number for commit "cciss: Update HPSA_BOUNDARY" (Joe Jin)
* Fri Sep 28 2012 Joe Jin <email@example.com> [2.6.39-200.1.8.el5uek]
If you look at the changes between 2.6.39-200.1.9 and 2.6.39-200.1.4 (which works for you) you can see scsi-related changes that might impact hardware specific to your R815s, and most of these have oracle bug IDs you can look up and comment on in MOS, and/or CVE IDs you can look up in cve.mitre.org so thats an route you can take for additional ideas on getting more info toward a resolution.
/dev/sda2 is mounted on (/) for Oracle VM server when its working right. Maybe you could try booting off a linux rescue CD or live CD for any recent distribution, running fsck over /dev/sda2 manually, checking if you can mount it manually after and that all the data and files in it look ok, and then see if your OS will boot?
Thank you for your reply. I did take a peek at the errata, but really didn't see much for this issue. However, about 10 minutes ago I was able to boot the server by blacklisting "sda" from multipath -- this was the latest suggestion from Oracle. The only change I had previously made to my multipath config was to add a Dell Powervault MD3620i other than that, it is default. So I am unsure why I am the only one experiencing this issue !
I will update this thread as soon as I hear back from Oracle.
Edit: Blacklisting "sda" in each of the servers multipath.conf did the trick.