I was the one who wrote the note to patch 144500-19 on support.oracle.com.
(We opened SR with Oracle support which resulted in suggestion that it's EMC issue... - and EMC support to be involved).
Here is our workaround:
0) Make backup of your /kernel/drv and /etc folders
1) Backup following files and move them somewhere else
/kernel/drv/emcp.conf (make sure there is no emcp.conf.saved )
/etc/powermt.custom (make sure there is no powermt.custom.saved )
emcp driver located within the following two directories:
Comment the emc-related stuff in /etc/system
3) Reboot in normal (not failsafe) mode - you should be able to boot now
4) Remove the new kernel pkgrm 144500-19
[this is supposed to bring you back to your pervious kernel. In our case this was 144488-17]
5) Reboot - make sure all runs fine with the new kernel, enable the emc drivers again (restoring the abovementioned config files and driver executables); reboot again
Hope that helps!
I hope your root file system is not on powerpath devices...?
So much for the Big-O owning up to the issue... Thanks for the suggestion. I guess we'll open a case with EMC.
Our root FS is mirrored with SVM. We split the mirror before patching so we were able to recover fairly quickly.
Me thinks the Big-O should recall this patch...
Found this in the patch readme:
<pre>NOTE 9: This patch causes a minor issue with systems using EMCPowerPath,
as explained below:
<pre>On changing the label of a EMC PowerPath pseudo disk from EFI to SMI,
the pseudo disk with a new SMI label may not have the :h device node
<pre>For example, an affected SMI labeled EMC PowerPath pseudo disk
contains only :a to :g device nodes and not :h device node, as
<pre>#ls -l /dev/rdsk/emcpower4*
lrwxrwxrwx 1 root other 33 Nov 13 21:43 /dev/rdsk/emcpower4a -> ../../devices/pseudo/emcp@4:a,raw
lrwxrwxrwx 1 root other 33 Nov 13 21:43 /dev/rdsk/emcpower4b -> ../../devices/pseudo/emcp@4:b,raw
lrwxrwxrwx 1 root other 33 Nov 13 21:43 /dev/rdsk/emcpower4c -> ../../devices/pseudo/emcp@4:c,raw
lrwxrwxrwx 1 root other 33 Nov 13 21:43 /dev/rdsk/emcpower4d -> ../../devices/pseudo/emcp@4:d,raw
lrwxrwxrwx 1 root other 33 Nov 13 21:43 /dev/rdsk/emcpower4e -> ../../devices/pseudo/emcp@4:e,raw
lrwxrwxrwx 1 root other 33 Nov 13 21:43 /dev/rdsk/emcpower4f -> ../../devices/pseudo/emcp@4:f,raw
lrwxrwxrwx 1 root other 33 Nov 13 21:43 /dev/rdsk/emcpower4g -> ../../devices/pseudo/emcp@4:g,raw
<pre>To resolve this issue, please update EMC PowerPath to version 5.5
or later release.</pre>
"Minor" issue??? Geez...
Edited by: bobthesungeek76036 on Aug 10, 2011 8:22 AM
We haven't changed any labels (from EFI to SMI or whatever else) and we've never had :h slices. I think this is a different issue (maybe a real minor one). The major issues are usually not listed in the readme :-)
I think the latest PowerPath version for Solaris is 126.96.36.199, isn't it? (Ours is 5.3.0 - I don't know whether 188.8.131.52 is affected).
Just received an update from EMC.
DO NOT INSTALL THIS PATCH IF YOU HAVE POWERPATH INSTALLED
EMC was not aware that Oracle was releasing this patch and there is no fix until PP 5.5 is released. (which will be at the end of the year when S10u10 is released)
Good question!!! My sources told me that PP 5.5 wasn't going to be ready until the "end of Q4". Now I don't know what EMC's fiscal year is; I was assuming CYQ4 but maybe EMC was meaning FYQ4? Does anyone know EMC's fiscal year cycle?
(CY = Calendar Year, FY = Fiscal Year)
The problem is reproducible with PowerPath 5.3 P01 too.
Interestingly - PowerPath 5.3 P01 (as per it's release notes) add support for Solaris 10 Update 9.
* Release date of PowerPath 5.3 P01 is May 2011
* Solaris 10 Update 9 has been released in September 2010
I have no idea if they'll add support for Solaris 10 Update 10 (which is bundled with kernel 144500-19) quicker.
This makes me think about a theoretical (yet) question: would it all work fine if EMC PowerPath is removed and Solaris MPXIO is used instead.
Yes, I saw that S10U10 has disappeared...
We also managed to setup an experiment with MPxIO instead of PowerPath - and it seems to work fine at first glimpse. (Though this was just an experiment - we just tested if the relevant file systems could be mounted. For production I still expect to get the problem with PowerPath fixed but it's good to know that there is some alternative).
In the meanwhile I got update from Oracle that they're working on the issue and they have some idea about the problem's root cause (so I hope it will be resolved soon).
Problem is that PowerPath and possible other multipath vendors are using a Solaris private interface which is just that, private. That's naughty. Private interfaces are subject to change and did indeed change in this case. A new element was added to the end of a structure to fix a race condition. That structure is referenced by a pointer. PowerPath was compiled with the old headers so is now experiencing an offset issue. I'm working directly with EMC to identify a quick(er) fix. I am also working with them to identify what private interfaces they are using, so that we can work together to wean them onto supported APIs. But first we're concentrating on helping them find a fix to the immediate issue.
Thanks for that last update. It's good to hear Oracle and storage vendors working together and having implementations based on clear, open and documented interfaces.
Sometimes people prefer to end up with quick "hacking" and "guessing" about how things work, ignoring any standards and recommendations.
To be fair I've been in situation when there is no public interface available (in Oracle Database RMAN) in order to implement my essential requirements. Oracle suggested me to use an undocumented functionality to solve my problem. I requested enhancement to be logged so that Oracle makes this functionality public and documented (or to provide some official alternative solution): Oracle refused to do so (since there is a solution, even though undocumented). When I shared my fear that probably that undocumented functionality may change without a notice in future versions - Oracle replied that "everything will eventually change".
So as I've mentioned - I'm happy to hear you're collaborating to solve the issue instead of just blaming each other.