I was the one who wrote the note to patch 144500-19 on support.oracle.com.
(We opened SR with Oracle support which resulted in suggestion that it's EMC issue... - and EMC support to be involved).
Here is our workaround:
0) Make backup of your /kernel/drv and /etc folders
1) Backup following files and move them somewhere else
/kernel/drv/emcp.conf (make sure there is no emcp.conf.saved )
/etc/powermt.custom (make sure there is no powermt.custom.saved )
emcp driver located within the following two directories:
Comment the emc-related stuff in /etc/system
3) Reboot in normal (not failsafe) mode - you should be able to boot now
4) Remove the new kernel pkgrm 144500-19
[this is supposed to bring you back to your pervious kernel. In our case this was 144488-17]
5) Reboot - make sure all runs fine with the new kernel, enable the emc drivers again (restoring the abovementioned config files and driver executables); reboot again
Hope that helps!
I hope your root file system is not on powerpath devices...?
We haven't changed any labels (from EFI to SMI or whatever else) and we've never had :h slices. I think this is a different issue (maybe a real minor one). The major issues are usually not listed in the readme :-)
I think the latest PowerPath version for Solaris is 220.127.116.11, isn't it? (Ours is 5.3.0 - I don't know whether 18.104.22.168 is affected).
Good question!!! My sources told me that PP 5.5 wasn't going to be ready until the "end of Q4". Now I don't know what EMC's fiscal year is; I was assuming CYQ4 but maybe EMC was meaning FYQ4? Does anyone know EMC's fiscal year cycle?
The problem is reproducible with PowerPath 5.3 P01 too.
Interestingly - PowerPath 5.3 P01 (as per it's release notes) add support for Solaris 10 Update 9.
* Release date of PowerPath 5.3 P01 is May 2011
* Solaris 10 Update 9 has been released in September 2010
I have no idea if they'll add support for Solaris 10 Update 10 (which is bundled with kernel 144500-19) quicker.
We also managed to setup an experiment with MPxIO instead of PowerPath - and it seems to work fine at first glimpse. (Though this was just an experiment - we just tested if the relevant file systems could be mounted. For production I still expect to get the problem with PowerPath fixed but it's good to know that there is some alternative).
In the meanwhile I got update from Oracle that they're working on the issue and they have some idea about the problem's root cause (so I hope it will be resolved soon).
Problem is that PowerPath and possible other multipath vendors are using a Solaris private interface which is just that, private. That's naughty. Private interfaces are subject to change and did indeed change in this case. A new element was added to the end of a structure to fix a race condition. That structure is referenced by a pointer. PowerPath was compiled with the old headers so is now experiencing an offset issue. I'm working directly with EMC to identify a quick(er) fix. I am also working with them to identify what private interfaces they are using, so that we can work together to wean them onto supported APIs. But first we're concentrating on helping them find a fix to the immediate issue.
Thanks for that last update. It's good to hear Oracle and storage vendors working together and having implementations based on clear, open and documented interfaces.
Sometimes people prefer to end up with quick "hacking" and "guessing" about how things work, ignoring any standards and recommendations.
To be fair I've been in situation when there is no public interface available (in Oracle Database RMAN) in order to implement my essential requirements. Oracle suggested me to use an undocumented functionality to solve my problem. I requested enhancement to be logged so that Oracle makes this functionality public and documented (or to provide some official alternative solution): Oracle refused to do so (since there is a solution, even though undocumented). When I shared my fear that probably that undocumented functionality may change without a notice in future versions - Oracle replied that "everything will eventually change".
So as I've mentioned - I'm happy to hear you're collaborating to solve the issue instead of just blaming each other.