1 2 Previous Next 21 Replies Latest reply on Oct 10, 2011 6:56 PM by javornikolov-JavaNet

    Patch 144500-19 and EMC PowerPath

      We're having issues with this patch on systems running PowerPath V5. System panics on reboot after patch install:

      <pre>panic[cpu0]/thread=2a100497ca0: BAD TRAP: type=31 rp=2a100496df0 addr=617564696f637338 mmu_fsr=0

      : trap type = 0x31
      pid=0, pc=0x13b6bc4, sp=0x2a100496691, tstate=0x80001603, context=0x0
      g1-g7: 19ae800, 0, 138, 1078614, 32, 10, 2a100497ca0

      000002a100496b10 unixie+9c (31, 2a100496df0, 617564696f637338, 0, 2a100496bd0, f85c2038)
      %l0-3: 0000000000001fff 0000000000000031 0000000001000000 0000000000002000
      %l4-7: 0000000000100000 00000000018b09c0 000002a100496ca4 00000000010a8400
      000002a100496bf0 unix:trap+9e0 (2a100496df0, 0, 1fff, 5, 617564696f636000, 1)
      %l0-3: 0000000000000000 00000000018b09c0 0000000000000031 0000000000001c00
      %l4-7: 0000000000000000 0000000000000001 ffffffffffffe000 0000000000000005
      000002a100496d40 unix:ktl0+48 (617564696f637300, 0, 0, 702305f8, 0, 0)
      %l0-3: 0000000000000004 0000000000001400 0000000080001603 000000000101cdcc
      %l4-7: 0000000001a15400 0000000000000000 0000000000000000 000002a100496df0
      000002a100496e90 emcpowerInitOpenCntHack+50 (20, 4c4b3f, 30002a0cd00, 300028c70d0, 18b3000, 1f4)
      %l0-3: 617564696f637300 0000000000000020 0000000000000500 0000030002a0c000
      %l4-7: 0000060058933b18 ffffffffffffffe2 0000000000000005 0000000000000001
      000002a100496fb0 emcpiocDaemon+1cc (0, 1a1a748, 3000a149cf8, 660500, 183c, 1)
      %l0-3: 0000000070230400 0000000000000005 00000000000010ec 000000000000fc00
      %l4-7: 000003000a149db8 0000000001a1a758 0000000000000000 000000000123adf0
      000002a100497080 emcpioc:_init+60 (0, 0, 0, 0, 1293000, 0)
      %l0-3: 000000007bf09450 0000000001a1a400 0000000000000000 00000000019a2b00
      %l4-7: 0000000000000001 0000000000000000 00000000019ab5b8 0000000000000000
      000002a100497140 genunix:modinstall+cc (0, 30009ff8b50, 0, 1950f08, 80000000, 7bf09e80)
      %l0-3: 0000000000000000 0000060058a43375 0000000000000000 00000000019a2800
      %l4-7: 0000000000000000 0000000020000000 0000000001950c00 0000060058a9dd93
      000002a100497200 genunix:mod_hold_installed_mod+9c (60058a43370, 0, 0, 2a10049736c, 1, 30009ff8b50)
      %l0-3: 0000000000000001 0000000000000003 0000000001957d40 0000000000000000
      %l4-7: 0000000000000000 0000000001957d40 0000000000000000 00000000019ab400
      000002a1004972b0 genunix:modrload+100 (19fba38, 19ab400, 0, 180c000, 16, 4)
      %l0-3: 000000000000000d 0000000000000000 0000060058a43370 00000000019a2800
      %l4-7: 000000000180c2f8 0000000000000034 0000000000000033 00000000fffd9c00
      000002a100497370 emcp:emcp_attach+ac8 (60052c62d90, 7, 0, 0, 0, 7fff)
      %l0-3: 0000000000000038 0000060058a9dd08 0000018014bb97d2 00000000019fb188
      %l4-7: 000000000000183c 0000000000000028 0000000000000000 0000000000001dda
      000002a1004974d0 genunixevi_attach+ac (60052c62d90, 0, 30002a14b20, 30002a0c000, 0, 13baf10)
      %l0-3: 00000300028c19f0 000000000000013e 00000000019446e8 0000000000008b20
      %l4-7: 00000300028c1000 00000000000009f0 0000000000000002 0000000000000001
      000002a1004975a0 genunix:attach_node+9c (60052c62d90, 1, 0, 0, 2, 60052c62df8)
      %l0-3: 00000300028c1000 0000000000000010 0000000000000002 00000000fffeffff
      %l4-7: 00000000fffefc00 0000000040010000 0000000000000000 0000000040010000
      000002a100497650 genunix:i_ndi_config_node+110 (60052c62d90, 11c, 10eef10, 10, 1944800, 10eec00)
      %l0-3: 0000000000000000 0000000000000006 00000000019449b0 0000000000000000
      %l4-7: 0000000001944800 0000000000000174 0000000000000174 0000000000000004
      000002a100497700 genunix:i_ddi_attachchild+38 (60052c62d90, 2a100497ca0, 4040, 19575e0, 0, 0)
      %l0-3: 0000030006ee6dc0 0000000000000001 0000030006ee6dc0 0000000000000000
      %l4-7: 0000000000000001 0000000001919000 000000000180b1b0 0000000000000000
      000002a1004977b0 genunixevi_attach_node+84 (60052c62d90, 4040, 0, 60052c62df8, 4040, 0)
      %l0-3: 0000030006ee6dc0 0000000000000000 00000000000003ff 0000000000000002
      %l4-7: 0000000000000001 0000000001919000 00000000800003cf 0000000000000004
      000002a100497860 genunix:config_immediate_children+d0 (30006ee6d58, 60052c62d90, 13e, 0, 4, 19beba0)
      %l0-3: 0000000000000000 0000000000004040 0000000000010000 0000030006ee6dc0
      %l4-7: 00000000b7fea9ac 000000000000002f 0000060052c62b60 ffffffffffffffff
      000002a100497920 genunixevi_config_common+c0 (30006ee6d58, 4, 13e, 13e, 0, 4040)
      %l0-3: ffffffffffffffff 0000000000000000 00000000b7fea9db 000000000000000c
      %l4-7: 00000000019beba0 00000000019bed18 000000000000000a 0000000000000000
      000002a1004979d0 genunix:mt_config_thread+60 (60052fcb020, 0, 18b09c0, 18b09c0, 60052ee5f40, 30006ee6d58)
      %l0-3: 0000000000000000 0000000000000001 0000018014bb97d2 000000000180e000
      %l4-7: 000000000000183c 00000000019a0800 000000000180c178 0000000000001dda</pre>

      Anybody else seeing this?
        • 1. Re: Patch 144500-19 and EMC PowerPath
          On support.oracle.com some man have same probleme. In discussion for this patche - recommend rallback to 144488-17.

          1 person found this helpful
          • 2. Re: Patch 144500-19 and EMC PowerPath

            I was the one who wrote the note to patch 144500-19 on support.oracle.com.
            (We opened SR with Oracle support which resulted in suggestion that it's EMC issue... - and EMC support to be involved).

            Here is our workaround:
            0) Make backup of your /kernel/drv and /etc folders
            1) Backup following files and move them somewhere else

            /kernel/drv/emcp.conf (make sure there is no emcp.conf.saved )
            /etc/powermt.custom (make sure there is no powermt.custom.saved )


            emcp driver located within the following two directories:

            Comment the emc-related stuff in /etc/system

            3) Reboot in normal (not failsafe) mode - you should be able to boot now
            4) Remove the new kernel pkgrm 144500-19
            [this is supposed to bring you back to your pervious kernel. In our case this was 144488-17]
            5) Reboot - make sure all runs fine with the new kernel, enable the emc drivers again (restoring the abovementioned config files and driver executables); reboot again

            Hope that helps!
            I hope your root file system is not on powerpath devices...?

            1 person found this helpful
            • 3. Re: Patch 144500-19 and EMC PowerPath
              So much for the Big-O owning up to the issue... Thanks for the suggestion. I guess we'll open a case with EMC.

              Our root FS is mirrored with SVM. We split the mirror before patching so we were able to recover fairly quickly.

              Me thinks the Big-O should recall this patch...
              • 4. Re: Patch 144500-19 and EMC PowerPath
                Found this in the patch readme:

                <pre>NOTE 9: This patch causes a minor issue with systems using EMCPowerPath,
                as explained below:

                <pre>On changing the label of a EMC PowerPath pseudo disk from EFI to SMI,
                the pseudo disk with a new SMI label may not have the :h device node

                <pre>For example, an affected SMI labeled EMC PowerPath pseudo disk
                contains only :a to :g device nodes and not :h device node, as
                shown below:

                <pre>#ls -l /dev/rdsk/emcpower4*
                lrwxrwxrwx 1 root other 33 Nov 13 21:43 /dev/rdsk/emcpower4a -> ../../devices/pseudo/emcp@4:a,raw
                lrwxrwxrwx 1 root other 33 Nov 13 21:43 /dev/rdsk/emcpower4b -> ../../devices/pseudo/emcp@4:b,raw
                lrwxrwxrwx 1 root other 33 Nov 13 21:43 /dev/rdsk/emcpower4c -> ../../devices/pseudo/emcp@4:c,raw
                lrwxrwxrwx 1 root other 33 Nov 13 21:43 /dev/rdsk/emcpower4d -> ../../devices/pseudo/emcp@4:d,raw
                lrwxrwxrwx 1 root other 33 Nov 13 21:43 /dev/rdsk/emcpower4e -> ../../devices/pseudo/emcp@4:e,raw
                lrwxrwxrwx 1 root other 33 Nov 13 21:43 /dev/rdsk/emcpower4f -> ../../devices/pseudo/emcp@4:f,raw
                lrwxrwxrwx 1 root other 33 Nov 13 21:43 /dev/rdsk/emcpower4g -> ../../devices/pseudo/emcp@4:g,raw

                <pre>To resolve this issue, please update EMC PowerPath to version 5.5
                or later release.</pre>

                "Minor" issue??? Geez...

                Edited by: bobthesungeek76036 on Aug 10, 2011 8:22 AM

                • 5. Re: Patch 144500-19 and EMC PowerPath
                  We haven't changed any labels (from EFI to SMI or whatever else) and we've never had :h slices. I think this is a different issue (maybe a real minor one). The major issues are usually not listed in the readme :-)

                  I think the latest PowerPath version for Solaris is, isn't it? (Ours is 5.3.0 - I don't know whether is affected).
                  • 6. Re: Patch 144500-19 and EMC PowerPath
                    Just received an update from EMC.


                    EMC was not aware that Oracle was releasing this patch and there is no fix until PP 5.5 is released. (which will be at the end of the year when S10u10 is released)
                    • 7. Re: Patch 144500-19 and EMC PowerPath
                      S10u10 has already been released today: http://www.oracle.com/technetwork/server-storage/solaris/downloads/index.html
                      So does it mean EMC PowerPath 5.5 is ready?!
                      • 8. Re: Patch 144500-19 and EMC PowerPath
                        Good question!!! My sources told me that PP 5.5 wasn't going to be ready until the "end of Q4". Now I don't know what EMC's fiscal year is; I was assuming CYQ4 but maybe EMC was meaning FYQ4? Does anyone know EMC's fiscal year cycle?

                        (CY = Calendar Year, FY = Fiscal Year)
                        • 9. Re: Patch 144500-19 and EMC PowerPath
                          The problem is reproducible with PowerPath 5.3 P01 too.

                          Interestingly - PowerPath 5.3 P01 (as per it's release notes) add support for Solaris 10 Update 9.
                          * Release date of PowerPath 5.3 P01 is May 2011
                          * Solaris 10 Update 9 has been released in September 2010

                          I have no idea if they'll add support for Solaris 10 Update 10 (which is bundled with kernel 144500-19) quicker.
                          1 person found this helpful
                          • 10. Re: Patch 144500-19 and EMC PowerPath
                            This makes me think about a theoretical (yet) question: would it all work fine if EMC PowerPath is removed and Solaris MPXIO is used instead.

                            • 11. Re: Patch 144500-19 and EMC PowerPath
                              Probably so. Unfortunately, I don't have that luxury.

                              BTW - If you go to the Solaris 10 download page it looks like they recalled Update 10. Download is showing Update 9 again...
                              • 12. Re: Patch 144500-19 and EMC PowerPath
                                Yes, I saw that S10U10 has disappeared...

                                We also managed to setup an experiment with MPxIO instead of PowerPath - and it seems to work fine at first glimpse. (Though this was just an experiment - we just tested if the relevant file systems could be mounted. For production I still expect to get the problem with PowerPath fixed but it's good to know that there is some alternative).

                                In the meanwhile I got update from Oracle that they're working on the issue and they have some idea about the problem's root cause (so I hope it will be resolved soon).
                                • 13. Re: Patch 144500-19 and EMC PowerPath
                                  Gerry Haskins-Oracle
                                  Problem is that PowerPath and possible other multipath vendors are using a Solaris private interface which is just that, private. That's naughty. Private interfaces are subject to change and did indeed change in this case. A new element was added to the end of a structure to fix a race condition. That structure is referenced by a pointer. PowerPath was compiled with the old headers so is now experiencing an offset issue. I'm working directly with EMC to identify a quick(er) fix. I am also working with them to identify what private interfaces they are using, so that we can work together to wean them onto supported APIs. But first we're concentrating on helping them find a fix to the immediate issue.
                                  • 14. Re: Patch 144500-19 and EMC PowerPath
                                    Thanks for that last update. It's good to hear Oracle and storage vendors working together and having implementations based on clear, open and documented interfaces.

                                    Sometimes people prefer to end up with quick "hacking" and "guessing" about how things work, ignoring any standards and recommendations.

                                    To be fair I've been in situation when there is no public interface available (in Oracle Database RMAN) in order to implement my essential requirements. Oracle suggested me to use an undocumented functionality to solve my problem. I requested enhancement to be logged so that Oracle makes this functionality public and documented (or to provide some official alternative solution): Oracle refused to do so (since there is a solution, even though undocumented). When I shared my fear that probably that undocumented functionality may change without a notice in future versions - Oracle replied that "everything will eventually change".

                                    So as I've mentioned - I'm happy to hear you're collaborating to solve the issue instead of just blaming each other.
                                    1 2 Previous Next