10 Replies Latest reply: Sep 15, 2011 6:30 PM by andy.colvin RSS

    Exadata - Storage Cells

    Dbacloud
      We had two flash disks go offline on a storage server!
      We restarted the storage cell on Friday. The physical disks never came back online. They are still showing as syncing or offline. ASM has dropped the disks.

      2 Of the DB's are not even coming up. Please advise.
        • 1. Re: Exadata - Storage Cells
          Dan.Norris-Oracle
          Open a SR with support, SEV 1 if this is a production system (apparently it isn't if the DBs have been down since Friday). There's too much required to troubleshoot a critical issue to do it via a forum.
          • 2. Re: Exadata - Storage Cells
            Dbacloud
            Already in the process of that. Thanks
            • 3. Re: Exadata - Storage Cells
              andy.colvin
              Are you using those flash disks in an ASM diskgroup or as flashcache? If they're set up as flashcache, it shouldn't take a database offline. If you're not on 11.2.2.3.5 (most likely you aren't), you may want to take a look at the metalink note "IDT switch on the PCI riser has a problem resulting in occasional loss of connectivity to pair of flash cards on the cells [ID 1351559.1]"

              There has been a problem with the PCI riser shutting off flash cards. There is a workaround available in this note.
              • 4. Re: Exadata - Storage Cells
                471852
                Hi,

                We had similar problem on UAT and PROD. As IT is impacting prod we have raised an SR with oracle and we will have an engineer this weekend to replace them.

                Regards,

                David Alejo-Marcos.
                • 5. Re: Exadata - Storage Cells
                  Vishal Gupta
                  David,

                  You may want to apply the fix mentioned in 1351559.1 until your Exadata storage server software is upgrade from 11.2.2.2.0 to 11.2.2.3.5. It will avoid the problem of two of flash cards on the same PCIe riser (ie. either 1 and 4 or 2 and 5) going offline at the same time on cell nodes.

                  Regards,
                  Vishal Gupta
                  • 6. Re: Exadata - Storage Cells
                    Vishal Gupta
                    David,

                    You may want to apply the fix mentioned in 1351559.1 until your Exadata storage server software is upgrade from 11.2.2.2.0 to 11.2.2.3.5. It will avoid the problem of two of flash cards on the same PCIe riser (ie. either 1 and 4 or 2 and 5) going offline at the same time on cell nodes.

                    Regards,
                    Vishal Gupta
                    • 7. Re: Exadata - Storage Cells
                      Vishal Gupta
                      David,

                      You may want to apply the fix mentioned in 1351559.1 until your Exadata storage server software is upgrade from 11.2.2.2.0 to 11.2.2.3.5. It will avoid the problem of two of flash cards on the same PCIe riser (ie. either 1 and 4 or 2 and 5) going offline at the same time on cell nodes.

                      Regards,
                      Vishal Gupta
                      • 8. Re: Exadata - Storage Cells
                        andy.colvin
                        Vishal Gupta wrote:
                        David,

                        You may want to apply the fix mentioned in 1351559.1 until your Exadata storage server software is upgrade from 11.2.2.2.0 to 11.2.2.3.5. It will avoid the problem of two of flash cards on the same PCIe riser (ie. either 1 and 4 or 2 and 5) going offline at the same time on cell nodes.

                        Regards,
                        Vishal Gupta
                        Vishal,

                        I took the contents of the note to mean that we still need the fix, even with 11.2.2.3.5. From the note:
                        >
                        Apply this solution to systems running Exadata Storage Server software 11.2.2.3.5 and earlier.
                        Systems patched to 11.2.2.3.5 that incorporated the 12832832_12886507_12413272.tbz workaround specified in Note 1334254.1 already have this in place. To verify, run the "service fixpciidt_12886507 status" command as described below.
                        This solution will be incorporated into the next Exadata Storage Server software release.
                        • 9. Re: Exadata - Storage Cells
                          andy.colvin
                          Vishal Gupta wrote:
                          David,

                          You may want to apply the fix mentioned in 1351559.1 until your Exadata storage server software is upgrade from 11.2.2.2.0 to 11.2.2.3.5. It will avoid the problem of two of flash cards on the same PCIe riser (ie. either 1 and 4 or 2 and 5) going offline at the same time on cell nodes.

                          Regards,
                          Vishal Gupta
                          Vishal,

                          I took the contents of the note to mean that we still need the fix, even with 11.2.2.3.5. From the note:
                          >
                          Apply this solution to systems running Exadata Storage Server software 11.2.2.3.5 and earlier.
                          Systems patched to 11.2.2.3.5 that incorporated the 12832832_12886507_12413272.tbz workaround specified in Note 1334254.1 already have this in place. To verify, run the "service fixpciidt_12886507 status" command as described below.
                          This solution will be incorporated into the next Exadata Storage Server software release.
                          • 10. Re: Exadata - Storage Cells
                            andy.colvin
                            Vishal Gupta wrote:
                            David,

                            You may want to apply the fix mentioned in 1351559.1 until your Exadata storage server software is upgrade from 11.2.2.2.0 to 11.2.2.3.5. It will avoid the problem of two of flash cards on the same PCIe riser (ie. either 1 and 4 or 2 and 5) going offline at the same time on cell nodes.

                            Regards,
                            Vishal Gupta
                            Vishal,

                            I took the contents of the note to mean that we still need the fix, even with 11.2.2.3.5. From the note:
                            >
                            Apply this solution to systems running Exadata Storage Server software 11.2.2.3.5 and earlier.
                            Systems patched to 11.2.2.3.5 that incorporated the 12832832_12886507_12413272.tbz workaround specified in Note 1334254.1 already have this in place. To verify, run the "service fixpciidt_12886507 status" command as described below.
                            This solution will be incorporated into the next Exadata Storage Server software release.