6 Replies Latest reply on May 2, 2019 6:39 AM by Kazuhiro

    Exadata Storage questions

    Arturo Gutierrez

      Hello,

      I've some questions regarding storage objects:

      1. Is about this output:

      CellCLI>  list physicaldisk

               252:0           RB8ETV                  normal

               252:1           RBBSNV                  normal

               252:2           RBG8AV                  normal

               252:3           RBJ5UV                  normal

               252:4           RBD0LV                  normal

               252:5           RBGGZV                  normal

               252:6           RB6Y2V                  normal

               252:7           RBJ79V                  normal

               252:8           RBB4GV                  normal

               252:9           RB90JV                  normal

               252:10          RBAAGV                  normal

               252:11          RBDDDV                  normal

               FLASH_10_1      PHLE816600ZE6P4BGN-1    normal

               FLASH_10_2      PHLE816600ZE6P4BGN-2    normal

               FLASH_4_1       PHLE816601BS6P4BGN-1    normal

               FLASH_4_2       PHLE816601BS6P4BGN-2    normal

               FLASH_5_1       PHLE816601MW6P4BGN-1    normal

               FLASH_5_2       PHLE816601MW6P4BGN-2    normal

               FLASH_6_1       PHLE816600ZD6P4BGN-1    normal

               FLASH_6_2       PHLE816600ZD6P4BGN-2    normal

               M2_SYS_0        PHDW816601Q4150A        normal

               M2_SYS_1        PHDW816601T0150A        normal

       

      Why for  every Flash Card create two phy disks?

       

       

      2. list celldisk detail

               name:                   CD_00_ro09celadm01

               comment:

               creationTime:           2018-11-05T15:15:18+01:00

               deviceName:             /dev/sdc

               devicePartition:        /dev/sdc

               diskType:               HardDisk

               errorCount:             0

               freeSpace:              192M

               id:                     14b48577-3eb7-43f9-b048-420bf9c17e65

               physicalDisk:           RB8ETV

               size:                   8.9094085693359375T

               status:                 normal

      Why every celldisk has 192 MB of free space ? Has any special use?

      Is not too much of course.

       

      Thanks

      Arturo

        • 1. Re: Exadata Storage questions
          Nishant Baurai

          Arturo,

           

          What is exadata version X6/X7 ?

          What is rack size ?

           

          Regards,

          Nishant

          • 2. Re: Exadata Storage questions
            Arturo Gutierrez

            Hi Nishant,

            This is a X7-2 QR

             

            Thanks

            Arturo

            • 3. Re: Exadata Storage questions
              Kazuhiro

              Hello Arturo,

               

              >Why for  every Flash Card create two phy disks?

               

              I had the same question when I first looked at our X7-2 storage server as follows:

              I'm not quite sure but it seems to me that there are two controllers for each flash card.

              That is probably why cell servers can identify as two separate disk (2.9TB) / card.

               

              [root@xxx ~]# cellcli -e list physicaldisk where disktype=flashdisk

                       FLASH_10_1      PHLE742200W96P4BGN-1    normal

                       FLASH_10_2      PHLE742200W96P4BGN-2    normal

                       FLASH_4_1       PHLE7422017B6P4BGN-1    normal

                       FLASH_4_2       PHLE7422017B6P4BGN-2    normal

                       FLASH_5_1       PHLE742200KW6P4BGN-1    normal

                       FLASH_5_2       PHLE742200KW6P4BGN-2    normal

                       FLASH_6_1       PHLE730300FQ6P4BGN-1    normal

                       FLASH_6_2       PHLE730300FQ6P4BGN-2    normal

               

               

              [root@xxx ~]# cellcli -e list physicaldisk where disktype=flashdisk detail

                       name:                   FLASH_10_1

                       deviceName:             /dev/nvme2n1

                       diskType:               FlashDisk

                       luns:                   10_0

                       makeModel:              "Oracle Flash Accelerator F640 PCIe Card"

                       physicalFirmware:       QDV1RD24

                       physicalInsertTime:     2018-04-12T10:59:21+09:00

                       physicalSerial:         PHLE742200W96P4BGN-1

                       physicalSize:           2.910957656800746917724609375T

                       slotNumber:             "PCI Slot: 10; FDOM: 1"

                       status:                 normal

               

                       name:                   FLASH_10_2

                       deviceName:             /dev/nvme3n1

                       diskType:               FlashDisk

                       luns:                   10_0

                       makeModel:              "Oracle Flash Accelerator F640 PCIe Card"

                       physicalFirmware:       QDV1RD24

                       physicalInsertTime:     2018-04-12T10:59:21+09:00

                       physicalSerial:         PHLE742200W96P4BGN-2

                       physicalSize:           2.910957656800746917724609375T

                       slotNumber:             "PCI Slot: 10; FDOM: 2"

                       status:                 normal

               

               

              https://docs.oracle.com/cd/E87231_01/pdf/E87233.pdf

              p.14

               

              Best regards,

              Kazuhiro

              1 person found this helpful
              • 4. Re: Exadata Storage questions
                robinsc

                I understand that every flash card has multiple independant modules that can be replaced independantly .For the X2-2 there were 4 modules per card so with 4 flash cards we used to have 16 devices. What this means is that if one flash module errors out only that module needs to be replaced rather than all the flash on the card.

                flash1.jpg

                flash2.jpg

                1 person found this helpful
                • 5. Re: Exadata Storage questions
                  Kazuhiro

                  Hello robinsc,

                   

                  >What this means is that if one flash module errors out only that module needs to be replaced rather than all the flash on the card.

                   

                  I just wonder if a CE can replace only a damaged module in a flash card in a data center,

                  because I always thought that they replace a whole flash card even if a part of the modules is damaged.

                  Technically, it will be possible to identify which module (FDOM) is damaged by sundiag, but I'm not sure

                  what they are doing exactly in data centers... at least I've never seen such a case.

                   

                  Best regards,

                  Kazuhiro

                  1 person found this helpful
                  • 6. Re: Exadata Storage questions
                    Kazuhiro

                    Hello,

                     

                    BTW, concerning your second question,

                     

                    >Why every celldisk has 192 MB of free space ? Has any special use?

                     

                    I think it is because the total size of griddisks is 192MB smaller than the size of celldisk.

                     

                    In my environment the freeSpace of celldisk CD_00_xxx01 is 0,

                    because the total size of griddisks (DATA_CD_00_xxx01 and RECO_CD_00_xxx01) in the celldisk

                    is almost exactly as the same as the size of the celldisk.

                     

                    DATA_CD_00_xxx01    6.681640625T

                    RECO_CD_00_xxx01    2.22772216796875T

                    -> sum 8.90936279296875 T

                        =~ 8.9094085693359375T

                     

                    So your original question, Why 192MB free space?

                    I think you have to ask someone who designed or created the

                    diskgroups and grid disks.

                     

                    CellCLI> list celldisk detail

                             name:                   CD_00_xxx01

                             comment:

                             creationTime:           2018-04-16T16:39:30+09:00

                             deviceName:             /dev/sda

                             devicePartition:        /dev/sda

                             diskType:               HardDisk

                             errorCount:             0

                             freeSpace:              0

                             id:                     6dcf8810-77d9-48ac-8339-b7eba6774905

                             physicalDisk:           R464WK

                             size:                   8.9094085693359375T

                             status:                 normal

                            

                            

                    CellCLI> list griddisk detail

                             name:                   DATA_CD_00_xxx01

                             asmDiskGroupName:       DATA

                             asmDiskName:            DATA_CD_00_XXX01

                             asmFailGroupName:       XXX01

                             availableTo:

                             cachedBy:               FD_00_xxx01

                             cachingPolicy:          default

                             cellDisk:               CD_00_xxx01

                             comment:                "Cluster dm01-cluster diskgroup DATA"

                             creationTime:           2018-04-16T16:48:00+09:00

                             diskType:               HardDisk

                             errorCount:             0

                             id:                     3157c312-f66c-4714-8440-334240b781a8

                             size:                   6.681640625T

                             status:                 active

                    ...

                             name:                   RECO_CD_00_xxx01

                             asmDiskGroupName:       RECO

                             asmDiskName:            RECO_CD_00_XXX01

                             asmFailGroupName:       XXX01

                             availableTo:

                             cachedBy:

                             cachingPolicy:          none

                             cellDisk:               CD_00_xxx01

                             comment:                "Cluster dm01-cluster diskgroup RECO"

                             creationTime:           2018-04-16T16:48:54+09:00

                             diskType:               HardDisk

                             errorCount:             0

                             id:                     b5b9de6a-703f-4dad-9282-5e7511e8bc78

                             size:                   2.22772216796875T

                             status:                 active

                     

                    Best regards,

                    Kazuhiro

                    1 person found this helpful