8 Replies Latest reply: May 9, 2013 10:53 AM by Omega3 RSS

    Questions from Linux System Admin on storage

    james_p
      Version Info
      +++++++++++
      Grid Infrstructure: 11.2.0.3
      Platform : Oracle Enterprise Linux 6.3
      ++++++++++++

      We are a big AIX, Solaris shop ! After some meetings with the Oracle Sales guys, our managers have decided to move some of our RAC DBs to Oracle Linux.

      Our Linux Admins (RHEL ) never configured machines for RAC before. They have only configured machines for App Servers before.

      For the shared LUNs meant for ASM Diskgroups they have asked me how I want to have the LUNs to be ? Physical Volume, Logical Volume .. etc. They spoke about things like pvcreate, vgcreate.. etc.
      Any idea how should the storage LUNs have to be presented to Linux RAC Nodes ?
        • 1. Re: Questions from Linux System Admin on storage
          user11977218
          Each node in a cluster requires external shared disks for storing the Oracle Clusterware (Oracle Cluster
          Registry and voting disk) files, and Oracle Database files.
          Once the LUNs have been presented from the SAN to ALL servers in the cluster, partition the LUNs from
          one node only, run fdisk to create a single whole-disk partition with exactly 1 MB offset on each LUN to be
          used as ASM Disk.
          • 2. Re: Questions from Linux System Admin on storage
            EdStevens
            james_p wrote:
            Version Info
            +++++++++++
            Grid Infrstructure: 11.2.0.3
            Platform : Oracle Enterprise Linux 6.3
            ++++++++++++

            We are a big AIX, Solaris shop ! After some meetings with the Oracle Sales guys, our managers have decided to move some of our RAC DBs to Oracle Linux.

            Our Linux Admins (RHEL ) never configured machines for RAC before. They have only configured machines for App Servers before.
            But you have AIX and/or Solaris admins who have configured for RAC?
            Why can't your AIX/Solaris guys help your Linux guys? They already speak the same language, the principles are the same ... I know some purists will say that Linux is not Unix, but from my perspective the difference between Linux and any particular Unix is no greater than the differences between the various particular Unix's. As a dba, I've worked with Solaris, AIX, HP-UX, and Oracle Linux and have seen some of the differences first hand, but they are so trivial that I can't even remember what they are.
            For the shared LUNs meant for ASM Diskgroups they have asked me how I want to have the LUNs to be ? Physical Volume, Logical Volume .. etc. They spoke about things like pvcreate, vgcreate.. etc.
            Any idea how should the storage LUNs have to be presented to Linux RAC Nodes ?
            • 3. Re: Questions from Linux System Admin on storage
              rarain
              Hi James,

              pvcreate Initilizes physical volume for later use by LVM and vgcreate creates a new volume group using the block special device previously configured for LVM with pvcreate. So if you are going to implement LVM then only you need to use pvcreate and vgcreate but since you are implementing ASM so LVM is not required because ASM itself offers LVM features where you can add the disk dynamically if your diskgroup is full and rebalancing is also performed automatically by ASM.

              You just have to Map configured LUN on RAC server which will be represented as block device let's say /dev/sdb and create logical partition under /dev/sdb let's say /dev/sdb1 and further use these devices to create the ASM diskgroup.

              Thanks
              • 4. Re: Questions from Linux System Admin on storage
                Levi Pereira
                james_p wrote:
                Version Info
                +++++++++++
                Grid Infrstructure: 11.2.0.3
                Platform : Oracle Enterprise Linux 6.3
                ++++++++++++

                We are a big AIX, Solaris shop ! After some meetings with the Oracle Sales guys, our managers have decided to move some of our RAC DBs to Oracle Linux.

                Our Linux Admins (RHEL ) never configured machines for RAC before. They have only configured machines for App Servers before.

                For the shared LUNs meant for ASM Diskgroups they have asked me how I want to have the LUNs to be ? Physical Volume, Logical Volume .. etc. They spoke about things like pvcreate, vgcreate.. etc.
                Any idea how should the storage LUNs have to be presented to Linux RAC Nodes ?
                Short answer is : Recommended present to ASM the Physical Disks (Luns)

                Install Docs is pretty clear about theses questions.
                http://docs.oracle.com/cd/E11882_01/install.112/e22489/storage.htm#CDECEBGH


                * All of the devices in an Oracle ASM disk group should be the same size and have the same performance characteristics.

                * Do not specify multiple partitions on a single physical disk as a disk group device. Each disk group device should be on a separate physical disk.

                * Although you can specify a logical volume as a device in an Oracle ASM disk group, Oracle does not recommend their use because it adds a layer of complexity that is unnecessary with Oracle ASM. In addition, Oracle RAC requires a cluster logical volume manager in case you decide to use a logical volume with Oracle ASM and Oracle RAC.

                Oracle recommends that if you choose to use a logical volume manager, then use the logical volume manager to represent a single LUN without striping or mirroring, so that you can minimize the impact of the additional storage layer.

                Using Logical Volume Managers with Oracle Grid Infrastructure and Oracle RAC
                http://docs.oracle.com/cd/E11882_01/install.112/e22489/storage.htm#CHDGEIIB
                • 5. Re: Questions from Linux System Admin on storage
                  james_p
                  Thank you. I have 2 more questions


                  Question1.
                  Should it be extended partition or Primary partition ? Should I choose 'e' or 'p' for below ?
                  # fdisk /dev/sdb
                  .
                  .
                  .
                  .
                  Command action
                     e   extended
                     p   primary partition (1-4)
                  Question2.
                  What should be the partition type ? What should I choose from the list below
                  Command (m for help): t
                  Selected partition 1
                  Hex code (type L to list codes): l
                  
                   0  Empty           1e  Hidden W95 FAT1 80  Old Minix       bf  Solaris        
                   1  FAT12           24  NEC DOS         81  Minix / old Lin c1  DRDOS/sec (FAT-
                   2  XENIX root      39  Plan 9          82  Linux swap / So c4  DRDOS/sec (FAT-
                   3  XENIX usr       3c  PartitionMagic  83  Linux           c6  DRDOS/sec (FAT-
                   4  FAT16 <32M      40  Venix 80286     84  OS/2 hidden C:  c7  Syrinx        
                   5  Extended        41  PPC PReP Boot   85  Linux extended  da  Non-FS data    
                   6  FAT16           42  SFS             86  NTFS volume set db  CP/M / CTOS / .
                   7  HPFS/NTFS       4d  QNX4.x          87  NTFS volume set de  Dell Utility  
                   8  AIX             4e  QNX4.x 2nd part 88  Linux plaintext df  BootIt        
                   9  AIX bootable    4f  QNX4.x 3rd part 8e  Linux LVM       e1  DOS access    
                   a  OS/2 Boot Manag 50  OnTrack DM      93  Amoeba          e3  DOS R/O        
                   b  W95 FAT32       51  OnTrack DM6 Aux 94  Amoeba BBT      e4  SpeedStor      
                   c  W95 FAT32 (LBA) 52  CP/M            9f  BSD/OS          eb  BeOS fs        
                   e  W95 FAT16 (LBA) 53  OnTrack DM6 Aux a0  IBM Thinkpad hi ee  EFI GPT        
                   f  W95 Ext'd (LBA) 54  OnTrackDM6      a5  FreeBSD         ef  EFI (FAT-12/16/
                  10  OPUS            55  EZ-Drive        a6  OpenBSD         f0  Linux/PA-RISC b
                  11  Hidden FAT12    56  Golden Bow      a7  NeXTSTEP        f1  SpeedStor      
                  12  Compaq diagnost 5c  Priam Edisk     a8  Darwin UFS      f4  SpeedStor      
                  14  Hidden FAT16 3 61  SpeedStor       a9  NetBSD          f2  DOS secondary  
                  16  Hidden FAT16    63  GNU HURD or Sys ab  Darwin boot     fb  VMware VMFS    
                  17  Hidden HPFS/NTF 64  Novell Netware  b7  BSDI fs         fc  VMware VMKCORE
                  18  AST SmartSleep  65  Novell Netware  b8  BSDI swap       fd  Linux raid auto
                  1b  Hidden W95 FAT3 70  DiskSecure Mult bb  Boot Wizard hid fe  LANstep        
                  1c  Hidden W95 FAT3 75  PC/IX           be  Solaris boot    ff  BBT            
                  Hex code (type L to list codes): 82
                  Changed system type of partition 1 to 82 (Linux swap / Solaris)
                  • 6. Re: Questions from Linux System Admin on storage
                    Levi Pereira
                    james_p wrote:
                    Thank you. I have 2 more questions


                    Question1.
                    Should it be extended partition or Primary partition ? Should I choose 'e' or 'p' for below ?
                    # fdisk /dev/sdb
                    .
                    .
                    .
                    .
                    Command action
                    e   extended
                    p   primary partition (1-4)
                    Use primary partition

                    >
                    Question2.
                    What should be the partition type ? What should I choose from the list below
                    Command (m for help): t
                    Selected partition 1
                    Hex code (type L to list codes): l
                    
                    0  Empty           1e  Hidden W95 FAT1 80  Old Minix       bf  Solaris        
                    1  FAT12           24  NEC DOS         81  Minix / old Lin c1  DRDOS/sec (FAT-
                    2  XENIX root      39  Plan 9          82  Linux swap / So c4  DRDOS/sec (FAT-
                    3  XENIX usr       3c  PartitionMagic  83  Linux           c6  DRDOS/sec (FAT-
                    4  FAT16 <32M      40  Venix 80286     84  OS/2 hidden C:  c7  Syrinx        
                    5  Extended        41  PPC PReP Boot   85  Linux extended  da  Non-FS data    
                    6  FAT16           42  SFS             86  NTFS volume set db  CP/M / CTOS / .
                    7  HPFS/NTFS       4d  QNX4.x          87  NTFS volume set de  Dell Utility  
                    8  AIX             4e  QNX4.x 2nd part 88  Linux plaintext df  BootIt        
                    9  AIX bootable    4f  QNX4.x 3rd part 8e  Linux LVM       e1  DOS access    
                    a  OS/2 Boot Manag 50  OnTrack DM      93  Amoeba          e3  DOS R/O        
                    b  W95 FAT32       51  OnTrack DM6 Aux 94  Amoeba BBT      e4  SpeedStor      
                    c  W95 FAT32 (LBA) 52  CP/M            9f  BSD/OS          eb  BeOS fs        
                    e  W95 FAT16 (LBA) 53  OnTrack DM6 Aux a0  IBM Thinkpad hi ee  EFI GPT        
                    f  W95 Ext'd (LBA) 54  OnTrackDM6      a5  FreeBSD         ef  EFI (FAT-12/16/
                    10  OPUS            55  EZ-Drive        a6  OpenBSD         f0  Linux/PA-RISC b
                    11  Hidden FAT12    56  Golden Bow      a7  NeXTSTEP        f1  SpeedStor      
                    12  Compaq diagnost 5c  Priam Edisk     a8  Darwin UFS      f4  SpeedStor      
                    14  Hidden FAT16 3 61  SpeedStor       a9  NetBSD          f2  DOS secondary  
                    16  Hidden FAT16    63  GNU HURD or Sys ab  Darwin boot     fb  VMware VMFS    
                    17  Hidden HPFS/NTF 64  Novell Netware  b7  BSDI fs         fc  VMware VMKCORE
                    18  AST SmartSleep  65  Novell Netware  b8  BSDI swap       fd  Linux raid auto
                    1b  Hidden W95 FAT3 70  DiskSecure Mult bb  Boot Wizard hid fe  LANstep        
                    1c  Hidden W95 FAT3 75  PC/IX           be  Solaris boot    ff  BBT            
                    Hex code (type L to list codes): 82
                    Changed system type of partition 1 to 82 (Linux swap / Solaris)
                    Leave to default.


                    You will need this:
                    http://docs.oracle.com/cd/E11882_01/install.112/e22489/storage.htm#CDEBFDEH
                    • 7. Re: Questions from Linux System Admin on storage
                      rarain
                      Hi James,

                      Use primary partition and you don't need to specify partition type because to create ASM diskgroup you need RAW devices. Better to follow docs as suggested.

                      http://docs.oracle.com/cd/E11882_01/install.112/e22489/storage.htm#CDEBFDEH

                      Thanks

                      Edited by: rarain on May 8, 2013 10:52 AM
                      • 8. Re: Questions from Linux System Admin on storage
                        Omega3
                        Question2.
                        What should be the partition type ?


                        I think this is all you need
                        [oracle@el6 Desktop]$ su -
                        Password:
                        [root@el6 ~]# fdisk /dev/sdc
                        Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
                        Building a new DOS disklabel with disk identifier 0x8c2dfecc.
                        Changes will remain in memory only, until you decide to write them.
                        After that, of course, the previous content won't be recoverable.
                        
                        Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
                        
                        WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
                                 switch off the mode (command 'c') and change display units to
                                 sectors (command 'u').
                        
                        Command (m for help): n
                        Command action
                           e   extended
                           p   primary partition (1-4)
                        p
                        Partition number (1-4): 1
                        First cylinder (1-1305, default 1):
                        Using default value 1
                        Last cylinder, +cylinders or +size{K,M,G} (1-1305, default 1305):
                        Using default value 1305
                        
                        Command (m for help): w
                        The partition table has been altered!
                        
                        Calling ioctl() to re-read partition table.
                        Syncing disks.
                        [root@el6 ~]#
                        now
                        /dev/sdc1
                        will be created and ASM will take it from here