1 2 Previous Next 15 Replies Latest reply: Feb 27, 2012 3:25 AM by 807928 RSS

    Solaris Cluster 3.2 - How to add a global file system

    user244261
      Dear Admins,

      I'm managing two nodes Solaris cluster 3.2 which is having Oralce db running (not RAC).

      Now I need to intorduce new file system from our sotrage (STK 6140) and mount it as a global file system which should be access by both the cluster nodes.

      I have managed upto populating the Luns and following is the devices,

      DID Device Name: /dev/did/rdsk/d10
      Full Device Path: degbkpsrv2:/dev/rdsk/c6t600A0B80004832CC0000275B4F3CC12Fd0
      Full Device Path: degbkpsrv1:/dev/rdsk/c6t600A0B80004832CC0000275B4F3CC12Fd0
      Replication: none
      default_fencing: global

      Now I need to add this volume as a global file system and mount it on /global/APP_ext

      I cannot find the precise steps to proceed from this point and help would be really appreciated.

      DK
        • 1. Re: Solaris Cluster 3.2 - How to add a global file system
          807928
          So, first you need to make sure your device is sliced the way you want. I assume you'll probably want to use the whole disk (s2), but even so, just check with:

          # format /dev/did/rdsk/d10s2
          # newfs /dev/did/rdsk/d10s2

          Then add an entry to /etc/vstab on all nodes:
          /dev/global/dsk/d10s2 /dev/global/rdsk/d10s2 /global/APP_ext ufs - yes global,logging

          That mounts the file system globally at boot time.

          Then in your Oracle resource group, you'll either need to augment your HAStoragePlus resource or create one so that your Oracle database is collocated with the global file system primary.

          Here is an example of how to create such a resource
          # clrt register SUNW.HAStoragePlus
          # clrs create -g your-oracle-rg -t SUNW.HAStoragePlus -p FileSystemMountPoints=/global/APP_ext -p AffinityOn=true db-hasp-rs

          The make your Oracle server resource depend on this:
          # clrs set -p resource_dependencies_offline_restart=db-hasp-rs my-oracle-svr-rs

          Regards,

          Tim
          ---

          Edited by: TimRead on Feb 16, 2012 2:00 PM
          • 2. Re: Solaris Cluster 3.2 - How to add a global file system
            user244261
            Hi Tim,

            Really appreciated your quick response. When I try to create the file system on the device it gives me an IO error.

            root@degbkpsrv1 # newfs /dev/did/rdsk/d10s2
            /dev/did/rdsk/d10s2: I/O error

            Any thought on this please ?

            DK
            • 3. Re: Solaris Cluster 3.2 - How to add a global file system
              807928
              Does it show up in format properly? Is there even a slice 2 defined (there should be). Are there any other useful messages in /var/adm/messages? Have you accidentally LUN masked it or zoned it away from these nodes from the storage box or SAN switch?

              Tim
              ---
              • 4. Re: Solaris Cluster 3.2 - How to add a global file system
                user244261
                Hi Tim

                <i><b>Does it show up in format properly?</b></i>

                Yes, the format command shows the 10GB disk properly.

                partition> p
                Volume: test
                Current partition table (original):
                Total disk cylinders available: 5118 + 2 (reserved cylinders)

                Part Tag Flag Cylinders Size Blocks
                0 usr wm 3 - 5117 9.99GB (5115/0/0) 20951040
                1 unassigned wm 0 0 (0/0/0) 0
                2 unassigned wm 0 0 (0/0/0) 0
                3 unassigned wm 0 0 (0/0/0) 0
                4 unassigned wm 0 0 (0/0/0) 0
                5 unassigned wm 0 0 (0/0/0) 0
                6 unassigned wm 0 0 (0/0/0) 0
                7 usr wu 0 - 2 6.00MB (3/0/0) 12288


                <i><b>Are there any other useful messages in /var/adm/messages?</b></i>

                There are no messages in for during the time I executed the commands.

                <i><b>Have you accidentally LUN masked it or zoned it away from these nodes from the storage box or SAN switch?</b></i>
                No

                Thanks in advance,
                DK
                • 5. Re: Solaris Cluster 3.2 - How to add a global file system
                  807928
                  OK, so as you can see, you don't have a slice 2 define, which is not normal. It's not required, but I think it's a convention.

                  You can continue by just substituting s0 for s2 in my original example, or define slice 2 to be the whole disk as well, i.e. equivalent to slice 0.

                  Regards,

                  Tim
                  ---
                  • 6. Re: Solaris Cluster 3.2 - How to add a global file system
                    user244261
                    Hi Tim,

                    Pardon for my ignorence of not noticing the s2 is not available.

                    I substituted s2 with s0 and the newfs command worked liked charm and required entries made in the /etc/vfstab as well.

                    Now I need to add this file system as a resource of the existing resource group scnb-harg.

                    root@degbkpsrv2 # clrg status

                    === Cluster Resource Groups ===

                    Group Name Node Name Suspended Status
                    ---------- --------- --------- ------
                    scnb-harg degbkpsrv2 No Online
                    degbkpsrv1 No Offline

                    I tried to use # clsetup command, but it only allowed me to create new resource group and doesn't provide an option to give the existing one.

                    Highly appreciate your help here.

                    DK
                    • 7. Re: Solaris Cluster 3.2 - How to add a global file system
                      807928
                      I wouldn't worry, it's often the obvious stuff you miss - me included.

                      So, I don't know what resources you have in your resource group scnb-harg, it would help to have the output from:

                      # clrs status scnb-harg

                      I'll assume you have everything in it except this file system resource. If not, post back again.

                      So, my original commands now become:
                      # clrt register SUNW.HAStoragePlus
                      # clrs create -g scnb-harg -t SUNW.HAStoragePlus -p FileSystemMountPoints=/global/APP_ext -p AffinityOn=true db-hasp-rs

                      Then make your Oracle server resource depend on this:
                      # clrs set -p resource_dependencies_offline_restart=db-hasp-rs my-oracle-svr-rs

                      Regards,

                      Tim
                      ---
                      • 8. Re: Solaris Cluster 3.2 - How to add a global file system
                        user244261
                        Hi Tim,

                        You are a superstar :)

                        I was strggling to achieve this for days following with Oracle (Sun) documentation and why this is not documented properly the way you have presented ?

                        I did this procedures (for testing) on a less critical cluster (not in the production Oracle DB cluster where I really wanted to add the file system).

                        <i><b># clrt register SUNW.HAStoragePlus </b></i> gave an error becuase the resource type is already registered.


                        I assume the command <i><b># clrs set -p resource_dependencies_offline_restart=db-hasp-rs my-oracle-svr-rs</b></i> would need the oracle resource to restart ?

                        What exactly is the "my-oracle-svr-rs" which you have given in the above command ?

                        Now I have the confidence to carry this out on the production Oracle cluster.

                        I found the other file system mounted are in the form of /dev/md/ . Is it required to add the above file system also into the same disk set before it's being added to the resource group ?
                        • 9. Re: Solaris Cluster 3.2 - How to add a global file system
                          807928
                          890717 wrote:
                          Hi Tim,

                          You are a superstar :)

                          I was strggling to achieve this for days following with Oracle (Sun) documentation and why this is not documented properly the way you have presented ?

                          I did this procedures (for testing) on a less critical cluster (not in the production Oracle DB cluster where I really wanted to add the file system).

                          <i><b># clrt register SUNW.HAStoragePlus </b></i> gave an error becuase the resource type is already registered.
                          That doesn't matter. It's just a warning.

                          I assume the command <i><b># clrs set -p resource_dependencies_offline_restart=db-hasp-rs my-oracle-svr-rs</b></i> would need the oracle resource to restart ?
                          What this is doing is telling Oracle Solaris Cluster to shutdown the Oracle database if the shared file system goes offline and only restart it if it comes back online. I'm assuming that your Oracle database files are on the shared file system?
                          What exactly is the "my-oracle-svr-rs" which you have given in the above command ?
                          This is the name of the Solaris Cluster resource that represents the Oracle server. I assume you have the database controlled by an Solaris Cluster SUNW.oracle_server resource?
                          Now I have the confidence to carry this out on the production Oracle cluster.
                          OK.
                          I found the other file system mounted are in the form of /dev/md/ . Is it required to add the above file system also into the same disk set before it's being added to the resource group ?
                          Almost impossible to answer without knowing a whole lot more about your cluster.

                          I have to say, I'm a little concerned about where this has gone. Are you sure you wanted a globally mounted file system and not a failover file system? It would help to know what you were really trying to achieve and where you had originally got to.

                          Anyway, all the above should have answered the original question.... :-)

                          Tim
                          ---
                          • 10. Re: Solaris Cluster 3.2 - How to add a global file system
                            user244261
                            Hi Tim,

                            Yes, my question is answered and lot more :)

                            Yes, I asked for the global file system and it would be available for both the nodes. I know that the existing file system which is configured here is a failover file system and I don't bother as long as the file system is available for both the nodes when ever the oracle resources are fail over to either one.

                            So thank you so much for your excellant assistance and you are truely a superstar :)

                            We even have platinum support for Solaris cluster with Oracle, but they refused to give these procedures mentioning I need to get it done through Oracle professional services. :)

                            But here you are for my rescue....!

                            Cheers Tim
                            • 11. Re: Solaris Cluster 3.2 - How to add a global file system
                              807928
                              I can't comment on what support services asked.

                              I will, however, make one more observation and that is that unless you need the functionality of a global file system, i.e. one that is mounted on all node simultaneously, you are better off using a failover file system. For an Oracle database, it probably won't matter, but for other services, opening, closing and writing to files on a global file system comes with a cost. Therefore a failover file system performs better in these cases.

                              <Shameless plug>If you want to know more about Solaris Cluster, I can recommend a very good book - "Oracle Solaris Cluster Essentials". I have a copy next to me and I refer to it regularly. Quite often when I need to post examples to this forum. :-)

                              Tim
                              ---
                              • 12. Re: Solaris Cluster 3.2 - How to add a global file system
                                user244261
                                Thank you very much Tim for the recommendations. I will definitly try to get the book you've mentioned.

                                If it's is not too much to ask, how to do it as the failover file system instead global file system ? Can we change the global file system as a failover file system later if we want ?


                                Wow !! So you are the author of the book !!! Great I'm going to order it right now from Amazon....!

                                Edited by: 890717 on Feb 16, 2012 8:41 AM
                                • 13. Re: Solaris Cluster 3.2 - How to add a global file system
                                  807928
                                  The only real difference between a failover file system (UFS or VxFS) and a global file system (UFS or VxFS) is the mount option. Therefore, you are free to change between the two options as you need.

                                  In general, a global file system has the 'mount at boot' flag in /etc/vfstab set to 'yes', while a failover file system has it set to 'no', and in addition, the global file system has the 'global' mount option. The failover file system is then mounted by an HAStoragePlus resource and if you have one of the resource_dependencies* properties set, then your applications that depend on it will wait for it to be mounted before they attempt to start.

                                  One other useful tip, while we're on this subject, is that if you need to 'bulk load' files onto a global file system, that is perform a restore, or install software, then it is often a lot faster to temporarily mount that file system as a failover file system and then, when the process is complete, remount it as a global file system

                                  Tim
                                  ---
                                  • 14. Re: Solaris Cluster 3.2 - How to add a global file system
                                    user244261
                                    Hi Tim,

                                    Could you please elaborate how do we revert back this dependency,

                                    <i><b>#clrs set -p resource_dependencies_offline_restart=db-hasp-rs my-oracle-svr-rs</b></i>
                                    1 2 Previous Next