Hi experts, please share with your experiences on setting up RAC/RAC1 nodes on a remote DR site using SAN replication. Here is our environment:
Primary site with RAC One nodes on 2 servers: p-node1, p-node2
Standy DR Site will have another RAC One nodes: s-node1, s-node2
Primary and DR Data/FRA are replicated by Hitachi HUR SAN replication.
Grid Home and RDBMS Home are local, not shared, and non-replicated cross site.
Clusterware/ASM (OCR/Voting disk) on CRS DG is independent on primary and DR -- not replicated cross site.
Note: with SAN mirror, the DR site Data/FRA hdisks are not visible until failover (or split mirror).
So, the plan is:
1. Primray site: Install Grid Home, RDBMS Home, and create RAC1 database (by standard installation procedures)
... now Primary site has two RAC1 nodes functioning ...
2. Next, on DR site: Install Grid Home/Clusterware and create another ASM instance on another non-replicated CRS diskgroup
... we have the 2nd Clusterware/ASM on DR, with CRS DG only
... at this point, the Data/FRA LUNs are replicated copy -- not visible to DR servers, until failover or split mirror.
3. Next, shutdown primary site database and Clusterware. Then, either failover or split SAN mirror to Dr -- to make Data/FRA LUNs visible to DR s-node1 and s-node2 servers.
1. Once the replicated Data/FRA LUNs are visible on DR, the DR ASM instance should be able to discover the LUNs automatically, correct?
2. Since Data/FRA DG metadata is already in LUN header, will DR ASM be able to automatically incoorporate the diskgroups? (thus no need to do create diskgroups?)
If not so, HOW do I add the Data/FRA DGs to DR ASM?
... there is no 'srvctl add diskgroup' command....
3. After DR ASM recognize and mount Data/FRA diskgroups, will Clusterware automatically register the diskgroups? The following 2 links says it will automatically register at first mount. Is this correct?
We run a similar setup to this - however its a RAC running on OCFS2 shared file system not ASM so not quite as complex.
We installed on 4 node rac on site 1. Then failed over all the LUNS (which does include ORACLE_HOME and CRS home) to site to where we have another 4 servers. These are running with the SAME IP's as Site 1.
Here we ran root.sh from the crs home (a one-off setup task) to create new Votedisks and the local files (/etc/ini.d scripts, /etc/oracle stuff) and install OCFS2 etc.
Once this setup is complete we can now failover quite easily from one site to another as they use the same IP, and Votedisks etc are stored locally. We simply present the LUNS, reboot the servers, they look and find all the LUNS expected and everything comes up. If you'd like any more detailsedinfo please advise.
All very well but we are now planning our migration to ASM and so are faced with same issue as you - how will the the DATA/FRA diskgroups simply mount to the Dr site when using ASM in the same way.
I'd be very interested in any info you receive - If i get any further I'll update.
Thanks to all your input. We were able to complete the installation, as follows:
- After primary RAC nodes are installed; srvctl stop database; srvctl stop diskgroups (replicated LUNs).
- SA mount the LUNs on standby site (split mirror), then, we installed Grid Home and RDBMS Home.
- We were able to mount the LUNs on standby ASM. ASM automatically recognized the LUNs as MEMBER.
Afterward, we have to run these 2 commands to register database and service:
srvctl add service -s SERVICE_NAME -d xxx -P BASIC
(I found the above 2 commands from DB creation scripts automatically generated by DBCA; the trick is that we have to add REDO1,REDO2 to the -a option, as the generated commands by DBCA did not include REDO1/REDO2. Make sure you have all replicated diskgroups in the srvctl add database -a dg_list.)
I will bring up the issue to our architect when primary SAN crashes and last DB block is not replicated.
We are having similar issue here. May I know if you encounter problem in the DR site that the ASM will try to access the disk when it is having replication (end up it come with I/O errors)? Do you have CRS/Grid on the standby site in your design?
From our Data Center we have a 2 node RAC and from our DR site we have a AIX server that also replicate the storage of our RAC from the data center. scenario will be, We will Install A single Instance and used the Replicate Copy of the storage of RAC as storage. Can you give me some advice on how can i make use of the replicate storage as the storage of the single instance? Is installing the Grid infrastructure and the RDBMS and copy the PFILE from the PRIMARY rac can make our Single Instance up?