This discussion is archived
1 2 3 Previous Next 32 Replies Latest reply: Oct 30, 2012 10:17 AM by 971499 Go to original post RSS
  • 15. Re: ACFS volume shows up as disabled after reboot or crs stop / start.
    Allan Newbie
    Currently Being Moderated
    Levi -
    I don't want to detract from the primary question asked here so I'm happy to continue this discussion offline. I'll try and explain what's going on with the ACFS resources below, please feel free to ask for more clarification if necessary.

    Almost all text below referring to the ACFS registry resource also applies to the single file system resource. These are known as CRS Managed file systems, and differ from unmanaged file systems due to their highly available nature, recovery characteristics, and the ability to be mounted and unmounted during CRS stack startup and shutdown.

    I recommend that you try the following experiment:
    a) add a file system to the registry.
    b) cd to that file system
    c) from another terminal session, stop the registry resource


    You will notice that the process on the file system gets killed.

    You will see in your alert logs the following message:
    ACFS-09126 - 09126, 0, "Attempting to terminate the program '%s' with OS process ID '%s'."
    // *Cause:  The process was using an ACFS file system, which was being forcibly
    // unmounted. A kill signal was sent to allow the unmount to proceed.
    // *Action: This message is informational.


    Again, reading the release note, you will note the words "may", "highly recommended", etc.

    Why?

    ACFS does a very good job of ensuring that all data is placed on disk. This prevents the "corruption" that you refer to. However, shutting down the CRS stack is similar to shutting down the system.

    Any system administrator that is shutting down the system will ensure that programs are properly shutdown. This prevents strange hangs during shutdown, it prevents unflushed data, it prevents other random issues. The same is true of ACFS and its resources.

    Since the ACFS registry resource does not allow for dependencies of applications to be placed on it, we can't guarantee that the ACFS resource will be stopped after applications are shutdown. Any programs running off the mount point outside the CRS managed space will effectively force the situation that you are concerned about - a process will attempt to be terminated during the clean stage of the ACFS registry resource (or single file system resource).


    If you don't ensure you have dismounted your ACFS file system by manually doing so, the file system itself will not be corrupt. Rather, you do risk potential data issues from processes that were caching their data, as they will not have the chance to write their data to disk. The ACFS file system itself will be coherent and can be remounted later on.

    A file system that has its backing storage go away will be marked as 'OFFLINE' in 'acfsutil info fs'. As soon as the storage is again available, you can unmount and remount the file system to bring it back online. If the file system is CRS managed - via ACFS registry or single file system, then the resource will attempt to do so, killing processes that are identified as using the file system.

    For reference, one can refer to the following 2 sections:
    1) ACFS registry section of the ASM Admin guide - http://docs.oracle.com/cd/E11882_01/server.112/e18951/asmfs_extra.htm#CACEDAJG
    2) Single file system section of the ASM Admin guide - http://docs.oracle.com/cd/E11882_01/server.112/e18951/asmfs_extra.htm#CACDDIBD


    If your file system is mounted, you will not hang ASM. You will see a message that the volume and disk group cannot be disabled, as there are open references on the volume\disk group. A forced dismount of the disk group will still dismount the disk group, even if the volume is still mounted. ASM can still be shutdown. But attempts to reenable the volume afterwards will fail until you unmount the volume. This is where the ACFS CRS resources come in - they automate this process for you.

    As a system administrator, I must profess caution - I always try and stop actively running programs properly before shutting things down. I also want a file system that can recover from storage failures properly - and ACFS provides that. I don't put it in a situation where I risk improperly shutdown programs, but when my storage has an issue, I'm secure in the knowledge that only the affected file system will be taken offline, and with a simple remount when the storage is available again I can have my data back.




    Back to the original poster's question:
    1) All ACFS CRS resources will enable\disable the volumes and mount\dismount the file system.
    2) You will see messages in the CRS alert log for mounts and unmounts.
    3) It is best practice to ensure that file systems are quiesced and unmounted before shutting the stack down to facilitate smooth stack shutdown. Another way to do this would be to ensure that every program accessing the ACFS mount point has a CRS resource and that your dependencies are properly placed so that the programs\resources are stopped before the ACFS file system resource is stopped.

    What we need to do is to ensure that the following items are true:
    1) ora.registry.acfs is online - crsctl stat res ora.registry.acfs
    2) there are no volume enable errors in the crs alert log.

    If those are true, one other thought - can you try adding the file system to the registry and not specify any nodes?

    Thanks,
    Allan
  • 16. Re: ACFS volume shows up as disabled after reboot or crs stop / start.
    Levi-Pereira Guru
    Currently Being Moderated
    Great arguments were convincing.

    I have used ACFS in version 11.2.0.2 and faced problems of hangs during shutdown of CRS because files were opened in ACFS.

    So, Oracle solution was make sure that you don't have openfiles on ACFS volumes during shutdown of CRS. It make no sense since in some situations when SA needed issue shutdown on Host, but before he had to clean all the openfiles ACFS volumes.

    So, The good news was I noticed some improvements in version 11.2.0.3 that handles openfiles on ACFS, as you mentioned.

    About "corrupted" word that I have used, I just means If ASM have a Shutdown Forced (due device busy) without disable Volumes or Diskgroup that can cause a potential corruption on that filesystem. That does not apply on all situation.
    What we need to do is to ensure that the following items are true:
    1) ora.registry.acfs is online - crsctl stat res ora.registry.acfs
    2) there are no volume enable errors in the crs alert log.
    If those are true, one other thought - can you try adding the file system to the registry and not specify any nodes?
    I have no env to test it for now. Far I could understand if we not specify any nodes that mean that filesystem should be mounted or dismounted on all nodes during sutdown of CRS on that node.

    Documentation:
    acfsutil registry with "-n" option:
    This is used in combination with -a and specifies an optional comma-delimited list of nodes, or the all keyword.
    This specifies which nodes should attempt to mount this device on the specified mount point. Host names should be specified.
    The all keyword is functionally equivalent to not specifying a list of nodes and indicates that the device should be mounted on all nodes. Mounting on all nodes is the default behavior.

    Soon I will make test using version 11.2.0.3 and take my conclusions on this matter.

    Last question is your last name "Graves"?

    Thanks,
    Levi Pereira
  • 17. Re: ACFS volume shows up as disabled after reboot or crs stop / start.
    Allan Newbie
    Currently Being Moderated
    Levi -
    Yes, my last name is really Graves. :)

    Regarding 11.2.0.3, we did make a number of improvements that handled open files on the mount point better. There will always be processes that cannot be killed, such as those on Windows marked 'do not kill', or those that are stuck in IO on *nix.  However, if you are using resources to manage your file systems and you have resources for all your binaries that run off the mount point, you should be able to stop and start the stack and have the resources automatically handle things.

    If you do not specify a node when registering a file system using 'acfsutil registry', then yes, it should mount and unmount on all nodes automatically during CRS stack start and stop.

    Please let me know if you encounter issues on 11.2.0.3 with your testing!
  • 18. Re: ACFS volume shows up as disabled after reboot or crs stop / start.
    Levi-Pereira Guru
    Currently Being Moderated
    Hi Allan,
    Thanks for correcting some points in this thread. I'll keep in touch about issues on this matter.

    I already have your e-mail :)
  • 19. Re: ACFS volume shows up as disabled after reboot or crs stop / start.
    SamTrashers Newbie
    Currently Being Moderated
    Hello,

    i get the same issue on same platform RHEL6/11.2.0.3.3
    => patch 14494305 is applyied

    from any situation (filesystem dismounted or not, before reboot),
    after a reboot, the acfs volume is always in a disabled state.

    i have to manualy perform
    ASMCMD> volenable -a
    # mount.acfs -o all

    i've recently install nearly the same on AIX6.1/11.2.0.3.2
    the filesystem is mounted normally after a reboot.

    ...

    recommended patch 13919095 has to be rolled back due to lack of information and :
    "11.2.0.3.3 Grid Infrastructure Patchset: 13919095 Installation On RH 6.3 Fails With ACFS-9459 . [ID 1493491.1]"
    so, the procedured folowed is
    - install 11.2.0.3 GI & DB
    - apply patch 13919095
    - rollback patch 13919095
    - install patch 14494305

    do i need to change the compatible attribute of diskgroups ?
    what should i do to automatically mount the acfs filesystems ?

    Thank's
  • 20. Re: ACFS volume shows up as disabled after reboot or crs stop / start.
    Allan Newbie
    Currently Being Moderated
    Sam -
    In order to help, more information is necessary:
    1) State of the ora.registry.acfs resource - 'crsctl stat res ora.registry.acfs'
    2) output of 'acfsutil registry'.


    Given that you installed on RHEL6.3 and then applied the patch for RHEL6.3, I'm going to suspect you don't have a registry resource.

    If that's the case, please try running 'acfsroot enable'. Then try a stack restart and see if that fixes your issue.
  • 21. Re: ACFS volume shows up as disabled after reboot or crs stop / start.
    SamTrashers Newbie
    Currently Being Moderated
    thank for your help ...

    after a reboot of node 1

    [grid@med-ca-pora1 ~]$ acfsutil registry
    acfsutil registry: CLSU-00100: Operating System function: open64 failed with error data: 2
    acfsutil registry: CLSU-00101: Operating System error message: No such file or directory
    acfsutil registry: CLSU-00103: error location: OOF_1
    acfsutil registry: CLSU-00104: additional error information: open64 (/dev/ofsctl)
    acfsutil registry: ACFS-00502: Failed to communicate with the ACFS driver. Verify the ACFS driver has been loaded.

    [grid@med-ca-pora1 ~]$ exit
    logout
    [root@med-ca-pora1 ~]# */u01/app/11.2.0/grid/bin/acfsload start -s*
    [root@med-ca-pora1 ~]# su - grid -c "asmcmd volenable -a"
    [root@med-ca-pora1 ~]# *mount.acfs -o all*


    [grid@med-ca-pora1 ~]$ *crsctl stat res ora.registry.acfs*
    CRS-2613: Could not find resource 'ora.registry.acfs'.


    [[root@med-ca-pora1 ~]# */u01/app/11.2.0/grid/bin/acfsroot enable*
    ACFS-9382: Modification of ADVM/ACFS drivers resource succeeded.
    ACFS-9398: Modification of ADVM/ACFS registry resource succeeded.

    [grid@med-ca-pora1 ~]$ *acfsutil registry*
    Mount Object:
    Device: /dev/asm/shared-369
    Mount Point: /shared
    Disk Group: DG_CFS
    Volume: SHARED
    Options: none
    Nodes: all

    [grid@med-ca-pora1 ~]$ *crsctl stat res ora.registry.acfs*
    NAME=ora.registry.acfs
    TYPE=ora.registry.acfs.type
    TARGET=ONLINE , ONLINE
    STATE=ONLINE on med-ca-pora1, OFFLINE

    [root@med-ca-pora1 ~]# uname -a
    Linux med-ca-pora1 2.6.32-279.5.2.el6.x86_64 #1 SMP Tue Aug 14 11:36:39 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux

    so you're right,
    rebbot is in progress
    come back in few minutes ...
  • 22. Re: ACFS volume shows up as disabled after reboot or crs stop / start.
    SamTrashers Newbie
    Currently Being Moderated
    well ...
    everything is all right now ...
    acfs file system automatically mount after reboot ....

    [grid@med-dc-pora1 ~]$ crsctl stat res ora.registry.acfs
    NAME=ora.registry.acfs
    TYPE=ora.registry.acfs.type
    TARGET=ONLINE , ONLINE
    STATE=ONLINE on med-ca-pora1, ONLINE on med-dc-pora1


    ... but where is the mistake
    acfsroot is part of root.sh ?
  • 23. Re: ACFS volume shows up as disabled after reboot or crs stop / start.
    Allan Newbie
    Currently Being Moderated
    Since your OS was unsupported by ACFS until you installed the patch, the root script did not create the resources for ACFS.



    Please see note #6 in https://support.us.oracle.com/oip/faces/secure/km/DocumentDisplay.jspx?id=1369107.1
  • 24. Re: ACFS volume shows up as disabled after reboot or crs stop / start.
    SamTrashers Newbie
    Currently Being Moderated
    ok,
    thank you very much ...
  • 25. Re: ACFS volume shows up as disabled after reboot or crs stop / start.
    user9308584 Newbie
    Currently Being Moderated
    Allan, Levi, all -- thank you. I tested registering ACFS with CRS and that solved the issue. Here's my summary / findings. You can see the users in the file system being terminated without having to manually un-mount the file system prior to shutdown -- CRS will handle all of this and bring it back up mounted -- unlike the 'general purpose' registration.

    When users are in the ACFS file system and it's a 'general-purpose' registration 'acfsutil registry -a -f -n lmmk87,lmmk88 /dev/asm/insracd_vg-107 /insracd' it will not un-mount the filesystem and it will cause ASM to hang on the way down requiring a forced shutdown of the clusterware resulting in general badness for the ACFS volume on restart.

    Messages from ASM log when users are logged into a general-purpose registered ACFS.

    ORA-15032: not all alterations performed
    ORA-15027: active use of diskgroup "FS_ORCL_CFS0195" precludes its dismount
    NOTE: LGWR doing non-clean dismount of group 4 (FS_ORCL_CFS0195)
    NOTE: LGWR sync ABA=8.74 last written ABA 8.74
    Thu Sep 20 11:32:01 2012

    Deregistered file system and registered with CRS using srvctl add filesystem

    $GRID_HOME/bin/srvctl add filesystem -d /dev/asm/insracd_vg-107 -v INSRACD_VG -g FS_ORCL_CFS0195 -m /insracd -u oracle

    We can see multiple users in the filesystem (they know it's going down) ….

    oracle@lmmk88:/insracd> sudo root.sh fuser -c /insracd
    /insracd: 11652c 11653c 18359c 20365c 20367c 23919c 26277c 26298c 31927c

    Shutdown the cluster ...

    oracle@lmmk88:/grid/oracle/product/11.2.0.3/bin> sudo root.sh ./crsctl stop crs
    CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'lmmk88'
    CRS-2673: Attempting to stop 'ora.crsd' on 'lmmk88'
    CRS-2677: Stop of 'ora.CR_VOTE.dg' on 'lmmk88' succeeded

    ...
    CRS-5014: Agent "/grid/oracle/product/11.2.0.3/bin/orarootagent.bin" timed out starting process "/grid/oracle/product/11.2.0.3/bin/acfssinglefsmount" for action "stop": details at "(:CLSN00009:)" in "/grid/oracle/product/11.2.0.3/log/lmmk88/agent/crsd/orarootagent_root/orarootagent_root.log"
    CRS-5017: The resource action "ora.fs_orcl_cfs0195.insracd_vg.acfs stop" encountered the following error:
    (:CLSN00009:)Utils:execCmd aborted. For details refer to "(:CLSN00108:)" in "/grid/oracle/product/11.2.0.3/log/lmmk88/agent/crsd/orarootagent_root/orarootagent_root.log".

    CRS-2675: Stop of 'ora.fs_orcl_cfs0195.insracd_vg.acfs' on 'lmmk88' failed
    CRS-2679: Attempting to clean 'ora.fs_orcl_cfs0195.insracd_vg.acfs' on 'lmmk88'
    CRS-2681: Clean of 'ora.fs_orcl_cfs0195.insracd_vg.acfs' on 'lmmk88' succeeded
    CRS-2673: Attempting to stop 'ora.FS_ORCL_CFS0195.dg' on 'lmmk88'
    CRS-2677: Stop of 'ora.FS_ORCL_CFS0195.dg' on 'lmmk88' succeeded
    CRS-2673: Attempting to stop 'ora.asm' on 'lmmk88'

    ...

    You can see users being terminated that were holding up the dismount of the file system in this log ...

    "/grid/oracle/product/11.2.0.3/log/lmmk88/agent/crsd/orarootagent_root/orarootagent_root.log".


    2012-09-28 14:02:58.133: [ora.fs_orcl_cfs0195.insracd_vg.acfs][939513600] {2:8358:29875} [stop] (:CLSN00010:)ACFS-9253: Failed to unmount mount point '/insracd'. Mount point likely in use.

    2012-09-28 14:03:02.931: [ora.fs_orcl_cfs0195.insracd_vg.acfs][805304064] {2:8358:29875} [clean] (:CLSN00010:)ACFS-9153: Program 'su - indev3' with OS process ID '11652' is using mount point '/insracd'.

    2012-09-28 14:03:02.931: [ora.fs_orcl_cfs0195.insracd_vg.acfs][805304064] {2:8358:29875} [clean] (:CLSN00010:)ACFS-9126: Attempting to terminate the program 'su - indev3' with OS process ID '11652'.

    2012-09-28 14:03:03.735: [ora.fs_orcl_cfs0195.insracd_vg.acfs][805304064] {2:8358:29875} [clean] (:CLSN00010:)ACFS-9251: Mount point '/insracd' unmounted.

    2012-09-28 14:03:03.735: [ora.fs_orcl_cfs0195.insracd_vg.acfs][805304064] {2:8358:29875} [clean] (:CLSN00010:)
    2012-09-28 14:03:03.735: [ora.fs_orcl_cfs0195.insracd_vg.acfs][805304064] {2:8358:29875} [clean] execCmd ret = 0
    2012-09-28 14:03:03.736: [ora.fs_orcl_cfs0195.insracd_vg.acfs][805304064] {2:8358:29875} [clean] AcfsAgent::clean returning: sid=+ASM2; volumeDevice=/dev/asm/insracd_vg-107; mountPath=/insracd
    2012-09-28 14:03:03.736: [ora.fs_orcl_cfs0195.insracd_vg.acfs][805304064] {2:8358:29875} [clean] (:CLSN00106:) clsn_agent::clean }
    2012-09-28 14:03:03.736: [    AGFW][805304064] {2:8358:29875} Command: clean for resource: ora.fs_orcl_cfs0195.insracd_vg.acfs lmmk88 1 completed with status: SUCCESS
    2012-09-28 14:03:03.736: [ora.fs_orcl_cfs0195.insracd_vg.acfs][805304064] {2:8358:29875} [check] AcfsAgent::check entered: sid=+ASM2; volumeDevice=/dev/asm/insracd_vg-107; mountPath=/insracd; checkType=CHECK_AFTER_CMD


    ora.fs_orcl_cfs0195.insracd_vg.acfs
    ONLINE ONLINE lmmk87 mounted on /insracd
    ONLINE ONLINE lmmk88 mounted on /insracd


    Thanks again,

    Malcolm.
  • 26. Re: ACFS volume shows up as disabled after reboot or crs stop / start.
    Allan Newbie
    Currently Being Moderated
    Malcolm -
    When users are in the ACFS file system and it's a 'general-purpose' registration 'acfsutil registry -a -f -n lmmk87,lmmk88 /dev/asm/insracd_vg-107 /insracd' it will not
    unmount the filesystem and it will cause ASM to hang on the way down requiring a forced shutdown of the clusterware resulting in general badness for the ACFS volume
    on restart.
    This is incorrect. The registry should do exactly the same thing that the single file system resource does.

    Can you provide the output of 'crsctl stop res ora.registry.acfs' when you have a file system registered in it?

    thanks,
    Allan
  • 27. Re: ACFS volume shows up as disabled after reboot or crs stop / start.
    user9308584 Newbie
    Currently Being Moderated
    Allan,

    I currently have ACFS registered with CRS. I checked for ACFS resources prior to changing ACFS to a CRS managed resource and I don't recall seeing a registry entry in CRS nor do I have a record from a prior crs stat to go back to. I would expect it at a minimum to have show up as 'offline' as it is supposed to be installed as part of the root scripts following GI installation. Is this missing entry the cause of the general-purpose ACFS unmount hanging when users are still using the filesystem?

    ORA-15032: not all alterations performed
    ORA-15027: active use of diskgroup "FS_ORCL_CFS0195" precludes its dismount


    Here's how it is now and so far, this hasn't happened again ...

    sudo root.sh $GRID_HOME/bin/crsctl stat res ora.registry.acfs
    CRS-2613: Could not find resource 'ora.registry.acfs'.

    oracle@lmmk88:/insracd/tune> sudo root.sh $GRID_HOME/bin/crsctl stat res -t
    --------------------------------------------------------------------------------
    NAME TARGET STATE SERVER STATE_DETAILS
    --------------------------------------------------------------------------------
    Local Resources
    --------------------------------------------------------------------------------
    ora.CR_VOTE.dg
    ONLINE ONLINE lmmk87
    ONLINE ONLINE lmmk88
    ora.DB_ORCL_DAT0195.dg
    ONLINE ONLINE lmmk87
    ONLINE ONLINE lmmk88
    ora.DB_ORCL_FRA0195.dg
    ONLINE ONLINE lmmk87
    ONLINE ONLINE lmmk88
    ora.FS_ORCL_CFS0195.dg
    ONLINE ONLINE lmmk87
    ONLINE ONLINE lmmk88
    ora.LISTENER.lsnr
    ONLINE ONLINE lmmk87
    ONLINE ONLINE lmmk88
    ora.LISTENER_ASM.lsnr
    ONLINE ONLINE lmmk87
    ONLINE ONLINE lmmk88
    ora.asm
    ONLINE ONLINE lmmk87 Started
    ONLINE ONLINE lmmk88 Started
    ora.fs_orcl_cfs0195.insracd_vg.acfs
    ONLINE ONLINE lmmk87 mounted on /insracd
    ONLINE ONLINE lmmk88 mounted on /insracd
    ora.gsd
    OFFLINE OFFLINE lmmk87
    OFFLINE OFFLINE lmmk88
    ora.net1.network
    ONLINE ONLINE lmmk87
    ONLINE ONLINE lmmk88
    ora.ons
    ONLINE ONLINE lmmk87
    ONLINE ONLINE lmmk88
    --------------------------------------------------------------------------------
    Cluster Resources
    --------------------------------------------------------------------------------
    ora.LISTENER_SCAN1.lsnr
    1 ONLINE ONLINE lmmk88
    ora.LISTENER_SCAN2.lsnr
    1 ONLINE ONLINE lmmk87
    ora.LISTENER_SCAN3.lsnr
    1 ONLINE ONLINE lmmk87
    ora.cvu
    1 ONLINE ONLINE lmmk87
    ora.insracd.db
    1 ONLINE ONLINE lmmk87 Open
    2 ONLINE ONLINE lmmk88 Open
    ora.insracd.ins_d1.svc
    1 ONLINE ONLINE lmmk87
    ora.insracd.insracd1.fmr.com.svc
    1 ONLINE ONLINE lmmk87
    ora.insracd.insracd2.fmr.com.svc
    1 ONLINE ONLINE lmmk88
    ora.insracd.insracdmaint.fmr.com.svc
    1 ONLINE ONLINE lmmk87
    2 ONLINE ONLINE lmmk88
    ora.insracd.xparallel.svc
    1 ONLINE ONLINE lmmk87
    2 ONLINE ONLINE lmmk88
    ora.insracs.db
    1 OFFLINE OFFLINE
    2 ONLINE ONLINE lmmk88 Open
    ora.lmmk87.vip
    1 ONLINE ONLINE lmmk87
    ora.lmmk88.vip
    1 ONLINE ONLINE lmmk88
    ora.oc4j
    1 ONLINE ONLINE lmmk87
    ora.scan1.vip
    1 ONLINE ONLINE lmmk88
    ora.scan2.vip
    1 ONLINE ONLINE lmmk87
    ora.scan3.vip
    1 ONLINE ONLINE lmmk87
    oracle@lmmk88:/insracd/tune>

    Malcolm
  • 28. Re: ACFS volume shows up as disabled after reboot or crs stop / start.
    Allan Newbie
    Currently Being Moderated
    Malcolm -
    If you don't have an ACFS registry resource, then you won't have any management of the file systems registered in it, resulting in the behavior you've seen - but only for file systems in the ACFS registry.

    CRS should definitely report an 'ora.registry.acfs' resource.

    If you are on 11.2.0.3 you can try the 'acfsroot enable' command.

    If you are on 11.2.0.2 or 11.2.0.1 you can try the 'acfsroot reregister' command.

    If you install CRS on an unsupported platform (such as a version of UEK that ACFS did not support) and then you upgrade your GI install or apply a patch that provides ACFS support, in 11.2 there would still be missing resources.
  • 29. Re: ACFS volume shows up as disabled after reboot or crs stop / start.
    user9308584 Newbie
    Currently Being Moderated
    Allan,

    I could probably live with this working under CRS, but there shouldn't be any need unless it's a shared $ORACLE_HOME, etc. The server is still in development so as soon as I can I'll de-registering from CRS, running acfsroot reregister and registering acfs as a general purpose FS again and from everything I've read here, what you have said and this note on Metalink, this should do it. This note 1451839.1 mentions the acfsroot reregister command and it will create "ora.drivers.acfs and registry resource ora.registry.acfs". That was the other side-effect, never remounting on clusterware restart. I'm going back over my tracks trying to figure where / how the registry entry was lost. I applied patch p13919095 to 11.2.0.3 successfully a month ago ... don't know.

    Thank you very much.

    Malcolm.

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points