Oracle Dismounter for Linux

Version 8
Visibility: Open to anyone

     

     

    1   About

     

    1.1   Why Oracle Dismounter?

     

    Oracle High Availability Services (OHAS) provide and extend the interoperability of computer nodes and
    also startup and shutdown Oracle Clusterware in RAC or Oracle Restart in standalone installations. Oracle
    Clusterware and Oracle Restart manage availability, startup, and shutdown of Oracle software components
    and dependencies, including the automated startup and shutdown of Oracle Database. It does, however,
    not ensure a clean detachment of Oracle mounted filesystems.

     

    For example:

     

    Modern journal or transactional based filesystems include automatic and efficient recovery when unmounted

    improperly, but the potential risk of losing data, however, remains.

     

     

    Note

    You may notice failures to unmount the filesystem of ORACLE_HOME when you have configured a persistent systemd journal and set up the ASM Filter Driver as part of the Oracle 19c Grid Infrastructure installation (19.3).

     

    Oracle Dismounter does not repair the source of any problems associated with Oracle High Availability Services (OHAS), but it attempts to ensure a clean unmount of Oracle filesystems and provides a persistent log file to review past OS shutdowns.

     

     

    1.2   How Does it Work?

     

    Oracle Dismounter is a systemd service that runs during a system shutdown:

     

    1. Activates after the shutdown completion or failure of Oracle OHAS and AFD services.
    2. Terminates (SIGKILL) remaining Clusterworks or Oracle Restart processes.
    3. Unloads the ASM filter driver.
    4. Unmounts specified filesystems.
    5. Provides a log file to verify the operation.

     

    Oracle Dismounter, by default, unmounts /u01, /u02, /u03, and /u04, but this can be customized.

     

    Tip:

    If you are familiar with shell scripting, you may also use oracle-dismounter to troubleshoot the shutdown process or execute your own code.

     

     

     

    All information herein is presented in the hope to be useful for members of the Oracle community and provided under
    the terms and conditions at http://www.oracle.com/us/legal/terms/index.html.

     

    Copyright (c) 2019, Dude! @ Oracle Communities

     

    You may copy and modify the scripts at your disposal, but you may not redistribute and plagiarize. Thanks!

     

     

    1.4   Comments

     

    For feedback, please use: Oracle Dismounter for Linux

     

     

     

    2   Installation

     

    2.1   Downloading

     

    You can download the zip archive containing the systemd oracle-dismounter.service file and oracle-dismounter

    shell script by clicking the following URL: oracle-dismounter download.

     

    If your Linux server has access to the Internet, you can also download the zip archive directly:

     

    [root@localhost ~]# curl --remote-name https://community.oracle.com/servlet/JiveServlet/downloadBody/1031470-102-9-190626/oracle-dismounter-3-1.zip

     

    If downloading the files is not an options, use copy and paste and create the files as outlined in the Files section.

     

     

    2.2   Installing

     

    Enter the following commands as root to install the oracle-dismounter service:

     

    [root@localhost ~]# unzip oracle-dismounter-3-1.zip

    Archive:  oracle-dismounter.zip

      inflating: oracle-dismounter      

      inflating: oracle-dismounter.service 


    [root@localhost ~]# sh oracle-dismounter -i

     

    ’oracle-dismounter.service’ -> /etc/systemd/system/oracle-dismounter.service’

    Service oracle-dismounter enabled successfully.

    Service oracle-dismounter started successfully.

     

     

     

    2.3   Customizing

     

    By default, oracle-dismounter unmounts standard Oracle mount points named /u01, /u02, /u03, and /u04.
    Directory mount points can be specified at the ExecStop= command line in the oracle-dismounter.service

    file or by editing the oracle-dismounter shell script.

     

     

     

    3   Usage

     

    3.1   Log Files

     

    [oracle@localhost ~]$ tail -n24 /opt/oracle/dismounter/oracle-dismounter.log

     

     

     

    3.2   Systemd Journal

     

    You may also review logs using journalctl, provided you have configured a persistent systemd journal log:

     

    [root@localhost ~]# journalctl -u oracle-dismounter

     

     

    3.3   Examples

     

    Without oracle-dismounter installed, unmounting /u01 would have failed.

     

    # journalctl -u oracle-ohasd

    # journalctl -u afd

    # journalctl -u oracle-dismounter

     

     

    4   Uninstalling

     

    To uninstall the service and any related files, open a terminal command prompt and enter the following:

     

    [oracle@localhost ~]$ su - root

    [root@localhost ~]# sh oracle-dismounter -u

     

     

    5   Files

     

    If you cannot download from the Internet, use copy and paste to create the files:

     

    5.1   oracle-dismounter.service

     

    cat > oracle-dismounter.service << 'EOF'
    # File: oracle-dismounter.services

    # Author: Dude! @ Oracle Communities, 2019
    # Version: 3.1
    # Documentation: https://community.oracle.com/docs/DOC-1030979
    #
    # This is the service unit for running oracle-dismounter at system restart
    # or shutdown. It verifies that Oracle filesystems are successfully unmounted.

    [Unit]

    Description=Oracle Dismounter.
    Before=afd.service ohasd.service
    Wants=afd.service ohasd-oracle.service

    [Service]

    Type=oneshot
    RemainAfterExit=true
    ExecStart=-/usr/bin/true
    ExecReload=-/usr/bin/true
    ExecStop=-/opt/oracle/dismounter/oracle-dismounter
    TimeoutStopSec=60min

    [Install]

    WantedBy=multi-user.target graphical.target

    #END
    EOF
    # Press Return to continue!

     

     

    5.2   oracle-dismounter

     

    cat > oracle-dismounter <<'EOF'
    #!/bin/bash
    # File: /opt/oracle/dismounter/oracle-dismounter
    # Version: 3.1
    # Author: Dude! @ Oracle Communities, 2019
    #
    # This script is part of /etc/systemd/system/oracle-dismounter.service
    # and executed during a OS shutdown or restart. It attempts to terminate
    # certain Oracle processes when a regular shutdown of OHASD and unloading
    # of AFD cannot be accomplished. When processes in ora_proc, afd_proc, and
    # clu_proc no longer exist, the script  will unmount named filesystems.
    # The script maintains a logfile in the /opt/oracle/dismounter directory.
    # Please review the following URL for instructions and other info:
    # https://community.oracle.com/docs/DOC-1030979
    #

    ora_mount=( ${@:-/u01 /u02 /u03 /u04} )

    ora_tries=13
    ora_ksig=SIGKILL

    ora_kill=on
    ora_proc=( asm_pmon ora_pmon )

    afd_tries=5
    afd_ksig=SIGKILL
    afd_kill=on
    afd_proc=( afd_log )

    clu_tries=13

    clu_ksig=SIGKILL
    clu_kill=on
    clu_proc=( bin/ohasd bin/ohasd bin/orarootagent bin/oraagent bin/evmd
               bin/evmlogger bin/cssdagent bin/ocssd.bin bin/tnslsnr )

    stop_ohasd=off
    stop_afd=on

    pause=2
    umnt_tries=5

    mn=${0##*/}
    mn_log=$mn.log
    mn_err=$mn.err
    md=/opt/oracle/dismounter
    f_log=$md/$mn_log
    f_err=$md/$mn_err
    hlp="Documentation: https://community.oracle.com/docs/DOC-1030979"

    pgrep=/usr/bin/pgrep
    pkill=/usr/bin/pkill
    mountpoint=/usr/bin/mountpoint
    date=/usr/bin/date
    umount=/usr/bin/umount
    hostname=/usr/bin/hostname
    systemctl=/usr/bin/systemctl
    egrep=/usr/bin/egrep
    cp=/usr/bin/cp
    rm=/usr/bin/rm
    install=/usr/bin/install
    sleep=/usr/bin/sleep

    custom() {
    # Custom function executed after stopping OHASD.
    #
      boo=foo
    }

    install() {
      echo
      if [ -f $mn.service ]; then
        if $systemctl -q is-enabled $mn 2> /dev/null; then
          echo "$mn: Already installed. Try $mn --uninstall."
          echo "$hlp"; stat=1
        else
          $cp -v $mn.service /etc/systemd/system/
          $install $mn -D $md/$mn
          $systemctl -q enable $mn
          $systemctl start $mn
          $systemctl status $mn
          $systemctl daemon-reload; unset stat
        fi
      else
        echo "$mn: File $mn.service missing. Aborted."
        echo "$hlp"; stat=1
      fi
    }

    uninstall() {
      echo
      $systemctl stop $mn && echo "Service $mn stopped successfully."
      $systemctl -q disable $mn && echo "Service $mn disabled successfully."
      $rm -f /etc/systemd/system/$mn.service
      $rm -rf $md
    }

    # Formats for logging and reporting.
    #
    log_report() {
      echo "$($date +'%b %d %T') $($hostname): $*" >> $f_log
      echo "$*"
    }

    err_report() {
      echo "$($date +'%b %d %T') $($hostname): $*" >> $f_err
    }

    proc_chk() {
    # Wait for processes to terminate or until number of tries is up.
    #
      proc_chk=( $1 )
      local i proc_tries=1 proc_maxtries=$2
      while [[ ${#proc_chk[@]} -ne 0 && $proc_tries -le $proc_maxtries ]]; do
        for i in ${proc_chk[@]}; do
          # Remove non-existing processes from proc array.
          $pgrep -f $i > /dev/null 2>&1 || proc_chk=( ${proc_chk[@]/$i} )
        done
        [ ${#proc_chk[@]} -ne 0 ] && $sleep $pause
        (( proc_tries++ ))
      done
    }

    proc_kill() {
    # Kill process as a final resort.
    #
      local proc_kill=( $1 ) i proc_ksig=$2
      for i in ${proc_kill[@]}; do
        $pkill -f --signal $proc_ksig $i > /dev/null 2>&1
        $sleep $pause
      done
    }

    umnt() {
    # Verify and unmount Oracle mountpoints. Remove non-existing or
    # successfully unmounted filesystems from the ora_mount array.
    #
      local i umnt_tries=1 umnt_maxtries=$1
      while [[ ${#ora_mount[@]} -ne 0 && $umnt_tries -le $umnt_maxtries ]]; do
        for i in ${ora_mount[@]}; do
          if $mountpoint "$i" > /dev/null 2>&1; then
            if $umount "$i" > /dev/null 2>&1; then
              ora_mount=( ${ora_mount[@]/$i} )
              log_report "Unmounting of $i successful."
            fi
          else
            ora_mount=( ${ora_mount[@]/$i} )
            log_report "$i not mounted."
          fi
        done
        [ ${#ora_mount[@]} -gt 0 ] && sleep $pause
        (( umnt_tries++ ))
      done
    }

    terminate() {
    # Wait for processes to terminate and use kill as a last resort
    # when specified. Then re-evaluate.
    #
      proc_1=( $1 ) tries_1=$2 kill_1=$3 ksig_1=$4
      if [ ${#proc_1[@]} -ne 0 ]; then
        proc_chk "${proc_1[*]}" $tries_1
        if [ ${#proc_chk[@]} -ne 0 ]; then
          if [ ${kill_1:-off} != off ]; then
            proc_kill "${proc_1[*]}" $ksig_1
            proc_chk "${proc_1[*]}" 1
            if [ ${#proc_chk[@]} -ne 0 ]; then
              log_report "Cannot terminate ${msg:-...}."
              err_report "Processes not terminating: ${proc_chk[*]}."
            else
              log_report "$msg processes killed."
            fi
          fi
        fi
      fi
    }

    ### Main

    # Early exit when the shell is interactive or when there is no system shutdown
    # or restart. Set debug to test the procedure under various scenarios without
    # performing a system restart or shutdown.
    #
    # 1: Debug ON (verbose).
    # a: Don't stop OHASD service.
    # b: Don't stop AFD (ASM Filter Driver).
    # c: Ignore ora_proc
    # d: Ignore afd_proc
    # e: Ignore clu_proc
    # e.g.: export debug=1ab
    #
    if [[ $debug == *1* ]]; then
      if [ $# -ne 0 ]; then
        echo; echo "$mn: Please unset debug and try again. Aborted."
        exit 1
      fi
      set -x
      [[ $debug == *a* ]] && unset stop_ohasd
      [[ $debug == *b* ]] && unset stop_afd
      [[ $debug == *c* ]] && unset ora_proc
      [[ $debug == *d* ]] && unset afd_proc
      [[ $debug == *e* ]] && unset clu_proc
    else
      if [[ -t 0 || -p /dev/stdin ]]; then
        [[ $* == *-i* ]] && { install; exit $stat; }
        [[ $* == *-u* ]] && { uninstall; exit $stat; }
        echo; echo "$mn: Invalid parameter or interactive mode. Aborted."
        echo "$hlp"; exit 1
      else
        jobs=$($systemctl list-jobs)
        [ ! "$($egrep "reboot.target.*start" <<< "$jobs")" ] && exit 0
        [ ! "$($egrep "shutdown.target.*start" <<< "$jobs")" ] && exit 0
      fi
    fi

    # Stop OHAS. Default=off.
    #
    [ ${stop_ohasd:-off} != off ] && /etc/init.d/ohasd stop

    # Execute a custom function.
    custom

    # Wait for processes to terminate and kill as a last resort when this
    # option is enabled. Then re-evaluate if there are still proccess running.

    msg="Oracle ASM and DB"
    terminate "${ora_proc[*]}" $ora_tries $ora_kill $ora_ksig

    msg="OHASD"
    terminate "${clu_proc[*]}" $clu_tries $clu_kill $clu_ksig

    # Stop AFD. Default=on.
    #
    if [ ${stop_afd:-off} != off ]; then
      proc_chk "${afd_proc[*]}" 1
      if [ ${#proc_chk[@]} -ne 0 ]; then
        /etc/init.d/afd stop
        msg="ASM Filter Driver"
        terminate "${afd_proc[*]}" $afd_tries $afd_kill $afd_ksig
      fi
    fi

    # Attempt to unmount ora_mount filesystems.
    #
    umnt $umnt_tries

    # Report mountpoints(s) that could not be unmounted.
    #
    if [ ${#ora_mount[@]} -gt 0 ]; then
      for i in ${ora_mount[@]}; do
        log_report "Failure unmounting ${i}."
      done
    fi

    exit 0
    #END
    EOF
    # Press Return to continue!

     

     

    Good luck!