1 2 Previous Next 20 Replies Latest reply: May 3, 2012 2:27 AM by 903807 RSS

    Oracle service shutdown problem (OEL6.1)

    903807
      Hello everybody, I have following script:

      #! /bin/bash
      # chkconfig: 35 98 02
      ORACLE_HOME=/opt/oracle/product/11.2.0/dbhome_1
      ORACLE=oracle
      PATH=${PATH}:$ORACLE_HOME/bin
      HOST=`hostname`
      PLATFORM=`uname`
      export ORACLE_HOME PATH

      case $1 in
      'start')
      su - $ORACLE -c "$ORACLE_HOME/bin/dbstart $ORACLE_HOME &"
      ;;
      'stop')
      su - $ORACLE -c "$ORACLE_HOME/bin/dbshut $ORACLE_HOME &"
      ;;
      *)
      echo "usage: $0 {start|stop}"
      exit
      ;;
      esac
      exit

      Which I added with chkconfig:

      [root@MIT ~]# chkconfig | grep dbora
      dbora      0:off     1:off     2:off     3:on     4:off     5:on     6:off

      It can be manually called and works with both start/stop options and also runs automatically on startup but not on shutdown (I can say for sure by logs it creates). Does anybody know why? Did I misconfigured something?

      And why it is not part of the list of services I can see on every start or shutdown with little green [OK] message confirming status? (not being run as one of build-in services => not in the list with smartd, sshd, etc..) I want it to be started/stopped BEFORE proceeding to the next service, I guess it has to be done in a different way, right?

      Edited by: Ren on Apr 26, 2012 2:08 PM
        • 1. Re: Oracle service shutdown problem (OEL6.1)
          Tommyreynolds-Oracle
          http://tldp.org/HOWTO/Oracle-9i-Fedora-3-Install-HOWTO/sect_08.html
          • 2. Re: Oracle service shutdown problem (OEL6.1)
            Catch-22
            When a service is started through an init script, a file is touched in the /var/lock/subsys/ directory with the same name as the init script. When the service is stopped, this file is removed. However, when switching runlevels, the rc scripts check for the existence of the file in /var/lock/subsys/. If this file is not found, the service will not start or stop between runlevels, even if there are start and kill symbolic links in the /etc/rc#.d/ directories.
            • 3. Re: Oracle service shutdown problem (OEL6.1)
              903807
              Hello Tommy and thanks but as I described I already did these steps, my problem comes a bit later. Manual execution (/etc/init.d/dbora start (or stop)) work as expected but after reboot of the box I don't see shutdown in the alert.log or $ORACLE_HOME/shutdown.log. Startup of the box brings up the Oracle service but after complete startup, when login prompt is ready; boot process doesn't wait for instance to start as it does with other services/daemons.
              • 4. Re: Oracle service shutdown problem (OEL6.1)
                903807
                Aha! Thanks Dude.:-) I'll add it to my script and will try again.
                • 5. Re: Oracle service shutdown problem (OEL6.1)
                  Catch-22
                  Please see my previous reply to find out what is missing in your script.
                  • 6. Re: Oracle service shutdown problem (OEL6.1)
                    903807
                    Thanks Dude, I didn't know about /var/lock/subsystem feature, it really does what I was looking for and I learned another bit.

                    But I still have some weird troubles. Never met such behavior before but now it goes like this:
                    Startup/shutdown of the box properly controls startup/shutdown of instance as expected. Root runs dbora, starts instance as oracle user, I can see instance processes owned by oracle as both root and oracle but oracle sees the instance as down and when I try to start it up, it fails 'cannot mount in EXCLUSIVE'. Of course it can't, because it already runs. With root I can normally login to sqlplus and it behaves properly. How is it possible oracle user doesn't know about instance running under his username? Just rough directions would be enough, if anybody has an idea for the cause of this. Thanks.
                    • 7. Re: Oracle service shutdown problem (OEL6.1)
                      Catch-22
                      With root I can normally login to sqlplus and it behaves properly.
                      Root shouldn't be able to run sqlplus and login as sysdba. What is the output of the following command:

                      <pre>
                      su - root
                      id
                      su - oracle
                      id
                      </pre>
                      I can see instance processes owned by oracle as both root and oracle
                      No oracle process should be owned by root. Can you show what you see?
                      • 8. Re: Oracle service shutdown problem (OEL6.1)
                        903807
                        I'm sorry, I see I gave a really poor description of the issue. Here's the situation. After start of the box db is brought up with init script.
                        While as root I can login to the db as 'sys as sysdba' and everything is ok, I can also see processes owned by oracle user:

                        [root@MIT ~]# export ORACLE_HOME=/opt/oracle/product/11.2.0/dbhome_1
                        [root@MIT ~]# export ORACLE_SID=EMREP
                        [root@MIT ~]# /opt/oracle/product/11.2.0/dbhome_1/bin/sqlplus

                        SQL*Plus: Release 11.2.0.3.0 Production on Mon Apr 30 10:35:16 2012

                        Copyright (c) 1982, 2011, Oracle. All rights reserved.

                        Enter user-name: sys as sysdba
                        Enter password:

                        Connected to:
                        Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
                        With the Partitioning option

                        SQL> exit
                        Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
                        With the Partitioning option
                        [root@MIT ~]# id
                        uid=0(root) gid=0(root) groups=0(root),1(bin),2(daemon),3(sys),4(adm),6(disk),10(wheel) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
                        [root@MIT ~]# ps -ef | grep oracle
                        oracle 1363 1 0 09:09 ? 00:00:04 /opt/oracle/product/11.2.0/dbhome_1/bin/tnslsnr LISTENER -inherit
                        oracle 1502 1 0 09:09 ? 00:00:07 ora_pmon_EMREP
                        oracle 1504 1 0 09:09 ? 00:00:17 ora_psp0_EMREP
                        oracle 1506 1 4 09:09 ? 00:03:33 ora_vktm_EMREP
                        oracle 1510 1 0 09:09 ? 00:00:03 ora_gen0_EMREP
                        oracle 1512 1 0 09:09 ? 00:00:03 ora_diag_EMREP
                        oracle 1514 1 0 09:09 ? 00:00:04 ora_dbrm_EMREP
                        oracle 1516 1 0 09:09 ? 00:00:11 ora_dia0_EMREP
                        oracle 1518 1 0 09:09 ? 00:00:05 ora_mman_EMREP
                        oracle 1520 1 0 09:09 ? 00:00:04 ora_dbw0_EMREP
                        oracle 1522 1 0 09:09 ? 00:00:04 ora_lgwr_EMREP
                        oracle 1524 1 0 09:09 ? 00:00:09 ora_ckpt_EMREP
                        oracle 1526 1 0 09:09 ? 00:00:02 ora_smon_EMREP
                        oracle 1528 1 0 09:09 ? 00:00:01 ora_reco_EMREP
                        oracle 1530 1 0 09:09 ? 00:00:09 ora_mmon_EMREP
                        oracle 1532 1 0 09:09 ? 00:00:13 ora_mmnl_EMREP
                        oracle 1632 1 0 09:09 ? 00:00:01 ora_arc0_EMREP
                        oracle 1634 1 0 09:09 ? 00:00:01 ora_arc1_EMREP
                        oracle 1636 1 0 09:09 ? 00:00:02 ora_arc2_EMREP
                        oracle 1638 1 0 09:09 ? 00:00:01 ora_arc3_EMREP
                        oracle 1640 1 0 09:09 ? 00:00:02 ora_qmnc_EMREP
                        oracle 1760 1 0 09:09 ? 00:00:01 ora_q000_EMREP
                        oracle 1762 1 0 09:09 ? 00:00:02 ora_q001_EMREP
                        oracle 1769 1 0 09:19 ? 00:00:03 ora_smco_EMREP
                        oracle 1886 1 0 10:19 ? 00:00:00 ora_w000_EMREP
                        root 1909 1785 0 10:35 pts/0 00:00:00 grep oracle

                        As oracle I can still see the processes owned by me but login /as sysdba (or sys as ssydba) results in 'idle instance'.

                        [root@MIT ~]# su - oracle
                        [oracle@MIT ~]$ sqlplus

                        SQL*Plus: Release 11.2.0.3.0 Production on Mon Apr 30 10:35:59 2012

                        Copyright (c) 1982, 2011, Oracle. All rights reserved.

                        Enter user-name: /as sysdba
                        Connected to an idle instance.

                        SQL> exit
                        Disconnected
                        [oracle@MIT ~]$ ps -ef | grep oracle
                        oracle 1363 1 0 09:09 ? 00:00:04 /opt/oracle/product/11.2.0/dbhome_1/bin/tnslsnr LISTENER -inherit
                        oracle 1502 1 0 09:09 ? 00:00:07 ora_pmon_EMREP
                        oracle 1504 1 0 09:09 ? 00:00:17 ora_psp0_EMREP
                        oracle 1506 1 4 09:09 ? 00:03:40 ora_vktm_EMREP
                        oracle 1510 1 0 09:09 ? 00:00:03 ora_gen0_EMREP
                        oracle 1512 1 0 09:09 ? 00:00:03 ora_diag_EMREP
                        oracle 1514 1 0 09:09 ? 00:00:04 ora_dbrm_EMREP
                        oracle 1516 1 0 09:09 ? 00:00:12 ora_dia0_EMREP
                        oracle 1518 1 0 09:09 ? 00:00:05 ora_mman_EMREP
                        oracle 1520 1 0 09:09 ? 00:00:04 ora_dbw0_EMREP
                        oracle 1522 1 0 09:09 ? 00:00:04 ora_lgwr_EMREP
                        oracle 1524 1 0 09:09 ? 00:00:09 ora_ckpt_EMREP
                        oracle 1526 1 0 09:09 ? 00:00:02 ora_smon_EMREP
                        oracle 1528 1 0 09:09 ? 00:00:01 ora_reco_EMREP
                        oracle 1530 1 0 09:09 ? 00:00:10 ora_mmon_EMREP
                        oracle 1532 1 0 09:09 ? 00:00:13 ora_mmnl_EMREP
                        oracle 1632 1 0 09:09 ? 00:00:02 ora_arc0_EMREP
                        oracle 1634 1 0 09:09 ? 00:00:02 ora_arc1_EMREP
                        oracle 1636 1 0 09:09 ? 00:00:02 ora_arc2_EMREP
                        oracle 1638 1 0 09:09 ? 00:00:01 ora_arc3_EMREP
                        oracle 1640 1 0 09:09 ? 00:00:02 ora_qmnc_EMREP
                        oracle 1760 1 0 09:09 ? 00:00:01 ora_q000_EMREP
                        oracle 1762 1 0 09:09 ? 00:00:02 ora_q001_EMREP
                        oracle 1769 1 0 09:19 ? 00:00:03 ora_smco_EMREP
                        oracle 1886 1 0 10:19 ? 00:00:00 ora_w000_EMREP
                        root 1910 1785 0 10:35 pts/0 00:00:00 su - oracle
                        oracle 1912 1910 0 10:35 pts/0 00:00:00 -bash
                        oracle 1931 1912 0 10:38 pts/0 00:00:00 ps -ef
                        oracle 1932 1912 0 10:38 pts/0 00:00:00 grep oracle
                        [oracle@MIT ~]$ id
                        uid=201(oracle) gid=201(dba) groups=201(dba) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023

                        Orcle user don't see it's own processes and I can't figure out why. When I try to start db from oracle's sqlplus, it of course fails 'mount exclusive' because db actually runs already. Result: doubled instance processes, all owned by oracle:

                        [oracle@MIT ~]$ sqlplus

                        SQL*Plus: Release 11.2.0.3.0 Production on Mon Apr 30 11:05:17 2012

                        Copyright (c) 1982, 2011, Oracle. All rights reserved.

                        Enter user-name: sys as sysdba
                        Enter password:
                        Connected to an idle instance.

                        SQL> startup
                        ORACLE instance started.

                        Total System Global Area 997953536 bytes
                        Fixed Size          2234520 bytes
                        Variable Size          578815848 bytes
                        Database Buffers     402653184 bytes
                        Redo Buffers          14249984 bytes
                        ORA-01102: cannot mount database in EXCLUSIVE mode


                        SQL> exit
                        Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
                        With the Partitioning option
                        [oracle@MIT ~]$ ps -ef | grep oracle
                        oracle 1363 1 0 09:09 ? 00:00:05 /opt/oracle/product/11.2.0/dbhome_1/bin/tnslsnr LISTENER -inherit
                        oracle 1502 1 0 09:09 ? 00:00:09 ora_pmon_EMREP
                        oracle 1504 1 0 09:09 ? 00:00:24 ora_psp0_EMREP
                        oracle 1506 1 4 09:09 ? 00:04:52 ora_vktm_EMREP
                        oracle 1510 1 0 09:09 ? 00:00:05 ora_gen0_EMREP
                        oracle 1512 1 0 09:09 ? 00:00:04 ora_diag_EMREP
                        oracle 1514 1 0 09:09 ? 00:00:05 ora_dbrm_EMREP
                        oracle 1516 1 0 09:09 ? 00:00:16 ora_dia0_EMREP
                        oracle 1518 1 0 09:09 ? 00:00:06 ora_mman_EMREP
                        oracle 1520 1 0 09:09 ? 00:00:06 ora_dbw0_EMREP
                        oracle 1522 1 0 09:09 ? 00:00:06 ora_lgwr_EMREP
                        oracle 1524 1 0 09:09 ? 00:00:12 ora_ckpt_EMREP
                        oracle 1526 1 0 09:09 ? 00:00:03 ora_smon_EMREP
                        oracle 1528 1 0 09:09 ? 00:00:02 ora_reco_EMREP
                        oracle 1530 1 0 09:09 ? 00:00:13 ora_mmon_EMREP
                        oracle 1532 1 0 09:09 ? 00:00:17 ora_mmnl_EMREP
                        oracle 1632 1 0 09:09 ? 00:00:02 ora_arc0_EMREP
                        oracle 1634 1 0 09:09 ? 00:00:02 ora_arc1_EMREP
                        oracle 1636 1 0 09:09 ? 00:00:02 ora_arc2_EMREP
                        oracle 1638 1 0 09:09 ? 00:00:02 ora_arc3_EMREP
                        oracle 1640 1 0 09:09 ? 00:00:03 ora_qmnc_EMREP
                        oracle 1760 1 0 09:09 ? 00:00:02 ora_q000_EMREP
                        oracle 1762 1 0 09:09 ? 00:00:02 ora_q001_EMREP
                        oracle 1769 1 0 09:19 ? 00:00:04 ora_smco_EMREP
                        oracle 1886 1 0 10:19 ? 00:00:01 ora_w000_EMREP
                        root 1910 1785 0 10:35 pts/0 00:00:00 su - oracle
                        oracle 1912 1910 0 10:35 pts/0 00:00:00 -bash
                        oracle 2200 1 0 11:05 ? 00:00:00 ora_w001_EMREP
                        oracle 2250 1 0 11:07 ? 00:00:00 ora_pmon_EMREP
                        oracle 2252 1 0 11:07 ? 00:00:00 ora_psp0_EMREP
                        oracle 2254 1 3 11:07 ? 00:00:01 ora_vktm_EMREP
                        oracle 2258 1 0 11:07 ? 00:00:00 ora_gen0_EMREP
                        oracle 2260 1 0 11:07 ? 00:00:00 ora_diag_EMREP
                        oracle 2262 1 0 11:07 ? 00:00:00 ora_dbrm_EMREP
                        oracle 2264 1 0 11:07 ? 00:00:00 ora_dia0_EMREP
                        oracle 2266 1 4 11:07 ? 00:00:01 ora_mman_EMREP
                        oracle 2268 1 0 11:07 ? 00:00:00 ora_dbw0_EMREP
                        oracle 2270 1 0 11:07 ? 00:00:00 ora_lgwr_EMREP
                        oracle 2272 1 0 11:07 ? 00:00:00 ora_ckpt_EMREP
                        oracle 2274 1 0 11:07 ? 00:00:00 ora_smon_EMREP
                        oracle 2276 1 0 11:07 ? 00:00:00 ora_reco_EMREP
                        oracle 2278 1 0 11:07 ? 00:00:00 ora_mmon_EMREP
                        oracle 2280 1 0 11:07 ? 00:00:00 ora_mmnl_EMREP
                        oracle 2378 1912 3 11:08 pts/0 00:00:00 ps -ef
                        oracle 2379 1912 0 11:08 pts/0 00:00:00 grep oracle
                        • 9. Re: Oracle service shutdown problem (OEL6.1)
                          Catch-22
                          I suggest to check the instance alert log. Verify you have the the Oracle shell environment variables of the "oracle" user account setup properly. Check the file/directory permissions of your database.
                          context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
                          Apparently you have SELinux turned on (/etc/selinux/config). Turn it off (disabled), restart the system and try again.
                          • 10. Re: Oracle service shutdown problem (OEL6.1)
                            903807
                            Nothing wrong in alert.log, I can see proper records for every startup/shutdown.

                            [oracle@MIT ~]$ set | grep oracle
                            ORACLE_BASE=/opt/oracle/
                            ORACLE_DIAG=/var/opt/oracle/diag/
                            ORACLE_HOME=/opt/oracle/product/11.2.0/dbhome_1/

                            No problem with file permissions:

                            [oracle@MIT ~]$ ls -lHA /var/opt/
                            total 4
                            drwxr-xr-x 6 oracle dba 4096 Apr 27 10:07 oracle
                            [oracle@MIT ~]$ ls -lHA /var/opt/oracle/
                            total 28
                            drwxrwxr-x. 4 oracle dba 4096 Apr 27 11:14 diag
                            drwxr-x---. 2 oracle dba 4096 Apr 27 10:08 EMREP
                            drwxr-x---. 3 oracle dba 4096 Apr 27 10:07 fra
                            drwx------ 2 root root 16384 Apr 26 15:36 lost+found
                            [oracle@MIT ~]$ ls -lHA /var/opt/oracle/EMREP/
                            total 1625312
                            -rw-r-----. 1 oracle dba 9748480 Apr 30 14:03 control01.ctl
                            -rw-r-----. 1 oracle dba 314573312 Apr 30 14:00 redo01.log
                            -rw-r-----. 1 oracle dba 314573312 Apr 30 14:00 redo02.log
                            -rw-r-----. 1 oracle dba 314573312 Apr 30 14:00 redo03.log
                            -rw-r-----. 1 oracle dba 209723392 Apr 30 14:00 sysaux01.dbf
                            -rw-r-----. 1 oracle dba 283123712 Apr 30 14:00 system01.dbf
                            -rw-r-----. 1 oracle dba 20979712 Apr 30 09:09 temp01.dbf
                            -rw-r-----. 1 oracle dba 209723392 Apr 30 14:00 undotbs01.dbf
                            -rw-r-----. 1 oracle dba 5251072 Apr 30 14:00 users01.dbf

                            SELinux is disabled:

                            [oracle@MIT ~]$ id
                            uid=201(oracle) gid=201(dba) groups=201(dba)

                            There is this weird "_" environment variable (I have no idea where it came from), the rest is ok:

                            [root@MIT ~]# set | grep ORACLE
                            [root@MIT ~]# su - oracle
                            [oracle@MIT ~]$ set | grep ORACLE
                            ORACLE_BASE=/opt/oracle/
                            ORACLE_DIAG=/var/opt/oracle/diag/
                            ORACLE_HOME=/opt/oracle/product/11.2.0/dbhome_1/
                            ORACLE_SID=EMREP
                            _=ORACLE_DIAG

                            I just found out I can login as oracle user to the instance through listener, username@dbname. When I do /as sysdba it is still the same, idle instance.
                            • 11. Re: Oracle service shutdown problem (OEL6.1)
                              Catch-22
                              The $_ underscore variable is special variable set to the argument of the previous command executed.

                              An instance can mount any database, but it can only open a single database at any time.

                              I slightly remember a similar problem like you described and I think the issue was a mismatch between the database name and instance. The oracle SID for example is case sensitive. I suggest you check your db_name, instance name and oracle SID and make sure you have the instance parameter and listener configuration files set accordingly.

                              I'm afraid further analysis will require a complete check of your database installation, which is off the forum topic.
                              • 12. Re: Oracle service shutdown problem (OEL6.1)
                                903807
                                All parameters are fine, same in the config files.
                                SQL> sho parameter name

                                NAME                    TYPE     VALUE
                                ------------------------------------ ----------- ------------------------------
                                db_file_name_convert          string
                                db_name                string     EMREP
                                db_unique_name               string     EMREP
                                global_names               boolean     FALSE
                                instance_name               string     EMREP
                                lock_name_space           string
                                log_file_name_convert          string
                                processor_group_name          string
                                service_names               string     EMREP.localdomain


                                I am DBA for a couple of years but this is my first real exposure to Linux administration, so I am kinda on a thin ice. It seems the problem goes in a different direction:

                                [oracle@MIT admin]$ ps -ef | grep LIST
                                oracle 1378 1 0 07:23 ? 00:00:07 /opt/oracle/product/11.2.0/dbhome_1/bin/tnslsnr LISTENER -inherit
                                oracle 2312 2074 0 10:39 pts/0 00:00:00 grep LIST

                                [oracle@MIT admin]$ lsnrctl status

                                LSNRCTL for Linux: Version 11.2.0.3.0 - Production on 02-MAY-2012 10:31:40

                                Copyright (c) 1991, 2011, Oracle. All rights reserved.

                                Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=MIT.localdomain)(PORT=1521)))
                                TNS-12541: TNS:no listener
                                TNS-12560: TNS:protocol adapter error
                                TNS-00511: No listener
                                Linux Error: 111: Connection refused
                                [oracle@MIT admin]$ lsnrctl start

                                LSNRCTL for Linux: Version 11.2.0.3.0 - Production on 02-MAY-2012 10:31:54

                                Copyright (c) 1991, 2011, Oracle. All rights reserved.

                                Starting /opt/oracle/product/11.2.0/dbhome_1//bin/tnslsnr: please wait...

                                TNSLSNR for Linux: Version 11.2.0.3.0 - Production
                                System parameter file is /opt/oracle/product/11.2.0/dbhome_1/network/admin/listener.ora
                                Log messages written to /var/opt/oracle/diag/tnslsnr/MIT/listener/alert/log.xml
                                Error listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=MIT.localdomain)(PORT=1521)))
                                TNS-12542: TNS:address already in use
                                TNS-12560: TNS:protocol adapter error
                                TNS-00512: Address already in use
                                Linux Error: 98: Address already in use

                                Listener failed to start. See the error message(s) above...


                                Oracle user once again don't see it's own process, started by init script. What conditions can cause it, any ideas?
                                • 13. Re: Oracle service shutdown problem (OEL6.1)
                                  Catch-22
                                  What happens when you start the listener and oracle database manually using the Oracle account. Does it give the same results?
                                  • 14. Re: Oracle service shutdown problem (OEL6.1)
                                    903807
                                    No. It works as expected.
                                    As root I stopped db with /etc/init.d/dbora stop, sued to oracle account and cleanely and nicely started instance. It seems there's a problem in the script or inner mechanism for sharing processes. OEL installation is non-default, it is hardened by company's rules. Wild guess but there might be some setting interfering.
                                    1 2 Previous Next