12 Replies Latest reply on Feb 10, 2012 12:04 PM by Shiyer-Oracle

    crsctl apache registration & start problem

    user1994048
      i have 2 nodes ( running OEL 5.5), and shared disk area iam using openfilter for that,

      i installed oracle grid, and i want to register my httpd daemon to cluster, each node has httpd,,

      the problem is i can 't start apachephp service..

      Question 1) Why i get these errors? i tried to change PLACEMENT to restrict but it didnt work,,

      [root@host1-pub ~]# /u01/11.2.0/grid/bin/crsctl start resource apachephp
      CRS-2672: Attempting to start 'apachephp' on 'host2-pub'
      CRS-2674: Start of 'apachephp' on 'host2-pub' failed
      CRS-2563: Attempt to start resource 'apachephp' on 'host2-pub' has failed. Will re-retry on 'host1-pub' now.
      CRS-2672: Attempting to start 'apachephp' on 'host1-pub'
      CRS-2674: Start of 'apachephp' on 'host1-pub' failed
      CRS-2632: There are no more servers to try to place resource 'apachephp' on that would satisfy its placement policy
      CRS-4000: Command Start failed, or completed with errors.

      details shown as below...



      resource add command for registration:
      /u01/11.2.0/grid/bin/crsctl add resource myApache -type cluster_resource -attr "ACTION_SCRIPT=/u01/AppBase/CRS_SCRIPTS/httpd.scr, PLACEMENT=restricted, SERVER_POOLS=myprojectpool,CHECK_INTERVAL=30,RESTART_ATTEMPTS=2,START_DEPENDENCIES=hard(appsvip),STOP_DEPENDENCIES=hard(appsvip)"

      if i try to run this script manually httpd starts and stops correctly but when tried to start it with crsctl command it fails, i configured all required settings for both machine (node1 and node2),, i have httpd.sh script on both machines too.


      *[root@host1-pub CRS_SCRIPTS]# cat httpd.sh*
      #!/bin/sh

      HTTPDCONFLOCATION=/etc/httpd/conf/httpd.conf
      WEBPAGECHECK=http://10.0.20.224:80/icons/apache_pb.gif

      case $1 in
      'start')
      /usr/bin/sudo /usr/sbin/apachectl -k start -f $HTTPDCONFLOCATION
      /bin/sleep 10
      RET=$?
      ;;
      'stop')
      /usr/bin/sudo /usr/sbin/apachectl -k stop
      RET=$?
      ;;
      'clean')
      /usr/bin/sudo /usr/sbin/apachectl -k stop
      RET=$?
      ;;
      'check')
      /usr/bin/sudo /usr/bin/wget -q --delete-after $WEBPAGECHECK
      RET=$?
      ;;
      *)
      RET=0
      ;;
      esac
      # 0: success; 1 : error
      if [ $RET -eq 0 ]; then
      exit 0
      else
      exit 1
      fi
        • 1. Re: crsctl apache registration & start problem
          user1994048
          DETAILED LOGS:

          2012-02-07 17:51:18.030: [  CRSCCL][1482291520]clscsend completed:msgTag= 0xcccccccc version= 0 msgType= 0 msgId= 266 msglen = 4568 clschdr.size_clscmsgh= 4656 src= (1, 4294264730) dest= (2, 4294200200)
          2012-02-07 17:51:27.674: [ CRSCOMM][1501186368] clsIpc: Adding msg (0x41253c0) to peer: 2
          2012-02-07 17:51:27.674: [ CRSCOMM][1494882624][F-ALGO] clsIpc: sendWork msg to peer 2
          2012-02-07 17:51:27.674: [ CRSCOMM][1494882624] IPCL:FC Looking for connection to member 28
          2012-02-07 17:51:27.675: [ CRSCOMM][1494882624] clsIpc: Sent msg: 0x41253c0 to member 2
          2012-02-07 17:51:27.675: [ CRSCOMM][1494882624][F-ALGO] clsIpc: sendWork waiting for msgs to send
          2012-02-07 17:51:27.677: [ CRSCOMM][1496983872] IPCL:FC Looking for connection to member 28
          2012-02-07 17:51:27.677: [ CRSCOMM][1496983872][F-ALGO] Found msgtype 0
          2012-02-07 17:51:27.677: [ CRSCOMM][1496983872][F-ALGO] MemberChangeData recd
          2012-02-07 17:51:28.215: [  CRSCCL][1471801664]clscreceive:msgTag= 0xcccccccc version= 0 msgType= 0 msgId= 311 msglen = 1814 clschdr.size_clscmsgh= 1902 src= (2, 4294200200) dest= (1, 4294264730)
          2012-02-07 17:51:28.216: [   CRSPE][1513793856] Received reply to action [Start] message ID: 44886
          2012-02-07 17:51:28.266: [  CRSCCL][1471801664]clscreceive:msgTag= 0xcccccccc version= 0 msgType= 0 msgId= 312 msglen = 1865 clschdr.size_clscmsgh= 1953 src= (2, 4294200200) dest= (1, 4294264730)
          2012-02-07 17:51:28.267: [   CRSPE][1513793856] Received reply to action [Start] message ID: 44886
          2012-02-07 17:51:28.267: [   CRSPE][1513793856] CRS-2674: Start of 'myApache' on 'host2-pub' failed

          2012-02-07 17:51:28.268: [UiServer][1517996352] Container [ Name: ORDER
          MESSAGE:
          TextMessage[CRS-2674: Start of 'myApache' on 'host2-pub' failed]
          MSGTYPE:
          TextMessage[1]
          OBJID:
          TextMessage[myApache]
          WAIT:
          TextMessage[0]
          ]
          2012-02-07 17:51:28.268: [   CRSPE][1513793856] Sequencer for [myApache 1 1] has completed with error: CRS-0215: Could not start resource 'myApache'.

          2012-02-07 17:51:28.268: [   CRSPE][1513793856] Sending message to agfw: id = 44900
          2012-02-07 17:51:28.269: [   CRSPE][1513793856] CRS-2673: Attempting to stop 'appsvip' on 'host2-pub'

          2012-02-07 17:51:28.271: [UiServer][1517996352] Container [ Name: ORDER
          MESSAGE:
          TextMessage[CRS-2673: Attempting to stop 'appsvip' on 'host2-pub']
          MSGTYPE:
          TextMessage[3]
          OBJID:
          • 2. Re: crsctl apache registration & start problem
            Shiyer-Oracle
            Hi,

            Can you re-confirm your resource name : 'apachephp' or 'myApache' and script file : httpd.sh or httpd.scr? Can you post the scriptagent logfile extract for the time period when the START was issued?

            Thanks.
            • 3. Re: crsctl apache registration & start problem
              user1994048
              ==============================
              ==== create pool ====
              ==============================
              /u01/11.2.0/grid/bin/crsctl add serverpool myprojectpool -attr "PARENT_POOLS=Generic, SERVER_NAMES=host1-pub host2-pub"


              ==============================
              ==== create virtual ip ====
              ==============================
              /u01/11.2.0/grid/bin/appvipcfg create -network=1 -ip=10.0.20.224 -vipname=appsvip -user=root


              ====================================
              ==== register app ====
              ====================================
              /u01/11.2.0/grid/bin/crsctl add resource myApache -type cluster_resource -attr "ACTION_SCRIPT=/u01/AppBase/CRS_SCRIPTS/httpd.scr, PLACEMENT=restricted, SERVER_POOLS=myprojectpool,CHECK_INTERVAL=30,RESTART_ATTEMPTS=2,START_DEPENDENCIES=hard(appsvip),STOP_DEPENDENCIES=hard(appsvip)"






              [root@host1-pub bin]# cat /u01/AppBase/CRS_SCRIPTS/httpd.scr
              #!/bin/sh

              HTTPDCONFLOCATION=/etc/httpd/conf/httpd.conf
              WEBPAGECHECK=http://10.0.20.224:80/icons/apache_pb.gif # 10.0.20.224 is vip that i created

              case $1 in
              'start')
              /usr/bin/sudo /usr/sbin/apachectl -k start -f $HTTPDCONFLOCATION
              /bin/sleep 10
              RET=$?
              ;;
              'stop')
              /usr/bin/sudo /usr/sbin/apachectl -k stop
              RET=$?
              ;;
              'clean')
              /usr/bin/sudo /usr/sbin/apachectl -k stop
              RET=$?
              ;;
              'check')
              /usr/bin/sudo /usr/bin/wget -q --delete-after $WEBPAGECHECK
              RET=$?
              ;;
              *)
              RET=0
              ;;
              esac
              # 0: success; 1 : error
              if [ $RET -eq 0 ]; then
              exit 0
              else
              exit 1
              fi






              [root@host1-pub bin]# ./crsctl start resource myApache
              CRS-2672: Attempting to start 'appsvip' on 'host1-pub'
              CRS-2676: Start of 'appsvip' on 'host1-pub' succeeded
              CRS-2672: Attempting to start 'myApache' on 'host1-pub'
              CRS-2674: Start of 'myApache' on 'host1-pub' failed
              CRS-2673: Attempting to stop 'appsvip' on 'host1-pub'
              CRS-2677: Stop of 'appsvip' on 'host1-pub' succeeded
              CRS-2563: Attempt to start resource 'myApache' on 'host1-pub' has failed. Will re-retry on 'host2-pub' now.
              CRS-2672: Attempting to start 'appsvip' on 'host2-pub'
              CRS-2676: Start of 'appsvip' on 'host2-pub' succeeded
              CRS-2672: Attempting to start 'myApache' on 'host2-pub'
              CRS-2674: Start of 'myApache' on 'host2-pub' failed
              CRS-2673: Attempting to stop 'appsvip' on 'host2-pub'
              CRS-2677: Stop of 'appsvip' on 'host2-pub' succeeded
              CRS-2632: There are no more servers to try to place resource 'myApache' on that would satisfy its placement policy
              CRS-4000: Command Start failed, or completed with errors.





              Error log (little big=) ): so pasted it to pastebin

              http://pastebin.com/SeG9sRm6
              • 4. Re: crsctl apache registration & start problem
                Shiyer-Oracle
                Can you post the scriptagent logfile? It will be in directory 'log/<hostname>/agent/crsd/scriptagent/'.

                Thanks.
                • 5. Re: crsctl apache registration & start problem
                  user1994048
                  [root@host1-pub crsd]# pwd
                  /u01/11.2.0/grid/log/host1-pub/agent/crsd

                  [root@host1-pub crsd]# ls
                  application_oracle cluster_resource_root local_resource_root ora_oc4j_type_grid
                  cluster_resource_oracle local_resource_oracle oraagent_grid orarootagent_root



                  how can i activate scriptagent logs?

                  Edited by: user1994048 on 09.Şub.2012 01:05
                  • 6. Re: crsctl apache registration & start problem
                    Shiyer-Oracle
                    Can you post the Grid Infrastructure version? Can you grep for messages with 'myApache' in the logfiles in subdirectories: cluster_resource_root and cluster_resource_oracle?

                    Thanks.
                    • 7. Re: crsctl apache registration & start problem
                      user1994048
                      linux.x64_11gR2_grid.zip ( version: 11.2.0.1 )

                      cluster_resource_root logs: http://pastebin.com/FFwXx3R4

                      cluster_resource_oracle logs: there is no log for myApache

                      Edited by: user1994048 on 09.Şub.2012 02:21
                      • 8. Re: crsctl apache registration & start problem
                        Shiyer-Oracle
                        There is some problem when executing the script :-

                        012-02-07 16:37:05.420: [myApache][1306265920] [start] Executing action script: /u01/AppBase/CRS_SCRIPTS/httpd.scr[start]
                        2012-02-07 16:37:05.421: [myApache][1306265920] [start] Could not execute script: /u01/AppBase/CRS_SCRIPTS/httpd.scr
                        2012-02-07 16:37:05.421: [    AGFW][1306265920] Command: start for resource: myApache 1 1 completed with invalid status: 3

                        Please verify execute permissions etc and also try to run the script manually.

                        There will be some output in the logfile between these 2 lines, please post that also :-

                        2012-02-07 16:37:05.421: [myApache][1306265920] [start] Could not execute script: /u01/AppBase/CRS_SCRIPTS/httpd.scr
                        2012-02-07 16:37:05.421: [    AGFW][1306265920] Command: start for resource: myApache 1 1 completed with invalid status: 3

                        .
                        • 9. Re: crsctl apache registration & start problem
                          user1994048
                          httpd.scr is working,

                          now the problem is about oid.scr

                          *: [myOid][1276234048] [clean] /u01/AppBase/Oracle_IDM1/opmn/bin/opmn: Error: LD_ASSUME_KERNEL is set* # <--------------------- this error
                          2012-02-09 16:31:56.258: [myOid][1276234048] [clean] opmnctl stopall: opmn is not running.
                          2012-02-09 16:31:56.258: [    AGFW][1276234048] Command: clean for resource: myOid 1 1 completed with status: SUCCESS
                          2012-02-09 16:31:56.260: [    AGFW][1309792576] Executing command: check for resource: myOid 1 1
                          2012-02-09 16:31:56.260: [    AGFW][1309792576] Entering script entry point...

                          Solution:
                          1)
                          my first problem was about shell script syntax error and check case RET definations,, check case is very important if you dont cant produce correct return exit code values,, crsctl will fail.

                          2)
                          if LD_ASSUME_KERNEL enviroment variable is defined before you run opmnctl script, opmnctl startup will fail with that error.. i added
                          unset LD_ASSUME_KERNEL line to the beginning of OID control script,, after that crsctl calls worked correctly


                          thank for answers,
                          best regards
                          • 10. Re: crsctl apache registration & start problem
                            Shiyer-Oracle
                            Is your apache resource working properly now? Can you post what was the error ?

                            I am not sure about OID. Is this Oracle LDAP server? How does the oid.scr script look like?

                            .
                            • 11. Re: crsctl apache registration & start problem
                              user1994048
                              i don't have any error now, everything is working well,,

                              httpd.scr:_
                              #!/bin/sh
                              # Oracle Clusterware Apache Registration Script
                              # Grid Version: 11.2.0.1 Platform: Oracle Enterprise Linux 5.5, Service: HTTPD
                              # Created by Ali Okan Yuksel - aokany[at]gmail[dot]com - www.siyahsapka.org
                              # Registration command:
                              # /u01/11.2.0/grid/bin/appvipcfg create -network=1 -ip=10.0.20.224 -vipname=appsvip -user=root#
                              # /u01/11.2.0/grid/bin/crsctl add resource myApache -type cluster_resource -attr "ACTION_SCRIPT=/u01/AppBase/CRS_SCRIPTS/httpd.scr, PLACEMENT=restricted, SERVER_POOLS=projectpool,CHECK_INTERVAL=30,RESTART_ATTEMPTS=2,START_DEPENDENCIES=hard(appsvip),STOP_DEPENDENCIES=hard(appsvip)"
                              #


                              . /etc/rc.d/init.d/functions

                              # check for 1.3 configuration
                              check13 () {
                              CONFFILE=/etc/httpd/conf/httpd.conf
                              GONE="(ServerType|BindAddress|Port|AddModule|ClearModuleList|"
                              GONE="${GONE}AgentLog|RefererLog|RefererIgnore|FancyIndexing|"
                              GONE="${GONE}AccessConfig|ResourceConfig)"
                              if LANG=C grep -Eiq "^[[:space:]]*($GONE)" $CONFFILE; then
                              echo
                              echo 1>&2 " Apache 1.3 configuration directives found"
                              echo 1>&2 " please read /usr/share/doc/httpd-2.2.3/migration.html"
                              failure "Apache 1.3 config directives test"
                              echo
                              exit 1
                              fi
                              }
                              HTTPDCONFLOCATION=/etc/httpd/conf/httpd.conf
                              WEBPAGECHECK=http://10.0.20.224:80/icons/apache_pb.gif # virtual ip: 10.0.20.224

                              case $1 in
                              'start')
                              #/usr/sbin/apachectl -k start -f $HTTPDCONFLOCATION
                              echo -n $"Starting $prog: "
                              check13 || exit 1
                              LANG=C daemon --pidfile=/var/run/httpd.pid httpd $OPTIONS
                              RET=$?
                              echo
                              [ $RET = 0 ] && touch /var/lock/subsys/httpd
                              ;;
                              'stop')
                              echo -n $"Stopping httpd: "
                              killproc -p /var/run/httpd.pid -d 10 httpd
                              RET=$?
                              echo
                              [ $RET = 0 ] && rm -f /var/lock/subsys/httpd /var/run/httpd.pid
                              # /usr/sbin/apachectl -k stop
                              ;;
                              'clean')
                              /usr/sbin/apachectl -k stop
                              /bin/sleep 10
                              RET=$?
                              ;;
                              'check')
                              if [ -f /var/run/httpd.pid ]; then
                              echo "runnning"
                              RET=0
                              else
                              echo "not running"
                              RET=1
                              fi
                              ;;
                              *)
                              RET=0
                              ;;
                              esac
                              # 0: success; 1 : error
                              if [ $RET -eq 0 ]; then
                              exit 0
                              else
                              exit 1
                              fi



                              oid.scr:_
                              #!/bin/sh
                              # Oracle Clusterware OID Registration Script
                              # Grid Version: 11.2.0.1 Platform: Oracle Enterprise Linux 5.5, Service: Oracle Identity Management LDAP
                              # Created by Ali Okan Yuksel - aokany[at]gmail[dot]com - www.siyahsapka.org
                              # Registration command:
                              # $ /u01/11.2.0/grid/bin/appvipcfg create -network=1 -ip=10.0.20.226 -vipname=appsvip2 -user=oracle
                              # $ /u01/11.2.0/grid/bin/crsctl add resource myOid -type cluster_resource -attr "ACTION_SCRIPT=/u01/AppBase/CRS_SCRIPTS/oid.scr, PLACEMENT=restricted, SERVER_POOLS=projectpool2,CHECK_INTERVAL=30,RESTART_ATTEMPTS=2,START_DEPENDENCIES=hard(appsvip2),STOP_DEPENDENCIES=hard(appsvip2)"
                              #

                              WEBPAGECHECK=http://10.0.20.226:80/welcome_images/SOA/mediator_over.png
                              LD_ASSUME_KERNEL="2.6.18"
                              export LD_ASSUME_KERNEL
                              unset LD_ASSUME_KERNEL
                              echo $LD_ASSUME_KERNEL
                              case $1 in
                              'start')
                              /u01/AppBase/asinst_1/bin/opmnctl startall
                              RET=$?
                              ;;
                              'stop')
                              /u01/AppBase/asinst_1/bin/opmnctl stopall
                              RET=$?
                              ;;
                              'clean')
                              /u01/AppBase/asinst_1/bin/opmnctl stopall
                              RET=$?
                              ;;
                              'check')
                              echo "Check OID entry point has been called.."
                              if /u01/AppBase/asinst_1/bin/opmnctl status ping
                              then
                              echo "Success: OID IS Running"
                              RET=0
                              else
                              echo "FAILED: OID NOT Running"
                              RET=1
                              fi
                              ;;
                              *)
                              RET=0
                              ;;
                              esac
                              # 0: success; 1 : error
                              if [ $RET -eq 0 ]; then
                              exit 0
                              else
                              exit 1
                              fi
                              • 12. Re: crsctl apache registration & start problem
                                Shiyer-Oracle
                                Great! I actually missed reading the last 2 points in your previous reply.