8 Replies Latest reply: Mar 6, 2013 9:18 AM by Erik Janssen RSS

    Please help me, opmn can't start![Urgent~!]

    994898
      i have 2 hp server installed hp-ux with service guard.

      2 servers has the same application. main server can't start, but backup server can work well.

      when i try to cmrunpkg -n mainserver pkg_app, i got follow opmn error message:

      13/03/05 21:15:54 [ons-internal] ONS server initiated
      13/03/05 21:15:54 [pm-internal] PM state directory exists: /oracle/oracle10g/product/10.1.3.1/OracleAS_1/opmn/logs/states
      13/03/05 21:15:54 [pm-internal] OPMN server ready. Request handling enabled.
      13/03/05 21:15:54 [ons-connect] Local connection 127.0.0.1,6100 invalid form factor
      <unknown>
      13/03/05 21:16:00 [ons-connect] Local connection 127.0.0.1,6100 invalid form factor
      <unknown>
      13/03/05 21:16:06 [ons-connect] Local connection 127.0.0.1,6100 invalid form factor
      <unknown>
      13/03/05 21:16:12 [ons-connect] Local connection 127.0.0.1,6100 invalid form factor
      <unknown>
      13/03/05 21:16:18 [ons-connect] Local connection 127.0.0.1,6100 invalid form factor
      <unknown>
      13/03/05 21:16:24 [ons-connect] Local connection 127.0.0.1,6100 invalid form factor
      <unknown>
      13/03/05 21:16:29 [ons-connect] Local connection 127.0.0.1,6100 invalid form factor
      <unknown>
      13/03/05 21:16:34 [ons-connect] Local connection 127.0.0.1,6100 invalid form factor
      <unknown>
      13/03/05 21:16:39 [ons-connect] Local connection 127.0.0.1,6100 invalid form factor
      <unknown>
      13/03/05 21:16:44 [ons-connect] Local connection 127.0.0.1,6100 invalid form factor
      <unknown>
      13/03/05 21:16:49 [ons-connect] Local connection 127.0.0.1,6100 invalid form factor
      <unknown>
      13/03/05 21:16:54 [ons-connect] Local connection 127.0.0.1,6100 invalid form factor


      and then i check cmcluster's control log:


      RCV: Permission denied
      Communication error with the OPMN server local port.
      Check the OPMN log files

      RCV: Permission denied
      Communication error with the OPMN server local port.
      Check the OPMN log files

      RCV: Permission denied
      Communication error with the OPMN server local port.
      Check the OPMN log files

      RCV: Permission denied
      Communication error with the OPMN server local port.
      Check the OPMN log files

      RCV: Permission denied
      Communication error with the OPMN server local port.
      Check the OPMN log files

      RCV: Permission denied
      Communication error with the OPMN server local port.
      Check the OPMN log files

      opmnctl: opmn is not running.


      /oracle/oracle10g/product/10.1.3.1/OracleAS_1/opmn/conf/.formfactor is below to oracle user.i'm sure. but this file have no permission.

      ------------------ .formfactor


      it's very strange. i tried to display opmn's status, i got 3 downs : ASG, default:home, HTTP_SERVER. it seems that opmn is already running. anyone help? if you need more infomation, tell me.

      Edited by: 991895 on 2013-3-5 上午7:21
        • 1. Re: Please help me, opmn can't start![Urgent~!]
          Erik Janssen
          Wild guess, but there was already a daemon process on port 6100 when you tried to start opmn? Try to shutdown opmn and check with lsof i tcp:6100 whether there is still a process listening on this port.

          Thanks,
          EJ
          • 2. Re: Please help me, opmn can't start![Urgent~!]
            994898
            hi, thanks.

            i checked the process list. it seems that opmn started twice. do you understand me? there is 2 'OPMN -d' processes running. i dont' know why it started twice. i didn't modify anything.

            Even i kill any processes below to oracle user, so that there's no opmn,no oracle processes. and then i startall, you guess what, it can't start!
            • 3. Re: Please help me, opmn can't start![Urgent~!]
              994898
              i try to run ./opmnctl startall by root. it works. right now i m sure that some file's permission is not correct!

              anyone can help me ? which files do i need to check?
              • 4. Re: Please help me, opmn can't start![Urgent~!]
                994898
                in /oracle/oracle10g/product/10.1.3.1/OracleAS_1/Apache/Apache/logs/error_log.XXXXXXX,


                [05/Mar/2013 20:45:52 14079] [error] System: Permission denied (errno: 13)
                [05/Mar/2013 20:45:52 14086] [error] Child could not open SSLMutex lockfile /oracle/oracle10g/product/10.1.3.1/OracleAS_1/Apache/Apache/logs/ssl_mutex.12947 (System error follows)


                what does it mean? logs directory have no permission to create file?
                • 5. Re: Please help me, opmn can't start![Urgent~!]
                  Erik Janssen
                  This would suggest that there are files (log/lock) which are owned by another user (presumably root) than the user who starts opmn. Notice that opmn should not be started by the root user.

                  Try to trace the OPEN() system calls with the trace utility available on your platform. From the trace output, you can identify which files are not accessible. Check Note [url https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&id=110888.1]How to Trace Unix System Calls for information about various utilities available for Unix platforms.

                  Thanks,
                  EJ
                  • 6. Re: Please help me, opmn can't start![Urgent~!]
                    994898
                    hi,Thanks your response.

                    that link about trace system calls can't open.

                    about log/lock,where is it normally? could you tell me?
                    • 7. Re: Please help me, opmn can't start![Urgent~!]
                      Erik Janssen
                      Not from the top of my head - check Support Note [url https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&id=1303000.1]1303000.1. It has a lot of troubleshooting tips and it also describes location of log files and the directory (search the note for the string 'unclean').

                      I reckon that cleaning out the states directory would already help. And you may want to do a 'find $ORACLE_HOME/opmn -user root -ls' to find any files owned by root in your OPMN directory.

                      Thanks,
                      EJ
                      • 8. Re: Please help me, opmn can't start![Urgent~!]
                        Erik Janssen
                        The link to trace unix system calls is pointing to a Support Note : https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&id=110888.1

                        You should be able to access this document provided you have a valid My Oracle Support user account.

                        Thanks,
                        EJ