OPMN is failed to start obiee 11.1.1.9 (OPMN server ready. Request handling enabled)

Anil Vemula

Hi All,

OPMN is failing to start.

opmn.log

[2017-07-24T13:32:20+10:00] [opmn] [TRACE:32] [] [internal] ORACLE_HOME: /home/oracle/Oracle/Middleware/Oracle_BI1

[2017-07-24T13:32:20+10:00] [opmn] [TRACE:32] [] [internal] ORACLE_INSTANCE: /home/oracle/Oracle/Middleware/instances/instance1

[2017-07-24T13:32:20+10:00] [opmn] [NOTIFICATION:1] [90] [ons-internal] ONS server initiated

[2017-07-24T13:32:20+10:00] [opmn] [TRACE:32] [] [internal] Host: anzdatabase; Remote Port: 9505; Local Port: 9504; Pid: 7078; 11.1.1.9.0

[2017-07-24T13:32:20+10:00] [opmn] [TRACE:1] [522] [pm-internal] PM state directory exists: /home/oracle/Oracle/Middleware/instances/instance1/config/OPMN/opmn/states

[2017-07-24T13:32:20+10:00] [opmn] [NOTIFICATION:1] [675] [pm-internal] OPMN server ready. Request handling enabled

opmn.out

[2017-07-24T12:54:34+10:00] [opmn] [TRACE:32] [] [internal] Server shut down: status 4200

[2017-07-24T13:23:17+10:00] [opmn] [TRACE:32] [] [internal] Server shut down: status 4200

[2017-07-24T13:30:39+10:00] [opmn] [TRACE:32] [] [internal] Server shut down: status 4200

Note: NodeManager is running

         AdminServer is running

         ManagedServers are running.

Tried solutions.

Tried the solution mentioned in the document (Doc ID 2261219.1) , that did not work.

All your suggestions are appreciated.

Thanks,

Anil.

FTisiot

Have you checked everything is properly shutdown before starting it? is there a process running that may have a port conflict? did you change anything lately that could have an impact?

Anil Vemula

Hi FTisiot

The first error I got was the port is being used (9500).

I have changed the ports local and remote in opmn.xml .

Now If I run ./opmnctl startall , it takes very very long time and finally says server failed.

opmn.log says "OPMN server ready. Request handling enabled"

opmn.out says "[2017-07-24T13:30:39+10:00] [opmn] [TRACE:32] [] [internal] Server shut down: status 4200"

All other services are up and running.

Thanks,

Anil.

Anil Vemula

Hi Christian Berg and FTisiot

I have changed back the ports to default numbers now.

Below is the port status in different stages, hosts file entry, ifconfig result.

1) Before database start

Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name   

tcp        0      0 0.0.0.0:5903            0.0.0.0:*               LISTEN      2502/Xvnc          

tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      -                  

tcp        0      0 0.0.0.0:6003            0.0.0.0:*               LISTEN      2502/Xvnc          

tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      -                  

tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      -                  

tcp        0      0 127.0.0.1:6010          0.0.0.0:*               LISTEN      -                  

tcp6       0      0 :::5903                 :::*                    LISTEN      2502/Xvnc          

tcp6       0      0 :::111                  :::*                    LISTEN      -                  

tcp6       0      0 :::6003                 :::*                    LISTEN      2502/Xvnc          

tcp6       0      0 :::22                   :::*                    LISTEN      -                  

tcp6       0      0 ::1:25                  :::*                    LISTEN      -                  

tcp6       0      0 ::1:6010                :::*                    LISTEN      -                

2) After database start

Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name   

tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      -                  

tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      -                  

tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      -                  

tcp        0      0 127.0.0.1:6010          0.0.0.0:*               LISTEN      -                  

tcp6       0      0 :::37769                :::*                    LISTEN      2703/ora_d000_ORCL

tcp6       0      0 :::111                  :::*                    LISTEN      -                  

tcp6       0      0 :::1521                 :::*                    LISTEN      3014/tnslsnr       

tcp6       0      0 :::22                   :::*                    LISTEN      -                  

tcp6       0      0 ::1:25                  :::*                    LISTEN      -                  

tcp6       0      0 ::1:6010                :::*                    LISTEN      -                  

tcp6       0      0 :::5500                 :::*                    LISTEN      3014/tnslsnr       

3) After NodeManager start

Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name   

tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      -                  

tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      -                  

tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      -                  

tcp        0      0 127.0.0.1:6010          0.0.0.0:*               LISTEN      -                  

tcp        0      0 127.0.0.1:6011          0.0.0.0:*               LISTEN      -                  

tcp6       0      0 :::37769                :::*                    LISTEN      2703/ora_d000_ORCL

tcp6       0      0 :::111                  :::*                    LISTEN      -                  

tcp6       0      0 :::1521                 :::*                    LISTEN      3014/tnslsnr       

tcp6       0      0 :::9556                 :::*                    LISTEN      3236/java          

tcp6       0      0 :::22                   :::*                    LISTEN      -                  

tcp6       0      0 ::1:25                  :::*                    LISTEN      -                  

tcp6       0      0 ::1:6010                :::*                    LISTEN      -                  

tcp6       0      0 ::1:6011                :::*                    LISTEN      -                  

tcp6       0      0 :::5500                 :::*                    LISTEN      3014/tnslsnr       

4) After AdminServer start

Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name   

tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      -                  

tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      -                  

tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      -                  

tcp        0      0 127.0.0.1:6010          0.0.0.0:*               LISTEN      -                  

tcp        0      0 127.0.0.1:6011          0.0.0.0:*               LISTEN      -                  

tcp6       0      0 :::37769                :::*                    LISTEN      2703/ora_d000_ORCL

tcp6       0      0 :::111                  :::*                    LISTEN      -                  

tcp6       0      0 :::1521                 :::*                    LISTEN      3014/tnslsnr       

tcp6       0      0 :::9556                 :::*                    LISTEN      3236/java          

tcp6       0      0 :::22                   :::*                    LISTEN      -                  

tcp6       0      0 ::1:7001                :::*                    LISTEN      3369/java          

tcp6       0      0 127.0.0.1:7001          :::*                    LISTEN      3369/java          

tcp6       0      0 192.168.1.224:7001      :::*                    LISTEN      3369/java          

tcp6       0      0 fe80::f534:adef:1f:7001 :::*                    LISTEN      3369/java          

tcp6       0      0 ::1:25                  :::*                    LISTEN      -                  

tcp6       0      0 ::1:6010                :::*                    LISTEN      -                  

tcp6       0      0 ::1:6011                :::*                    LISTEN      -                  

tcp6       0      0 :::5500                 :::*                    LISTEN      3014/tnslsnr  

5) After ManagedServers start

Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name   

tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      -                  

tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      -                  

tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      -                  

tcp        0      0 127.0.0.1:6010          0.0.0.0:*               LISTEN      -                  

tcp        0      0 127.0.0.1:6011          0.0.0.0:*               LISTEN      -                  

tcp6       0      0 127.0.0.1:9704          :::*                    LISTEN      3663/java          

tcp6       0      0 fe80::f534:adef:1f:9704 :::*                    LISTEN      3663/java          

tcp6       0      0 192.168.1.224:9704      :::*                    LISTEN      3663/java          

tcp6       0      0 ::1:9704                :::*                    LISTEN      3663/java          

tcp6       0      0 :::37769                :::*                    LISTEN      2703/ora_d000_ORCL

tcp6       0      0 :::10378                :::*                    LISTEN      3663/java          

tcp6       0      0 :::8205                 :::*                    LISTEN      3663/java          

tcp6       0      0 :::34477                :::*                    LISTEN      3663/java          

tcp6       0      0 :::111                  :::*                    LISTEN      -                  

tcp6 <span class

Sherry George

Was the environment working fine before? If yes, what changed?

Anil Vemula

Hi Sherry,

Good day to you,

Yes the environment was working before, for some reason the vm was down and I found this issues while bringing the environment up.

Thanks,

Anil.

Anil Vemula

Hi All,

I got the OPMN issue fixed. Below is the solution  (referred doc : Doc ID 1303000.1)

Provided Solution :
1) Killed all the opmn processes (more than 1 process)
$ netstat -tulpn
$ kill -9 <opmn pid>
2) Verified the hosts file for the correct ip
3) Deleted the opmn .log and .out files from /Middleware/instances/instance1/diagnostics/logs/OPMN/opmn
4) Deleted all the files from states directory (including .local and .opmndat)
/Middleware/instances/instance1/config/OPMN/opmn/states .
5) Verify the opmn.xml for the correct ports (remote and local)
4) ./opmnctl validate (to make sure the opmn.xml is correct)
5) ./opmnctl start (to start OPMN Process Manager)
6) ./opmnctl status -l
7) Check the States directory has got new set of files (we deleted in step 4)
8) ./opmnctl stopall (if required)
9) ./opmnctl startall
10) ./opmnctl status

Thanks Christian Berg and Sherry George for your comments.

Anil.