This discussion is archived
1 2 Previous Next 15 Replies Latest reply: Jun 3, 2013 7:25 AM by Levi-Pereira RSS

Redirect with wrong IP or PORT from SCAN LISTENER.

Actitud Newbie
Currently Being Moderated
Hello guys.

I'll try to be as brief/clear as I can. I've been struggling with this issue since couple months now and yet, I couldn't find a solution besides I've tried everything (or at least, that's what I think).

Scenario*:

<li> Two nodes cluster
<li> Oracle CRS and DBMS 11.2.0.3 (PSU 5 has been applied to GI)
<li> Many databases created on it, all of them 11.2.0.3.
<li> Three LISTENER_SCANs
<li> One local LISTENER per node
<li> SINGLETON and UNIFORM services (I'll show you SINGLETON example).
<li> SHARED server mode, three dispatchers on random ports. NO, i don't have a firewall issue :)
<li> SERVER B with LISTENER_SCAN1
<li> SERVER A with LISTENER_SCAN2 and LISTENER_SCAN3

What I want to do?*:

Connect to the SINGLETON service using a SHARED server and of course, using the cluster (SCAN) name.

What is the result?*:

<li> IF I use the cluster name (cluster.domain.com) the connection sometime works, sometimes fails.
<li> IF I use the LISTENER_SCAN running on the server where the SINGLETON service is running (B), connection succeeds.
<li> IF I use the LISTENER_SCAN running on the OTHER server (A), connection fails.

LISTENER-SCAN3 log from server A*
04-APR-2013 14:05:37 * (CONNECT_DATA=(CID=(PROGRAM=JDBC Thin Client)(HOST=__jdbc__)(USER=bogdan))(SERVICE_NAME=dbname_srv_s)(CID=(PROGRAM=JDBC Thin Client)(HOST=__jdbc__)(USER=bogdan))) * (ADDRESS=(PROTOCOL=tcp)(HOST=bogdansIP)(PORT=58611)) * establish * dbname_srv_s * 0
04-APR-2013 14:05:39 * service_update * asuadev4_2 * 0
04-APR-2013 14:05:40 * (CONNECT_DATA=(CID=(PROGRAM=JDBC Thin Client)(HOST=__jdbc__)(USER=bogdan))(SERVICE_NAME=dbname_srv_s)(CID=(PROGRAM=JDBC Thin Client)(HOST=__jdbc__)(USER=bogdan))) * (ADDRESS=(PROTOCOL=tcp)(HOST=bogdansIP)(PORT=58613)) * establish * dbname_srv_s * 0
04-APR-2013 14:06:38 * (CONNECT_DATA=(CID=(PROGRAM=JDBC Thin Client)(HOST=__jdbc__)(USER=bogdan))(SERVICE_NAME=dbname_srv_s)(CID=(PROGRAM=JDBC Thin Client)(HOST=__jdbc__)(USER=bogdan))) * (ADDRESS=(PROTOCOL=tcp)(HOST=bogdansIP)(PORT=58624)) * establish * dbname_srv_s * 0
From Wireshark I'm getting this*

I tried to snif the packages to see what is going on, and what I receive is a REDIRECT to LISTENER_SCAN3 IP (running on A) with a dispatcher's port which is running on server B.
Redirect Data: (ADDRESS=(PROTOCOL=tcp)(HOST=x.x.x.241)(PORT=56270))
Redirect Data: (ADDRESS=(PROTOCOL=tcp)(HOST=x.x.x.241)(PORT=9317))
Redirect Data: (ADDRESS=(PROTOCOL=tcp)(HOST=x.x.x.241)(PORT=50045))
bash>srvctl config scan
SCAN name: cluster.domain.com, Network: 1/x.x.x.0/255.255.254.0/bond0
SCAN VIP name: scan1, IP: /cluster.domain.com/x.x.x.242
SCAN VIP name: scan2, IP: /cluster.domain.com/x.x.x.243
SCAN VIP name: scan3, IP: /cluster.domain.com/x.x.x.241

bash>crsctl status res -t
...
ora.LISTENER_SCAN1.lsnr     1        ONLINE  ONLINE       B
ora.LISTENER_SCAN2.lsnr     1        ONLINE  ONLINE       A                                      
ora.LISTENER_SCAN3.lsnr     1        ONLINE  ONLINE       A   
...
ora.scan1.vip               1        ONLINE  ONLINE       B                                     
ora.scan2.vip               1        ONLINE  ONLINE       A                                      
ora.scan3.vip               1        ONLINE  ONLINE       A  
...
Database and services status*
ora.dbname.dbname_srv_s.svc     1        ONLINE  ONLINE       B                                     
ora.dbname.dbname_srv_u.svc     1        ONLINE  ONLINE       B                                     
                                2        ONLINE  ONLINE       A                                      
ora.dbname.db                   1        ONLINE  ONLINE       A                 Open                
                                2        ONLINE  ONLINE       B                 Open 
Some parameters*
NAME                   SID        VALUE                               IsSpec
---------------------- ---------- ----------------------------------- ------
dispatchers                  *          (protocol=tcp)(dispatchers=3)       TRUE
local_listener               *                                                 FALSE
max_shared_servers           *          500                                 TRUE
processes                    *          2000                                TRUE
remote_listener              *          cluster.domain.com:1521    TRUE
sessions                     *          445                                 TRUE
shared_server_sessions       *          100                                 TRUE
shared_servers               *          40                                  TRUE
These are the open ports tied up to dispatcher processes*
tcp        0      0 :::56270                    :::*                        LISTEN      1356/ora_d000_dbnam 
tcp        0      0 :::9317                     :::*                        LISTEN      1361/ora_d001_dbnam 
tcp        0      0 :::50045                    :::*                        LISTEN      1368/ora_d002_dbnam 
Let me know if you need ANY aditional information and thanks a LOT for any help you can provide me.
Alex.
  • 1. Re: Redirect with wrong IP or PORT from SCAN LISTENER.
    Levi-Pereira Guru
    Currently Being Moderated
    Hi,

    Can you perform the same test as I did and post output here:
    http://levipereira.wordpress.com/2011/05/03/configuring-client-to-use-scan-11-2-0/


    Little note about LOCAL_LISTENER parameter (it's just a comment):
    The Database Agent (previously known as racgimon) configures the LOCAL_LISTENER initialization parameters automatically. You do not need to alter this setting.
    You can set a value manually for LOCAL_LISTENER. However, Oracle recommends that you leave the parameter unset so that the Database Agent can maintain it automatically.
  • 2. Re: Redirect with wrong IP or PORT from SCAN LISTENER.
    Actitud Newbie
    Currently Being Moderated
    Hello Levi.

    First of all, congrats for your notes and documents. I always enjoy the way you analize things.
    BTW, I've ran all those tests many times but since I haven't save the results, here they are again.

    Thanks,
    Alex.

    <font color="red">Server side tests*</font>
    A@/u01/oracle> olsnodes -i  -s -n
    A 1       A-vip     Active
    B        2       B-vip    Active
    
    A@/u01/oracle> srvctl config vip -n A
    VIP exists: /A-vip/x.x.x.221/x.x.x.0/255.255.254.0/bond0, hosting node A
    
    A@/u01/oracle> srvctl config vip -n B
    VIP exists: /B-vip/x.x.x.222/x.x.x.0/255.255.254.0/bond0, hosting node B
    
    A@/u01/oracle> srvctl config scan
    SCAN name: cluster.domain.com, Network: 1/x.x.x.0/255.255.254.0/bond0
    SCAN VIP name: scan1, IP: /cluster.domain.com/x.x.x.242
    SCAN VIP name: scan2, IP: /cluster.domain.com/x.x.x.243
    SCAN VIP name: scan3, IP: /cluster.domain.com/x.x.x.241
    
    SQL> select INST_ID, NAME, VALUE from gv$parameter where name like '%_listener%';
    
       INST_ID NAME              VALUE
       ------------------------------------------------------------------------------------------------------------
       2       local_listener    (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=x.x.x.222)(PORT=1521))))
       2       remote_listener    cluster.domain.com:1521
       1       local_listener    (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=x.x.x.221)(PORT=1521))))
       1       remote_listener    cluster.domain.com:1521
       
    <font color="red">NODE A*</font>


    LISTENER-SCAN2 on A*
    Service "dbname" has 2 instance(s).
      Instance "dbname_1", status READY, has 4 handler(s) for this service...
        Handler(s):
          "D002" established:5 refused:0 current:0 max:1022 state:ready
             DISPATCHER <machine: A, pid: 1221>
             (ADDRESS=(PROTOCOL=tcp)(HOST=A.domain.com)(PORT=59319))
          "D001" established:5 refused:0 current:1 max:1022 state:ready
             DISPATCHER <machine: A, pid: 1171>
             (ADDRESS=(PROTOCOL=tcp)(HOST=A.domain.com)(PORT=43696))
          "D000" established:5 refused:0 current:0 max:1022 state:ready
             DISPATCHER <machine: A, pid: 1120>
             (ADDRESS=(PROTOCOL=tcp)(HOST=A.domain.com)(PORT=26500))
          "DEDICATED" established:0 refused:0 state:ready
             REMOTE SERVER
             (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=x.x.x.221)(PORT=1521)))
      Instance "dbname_2", status READY, has 6 handler(s) for this service...
        Handler(s):
          "D002" established:61 refused:0 current:0 max:1022 state:ready
             DISPATCHER <machine: B, pid: 1368>
             (ADDRESS=(PROTOCOL=tcp)(HOST=B.domain.com)(PORT=50045))
          "D001" established:61 refused:0 current:1 max:1022 state:ready
             DISPATCHER <machine: B, pid: 1361>
             (ADDRESS=(PROTOCOL=tcp)(HOST=B.domain.com)(PORT=9317))
          "D000" established:54 refused:0 current:1 max:1022 state:ready
             DISPATCHER <machine: B, pid: 1356>
             (ADDRESS=(PROTOCOL=tcp)(HOST=B.domain.com)(PORT=56270))
          "DEDICATED" established:0 refused:0 state:ready
             REMOTE SERVER
             (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=x.x.x.222)(PORT=1521)))
          "D003" established:9 refused:0 current:0 max:1022 state:ready
             DISPATCHER <machine: B, pid: 25363>
             (ADDRESS=(PROTOCOL=tcp)(HOST=x.x.x.222)(PORT=52661))
          "D004" established:9 refused:0 current:0 max:1022 state:ready
             DISPATCHER <machine: B, pid: 25482>
             (ADDRESS=(PROTOCOL=tcp)(HOST=x.x.x.214)(PORT=54368))
    Service "dbname_srv_s" has 1 instance(s).
      Instance "dbname_2", status READY, has 6 handler(s) for this service...
        Handler(s):
          "D002" established:61 refused:0 current:0 max:1022 state:ready
             DISPATCHER <machine: B, pid: 1368>
             (ADDRESS=(PROTOCOL=tcp)(HOST=B.domain.com)(PORT=50045))
          "D001" established:61 refused:0 current:1 max:1022 state:ready
             DISPATCHER <machine: B, pid: 1361>
             (ADDRESS=(PROTOCOL=tcp)(HOST=B.domain.com)(PORT=9317))
          "D000" established:54 refused:0 current:1 max:1022 state:ready
             DISPATCHER <machine: B, pid: 1356>
             (ADDRESS=(PROTOCOL=tcp)(HOST=B.domain.com)(PORT=56270))
          "DEDICATED" established:0 refused:0 state:ready
             REMOTE SERVER
             (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=x.x.x.222)(PORT=1521)))
          "D003" established:9 refused:0 current:0 max:1022 state:ready
             DISPATCHER <machine: B, pid: 25363>
             (ADDRESS=(PROTOCOL=tcp)(HOST=x.x.x.222)(PORT=52661))
          "D004" established:9 refused:0 current:0 max:1022 state:ready
             DISPATCHER <machine: B, pid: 25482>
             (ADDRESS=(PROTOCOL=tcp)(HOST=x.x.x.214)(PORT=54368))
    Service "dbname_srv_u" has 2 instance(s).
      Instance "dbname_1", status READY, has 4 handler(s) for this service...
        Handler(s):
          "D002" established:5 refused:0 current:0 max:1022 state:ready
             DISPATCHER <machine: A, pid: 1221>
             (ADDRESS=(PROTOCOL=tcp)(HOST=A.domain.com)(PORT=59319))
          "D001" established:5 refused:0 current:1 max:1022 state:ready
             DISPATCHER <machine: A, pid: 1171>
             (ADDRESS=(PROTOCOL=tcp)(HOST=A.domain.com)(PORT=43696))
          "D000" established:5 refused:0 current:0 max:1022 state:ready
             DISPATCHER <machine: A, pid: 1120>
             (ADDRESS=(PROTOCOL=tcp)(HOST=A.domain.com)(PORT=26500))
          "DEDICATED" established:0 refused:0 state:ready
             REMOTE SERVER
             (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=x.x.x.221)(PORT=1521)))
      Instance "dbname_2", status READY, has 6 handler(s) for this service...
        Handler(s):
          "D002" established:61 refused:0 current:0 max:1022 state:ready
             DISPATCHER <machine: B, pid: 1368>
             (ADDRESS=(PROTOCOL=tcp)(HOST=B.domain.com)(PORT=50045))
          "D001" established:61 refused:0 current:1 max:1022 state:ready
             DISPATCHER <machine: B, pid: 1361>
             (ADDRESS=(PROTOCOL=tcp)(HOST=B.domain.com)(PORT=9317))
          "D000" established:54 refused:0 current:1 max:1022 state:ready
             DISPATCHER <machine: B, pid: 1356>
             (ADDRESS=(PROTOCOL=tcp)(HOST=B.domain.com)(PORT=56270))
          "DEDICATED" established:0 refused:0 state:ready
             REMOTE SERVER
             (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=x.x.x.222)(PORT=1521)))
          "D003" established:9 refused:0 current:0 max:1022 state:ready
             DISPATCHER <machine: B, pid: 25363>
             (ADDRESS=(PROTOCOL=tcp)(HOST=x.x.x.222)(PORT=52661))
          "D004" established:9 refused:0 current:0 max:1022 state:ready
             DISPATCHER <machine: B, pid: 25482>
             (ADDRESS=(PROTOCOL=tcp)(HOST=x.x.x.214)(PORT=54368))
    LISTENER-SCAN3 on A*
    Service "dbname" has 2 instance(s).
      Instance "dbname_1", status READY, has 4 handler(s) for this service...
        Handler(s):
          "D002" established:7 refused:0 current:0 max:1022 state:ready
             DISPATCHER <machine: A, pid: 1221>
             (ADDRESS=(PROTOCOL=tcp)(HOST=A.domain.com)(PORT=59319))
          "D001" established:10 refused:0 current:1 max:1022 state:ready
             DISPATCHER <machine: A, pid: 1171>
             (ADDRESS=(PROTOCOL=tcp)(HOST=A.domain.com)(PORT=43696))
          "D000" established:6 refused:0 current:0 max:1022 state:ready
             DISPATCHER <machine: A, pid: 1120>
             (ADDRESS=(PROTOCOL=tcp)(HOST=A.domain.com)(PORT=26500))
          "DEDICATED" established:0 refused:0 state:ready
             REMOTE SERVER
             (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=x.x.x.221)(PORT=1521)))
      Instance "dbname_2", status READY, has 6 handler(s) for this service...
        Handler(s):
          "D002" established:40 refused:0 current:0 max:1022 state:ready
             DISPATCHER <machine: B, pid: 1368>
             (ADDRESS=(PROTOCOL=tcp)(HOST=B.domain.com)(PORT=50045))
          "D001" established:40 refused:0 current:1 max:1022 state:ready
             DISPATCHER <machine: B, pid: 1361>
             (ADDRESS=(PROTOCOL=tcp)(HOST=B.domain.com)(PORT=9317))
          "D000" established:32 refused:0 current:1 max:1022 state:ready
             DISPATCHER <machine: B, pid: 1356>
             (ADDRESS=(PROTOCOL=tcp)(HOST=B.domain.com)(PORT=56270))
          "DEDICATED" established:0 refused:0 state:ready
             REMOTE SERVER
             (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=x.x.x.222)(PORT=1521)))
          "D003" established:0 refused:0 current:0 max:1022 state:ready
             DISPATCHER <machine: B, pid: 25363>
             (ADDRESS=(PROTOCOL=tcp)(HOST=x.x.x.222)(PORT=52661))
          "D004" established:0 refused:0 current:0 max:1022 state:ready
             DISPATCHER <machine: B, pid: 25482>
             (ADDRESS=(PROTOCOL=tcp)(HOST=x.x.x.214)(PORT=54368))
    Service "dbname_srv_s" has 1 instance(s).
      Instance "dbname_2", status READY, has 6 handler(s) for this service...
        Handler(s):
          "D002" established:40 refused:0 current:0 max:1022 state:ready
             DISPATCHER <machine: B, pid: 1368>
             (ADDRESS=(PROTOCOL=tcp)(HOST=B.domain.com)(PORT=50045))
          "D001" established:40 refused:0 current:1 max:1022 state:ready
             DISPATCHER <machine: B, pid: 1361>
             (ADDRESS=(PROTOCOL=tcp)(HOST=B.domain.com)(PORT=9317))
          "D000" established:32 refused:0 current:1 max:1022 state:ready
             DISPATCHER <machine: B, pid: 1356>
             (ADDRESS=(PROTOCOL=tcp)(HOST=B.domain.com)(PORT=56270))
          "DEDICATED" established:0 refused:0 state:ready
             REMOTE SERVER
             (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=x.x.x.222)(PORT=1521)))
          "D003" established:0 refused:0 current:0 max:1022 state:ready
             DISPATCHER <machine: B, pid: 25363>
             (ADDRESS=(PROTOCOL=tcp)(HOST=x.x.x.222)(PORT=52661))
          "D004" established:0 refused:0 current:0 max:1022 state:ready
             DISPATCHER <machine: B, pid: 25482>
             (ADDRESS=(PROTOCOL=tcp)(HOST=x.x.x.214)(PORT=54368))
    Service "dbname_srv_u" has 2 instance(s).
      Instance "dbname_1", status READY, has 4 handler(s) for this service...
        Handler(s):
          "D002" established:7 refused:0 current:0 max:1022 state:ready
             DISPATCHER <machine: A, pid: 1221>
             (ADDRESS=(PROTOCOL=tcp)(HOST=A.domain.com)(PORT=59319))
          "D001" established:10 refused:0 current:1 max:1022 state:ready
             DISPATCHER <machine: A, pid: 1171>
             (ADDRESS=(PROTOCOL=tcp)(HOST=A.domain.com)(PORT=43696))
          "D000" established:6 refused:0 current:0 max:1022 state:ready
             DISPATCHER <machine: A, pid: 1120>
             (ADDRESS=(PROTOCOL=tcp)(HOST=A.domain.com)(PORT=26500))
          "DEDICATED" established:0 refused:0 state:ready
             REMOTE SERVER
             (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=x.x.x.221)(PORT=1521)))
      Instance "dbname_2", status READY, has 6 handler(s) for this service...
        Handler(s):
          "D002" established:40 refused:0 current:0 max:1022 state:ready
             DISPATCHER <machine: B, pid: 1368>
             (ADDRESS=(PROTOCOL=tcp)(HOST=B.domain.com)(PORT=50045))
          "D001" established:40 refused:0 current:1 max:1022 state:ready
             DISPATCHER <machine: B, pid: 1361>
             (ADDRESS=(PROTOCOL=tcp)(HOST=B.domain.com)(PORT=9317))
          "D000" established:32 refused:0 current:1 max:1022 state:ready
             DISPATCHER <machine: B, pid: 1356>
             (ADDRESS=(PROTOCOL=tcp)(HOST=B.domain.com)(PORT=56270))
          "DEDICATED" established:0 refused:0 state:ready
             REMOTE SERVER
             (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=x.x.x.222)(PORT=1521)))
          "D003" established:0 refused:0 current:0 max:1022 state:ready
             DISPATCHER <machine: B, pid: 25363>
             (ADDRESS=(PROTOCOL=tcp)(HOST=x.x.x.222)(PORT=52661))
          "D004" established:0 refused:0 current:0 max:1022 state:ready
             DISPATCHER <machine: B, pid: 25482>
             (ADDRESS=(PROTOCOL=tcp)(HOST=x.x.x.214)(PORT=54368))
    LOCAL LISTENER ON A*
    Service "dbname" has 1 instance(s).
      Instance "dbname_1", status READY, has 4 handler(s) for this service...
        Handler(s):
          "D002" established:1 refused:0 current:0 max:1022 state:ready
             DISPATCHER <machine: A, pid: 1221>
             (ADDRESS=(PROTOCOL=tcp)(HOST=A.domain.com)(PORT=59319))
          "D001" established:0 refused:0 current:1 max:1022 state:ready
             DISPATCHER <machine: A, pid: 1171>
             (ADDRESS=(PROTOCOL=tcp)(HOST=A.domain.com)(PORT=43696))
          "D000" established:0 refused:0 current:0 max:1022 state:ready
             DISPATCHER <machine: A, pid: 1120>
             (ADDRESS=(PROTOCOL=tcp)(HOST=A.domain.com)(PORT=26500))
          "DEDICATED" established:0 refused:0 state:ready
             LOCAL SERVER
    Service "dbname_srv_u" has 1 instance(s).
      Instance "dbname_1", status READY, has 4 handler(s) for this service...
        Handler(s):
          "D002" established:1 refused:0 current:0 max:1022 state:ready
             DISPATCHER <machine: A, pid: 1221>
             (ADDRESS=(PROTOCOL=tcp)(HOST=A.domain.com)(PORT=59319))
          "D001" established:0 refused:0 current:1 max:1022 state:ready
             DISPATCHER <machine: A, pid: 1171>
             (ADDRESS=(PROTOCOL=tcp)(HOST=A.domain.com)(PORT=43696))
          "D000" established:0 refused:0 current:0 max:1022 state:ready
             DISPATCHER <machine: A, pid: 1120>
             (ADDRESS=(PROTOCOL=tcp)(HOST=A.domain.com)(PORT=26500))
          "DEDICATED" established:0 refused:0 state:ready
             LOCAL SERVER
    <font color="red">NODE B (where the singleton service is running)*</font>

    LISTENER-SCAN1 on B*
    Service "dbname" has 2 instance(s).
      Instance "dbname_1", status READY, has 4 handler(s) for this service...
        Handler(s):
          "D002" established:3 refused:0 current:0 max:1022 state:ready
             DISPATCHER <machine: A, pid: 1221>
             (ADDRESS=(PROTOCOL=tcp)(HOST=A.domain.com)(PORT=59319))
          "D001" established:4 refused:0 current:1 max:1022 state:ready
             DISPATCHER <machine: A, pid: 1171>
             (ADDRESS=(PROTOCOL=tcp)(HOST=A.domain.com)(PORT=43696))
          "D000" established:4 refused:0 current:0 max:1022 state:ready
             DISPATCHER <machine: A, pid: 1120>
             (ADDRESS=(PROTOCOL=tcp)(HOST=A.domain.com)(PORT=26500))
          "DEDICATED" established:0 refused:0 state:ready
             REMOTE SERVER
             (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=x.x.x.221)(PORT=1521)))
      Instance "dbname_2", status READY, has 6 handler(s) for this service...
        Handler(s):
          "D002" established:5044 refused:0 current:0 max:1022 state:ready
             DISPATCHER <machine: B, pid: 1368>
             (ADDRESS=(PROTOCOL=tcp)(HOST=B.domain.com)(PORT=50045))
          "D001" established:5036 refused:0 current:1 max:1022 state:ready
             DISPATCHER <machine: B, pid: 1361>
             (ADDRESS=(PROTOCOL=tcp)(HOST=B.domain.com)(PORT=9317))
          "D000" established:4978 refused:0 current:1 max:1022 state:ready
             DISPATCHER <machine: B, pid: 1356>
             (ADDRESS=(PROTOCOL=tcp)(HOST=B.domain.com)(PORT=56270))
          "DEDICATED" established:0 refused:0 state:ready
             REMOTE SERVER
             (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=x.x.x.222)(PORT=1521)))
          "D003" established:0 refused:0 current:0 max:1022 state:ready
             DISPATCHER <machine: B, pid: 25363>
             (ADDRESS=(PROTOCOL=tcp)(HOST=x.x.x.222)(PORT=52661))
          "D004" established:0 refused:0 current:0 max:1022 state:ready
             DISPATCHER <machine: B, pid: 25482>
             (ADDRESS=(PROTOCOL=tcp)(HOST=x.x.x.214)(PORT=54368))
    Service "dbname_srv_s" has 1 instance(s).
      Instance "dbname_2", status READY, has 6 handler(s) for this service...
        Handler(s):
          "D002" established:5044 refused:0 current:0 max:1022 state:ready
             DISPATCHER <machine: B, pid: 1368>
             (ADDRESS=(PROTOCOL=tcp)(HOST=B.domain.com)(PORT=50045))
          "D001" established:5036 refused:0 current:1 max:1022 state:ready
             DISPATCHER <machine: B, pid: 1361>
             (ADDRESS=(PROTOCOL=tcp)(HOST=B.domain.com)(PORT=9317))
          "D000" established:4978 refused:0 current:1 max:1022 state:ready
             DISPATCHER <machine: B, pid: 1356>
             (ADDRESS=(PROTOCOL=tcp)(HOST=B.domain.com)(PORT=56270))
          "DEDICATED" established:0 refused:0 state:ready
             REMOTE SERVER
             (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=x.x.x.222)(PORT=1521)))
          "D003" established:0 refused:0 current:0 max:1022 state:ready
             DISPATCHER <machine: B, pid: 25363>
             (ADDRESS=(PROTOCOL=tcp)(HOST=x.x.x.222)(PORT=52661))
          "D004" established:0 refused:0 current:0 max:1022 state:ready
             DISPATCHER <machine: B, pid: 25482>
             (ADDRESS=(PROTOCOL=tcp)(HOST=x.x.x.214)(PORT=54368))
    Service "dbname_srv_u" has 2 instance(s).
      Instance "dbname_1", status READY, has 4 handler(s) for this service...
        Handler(s):
          "D002" established:3 refused:0 current:0 max:1022 state:ready
             DISPATCHER <machine: A, pid: 1221>
             (ADDRESS=(PROTOCOL=tcp)(HOST=A.domain.com)(PORT=59319))
          "D001" established:4 refused:0 current:1 max:1022 state:ready
             DISPATCHER <machine: A, pid: 1171>
             (ADDRESS=(PROTOCOL=tcp)(HOST=A.domain.com)(PORT=43696))
          "D000" established:4 refused:0 current:0 max:1022 state:ready
             DISPATCHER <machine: A, pid: 1120>
             (ADDRESS=(PROTOCOL=tcp)(HOST=A.domain.com)(PORT=26500))
          "DEDICATED" established:0 refused:0 state:ready
             REMOTE SERVER
             (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=x.x.x.221)(PORT=1521)))
      Instance "dbname_2", status READY, has 6 handler(s) for this service...
        Handler(s):
          "D002" established:5044 refused:0 current:0 max:1022 state:ready
             DISPATCHER <machine: B, pid: 1368>
             (ADDRESS=(PROTOCOL=tcp)(HOST=B.domain.com)(PORT=50045))
          "D001" established:5036 refused:0 current:1 max:1022 state:ready
             DISPATCHER <machine: B, pid: 1361>
             (ADDRESS=(PROTOCOL=tcp)(HOST=B.domain.com)(PORT=9317))
          "D000" established:4978 refused:0 current:1 max:1022 state:ready
             DISPATCHER <machine: B, pid: 1356>
             (ADDRESS=(PROTOCOL=tcp)(HOST=B.domain.com)(PORT=56270))
          "DEDICATED" established:0 refused:0 state:ready
             REMOTE SERVER
             (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=x.x.x.222)(PORT=1521)))
          "D003" established:0 refused:0 current:0 max:1022 state:ready
             DISPATCHER <machine: B, pid: 25363>
             (ADDRESS=(PROTOCOL=tcp)(HOST=x.x.x.222)(PORT=52661))
          "D004" established:0 refused:0 current:0 max:1022 state:ready
             DISPATCHER <machine: B, pid: 25482>
             (ADDRESS=(PROTOCOL=tcp)(HOST=x.x.x.214)(PORT=54368))
    LOCAL LISTENER ON B*
    Service "dbname" has 1 instance(s).
      Instance "dbname_2", status READY, has 6 handler(s) for this service...
        Handler(s):
          "D002" established:0 refused:0 current:0 max:1022 state:ready
             DISPATCHER <machine: B, pid: 1368>
             (ADDRESS=(PROTOCOL=tcp)(HOST=B.domain.com)(PORT=50045))
          "D001" established:0 refused:0 current:1 max:1022 state:ready
             DISPATCHER <machine: B, pid: 1361>
             (ADDRESS=(PROTOCOL=tcp)(HOST=B.domain.com)(PORT=9317))
          "D000" established:0 refused:0 current:1 max:1022 state:ready
             DISPATCHER <machine: B, pid: 1356>
             (ADDRESS=(PROTOCOL=tcp)(HOST=B.domain.com)(PORT=56270))
          "DEDICATED" established:0 refused:0 state:ready
             LOCAL SERVER
          "D003" established:1 refused:0 current:0 max:1022 state:ready
             DISPATCHER <machine: B, pid: 25363>
             (ADDRESS=(PROTOCOL=tcp)(HOST=x.x.x.222)(PORT=52661))
          "D004" established:1 refused:0 current:0 max:1022 state:ready
             DISPATCHER <machine: B, pid: 25482>
             (ADDRESS=(PROTOCOL=tcp)(HOST=x.x.x.214)(PORT=54368))
    Service "dbname_srv_s" has 1 instance(s).
      Instance "dbname_2", status READY, has 6 handler(s) for this service...
        Handler(s):
          "D002" established:0 refused:0 current:0 max:1022 state:ready
             DISPATCHER <machine: B, pid: 1368>
             (ADDRESS=(PROTOCOL=tcp)(HOST=B.domain.com)(PORT=50045))
          "D001" established:0 refused:0 current:1 max:1022 state:ready
             DISPATCHER <machine: B, pid: 1361>
             (ADDRESS=(PROTOCOL=tcp)(HOST=B.domain.com)(PORT=9317))
          "D000" established:0 refused:0 current:1 max:1022 state:ready
             DISPATCHER <machine: B, pid: 1356>
             (ADDRESS=(PROTOCOL=tcp)(HOST=B.domain.com)(PORT=56270))
          "DEDICATED" established:0 refused:0 state:ready
             LOCAL SERVER
          "D003" established:1 refused:0 current:0 max:1022 state:ready
             DISPATCHER <machine: B, pid: 25363>
             (ADDRESS=(PROTOCOL=tcp)(HOST=x.x.x.222)(PORT=52661))
          "D004" established:1 refused:0 current:0 max:1022 state:ready
             DISPATCHER <machine: B, pid: 25482>
             (ADDRESS=(PROTOCOL=tcp)(HOST=x.x.x.214)(PORT=54368))
    Service "dbname_srv_u" has 1 instance(s).
      Instance "dbname_2", status READY, has 6 handler(s) for this service...
        Handler(s):
          "D002" established:0 refused:0 current:0 max:1022 state:ready
             DISPATCHER <machine: B, pid: 1368>
             (ADDRESS=(PROTOCOL=tcp)(HOST=B.domain.com)(PORT=50045))
          "D001" established:0 refused:0 current:1 max:1022 state:ready
             DISPATCHER <machine: B, pid: 1361>
             (ADDRESS=(PROTOCOL=tcp)(HOST=B.domain.com)(PORT=9317))
          "D000" established:0 refused:0 current:1 max:1022 state:ready
             DISPATCHER <machine: B, pid: 1356>
             (ADDRESS=(PROTOCOL=tcp)(HOST=B.domain.com)(PORT=56270))
          "DEDICATED" established:0 refused:0 state:ready
             LOCAL SERVER
          "D003" established:1 refused:0 current:0 max:1022 state:ready
             DISPATCHER <machine: B, pid: 25363>
             (ADDRESS=(PROTOCOL=tcp)(HOST=x.x.x.222)(PORT=52661))
          "D004" established:1 refused:0 current:0 max:1022 state:ready
             DISPATCHER <machine: B, pid: 25482>
             (ADDRESS=(PROTOCOL=tcp)(HOST=x.x.x.214)(PORT=54368))
    <font color="red">Client Side Tests*</font>

    C:\>nslookup cluster
    Server:  any.domain.com
    Address:  x.x.x.x
    
    Name:    cluster.domain.com
    Addresses:  x.x.x.243
              x.x.x.241
              x.x.x.242
    
    
    C:\>ping cluster
    
    Pinging cluster.domain.com [x.x.x.241] with 32 bytes of data:
    Reply from x.x.x.241: bytes=32 time=208ms TTL=58
    Reply from x.x.x.241: bytes=32 time=208ms TTL=58
    Reply from x.x.x.241: bytes=32 time=213ms TTL=58
    Reply from x.x.x.241: bytes=32 time=208ms TTL=58
    
    Ping statistics for x.x.x.241:
        Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
    Approximate round trip times in milli-seconds:
        Minimum = 208ms, Maximum = 213ms, Average = 209ms
    
    C:\>ping A-vip
    
    Pinging A-vip.domain.com [x.x.x.221] with 32 bytes of data:
    Reply from x.x.x.221: bytes=32 time=209ms TTL=58
    Reply from x.x.x.221: bytes=32 time=206ms TTL=58
    Reply from x.x.x.221: bytes=32 time=209ms TTL=58
    Reply from x.x.x.221: bytes=32 time=208ms TTL=58
    
    Ping statistics for x.x.x.221:
        Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
    Approximate round trip times in milli-seconds:
        Minimum = 206ms, Maximum = 209ms, Average = 208ms
    
    C:\>ping B-vip
    
    Pinging B-vip.domain.com [x.x.x.222] with 32 bytes of data:
    Reply from x.x.x.222: bytes=32 time=206ms TTL=58
    Reply from x.x.x.222: bytes=32 time=207ms TTL=58
    Reply from x.x.x.222: bytes=32 time=208ms TTL=58
    Reply from x.x.x.222: bytes=32 time=209ms TTL=58
    
    Ping statistics for x.x.x.222:
        Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
    Approximate round trip times in milli-seconds:
        Minimum = 206ms, Maximum = 209ms, Average = 207ms
         
    ##### TELNET TESTS WORKED IN ALL CASES #####
    
    C:\>sqlplus system/oracle@B-vip:1521/dbname_srv_s
    
    SQL*Plus: Release 11.2.0.1.0 Production on Fri Apr 5 10:47:57 2013
    
    Copyright (c) 1982, 2010, Oracle.  All rights reserved.
    
    
    Connected to:
    Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
    With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
    Data Mining and Real Application Testing options
    
    SQL>
    
    
    C:\>sqlplus system/oracle@A-vip:1521/dbname
    
    SQL*Plus: Release 11.2.0.1.0 Production on Fri Apr 5 10:48:35 2013
    
    Copyright (c) 1982, 2010, Oracle.  All rights reserved.
    
    
    Connected to:
    Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
    With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
    Data Mining and Real Application Testing options
    
    
    SQL>
    Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
    With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
    Data Mining and Real Application Testing options
    
    C:\>sqlplus system/oracle@B-vip:1521/dbname
    
    SQL*Plus: Release 11.2.0.1.0 Production on Fri Apr 5 10:48:46 2013
    
    Copyright (c) 1982, 2010, Oracle.  All rights reserved.
    
    
    Connected to:
    Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
    With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
    Data Mining and Real Application Testing options
    
    SQL>
    
    TNSconnection
    
    C:\>tnsping dbname_srv_s_A_scan3.domain.com
    
    TNS Ping Utility for 32-bit Windows: Version 11.2.0.1.0 - Production on 05-APR-2013 10:55:17
    
    Copyright (c) 1997, 2010, Oracle.  All rights reserved.
    
    Used parameter files:
    C:\oracle\product\11.2.0\client_1\network\admin\sqlnet.ora
    
    
    Used TNSNAMES adapter to resolve the alias
    Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP) (HOST = x.x.x.241) (PORT = 1521)) (CONNECT_DATA = (SERVER = SHARED) (SERV
    ICE_NAME = dbname_srv_s)))
    OK (3450 msec)
    
    C:\>tnsping dbname_srv_s_B_scan1.domain.com
    
    TNS Ping Utility for 32-bit Windows: Version 11.2.0.1.0 - Production on 05-APR-2013 10:55:24
    
    Copyright (c) 1997, 2010, Oracle.  All rights reserved.
    
    Used parameter files:
    C:\oracle\product\11.2.0\client_1\network\admin\sqlnet.ora
    
    
    Used TNSNAMES adapter to resolve the alias
    Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP) (HOST = x.x.x.242) (PORT = 1521)) (CONNECT_DATA = (SERVER = SHARED) (SERV
    ICE_NAME = dbname_srv_s)))
    OK (430 msec)
    
    NOW, the TNS tests.
    
    Connecting from LISTENER_SCAN1 running on B
    -------------------------------------------
    
    Connection from SQLPLUS
    C:\>sqlplus system/oracle@dbname_srv_s_B_scan1.domain.com
    SQL*Plus: Release 11.2.0.1.0 Production on Fri Apr 5 11:05:16 2013
    Copyright (c) 1982, 2010, Oracle.  All rights reserved.
    Connected to:
    Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
    With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
    Data Mining and Real Application Testing options
    SQL>
    ### Results from Wireshark : NO ERRORS NOR REDIRECTS.
    
    
    Connecting from LISTENER_SCAN2 running on A
    -------------------------------------------
    Connection from SQLPLUS
    C:\>sqlplus system/oracle@dbname_srv_s_A_scan2.domain.com
    SQL*Plus: Release 11.2.0.1.0 Production on Fri Apr 5 11:02:54 2013
    Copyright (c) 1982, 2010, Oracle.  All rights reserved.
    ERROR:
    ORA-12541: TNS:no listener
    Enter user-name:
    ### Results from Wireshark : Redirect Data: (ADDRESS=(PROTOCOL=tcp)(HOST=x.x.x.243)(PORT=50045))
    
    
    Connecting from LISTENER_SCAN3 running on A
    -------------------------------------------
    Connection from SQLPLUS
    C:\>sqlplus system/oracle@dbname_srv_s_A_scan3.domain.com
    SQL*Plus: Release 11.2.0.1.0 Production on Fri Apr 5 11:04:42 2013
    Copyright (c) 1982, 2010, Oracle.  All rights reserved.
    ERROR:
    ORA-12541: TNS:no listener
    Enter user-name:
    ### Results from Wireshark : Redirect Data: (ADDRESS=(PROTOCOL=tcp)(HOST=x.x.x.241)(PORT=9317))
  • 3. Re: Redirect with wrong IP or PORT from SCAN LISTENER.
    Levi-Pereira Guru
    Currently Being Moderated
    I not found reason of this error happen, because service is registred on all SCAN listeners and show redirecting to node B.

    Can you enable NET trace and post here the trace result:

    I nice example how do that is on link below:
    http://www.pythian.com/blog/pythian-video-oracle-rac-vips-troubleshooting-connectivity-issues/
  • 4. Re: Redirect with wrong IP or PORT from SCAN LISTENER.
    Actitud Newbie
    Currently Being Moderated
    Hello Levi.

    Here is the result for the FAILED connection.
    Fatal NI connect error 12170.
    
      VERSION INFORMATION:
         TNS for 32-bit Windows: Version 11.2.0.1.0 - Production
         Windows NT TCP/IP NT Protocol Adapter for 32-bit Windows: Version 11.2.0.1.0 - Production
      Time: 08-APR-2013 10:51:57
      Tracing to file: T‚
      Tns error struct:
        ns main err code: 12535
        
    TNS-12535: TNS:operation timed out
        ns secondary err code: 12560
        nt main err code: 505
        
    TNS-00505: Operation timed out
        nt secondary err code: 60
        nt OS err code: 0
      Client address: <unknown>
    On the ora_# file, here is what I see:
    2013-04-08 10:51:38.450611 : nscall:connecting...
    2013-04-08 10:51:38.450681 : nsc2addr:(ADDRESS=(PROTOCOL=tcp)(HOST=*.*.*.243)(PORT=9317))
    Again, x.x.x.243 is the LISTENER_SCAN from node A, and port *9317* is one of the ports on the instance running on node B so, it will never be able to find it.


    BTW, no matter what I tried on the SQLNET tracing settings, I couldn't make to force them to be created on a particular location.
    I found the trace files inside something like this:
    C:\Users\my_id\Oracle\oradiag_my_id\diag\clients\user_my_id\host_4018301566_76\trace
    Thanks,
    Alex.

    Edited by: Actitud on Apr 8, 2013 11:03 AM
  • 5. Re: Redirect with wrong IP or PORT from SCAN LISTENER.
    Levi-Pereira Guru
    Currently Being Moderated
    Hi Alex,

    I'm deploying a RAC env to do some tests.

    Can you post here Listener.ora and SQLNET.ora of LSNR SCAN from nodes where problem happens.
  • 6. Re: Redirect with wrong IP or PORT from SCAN LISTENER.
    Actitud Newbie
    Currently Being Moderated
    Hello Levi.
    nodeA@/CRS_HOME> cat network/admin/listener.ora
    LISTENER_SCAN2=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN2))))                # line added by Agent
    LISTENER_SCAN3=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN3))))                # line added by Agent
    LISTENER=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER))))            # line added by Agent
    LISTENER_SCAN1=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1))))                # line added by Agent
    ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN1=ON                # line added by Agent
    ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER=ON              # line added by Agent
    ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN3=ON                # line added by Agent
    ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN2=ON                # line added by Agent
    I'm not using any sqlnet.ora though.

    Thanks,
    Alex.

    Edited:

    This could be helpful as well
    nodeA@/CRS_HOME> cat network/admin/endpoints_listener.ora
    LISTENER_NODEA=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=nodeA-vip)(PORT=1521))(ADDRESS=(PROTOCOL=TCP)(HOST=*.*.*.215)(PORT=1521)(IP=FIRST))))              # line added by Agent
    Edited by: Actitud on Apr 8, 2013 4:17 PM
  • 7. Re: Redirect with wrong IP or PORT from SCAN LISTENER.
    Levi-Pereira Guru
    Currently Being Moderated
    Hi Alex,

    I did some test and found more issue on this case.

    You are using POLICY MANAGED DATABASE, then Oracle Manage all instance automatically on demand without dba intervention. For the whole thing work well is mandatory use SCAN and VIP because these resource will manage it automatically the services and is this way how the client will find a Oracle Instance.

    It is important to understand that the dispatchers must spawn on the VIP addresses, in your case it's spawn on Node Address.


    You configured a SHARED SERVER which depend on of DISPATCHERS parameter to work.

    DISPATCHERS parameter is NOT Like LOCAL_LISTENER parameter which automatically updated by Database Agent. Then you must set DISPATCHERS manually using VIP of current Node on each Instance. As explained in note *How To Configure Shared Server Dispatchers For RAC Environment [ID 578524.1]*. BUT this will work only if you have a ADMIN-MANAGED database because instance is fixed on that node.

    POLICY-MANAGED database has not fixed instance on specific node and also has not a specific INSTANCE_NAME, because INSTANCE_NAME can be changed automatically on demand or when you change configuration of your SERVER POOL.

    I didn't found:
    How configure DISPATCHERS parameter when database is POLICY-MANAGED ? Oracle must give us a explanation about it... Can you open a SR?

    Another Issue:
    If you stop the service on SHARED SERVER with srvctl stop service or relocate service, the service will still registered with on the previous node listener and accepting connections.
    This service registered cannot be managed by the dba.(I can be missing something here)

    If this above is true, SINGLETON will not be a SINGLE service on SHARED SERVER and can be accepting connection from two or more nodes since they was relocated.


    I did some test about your problem and all succeeds. Even using NODENAME (i.e not VIP) the connection using SCAN from all nodes redirected to right node.


    Can you post here configuration (parameters) of your Shared Server?


    Regards,
    Levi Pereira
  • 8. Re: Redirect with wrong IP or PORT from SCAN LISTENER.
    Actitud Newbie
    Currently Being Moderated
    Hello Levi.

    I have open an SR yesterday and the answer I've got is (BTW, just asked them how to configure DISPATCHERS for PM databases):

    +"You need to make sure the dispatchers know where to find the local listener. The dispatcher parameter would look something like: DISPATCHERS="(PROTOCOL=TCP)(LISTENER=(ADDRESS=(PROTOCOL=tcp)(HOST=sales1-server)(PORT=1521)))(DISPATCHERS=2)""+

    OF course, as you stated this doesn't work with POLICY MANAGED databases IF the instance got relocated.
    It works if the instance is not moved though. I've tested it.

    Just as an example, I've picked up one of the databases on the cluster and set this:
    alter system set DISPATCHERS="(PROTOCOL=TCP)(LISTENER=(ADDRESS=(PROTOCOL=tcp)(HOST=node_A)(PORT=1521)))(DISPATCHERS=2)" sid='instance_2' scope=both;
    alter system set DISPATCHERS="(PROTOCOL=TCP)(LISTENER=(ADDRESS=(PROTOCOL=tcp)(HOST=node_B)(PORT=1521)))(DISPATCHERS=2)" sid='instance_1' scope=both;
    Again, this will NOT work if the instance #1 is started on node A or the other way arround. True and correct.

    Here are the results for local listeners and scan listeners services if that change is applied:

    IPs shown below:*
    *.*.*.221 is node_A-vip
    *.*.*.222 is node_B-vip


    SCAN LISTENERs*
    _==============_
    #### LISTENER_SCAN1 (node_B) for dbname ####
    ---------------------------------------------------------
    Service "dbname" has 2 instance(s).
      Instance "dbname_1", status READY, has 1 handler(s) for this service...
        Handler(s):
          "DEDICATED" established:1 refused:0 state:ready
             REMOTE SERVER
             (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=*.*.*.221)(PORT=1521)))
      Instance "dbname_2", status READY, has 1 handler(s) for this service...
        Handler(s):
          "DEDICATED" established:8 refused:0 state:ready
             REMOTE SERVER
             (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=*.*.*.222)(PORT=1521)))
    Service "dbname_srv_s" has 1 instance(s).
      Instance "dbname_2", status READY, has 1 handler(s) for this service...
        Handler(s):
          "DEDICATED" established:8 refused:0 state:ready
             REMOTE SERVER
             (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=*.*.*.222)(PORT=1521)))
    Service "dbname_srv_u" has 2 instance(s).
      Instance "dbname_1", status READY, has 1 handler(s) for this service...
        Handler(s):
          "DEDICATED" established:1 refused:0 state:ready
             REMOTE SERVER
             (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=*.*.*.221)(PORT=1521)))
      Instance "dbname_2", status READY, has 1 handler(s) for this service...
        Handler(s):
          "DEDICATED" established:8 refused:0 state:ready
             REMOTE SERVER
             (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=*.*.*.222)(PORT=1521)))
    
               
    #### LISTENER_SCAN2 (node_A) for dbname ####
    --------------------------------------------------------
    Service "dbname" has 2 instance(s).
      Instance "dbname_1", status READY, has 1 handler(s) for this service...
        Handler(s):
          "DEDICATED" established:0 refused:0 state:ready
             REMOTE SERVER
             (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=*.*.*.221)(PORT=1521)))
      Instance "dbname_2", status READY, has 1 handler(s) for this service...
        Handler(s):
          "DEDICATED" established:6 refused:0 state:ready
             REMOTE SERVER
             (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=*.*.*.222)(PORT=1521)))
    Service "dbname_srv_s" has 1 instance(s).
      Instance "dbname_2", status READY, has 1 handler(s) for this service...
        Handler(s):
          "DEDICATED" established:6 refused:0 state:ready
             REMOTE SERVER
             (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=*.*.*.222)(PORT=1521)))
    Service "dbname_srv_u" has 2 instance(s).
      Instance "dbname_1", status READY, has 1 handler(s) for this service...
        Handler(s):
          "DEDICATED" established:0 refused:0 state:ready
             REMOTE SERVER
             (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=*.*.*.221)(PORT=1521)))
      Instance "dbname_2", status READY, has 1 handler(s) for this service...
        Handler(s):
          "DEDICATED" established:6 refused:0 state:ready
             REMOTE SERVER
             (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=*.*.*.222)(PORT=1521)))
    
    
    #### LISTENER_SCAN3 (node_A) for dbname ####
    --------------------------------------------------------
    Service "dbname" has 2 instance(s).
      Instance "dbname_1", status READY, has 1 handler(s) for this service...
        Handler(s):
          "DEDICATED" established:0 refused:0 state:ready
             REMOTE SERVER
             (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=*.*.*.221)(PORT=1521)))
      Instance "dbname_2", status READY, has 1 handler(s) for this service...
        Handler(s):
          "DEDICATED" established:4 refused:0 state:ready
             REMOTE SERVER
             (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=*.*.*.222)(PORT=1521)))
    Service "dbname_srv_s" has 1 instance(s).
      Instance "dbname_2", status READY, has 1 handler(s) for this service...
        Handler(s):
          "DEDICATED" established:4 refused:0 state:ready
             REMOTE SERVER
             (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=*.*.*.222)(PORT=1521)))
    Service "dbname_srv_u" has 2 instance(s).
      Instance "dbname_1", status READY, has 1 handler(s) for this service...
        Handler(s):
          "DEDICATED" established:0 refused:0 state:ready
             REMOTE SERVER
             (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=*.*.*.221)(PORT=1521)))
      Instance "dbname_2", status READY, has 1 handler(s) for this service...
        Handler(s):
          "DEDICATED" established:4 refused:0 state:ready
             REMOTE SERVER
             (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=*.*.*.222)(PORT=1521)))
    LISTENERs*
    _=========_
    node_B
    ------
    Service "dbname" has 1 instance(s).
      Instance "dbname_2", status READY, has 3 handler(s) for this service...
        Handler(s):
          "D001" established:0 refused:0 current:0 max:1022 state:ready
             DISPATCHER <machine: node_B, pid: 8271>
             (ADDRESS=(PROTOCOL=tcp)(HOST=node_B.domain.com)(PORT=28054))
          "D000" established:0 refused:0 current:0 max:1022 state:ready
             DISPATCHER <machine: node_B, pid: 8269>
             (ADDRESS=(PROTOCOL=tcp)(HOST=node_B.domain.com)(PORT=35416))
          "DEDICATED" established:0 refused:0 state:ready
             LOCAL SERVER
    Service "dbname_srv_s" has 1 instance(s).
      Instance "dbname_2", status READY, has 3 handler(s) for this service...
        Handler(s):
          "D001" established:0 refused:0 current:0 max:1022 state:ready
             DISPATCHER <machine: node_B, pid: 8271>
             (ADDRESS=(PROTOCOL=tcp)(HOST=node_B.domain.com)(PORT=28054))
          "D000" established:0 refused:0 current:0 max:1022 state:ready
             DISPATCHER <machine: node_B, pid: 8269>
             (ADDRESS=(PROTOCOL=tcp)(HOST=node_B.domain.com)(PORT=35416))
          "DEDICATED" established:0 refused:0 state:ready
             LOCAL SERVER
    Service "dbname_srv_u" has 1 instance(s).
      Instance "dbname_2", status READY, has 3 handler(s) for this service...
        Handler(s):
          "D001" established:0 refused:0 current:0 max:1022 state:ready
             DISPATCHER <machine: node_B, pid: 8271>
             (ADDRESS=(PROTOCOL=tcp)(HOST=node_B.domain.com)(PORT=28054))
          "D000" established:0 refused:0 current:0 max:1022 state:ready
             DISPATCHER <machine: node_B, pid: 8269>
             (ADDRESS=(PROTOCOL=tcp)(HOST=node_B.domain.com)(PORT=35416))
          "DEDICATED" established:0 refused:0 state:ready
             LOCAL SERVER
               
    node_A
    ------
    Service "dbname" has 1 instance(s).
      Instance "dbname_1", status READY, has 3 handler(s) for this service...
        Handler(s):
          "D001" established:0 refused:0 current:0 max:1022 state:ready
             DISPATCHER <machine: node_A, pid: 31073>
             (ADDRESS=(PROTOCOL=tcp)(HOST=node_A.domain.com)(PORT=18586))
          "D000" established:1 refused:0 current:1 max:1022 state:ready
             DISPATCHER <machine: node_A, pid: 31069>
             (ADDRESS=(PROTOCOL=tcp)(HOST=node_A.domain.com)(PORT=26036))
          "DEDICATED" established:0 refused:0 state:ready
             LOCAL SERVER
    Service "dbname_srv_u" has 1 instance(s).
      Instance "dbname_1", status READY, has 3 handler(s) for this service...
        Handler(s):
          "D001" established:0 refused:0 current:0 max:1022 state:ready
             DISPATCHER <machine: node_A, pid: 31073>
             (ADDRESS=(PROTOCOL=tcp)(HOST=node_A.domain.com)(PORT=18586))
          "D000" established:1 refused:0 current:1 max:1022 state:ready
             DISPATCHER <machine: node_A, pid: 31069>
             (ADDRESS=(PROTOCOL=tcp)(HOST=node_A.domain.com)(PORT=26036))
          "DEDICATED" established:0 refused:0 state:ready
             LOCAL SERVER
    Just to clarify. Is my previous configuration working for you? If it's not the same configuration, what have you changed?

    Here is my current shared server parameters settings*
    alter system set dispatchers="(protocol=tcp)(dispatchers=3)";
    alter system set max_shared_servers=500 scope=both sid='*';
    alter system set processes=2000 scope=spfile sid='*';
    alter system set shared_servers=40 scope=both sid='*';
    alter system set shared_server_sessions=400 scope=both sid='*';
    Thanks,
    Alex.
  • 9. Re: Redirect with wrong IP or PORT from SCAN LISTENER.
    Levi-Pereira Guru
    Currently Being Moderated
    Check my test.
    [oracle@node01 ~]$  srvctl status database -d orcl
    Instance orcl_1 is running on node node01
    Instance orcl_2 is running on node node02
    
    [oracle@node01 ~]$  srvctl status service -d orcl -s inbound_single
    Service inbound_single is running on nodes: node01
    
    [oracle@node01 ~]$  srvctl config service -d orcl -s inbound_single
    Service name: inbound_single
    Service is enabled
    Server pool: sporcl
    Cardinality: SINGLETON
    Disconnect: false
    Service role: PRIMARY
    Management policy: AUTOMATIC
    DTP transaction: false
    AQ HA notifications: false
    Failover type: NONE
    Failover method: NONE
    TAF failover retries: 0
    TAF failover delay: 0
    Connection Load Balancing Goal: LONG
    Runtime Load Balancing Goal: NONE
    TAF policy specification: NONE
    Edition:
    Service is enabled on nodes:
    Service is disabled on nodes:
    
    [oracle@node01 ~]$  srvctl status scan_listener
    SCAN Listener LISTENER_SCAN1 is enabled
    SCAN listener LISTENER_SCAN1 is running on node node02
    SCAN Listener LISTENER_SCAN2 is enabled
    SCAN listener LISTENER_SCAN2 is running on node node01
    SCAN Listener LISTENER_SCAN3 is enabled
    SCAN listener LISTENER_SCAN3 is running on node node01
    
    [oracle@node01 ~]$  lsnrctl service LISTENER_SCAN2
    ...
    Service "inbound_single" has 1 instance(s).
      Instance "orcl_1", status READY, has 4 handler(s) for this service...
        Handler(s):
          "D002" established:1 refused:0 current:3 max:1022 state:ready
             DISPATCHER <machine: node01, pid: 11074>
             (ADDRESS=(PROTOCOL=tcp)(HOST=node01.oracle.com)(PORT=14910))
          "D001" established:3 refused:0 current:2 max:1022 state:ready
             DISPATCHER <machine: node01, pid: 11070>
             (ADDRESS=(PROTOCOL=tcp)(HOST=node01.oracle.com)(PORT=38283))
          "D000" established:5 refused:0 current:2 max:1022 state:ready
             DISPATCHER <machine: node01, pid: 11066>
             (ADDRESS=(PROTOCOL=tcp)(HOST=node01.oracle.com)(PORT=52178))
          "DEDICATED" established:0 refused:0 state:ready
             REMOTE SERVER
             (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=154.120.120.152)(PORT=1521)))
    ....
    
    [oracle@node01 ~]$  lsnrctl service LISTENER_SCAN3
    ...
    Service "inbound_single" has 1 instance(s).
      Instance "orcl_1", status READY, has 4 handler(s) for this service...
        Handler(s):
          "D002" established:0 refused:0 current:3 max:1022 state:ready
             DISPATCHER <machine: node01, pid: 11074>
             (ADDRESS=(PROTOCOL=tcp)(HOST=node01.oracle.com)(PORT=14910))
          "D001" established:0 refused:0 current:2 max:1022 state:ready
             DISPATCHER <machine: node01, pid: 11070>
             (ADDRESS=(PROTOCOL=tcp)(HOST=node01.oracle.com)(PORT=38283))
          "D000" established:0 refused:0 current:2 max:1022 state:ready
             DISPATCHER <machine: node01, pid: 11066>
             (ADDRESS=(PROTOCOL=tcp)(HOST=node01.oracle.com)(PORT=52178))
          "DEDICATED" established:0 refused:0 state:ready
             REMOTE SERVER
             (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=154.120.120.152)(PORT=1521)))
    ...
    
    [oracle@node02 ~]$  lsnrctl service LISTENER_SCAN1
    ...
    
    Service "inbound_single" has 1 instance(s).
      Instance "orcl_1", status READY, has 4 handler(s) for this service...
        Handler(s):
          "D002" established:0 refused:0 current:3 max:1022 state:ready
             DISPATCHER <machine: node01, pid: 11074>
             (ADDRESS=(PROTOCOL=tcp)(HOST=node01.oracle.com)(PORT=14910))
          "D001" established:1 refused:0 current:2 max:1022 state:ready
             DISPATCHER <machine: node01, pid: 11070>
             (ADDRESS=(PROTOCOL=tcp)(HOST=node01.oracle.com)(PORT=38283))
          "D000" established:2 refused:0 current:2 max:1022 state:ready
             DISPATCHER <machine: node01, pid: 11066>
             (ADDRESS=(PROTOCOL=tcp)(HOST=node01.oracle.com)(PORT=52178))
          "DEDICATED" established:0 refused:0 state:ready
             REMOTE SERVER
             (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=154.120.120.152)(PORT=1521)))
    ...
    
    
    [oracle@node01 ~]$  lsnrctl service
    ...
    Service "inbound_single" has 1 instance(s).
      Instance "orcl_1", status READY, has 4 handler(s) for this service...
        Handler(s):
          "D002" established:5 refused:0 current:3 max:1022 state:ready
             DISPATCHER <machine: node01, pid: 11074>
             (ADDRESS=(PROTOCOL=tcp)(HOST=node01.oracle.com)(PORT=14910))
          "D001" established:10 refused:0 current:2 max:1022 state:ready
             DISPATCHER <machine: node01, pid: 11070>
             (ADDRESS=(PROTOCOL=tcp)(HOST=node01.oracle.com)(PORT=38283))
          "D000" established:7 refused:0 current:3 max:1022 state:ready
             DISPATCHER <machine: node01, pid: 11066>
             (ADDRESS=(PROTOCOL=tcp)(HOST=node01.oracle.com)(PORT=52178))
          "DEDICATED" established:0 refused:0 state:ready
             LOCAL SERVER
    ...
    
    [oracle@node02 ~]$ lsnrctl service
    ...
    The command completed successfully
    
    
    NAME                                 TYPE        VALUE
    ------------------------------------ ----------- ------------------------------
    dispatchers                          string      (protocol=tcp)(dispatchers=3)
    max_shared_servers                   integer     500
    shared_server_sessions               integer     400
    shared_servers                       integer     40
    remote_listener                      string      node-scan.oracle.com:1521
    Client side
    C:\Users\Levi>nslookup node01.oracle.com
    Server:  UnKnown
    Address:  154.120.120.150
    
    Nome:    node01.oracle.com
    Address:  154.120.120.150
    
    
    C:\Users\Levi>nslookup node-scan.oracle.com
    Server:  UnKnown
    Address:  154.120.120.150
    
    Nome:    node-scan.oracle.com
    Addresses:  154.120.120.105
              154.120.120.103
              154.120.120.104
    
    
    C:\Users\Levi>tnsping inbound_single
    
    TNS Ping Utility for 64-bit Windows: Version 11.2.0.3.0 - Production on 10-ABR-2013 17:36:09
    
    Copyright (c) 1997, 2011, Oracle.  All rights reserved.
    
    Used parameter files:
    C:\app\oracle\product\11.2.0\dbhome_1\network\admin\sqlnet.ora
    
    Used TNSNAMES adapter to resolve the alias
    Attempting to contact (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = node-scan)(PORT = 1521))) (CONNECT_DATA = (SERVER = SHARED) (SERVICE_NAME = inbound_single)))
    OK (70 msec)
    
    
    
    C:\Users\Levi>tnsping inbound_single_a
    
    TNS Ping Utility for 64-bit Windows: Version 11.2.0.3.0 - Production on 10-ABR-2013 17:37:07
    
    Copyright (c) 1997, 2011, Oracle.  All rights reserved.
    
    Used parameter files:
    C:\app\oracle\product\11.2.0\dbhome_1\network\admin\sqlnet.ora
    
    
    Used TNSNAMES adapter to resolve the alias
    Attempting to contact (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = 154.120.120.103)(PORT = 1521))) (CONNECT_DATA = (SERVER = SHARED) (SERVICE_NAME = inbound_single)))
    OK (10 msec)
    
    
    C:\Users\Levi>tnsping inbound_single_b
    
    TNS Ping Utility for 64-bit Windows: Version 11.2.0.3.0 - Production on 10-ABR-2013 17:37:46
    
    Copyright (c) 1997, 2011, Oracle.  All rights reserved.
    
    Used parameter files:
    C:\app\oracle\product\11.2.0\dbhome_1\network\admin\sqlnet.ora
    
    
    Used TNSNAMES adapter to resolve the alias
    Attempting to contact (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = 154.120.120.104)(PORT = 1521))) (CONNECT_DATA = (SERVER = SHARED) (SERVICE_NAME = inbound_single)))
    OK (0 msec)
    
    C:\Users\Levi>tnsping inbound_single_c
    
    TNS Ping Utility for 64-bit Windows: Version 11.2.0.3.0 - Production on 10-ABR-2013 17:38:14
    
    Copyright (c) 1997, 2011, Oracle.  All rights reserved.
    
    Used parameter files:
    C:\app\oracle\product\11.2.0\dbhome_1\network\admin\sqlnet.ora
    
    
    Used TNSNAMES adapter to resolve the alias
    Attempting to contact (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = 154.120.120.105)(PORT = 1521))) (CONNECT_DATA =
     (SERVER = SHARED) (SERVICE_NAME = inbound_single)))
    OK (0 msec)
    
    
    C:\Users\Levi>sqlplus test/teste@inbound_single
    
    SQL*Plus: Release 11.2.0.3.0 Production on Qua Abr 10 17:38:35 2013
    
    Copyright (c) 1982, 2011, Oracle.  All rights reserved.
    
    
    Connected to:
    Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - Production
    With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
    Data Mining and Real Application Testing options
    
    SQL> show parameter instance_name
    
    NAME                                 TYPE        VALUE
    ------------------------------------ ----------- ------------------------------
    instance_name                        string      orcl_1
    SQL>
    
    
    C:\Users\Levi>sqlplus test/teste@inbound_single_a
    
    SQL*Plus: Release 11.2.0.3.0 Production on Qua Abr 10 17:38:40 2013
    
    Copyright (c) 1982, 2011, Oracle.  All rights reserved.
    
    
    Connected to:
    Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - Production
    With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
    Data Mining and Real Application Testing options
    
    SQL> show parameter instance_name
    
    NAME                                 TYPE        VALUE
    ------------------------------------ ----------- ------------------------------
    instance_name                        string      orcl_1
    SQL>
    
    C:\Users\Levi>sqlplus test/teste@inbound_single_b
    
    SQL*Plus: Release 11.2.0.3.0 Production on Qua Abr 10 17:38:55 2013
    
    Copyright (c) 1982, 2011, Oracle.  All rights reserved.
    
    
    Connected to:
    Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - Production
    With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
    Data Mining and Real Application Testing options
    
    SQL> show parameter instance_name
    
    NAME                                 TYPE        VALUE
    ------------------------------------ ----------- ------------------------------
    instance_name                        string      orcl_1
    SQL>
    
    
    C:\Users\Levi>sqlplus test/teste@inbound_single_c
    
    SQL*Plus: Release 11.2.0.3.0 Production on Qua Abr 10 17:39:51 2013
    
    Copyright (c) 1982, 2011, Oracle.  All rights reserved.
    
    
    Connected to:
    Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - Production
    With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
    Data Mining and Real Application Testing options
    
    SQL> show parameter instance_name
    
    NAME                                 TYPE        VALUE
    ------------------------------------ ----------- ------------------------------
    instance_name                        string      orcl_1
    SQL>
  • 10. Re: Redirect with wrong IP or PORT from SCAN LISTENER.
    Actitud Newbie
    Currently Being Moderated
    Hello Levi.

    I've checked it and compare it with my settings, and it's pretty much the same.

    New data! (maybe good news)

    1 - I've tested it from another server (as a client) instead of my box and it works with each SCAN IP and with the cluster name :P
    2 - SR with Oracle is not being productive. I've got things like " I am not sure what you mean by "Policy Managed" databases." and "dispatchers don't register with the scan listeners"

    This is a client side problem in some cases.
    I'll do some other tests and let you know.

    Thanks for all your time.
    Alex.
  • 11. Re: Redirect with wrong IP or PORT from SCAN LISTENER.
    Levi-Pereira Guru
    Currently Being Moderated
    2 - SR with Oracle is not being productive. I've got things like " I am not sure what you mean by "Policy Managed" databases." and "dispatchers don't register with the scan listeners"
    Don't waste time with dumb question..Escalate it.

    *How To Escalate a Service Request (SR) with Oracle Support Services [ID 199389.1]*
  • 12. Re: Redirect with wrong IP or PORT from SCAN LISTENER.
    Actitud Newbie
    Currently Being Moderated
    "My Oracle Support is currently experiencing an unscheduled outage.
    We are working to resolve this outage as quickly as possible. If you have an urgent issue please call your local support center. Click here for a list of Oracle Support phone numbers. Click here for a list of CRM On Demand Support phone numbers.
    Thank you in advance for your patience, and we apologize for any inconvenience.
    The My Oracle Support Team"

    Nothing else can be wrong.
    I guess my SR blow it all from the inside somehow :P

    Thanks Levi and I'll let you know if I can find the solution.
    Alex.
  • 13. Re: Redirect with wrong IP or PORT from SCAN LISTENER.
    Actitud Newbie
    Currently Being Moderated
    Levi.

    I'm still dealing with the Oracle SR.

    Question: I'm being able to connect using scan name or any of the scan listeners (IPs) from inside all the servers on the same datacenter than the cluster but NOT from developers laptops. Does that tells you anything?
    Can you test your cluster from a remote location? (if you still have it).

    Thanks!
    Alex.
  • 14. Re: Redirect with wrong IP or PORT from SCAN LISTENER.
    Actitud Newbie
    Currently Being Moderated
    Levi.

    I just noticed I didn't updated this topic. I'm very sorry.
    The issue is finally resolved!

    This was a major issue across the company but the thing is, we have different environments with different setups and teams taking care of so, a lot of miss communication happened.

    Basically, Prod and Cert have firewalls with ports opened (that's for dispatchers and so). I've request networking team about the firewalls we have and they say we had no restrictions so I just stop considering a "firewall issue". That as my mistake.

    Besides we don't have any restrictions on the firewall we have, there are some features which after long time of investigation at networking level (using Wireshark to read packages and see the redirect results, among other things), we could see something in the middle was messing it up.

    That was, "SQLNet Packet Inspection" feature enabled by default!. I've requested to disable that and everything started to work after that.

    Of course, Oracle Support didn't helped AT ALL which is really bad since we spend lots of money on support. I told them about the firewall at the end and they say "contact your networking team" instead of providing any sort of clue to this issue.

    Anyhow, Thanks a LOT Levi for your dedication to my issue.

    Thanks from Uruguay.
    Alex.
1 2 Previous Next

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points