This discussion is archived
11 Replies Latest reply: Sep 25, 2012 4:48 AM by MahirM.Quluzade RSS

RFS not running on my 1st and 3rd instances - physical standby

963313 Newbie
Currently Being Moderated
Hi,

In one of my physical standby db -3 node RAC, rfs process is not running on 1st and 3rd instances, however 14 rfs processes are running on the 2nd instances fetching all 3 threads.
1)Is rfs instance specific?
2)How can we increase the rfs process is it dependent on any parameter?
log_archive_max_processes on primary and standby has been set to 8 on all 3 instances, but i have on physical standby 1st inst: rfs-0,2nd inst: rfs-14 and 3rd inst: rfs -0

Regards,
Sebastian
  • 1. Re: RFS not running on my 1st and 3rd instances - physical standby
    CKPT Guru
    Currently Being Moderated
    960310 wrote:
    Hi,

    In one of my physical standby db -3 node RAC, rfs process is not running on 1st and 3rd instances, however 14 rfs processes are running on the 2nd instances fetching all 3 threads.
    1)Is rfs instance specific?
    2)How can we increase the rfs process is it dependent on any parameter?
    log_archive_max_processes on primary and standby has been set to 8 on all 3 instances, but i have on physical standby 1st inst: rfs-0,2nd inst: rfs-14 and 3rd inst: rfs -0

    Regards,
    Sebastian
    RFS process is specific to standby database to receive redo data. Every primary database of all active instances will have its own thread and its own LNS process to transmit redo to the standby database. These redos are transmitted by the LNS and they will be received by RFS process and they will be written into Standby redo log files.
    1)Is rfs instance specific?
    for your questions, yes RFS specific to Standby Instance, If you have 5 standby database, you will have 5 RFS process on each standby database.
    2)How can we increase the rfs process is it dependent on any parameter?
    log_archive_max_processes on primary and standby has been set to 8 on all 3 instances, but i have on physical standby 1st inst: rfs-0,2nd inst: rfs-14 and 3rd inst: rfs -0
    Here is one example, I have a RAC environment of two Instances.
    SQL> select count(*) from gv$instance;
    
      COUNT(*)
    ----------
              "2"
    SQL>
    And Log archive maximum processes been set to 10
    SQL> show parameter log_archive_max_processes
    NAME                                 TYPE                                             VALUE
    ------------------------------------ --------------------------------       ------------------------------
    "log_archive_max_processes            integer                               10"
    SQL>
    Lets Assume A=10

    Here each instance will have one thread as we discussed, at one time one of the LNS process is sending redo to the standby database directly to Remote file server which wirtes them into standby redo log files.
    SQL> select thread#,PROCESS, STATUS, THREAD#, SEQUENCE#, BLOCK#, BLOCKS FROM GV$MANAGED_STANDBY where process='LNS';
    
       THREAD# PROCESS                     STATUS                                  THREAD#  SEQUENCE#     BLOCK#     BLOCKS
    ---------- --------------------------- ------------------------------------ ---------- ---------- ---------- ----------
             1 LNS                         WRITING                                       1     271871     745474      16384
             2 LNS                         WRITING                                       2     336857        261         53
    
    SQL>
    Lets Assume B=2
    Note:- LNS process will be active on each instance of RAC database.

    Now Let's see how many processes are active and how many are Idle and how many processes are writing in standby redo log files.
    Here is calculation.

    >
    RFS processes on standby = (Log_archive_max_processes of primary) * (Number_of_Threads)
    RFS Processes on standby = A * B
    RFS Processes on standby = 10 * 2 = 20
    >

    Now come to standby database and see how many processes with RFS are connected and how many are idle, and how many RFS processes are writing on standby of each thread.
    SQL> select thread#,PROCESS, STATUS, THREAD#, SEQUENCE#, BLOCK#, BLOCKS FROM V$MANAGED_STANDBY where process='RFS';
    
       THREAD# PROCESS                     STATUS                                  THREAD#  SEQUENCE#     BLOCK#     BLOCKS
    ---------- --------------------------- ------------------------------------ ---------- ---------- ---------- ----------
             0 RFS                         IDLE                                          0          0          0          0
             0 RFS                         IDLE                                          0          0          0          0
             0 RFS                         IDLE                                          0          0          0          0
             0 RFS                         IDLE                                          0          0          0          0
             0 RFS                         IDLE                                          0          0          0          0
             2 RFS                         IDLE                                          2     336855       2060       1274
             0 RFS                         IDLE                                          0          0          0          0
             0 RFS                         IDLE                                          0          0          0          0
             0 RFS                         IDLE                                          0          0          0          0
             0 RFS                         IDLE                                          0          0          0          0
             0 RFS                         IDLE                                          0          0          0          0
    
       THREAD# PROCESS                     STATUS                                  THREAD#  SEQUENCE#     BLOCK#     BLOCKS
    ---------- --------------------------- ------------------------------------ ---------- ---------- ---------- ----------
             0 RFS                         IDLE                                          0          0          0          0
             0 RFS                         IDLE                                          0          0          0          0
             0 RFS                         IDLE                                          0          0          0          0
             1 RFS                         IDLE                                          1     271869     757737      16384
             0 RFS                         IDLE                                          0          0          0          0
             0 RFS                         IDLE                                          0          0          0          0
             0 RFS                         IDLE                                          0          0          0          0
             0 RFS                         IDLE                                          0          0          0          0
             0 RFS                         IDLE                                          0          0          0          0
    
    20 rows selected.
    
    SQL>
    So even if you have number of RFS process, once one RFS will keep writes only on one sequence, Once that is full then another RFS process will be assigned to new stanbdy log sequence.


    Here the relation ship between LNS & RFS is below.
    Always from One instance there will be redo send by LNS process to the standby database using RFS process to Standby redo log files, Behind logic is Even in single instance production database only one of the online redo log will be in "CURRENT" remaining can be either "ACTIVE" or "INACTIVE" that mean only one online redo log can ready to hold the live data, So here communication is One-to-One from Primary Online redo logs to Standby Redo log files. So always Only one RFS process will be actively writing in standby of each thread.

    Of course there can be more RFS processes will be active, If in case there is any GAP between primary and standby , Then RFS process will keep try to fetch those archives. So the relationship will be from n-to-n between Log_archive_max_processes and RFS processes.
    Note:- If you have any Lag of archive logs, then network server try to fetch using FAL_SERVER and then number of RFS active processes may vary.

    When there is Standby is out of SYNC with primary Database, Then the status and the number of active RFS processes may vary because they are surviving archives to ship from primary to standby database.
    PROCES CLIENT_PROCESS           PROCES STATUS        THREAD#  SEQUENCE#     BLOCK#     BLOCKS
    ------ ------------------------ ------ ---------- ---------- ---------- ---------- ----------
    RFS    UNKNOWN                  RFS    IDLE                0          0          0          0
    RFS    UNKNOWN                  RFS    RECEIVING           0          0          0          0
    RFS    UNKNOWN                  RFS    IDLE                0          0          0          0
    RFS    UNKNOWN                  RFS    IDLE                0          0          0          0
    RFS    UNKNOWN                  RFS    IDLE                0          0          0          0
    RFS    LGWR                     RFS    IDLE                2     336860      53811      16384
    RFS    UNKNOWN                  RFS    IDLE                0          0          0          0
    RFS    UNKNOWN                  RFS    WRITING             1     271880     532481      20480
    RFS    UNKNOWN                  RFS    IDLE                0          0          0          0
    RFS    UNKNOWN                  RFS    IDLE                0          0          0          0
    RFS    UNKNOWN                  RFS    IDLE                0          0          0          0
    
    PROCES CLIENT_PROCESS           PROCES STATUS        THREAD#  SEQUENCE#     BLOCK#     BLOCKS
    ------ ------------------------ ------ ---------- ---------- ---------- ---------- ----------
    RFS    UNKNOWN                  RFS    IDLE                0          0          0          0
    RFS    UNKNOWN                  RFS    IDLE                0          0          0          0
    RFS    UNKNOWN                  RFS    IDLE                0          0          0          0
    RFS    LGWR                     RFS    IDLE                1     271884     270338      16384
    RFS    UNKNOWN                  RFS    IDLE                1     271883     163841      20480
    RFS    UNKNOWN                  RFS    IDLE                0          0          0          0
    RFS    UNKNOWN                  RFS    WRITING             1     271882     286721      20480
    RFS    UNKNOWN                  RFS    IDLE                0          0          0          0
    RFS    UNKNOWN                  RFS    IDLE                1     271879     737281      20480
    but i have on physical standby 1st inst: rfs-0,2nd inst: rfs-14 and 3rd inst: rfs -0
    Well, Always only one Physical standby will be active in Recovery mode. Can you please confirm whether your second standby instance is running MRP, and remaining instances must be in IDLE, so number of processes on other instances are Null.
    Pleas use above queries and compare once if you have questions let us know.

    Thank you.

    Edited by: CKPT on Sep 21, 2012 9:54 AM
  • 2. Re: RFS not running on my 1st and 3rd instances - physical standby
    963313 Newbie
    Currently Being Moderated
    Thanks for the reply.

    1) Since rfs is instance specific, in my case rfs is running only on the 2nd instance and 1st and 3rd no rfs processes are running.
    The primary and standby are in sync and the mrp runs from 1st instance.

    =
    INST_ID PROCESS THREAD# SEQUENCE# STATUS CLIENT_P PID
    ---------- --------- ---------- ---------- ------------ -------- ----------
    1 ARCH 0 0 CONNECTED ARCH 21863
    1 ARCH 3 554 CLOSING ARCH 21867
    1 ARCH 0 0 CONNECTED ARCH 21871
    1 ARCH 0 0 CONNECTED ARCH 21875
    1 ARCH 3 549 CLOSING ARCH 21879
    1 ARCH 0 0 CONNECTED ARCH 21883
    1 ARCH 0 0 CONNECTED ARCH 21887
    1 ARCH 0 0 CONNECTED ARCH 21891
    2 ARCH 0 0 CONNECTED ARCH 9754
    2 ARCH 1 809 CLOSING ARCH 9774
    2 ARCH 2 633 CLOSING ARCH 9770

    INST_ID PROCESS THREAD# SEQUENCE# STATUS CLIENT_P PID
    ---------- --------- ---------- ---------- ------------ -------- ----------
    2 ARCH 1 810 CLOSING ARCH 9766
    2 ARCH 3 579 CLOSING ARCH 9762
    2 ARCH 2 635 CLOSING ARCH 9758
    2 ARCH 3 578 CLOSING ARCH 9750
    2 ARCH 2 634 CLOSING ARCH 9746
    3 ARCH 2 575 CLOSING ARCH 19355
    3 ARCH 2 560 CLOSING ARCH 19359
    3 ARCH 0 0 CONNECTED ARCH 19363
    3 ARCH 2 574 CLOSING ARCH 19367
    3 ARCH 0 0 CONNECTED ARCH 19371
    3 ARCH 3 527 CLOSING ARCH 19375

    INST_ID PROCESS THREAD# SEQUENCE# STATUS CLIENT_P PID
    ---------- --------- ---------- ---------- ------------ -------- ----------
    3 ARCH 2 547 CLOSING ARCH 19379
    3 ARCH 3 470 CLOSING ARCH 19383
    1 MRP0 2 636 WAIT_FOR_LOG N/A 25507
    2 RFS 0 0 IDLE ARCH 18501
    2 RFS 0 0 IDLE UNKNOWN 31522
    2 RFS 2 636 IDLE LGWR 18505 --------------------------------> thread 2
    2 RFS 0 0 IDLE UNKNOWN 12833
    2 RFS 0 0 IDLE UNKNOWN 12817
    2 RFS 0 0 IDLE ARCH 31931
    2 RFS 0 0 IDLE UNKNOWN 2610
    2 RFS 3 580 IDLE LGWR 2605 ---------------------------------->thread 3

    INST_ID PROCESS THREAD# SEQUENCE# STATUS CLIENT_P PID
    ---------- --------- ---------- ---------- ------------ -------- ----------
    2 RFS 0 0 IDLE ARCH 2639
    2 RFS 0 0 IDLE UNKNOWN 31530
    2 RFS 1 811 IDLE LGWR 31526-------------------------------------->thread 1
    2 RFS 0 0 IDLE UNKNOWN 2614
    2 RFS 0 0 IDLE UNKNOWN 18509
    2 RFS 0 0 IDLE UNKNOWN 14607

    =

    Also, as per the RFS calculation
    =
    RFS processes on standby = (Log_archive_max_processes of primary) * (Number_of_Threads)
    RFS Processes on standby = A * B,
    here my A -> 8 and B->3
    so, RFS Processes on standby = 8*3 = 24

    However i'm able to see only 14 processes ( from the 2nd instance).

    Could you please explain this?


    2) Does RFS process on the standby relocate, if so under which conditions?

    As these are my findings,

    2605,31526,18505 --> these are the pid's

    =
    INST_ID PROCESS THREAD# SEQUENCE# STATUS CLIENT_P PID
    ---------- --------- ---------- ---------- ------------ -------- ----------
    2 RFS 1 811 IDLE LGWR 31526
    2 RFS 2 636 IDLE LGWR 18505
    2 RFS 3 580 IDLE LGWR 2605

    [oracle@swedcvfrien02 ~]$ ps -ef |grep -i 2605 |grep -i local
    oracle 2605 1 0 Sep09 ? 00:03:20 oracleetrspe2 (LOCAL=NO)
    [oracle@swedcvfrien02 ~]$ ps -ef |grep -i 31526 |grep -i local
    oracle 31526 1 0 Sep14 ? 00:02:07 oracleetrspe2 (LOCAL=NO)
    [oracle@swedcvfrien02 ~]$ ps -ef |grep -i 18505 |grep -i local
    oracle 18505 1 0 Sep12 ? 00:02:41 oracleetrspe2 (LOCAL=NO)

    standby:
    =

    OPEN_MODE DATABASE_ROLE INSTANCE_NUMBER startup time
    -------------------- ---------------- --------------- -----------------------------
    READ ONLY WITH APPLY PHYSICAL STANDBY 1 31-AUG-2012:12:39:11
    READ ONLY WITH APPLY PHYSICAL STANDBY 2 31-AUG-2012:12:39:11
    READ ONLY WITH APPLY PHYSICAL STANDBY 3 31-AUG-2012:12:39:11

    primary:
    =
    OPEN_MODE DATABASE_ROLE INSTANCE_NUMBER startup time
    -------------------- ---------------- --------------- -----------------------------
    READ WRITE PRIMARY 1 30-AUG-2012:09:11:22
    READ WRITE PRIMARY 2 30-AUG-2012:17:04:20
    READ WRITE PRIMARY 3 09-SEP-2012:17:25:41


    =
    Does restarting an instance on primary affects the rfs process on standby?

    Thanks in advances
    Regards,
    Sebastian
  • 3. Re: RFS not running on my 1st and 3rd instances - physical standby
    CKPT Guru
    Currently Being Moderated
    1) Since rfs is instance specific, in my case rfs is running only on the 2nd instance and 1st and 3rd no rfs processes are running. The primary and standby are in sync and the mrp runs from 1st instance.
    I think you have not read completely, I said MRP will run only any one of the instance. Can you confirm that your primary has 3 node and standby has 3 nodes?
    And RFS process will be served only on the instance where recovery is in progress, if there is any other standby instances are in idle then RFS process wont write on standby because recovery will be performed only one of the standby instance.

    However i'm able to see only 14 processes ( from the 2nd instance). Could you please explain this?
    Well, i said total number of RFS processes applicable for the calculation what i have mentioned. You can use below query. The calculation still works.
    SQL> select count(*) from v$managed_standby where process='RFS';
    
      COUNT(*)
    ----------
            20
    SQL>
    2) Does RFS process on the standby relocate, if so under which conditions?
    I didnt understand this question completely. You mean relocate of RFS process from one standby to another standby?
    Usually NO, if you stop MRP and start MRP in another instance of standby then RFS processes will be active in new instance.
    BTW, If it is clustered standby database, how many nodes it have?

    Does restarting an instance on primary affects the rfs process on standby?
    It wont affect, because standby FAL_SERVER parameter will keep ping to the service identifier you have mentioned, whenever primary is accessible then it will fetches archives/redo automatically based on the sequence and recovery status.
  • 4. Re: RFS not running on my 1st and 3rd instances - physical standby
    963313 Newbie
    Currently Being Moderated
    Thanks for the reply.

    1)Can you confirm that your primary has 3 node and standby has 3 nodes?

    Yes both primary - 3 node RAC
    standby - 3 Node RAC

    2)And RFS process will be served only on the instance where recovery is in progress, if there is any other standby instances are in idle then RFS process wont write on standby because recovery will be performed only one of the standby instance.

    True, kindly help me to understand and correct if i'm wrong.

    =
    COUNT(*) PROCESS STATUS INST_ID
    ---------- --------- ------------ ----------
    2 ARCH CLOSING 1
    6 ARCH CONNECTED 1
    1 MRP0 WAIT_FOR_LOG 1
    7 ARCH CLOSING 2
    1 ARCH CONNECTED 2
    14 RFS IDLE 2
    6 ARCH CLOSING 3
    2 ARCH CONNECTED 3

    =

    In the above output, RFS processes from 2nd instance will be serving the 1st instance since the mrp is running from instance 1?

    3) Calculation seems to be different in my case. -- using gv$managed_standby

    =
    SQL> select count(*) from gv$managed_standby where process='RFS';

    COUNT(*)
    ----------
    14
    =

    4)Usually NO, if you stop MRP and start MRP in another instance of standby then RFS processes will be active in new instance.

    Actually i tried first stopping the mrp in the 3rd instance and starting the mrp in the 1st instance, but no RFS process was/is active in the 1st instance ( please find the details from the above output: question 2)

    Regards,
    Sebastian
  • 5. Re: RFS not running on my 1st and 3rd instances - physical standby
    CKPT Guru
    Currently Being Moderated
    1)Can you confirm that your primary has 3 node and standby has 3 nodes?

    Yes both primary - 3 node RAC
    standby - 3 Node RAC
    Guess is correct.
    2)And RFS process will be served only on the instance where recovery is in progress, if there is any other standby instances are in idle then RFS process wont write on standby because recovery will be performed only one of the standby instance.

    True, kindly help me to understand and correct if i'm wrong.

    =
    COUNT(*) PROCESS STATUS INST_ID
    ---------- --------- ------------ ----------
    2 ARCH CLOSING 1
    6 ARCH CONNECTED 1
    1 MRP0 WAIT_FOR_LOG 1
    7 ARCH CLOSING 2
    1 ARCH CONNECTED 2
    14 RFS IDLE 2
    6 ARCH CLOSING 3
    2 ARCH CONNECTED 3

    =

    In the above output, RFS processes from 2nd instance will be serving the 1st instance since the mrp is running from instance 1?
    4)Usually NO, if you stop MRP and start MRP in another instance of standby then RFS processes will be active in new instance.

    Actually i tried first stopping the mrp in the 3rd instance and starting the mrp in the 1st instance, but no RFS process was/is active in the 1st instance ( please find the details from the above output: question 2)
    When any session is connected, even from remote server using oracle net service, They wont be disconnected until you kill or bounce the standby database. I suggest you to bounce the instance 2. and then see where RFS processes are connected again.

    3) Calculation seems to be different in my case. -- using gv$managed_standby

    =
    SQL> select count(*) from gv$managed_standby where process='RFS';

    COUNT(*)
    ----------
    14
    What is LOG_ARCHIVE_MAX_PROCESSES parameter value across all the instances of clustered database?
    SQL>  SELECT name, display_value FROM gv$parameter where name='log_archive_max_processes';
    
    NAME                           DISPL
    ------------------------------ -----
    log_archive_max_processes      10
    log_archive_max_processes      10
    
    SQL>
    BTW, Its interesting thread. Thanks for sharing.
  • 6. Re: RFS not running on my 1st and 3rd instances - physical standby
    963313 Newbie
    Currently Being Moderated
    Thanks for the quick response.
    log_archive_max_processes=8 (cluster wide) on the primary.

    Since standby is used for reporting(read only), we need downtime to proceed with standby 2nd instance restart.

    Regards,
    Sebastian
  • 7. Re: RFS not running on my 1st and 3rd instances - physical standby
    MahirM.Quluzade Guru
    Currently Being Moderated
    Hi Sebastian.

    I read your and my friend CKPT's all replys.
    This is very interesting threa, thanks for sharing.

    Can you check tnsnames which standby service you are using on primary for send archived logs ? Did you use scan name for standby database?


    Regards
    Mahir M. Quluzade
  • 8. Re: RFS not running on my 1st and 3rd instances - physical standby
    963313 Newbie
    Currently Being Moderated
    Hi Mahir,

    We are using vip and not scan ip to define the service of the standby.
    Also checked the tns entries(tnsping confirmed) across all the 3 nodes and its the same.

    We tried restarting the standby and this time, we were able to see 24 RFS process and the calculation part we are fine, however all 24 processes has been started from node 2 only even after restart.

    Regards,
    Sebastian
  • 9. Re: RFS not running on my 1st and 3rd instances - physical standby
    MahirM.Quluzade Guru
    Currently Being Moderated
    HI Sebastian!

    I think now, working fine, because you have 3 nodes and Log_archive_max_processes of primary is equal 8.
    RFS processes on standby = (Log_archive_max_processes of primary) * (Number_of_Threads)  = 3*8 = 24 
    and thanks for interesting thread.


    Best Regards
    Mahir M. Quluzade
    www.mahir-quluzade.com
  • 10. Re: RFS not running on my 1st and 3rd instances - physical standby
    963313 Newbie
    Currently Being Moderated
    Thanks Mahir, rasied an SR with oracle for this.

    Regards,
    Sebastian
  • 11. Re: RFS not running on my 1st and 3rd instances - physical standby
    MahirM.Quluzade Guru
    Currently Being Moderated
    You are welcome !


    Thanks
    Mahir


    PS:Please dont forget to change thread status to answered if it possible when you are belive your thread has been answered.

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points