13 Replies Latest reply on Jun 20, 2012 11:58 AM by User10313095-Oracle

    Datapump process cannot connect to REMOTE HOST.

    VIRU
      hi,

      I am using ORACLE DATABASE 11g on 2 servers. Both have IBM AIX OS in it. I am trying to implement Golden Gate(GG) on it. The scenario is as follows :-

      Server 1 :-

      I have created 1 extract and 1 datapump processes.
      GGSCI (JLFLXFC1) 228> info all
      
      Program     Status      Group       Lag           Time Since Chkpt
      
      MANAGER     RUNNING
      EXTRACT     RUNNING     R2B         00:00:00      00:00:03
      EXTRACT     ABENDED     R2BDP       00:00:00      00:06:40
      The datapump process gets aborted after a few tries to connect to the remote host server. The log of datapump is as follows :-
      GGSCI (JLFLXFC1) 230> view report r2bdp
      
      
      ***********************************************************************
                       Oracle GoldenGate Capture for Oracle
              Version 11.1.1.1 OGGCORE_11.1.1_PLATFORMS_110421.2040
        AIX 5L, ppc, 64bit (optimized), Oracle 11g on Apr 22 2011 03:25:30
      
      Copyright (C) 1995, 2011, Oracle and/or its affiliates. All rights reserved.
      
      
                          Starting at 2012-06-14 15:13:07
      ***********************************************************************
      
      Operating System Version:
      AIX
      Version 5, Release 3
      Node: JLFLXFC1
      Machine: 00C576D24C00
                               soft limit   hard limit
      Address Space Size   :    unlimited    unlimited
      Heap Size            :    unlimited    unlimited
      File Size            :    unlimited    unlimited
      CPU Time             :    unlimited    unlimited
      
      Process id: 278676
      
      Description:
      
      ***********************************************************************
      **            Running with the following parameters                  **
      ***********************************************************************
      EXTRACT R2BDP
      --setenv (ORACLE_SID=NGPR2B)
      USERID ggs_owner, PASSWORD *********
      RMTHOST FIFLX595,MGRPORT 7809
      RMTTRAIL /data01/GoldenGate/dirdat/NGPR2B/rt
      PASSTHRU
      TABLE R2B.*;
      
      CACHEMGR virtual memory values (may have been adjusted)
      CACHEBUFFERSIZE:                         64K
      CACHESIZE:                                8G
      CACHEBUFFERSIZE (soft max):               4M
      CACHEPAGEOUTSIZE (normal):                4M
      PROCESS VM AVAIL FROM OS (min):          16G
      CACHESIZEMAX (strict force to disk):  13.99G
      
      Database Version:
      Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
      PL/SQL Release 11.2.0.3.0 - Production
      CORE    11.2.0.3.0      Production
      TNS for IBM/AIX RISC System/6000: Version 11.2.0.3.0 - Production
      NLSRTL Version 11.2.0.3.0 - Production
      
      Database Language and Character Set:
      NLS_LANG environment variable specified has invalid format, default value will be used.
      NLS_LANG environment variable not set, using default value AMERICAN_AMERICA.US7ASCII.
      NLS_LANGUAGE     = "AMERICAN"
      NLS_TERRITORY    = "AMERICA"
      NLS_CHARACTERSET = "AL32UTF8"
      
      Warning: your NLS_LANG setting does not match database server language setting.
      Please refer to user manual for more information.
      Opened trail file /data1/GoldenGate/dirdat/R2B/lt000000 at 2012-06-14 15:13:07
      
      2012-06-14 15:13:07  WARNING OGG-01015  Positioning with begin time: Jun 14, 2012 3:13:04 PM, waiting for data: at extseqno 0, extrba 0.
      
      2012-06-14 15:13:37  WARNING OGG-01223  TCP/IP error 79 (Connection refused).
      
      2012-06-14 15:14:17  WARNING OGG-01223  TCP/IP error 79 (Connection refused).
      
      2012-06-14 15:14:57  WARNING OGG-01223  TCP/IP error 79 (Connection refused).
      
      2012-06-14 15:15:37  WARNING OGG-01223  TCP/IP error 79 (Connection refused).
      
      2012-06-14 15:16:17  WARNING OGG-01223  TCP/IP error 79 (Connection refused).
      
      2012-06-14 15:16:57  WARNING OGG-01223  TCP/IP error 79 (Connection refused).
      
      2012-06-14 15:17:37  WARNING OGG-01223  TCP/IP error 79 (Connection refused).
      
      2012-06-14 15:18:17  WARNING OGG-01223  TCP/IP error 79 (Connection refused).
      
      2012-06-14 15:18:57  WARNING OGG-01223  TCP/IP error 79 (Connection refused).
      
      2012-06-14 15:19:37  WARNING OGG-01223  TCP/IP error 79 (Connection refused).
      
      2012-06-14 15:20:17  WARNING OGG-01223  TCP/IP error 79 (Connection refused).
      
      Source Context :
        SourceModule            : [er.extrout]
        SourceID                : [/scratch/aime1/adestore/views/aime1_staxk09/oggcore/OpenSys/src/app/er/extrout.c]
        SourceFunction          : [tcp_send_data(extract_def *)]
        SourceLine              : [1257]
      
      2012-06-14 15:20:57  ERROR   OGG-01224  TCP/IP error 79 (Connection refused); retries exceeded.
      
      2012-06-14 15:20:57  ERROR   OGG-01668  PROCESS ABENDING.
      
      
      GGSCI (JLFLXFC1) 231>
      I am able to ping the remote server but GG datapump is not able to connect it.

      The manager at the remote server shows the following messages in report :-
      ................................
      
      2012-06-14 14:54:03  INFO    OGG-00953  Purging log history from GGS_OWNER.GGS_DDL_HIST older than 2012-06-12 13:54:03.290718: 0 rows deleted from GGS_OWNER
      .GGS_DDL_HIST.
      
      2012-06-14 14:54:03  INFO    OGG-00953  Purging log history from GGS_OWNER.GGS_DDL_HIST_ALT older than 2012-06-12 13:54:03.297951: 0 rows deleted from GGS_O
      WNER.GGS_DDL_HIST_ALT.
      
      2012-06-14 14:54:08  INFO    OGG-00963  Command received from EXTRACT on host 10.180.22.246 (START SERVER CPU -1 PRI -1  TIMEOUT 300 PARAMS ).
      
      2012-06-14 14:54:43  INFO    OGG-00974  Manager started collector process (Port 7847).
      
      2012-06-14 14:54:48  INFO    OGG-00963  Command received from EXTRACT on host 10.180.22.246 (START SERVER CPU -1 PRI -1  TIMEOUT 300 PARAMS ).
      
      2012-06-14 14:55:23  INFO    OGG-00974  Manager started collector process (Port 7848).
      
      2012-06-14 14:55:28  INFO    OGG-00963  Command received from EXTRACT on host 10.180.22.246 (START SERVER CPU -1 PRI -1  TIMEOUT 300 PARAMS ).
      
      2012-06-14 14:56:03  INFO    OGG-00974  Manager started collector process (Port 7849).
      
      2012-06-14 14:56:08  INFO    OGG-00963  Command received from EXTRACT on host 10.180.22.246 (START SERVER CPU -1 PRI -1  TIMEOUT 300 PARAMS ).
      
      2012-06-14 14:56:43  INFO    OGG-00974  Manager started collector process (Port 7850).
      
      2012-06-14 14:56:48  INFO    OGG-00963  Command received from EXTRACT on host 10.180.22.246 (START SERVER CPU -1 PRI -1  TIMEOUT 300 PARAMS ).
      
      2012-06-14 14:57:23  INFO    OGG-00974  Manager started collector process (Port 7851).
      
      2012-06-14 14:57:28  INFO    OGG-00963  Command received from EXTRACT on host 10.180.22.246 (START SERVER CPU -1 PRI -1  TIMEOUT 300 PARAMS ).
      
      2012-06-14 14:58:03  INFO    OGG-00974  Manager started collector process (Port 7852).
      
      2012-06-14 15:09:30  INFO    OGG-00963  Command received from EXTRACT on host 10.180.22.246 (START SERVER CPU -1 PRI -1  TIMEOUT 300 PARAMS ).
      
      2012-06-14 15:10:05  INFO    OGG-00974  Manager started collector process (Port 7841).
      
      2012-06-14 15:10:10  INFO    OGG-00963  Command received from EXTRACT on host 10.180.22.246 (START SERVER CPU -1 PRI -1  TIMEOUT 300 PARAMS ).
      
      2012-06-14 15:10:45  INFO    OGG-00974  Manager started collector process (Port 7842).
      
      2012-06-14 15:10:50  INFO    OGG-00963  Command received from EXTRACT on host 10.180.22.246 (START SERVER CPU -1 PRI -1  TIMEOUT 300 PARAMS ).
      
      2012-06-14 15:11:25  INFO    OGG-00974  Manager started collector process (Port 7843).
      
      2012-06-14 15:11:30  INFO    OGG-00963  Command received from EXTRACT on host 10.180.22.246 (START SERVER CPU -1 PRI -1  TIMEOUT 300 PARAMS ).
      
      2012-06-14 15:12:05  INFO    OGG-00974  Manager started collector process (Port 7844).
      
      2012-06-14 15:12:10  INFO    OGG-00963  Command received from EXTRACT on host 10.180.22.246 (START SERVER CPU -1 PRI -1  TIMEOUT 300 PARAMS ).
      
      2012-06-14 15:12:45  INFO    OGG-00974  Manager started collector process (Port 7845).
      
      2012-06-14 15:12:50  INFO    OGG-00963  Command received from EXTRACT on host 10.180.22.246 (START SERVER CPU -1 PRI -1  TIMEOUT 300 PARAMS ).
      
      2012-06-14 15:13:25  INFO    OGG-00974  Manager started collector process (Port 7846).
      
      2012-06-14 15:13:30  INFO    OGG-00963  Command received from EXTRACT on host 10.180.22.246 (START SERVER CPU -1 PRI -1  TIMEOUT 300 PARAMS ).
      
      2012-06-14 15:14:05  INFO    OGG-00974  Manager started collector process (Port 7847).
      
      2012-06-14 15:14:10  INFO    OGG-00963  Command received from EXTRACT on host 10.180.22.246 (START SERVER CPU -1 PRI -1  TIMEOUT 300 PARAMS ).
      
      2012-06-14 15:14:45  INFO    OGG-00974  Manager started collector process (Port 7848).
      
      2012-06-14 15:14:50  INFO    OGG-00963  Command received from EXTRACT on host 10.180.22.246 (START SERVER CPU -1 PRI -1  TIMEOUT 300 PARAMS ).
      
      2012-06-14 15:15:25  INFO    OGG-00974  Manager started collector process (Port 7849).
      
      2012-06-14 15:15:30  INFO    OGG-00963  Command received from EXTRACT on host 10.180.22.246 (START SERVER CPU -1 PRI -1  TIMEOUT 300 PARAMS ).
      
      2012-06-14 15:16:05  INFO    OGG-00974  Manager started collector process (Port 7850).
      
      2012-06-14 15:16:10  INFO    OGG-00963  Command received from EXTRACT on host 10.180.22.246 (START SERVER CPU -1 PRI -1  TIMEOUT 300 PARAMS ).
      
      2012-06-14 15:16:45  INFO    OGG-00974  Manager started collector process (Port 7851).
      
      2012-06-14 15:16:50  INFO    OGG-00963  Command received from EXTRACT on host 10.180.22.246 (START SERVER CPU -1 PRI -1  TIMEOUT 300 PARAMS ).
      
      2012-06-14 15:17:25  INFO    OGG-00974  Manager started collector process (Port 7852).
      Can anyone suggest what could be the problem ?
      Please guide me how to troubleshoot this problem.

      Thanks in advance.
        • 1. Re: Datapump process cannot connect to REMOTE HOST.
          VIRU
          any updates please ?
          • 2. Re: Datapump process cannot connect to REMOTE HOST.
            EmaxG
            please post the parameters of your MGR
            • 3. Re: Datapump process cannot connect to REMOTE HOST.
              amardeep.sidhu
              Most likely a firewall that is blocking these ports between source and the target.

              What does your target mgr param file look like ?
              • 4. Re: Datapump process cannot connect to REMOTE HOST.
                938450
                Hi

                It looks like a fire wall issue to me.

                Do one thing contact your network admin and tell him that you are using "FIFLX595" and port xxxxxxxx to connect to target.

                Tell him if he can sniff any thing.

                Cause: Not able to make a connect.

                Sol: Either you are using incorrect RMTHOST and port, or firewall blocking the connection.
                • 5. Re: Datapump process cannot connect to REMOTE HOST.
                  VIRU
                  Hi,
                  Thanks for your replies.

                  My HOST MGR is :-
                  GGSCI (JLFLXFC1) 20>  view params MGR
                  
                  PORT 7809
                  USERID ggs_owner, PASSWORD ggs_owner
                  PURGEOLDEXTRACTS /data1/GoldenGate/dirdat/ex, USECHECKPOINTS, MINKEEPDAYS 3
                  PURGEMARKERHISTORY MINKEEPDAYS 3, MAXKEEPDAYS 5, FREQUENCYMINUTES 30
                  PURGEDDLHISTORY MINKEEPDAYS 1, MAXKEEPDAYS 2, FREQUENCYMINUTES 30
                  My TARGET MGR is :-
                  GGSCI (FIFLX595) 17> view params MGR
                  
                  PORT 7853
                  USERID ggs_owner, PASSWORD ggs_owner
                  PURGEOLDEXTRACTS /data01/GoldenGate/dirdat/ex, USECHECKPOINTS, MINKEEPDAYS 3
                  PURGEMARKERHISTORY MINKEEPDAYS 3, MAXKEEPDAYS 5, FREQUENCYMINUTES 30
                  PURGEDDLHISTORY MINKEEPDAYS 1, MAXKEEPDAYS 2, FREQUENCYMINUTES 30
                  Yes even i think that this could be an firewall issue.
                  I have contacted the network admin people but till now i came to know that their is not a firewall between 2 servers. As both are on IBM AIX machine and same hardware is divided and given to the two servers. still i am trying to dig out more.

                  Please have a look at the MGR's and let me know if you people find any mistake.

                  Thanks.
                  • 6. Re: Datapump process cannot connect to REMOTE HOST.
                    amardeep.sidhu
                    My suggestion is:

                    Use DYNAMICPORTLIST parameter in the target mgr's param file to define a range of ports (say 7820-7830). Restart all the extract pump processes on source and mgr on target side. See if the error occurs again. If it does, ask the network guys to monitor the traffic on the mentioned ports and see if it is getting blocked due to some reason.
                    • 7. Re: Datapump process cannot connect to REMOTE HOST.
                      902642
                      change port MGR on data pump same with mgr port on the target manager

                      EXTRACT R2BDP
                      --setenv (ORACLE_SID=NGPR2B)
                      USERID ggs_owner, PASSWORD *********
                      RMTHOST FIFLX595,MGRPORT 7809
                      RMTTRAIL /data01/GoldenGate/dirdat/NGPR2B/rt
                      PASSTHRU
                      TABLE R2B.*;
                      to
                      EXTRACT R2BDP
                      --setenv (ORACLE_SID=NGPR2B)
                      USERID ggs_owner, PASSWORD *********
                      RMTHOST FIFLX595,MGRPORT 7853
                      RMTTRAIL /data01/GoldenGate/dirdat/NGPR2B/rt
                      PASSTHRU
                      TABLE R2B.*;
                      thanks,
                      Riyas
                      • 8. Re: Datapump process cannot connect to REMOTE HOST.
                        VIRU
                        I tried to use the DYNAMICPORTLIST option but still no success.
                        • 9. Re: Datapump process cannot connect to REMOTE HOST.
                          VIRU
                          Yes, I have changed that previously for testing purpose.

                          Now i have kept DATAPUMP port and TARGET MGR port to 7809 . Still i am getting the error.
                          I also checked with TELNET if the port is blocked but its not.

                          Below is the report of my DATAPUMP and the attempt of TELNET Command.
                          GGSCI (JLFLXFC1) 30>  view report r2bdp
                          
                          
                          ***********************************************************************
                                           Oracle GoldenGate Capture for Oracle
                                  Version 11.1.1.1 OGGCORE_11.1.1_PLATFORMS_110421.2040
                            AIX 5L, ppc, 64bit (optimized), Oracle 11g on Apr 22 2011 03:25:30
                          
                          Copyright (C) 1995, 2011, Oracle and/or its affiliates. All rights reserved.
                          
                          
                                              Starting at 2012-06-18 11:03:30
                          ***********************************************************************
                          
                          Operating System Version:
                          AIX
                          Version 5, Release 3
                          Node: JLFLXFC1
                          Machine: 00C576D24C00
                                                   soft limit   hard limit
                          Address Space Size   :    unlimited    unlimited
                          Heap Size            :    unlimited    unlimited
                          File Size            :    unlimited    unlimited
                          CPU Time             :    unlimited    unlimited
                          
                          Process id: 823546
                          
                          Description:
                          
                          ***********************************************************************
                          **            Running with the following parameters                  **
                          ***********************************************************************
                          EXTRACT R2BDP
                          --setenv (ORACLE_SID=NGPR2B)
                          USERID ggs_owner, PASSWORD *********
                          RMTHOST FIFLX595,MGRPORT 7809
                          RMTTRAIL /data01/GoldenGate/dirdat/NGPR2B/rt
                          PASSTHRU
                          TABLE R2B.*;
                          
                          CACHEMGR virtual memory values (may have been adjusted)
                          CACHEBUFFERSIZE:                         64K
                          CACHESIZE:                                8G
                          CACHEBUFFERSIZE (soft max):               4M
                          CACHEPAGEOUTSIZE (normal):                4M
                          PROCESS VM AVAIL FROM OS (min):          16G
                          CACHESIZEMAX (strict force to disk):  13.99G
                          
                          Database Version:
                          Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
                          PL/SQL Release 11.2.0.3.0 - Production
                          CORE    11.2.0.3.0      Production
                          TNS for IBM/AIX RISC System/6000: Version 11.2.0.3.0 - Production
                          NLSRTL Version 11.2.0.3.0 - Production
                          
                          Database Language and Character Set:
                          NLS_LANG environment variable specified has invalid format, default value will be used.
                          NLS_LANG environment variable not set, using default value AMERICAN_AMERICA.US7ASCII.
                          NLS_LANGUAGE     = "AMERICAN"
                          NLS_TERRITORY    = "AMERICA"
                          NLS_CHARACTERSET = "AL32UTF8"
                          
                          Warning: your NLS_LANG setting does not match database server language setting.
                          Please refer to user manual for more information.
                          Opened trail file /data1/GoldenGate/dirdat/R2B/lt000000 at 2012-06-18 11:03:30
                          
                          2012-06-18 11:03:30  INFO    OGG-01014  Positioning with begin time: Jun 18, 2012 11:03:22 AM, starting record time: Jun 18, 2012 6:50:11 AM at extseqno 0, e
                          xtrba 4557.
                          
                          2012-06-18 11:04:00  WARNING OGG-01223  TCP/IP error 79 (Connection refused).
                          
                          
                          GGSCI (JLFLXFC1) 31> shell telnet FIFLX595 7809
                          
                          Trying...
                          Connected to FIFLX595.
                          Escape character is '^]'.
                          
                          
                          telnet>q
                          My MGR at target schema is as follows :-
                          PORT 7809
                          USERID ggs_owner, PASSWORD ggs_owner
                          PURGEOLDEXTRACTS /data01/GoldenGate/dirdat/ex, USECHECKPOINTS, MINKEEPDAYS 3
                          PURGEMARKERHISTORY MINKEEPDAYS 3, MAXKEEPDAYS 5, FREQUENCYMINUTES 30
                          PURGEDDLHISTORY MINKEEPDAYS 1, MAXKEEPDAYS 2, FREQUENCYMINUTES 30
                          Please let me know if you find any mistake in this.
                          Thanks.
                          • 10. Re: Datapump process cannot connect to REMOTE HOST.
                            amardeep.sidhu
                            Most likely problem is not with the mgr port. It is with the port on which Collectors are going to do their work.
                            • 11. Re: Datapump process cannot connect to REMOTE HOST.
                              amardeep.sidhu
                              VIRU wrote:
                              I tried to use the DYNAMICPORTLIST option but still no success.
                              What all stuff you did ? and what issues you faced ?

                              Did you follow what i posted above
                              My suggestion is:
                              
                              Use DYNAMICPORTLIST parameter in the target mgr's param file to define a range of ports (say 7820-7830). Restart all the extract pump processes on source and mgr on target side.
                               See if the error occurs again. If it does, ask the network guys to monitor the traffic on the mentioned ports and see if it is getting blocked due to some reason.
                              If YES, then what was the outcome ?
                              • 12. Re: Datapump process cannot connect to REMOTE HOST.
                                902642
                                You can ping RMTHOST Name?
                                Like
                                ping  FIFLX595
                                • 13. Re: Datapump process cannot connect to REMOTE HOST.
                                  User10313095-Oracle
                                  Thanks for your replies.

                                  Using DYNAMICPORTLIST in destination MGR solved my issue.

                                  Thanks.