9 Replies Latest reply: Dec 20, 2012 1:00 AM by 980958 RSS

    ORA-17629: Cannot connect to the remote database server

    856988
      Hi there,

      I have Dataguard set up with primary and 1 physical standby database for learning/testing:

      I do NOT plan on off-loading RMAN backups onto the physical standby;

      Registered primary database with RMAN and when "resync catalog from db_unique_name all; " is issued,
      I get the following error:
      ---------------------------------------------------------------------------------------------------------------------
      RMAN> resync catalog from db_unique_name all;
      starting full resync of recovery catalog
      full resync complete
      resyncing from database with DB_UNIQUE_NAME ASTTESTB
      RMAN-00571: ===========================================================
      RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
      RMAN-00571: ===========================================================
      RMAN-03002: failure of resync from db_unique_name command at 09/28/2011 11:34:23
      ORA-17629: Cannot connect to the remote database server
      ORA-17628: Oracle error 17629 returned by remote Oracle server
      ------------------------------------------------------------------------------------------------------------------
      RMAN configuration for the primary instance are as follows:

      RMAN> show all for db_unique_name asttest;

      RMAN configuration parameters for database with db_unique_name ASTTEST are:
      CONFIGURE RETENTION POLICY TO RECOVERY WINDOW OF 7 DAYS;
      CONFIGURE BACKUP OPTIMIZATION OFF; # default
      CONFIGURE DEFAULT DEVICE TYPE TO DISK; # default
      CONFIGURE CONTROLFILE AUTOBACKUP OFF; # default
      CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '%F'; # default
      CONFIGURE DEVICE TYPE DISK PARALLELISM 1 BACKUP TYPE TO BACKUPSET; # default
      CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
      CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
      CONFIGURE MAXSETSIZE TO UNLIMITED; # default
      CONFIGURE ENCRYPTION FOR DATABASE OFF; # default
      CONFIGURE ENCRYPTION ALGORITHM 'AES128'; # default
      CONFIGURE COMPRESSION ALGORITHM 'BASIC' AS OF RELEASE 'DEFAULT' OPTIMIZE FOR LOAD TRUE ; # default
      CONFIGURE DBUNIQUE_NAME 'asttest' CONNECT IDENTIFIER 'ASTTEST.UMDNJ.EDU';_
      CONFIGURE DBUNIQUE_NAME 'asttestb' CONNECT IDENTIFIER 'asttestb';_
      CONFIGURE ARCHIVELOG DELETION POLICY TO APPLIED ON ALL STANDBY;
      CONFIGURE SNAPSHOT CONTROLFILE NAME TO '/opt/oracle/product/11.2.0/dbhome_1/dbs/snapcf_asttest.f'; # default

      RMAN configuration for the physical standby instance are as follows:

      RMAN> show all for db_unique_name asttestb;

      RMAN configuration parameters for database with db_unique_name ASTTESTB are:
      CONFIGURE RETENTION POLICY TO RECOVERY WINDOW OF 7 DAYS;
      CONFIGURE BACKUP OPTIMIZATION OFF; # default
      CONFIGURE DEFAULT DEVICE TYPE TO DISK; # default
      CONFIGURE CONTROLFILE AUTOBACKUP ON;
      CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '%F'; # default
      CONFIGURE DEVICE TYPE DISK PARALLELISM 1 BACKUP TYPE TO BACKUPSET; # default
      CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
      CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
      CONFIGURE MAXSETSIZE TO UNLIMITED; # default
      CONFIGURE ENCRYPTION FOR DATABASE OFF; # default
      CONFIGURE ENCRYPTION ALGORITHM 'AES128'; # default
      CONFIGURE COMPRESSION ALGORITHM 'BASIC' AS OF RELEASE 'DEFAULT' OPTIMIZE FOR LOAD TRUE ; # default
      CONFIGURE DBUNIQUE_NAME 'asttest' CONNECT IDENTIFIER 'ASTTEST.UMDNJ.EDU';_
      CONFIGURE DBUNIQUE_NAME 'asttestb' CONNECT IDENTIFIER 'asttestb';_
      CONFIGURE ARCHIVELOG DELETION POLICY TO APPLIED ON ALL STANDBY;
      CONFIGURE SNAPSHOT CONTROLFILE NAME TO '/opt/oracle/product/11.2.0/dbhome_1/dbs/snapcf_asttest.f'; # default

      ----------------------------------------------------------------------------------------------------------------------

      When I tested missing datafile and missing control file scenario on the primary database, it worked (used data recover advisor)

      However, missing datafile scenario is not working on the physical standby.
      Below are the steps I followed:
      ********************************************
      1. Renamed a datafile on physical-standby:

      bash-3.2$ ls -ltr

      total 107832

      -rw-r----- 1 oracle oinstall 52429312 Sep 16 12:19 asttest_redo03.log

      -rw-r----- 1 oracle oinstall 61874176 Sep 29 15:41 asttest_temp01.dbf

      -rw-r----- 1 oracle oinstall 52436992 Oct 5 11:00 rmandg_data01.dbf

      bash-3.2$ mv rmandg_data01.dbf rmandg_data01.dbf_old

      bash-3.2$ ls -ltr

      total 107832

      -rw-r----- 1 oracle oinstall 52429312 Sep 16 12:19 asttest_redo03.log

      -rw-r----- 1 oracle oinstall 61874176 Sep 29 15:41 asttest_temp01.dbf

      -rw-r----- 1 oracle oinstall 52436992 Oct 5 11:00 rmandg_data01.dbf_old


      2. Restore datafile (find the datafile# from v$recover_file view):
      RMAN> restore datafile 5;



      Starting restore at 05-OCT-11

      Segmentation fault


      3. Inspite of the error, issued “recover datafile command”

      4. RMAN> recover datafile 5;



      Starting recover at 05-OCT-11

      allocated channel: ORA_DISK_1

      channel ORA_DISK_1: SID=10 device type=DISK

      RMAN-00571: ===========================================================

      RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============

      RMAN-00571: ===========================================================

      RMAN-03002: failure of recover command at 10/05/2011 12:00:40

      RMAN-06094: datafile 5 must be restored


      5. Manually copy datafile from primary to physical-standby, & then issue recover command in RMAN:
      bash-3.2$ scp oracle@msudrainier:/ora03/oradata/asttest/rmandg_data01.dbf .

      rmandg_data01.dbf 100% 50MB 16.7MB/s 00:03

      bash-3.2$ ls -ltr

      total 159096

      -rw-r----- 1 oracle oinstall 52429312 Sep 16 12:19 asttest_redo03.log

      -rw-r----- 1 oracle oinstall 61874176 Sep 29 15:41 asttest_temp01.dbf

      -rw-r----- 1 oracle oinstall 52436992 Oct 5 11:49 rmandg_data01.dbf_old

      -rw-r----- 1 oracle oinstall 52436992 Oct 5 12:01 rmandg_data01.dbf


      RMAN> recover datafile 5;



      Starting recover at 05-OCT-11

      using channel ORA_DISK_1



      RMAN-00571: ===========================================================

      RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============

      RMAN-00571: ===========================================================

      RMAN-03002: failure of recover command at 10/05/2011 12:01:59

      RMAN-06067: RECOVER DATABASE required with a backup or created control file


      6. Obviously RMAN commands were NOT successful in restoring the missing datafile;
      So I renamed the old file back to original and restarted redo-apply services – this did restore the datafile and now DG configuration status is normal.
      *********************************************************************************

      Not quite sure if I'm even recovering it the right way or if the error above is related to RMAN not able to sync with the physical standby.

      The documentation I'm referring to is here (1st paragraph):

      [http://download.oracle.com/docs/cd/B28359_01/server.111/b28294/rman.htm#BAJDFBDD|http://download.oracle.com/docs/cd/B28359_01/server.111/b28294/rman.htm#BAJDFBDD]


      Any pointers/direction and advise is greatly appreciated.

      Best regards,
      NK
        • 1. Re: ORA-17629: Cannot connect to the remote database server
          mseberg
          Hello;

          I'm thinking when you connected to RMAN you did not use a password.

          This is required with your setup.

          The password file must be identical on all of the remote databases for this feature to work properly. You must manually copy it to all the remote databases in the configuration.

          I ran into this between Oracle 10 and 11. On Oracle 11 you have to copy the password file for sure. At the time I don't think this was documented.

          Best Regards

          mseberg
          • 2. Re: ORA-17629: Cannot connect to the remote database server
            856988
            mseberg,
            The password file was copied from the primary to the physical standby during set-up;
            I did re-copy & still get the same error.

            *************************************************************
            [oracle@devldb dbs]$ pwd
            /opt/oracle/product/11.2.0/dbhome_1/dbs

            [oracle@devldb dbs]$ ls -l orapwasttest*
            -rw-r----- 1 oracle oinstall 1536 Aug 23 08:52 orapwasttest

            [oracle@devldb dbs]$

            bash-3.2$ ls -ltr orapwast*
            -rw-r----- 1 oracle oinstall 1536 Oct 7 08:36 orapwasttest_goldcopy
            -rw-r----- 1 oracle oinstall 1536 Oct 7 08:37 orapwasttest

            *****************************************************************

            best regards,
            NK
            • 3. Re: ORA-17629: Cannot connect to the remote database server
              mseberg
              Hello NK;

              Odd, I later found this Oracle note which matches your error and commands

              *RESYNC CATALOG FROM DB_UNIQUE_NAME ALL fails ORA-17629, ORA-17628 [ID 1327156.1]*

              It shows additional RMAN connect info etc.

              Best Regards

              mseberg

              I'm looking at your password copy and I'm concerned you may not be doing a rename

              Ex
              I copy orapwPRIMARY but rename it to orapwSTANDBY
              
              (Renaming the current standby password first to avoid conflicts)
              Does this make sense? So my primary database is PRIMARY and my standby database is STANDBY. The Oracle doc does the BOSTON - CHICAGO thing.

              Edited by: mseberg on Oct 7, 2011 11:11 AM
              • 4. Re: ORA-17629: Cannot connect to the remote database server
                856988
                mseberg: First off - thanks for taking time to respond..

                the instance name and the service name on the standby is asttest (same as primary - to minimize any connect string changes at application level);

                bash-3.2$ ps -ef | grep pmon
                oracle 16793 1 0 08:38 ? 00:00:00 ora_pmon_asttest

                I had previously copied the password file as asttestb too, but that too does not seem to make any difference.

                bash-3.2$ ls -ltr orapwast*
                -rw-r----- 1 oracle oinstall 1536 Oct 7 08:16 orapwasttestb
                -rw-r----- 1 oracle oinstall 1536 Oct 7 08:36 orapwasttest_goldcopy
                -rw-r----- 1 oracle oinstall 1536 Oct 7 08:37 orapwasttest

                ****************************************************************
                1 select name, db_unique_name, PRIMARY_DB_UNIQUE_NAME, database_role
                2* from v$database
                SQL> /

                NAME DB_UNIQUE_NAME PRIMARY_DB_UNIQUE_NAME DATABASE_ROLE
                --------- ------------------------------ ---------------------------------- ---------------------
                ASTTEST asttestb asttest PHYSICAL STANDBY


                SQL> select instance_name from V$instance;

                INSTANCE_NAME
                ----------------
                asttest

                **************************************************************************
                Thanks & regards,
                NK

                Edited by: NK on Oct 7, 2011 11:12 AM
                • 5. Re: ORA-17629: Cannot connect to the remote database server
                  mseberg
                  Got it. Since you made no comment on the Oracle note I guess you had already found that.

                  You may be interested in this Transparent Application document at some point :

                  http://uhesse.wordpress.com/2009/08/19/connect-time-failover-transparent-application-failover-for-data-guard/

                  Best Regards

                  mseberg

                  LATER

                  I'm able to recreate your issue on my system!!
                  MAN> resync catalog from db_unique_name all; 
                  
                  
                  starting full resync of recovery catalog
                  full resync complete
                  
                  resyncing from database with DB_UNIQUE_NAME STANDBY
                  RMAN-00571: ===========================================================
                  RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
                  RMAN-00571: ===========================================================
                  RMAN-03002: failure of resync from db_unique_name command at 10/07/2011 13:32:45
                  ORA-17629: Cannot connect to the remote database server
                  ORA-17627: ORA-01017: invalid username/password; logon denied
                  ORA-17629: Cannot connect to the remote database server
                  
                  RMAN> 
                  Edited by: mseberg on Oct 7, 2011 1:33 PM

                  Still Later


                  Managed to get a new error
                  using target database control file instead of recovery catalog
                  RMAN-00571: ===========================================================
                  RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
                  RMAN-00571: ===========================================================
                  RMAN-03002: failure of resync from db_unique_name command at 10/07/2011 13:41:44
                  RMAN-05037: FOR DB_UNIQUE_NAME option cannot be used in nocatalog mode
                  I started RMAN can used this connect string ( not that it helps )
                  $ORACLE_HOME/bin/rman target sys/<password>@STANDBY
                  Edited by: mseberg on Oct 7, 2011 1:42 PM

                  Even later
                  RMAN> resync catalog from db_unique_name 'standby';
                  
                  resyncing from database with DB_UNIQUE_NAME STANDBY
                  RMAN-00571: ===========================================================
                  RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
                  RMAN-00571: ===========================================================
                  RMAN-03009: failure of resync from db_unique_name command on default channel at 10/07/2011 13:52:04
                  ORA-17629: Cannot connect to the remote database server
                  ORA-17627: ORA-01017: invalid username/password; logon denied
                  ORA-17629: Cannot connect to the remote database server
                  I hate when I'm getting clues and I'm too thick to put them together!

                  Edited by: mseberg on Oct 7, 2011 1:52 PM

                  But this works :
                  RMAN> resync catalog from db_unique_name 'primary';
                  
                  starting full resync of recovery catalog
                  full resync complete
                  
                  RMAN> 
                  Edited by: mseberg on Oct 7, 2011 1:54 PM


                  I love this stuff

                  If I try from the standby side I get this :
                  resync catalog from db_unique_name all;
                  
                  
                  resyncing from database with DB_UNIQUE_NAME PRIMARY
                  RMAN-00571: ===========================================================
                  RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
                  RMAN-00571: ===========================================================
                  RMAN-03002: failure of resync from db_unique_name command at 10/07/2011 13:58:42
                  ORA-17629: Cannot connect to the remote database server
                  ORA-17627: ORA-01017: invalid username/password; logon denied
                  ORA-17629: Cannot connect to the remote database server
                  I'm going to copy the password file again here and shutdown the standby and try again.

                  Edited by: mseberg on Oct 7, 2011 2:00 PM
                  However what is missing is the ability for [RMAN to connect to the standby (while primary is the target) and vice versa]so it sync with the catalog. 
                  Agreed. I get the same results here and I know that does not help.

                  Edited by: mseberg on Oct 7, 2011 2:10 PM
                  • 6. Re: ORA-17629: Cannot connect to the remote database server
                    856988
                    When I look for the Oracle note, I hit this very thread! I must be missing something or am I toast after a week of reading!!!

                    NK

                    in response to mseberg "still later"

                    [oracle@msudrainier datagaurd]$ rman target sys@ASTTESTB catalog rman@rvct

                    Recovery Manager: Release 11.2.0.2.0 - Production on Fri Oct 7 14:49:56 2011

                    Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved.

                    target database Password:
                    connected to target database: ASTTEST (DBID=1661319998, not open)
                    recovery catalog database Password:
                    connected to recovery catalog database

                    RMAN> resync catalog from db_unique_name all;


                    RMAN> resync catalog from db_unique_name all;


                    resyncing from database with DB_UNIQUE_NAME ASTTEST
                    RMAN-00571: ===========================================================
                    RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
                    RMAN-00571: ===========================================================
                    RMAN-03002: failure of resync from db_unique_name command at 10/07/2011 15:09:54
                    ORA-17629: Cannot connect to the remote database server
                    ORA-17628: Oracle error 17629 returned by remote Oracle server


                    RMAN> Exit


                    [oracle@msudrainier datagaurd]$ rman target / catalog rman@rvct

                    Recovery Manager: Release 11.2.0.2.0 - Production on Fri Oct 7 14:59:50 2011

                    Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved.

                    connected to target database: ASTTEST (DBID=1661319998)
                    recovery catalog database Password:
                    connected to recovery catalog database

                    RMAN> resync catalog from db_unique_name 'asttest';

                    starting full resync of recovery catalog
                    full resync complete

                    RMAN>

                    And on Standby:
                    [oracle@nwpdenalib datagaurd]$ rman target / catalog rman@rvct

                    Recovery Manager: Release 11.2.0.2.0 - Production on Fri Oct 7 15:02:15 2011

                    Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved.

                    connected to target database: ASTTEST (DBID=1661319998, not open)
                    recovery catalog database Password:
                    connected to recovery catalog database

                    RMAN> resync catalog from db_unique_name 'asttestb';


                    RMAN> resync catalog from db_unique_name all;


                    resyncing from database with DB_UNIQUE_NAME ASTTEST
                    RMAN-00571: ===========================================================
                    RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
                    RMAN-00571: ===========================================================
                    RMAN-03002: failure of resync from db_unique_name command at 10/07/2011 15:09:54
                    ORA-17629: Cannot connect to the remote database server
                    ORA-17628: Oracle error 17629 returned by remote Oracle server



                    --- However what is missing is the ability for [RMAN to connect to the standby (while primary is the target) and vice versa]so it sync with the catalog.

                    NK

                    Edited by: NK on Oct 7, 2011 12:13 PM
                    • 7. Re: ORA-17629: Cannot connect to the remote database server
                      mseberg
                      I got it!
                      /home/oracle:STANDBY >$ORACLE_HOME/bin/rman target sys/<password> catalog <user_name>/<password>@rcatalog
                      
                      Recovery Manager: Release 11.2.0.1.0 - Production on Fri Oct 7 14:12:10 2011
                      
                      Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved.
                      
                      connected to target database: PRIMARY (DBID=1562898590, not open)
                      connected to recovery catalog database
                      
                      RMAN> resync catalog from db_unique_name all;
                      
                      
                      resyncing from database with DB_UNIQUE_NAME PRIMARY
                      starting full resync of recovery catalog
                      RMAN output not resynced for database with DB_UNIQUE_NAME PRIMARY
                      full resync complete
                      So the connect string needed a target/sys and a catalog

                      rman target sys/<password> catalog <user_name>/<password>@rcatalog

                      I did this from my standby

                      Will try from the primary

                      Best Regards

                      mseberg

                      Works on both. Great question!!
                      connected to target database: PRIMARY (DBID=1562898590)
                      connected to recovery catalog database
                      
                      RMAN> resync catalog from db_unique_name all;
                      
                      
                      starting full resync of recovery catalog
                      full resync complete
                      
                      resyncing from database with DB_UNIQUE_NAME STANDBY
                      RMAN output not resynced for database with DB_UNIQUE_NAME STANDBY
                      
                      RMAN> 
                      Edited by: mseberg on Oct 7, 2011 2:16 PM


                      Thanks a ton for sticking with this. I will add this little trick to my web site.

                      Edited by: mseberg on Oct 7, 2011 2:22 PM
                      • 8. Re: ORA-17629: Cannot connect to the remote database server
                        856988
                        Yes sir it works for me too!!
                        thanks so much..

                        I did it on primary:

                        [oracle@msudrainier datagaurd]$ rman target sys catalog rman@rvct

                        Recovery Manager: Release 11.2.0.2.0 - Production on Fri Oct 7 15:18:50 2011

                        Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved.

                        target database Password:
                        connected to target database: ASTTEST (DBID=1661319998)
                        recovery catalog database Password:
                        connected to recovery catalog database

                        RMAN> resync catalog from db_unique_name all;


                        starting full resync of recovery catalog
                        full resync complete

                        resyncing from database with DB_UNIQUE_NAME ASTTESTB

                        RMAN>
                        • 9. Re: ORA-17629: Cannot connect to the remote database server
                          980958
                          Many Thanks. This worked perfectly for me. Trick is to provide the target & catalog credentials at the same time.