GG REPLICAT ABEND WITH Unknown exception

3857189

    Hello all,

     

    I keep getting this error notification which makes replicat not to apply captured data even when it show online but really it is not applying all data change to the target DB.

     

    ""

    Code:OGG-12025

    Severity: ERROR

    Title: The handler method registered for 'POST /services/v2/replicats/REP_HCE/command' caused an exception: No error code - Unknown exception.""

     

    I have raised SR several times on this and the best Oracle engineers could advice was for me to upgrade my GG to 19c which i did in DEV env and it not not resolve the error.

     

    Please i need help if anyone has encountered this before.

     

    I use OGG 12.3 Microservices on 12c DB.

     

    Thank you

      • 1. Re: GG REPLICAT ABEND WITH Unknown exception
        ORASCN

        Are you seeing any errors in report file?

         

        Could you please share the parameter file and report file of the repilcat process.

         

        Regards,

        Veera

        • 2. Re: GG REPLICAT ABEND WITH Unknown exception
          3857189

          No errors seen in the report file or log file but the errors shows up in the notification. And when you look at the admin dashboard you'll see replicat not showing Lag Seconds even Zero seconds even it's running. So, as such it is not picking the data from the trail to apply. Nothing is been apply.

           

          When i refresh or bounce DB, and start replicat again, it now shows the hours of Lag and after a few minute i refresh it goes back to the previous state not replicating.

           

          Below is the param file and report file.

           

          Thank you

           

          PARAMETER FILE:-

           

          replicat REP_HCE

          useridalias gg_db_ccnncprd_stnd domain fscGoldenGate

           

          DBOPTIONS INTEGRATEDPARAMS (PARALLELISM 40)

          DISCARDFILE ./irdl.dsc, APPEND, MEGABYTES 4000

          DBOPTIONS NOSUPPRESSTRIGGERS

          EOFDelayCSecs 1

          APPLYNOOPUPDATES

           

          MAP ECAMS_HCE.PEOPLE_IDENTIFIER, TARGET ECAMS_HCE.PEOPLE_IDENTIFIER, KEYCOLS (PEO_IDNTFR_SID);

          MAP ECAMS_HCE.PROCEDURE_INDICATOR, TARGET PDBCCNNCPRD.ECAMS_HCE.PROCEDURE_INDICATOR, KEYCOLS (PRCDR_INDCTR_SID);

          MAP ECAMS_HCE.VENDOR_DETAIL, TARGET PDBCCNNCPRD.ECAMS_HCE.VENDOR_DETAIL, KEYCOLS (VNDR_DETAIL_SID);

          REPERROR (0001, DISCARD)

          REPERROR (1403 DISCARD)

          ddlerror 1418 ignore

          reperror 01169 ignore

          ddlerror 01169 ignore

          ddlerror 01654 IGNORE

          reperror 01654 IGNORE

          ddlerror 2431 ignore

          ddlerror 2431 ignore

          ddlerror 01296 discard

          ddlerror 2403 ignore

          ddlerror 02264 ignore

          reperror 02264 ignore

          ddlerror 2443 ignore

          ddlerror 2430 ignore

          reperror 01296 discard

          ddlerror 02292 IGNORE

          REPERROR 02292 IGNORE

          ddlerror 02443 ignore

          reperror 02443 transdiscard

          ddlerror 1430 ignore

          ddlerror 02430 IGNORE

          ddlerror 1432 ignore

          ddlerror 1435 ignore

          ddlerror 942 ignore

          ddlerror 904 ignore

          ddlerror 2291 ignore

          ddlerror 02437 ignore

          reperror 02437 ignore

          REPERROR 02291, IGNORE

          DDLERROR 02291, IGNORE

          REPERROR 00519, IGNORE

          DDLERROR 00519, IGNORE

          ALLOWNOOPUPDATES

          DDLERROR DEFAULT IGNORE RETRYOP

          REPERROR 01403, ignore

          DDLERROR 1435 IGNORE INCLUDE OPTYPE ALTER OBJTYPE SESSION

          DDLERROR 2292 IGNORE INCLUDE OPTYPE ALTER OBJTYPE SESSION

          REPLACEBADCHAR ESCAPE

          MAP ECAMS_HCE.*, TARGET ECAMS_HCE.*;

           

          REPORT FILE:-

           

           

          ***********************************************************************

                           Oracle GoldenGate Delivery for Oracle

              Version 12.3.0.1.4 OGGCORE_12.3.0.1.0_PLATFORMS_180415.0359_FBO

             Linux, x64, 64bit (optimized), Oracle 12c on Apr 16 2018 01:10:23

           

          Copyright (C) 1995, 2018, Oracle and/or its affiliates. All rights reserved.

           

           

           

           

                              Starting at 2020-08-07 23:43:58

          ***********************************************************************

           

           

          Operating System Version:

          Linux

          Version #1 SMP Fri Jun 12 14:34:17 EDT 2020, Release 3.10.0-1127.13.1.el7.x86_64

          Node: vafscmul5531.aac.dva.va.gov

          Machine: x86_64

                                   soft limit   hard limit

          Address Space Size   :    unlimited    unlimited

          Heap Size            :    unlimited    unlimited

          File Size            :    unlimited    unlimited

          CPU Time             :    unlimited    unlimited

           

           

          Process id: 129435

           

           

          Description:

           

           

          ***********************************************************************

          **            Running with the following parameters                  **

          ***********************************************************************

           

           

          2020-08-07 23:43:58  INFO    OGG-03059  Operating system character set identified as US-ASCII.

           

           

          2020-08-07 23:43:58  INFO    OGG-02695  ANSI SQL parameter syntax is used for parameter parsing.

           

           

          2020-08-07 23:43:59  INFO    OGG-01360  REPLICAT is running in Integrated mode.

          replicat REP_HCE

          useridalias gg_db_ccnncprd_stnd domain fscGoldenGate

           

           

          2020-08-07 23:44:00  INFO    OGG-02679  The Replicat process logged on to database PDBCCNNCPRD and can only apply to that database.

          DBOPTIONS INTEGRATEDPARAMS (PARALLELISM 40)

          DISCARDFILE ./irdl.dsc, APPEND, MEGABYTES 4000

          DBOPTIONS NOSUPPRESSTRIGGERS

          EOFDelayCSecs 1

          APPLYNOOPUPDATES

          MAP ECAMS_HCE.PEOPLE_IDENTIFIER, TARGET ECAMS_HCE.PEOPLE_IDENTIFIER, KEYCOLS (PEO_IDNTFR_SID);

           

           

          2020-08-07 23:44:00  INFO    OGG-02669  Default catalog name PDBCCNNCPRD will be used for target table name ECAMS_HCE.PEOPLE_IDENTIFIER mapping.

          MAP ECAMS_HCE.PROCEDURE_INDICATOR, TARGET PDBCCNNCPRD.ECAMS_HCE.PROCEDURE_INDICATOR, KEYCOLS (PRCDR_INDCTR_SID);

          MAP ECAMS_HCE.VENDOR_DETAIL, TARGET PDBCCNNCPRD.ECAMS_HCE.VENDOR_DETAIL, KEYCOLS (VNDR_DETAIL_SID);

          REPERROR (0001, DISCARD)

          REPERROR (1403 DISCARD)

          ddlerror 1418 ignore

          reperror 01169 ignore

          ddlerror 01169 ignore

          ddlerror 01654 IGNORE

          reperror 01654 IGNORE

          ddlerror 2431 ignore

          ddlerror 2431 ignore

          ddlerror 01296 discard

          ddlerror 2403 ignore

          ddlerror 02264 ignore

          reperror 02264 ignore

          ddlerror 2443 ignore

          ddlerror 2430 ignore

          reperror 01296 discard

          ddlerror 02292 IGNORE

          REPERROR 02292 IGNORE

          ddlerror 02443 ignore

          reperror 02443 transdiscard

          ddlerror 1430 ignore

          ddlerror 02430 IGNORE

          ddlerror 1432 ignore

          ddlerror 1435 ignore

          ddlerror 942 ignore

          ddlerror 904 ignore

          ddlerror 2291 ignore

          ddlerror 02437 ignore

          reperror 02437 ignore

          REPERROR 02291, IGNORE

          DDLERROR 02291, IGNORE

          REPERROR 00519, IGNORE

          DDLERROR 00519, IGNORE

          ALLOWNOOPUPDATES

          DDLERROR DEFAULT IGNORE RETRYOP

          REPERROR 01403, ignore

          DDLERROR 1435 IGNORE INCLUDE OPTYPE ALTER OBJTYPE SESSION

          DDLERROR 2292 IGNORE INCLUDE OPTYPE ALTER OBJTYPE SESSION

          REPLACEBADCHAR ESCAPE

          MAP ECAMS_HCE.*, TARGET ECAMS_HCE.*;

           

           

          2020-08-07 23:44:00  INFO    OGG-02669  Default catalog name PDBCCNNCPRD will be used for target table name ECAMS_HCE.* mapping.

           

           

          2020-08-07 23:44:00  INFO    OGG-01815  Virtual Memory Facilities for: COM

              anon alloc: mmap(MAP_ANON)  anon free: munmap

              file alloc: mmap(MAP_SHARED)  file free: munmap

              target directories:

              /u01/app/oracle/goldengate/deployments/ccnc_prod_ggate/var/temp.

           

           

          Database Version:

          Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

          PL/SQL Release 12.2.0.1.0 - Production

          CORE 12.2.0.1.0 Production

          TNS for Linux: Version 12.2.0.1.0 - Production

          NLSRTL Version 12.2.0.1.0 - Production

           

           

          Database Language and Character Set:

          NLS_LANGUAGE     = "AMERICAN"

          NLS_TERRITORY    = "AMERICA"

          NLS_CHARACTERSET = "WE8ISO8859P1"

           

           

          2020-08-07 23:44:00  INFO    OGG-06604  Database PDBCCNNCPRD.AAC.DVA.VA.GOV CPU info: CPU Count 32, CPU Core Count 32, CPU Socket Count 8.

           

           

          2020-08-07 23:44:00  WARNING OGG-05673  CSN-based duplicate suppression is disabled because there is no checkpoint table for this Replicat.

           

           

          2020-08-07 23:44:00  INFO    OGG-02545  Parameter GROUPTRANSOPS is ignored by Integrated Replicat when parallelism is greater than 1.

           

           

          2020-08-07 23:44:00  INFO    OGG-02527  Integrated Replicat does not populate a trace table.

           

           

          2020-08-07 23:44:22  INFO    OGG-02530  Integrated replicat successfully attached to inbound server OGG$REP_HCE.

           

           

          ***********************************************************************

          **                     Run Time Messages                             **

          ***********************************************************************

           

           

           

           

          2020-08-07 23:44:22  INFO    OGG-01021  Command received from REST API: GETLAG.

           

           

          2020-08-07 23:44:22  INFO    OGG-02243  Opened trail file /u01/app/oracle/goldengate/deployments/ccnc_prod_ggate/var/lib/data/rt000000089 at 2020-08-07 23:44:22.914033.

           

           

          2020-08-07 23:44:22  INFO    OGG-03522  Setting session time zone to source database time zone 'GMT'.

           

           

          2020-08-07 23:44:22  INFO    OGG-03506  The source database character set, as determined from the trail file, is UTF-8.

           

           

          2020-08-07 23:44:23  INFO    OGG-06506  Wildcard MAP resolved (entry ECAMS_HCE.*): MAP "ECAMS_HCE"."AD_CLAIM_LINE", TARGET "PDBCCNNCPRD".ECAMS_HCE."AD_CLAIM_LINE".

           

           

          2020-08-07 23:44:23  INFO    OGG-02756  The definition for table ECAMS_HCE.AD_CLAIM_LINE is obtained from the trail file.

           

           

          2020-08-07 23:44:23  INFO    OGG-06511  Using following columns in default map by name: CLAIM_LINE_SID, CLAIM_HEADER_SID, FACILITY_TYPE_CODE, PROCEDURE_IID, REVENUE_IID, FROM_SERVICE_DATE, TO_SERVICE_DATE, CLAIM_LINE_TCN, LINE_ITEM_CONTROL_NUMBER, CREATED_BY, CREATED_DATE, MODIFIED_BY, MODIFIED_DATE, PARENT_LINE_SID, SRVCNG_PRVDR_LCTN_IID, SRVCNG_PRVDR_LCTN_IDENTIFIER, SRVCNG_PRVDR_IDNTFR_TYPE_CID, SRVCNG_PRVDR_LCTN_TXNMY_CODE, APLCTN_STATUS_TYPE_CID, APLCTN_STATUS_CID, BSNS_STATUS_TYPE_CID, BSNS_STATUS_CID, STATUS_EFFECTIVE_DATE, LAST_RUN_NMBR, LAST_RUN_DATE, CLM_TYPE_CID, PROGRAM_CID, TPL_LKPCD, LAST_SUSPEND_DATE, SYSTEM_IN_DATE, ADJUSTMENT_SOURCE_TYPE_LKPCD, ADJUSTMENT_TYPE_LKPCD, CLSFCTN_GROUP_CID, BILLED_AMOUNT, BILLED_UNITS, PRCDR_CODE, REVENUE_CODE, PRIMARY_DIAGNOSIS_IID, PRIMARY_PA_RQST_SID, PRIMARY_PA_RQST_IDENTIFIER, MDFR_CODE1, MDFR_CODE2, MDFR_CODE3, MDFR_CODE4, PRIMARY_RFRL_IDNTFCTN_NMBR, ADMISSION_DATE, SRVCNG_NATIONAL_PRVDR_IDNTFR, SRVCNG_PRVDR_LAST_NAME, SRVCNG_PRVDR_FIRST_NAME, SRVCNG_PRVDR_MIDDLE_NAME, SRVCNG_PRVDR_SUFFIX_LKPCD, PRCDR_CODE_QLFR_LKPCD, RMTNC_ADVC_NUMBER, SRVCNG_PRVDR_LEGACY_ID, SBMTD_PRCDR_CODE, SBMTD_PRCDR_IID, SBMTD_PRCDR_DESC, BILLED_UNIT_UOM_CODE, SSBP_ATHRZTN_HDR_SID, ATHRZTN_NMBR.

           

           

          2020-08-07 23:44:23  INFO    OGG-06510  Using the following key columns for target table PDBCCNNCPRD.ECAMS_HCE.AD_CLAIM_LINE: CLAIM_LINE_SID.

           

           

           

           

          2020-08-07 23:44:24  INFO    OGG-03010  Performing implicit conversion of column data from character set UTF-8 to we8iso8859p1.

           

           

          2020-08-07 23:44:25  INFO    OGG-01021  Command received from REST API: GETLAG.

           

           

          2020-08-07 23:44:50  INFO    OGG-02232  Switching to next trail file /u01/app/oracle/goldengate/deployments/ccnc_prod_ggate/var/lib/data/rt000000090 at 2020-08-07 23:44:50.110110 due to EOF. with current RBA 524,287,187.

           

           

          2020-08-08 00:06:55  INFO    OGG-01971  The previous message, 'INFO OGG-01021', repeated 1 times.

           

           

          2020-08-08 00:16:55  INFO    OGG-01971  The previous message, 'INFO OGG-01021', repeated 1 times.

           

           

          2020-08-08 00:20:21  INFO    OGG-01021  Command received from REST API: STATS TOTAL.

           

           

          2020-08-08 00:20:51  INFO    OGG-01971  The previous message, 'INFO OGG-01021', repeated 1 times.

           

           

          2020-08-08 00:28:35  INFO    OGG-01021  Command received from REST API: GETLAG.

          • 3. Re: GG REPLICAT ABEND WITH Unknown exception
            ORASCN

            May I know if someone suggested to use the below parameter file?

             

            DBOPTIONS INTEGRATEDPARAMS (PARALLELISM 40)

            DISCARDFILE ./irdl.dsc, APPEND, MEGABYTES 4000

            DBOPTIONS NOSUPPRESSTRIGGERS

            EOFDelayCSecs 1

            APPLYNOOPUPDATES

             

            MAP ECAMS_HCE.PEOPLE_IDENTIFIER, TARGET ECAMS_HCE.PEOPLE_IDENTIFIER, KEYCOLS (PEO_IDNTFR_SID);

            MAP ECAMS_HCE.PROCEDURE_INDICATOR, TARGET PDBCCNNCPRD.ECAMS_HCE.PROCEDURE_INDICATOR, KEYCOLS (PRCDR_INDCTR_SID);

            MAP ECAMS_HCE.VENDOR_DETAIL, TARGET PDBCCNNCPRD.ECAMS_HCE.VENDOR_DETAIL, KEYCOLS (VNDR_DETAIL_SID);

            REPERROR (0001, DISCARD)

            REPERROR (1403 DISCARD)

            ddlerror 1418 ignore

            reperror 01169 ignore

            ddlerror 01169 ignore

            ddlerror 01654 IGNORE

            reperror 01654 IGNORE

            ddlerror 2431 ignore

            ddlerror 2431 ignore

            ddlerror 01296 discard

            ddlerror 2403 ignore

            ddlerror 02264 ignore

            reperror 02264 ignore

            ddlerror 2443 ignore

            ddlerror 2430 ignore

            reperror 01296 discard

            ddlerror 02292 IGNORE

            REPERROR 02292 IGNORE

            ddlerror 02443 ignore

            reperror 02443 transdiscard

            ddlerror 1430 ignore

            ddlerror 02430 IGNORE

            ddlerror 1432 ignore

            ddlerror 1435 ignore

            ddlerror 942 ignore

            ddlerror 904 ignore

            ddlerror 2291 ignore

            ddlerror 02437 ignore

            reperror 02437 ignore

            REPERROR 02291, IGNORE

            DDLERROR 02291, IGNORE

            REPERROR 00519, IGNORE

            DDLERROR 00519, IGNORE

            ALLOWNOOPUPDATES

            DDLERROR DEFAULT IGNORE RETRYOP

            REPERROR 01403, ignore

            DDLERROR 1435 IGNORE INCLUDE OPTYPE ALTER OBJTYPE SESSION

            DDLERROR 2292 IGNORE INCLUDE OPTYPE ALTER OBJTYPE SESSION

            REPLACEBADCHAR ESCAPE

            MAP ECAMS_HCE.*, TARGET ECAMS_HCE.*;

             

            You don't need to ignore some many things.

             

            Was the initial load done before starting the replication?

             

            Instead of GUI, use the AdminClient which is command line interface for OGG MA. Issue the below command,

             

            stats replicat REP_HCE

             

            Regards,

            Veera

            • 4. Re: GG REPLICAT ABEND WITH Unknown exception
              3857189

              Hi Veera,

               

              DBOPTIONS INTEGRATEDPARAMS (PARALLELISM 40) ==> This was used to speed up replication apply process the last time this same error gave us 48hrs of Lag.

               

              My environment carry out Code change Deployment every week and as such i see replicat abending so the rep/ddl error was to enable replicat continue working.

               

              The KEYCOLS was also used when needed to replicat to pick up again especially when it abends on those tables.

               

              adminclient or ggsci does'nt work in my environment or maybe you can give me a better command to use to invoke it.

               

              Menawhile what change will it make if i use adminclient to start it instead of just clicking start on the GUI

               

               

               

              • 5. Re: GG REPLICAT ABEND WITH Unknown exception
                ORASCN

                Hi ,

                 

                There is no GGSCI in OGG MA.

                 

                In OGG Microservices, you can either use GUI or Command Interface (AdminClient). Please check the below for more details,

                 

                https://docs.oracle.com/goldengate/c1230/gg-winux/GGSAU/how-use-admin-client.htm

                 

                Since there is an option for PARALLELISM in IR, you shouldn't give a higher number. You need to understand why there is a Lag in it. For that you need to know how the Integrated Replicat process works. The pros and cons of it. Check the below,

                 

                https://www.oracle-scn.com/oracle-goldengate-integrated-replicat/

                 

                Regards,

                Veera