4 Replies Latest reply: May 13, 2014 6:32 PM by MikeN RSS

    Hadoop integration with GlodenGate

    9b72258f-9bba-4426-9c3d-93bd24aa908c

      Hi,

       

         I need to integrate GG with Cloudera Hadoop setup and I am not able to find any solutions around it.It would be great help if someone can provide me some insight into it or a sample HBASE adapter code would be

       

      Thanks

      Haider

        • 1. Re: Hadoop integration with GlodenGate
          Ananth R. Tiru

          Hi Haider,

           

          Please refer to Knowledge ID 1586211.1 on MOS. The paper has code and instructions to integrate OGG with HBase

           

          Thanks

          Ananth

          • 2. Re: Hadoop integration with GlodenGate
            124886cd-1b88-43dc-8526-aab9b5d7eb3c

            Hi Ananth,

             

            We did follow the steps in the support ticket. However we have encountered an issue where the pump process keeps getting stopped. It looks like it is not able to trigger the Sample Hbase Adapter jar file instead a core file is getting generated

             

             

            Thanks,

            Hussain

            • 3. Re: Hadoop integration with GlodenGate
              124886cd-1b88-43dc-8526-aab9b5d7eb3c

              So it looks like the Pump process is not getting triggered due to the below error. Any idea on how this can be fixed?

               

               

              Error occurred (Java exception): Error calling static void Method UserExitMain.main:

                org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'userExitDataSource' defined in class path resource [com/goldengate/atg/datasource/DataSource-context.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.factory.BeanDefinitionStoreException: Factory method [public final com.goldengate.atg.datasource.GGDataSource com.goldengate.atg.datasource.factory.DataSourceFactory.getDataSource()] threw exception; nested exception is com.goldengate.atg.util.ConfigException: Unable to set property on handler 'hdfshandler' (com.mycompany.bigdata.SampleHandlerHDFS).An exception was thrown while attempting to set property on 'class com.mycompany.bigdata.SampleHandlerHDFS', ' property: 'mode', value: 'op ', exception: java.lang.reflect.InvocationTargetException, message: null

              • 4. Re: Hadoop integration with GlodenGate
                MikeN

                124886cd-1b88-43dc-8526-aab9b5d7eb3c wrote:

                 

                exception is com.goldengate.atg.util.ConfigException: Unable to set property on handler 'hdfshandler' (com.mycompany.bigdata.SampleHandlerHDFS).An exception was thrown while attempting to set property on 'class com.mycompany.bigdata.SampleHandlerHDFS', ' property: 'mode', value: 'op '

                 

                When using a custom event handler, the property file can be used to configure the Java object instance; e.g, if a class has a method "setFoo(String x)", then the property file can use "gg.handler.{instance_name}.foo=bar", and that will call "setFoo("bar")" on the instance.  Custom handlers should extend AbstractHandler (in the current API).

                 

                For example,

                 

                package com.mycompany.bigdata;
                import com.goldengate.atg.datasource.*;
                .
                public class SampleHandlerHDFS extends AbstractHandler {
                private long numOps = 0;
                private YourCustomTarget target = YourCustomTarget.create();
                .
                public void setFoo(String bar) {
                target.setSomethingFooey(bar);
                }
                .
                public Status operationAdded(DsEvent e, DsTransaction tx, DsOperation op) {
                super.operationAdded(e, tx, op);
                target.write(++numOps, op.toString());
                return Status.OK;
                }
                .
                public String reportStatus() {
                    return "Processed "+ numOps +" operations: " + target.getReport();
                }
                }
                
                
                

                 

                 

                The param file (dirprm/sample.prm):

                 

                extract sample
                CUserExit libggjava_ue.so CUSEREXIT PassThru IncludeUpdateBefores
                GetUpdateBefores
                SourceDefs dirdef/bb.def
                Table GG.*;
                
                
                

                 

                 

                And the associated property file (dirprm/sample.properties) -- note "sample.foo=some value" calls the method setFoo("some value"):

                 

                # java config
                gg.handlerlist=sample
                gg.handler.sample.type=com.mycompany.bigdata.SampleHandlerHDFS
                gg.handler.sample.foo=some value...
                .
                # these configure the native ggjava dll/so library and the jvm
                goldengate.userexit.nochkpt=TRUE
                jvm.stats.display=TRUE
                jvm.stats.full=TRUE
                jvm.bootoptions=-Djava.class.path=ggjava/ggjava.jar:dirprm:path/to/someother.jar -Dlogback.configurationFile=logback-debug.xml
                
                
                

                 

                Message was edited by: MikeN