0 Replies Latest reply: Aug 18, 2013 12:40 AM by Joseph Hwang RSS

    java.lang.IllegalStateException: "Reducer has been already set" Exception when invoking ChainReducer.setReducer 2 times

    Joseph Hwang

      I chained several Mappers and Reducers like below

       

      MapperFirstRelocalize(Mapper) => MapperSecondSortNCalc(Mapper) => ReducerThirdSortNCalc(Reducer) => MapperFifthCarrierCode(Mapper) => ReducerFinalOutput(Reducer)

       

      Thelast ReducerFinalOutput reducer throws exception like this;

       

      Exception in thread "main" java.lang.IllegalStateException: Reducer has been already set at org.apache.hadoop.mapred.lib.Chain.setReducer(Chain.java:278)

       

      Is it impossible to invoke ChainReducer.setReducer method 2 times in the same JobConf? How can I chain 2 reduers in one JobConf?

       

      Any advice will be appreciated.

       

      This is the driver class source.

       

      public class HadoopETLDriver extends Configured implements Tool {

       

        @Override
        public int run(String[] arg0) throws Exception {
          // TODO Auto-generated method stub
          JobConf conf1 = new JobConf(getConf(),HadoopETLDriver.class);
          conf1.setJobName("HadoopETLDriver_1");
                     
          FileInputFormat.setInputPaths(conf1, new Path("/home/user01/input_temp"));
          FileOutputFormat.setOutputPath(conf1, new Path("/home/user01/output"));

       

          conf1.setInputFormat(TextInputFormat.class);
          conf1.setOutputFormat(TextOutputFormat.class);
          conf1.setPartitionerClass(GroupKeyPartitioner.class);
             
          JobConf mapperFirstConf = new JobConf(false);
          ChainMapper.addMapper(conf1, MapperFirstRelocalize.class, LongWritable.class, Text.class, CarrierKey.class, Text.class, true, mapperFirstConf);
             
          JobConf mapperSecondConf = new JobConf(false);
          ChainMapper.addMapper(conf1, MapperSecondSortNCalc.class, CarrierKey.class, Text.class, CarrierKey.class, IntWritable.class, true, mapperSecondConf);
             
          JobConf reducerThirdConf = new JobConf(false);
          ChainReducer.setReducer(conf1, ReducerThirdSortNCalc.class, CarrierKey.class, IntWritable.class, CarrierKey.class, IntWritable.class, true, reducerThirdConf);
             
          JobConf mapperFifthConf = new JobConf(false);
          ChainReducer.addMapper(conf1, MapperFifthCarrierCode.class, CarrierKey.class, IntWritable.class, Text.class, Text.class, true, mapperFifthConf);

       

         // It works well until this line
         
          JobConf reducerFinalConf = new JobConf(false);

       

          // java.lang.IllegalStateException: Reducer has been already set Exception is throws on this line
          ChainReducer.setReducer(conf1, ReducerFinalOutput.class, Text.class, Text.class, Text.class, Text.class, true, reducerFinalConf);

       

          JobClient.runJob(conf1);

       

             
          return 0;
        }

       

        public static void main(String[] args) throws Exception {
          // TODO Auto-generated method stub
          int res = ToolRunner.run(new Configuration(), new HadoopETLDriver(), args);
          System.exit(res);
        }