2 Replies Latest reply on Sep 26, 2019 1:21 PM by 3930347

    tableInsertDao.defaultBatchSize impact if changed from default?

    3930347

      In override.properties file currently the line is commented

      tableInsertDao.defaultBatchSize

      Iam assuming default is 100 if this commented.

       

      I have Merge Data Streams and Group and Merge with around 500 columns and 500k records.

      With the default value of 100, the job is taking very long time.

       

      Can you let me know the impact if default value is changed to 1000 to 10000 ?

        • 1. Re: tableInsertDao.defaultBatchSize impact if changed from default?
          Mike-Matthews-Oracle

          In most tests, changes to this parameter make little difference to end-to-end performance.

           

          Things to check:

           

          1. Do you really need to use Merge Data Streams? You should only ever use this if you are merging from multiple readers - NOT from multiple paths from the same reader (where any standard processor can unite paths). Unfortunately, it is very often misused.

           

          2. Could you re-configure your process such that the Group and Merge acts only on the columns it really needs to and you use a Lookup and Return for the rest? This would avoid passing so many columns through and would likely radically improve performance.

           

          Regards,

           

          Mike

          • 2. Re: tableInsertDao.defaultBatchSize impact if changed from default?
            3930347

            Thanks Mike.

             

            1. Do you really need to use Merge Data Streams? You should only ever use this if you are merging from multiple readers - NOT from multiple paths from the same reader (where any standard processor can unite paths). Unfortunately, it is very often misused.

            Ans) In this case we are reading from multiple readers and merging using 'Merge Data Streams'.

             

            2. Could you re-configure your process such that the Group and Merge acts only on the columns it really needs to and you use a Lookup and Return for the rest? This would avoid passing so many columns through and would likely radically improve performance.

            Ans) Thanks. I will work on this option.

             

            Separately,

            What is the schema and table name in EDQ Database where I can get the configuration of EDQ Objects like process/reference data,snapshot etc.,

            You help me with 1 table. I can find other tables.