This discussion is archived
3 Replies Latest reply: Nov 27, 2012 6:15 AM by _Phil RSS

ODI Full Load Vs Incremental Load

970475 Newbie
Currently Being Moderated
Hi,

I am absolutely new to ODI and here in our environments, the development is done in such a way that we need to do the full load everytime when we add a new table in our MIS schema or we add new columns in MIS tables. The whole ODI structure is built in such a way that whenever we make any table structure changes and give the build as Work_Repo.zip file, after the deployment and regenarating the scenarios, a full load is required to be executed to populate newly added tables and columns.

And since the full load takes a very long time to get completed, it really affects the business, so our major concern is to avoid this full load. We have our staging schema different than our source and destination schemas. Which means, we have three schemas: source, staging and target schemas.

My question is, is there any way to avoid this full load after every deployment? If yes, then how can we do that? I have a very less knowledge of ODI till now, please advice me some solution for this. The ODI version is 10.1.3.6.2.

Your help is highly appreciated.

Thanks.
  • 1. Re: ODI Full Load Vs Incremental Load
    Bhabani Ranjan Guru
    Currently Being Moderated
    I dont know whether i understand you properly or not. When you add new columns or new tables, does it have relation with other tables ? If you are adding a new column to your source table, what is the process you are following to update that particular column. If you know that then same logic you can apply in your odi process.

    So its quite difficult to suggest you without knowing the design diagram of your process.
  • 2. Re: ODI Full Load Vs Incremental Load
    970475 Newbie
    Currently Being Moderated
    Hi,

    Thanks for your response. Most of the time newly added columns do not have any relation with any other table. But then also it doesnot get populated by only incremental load. Below is the details:

    Whenever a new column is added into the source schema, it gets populated by our online application form. Now we add that column in our staging as well as target schemas, update the mappings of ODI interfaces for that particular table used for incremental and full load. Now when we regenerate the scenarios and do the incremental load then the new column in staging and target schemas do not get populated with the values. To get it populated, first we have to run a full load. After that only the newly added columns gets populated in by data during our everydays incremental load. Same applies to addition of a new table also.

    Its something like our source schema which is a transcational database, gets populated by our online application. That data should be loaded into our MIS schema for BI work. So populating the MIS schemas is not getting done without a first full load.


    Please let me know if your require any further information.

    Thanks,

    Edited by: user2640279 on 27-Nov-2012 06:07

    Edited by: user2640279 on 27-Nov-2012 06:09
  • 3. Re: ODI Full Load Vs Incremental Load
    _Phil Journeyer
    Currently Being Moderated
    When you say "then the new column in staging and target schemas do not get populated with the values" do you mean historically, or not at all, i.e. even for newly created source records?

    How are you determining what data to pull from your source - are you using a last modified date or ID as a key?

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points