4 Replies Latest reply on Apr 16, 2014 5:16 AM by Savi17

    Partial Update not running

    Savi17

      Hi All,

      I am trying to build a partial update pipeline for the first time. I tried to follow the steps given in Partial updates guide. But when i tried to run the partial update script, initially it  was throwing an exception "SEVERE: Partial updates not implemented.

      Occurred while executing line 3 of valid BeanShell script". But I modified the "DataIngest" file in C:\Endeca\Apps\Discover\config\script by commenting the exception part.

      Now my partial update script is running but it gives me a following warning message and nothing gets updated or added in my jspref page.

       

       

      C:\Endeca\Apps\Discover\control>partial_update.bat

       

       

      [04.15.14 17:31:45] INFO: Checking definition from AppConfig.xml against existin

      g EAC provisioning.

      [04.15.14 17:31:45] INFO: Definition has not changed.

      [04.15.14 17:31:45] INFO: Starting partial update script.

      [04.15.14 17:31:45] INFO: Acquired lock 'update_lock'.

      [04.15.14 17:31:45] WARNING: Partial update data extract flag 'partial_extract::

      adds.txt.gz' exists with no associated file in incoming directory 'C:\Endeca\App

      s\Discover\config\script\..\..\.\data\partials\incoming'.

      [04.15.14 17:31:45] WARNING: Partial update data extract flag 'partial_extract::

      deletes.txt.gz' exists with no associated file in incoming directory 'C:\Endeca\

      Apps\Discover\config\script\..\..\.\data\partials\incoming'.

      [04.15.14 17:31:45] WARNING: Partial update data extract flag 'partial_extract::

      updates.txt.gz' exists with no associated file in incoming directory 'C:\Endeca\

      Apps\Discover\config\script\..\..\.\data\partials\incoming'.

      [04.15.14 17:31:45] WARNING: No partial update data ready for processing.

      [04.15.14 17:31:45] INFO: Released lock 'update_lock'.

      [04.15.14 17:31:45] INFO: Partial update script finished.

       

      Please help

       

      Thanks in Advance.

      Savi

        • 1. Re: Partial Update not running
          TimK

          You need to set partial update locks on exactly the input files that you reference in your partial_pipeline.epx diagram - at least one and all of the required ones (if you have several and some are optional)

           

          Out of the box, the load_partial_test_data script sets flags for adds.txt.gz, updates.txt.gz, and deletes.txt.gz.  Edit the load_partial_test_data script to see how it is setting this flag.  Then swap out the files it references to the ones your partial pipeline references.  For example, change to

          "${WORKING_DIR}/set_partial_data_ready_flag.sh" my-test-file.txt

           

          After you do that, running the partial_update script should work just fine.

          • 2. Re: Partial Update not running
            Savi17

            Thank you very much. It worked

            I changed the file names in load_partial_test_data script to the ones my pipeline references and it worked.

            But what do you mean by "You need to set partial update locks on exactly the input files that you reference in your partial_pipeline.epx diagram - at least one and all of the required ones (if you have several and some are optional)"? can you please explain in detail.

            • 3. Re: Partial Update not running
              TimK

              Not much else to say, you did it!

               

              But lets say in your partial update pipeline you had 4 sources, and 2 were set to data required, I just meant that you would need to make sure that you set the flags for those two, and optionally the remaining two.

               

              This framework ensures that a partial update will only process data files that you explicitly set as being ready via these flags.  I'll leave it to you to think through the implications especially if you have a complex partial update strategy/requirements.

              1 person found this helpful
              • 4. Re: Partial Update not running
                Savi17

                Grt! Thanks again