I have got a requirement to load some files from a remote machine ("SOURCE") into the Database.
I am using the OdiSftpGet to pull the files across from the remote machine and then process the files.
The Remote machine user can edit or add any files but they never delete/archive any files on their side.
This is an issue for me since every "pull" brings ** ALL** the files across and I end up re-processing these again !
I simply do not want any files which were already pulled over by me in a previous run (provided they weren't modified)
Is there anyway I can restrict only
(a) Newly added OR
files on source to be pulled across to my machine ?
Kindly note that I cannot write any scripts on the SOURCE side. Nor am I supposed to ask them to change their processes (I cannot ask them to rename files after I have loaded them)
Is there any ODI component or setting that can help me here ?
thanks in advance