Remove Duplicate Records from CSV using OIC
SummaryRemove Duplicate records from CSV
We have a csv file which we download from source ( EPM Cloud). We need to remove duplicate records from the csv in OIC before we insert the data in target. File is sorted and hence duplicate records will be present one after the other
The only option I think of is to read file record by record using for-each and compare current and previous record. However, since the file is huge, we have to read it in segments and do record by record comparison.
Is there any other simpler way of doing this?