Hello. I have a file with over 100 000 lines.
I want to locate the 5th element of each line which uses delim '|' and check the value against the next line, of the 5th element with delim '|'. And if this is a duplicate I want to remove the line.
Can someone please explain me how to do this?
Message was edited by:
Read two lines, use String.split("\\|") to split those lines, look at the 5th element of each resulting array, compare them using equals. If they're equal, write just one line into a new file, if they'Re not equal, write both lines.
I guess the neatest way would be to create a Row class with a String (line) c'tor and an appropriate equals method, and throwing each Row into a Set. Then write them and you're done. 100000 lines are no big issue, memory-wise.