Reading file and validate data issue

882656

Hey all !

I'm developing an application where I get info from a file and I have to validate the data. What I want is to improve performance. Now I'm thinking about using a PRODUCER-CONSUMER approach to improve performance, having 1 thread reading data from file and having 1 thread validating data.

What do you think? Is there a better approach on this kind of problem?

Thanks in advance

796440

879653 wrote:
What I want is to improve performance.

Why? Have you measured it and determined that it does not meet your specific, documented requirements?

Or is it that you have an assumption that something you are doing is inherently "slow" or "inefficient."

Now I'm thinking about using a PRODUCER-CONSUMER approach to improve performance, having 1 thread reading data from file and having 1 thread validating data.

Using multiple threads may or may not improve performance. The issue you have to consider before you even put multithreading on the table is whether the tasks you're planning to multithread can logically be run independently of each other. Certainly for the most basic form--a single producer thread reading from the file and inserting into the queue and a single consumer thread reading from the queue and processing the data--should be viable. But the next logical step--multiple consumer threads--may not be viable, if processing needs to occur in the same order as reading.

And of course, it doesn't make sense to invest in improving performance unless there is an actual performance problem in the first place.

And if the processing time after reading a chunk of the file is not on the same order of magnitude as the I/O to read that chunk, then you won't gain any noticeable performance by adding another thread.

What do you think? Is there a better approach on this kind of problem?

I think you haven't really defined an actual problem yet.

DrClap

I think that particular idea isn't likely to improve the performance (by which I assume you mean the elapsed time to completion) of your code much. But then that's just a guess; if you really want to improve the performance of a task then your first step should be to find out which parts of the task are the worst-performing (for whatever definition of "performance" you have in mind).