6 Replies Latest reply on Jan 30, 2013 8:14 PM by jschellSomeoneStoleMyAlias

    Why is STDOUT to STDIN so dang slow in Windows?

    802457
      I have a JAR file that reads a specific file format and writes the data to STDOUT.

      I have another JAR file that reads this data from STDIN and processes it.

      This data has hundreds of millions of records, each of which are ~2K bytes long.

      When i read this data from a text file i can read it at a blazing speed, like multiple million records per minute.

      However, when i run:

      java -jar DataReader.jar | java -jar DataProcessor.jar (this is an example, not the EXACT name of the jars...)

      the processing time is MUCH MUCH MUCH less. We're talking MAYBE 20-30K per minute.

      I use BufferedOutputStream in the class that writes to STDOUT AND for the class that reads from STDIN.

      I have tried increasing the buffersize for the above classes in multiple increments, but it doesn't seem to change anything by much.

      Is there some sort of an Achilles Heel that i'm missing that can cripple the speed of this operation if i'm not aware of it, or is this just a naturally doomed process because of something in Windows?

      What kind of throughput should i expect in this situation? What is fair/expected?

      Any info is greatly appreciated!