You can simply google for "oracle reports slow toad fast" and you will find that this question has been asked and answered so many times. One answer that says it all:
One of the things I always comment on is that TOAD does not bring all rows back. It only brings back the first 25 or so, depending upon how you have it configured. If you have asked Discoverer to create a Crosstab or Page Items then you are forcing it to retrieve ALL rows before it can even build the worksheet.
(replace Discoverer by Reports and Crosstab by Matrix Report).
In Reports you can also have program units that work on your data.