You do not state which SQL Developer version you use or the specifics of how you export the data, but as a general rule ...
- Exporting in csv format takes less memory than xls format
- If the query results grid does not contain all rows yet, the full query runs again, taking even more memory
The xls memory issue is fixed in the next release. You can find many similar discussions if you search the forum.
I forgot to add, you can always try increasing the JVM memory limit:
- Put an "AddVMOption -Xmx1024M" line in your sqldeveloper.conf file (sqldeveloper-Darwin.conf on Mac OSX), where the new memory limit exceeds the 800M default value in the ide.conf file.
And, no, tkprof will not help -- that pertains to SQL execution performance on the database, not Java memory consumption / limits.