This is a browser/java plugin issue and has nothing to do with Oracle Forms. If you open an applet in a browser, as long as that browser remains open the plugin will remain loaded and the java.exe process will remain in the task manager. Because the tabs in the browser are of the same browser session, it won't matter if you close the tab or not. The parent browser session is responsible for the java process. Here are two of many discussions about this topic which began in IE8 and continue today.
I am not aware of any way to do that. Unfortunately, using Java script to open a window will create a shared session. This is really something you need to ask of the Microsoft guys. I know that many people have complained about this new behavior, but I have not yet seen any response from Microsoft that addresses this undesirable behavior. Here is another MS blog that migth offer some clarity on what is happening:
Is there some reason why you are worried about the java.exe proc remaining? Once the browser is closed, it should also close the java proc.