is any other way to open exe file as tab in javafx?
No, you can't open an exe file in a tab.
The only thing you can do with an exe file to spawn a process to run it. If the exe has it's own UI, then that UI will be displayed in UI windows created by the exe process. Those windows would be completely separate from any windows managed by JavaFX.
The process API in Java is sometimes unwieldy and difficult to work with (at least if you need to communicate in arguments and get results out of the spawned process).
Spawning processes in Java has nothing to do with JavaFX. If that is what you want to do, you will get better help sooner if you post a question "How do I spawn a process in Java?" to a general purpose Java forum rather than a JavaFX specific forum.