Is the issue with all browsers or specific ones only? Do you have pop-up blockers disabled?
Can you find any errors in the Java console window? -- How To View The Java Console And Enable Tracing For Sun JRE (Native Plug-in) (Doc ID 452853.1)
Any errors in the application log files?
E-Business Suite Forms Do Not Open After Deploying Java JRE 7 (Doc ID 2010266.1)
Forms Can Not Open With Jre8 Plugin (Doc ID 2012607.1)
After Upgrade Client To JRE 8 The Forms Applet Fails with ClassNotFoundException Where SSL Is Offloaded Before EBS Apache (Doc ID 1998612.1)
How to diagnose and resolve "Java has discovered application components that could indicate a security concern" in Forms (Doc ID 1120573.1)
have you tried running adautocfg.sh?