I've Googled and searched the OTN forums and MOS to no avail with this particular issue. We're in the process of filing an SR, but I thought of posting here as well to see if anyone else has experienced the same issue.
Windows Server 2008 R2
JDK 7 64-bit (1.7u11)
Latest Forms/Reports 11gR2
We have installed Forms/Reports in a cluster consisting of 9 servers. As a note, we have done this same setup/configuration with 2-server clusters and have not had this issue. After installing and configuring all of the nodes, we've rebooted the servers and ensured node manager is started on each server. Fusion Middleware control shows all of the servers and their correct state. Starting Forms/Reports servers on either of the first two nodes in the cluster works fine. However, starting the managed servers on any of the other nodes does not work. Fusion Middleware Control "hangs" for a long time (~15 minutes) during this attempted start. Going to the server on which we are trying to start the managed server shows a java.exe process consuming all or nearly all of the CPU. The managed server log shows "java.lang.OutOfMemoryError: PermGen space" errors. I know what PermGen space is and how to increase it, but I was unable to find any information about it in a Forms/Reports environment. The other unusual thing is that there are errors early on in the managed server log related to classes not being found (oracle.dms.http.Request and oracle.security.jps.wls.listeners.JpsApplicationLifecycleListener for example) - this would seem to point to a failed installation, but as far as I know, there were no errors during install/config.
This issue doesn't appear to be related to having more than 2 managed servers started, as we get the same issue if we go just to (for example) server 3 and start the managed forms server.
At the moment, I'm leaning towards a re-installation of the Forms/Reports cluster, but with 9 servers, this takes a bit of time.
Has anyone else experienced something like this, and if so, how did you resolve the issue? I can provide detailed logs here if it helps.
I guess if you don't start the managed servers with NodeManager they do come up?
Not sure if this is related, but did you take a look at MOS note 1336100.1? It basically says there was a failure during configuration of NodeManager; it is not entirely related to forms but to FMW 126.96.36.199 which is somehow the version number of forms 11.1.2. A cause of it is that NodeManager isn't using the correct script when starting a managed server; this also sounds a little bit like your problem...
Regarding the OutOfMemory exception you might look at note 1312573.1; it is related to running WLS as windows services (without NodeManager); but it might as well apply to running the Servers with NodeManager.
It might be a shot in the dark, but it might be worth a try.
In any case when running into (forms related) troubles with weblogic I found very little specific forms related informations on weblogic; this is probably due to the fact that forms doesn't make much use of weblogic at all (all in all most of the work is done by the good old fashioned C- forms runitmes); but the generic notes on weblogic often did the trick.
I'm going to close this thread for now.
A re-installation made the issue go away. I wish I'd had time to do more troubleshooting, especially to look at the possible solution mentioned by Christian, but time pressures led me down the road of a re-install. The MOS note 1336100.1 looked like a very promising solution, given that the servers would start fine from the command line but not via node manager. Unfortunately, I didn't back up any of the nodemanager.properties files before uninstalling, so I cannot confirm with any certainty the exact cause.
Thanks, Christian, for your suggestion.