I guess if you don't start the managed servers with NodeManager they do come up?
Not sure if this is related, but did you take a look at MOS note 1336100.1? It basically says there was a failure during configuration of NodeManager; it is not entirely related to forms but to FMW 18.104.22.168 which is somehow the version number of forms 11.1.2. A cause of it is that NodeManager isn't using the correct script when starting a managed server; this also sounds a little bit like your problem...
Regarding the OutOfMemory exception you might look at note 1312573.1; it is related to running WLS as windows services (without NodeManager); but it might as well apply to running the Servers with NodeManager.
It might be a shot in the dark, but it might be worth a try.
In any case when running into (forms related) troubles with weblogic I found very little specific forms related informations on weblogic; this is probably due to the fact that forms doesn't make much use of weblogic at all (all in all most of the work is done by the good old fashioned C- forms runitmes); but the generic notes on weblogic often did the trick.
I'm going to close this thread for now.
A re-installation made the issue go away. I wish I'd had time to do more troubleshooting, especially to look at the possible solution mentioned by Christian, but time pressures led me down the road of a re-install. The MOS note 1336100.1 looked like a very promising solution, given that the servers would start fine from the command line but not via node manager. Unfortunately, I didn't back up any of the nodemanager.properties files before uninstalling, so I cannot confirm with any certainty the exact cause.
Thanks, Christian, for your suggestion.