I've just installed Solaris 11 x86 and it's mostly running fine and dandy except that I have a strange problem under gnome. After some minutes/hours (varies), applications (any applications) will stop launching. Instead they will just spend about 15 seconds at the bottom of the screen as 'Starting "app name"' and then the message just disappears and the app never actually launches. Existing apps will continue to run fine but even a new instance of the same app will fail to run. If I restart gdm, log out, bounce the box etc, everything will be fine again but after awhile the problem returns.
I've seen nothing in the logs, or from top. I tried a ps ax | grep firefox as a test after loading firefox (as an easy example) just to see if it was a case of the app window not rendering but sure enough it isn't appearing in the process list.
I've searched google and the forums but not found anything specific to this.
Take a look at the .xsession-errors file in your home directory. That may contain error messages that explain things. The applications may also just be crashing with a core dump, the coreadm command will tell you where core files are being generated.