I get this error message all the time. Check to see if you have an open MS Word, MS Excel or Outlook with the ADF Component loaded in it. By closing the offending MS application, JDeveloper loads. So annoying.
Let me provide a little more information. I installed the ADF Data visualization components for MS Office - that's when I started to see this error message. I also changed my ide.conf settings to increase memory - but that didn't make any difference. I'm running Windows 7 (64-bit) on a 4Gb Dell Notebook. I don't get the error so long as I'm not running any MS Office programs.
Well, the mx option is only one part of the memory pool used by the jvm. So it might be that you are running out of memory with the bigger value. Even if you see more memory in total, there might not be enough memory in one peace to fulfill the request.