I assume, there is no limit for the number of open files per Solaris system. The number of files which can be opened on Solaris 10 and above is only limited by the amount of available memory.
Can you describe what you try to achieve, why and how?
To the best of my understanding you are looking at a per process limitation and removing the limit entirely seems quirky to dangerous. Defaults for process.max-file-descriptor should be around 256, 65K, 2.15G.
I'd try to investigate whether it is actually necessary to have more than 65K descriptors open (maybe being more aggressive in e.g. closing tcp connections might bring some relief) and if that's really the case, define a project and provide the adjusted values there.