Solved the problem myself, it was a simple thing.
All i had to do was rename the automount map in LDAP from auto.nethome to auto_nethome and now everything works.
There is a delay though of about 20 seconds when logging in or doing su - user, will investigate that with pam module debug logging.
Seems as if the whole pam configuration have changed for Solaris 11.1 but the documentation like man pages have not, still reference /etc/pam.conf pam configuration instead of /etc/pam.d/. :)