I'm currently installing RAC using Oracle 11g Release 2 but unfortunately while installing the Grid Infrustructure, it failed to continue because of the error "INS-40904 ORACLE_HOSTNAME does not resolve to a valid host name", I have configured the public hostnames including vip and scan in DNS Server. I am able to nslookup all the configured hostnames.
ORACLE_HOSTNAME is an environment variable. Check your .profile (.bash_profile etc) and see if its being set there. echo $ORACLE_HOSTNAME to se what its set to currently. My bet is that $ORACLE_HOSTNAME is being set somewhere in your environment and simply needs changing to be correct;
I know this as when I was following the www.ORACLE-BASE.com 11g RAC install I used their profile and missed the fact that it was setting ORACLE_HOSTNAME to RAC1!!!!