I am not able to ssh ipvip from outisde the RAC nodes
You should not be using ssh with virtual IPs at all. Not inside the RAC. Not outside the RAC.
A virtual IP can reside, over time, on a number of DIFFERENT host platforms.
A ssh expects that IP to reside (always) on a SINGLE host platform. (it stores the host platform's signature against that IP as a known host)
So when a virtual IP address moves from one platform to another, ssh will fail with a man-in-the-middle-attack. The new host signature of that virtual IP is different from the host the IP was on previously.
Yes, you an manually fix it.
No, it does not make sense.