TLS Handshake - Server Authentication

Hi Fellows,

I was Reading about the tls handshake and a doubt come to me.

After sending Client Key Exchange, the Client and Server make the calculation of Secret Key – RSA or DH.

How Server avoid that an intruder can be who is trying to make the establishing of the TLS and not real Client that I server is expecting?

My question comes cause in the server authentication the interchange of the information is from Client and Server and only in the server is where I have imported a certificated signed.

If a client could be anyone that sends Client Key Exchange and Client and Server calculates independently the RSA or DH secret keys how could not be an intruder client?