This is a kind of broad question and the answer depends upon what you are doing and doesn't really have much to do with JavaFX.
What your want is a secure transport (e.g. TLS/SSL) and a way for your client to authenticate that your server is trusted and your server to authenticate that your client is trusted.
But briefly, assuming that you control both the client and the server and your application.
2. Export the self signed certificate from the keystore used on your server and import it into it's own keystore as a trusted certificate authority (ca).
3. Distribute the keystore file containing your trusted ca with your application and use the trusted ca keystore from your client when establishing an https connection.
4. For the client login, collect a user/password and transmit it over your https connection using http basic authentication.
Some alternative variations on the above are:
1. Rather than a self-signed certificate, you could purchase a certificate from a certificate authority (then you wouldn't need your own trusted ca cert store on the client).
2. Use form authentication in step 4 and 5 rather than basic authentication.
3. You could generate a client keypairs for your app or it's users and use ssl mutual auth as a first or second factor.