Physics background:
http://en.m.wikipedia.org/wiki/Deflection_(physics)
Programmatic formula (elastic deflection section) - you'll need to convert that code snippet to Java:
http://xnawiki.com/index.php?title=Vector_Math
Quoting a gamedev forum 2d answer (last post here: http://www.gamedev.net/topic/615690-calculating-angle-of-deflection/)
the reflected vector is: V - 2N(N.V) where N.V is the dot product of N and V, if you don't know what the dot procut is, then N.V = Nx*Vx + Ny*Vy, so that you end up with the vector { Vx - 2*Nx*dot, Vy - 2*Ny*dot } <-- no trigonometric functions needed.
Just use 3d vectors (with z coords) rather than a 2d ones with only x and y. Also reduce the velocity to account for the energy lost in the bounce.
You'll find the functions you need in the Point3D class:
http://hg.openjdk.java.net/openjfx/8/master/rt/file/54bb3abd41fe/javafx-ui-common/src/javafx/geometry/Point3D.java
Use jdk8 to get a version of JavaFX that includes these methods:
http://jdk8.java.net/download.html