I would say that makes sense as a default behaviour.
The user naturally assumes that nothing will happen, if he just exits the modal dialog.
Why would you want to change this behaviour and go against the user expectation?
The problem we face is that we are limited to the requirement. We are replacing a flash based application and the X is the default action in the existing application. That is the only reason. I am trying to convince the users that this should be changed.