The problem is that the error notification displays on the Apex page and not within the Modal Region itself. Is there any way to get the error notification to display within the modal region template?
Did you consider putting your popup on another page and using an iFrame?
That way close popup is just a process on that page, and won't be run when the validations fail, so your notification will show on your pop-up.
Alternatively you could do "live" validations by using jQuery/Ajax and do the validation while the user is typing, but I think using an iFrame is better.
This is the reason why i like using the skillbuilders modal page plugin so much too. It does all of the handling and it uses an iframe. When i need dml done in a modal popup i always create a new page and use that. This gives you much more control over what is happening: there is a page template you can adjust and the dml processes are contained.