I can provide some background about how the modal date picker window deals with this issue - and I think that you can use the same technique...
The UIX body component has a built-in unload handler that gets called when the page is unloaded - for example, when you post data in your modal window. The unload handler will check to see if the page is a modal child of its parent window - and if so, will re-enable the parent window (the assumption here is that once the page has been unloaded, the modal child will be hidden).
The solution that UIX uses for the date picker window is to place the contents of the child window inside of a frame set. The actual contents of the window are embedded inside of a frame inside of the frame set. That way, when you navigate inside of the child window, the body's built-in unload handler is still called - but, it no longer enables the parent window, since the body is no longer a child of the parent window - but is a child of the frame set.
I think that you can implement a similar solution by embedding your modal window contents inside of a UIX frameBorderLayout component.
I was having a problem similar to Chrisitan's so I tried embedding my modal window contents inside of a UIX frameBorderLayout component as you suggested. This works in part but I am still able to click on the parent window and bring it to the foreground. Is there a way to disable this behavior.
I took a look at the Uix 2.1.7 Demo UIX Component Guide and noticed the same behavior for the lovField widget. This widget is set up to be model but its not. If you test this widget you will see what I mean.
Thanks for the info. I am using an older version of IE (IE 5.5, Windows 2000) - and the lovField child window modality seems to be working as expected. I'll see if I can track down a version of IE/Windows that is closer to what you are using. If you happen to have access to other versions of IE/Windows, can you let us know if this problem repros consistently across versions?