design rules (including custom ones) are always applied on one relational model. I wasn't able to reproduce a case when that rule is broken.
However there is a bug in "Custom Design Rules" functionality and rule is applied on so called "current" model and selected model is ignored.
That works fine in "Design rules" functionality where selected model is used.
As workaround for "Custom Design Rules" - you can generate DDL using "Generate DDL" icon and that model will become as "current' relational model.
I have explained poorly the issue of the script runs for *all* open models. Indeed, the custom design rule runs for tables on *one* other than the intended relational model, no matter what model I've selected. So, the bug is the same that you have explained.
And as you said, the workaround of Generate DDL for the intended relational model to becomes the "selected" model works fine. After that, the script runs smoothly for the intended model.
Thanks for the right answer.