It would be fantastic if the Custom Libraries window was non-modal. Since there is no debug on that screen or even any way to run the code from that screen, you have to close this window to try the code in the Rules or Transformations windows. Of course this means that changes in the Libraries window will need to be automatically reflected in the other two windows, which currently is not the case. To get changes to show up, you have to switch to another script and switch back.
I will log an enhancement request on this.