How to restrict Business Unit from showing in Manage AP Periods?

We have a scenario that Business Units that are closed for business activity should not appear in Manage AP Period page, However, Invoices entered in these business units should be visible to the relevant users.

Please guide us how to achieve this requirement as we are currently facing the issue that when we remove the user access from AP Period Closure role, he is still viewing these business units in Manage AP Period page as he has access of Invoice Creation / View for these Business Units.