MOAC Responsibility Business Group access, how to restrict from several Operating Units?

We have a Payables responsibility that is setup to access multiple operating units by using security profile settings to a business group. We need to restrict this responsibility from accessing a few of the operating units. Is there a way to have MOAC access but limit specific operating units? Or is there a way to restrict through Grants?

Note: the responsibility is primarily used for Inquiry and Reporting activities.