Restrict Inventory Organization Access

Hi Gurus,
Here is the requirement: We have 5 inventory organizations, and have created a single Inventory responsibility that has org access to all 5 of these inventory organizations. User wants to restrict this responsibility so that every user has access to only one inventory organization. Instead of creating 5 inventory org specific responsibilities, is it possible to use user management and data security policy to restrict users from accessing/changing inventory organizations other than what has been granted to them by data security policy?

Thanks,
Amit.