Can you restrict case manager access by queues that they are in?

Summary:

We have a requirement for a case manager to only be able to see cases for which queues they are assigned to. We have a custom security role built which is still allowing the case manager to view all cases. When testing with the seeded case manager role, the user can see all cases even those they are not in the queue for. Any ideas on how to remove this access? 

Content (please ensure you mask any confidential information):

Version (include the version you are using, if applicable):

Code Snippet (add any code snippets that support your topic, if applicable):