Requirement to hide a subset of users from Public Worker View and Specialist Role View

We want to hide a small subset of 'protect users' with a record in the env from all roles/users (except a single specialist role we have created). This group is identified by a Person DFF (a tick box).

We have created a custom job role that provides access to this small group of users, via the DFF identifier. However we are facing issues with removing visibility of these users for everyone else. This is two prong (public access and specialist access).

Public Access:

• Current set up for Connections is a custom SQL to restrict the Public Person Security Profile to only show employees other employees in their own Legal Employer. Paired with the Public Worker Access configured to Workers in Users Legal Employer.