Restrict Employee to view Managers and Peers in the Direcory

Summary

Is there any way to restrict employees view in Directory using security

Content

We have an requirement to restrict employees to view their peers and managers in the Directory Search. Currently predefined Employee role Public Security Profile is set ot View All Workers. 

Steps I followed,

- Created custom employee role by copying from seeded employee role

- Updated the data role for custom employee role and set the Public Security Profile value to "View Own Record".

Is there anyway to restrict via security? If yes, what will happen if I have a combination of roles. lets say, Employee & HR Specialsit