Restrict Agent Access to SR details when Agent is Primary Contact

Summary

How to restrict an Agent from seeing internal notes and other content when the agent is the primary contact

Content

We are trying to solve the issue where a user who submits a Service Request an issue relating to things worked by HR Dept "1" happens to also be a Help Desk Agent working in HR Dept "2," and as the primary contact, by virtue of his security roles, is getting the agent layout for his ticket, and can see things that resources on the SR team can see, like Internal Notes.

Has anyone arrived at a way to assure a Primary Contact who