Ability to allow hiring managers access to see applicant notes

Is there a way to allow hiring managers the ability to see applicant notes without also allowing Interviewers the same access?

When we grant the Hiring Manager access thru security thru the Menu: HRS_HRAT and Component: HRS_APPLICATION this will allow the Hiring manager to have access to the Questionnaire, Notes, Offer and Hire pages on the applicant application. However, it also allows members of the Recruiting Team (Interviewers we add to the job opening) access to see this information as well if they have the Hiring Manager role.

This creates a problem if the interviewer (recruiting team member) is also a peer of the position being hired.