Can you limit the visibility of the candidate pool for employees assigned as an Interviewer?

We are operating on PS 9.2, PUM 45, Tax Update 23D, PT 8.56.25

For open job postings, we assign interviewers to assist in the interview process. We understand that assigning someone as an interviewer will add them to the recruiting team and will grant them various access to the job opening.

Is there any way to limit the access that Interviewers are granted and limit the visibility of the candidate pool for the job opening? Often times, interviewers could potentially be peers for the open position and therefore we want to limit who and what the interviewer can see. The other thing is that some of the interviewers may be Hiring Managers on other job postings and with the Hiring Manager security access they see more information than what we would like them to see for job openings where they are only part of the interview panel.