Can we restrict Population visibility while creating Talent Review meeting for Custom Role?

Summary:

Business Scenario: We want the line manager to have access to the talent review springboard under My Client Group

Line Manager should be able to create Talent Review Meeting from My Client Group> Talent Review

However, while adding a population to the talent review meeting, the Line Manager should only be able to view his or her direct reports and add them to the review population. We have used Person Profile Security as Manager Hierarchy and Manager Type as Line Manager. Yet, the Line Manager is able to add and view other participants as well. Is there any workaround or solution to restrict visibility to only direct reports?