Can we restrict manager from 'Confirm Review Meeting' so only employee confirms the meeting?

We are currently configuring the performance document workflow and would like to customize the "Confirm Review Meeting Held" activity.

Our requirement is:

• Only the employee should confirm that the review meeting was held.
• We do not want the manager to have the ability to schedule a call or confirm the meeting in this task.

Is there a way to restrict the manager's involvement in this specific activity, or configure the task so that only the employee can take action?