How to limit only the assigned Approver can respond the Invoice approve action?

Customer found that not only the original approver, anybody who has the Invoice Email Notification respond template can perform the approve action to the invoice

How to limit only the original or assigned approver can respond the invoice email approval notification

Eg.

1. Approver A received the Invoice approval email notification

2. Approver A accidently fwd the email to a person B who is a external person and should now have the authorization to approve the invoice

3. Person B responded the invoice email notification and Approved the invoice without Approver A's consent 

4. The system accepts the Invoice approval from person B and change the invoice status to Workflow Approved