Document Records - Restrict Line Manager from updating documents for self

We have a need to restrict Line Managers from updating documents for themselves but still have the ability to update documents for users that report to them.

We have tried creating a sandbox to remove the Add button from Me>Document Records but this also removes the Add Button from My Team>Document Records for Direct Report