How to restrict Line Manager's to Time Card incase of multiple assignments

Summary:

Employee has multiple assignments, each assignment has a different line manager.

Primary Line Manager is able to see all assignments on Assignment dropdown on Time Card UI and vice-a-versa.
Therefore able to create\edit\view time entries of all assignments.

We need to restrict Primary manager to view or edit or create Secondary assignment time entries and vice-a-versa.

How do we restrict line managers access in this case ?

Content (please ensure you mask any confidential information):

Version (include the version you are using, if applicable):

Code Snippet (add any code snippets that support your topic, if applicable):