Documents of Records (DoRs) visibility restrictions within multiple Line Managers

Summary

Documents of Records (DoRs) visibility restrictions within multiple Line Managers

Content

At the moment, we are seeing, a worker who have got multiple assignments and have multiple Line Managers, say for the below instance, 

Worker - Eden Gardens 

Assignment 1 - Line Manager 1

Assignment 2 - Line Manager 2 

Assignment 3 - Line Manager 3 

When Line Manager 3 uploads a document of record on behalf of the worker, it is currently visible to both Line Manager 1 and 2 - they can see the documents uploaded by other Line Managers. Is there a way to restrict this? We want the documents to be made visible to the respective Line Managers uploading the documents not to all, however employee should see everything. Please advise.