Terminated Worker Records Shows Even if The HR doesn't have access to the BU

There are 4 BUs and HR Users are configured see only employees in their BU including terminated employees.

For example Employee A is part of BU A and HR from XYZ BU cannot see the employee A but when employee A is rehired in XYZ the HR can see the old records from BU A including last salary etc information.

We would like our security to works the same way for rehire as well. Our expectation is that they shouldn't see the old or terminated work records.

Is there a way to control it?