Assignment level security is not working as expected.

We have a case in production where, User - TEST_USER should be able to access only the employee who belong to only assignment category 'Contact Center'. However, there is one employee belong to 'Contact center' but this assignment got terminated and got rehired with new assignment category called 'Consumer sales'. Where the expectation is TEST_USER should not access the active assignment which belong to 'Consumer sales' assignment category. But he is able to access the record. Also, we have enabled the assignment level security by enabling the profile option, still he is able to access the other assignment record.