Unable to restrict access using custom person security profile and AoR

Summary:

We have a requirement where we need to create a new role that gives us access to only users belonging to an agency DFF "Agency", and further restricts access by Department Hierarchy.

We created a new person security profile with custom criteria and attached it to the data role.

The issue is if we select only the "Custom Criteria" in the Person Security Profile and do not try to restrict it further by selecting "Area of Responsibility" it works fine.

But if we enable both at the same time so that it restricts using the custom criteria and also restricts by Dept Hierarchy the role doesn't work. The person inheriting the role is unable to search for anyone.