Restrict the fields available in List Views configuration

We have created an EFF for "Preferred Gender" in Person EIT because we are using the out of the box field Gender field for "Birth Gender". We have restricted the ability for recruiters to see this using Visual Builder on the candidate profile. However, the ability to add this and any other Person EFF in list views.

We are looking for a solution for how to restrict the fields that are available in list views to remove these EFFs. I tried to user page composer but there is no ability to customize on the list view configuration page. Alternatively, I tried to remove the gear to access the list view configuration on the pages (candidate search, job applications, candidate pools, etc.) but was also unsuccessful. Note that we are using the redwood experience in recruitment.