Adding access groups to category and topic communities

Summary:

We have multiple departments within one oracle platform so all employees will be using the learning catalogue. We are using different category and topic communities in the learning catalogue for each department. I thought we'd be able to set the category to secret (only learning administrators, the community managers, and members can access the community. No one can recommend the community) then add department access groups which would mean only employees in the access group department could see the categories at the top. However, when I do this, the category disappears for everyone, including members. Should this happen?

Content (please ensure you mask any confidential information):