Hide/Remove Assign Goal Button under Manager Goals Center

Summary:

Hide/Remove Assign Goal Button under My Team » Goals Center

Content (please ensure you mask any confidential information):

I would like to remove the option from managers to assign/add goals to their reports. I expected with 24C, not having the HRG_ASSIGN_PERFORMANCE_GOALS_PRIV privilege assigned to our custom roles will solve this requirement. However, it turned out this privilege only removes the assign action under Me » Goals Center » Actions button but not from the manager page.
I couldn't find any option in Visual Builder Studio to hide this button, either.

Do you know how the requirement of hiding the Assign Goal button could be reached?