How to remove "To Supplier" Attachment Category from Supplier Profile

We are trying to remove one attachment category from supplier attachment entities. This is because we need to default "To Supplier" for PO attachments, but it defaults for all attachment entities. Supplier attachments shouldn't default to go to Supplier. I can't delete it from Manage Attachment Entities, so how else can I achieve this? Or can I default a category for just the PO entity?