Restrict visibility of attachments to vendors

Summary: We recently discovered that attachments intended to be internal are visible to vendors when they use the vendor portal. Is there any way to have an internal and external sets of attachments?

Content (please ensure you mask any confidential information): N/A

Version (include the version you are using, if applicable): 23D

Code Snippet (add any code snippets that support your topic, if applicable): N/A