Need ability to restrict Ship-to location values for AP Invoices

Summary: We are able to see that Ship-to location field (on invoice work area and Supplier Portal invoice creation page) shows ship-to locations across Location sets in the list of values. We have locations setup on the application, along with the Location sets.

We are able to see US/Canada locations while creating invoices under the AU/NZ Business Unit. Since these are not AU or NZ locations we would like to restrict the list of values under the Ship-to Location.

Due to incorrect Ship-to locations being selected by Suppliers (on Supplier Portal) and AP invoice processors on AP invoices, this is resulting in incorrect and no tax calculation on the invoices.