Pros of Remove: If there were any human-errors while entering a Supplier Site and if they have to be discarded then you may want to use this option. For example, incorrect Supplier Siite imported from external sources via interfaces or data conversion.
Cons of Remove: Auditors will be unhappy since the record itself is no more.
Pros of Inactive: Consider a scenario wherein a Supplier on-site Audit had happened and the auditors found non-compliance with a specific Supplier Site's operations. In such a case your company had decided not to create any more transactions (or business) with that Supplier Site until the Supplier complies with agreed contractual policies and Auditors gives a green signal. And that's where you would use 'inactive' to simply inactivate the Supplier Site, and then reactivate after it passes a second audit.
Cons of Inactive: You are not recording the date on which it was inactivated. However database captures the 'last_update_date' column (which is not helpful unless you have turned on the Audit feature).
Pros of Inactive Date: Consider a scenario wherein your supplier had an warehouse in Auckland. But now they have closed that warehouse and moved it to China. In this case you want to still maintain the previous transactions associated in the system, plus from an audit perspective after inactivating it you add some comments there. That's where the feature of 'inactive date' will be handy wherein you clearly show in the system when the site became inactive (and the reason for such an action).
Hope this helps.