Nbr of Addr Book records required when One Vendor has multiple ordering locations

I will also post this question in the Supply Chain Mgmt community.

We are attempting to clean up our Addr Book from past abuses.  I have come across a situation that I am not quite sure how to handle.  We have some vendors that have multi ordering locations with only one remit to address.  I know about the Spec/Factor Payee but if we want to get down to ONE Addr Book number for both ordering and paying how can we do that in this situation when the purchase order would need to print the correct address for the vendor we are ordering from?