Restrict BU from being recognized by IDR

Summary: We have many BUs and most of them have same address. We have enabled the BU defaulting based on recipient's email ID. But when an invoice is sent to the recipient mapped to a specific BU then IDR is not recognizing that specific BU but its picking another wrong BU which has same address. We have observed that its picking a specific BU which we don't want IDR to recognize. So is there a way we can restrict some specific BU from being recognized by IDR?

Content (please ensure you mask any confidential information):-

We have many BUs and most of them have same address. We have enabled the BU defaulting based on recipient's email ID. But when an invoice is sent to the recipient mapped to a specific BU then IDR is not recognizing that specific BU but its picking another wrong BU which has same address. We have observed that its picking a specific BU which we don't want IDR to recognize. So is there a way we can restrict some specific BU from being recognized by IDR?