Unexpected Collector Assignment When Party Has Multiple Accounts

We have set up territory management to assign collectors at the ACCOUNT level based on various business rules.  The assignment overall is working as expected except when there are multiple accounts belonging to the same party.  Based on the rules we expect each account to be assigned to different collectors.  However we are seeing that within a party all accounts get assigned to one collector regardless of the rules.  We did not choose to assign collectors at the party level.  Is there a way to get the individual accounts within a party automatically assigned to the correct individual collectors?