Removing Locations

We are on 9.1 and are not using advanced warehousing.

In one of our locations, we are removing some of our rack locations so we want to delete them from our location master.  We want to make sure that we can no longer assign inventory to those locations.  As long as we remove them from the location master, do we need to be concerned about the item/branch/location file?  I don't want to have to remove it from that file given the amount of records that it would affect.  I'm also concerned about links to the file for history records (cardex, etc).