Importing Multiple Categories Problem?

I import thousands of products at a time and our prices change quite often.  We need to move products into certain categories then move them out again.  I need to be able to update this in a CSV import.

In my CSV Import doc I have a column for "Item Site Category 1" and "Item Site Category 2" etc.  I have no problem adding a product to multiple categories by following this format BUT when I need to remove the product for one of the categories or several of the categories I am having a problem. 

Example: Say I have a product in Category A and Category B.  In my file, Category A will be mapped to column "Item Site Category 1" and Category B will be mapped to column "Item Site Category 2".  I can upload the file and everything works great.  Now say I want to keep my product in Category A but remove it from Category B.. how do I format my file.  Currently I am keeping my Category A in column "Item Site Category 1" but leaving my column "Item Site Category 2" blank.  I was hoping this would overwrite the field