Thats odd, if you have access to open the breakdown you should have access to delete it. No special role should be needed. I tried setting up a breakdown where I didnt have access then gave myself access it in my 18.104.22.168 local environment and it appears to be functioning.
1. If the breakdown name is a blue link and you still don't have a delete icon then I think you have found a bug.
2. If the breakdown name isn't a blue link but you are in a group that is listed in ADMN as has access to the formula classification. Cache may need to be flushed or you may have found a bug.
Hope this helps a little,