HELP!!!!! Work flows are not working on activities

I will try and describe what it is I am looking for as well as the current work flows I have set up and hopefully someone can give me some advise.....

I have created 4 different dynamic page layouts for tasks. Certain pick list items drive the different pages. Setting those up was the easy part. The owner of the task I wanted to be the person completing the task and I created a seperate field called Delegated By which is the person creating the task.

When a new activity is saved, I want the Owner to receive an email notification. When this person modifies the task, I want the delegator to receive an email notification. If the delegator modifies it again, I want the owner to receive another notification until it is finally "completed". As it is right now...each person is receiving emails and it's too confusing. Here is what is set up...