Restrict Activity Access to specific user(s)

We've used access groups to allow our organizations users visibility into all records (Accounts, Contacts, Activities, etc.) so that any user can assist a customer. However, some sales reps have confidential information that they'd like to log on their Account records via activities. Is there a way for them to log an activity but mark it a private so only they (and maybe their manager) can view it on the customers record?