How to map email activity data to Custom Object Recrod

Hello, I have a question about email activity mapping and will really appreciate your help. We send campaign emails based on custom object records (not contact because we may have multiple custom objects per contact). Is there any way we can map the email activity data (open, clickthrough, etc.) to the custom object records? We can't just map to email addresses because we may have multiple custom object records per email address. Perhaps by recipientId? But I couldn't find a way to retrieve custom object record by recipientId.

Thank you in advance.