Remove / delete journey assignment

Summary:

Is there a simple way I am missing to only allow certain people to delete journey assignments one-by-one? I know within Display Settings, I can control who will have the option to delete, but none of the options fit. We really only want HRIS, as an example, to be able to delete journey assignments (not the Assignee, LM, Other User, or Initiator). I do have an HSDL to do this, which is fine, but I feel like I must be missing a way to delete just one journey that was erroneously assigned at some point using an easier method?