How do limit views of tasks in Task Manager

Summary: We were under the impression that viewing a task was based on the Org ID in task manager. We have assigned the Org ID to the Template and to the Schedules. However, the associated tasks did not inherit the Org ID so we assigned the Org ID to the individual tasks also. It is important to us that if a person does person does not have access to an Org ID that they cannot see the task. During testing we wanted to confirm this and found that even if we did not assign and Org