Is there a way to restrict Alerts in a Dev environment?

Summary:

We have an unmasked Dev environment with real employee emails where we were doing some testing. The "Notification" setting is turned off to prevent any mass email notices going out, but we now have a case where on employee received an email alert due to their Personal Payment Method being changed. We went to Alerts Composer and identified the particular alert that was triggered and have turned it off, but we have 400+ alerts and are concerned other unintended notices could go out, since they are not being stopped by the "Notification" setting. Is there another setting