The timeout and max retries are specified in the App Server configuration file. In this case I think it would be the ones under PSSUBHND_dflt, or possibly under PSPUBHND_dflt. However, these settings will have a global scope affecting all service operations so I wouldn't recommend changing them.
For outbound operations you can change the delivery mode in the routing from Guaranteed to Best Effort to eliminate the retries, but in this case the routing is Local so it isn't an option.
If you can recreate the problem in a non-production environment I would recommend enabling a PeopleCode trace to figure out why the operations are timing out. Depending on your tools version, there's also an app server setting described in the documentation under the SMTP Settings section called SMTPTimeToWaitForResult that may not be listed in the default app server config file. It says the default value for this setting is 10000 ms (10 seconds). You could try increasing it if you discover that the delay is on the mail server side, but 10 seconds is a long time and there could be a DNS or other network issue that needs to be addressed. You could also enable the SMTPTrace setting and see what it gives you (requires LogFence=5 as well).