Could it be that you miss the hostname reference in the host file on the gateway webserver to the correct ip adress?
That's not the case. I've removed the hostname from the error message in my original post, but it does mention the hostname of the server. It's more like this:
HTTPS hostname wrong: should be <peoplesoft.oracle.com>, but cert says <CN=www.oracle.com, O=City Province, L=City, ST=ProvinceBogota, C=CO>"
I'll think about this one for a bit. But while I'm thinking one option may be to try a wild card cert it would be for *.oracle.com for example so it can be used for multiple systems on the same domain name. I've used them heavily in large non prod environments to save money and time on cert maintenance tasks. Then again, I've never used one with 8.53, that was 8.48 - 8.50. Maybe worth a try if you are using self signed certs.
It would make sense that this would impact plenty of people, anyone using a load balancer with SSL config would run into this right? Hostname would never match the URL in that case.
It's not a self-signed certificate, but I am discussing trying this with the person responsible for the certificates on the proxy.
When it comes to load balancers; the URL for the Integration Broker gateway would be the URL of the load balancer. Normally that would match the CN value in the certificate. I do agree that probably more people should run into this, because I doubt every organisation has a separate certificate for their PeopleSoft environment.
Okay, I think I was misunderstanding the error and your configuration originally, but I wanted to get that thought of wildcard certs out there since it immediately popped in my head as a solution to any naming mismatch.
Can you explain your setup in more detail... your at a client, demo'ing PS, and connecting back to the home office. What cert is where, is the proxy a reverse proxy? I thought I had it figured out but then what I was thinking should not have created a cert warning at the browser. But I've never done SSL over RPS so I might be missing something. Plus it's the weekend now so my brain is on vacation for 2 days.
Anyway, I'll pull the old PeopleSoft response out of the bag and say it sounds like "it's working as intended". If you have a name mismatch you should be warned/denied access. Perhaps it was working due to a bug.
For what it's worth I tried putting the wrong cert on my 8.52.07 environment and got the following right away when trying to reload the connectors.
HTTPS hostname wrong: should be <psweb1.domain.com>, but cert says <CN=psweb2.domain.com
Another option for now would be to drop the app to gateway connectivity back to non-ssl. Or depending on your setup can you get a new external only DNS entry added just for your env. So say from external you hit ps.yourdomain.com which hits your inbound firewall device and sends it on in to ps.yourdomain.com. Even if it's going through an RPS then at least names match along the way.