Enabling SAML2 SSO on both Admin and Managed servers

I followed this example (https://blogs.oracle.com/blogbypuneeth/steps-to-configure-saml-sso-with-azure-as-idp-and-weblogic-server-as-sp ) and setup SSO on the Admin Server. Then I setup SSO on the Managed server running on a different port using the same Asserter but different Identity Provider Partner. The login kept failing because it was always authenticating using the first Identity Provider Partner. I had to remove that first Identity Provider Partner and got it to work on the Managed server. Obviously, it disabled SSO on the Admin server. Is there any way to have SSO enabled on both Admin and Managed servers (or on two Managed servers running on tow different pots)?