External Agents access to fusion

Hello,

We are adding External agents via ORC and they will role and particular User_Category assigned automatically.

We have single sign on enabled ,so Agents are not able to direct sign on to fusion since its directing to IDP page.

Now we have added those agents details (first name, last name, email id ) in Azure and in Fusion agents created through ORC.

is there any additional details need to be added in Azure/ Fusion to get agents sign on? does any one has come across this kind of issue , if yes, what solutions you have used for this?

( As per business "Enable chooser login " we can't apply , so this solution we are not using)