How to keep portals off the search engine list

Summary:

Customer would like to setup vanity URL for the non-prod portals.

Does setting up vanity URL risk portal to be listed on the search engine?

Does DCS have ability to update robots.txt to disallow indexing?

Is there a way to enable basic auth to keep crawlers out?

Content (please ensure you mask any confidential information):

Version (include the version you are using, if applicable):

Code Snippet (add any code snippets that support your topic, if applicable):