You're almost there! Please answer a few more questions for access to the Applications content. Complete registration
Interested in joining? Complete your registration by providing Areas of Interest here. Register

About editing robots.txt

Received Response
21
Views
1
Comments

Summary: The following table shows the number of the

There are search engines that are allowed access to all directories by default in robots.txt.

We would like to prohibit robot collection from that search engine for the contact form.

If I add a ban on access to the contact form to robots.txt, does this mean that the default setting will take precedence?

Does the content with #CUSTOM at the end of the line take precedence over the default setting?


-------

example

User-agent: Googlebot # Google # ADDED BY HMS

Disallow: /app/ask #CUSTOM

User-agent: Googlebot # Google # ADDED BY HMS

Disallow: # ADDED BY HMS

Howdy, Stranger!

Log In

To view full details, sign in.

Register

Don't have an account? Click here to get started!