User-agent: googlebot Disallow: /*jsessionid=* Disallow: /*?sc=* User-agent: * Disallow: /s.nl/ Disallow: /core/
I'm looking at my robots.txt for the first time in a while and I am now concerned with the blocking of the ?sc=* pages... I originally did this because I figured it would eliminate the duplicate content that results from an item having a unique query string for every category it's in. But why is Yahoo not showing the query-stringed URLs in it's search results when that part of my robots.txt doesn't apply to them? Any SEO experts have advice on this?Thanks,Jacob