Can anyone help me out on this?
If you try the same URL using wget are you able to get the file?
wget <site URL>
Also check if your seed url is http and the site is redirecting to a https.
I have checked it and i am able to ping the site using wget siteURL and am able to get the response.
It is also not redirecting to https. The major problem is that if I try crawling without the proxy server I am able to crawl and if I try to connect through with proxy I am getting all the sites blocked.
Thanks & Regards,
Karthik, you did not mention that you are not able to crawl through proxy before.
It seems like your proxy is blocking it. Did you test your wget with a proxy?
wget -e use_proxy=yes -e http_proxy=proxy.abcdcorp.com:5555 <URL>
If wget can't get the content the crawler can't either.