When I added Google Sitemap to our site, and checked on Webmaster Tools section, the Web Crawl report showed several "URLs restricted by robots.txt" including index page. Our robots.txt file has only following lines.
User-agent: *
Disallow:
Since then, I have removed Sitemap fearing problems with Google search results. What could be the reason for this?
For your information, we do have some 301 redirect pages, and sitemap was showing old as well as new urls.
Please guide me, so that I can generate a correct Google sitemap and add to my site.
URL of our site http://www.lth-hotels.com
Thanks in advance….
What is the site your are working on and I’ll take a look.
Regards,
Jim.
Url of our site is lth-hotels.com
Sorry, I see that you posted the url in the first question and didn’t see it.
You are talking about Google webmaster tools and their report on URLs.
I too am having this problem, but it has nothing to do with the sitemap.
In fact, you can take the sitemap off and you’ll still see the same thing.
Google has been reporting that some of my urls are restricted by my robots.txt file, but in that file I don’t have the url in question listed – and – I do not, never have, had a sitemap on that site.
I thought perhaps I had a typo and it read it as a wildcard or something, but the robots.txt file in question is 100% accurate.
I think it is a Glitch in Google – but It has nothing to do with sitemaps.
I hope that helps! If you find more information, please post it here – I would like to follow this!
Best regards,
Jim.