Jim,
Is there anyway I can make my robots.txt file any more accessible?
Right now I have
User-agent: *
Disallow:
and my awstats for my site, albeit the free version (6.6), for example google stats are 64 +4
and
"Robots shown here gave hits or traffic "not viewed" by visitors, so they are not included in other charts. Numbers after + are successful hits on "robots.txt" files."
Can you think a way to improve the + number or doesn’t it matter becuase I don’t make as many changes to my site as I use to.
Thanks Jim,
Roger
I just set the file up and leave it – nothing else to do.
Also, see the first entry, visit.php, that tells the bots that they can’t access visit.php (other files are fine, but not this file). I don’t really have a visit.php file, but I do monitor calls to it and when a bot or someone attempts to access it, I can act accordingly.
Many website owners place the names of files and directories that they don’t want the bots to visit in their robots.txt file and this is one of the first places a malicious hacker will look for information.
It’s most interesting when you find a search engine that you think would / should respect the robots.txt file trying to access the file anyway!
Regards,
Jim.
Jim,
Holy Toledo. I’ll have to add that to my robots.txt file.
Thanks Jim,
Roger
Robots.txt is a file that you create and upload to your websites root directory that is used by search engine spiders to determine which parts of your website they should index and which folders/pages that you, the website owner, don’t want listed in search engine indexes.