firewall Spyware removal internet speed test web security service free software and tips


Go Back   Web Security > Our Free Tools > Webmaster Tool - Sitemap Generator
FAQ Members List Search Today's Posts Mark Forums Read

Webmaster Tool - Sitemap Generator Questions about building a sitemap, including the XML Sitemap, HTML sitemap, Google and Yahoo! Sitemaps.

Advertisements

frustrated - timing out and exclusions

Reply
 
LinkBack Thread Tools Display Modes
  #1 (permalink)  
Old 01-12-2009,
Junior Member
 
Join Date: Jan 2009
Location: NC
Posts: 1
Default frustrated - timing out and exclusions

I am really abou to pull my hair out. The sitemap generator looks great. However, I kept having time out issues so i was able to increase the amount of memory by following the instuctions and I got a lot farther before it times out the next time.

So I tried escluding my images (which I am not sure I really want to do) but I really want to exclude a folder /phpbb/. This folder seems to have tons of files. I put */phpbb/* in the exclusions but it didn't work. I have read through all of the forums to make sure I am not missing something but I am at the point where I really need help. My url is http://www.bonniesplants.com

Thanks for you help in advance
Layla
Reply With Quote
  #2 (permalink)  
Old 01-12-2009,
AMPC's Avatar
Administrator
 
Join Date: Jan 2007
Posts: 1,415
Default BonniesPlants.com

Hello Layla,

You don't want to do that - it takes a long time for it to grow back and the older we get, the greater chance there is it may never come back

I ran your site and found a number of items, and here is the solution.

I noticed a log of phpbb pages, a ton of login pages and other data you don't need for the sitemap. I noticed that all the important content seems to be on pages ending in .html

So, I add one INCLUDE filter of:
*.html
This tells the sitemap generator to grab only these type of files.

If you do that, then you'll see that the sitemap generator will find a total of 74 good pages and 13 404 errors. The 404 errors start with page:
http://www.bonniesplants.com/HardyLilies2.html

If you follow the row with that error on the sitemap generator SITEMAP page, you'll see it is being called from this page:
http://www.bonniesplants.com/HardyLilies2.html

I'm guessing an old sitemap file that has invalid (expired) links in it?

So, just as a test, remove that page, then run the sitemap generator with the settings above and let me know what happens.

Cool site by the way! I can tell you have put a LOT of work into BonniesPlants.com

Best regards,

Jim.
Reply With Quote
  #3 (permalink)  
Old 01-12-2009,
AMPC's Avatar
Administrator
 
Join Date: Jan 2007
Posts: 1,415
Default PHPBB sitemap generator

I did a little more research. I see you are using PHPBB and this is adding a lot of unnecessary pages, so here is what you need to do for your PHPBB sitemap generator.

On the settings page, add these exclude filters:
*.JPG
*.jpg
sitemap.html
*.mpg
*.wmv
style.*
generate_feed.*
ucp.*
create_syn*
download*
*-u*
posting*
*/images/*
faq.php*
*/download/*
viewtopic.*
*/styles/*
post*

Then add a check next to:
Choose Ignore invalid relative links (extra "../" tokens)
Choose Exclude images

That should do it - you could run with this alone, but I'd take a look and fix any 404 errors before submitting your sitemap. Also, on your for sale page, I see a lot of duplicate TITLE tags, something you'll want to fix if you can.

That's about it - enjoy your new sitemap!

Regards,

Jim.
Reply With Quote
Reply


Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off



All times are GMT -5. The time now is .


Powered by a CPU
vBulletin® v3.8.4, Copyright ©2000-2009, Jelsoft Enterprises Ltd.
LinkBacks Enabled by vBSEO 3.3.2 © 2009, Crawlability, Inc.