Large amount of Queued url’s

Comments

1. AMPC says:
   April 5, 2008 at 2:35 am

   You can stop the crawling and save out the project (project tab) or you can allocate more memory to the program, see auditmypc.com/free-sitemap-generator.asp

   Best regards,

   Jim.
2. ribor39 says:
   April 6, 2008 at 2:35 am

   Thanks Jim
   I’ve been doing that but everytime I try to change the amount of memory that Java is using Java fails to start . I have seen this problem on Sun Micro’s Forums. The only way I’ve been able to fix it is to leave the Java setting blank. At the moment I have processed around 17’000 url’s but I have had to save and restart a number of times. It takes a long time to load the sitemap back up and stops shortly after that the time keep on going up but no processing. I think this is a Java problem but if you have any help that would be great.
   Many Thanks for your help Great Site

   Kindest Regards,

   Robert
3. ribor39 says:
   April 7, 2008 at 2:35 am

   Thank Jim

   I’m been doing that. If I leave the craw going it will stop processing url’s but the time keep counting up. The bigger (34’000 url’s 18’000 Finished)the sitemap(20mb) gets the slower it is to load. I have added 128mb in to the Java setting if I put in 300mb Java will not load. I added 512mb of ram but still could not boost the Java memory setting. It seems to use a lot of processor power. Is it just that the sitemap is getting to big or my computer is to slow?
   Many thanks for your help Great site Keep up the good work.

   Kindest Regards,

   Robert
   nibor39
4. AMPC says:
   April 14, 2008 at 2:35 am

   Hi Robert,

   Sorry for the delay in getting back to you!

   There are a ton of calculations going on in the background and depending on how complex the site is, the sitemap processing may stop. This may happen on very large sites with a lot of internal linking.

   You could try it on a newer system, perhaps a friends computer and see if the results change, but chances are results will be the same.

   I’m working on solution, just a matter of time. Time, time time….

   Best regards,

   Jim.
5. c2h5oh says:
   April 21, 2008 at 2:35 am

   Increasing the memory available to applets doesn’t seem to be the right solution – what about very large sites with extensive internal linking? With max depth set to 3 I get over 200 000 queued, valid urls with one of my sites…
   The other thing I have noticed is that sitemap generator seems to cache downloaded content in memory – why the hell would you do that? Dump it as soon as it is no longer needed or use disc cache..

Speak Your Mind Cancel reply