Re: Search Spiders

This WebDNA talk-list message is from

2002

It keeps the original formatting. numero = 44890
interpreted = N
texte = FWIW:>Asked to clarify what URL paths AlltheWeb follows, FAST engineer>Frode Lundgren indicated that while crawling the spider will>follow links from a static page to a dynamic one, but, to avoid>looping situations, will not follow links from a dynamic page to>another dynamic page.http://www.webreference.com/new/020620.html#feature>There are millions of dynamic pages, many of which are exactly what you may be looking for. Indexing dynamic pages has the potential to stall a search engine's crawler, which is why most don't do this. Google does index dynamic pages by following links from static ones.> Quigo indexes only dynamic pages, but not in the same was as Google. Quigo's method is quite high-tech, and while it is still in beta, it is a very good search tool.>This page was last updated on March 8, 2002.http://www.faganfinder.com/invis/index.shtmlGoogle Groups has several threads about dynamic links.http://groups.google.com/groups?q=google.public.support.generalThe consensus seems to be that Google will follow dynamic links (those with a ? in the), but that using Apache's mod-rewrite to remove the ? and & characters is the way to go, not just for Google but for the other SE's as well.Google's algorithm isn't carved in stone. They're known to change stuff with no warning.-------------------------------------------------------------This message is sent to you because you are subscribed to the mailing list .To unsubscribe, E-mail to: To switch to the DIGEST mode, E-mail to Web Archive of this list is at: http://search.smithmicro.com/ Associated Messages, from the most recent to the oldest:

Re: Search Spiders (Glenn Busbin 2002)
Re: Search Spiders (Glenn Busbin 2002)
Re: Search Spiders (Glenn Busbin 2002)
Search Spiders (Alain Russell 2002)

FWIW:>Asked to clarify what URL paths AlltheWeb follows, FAST engineer>Frode Lundgren indicated that while crawling the spider will>follow links from a static page to a dynamic one, but, to avoid>looping situations, will not follow links from a dynamic page to>another dynamic page.http://www.webreference.com/new/020620.html#feature>There are millions of dynamic pages, many of which are exactly what you may be looking for. Indexing dynamic pages has the potential to stall a search engine's crawler, which is why most don't do this. Google does index dynamic pages by following links from static ones.> Quigo indexes only dynamic pages, but not in the same was as Google. Quigo's method is quite high-tech, and while it is still in beta, it is a very good search tool.>This page was last updated on March 8, 2002.http://www.faganfinder.com/invis/index.shtmlGoogle Groups has several threads about dynamic links.http://groups.google.com/groups?q=google.public.support.generalThe consensus seems to be that Google will follow dynamic links (those with a ? in the), but that using Apache's mod-rewrite to remove the ? and & characters is the way to go, not just for Google but for the other SE's as well.Google's algorithm isn't carved in stone. They're known to change stuff with no warning.-------------------------------------------------------------This message is sent to you because you are subscribed to the mailing list .To unsubscribe, E-mail to: To switch to the DIGEST mode, E-mail to Web Archive of this list is at: http://search.smithmicro.com/ Glenn Busbin

DOWNLOAD WEBDNA NOW!

Re: Search Spiders

2002

Top Articles:

Related Readings: