Re: Indexing Web sites?
This WebDNA talk-list message is from 2000
It keeps the original formatting.
numero = 34015
interpreted = N
texte = So, if I have 14000+ items, my index page will have an additional ~550k of text to deal with these stupid bots (figuring on 40 bytes/item)!!!That's it, I am going to write a WebCat-based spider catcher. I wonder if most spiders report that the referrer page was robots.txt when they hit the main index page? Anyone who wants to help (I'll need to know IP address and browser names for spiders, for example) can contact me offline. I hope to create an include file which could be easily linked to any page to redirect spiders to another section of the website.John Peacock jpeacock@univpress.com John Butler wrote: > > storebuilder makes a bunch of links at the bottom of a page which are transparent gifs > linked to those static pages... > (at least that is what we determined in a back & forth between me - who never looked at > storebuilder code, and Glenn who posted some of it) > -John > > John Peacock wrote: > > > And how are you directing the spiders/bots to the static pages, while > > your users are getting the active pages? This is always the problem, > > not creating the static pages. Whoever came up with the robots.txt file > > in the first place was a complete idiot! > > > > John Peacock > > > > Paul Uttermohlen wrote: > > > > > > > >The documentation for version 4 says it has the following new facility: > > > > > > > > > >> * Automatically index your web site for maximum search engine exposure. > > > > > > > > > >but I can't find any reference to this in the documentation. Where is it? > > > > > > > > > >Thanks. > > > > > > > > > > > > >Neither can I. > > > > > > > >If you use StoreBuilder, it will create files in a folder named > > > >staticpages. That's the stuff for search engines to find. You can get the > > > >general idea of what StoreBuilder does and use it for sites built by hand. > > > > > > > >Glenn > > > > > > > > > > if you don't want to use store builder and you still want to create > > > static pages from a products database, you can easily do this without > > > upgrading to 4.0. > > > > > > I just did this last week: > > > > > > if the products database modification date is newer than the static > > > pages folder then the static pages folder is deleted and a new folder > > > is created with a static page for each product. Any time the products > > > database is updated new static pages, or Spider bait, is created. I > > > did another version that is triggered manually for a site that has > > > 10's of thousands of products where rewriting the pages is too > > > intensive to have occur automatically. > > > > > > Paul > > > > > > _/_/_/_/_/_/_/_/_/_/_/_/|\_\_\_\_\_\_\_\_\_\_\_\_ > > > _/_/_/Paul Uttermohlen, Interactive Ink, Inc._\_\_ \_\_\_\_ > > > _/_/_/ mailto:paul@ims1.com - Website Development \_\_\_\_ > > > _/_/_/ Business -
_\_\_\_\_\_\_ > > > _/_/_/ Real Estate - _\_\_\_\_ > > > _/_/_/_/_/_/_/_/_/_/_/_/_/_/ | \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ > > > > > > ------------------------------------------------------------- > > > This message is sent to you because you are subscribed to > > > the mailing list . > > > To unsubscribe, E-mail to: > > > To switch to the DIGEST mode, E-mail to > > > Web Archive of this list is at: http://search.smithmicro.com/ > > > > ------------------------------------------------------------- > > This message is sent to you because you are subscribed to > > the mailing list . > > To unsubscribe, E-mail to: > > To switch to the DIGEST mode, E-mail to > > Web Archive of this list is at: http://search.smithmicro.com/ > > ------------------------------------------------------------- > This message is sent to you because you are subscribed to > the mailing list . > To unsubscribe, E-mail to: > To switch to the DIGEST mode, E-mail to > Web Archive of this list is at: http://search.smithmicro.com/------------------------------------------------------------- This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: To switch to the DIGEST mode, E-mail to Web Archive of this list is at: http://search.smithmicro.com/
Associated Messages, from the most recent to the oldest:
So, if I have 14000+ items, my index page will have an additional ~550k of text to deal with these stupid bots (figuring on 40 bytes/item)!!!That's it, I am going to write a WebCat-based spider catcher. I wonder if most spiders report that the referrer page was robots.txt when they hit the main index page? Anyone who wants to help (I'll need to know IP address and browser names for spiders, for example) can contact me offline. I hope to create an include file which could be easily linked to any page to redirect spiders to another section of the website.John Peacock jpeacock@univpress.com John Butler wrote: > > storebuilder makes a bunch of links at the bottom of a page which are transparent gifs > linked to those static pages... > (at least that is what we determined in a back & forth between me - who never looked at > storebuilder code, and Glenn who posted some of it) > -John > > John Peacock wrote: > > > And how are you directing the spiders/bots to the static pages, while > > your users are getting the active pages? This is always the problem, > > not creating the static pages. Whoever came up with the robots.txt file > > in the first place was a complete idiot! > > > > John Peacock > > > > Paul Uttermohlen wrote: > > > > > > > >The documentation for version 4 says it has the following new facility: > > > > > > > > > >> * Automatically index your web site for maximum search engine exposure. > > > > > > > > > >but I can't find any reference to this in the documentation. Where is it? > > > > > > > > > >Thanks. > > > > > > > > > > > > >Neither can I. > > > > > > > >If you use StoreBuilder, it will create files in a folder named > > > >staticpages. That's the stuff for search engines to find. You can get the > > > >general idea of what StoreBuilder does and use it for sites built by hand. > > > > > > > >Glenn > > > > > > > > > > if you don't want to use store builder and you still want to create > > > static pages from a products database, you can easily do this without > > > upgrading to 4.0. > > > > > > I just did this last week: > > > > > > if the products database modification date is newer than the static > > > pages folder then the static pages folder is deleted and a new folder > > > is created with a static page for each product. Any time the products > > > database is updated new static pages, or Spider bait, is created. I > > > did another version that is triggered manually for a site that has > > > 10's of thousands of products where rewriting the pages is too > > > intensive to have occur automatically. > > > > > > Paul > > > > > > _/_/_/_/_/_/_/_/_/_/_/_/|\_\_\_\_\_\_\_\_\_\_\_\_ > > > _/_/_/Paul Uttermohlen, Interactive Ink, Inc._\_\_ \_\_\_\_ > > > _/_/_/ mailto:paul@ims1.com - Website Development \_\_\_\_ > > > _/_/_/ Business - _\_\_\_\_\_\_ > > > _/_/_/ Real Estate - _\_\_\_\_ > > > _/_/_/_/_/_/_/_/_/_/_/_/_/_/ | \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ > > > > > > ------------------------------------------------------------- > > > This message is sent to you because you are subscribed to > > > the mailing list . > > > To unsubscribe, E-mail to: > > > To switch to the DIGEST mode, E-mail to > > > Web Archive of this list is at: http://search.smithmicro.com/ > > > > ------------------------------------------------------------- > > This message is sent to you because you are subscribed to > > the mailing list . > > To unsubscribe, E-mail to: > > To switch to the DIGEST mode, E-mail to > > Web Archive of this list is at: http://search.smithmicro.com/ > > ------------------------------------------------------------- > This message is sent to you because you are subscribed to > the mailing list . > To unsubscribe, E-mail to: > To switch to the DIGEST mode, E-mail to > Web Archive of this list is at: http://search.smithmicro.com/------------------------------------------------------------- This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: To switch to the DIGEST mode, E-mail to Web Archive of this list is at: http://search.smithmicro.com/
John Peacock
DOWNLOAD WEBDNA NOW!
Top Articles:
Talk List
The WebDNA community talk-list is the best place to get some help: several hundred extremely proficient programmers with an excellent knowledge of WebDNA and an excellent spirit will deliver all the tips and tricks you can imagine...
Related Readings:
Upgrade to web DNA6.0 (2004)
Why don't Typhoon & Firesite work together? (and webcat) (1998)
Weird happenings (1998)
RE: [isfolder] and [filename] (1997)
[WebDNA] 301 redirect on linux (2010)
Help name our technology! (1997)
includes and cart numbers (1997)
Imposible dream? (2000)
Separate SSL Server (1997)
Mac v. NT (1998)
WebMerchant 1.6 and https (1997)
AppleScript: Tell application:app location? (1998)
URGENT: WebCatalog Stalls (1998)
The Guru Speaks-Is That Really the Only Manuel? (1998)
[listfiles] (2004)
WebCommerce: Folder organization ? (1997)
OBDC Support (1997)
Shell traceroute (2001)
WebCat2b13MacPlugIn - [showif][search][/showif] (1997)
Emailer Set Up (1997)