More Search Engine

This WebDNA talk-list message is from

2002


It keeps the original formatting.
numero = 44948
interpreted = N
texte = Since the search engine bots don't like to follow the query strings links, I may have a workaround that will let them spider pages anyhow.GoogleBot the FAST crawler, and others are good at obeying robots.txt instructions, so instead of using that to keep bots away from pages, maybe it can be used to attract them to others.I rewrote my robots.txt file to include a lot of Allow: abc.html along with my old Disallow: xyz.html instructions. In fact, I used writefile to create a new robobts.txt file so it can be updated as required.Maybe this will work on those templates that don't depend on a query string to tell them what to display. I'll have to wait for another crawl to find out.Glenn------------------------------------------------------------- This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: To switch to the DIGEST mode, E-mail to Web Archive of this list is at: http://search.smithmicro.com/ Associated Messages, from the most recent to the oldest:

    
  1. More Search Engine (Glenn Busbin 2002)
Since the search engine bots don't like to follow the query strings links, I may have a workaround that will let them spider pages anyhow.GoogleBot the FAST crawler, and others are good at obeying robots.txt instructions, so instead of using that to keep bots away from pages, maybe it can be used to attract them to others.I rewrote my robots.txt file to include a lot of Allow: abc.html along with my old Disallow: xyz.html instructions. In fact, I used writefile to create a new robobts.txt file so it can be updated as required.Maybe this will work on those templates that don't depend on a query string to tell them what to display. I'll have to wait for another crawl to find out.Glenn------------------------------------------------------------- This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: To switch to the DIGEST mode, E-mail to Web Archive of this list is at: http://search.smithmicro.com/ Glenn Busbin

DOWNLOAD WEBDNA NOW!

Top Articles:

Talk List

The WebDNA community talk-list is the best place to get some help: several hundred extremely proficient programmers with an excellent knowledge of WebDNA and an excellent spirit will deliver all the tips and tricks you can imagine...

Related Readings:

HELP WITH DATES (1997) Separate SSL Server (1997) Textarea Breaks (2000) Thanks Grant (1997) using showpage and showcart commands (1996) Site Search Concepts (2003) [OT] Bad Hard Drive? (2003) webcat NT (1998) AutoCommit Preference? (1998) PIXO Support (1997) search results not sorted with 'cl' (1998) RE: [WebDNA] Use of MySQL with WebDNA (2010) Separate SSL Server (1997) blank page from template (1997) WebCat 5 Intranet Question (2003) problems with 2 tags (1997) I forgot (1998) How to put the Trademark Symbol (%99) in Sendmail? (2003) Search in 2 or more catalogs (1997) Part Html part WebDNA (1997)