Re: Search Engine bots
This WebDNA talk-list message is from 2002
It keeps the original formatting.
numero = 44758
interpreted = N
texte = >Hi, I need this clarified once and for all.>I have a domain with a default page at the root>called default.tpl.>>Will a search engine robot (specifically googlebot)>see this page?Probably, and it may also follow some dynamically generated links on that page, but it may also refuse to follow some of them. I'd use Default.html just to be a bit on the safe side. Google has info for webmasters about this. See their site map. It doesn't mention .tpl suffixes, but that's no surprise.AllTheWeb won't follow dynamically generated links, so I make them static. I don't know about the other spiders.Your robots.txt file is ok, but there are bad bots who not only ignore the Disallow stuff, they use it to crawl pages they wouldn't otherwise know about.Glenn-------------------------------------------------------------This message is sent to you because you are subscribed to the mailing list
.To unsubscribe, E-mail to: To switch to the DIGEST mode, E-mail to Web Archive of this list is at: http://search.smithmicro.com/
Associated Messages, from the most recent to the oldest:
>Hi, I need this clarified once and for all.>I have a domain with a default page at the root>called default.tpl.>>Will a search engine robot (specifically googlebot)>see this page?Probably, and it may also follow some dynamically generated links on that page, but it may also refuse to follow some of them. I'd use Default.html just to be a bit on the safe side. Google has info for webmasters about this. See their site map. It doesn't mention .tpl suffixes, but that's no surprise.AllTheWeb won't follow dynamically generated links, so I make them static. I don't know about the other spiders.Your robots.txt file is ok, but there are bad bots who not only ignore the Disallow stuff, they use it to crawl pages they wouldn't otherwise know about.Glenn-------------------------------------------------------------This message is sent to you because you are subscribed to the mailing list .To unsubscribe, E-mail to: To switch to the DIGEST mode, E-mail to Web Archive of this list is at: http://search.smithmicro.com/
Glenn Busbin
DOWNLOAD WEBDNA NOW!
Top Articles:
Talk List
The WebDNA community talk-list is the best place to get some help: several hundred extremely proficient programmers with an excellent knowledge of WebDNA and an excellent spirit will deliver all the tips and tricks you can imagine...
Related Readings:
Problem with textA (2000)
Grep help needed (2004)
Nested tags count question (1997)
Templates on Unix & CGI on Mac? (1997)
(1997)
Re[2]: Hiding files/images. (2000)
2.0 Info (1997)
Showif, Hideif reverse logic ? (1997)
form data submission gets truncated (1997)
[WebDNA] Connect to MS SQL (2013)
problems with 2 tags shakur (1997)
sendmail Q (2001)
Quick Question About Sorting (1997)
Shownext never shows next...still (1997)
NetCloak as Emailer substitute? (1999)
Replacing a Word (1999)
[WebDNA] total (2010)
[Semi-OT] Eaaaaaasy OSX FTP (2003)
Math Function (1997)
Associative lookup style? + bit more (1997)