Re: Search Engine bots

This WebDNA talk-list message is from

2002


It keeps the original formatting.
numero = 44758
interpreted = N
texte = >Hi, I need this clarified once and for all. >I have a domain with a default page at the root >called default.tpl. > >Will a search engine robot (specifically googlebot) >see this page?Probably, and it may also follow some dynamically generated links on that page, but it may also refuse to follow some of them. I'd use Default.html just to be a bit on the safe side. Google has info for webmasters about this. See their site map. It doesn't mention .tpl suffixes, but that's no surprise.AllTheWeb won't follow dynamically generated links, so I make them static. I don't know about the other spiders.Your robots.txt file is ok, but there are bad bots who not only ignore the Disallow stuff, they use it to crawl pages they wouldn't otherwise know about.Glenn------------------------------------------------------------- This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: To switch to the DIGEST mode, E-mail to Web Archive of this list is at: http://search.smithmicro.com/ Associated Messages, from the most recent to the oldest:

    
  1. Re: Browser compatibility (Was: Re: Search Engine bots) (Donovan Brooke 2002)
  2. Re: Browser compatibility (Was: Re: Search Engine bots) (Frank Nordberg 2002)
  3. Re: Browser compatibility (Was: Re: Search Engine bots) (Dan Strong 2002)
  4. Re: Browser compatibility (Was: Re: Search Engine bots) (Donovan Brooke 2002)
  5. Re: Browser compatibility (Was: Re: Search Engine bots) (Alain Russell 2002)
  6. Browser compatibility (Was: Re: Search Engine bots) (Frank Nordberg 2002)
  7. Re: Search Engine bots (Alain Russell 2002)
  8. Re: Search Engine bots (Dan Strong 2002)
  9. Re: Search Engine bots (Donovan Brooke 2002)
  10. Re: Search Engine bots (Andrew Simpson 2002)
  11. Re: Search Engine bots (Alain Russell 2002)
  12. Re: Search Engine bots (Andrew Simpson 2002)
  13. Re: Search Engine bots (Dan Strong 2002)
  14. Re: Search Engine bots (Andrew Simpson 2002)
  15. Re: Search Engine bots (Dan Strong 2002)
  16. Re: Search Engine bots (Donovan Brooke 2002)
  17. Re: Search Engine bots (Andrew Simpson 2002)
  18. Re: Search Engine bots (Donovan Brooke 2002)
  19. Re: Search Engine bots (Alain Russell 2002)
  20. Re: Search Engine bots (Glenn Busbin 2002)
  21. Re: Search Engine bots (Andrew Simpson 2002)
  22. Re: Search Engine bots (Donovan Brooke 2002)
  23. Re: Search Engine bots (Alain Russell 2002)
  24. Re: Search Engine bots (Donovan Brooke 2002)
  25. Re: Search Engine bots (Alain Russell 2002)
  26. Re: Search Engine bots (Andrew Simpson 2002)
  27. Search Engine bots (Donovan Brooke 2002)
>Hi, I need this clarified once and for all. >I have a domain with a default page at the root >called default.tpl. > >Will a search engine robot (specifically googlebot) >see this page?Probably, and it may also follow some dynamically generated links on that page, but it may also refuse to follow some of them. I'd use Default.html just to be a bit on the safe side. Google has info for webmasters about this. See their site map. It doesn't mention .tpl suffixes, but that's no surprise.AllTheWeb won't follow dynamically generated links, so I make them static. I don't know about the other spiders.Your robots.txt file is ok, but there are bad bots who not only ignore the Disallow stuff, they use it to crawl pages they wouldn't otherwise know about.Glenn------------------------------------------------------------- This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: To switch to the DIGEST mode, E-mail to Web Archive of this list is at: http://search.smithmicro.com/ Glenn Busbin

DOWNLOAD WEBDNA NOW!

Top Articles:

Talk List

The WebDNA community talk-list is the best place to get some help: several hundred extremely proficient programmers with an excellent knowledge of WebDNA and an excellent spirit will deliver all the tips and tricks you can imagine...

Related Readings:

Problem with textA (2000) Grep help needed (2004) Nested tags count question (1997) Templates on Unix & CGI on Mac? (1997) (1997) Re[2]: Hiding files/images. (2000) 2.0 Info (1997) Showif, Hideif reverse logic ? (1997) form data submission gets truncated (1997) [WebDNA] Connect to MS SQL (2013) problems with 2 tags shakur (1997) sendmail Q (2001) Quick Question About Sorting (1997) Shownext never shows next...still (1997) NetCloak as Emailer substitute? (1999) Replacing a Word (1999) [WebDNA] total (2010) [Semi-OT] Eaaaaaasy OSX FTP (2003) Math Function (1997) Associative lookup style? + bit more (1997)