Re: Spiders

This WebDNA talk-list message is from

1998


It keeps the original formatting.
numero = 17010
interpreted = N
texte = Kevin,Examples, as well as other information about the robots.txt file and the Standard for Robot Exclusion can be found at:WendellKevin Mueller wrote:>This may seem like a rookie question, but here goes. > >I have noticed that when my site is accessed by a spider it first looks >for a file called SPIDER.TXT. >What is (or should be) contained in this file? >Is the presence of this file a solution the problems indexing sites with >dynamic URLs like those in a WebCat site? >I know this is on the fringes of WebCat but if someone can pull me out >of the dark on this one... : ) > >Thanks, >Kevin Mueller >iFolio Digital Image Library Associated Messages, from the most recent to the oldest:

    
  1. Re: Search Spiders (Glenn Busbin 2002)
  2. Re: Search Spiders (Glenn Busbin 2002)
  3. Re: Search Spiders (Glenn Busbin 2002)
  4. Search Spiders (Alain Russell 2002)
  5. Re: Spiders and Bots (Paul Uttermohlen 2000)
  6. Re: OT - Re: Spiders and Bots (Glenn Busbin 2000)
  7. OT - Re: Spiders and Bots (John Butler 2000)
  8. Re: Spiders and Bots (John Butler 2000)
  9. Re: Spiders and Bots (Glenn Busbin 2000)
  10. Re: Spiders and Bots (Glenn Busbin 2000)
  11. Re: Spiders and Bots (John Butler 2000)
  12. Re: Spiders and Bots (Jesse Proudman 2000)
  13. Re: Spiders and Bots (Bob Sneidar 2000)
  14. Re: Spiders and Bots (John Butler 2000)
  15. Re: Spiders and Bots (Aaron Lynch 2000)
  16. Re: Spiders and Bots (Glenn Busbin 2000)
  17. Re: Spiders and Bots (John Butler 2000)
  18. Spiders and Bots (Glenn Busbin 2000)
  19. Re: Spiders (Glenn Busbin 2000)
  20. Re: Spiders (Rob Marquardt 2000)
  21. Spiders (Glenn Busbin 2000)
  22. Re: Spiders, robots, whatever you want to call them (Christer Olsson 1998)
  23. Spiders, robots, whatever you want to call them (Sue Ann Sanders 1998)
  24. Re: Spiders (Wendell B. Kozak 1998)
  25. Re: Spiders (Bob Minor 1998)
  26. Re: Spiders (Kevin Mueller 1998)
  27. Spiders (Kevin Mueller 1998)
Kevin,Examples, as well as other information about the robots.txt file and the Standard for Robot Exclusion can be found at:WendellKevin Mueller wrote:>This may seem like a rookie question, but here goes. > >I have noticed that when my site is accessed by a spider it first looks >for a file called SPIDER.TXT. >What is (or should be) contained in this file? >Is the presence of this file a solution the problems indexing sites with >dynamic URLs like those in a WebCat site? >I know this is on the fringes of WebCat but if someone can pull me out >of the dark on this one... : ) > >Thanks, >Kevin Mueller >iFolio Digital Image Library Wendell B. Kozak

DOWNLOAD WEBDNA NOW!

Top Articles:

Talk List

The WebDNA community talk-list is the best place to get some help: several hundred extremely proficient programmers with an excellent knowledge of WebDNA and an excellent spirit will deliver all the tips and tricks you can imagine...

Related Readings:

Organisation website (2005) Occasional crashes with $remove (1997) Forbidden CGI Error (1997) HELP WITH DATES (1997) Extended [ConvertChars] (1997) Newbie from and old bee (1997) [SearchString] problem with [search] context (1997) dates and hex formatting (1997) WebCat2b13MacPlugIn - [showif][search][/showif] (1997) Forms Search Questions (1997) Web Catalog Book? (1998) SSL and reg web* (1997) WebCatalog Mac 2.1b3 (1997) IE problems persist (2004) Using Cookie for client specific info? (1997) [WebDNA] v7 Stress Test and Strange Behaviour (2012) test (2005) db updating? (2000) U&P IIS concept (1998) WebDNA Solutions ... sorry! (1997)