Re: Search Engine questions ...

This WebDNA talk-list message is from

2002


It keeps the original formatting.
numero = 44843
interpreted = N
texte = >At 10:50 PM 11/4/02, Glenn Busbin wrote: >>In my case, I recently put up a site that uses WC and, after >>keeping it blocked for a while with the robots.txt file, I opened >>it up for spidering and asked Google and AllTheWeb to take a look. >>Some of the content on the pages is static. Some is dynamic. >>AllTheWeb crawled the site and followed 100% of the links that were >>written manually (they do contain ?cart=[cart] and maybe some other >>variable). It did not follow any link that was written dynamically. >>Searching AllTheWeb for words and phrases that were displayed >>dynamically produced zero returns, even when I used the URL in the >>search. > > >I don't know what your code looks like or how it determines what >links to write dynamically, but is there any condition that would >cause no dynamic links to be generated, based, perhaps, on the >parameters passed to the page? Every link created by the webdna engine is a dynamically-generated link. In fact, every character in every HTML file which is generated by the webdna engine is a dynamically-generated character. This means every webdna page is 100% dynamic because every single character is written on-the-fly instead of simply being read from disk and served immediately.And the terms dynamic or static have nothing to do with links. Dynamic and static are terms used to define entire web pages, not parts of pages and definitely not hyperlinks. >There is absolutely no difference between >written by hand and written by WebDNA. As far as >the browser (spider) is concerned, it's just text.Damn right! And even if the links look like or there is still no difference when they are written by hand or written by webdna. They are all still JUST PLAIN TEXT to the spider ...... and whether or not the spider actually follows a link does not depend at all on how the link was created, it ONLY depends upon the character of the link itself!If a spider is programmed NOT to follow links with question marks in them (for example) then it simply will not follow that link. And if that link leads to a page where someone has placed his webdna code in an attempt to record the spider's behavior, the spider's behavior will not be recorded for that page -- because the spider never visited that page. Sincerely, Kenneth Grome--------------------------------------------------- WebDNA Professional Training and Development Center 175 J. Llorente Street +63 (32) 255-6921 Cebu City, Cebu 6000 kengrome@webdna.net Philippines http://www.webdna.net ---------------------------------------------------------------------------------------------------------------- This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: To switch to the DIGEST mode, E-mail to Web Archive of this list is at: http://search.smithmicro.com/ Associated Messages, from the most recent to the oldest:

    
  1. Re: Search Engine questions ... (Pedro Rivera 2002)
  2. Re: Search Engine questions ... (Wendell Kozak 2002)
  3. Re: Search Engine questions ... (dale's stuff 2002)
  4. Re: Search Engine questions ... (Gary Krockover 2002)
  5. Re: Search Engine questions ... (Gary Krockover 2002)
  6. Re: Search Engine questions ... (Kenneth Grome 2002)
  7. Re: Search Engine questions ... (Kenneth Grome 2002)
  8. Re: Search Engine questions ... (Kenneth Grome 2002)
  9. Re: Search Engine questions ... (Glenn Busbin 2002)
  10. Re: Search Engine questions ... (Kenneth Grome 2002)
  11. Re: Search Engine questions ... (Glenn Busbin 2002)
  12. Re: Search Engine questions ... (Kenneth Grome 2002)
  13. Re: Search Engine questions ... (Brian Fries 2002)
  14. Re: Search Engine questions ... (Glenn Busbin 2002)
  15. Re: Search Engine questions ... (Andrew Simpson 2002)
  16. Re: Search Engine questions ... (Glenn Busbin 2002)
  17. Re: Search Engine questions ... (Kenneth Grome 2002)
  18. Re: Search Engine questions ... (Glenn Busbin 2002)
  19. Re: Search Engine questions ... (Kenneth Grome 2002)
  20. Re: Search Engine questions ... (Glenn Busbin 2002)
  21. Re: Search Engine questions ... (dale's stuff 2002)
  22. Re: Search Engine questions ... (Glenn Busbin 2002)
  23. Re: Search Engine questions ... (John Peacock 2002)
  24. Re: Search Engine questions ... (Glenn Busbin 2002)
  25. Re: Search Engine questions ... (Alain Russell 2002)
  26. Re: Search Engine questions ... (Glenn Busbin 2002)
  27. Re: Search Engine questions ... (Dan Strong 2002)
  28. Re: Search Engine questions ... (Oleg Kremiansky 2002)
  29. Re: Search Engine questions ... (Glenn Busbin 2002)
  30. Re: Search Engine questions ... (Alain Russell 2002)
  31. Re: Search Engine questions ... (Kenneth Grome 2002)
  32. Re: Search Engine questions ... (Glenn Busbin 2002)
  33. Re: Search Engine questions ... (Glenn Busbin 2002)
  34. Re: Search Engine questions ... (Glenn Busbin 2002)
  35. Re: Search Engine questions ... (Glenn Busbin 2002)
  36. Re: Search Engine questions ... (Clayton Randall 2002)
  37. Re: Search Engine questions ... (Kenneth Grome 2002)
  38. Re: Search Engine questions ... (Glenn Busbin 2002)
  39. Re: Search Engine questions ... (Donovan Brooke 2002)
  40. Re: Search Engine questions ... (Alain Russell 2002)
  41. Re: Search Engine questions ... (Donovan Brooke 2002)
  42. Re: Search Engine questions ... (Alain Russell 2002)
  43. Search Engine questions ... (Kenneth Grome 2002)
>At 10:50 PM 11/4/02, Glenn Busbin wrote: >>In my case, I recently put up a site that uses WC and, after >>keeping it blocked for a while with the robots.txt file, I opened >>it up for spidering and asked Google and AllTheWeb to take a look. >>Some of the content on the pages is static. Some is dynamic. >>AllTheWeb crawled the site and followed 100% of the links that were >>written manually (they do contain ?cart=[cart] and maybe some other >>variable). It did not follow any link that was written dynamically. >>Searching AllTheWeb for words and phrases that were displayed >>dynamically produced zero returns, even when I used the URL in the >>search. > > >I don't know what your code looks like or how it determines what >links to write dynamically, but is there any condition that would >cause no dynamic links to be generated, based, perhaps, on the >parameters passed to the page? Every link created by the webdna engine is a dynamically-generated link. In fact, every character in every HTML file which is generated by the webdna engine is a dynamically-generated character. This means every webdna page is 100% dynamic because every single character is written on-the-fly instead of simply being read from disk and served immediately.And the terms dynamic or static have nothing to do with links. Dynamic and static are terms used to define entire web pages, not parts of pages and definitely not hyperlinks. >There is absolutely no difference between >written by hand and written by WebDNA. As far as >the browser (spider) is concerned, it's just text.Damn right! And even if the links look like [cart]> or there is still no difference when they are written by hand or written by webdna. They are all still JUST PLAIN TEXT to the spider ...... and whether or not the spider actually follows a link does not depend at all on how the link was created, it ONLY depends upon the character of the link itself!If a spider is programmed NOT to follow links with question marks in them (for example) then it simply will not follow that link. And if that link leads to a page where someone has placed his webdna code in an attempt to record the spider's behavior, the spider's behavior will not be recorded for that page -- because the spider never visited that page. Sincerely, Kenneth Grome--------------------------------------------------- WebDNA Professional Training and Development Center 175 J. Llorente Street +63 (32) 255-6921 Cebu City, Cebu 6000 kengrome@webdna.net Philippines http://www.webdna.net ---------------------------------------------------------------------------------------------------------------- This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: To switch to the DIGEST mode, E-mail to Web Archive of this list is at: http://search.smithmicro.com/ Kenneth Grome

DOWNLOAD WEBDNA NOW!

Top Articles:

Talk List

The WebDNA community talk-list is the best place to get some help: several hundred extremely proficient programmers with an excellent knowledge of WebDNA and an excellent spirit will deliver all the tips and tricks you can imagine...

Related Readings:

Problem with version 4 browsers (1998) [WriteFile] problems (1997) [WriteFile] problems (1997) WCf2 and nested tags (1997) WebCat 4.0 & WebMerchant (2000) OK, here goes... (1997) Fwd: About WebCatalog and iTools 7 (2003) Redirect from Domain1 to Domain2 (2004) Unexpected comparison behavior change in 4.5.1 (2003) [delete] problem (1997) Country & Ship-to address & other fields ? (1997) Pager notification of completed orders (1998) PCS Frames (1997) WebCat2b15MacPlugin - showing [math] (1997) WebCat2: Items xx to xx shown, etc. (1997) using showpage and showcart commands (1996) Robots fill event log (1997) StoreBuilder/SiteBuilder (2005) Upcoming 2.1 Release and PCS Committment (1997) cookies and [returnraw] (1997)