Re: How do I get Google to crawl a WebCat site?
This WebDNA talk-list message is from 2003
It keeps the original formatting.
numero = 48648
interpreted = N
texte = Dennis J. Bonsall, Jr. wrote:>I've developed numerous WebCat sites over the years, but have always had >dismal results with search engines. While I have not had problems >getting the home pages listed, I have never been able to get the >remainder of the site crawled. I figured that part of the problem is >related to the fact that all the links contain a '?' with at least >'cart=[cart]' after that point. However, on my most recent project, I >have created a link on the home page that takes the search engine >directly to the catalog without a question mark in the URL at all. But, >Googlebot has visited the site three times over the past few months. >According to my logs, it has requested robots.txt (as expected), then >the home page. After that, it simply leaves and goes no further. There >is no robots.txt file at all, so there is no restrictions on the search >engines. I was hoping that someone on this list who knows more about >search engines than I do might be able to tell me what I am doing >wrong. The url for the site is http://www.cxsonline.com. > The most important things to remember is no tricks. do not include any links with invisible gifs or text links that are the same color as the back ground.IME, google will follow any link from the home page including ? linksMy best results come from a static link to a site mapyou might want to put a basic robots.txt file in your root of the domain.# # robots.txt for http://www.cxsonline.com/ # # $Id: robots.txt,v 1.22 2002/04/18 20:23:04 ted Exp $ ## For use by all robots User-agent: *Don't use the keywords meta tagDo use the Description meta tagCode in all the alt tags for graphics.HTH, Donovan> >Thanks in advance for the assistance, > >Dennis > > -- -Donovan Brooke -Administrator of IT / -Assc. Art Director -Creative Services -Epsen Hillmer Graphics-402.829.0123 ------------------------------------------------------------- This message is sent to you because you are subscribed to the mailing list
. To unsubscribe, E-mail to: To switch to the DIGEST mode, E-mail to Web Archive of this list is at: http://webdna.smithmicro.com/
Associated Messages, from the most recent to the oldest:
Dennis J. Bonsall, Jr. wrote:>I've developed numerous WebCat sites over the years, but have always had >dismal results with search engines. While I have not had problems >getting the home pages listed, I have never been able to get the >remainder of the site crawled. I figured that part of the problem is >related to the fact that all the links contain a '?' with at least >'cart=[cart]' after that point. However, on my most recent project, I >have created a link on the home page that takes the search engine >directly to the catalog without a question mark in the URL at all. But, >Googlebot has visited the site three times over the past few months. >According to my logs, it has requested robots.txt (as expected), then >the home page. After that, it simply leaves and goes no further. There >is no robots.txt file at all, so there is no restrictions on the search >engines. I was hoping that someone on this list who knows more about >search engines than I do might be able to tell me what I am doing >wrong. The url for the site is http://www.cxsonline.com. > The most important things to remember is no tricks. do not include any links with invisible gifs or text links that are the same color as the back ground.IME, google will follow any link from the home page including ? linksMy best results come from a static link to a site mapyou might want to put a basic robots.txt file in your root of the domain.# # robots.txt for http://www.cxsonline.com/ # # $Id: robots.txt,v 1.22 2002/04/18 20:23:04 ted Exp $ ## For use by all robots User-agent: *Don't use the keywords meta tagDo use the Description meta tagCode in all the alt tags for graphics.HTH, Donovan> >Thanks in advance for the assistance, > >Dennis > > -- -Donovan Brooke -Administrator of IT / -Assc. Art Director -Creative Services -Epsen Hillmer Graphics-402.829.0123 ------------------------------------------------------------- This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: To switch to the DIGEST mode, E-mail to Web Archive of this list is at: http://webdna.smithmicro.com/
Donovan
DOWNLOAD WEBDNA NOW!
Top Articles:
Talk List
The WebDNA community talk-list is the best place to get some help: several hundred extremely proficient programmers with an excellent knowledge of WebDNA and an excellent spirit will deliver all the tips and tricks you can imagine...
Related Readings:
watch out for format_to_days on NT (1997)
READFILE command? (1998)
New Site Announcement (1998)
Search returns all, not 20 (1997)
Change to 5.0 per website licensing (2003)
WebCat2 several catalogs? (1997)
suffix mapping, use of cache, etc. (1997)
Upgrading old WebCat Database Files (1997)
Roundup function? (1997)
AGAIN: Who's Security? (2000)
2 easy questions re: [showif] and [sendmail] (1997)
Setting up shop (1997)
2.0Beta Command Ref (can't find this instruction) (1997)
No. of items in shopping cart (2000)
Quit revisited (1997)
multi-paragraph fields (1997)
Mac v. NT (1998)
Emailer pref's won't save (2005)
What am I missing (1997)
Running _every_ page through WebCat ? (1997)