Re: How do I get Google to crawl a WebCat site?
This WebDNA talk-list message is from 2003
It keeps the original formatting.
numero = 48650
interpreted = N
texte = Dennis J. Bonsall, Jr. wrote:> wrong. The url for the site is http://www.cxsonline.com.> Here are the headers your server is sending:HTTP/1.0 200 OKConnection: CloseContent-Length: 12010Expires: Thu Mar 13 17:42:01 2003 GMTLast-Modified: Thu Mar 13 17:42:01 2003 GMTDate: Thu Mar 13 17:42:01 2003 GMTMIME-Version: 1.0Connection: CloseServer: 4D_WebSTAR_S/5.2.4 (MacOS X)Set-Cookie: cxsonline=1047577321700; expires=Thursday, 31-Dec-2099 23:59:59 GMT; path=/; domain=www.cxsonline.comP3P: CP=NON COR CUR PSAa CONo OTPa OUR IND UNI COM NAV STA PRE OTCContent-type: text/htmlYou have a couple of bot-unfriendly features:1) you are setting a cookie, which is not fatal to all bots, but it will limit which ones will crawl your site; leave this off until the user looks like they are starting to get ready to order something.2) Your page is not cacheable, in the sense that Expires, Last-Modified, and Date are all the same value. Search the list archive for my most recent MIMEHeaders.inc files and set a reasonable Expires for all pages which are not the result of a search. If you database is slowchanging (i.e. over weeks or days but not minutes), you can even set your search pages to be cacheable (which we do).HTHJohn-- John PeacockDirector of Information Research and TechnologyRowman & Littlefield Publishing Group4501 Forbes BoulevardSuite HLanham, MD 20706301-459-3366 x.5010fax 301-429-5748-------------------------------------------------------------This message is sent to you because you are subscribed to the mailing list
.To unsubscribe, E-mail to: To switch to the DIGEST mode, E-mail to Web Archive of this list is at: http://webdna.smithmicro.com/
Associated Messages, from the most recent to the oldest:
Dennis J. Bonsall, Jr. wrote:> wrong. The url for the site is http://www.cxsonline.com.> Here are the headers your server is sending:HTTP/1.0 200 OKConnection: CloseContent-Length: 12010Expires: Thu Mar 13 17:42:01 2003 GMTLast-Modified: Thu Mar 13 17:42:01 2003 GMTDate: Thu Mar 13 17:42:01 2003 GMTMIME-Version: 1.0Connection: CloseServer: 4D_WebSTAR_S/5.2.4 (MacOS X)Set-Cookie: cxsonline=1047577321700; expires=Thursday, 31-Dec-2099 23:59:59 GMT; path=/; domain=www.cxsonline.comP3P: CP=NON COR CUR PSAa CONo OTPa OUR IND UNI COM NAV STA PRE OTCContent-type: text/htmlYou have a couple of bot-unfriendly features:1) you are setting a cookie, which is not fatal to all bots, but it will limit which ones will crawl your site; leave this off until the user looks like they are starting to get ready to order something.2) Your page is not cacheable, in the sense that Expires, Last-Modified, and Date are all the same value. Search the list archive for my most recent MIMEHeaders.inc files and set a reasonable Expires for all pages which are not the result of a search. If you database is slowchanging (i.e. over weeks or days but not minutes), you can even set your search pages to be cacheable (which we do).HTHJohn-- John PeacockDirector of Information Research and TechnologyRowman & Littlefield Publishing Group4501 Forbes BoulevardSuite HLanham, MD 20706301-459-3366 x.5010fax 301-429-5748-------------------------------------------------------------This message is sent to you because you are subscribed to the mailing list .To unsubscribe, E-mail to: To switch to the DIGEST mode, E-mail to Web Archive of this list is at: http://webdna.smithmicro.com/
John Peacock
DOWNLOAD WEBDNA NOW!
Top Articles:
Talk List
The WebDNA community talk-list is the best place to get some help: several hundred extremely proficient programmers with an excellent knowledge of WebDNA and an excellent spirit will deliver all the tips and tricks you can imagine...
Related Readings:
[OT] Ave. Hits Per Day (2006)
changing banners on a page without refresh? (2000)
Multiple Users.db Possible? (1997)
foriegn characters and webcatalog (1997)
PhotoMill -> PhotoMaster (1997)
Nesting format tags (1997)
MasterCounter and capitalization (1997)
[AppendFile] problem (WebCat2b13 Mac .acgi) (1997)
Template Not Found (1998)
HomePage Caution (1997)
WebTen and WebCat (1997)
Strange Crash (1998)
Configuration Q (1998)
WebCatalog Mac and cgi-bin (WebSTAR 2.0) (1997)
Error Log.db --however (1997)
emailer w/F2 (1997)
Checkbox question (1997)
[delete] problem (1997)
WebCat2b15MacPlugin - showing [math] (1997)
[WebDNA] Best practice re: password storage (2013)