Re: How do I get Google to crawl a WebCat site?
This WebDNA talk-list message is from 2003
It keeps the original formatting.
numero = 48650
interpreted = N
texte = Dennis J. Bonsall, Jr. wrote: > wrong. The url for the site is http://www.cxsonline.com. > Here are the headers your server is sending:HTTP/1.0 200 OK Connection: Close Content-Length: 12010 Expires: Thu Mar 13 17:42:01 2003 GMT Last-Modified: Thu Mar 13 17:42:01 2003 GMT Date: Thu Mar 13 17:42:01 2003 GMT MIME-Version: 1.0 Connection: Close Server: 4D_WebSTAR_S/5.2.4 (MacOS X) Set-Cookie: cxsonline=1047577321700; expires=Thursday, 31-Dec-2099 23:59:59 GMT; path=/; domain=www.cxsonline.com P3P: CP=NON COR CUR PSAa CONo OTPa OUR IND UNI COM NAV STA PRE OTC Content-type: text/htmlYou have a couple of bot-unfriendly features:1) you are setting a cookie, which is not fatal to all bots, but it will limit which ones will crawl your site; leave this off until the user looks like they are starting to get ready to order something.2) Your page is not cacheable, in the sense that Expires, Last-Modified, and Date are all the same value. Search the list archive for my most recent MIMEHeaders.inc files and set a reasonable Expires for all pages which are not the result of a search. If you database is slowchanging (i.e. over weeks or days but not minutes), you can even set your search pages to be cacheable (which we do).HTHJohn-- John Peacock Director of Information Research and Technology Rowman & Littlefield Publishing Group 4501 Forbes Boulevard Suite H Lanham, MD 20706 301-459-3366 x.5010 fax 301-429-5748 ------------------------------------------------------------- This message is sent to you because you are subscribed to the mailing list
. To unsubscribe, E-mail to: To switch to the DIGEST mode, E-mail to Web Archive of this list is at: http://webdna.smithmicro.com/
Associated Messages, from the most recent to the oldest:
Dennis J. Bonsall, Jr. wrote: > wrong. The url for the site is http://www.cxsonline.com. > Here are the headers your server is sending:HTTP/1.0 200 OK Connection: Close Content-Length: 12010 Expires: Thu Mar 13 17:42:01 2003 GMT Last-Modified: Thu Mar 13 17:42:01 2003 GMT Date: Thu Mar 13 17:42:01 2003 GMT MIME-Version: 1.0 Connection: Close Server: 4D_WebSTAR_S/5.2.4 (MacOS X) Set-Cookie: cxsonline=1047577321700; expires=Thursday, 31-Dec-2099 23:59:59 GMT; path=/; domain=www.cxsonline.com P3P: CP=NON COR CUR PSAa CONo OTPa OUR IND UNI COM NAV STA PRE OTC Content-type: text/htmlYou have a couple of bot-unfriendly features:1) you are setting a cookie, which is not fatal to all bots, but it will limit which ones will crawl your site; leave this off until the user looks like they are starting to get ready to order something.2) Your page is not cacheable, in the sense that Expires, Last-Modified, and Date are all the same value. Search the list archive for my most recent MIMEHeaders.inc files and set a reasonable Expires for all pages which are not the result of a search. If you database is slowchanging (i.e. over weeks or days but not minutes), you can even set your search pages to be cacheable (which we do).HTHJohn-- John Peacock Director of Information Research and Technology Rowman & Littlefield Publishing Group 4501 Forbes Boulevard Suite H Lanham, MD 20706 301-459-3366 x.5010 fax 301-429-5748 ------------------------------------------------------------- This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: To switch to the DIGEST mode, E-mail to Web Archive of this list is at: http://webdna.smithmicro.com/
John Peacock
DOWNLOAD WEBDNA NOW!
Top Articles:
Talk List
The WebDNA community talk-list is the best place to get some help: several hundred extremely proficient programmers with an excellent knowledge of WebDNA and an excellent spirit will deliver all the tips and tricks you can imagine...
Related Readings:
Re2: AAgghh!! Help, please. SSL strikes again. (1997)
If Empty ? (1997)
Multiple prices (1997)
Not really WebCat (1997)
flushdatabases (2000)
[isfile] ? (1997)
Mem per threads (1998)
Are you an WC expert? (1997)
WC2f3 (1997)
Where's Cart Created ? (1997)
Moving encrypted fields to different boxes/platforms (2000)
Mime Headers for Mail (1998)
Trouble with my database (1998)
NT challenges default admin/admin password (2000)
Solution to truncated email problem! (1998)
Grep support request for SMSI ... (2003)
WebCat editing, SiteGuard WAS:SiteAssociative lookup style? (1997)
Shipping charge question (2003)
Help! WebCat2 bug (1997)
Need help with form (1998)