Re: How do I get Google to crawl a WebCat site?
This WebDNA talk-list message is from 2003
It keeps the original formatting.
numero = 48652
interpreted = N
texte = > > >Don't use the keywords meta tagYou can use keywords META tags all you want. Some spiders ignore them, some use them, none penalize you for their use.Use
in the page headers. Modify as required.> >Do use the Description meta tag > >Code in all the alt tags for graphics.These may help ranking, but do nothing for getting a site spidered.See Google's webmaster tips for getting spidered by Googlebot.Googlebot reads the robots.txt file before spidering and looks to see if a home page exists before ever trying to spider a site. It also obeys the Allow: command in the robots.txt file, even though it's not in the robots.txt RFC. Use it anyway.Some bad bots will use the templates in the Disallow: command to spider pages you want left alone. Don't use that command and do not link to such templates from those that can be spidered unless you have good security for those templates (U/N and P/W's, for example).I doubt if Google or any other bot knows what a .tmpl or .tpl suffixes are. SM should, but prolly never has, tried to educated the bot owners about this. Use .htm or .html instead.Query strings can be read by some bots, but not all. In any event, URL's with query strings do not rank well compared to those without them.Some bots can accept a cookie now, but don't use it. It's just a way of spidering without being restricted to only those pages which require one. They still hit links they find, but without following them from one page to the next. Hence, no referrer for those hits shows in the logs.Glenn------------------------------------------------------------- This message is sent to you because you are subscribed to the mailing list
. To unsubscribe, E-mail to: To switch to the DIGEST mode, E-mail to Web Archive of this list is at: http://webdna.smithmicro.com/
Associated Messages, from the most recent to the oldest:
> > >Don't use the keywords meta tagYou can use keywords META tags all you want. Some spiders ignore them, some use them, none penalize you for their use.Use in the page headers. Modify as required.> >Do use the Description meta tag > >Code in all the alt tags for graphics.These may help ranking, but do nothing for getting a site spidered.See Google's webmaster tips for getting spidered by Googlebot.Googlebot reads the robots.txt file before spidering and looks to see if a home page exists before ever trying to spider a site. It also obeys the Allow: command in the robots.txt file, even though it's not in the robots.txt RFC. Use it anyway.Some bad bots will use the templates in the Disallow: command to spider pages you want left alone. Don't use that command and do not link to such templates from those that can be spidered unless you have good security for those templates (U/N and P/W's, for example).I doubt if Google or any other bot knows what a .tmpl or .tpl suffixes are. SM should, but prolly never has, tried to educated the bot owners about this. Use .htm or .html instead.Query strings can be read by some bots, but not all. In any event, URL's with query strings do not rank well compared to those without them.Some bots can accept a cookie now, but don't use it. It's just a way of spidering without being restricted to only those pages which require one. They still hit links they find, but without following them from one page to the next. Hence, no referrer for those hits shows in the logs.Glenn------------------------------------------------------------- This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: To switch to the DIGEST mode, E-mail to Web Archive of this list is at: http://webdna.smithmicro.com/
Glenn Busbin
DOWNLOAD WEBDNA NOW!
Top Articles:
Talk List
The WebDNA community talk-list is the best place to get some help: several hundred extremely proficient programmers with an excellent knowledge of WebDNA and an excellent spirit will deliver all the tips and tricks you can imagine...
Related Readings:
Configuring E-mail (1997)
Scrape Interest Rates (2005)
Credit card arrangement (2005)
WebCat2b13MacPlugin - [math][date][/math] problem (1997)
Images in the page without using separate image files ... (2003)
Time to opensource? was [SMSI] WebDNA is too good to (2006)
authnet good news (2003)
label?! (2005)
WebCat2b12plugin - [search] is broken ... not! (1997)
strange [Shell] things / Linux.2 (2000)
Showif, Hideif reverse logic ? (1997)
Help name our technology! (1997)
webcat license???? (1997)
Highlighting words found in a keyword search (2003)
absolute path (*) - how does it work? (2007)
Initiating NewCart (1997)
WebCatalog for Postcards ? (1997)
Variables (1999)
can WC render sites out? (1997)
Unix Guide (2004)