Re: SOP for WebDNA talk - MSNBot Crashing
This WebDNA talk-list message is from 2004
It keeps the original formatting.
numero = 57807
interpreted = N
texte = How about 10,000 page requests from MSNBot in about 3 hours. I created robots.txt ---------------- # MSNBot Search User-agent: msnbot Disallow: / ------------- on all the root directory of all our sites just to stop webDNA from crashing. I will refine the file later. Right now w/out this file its like MSNBot is doing a DoS on us. Eduardo ----- Original Message ----- From: "Alain Russell"
To: "WebDNA Talk" Sent: Thursday, May 13, 2004 12:13 PM Subject: Re: SOP for WebDNA talk - MSNBot Crashing > I don't mean to cause trouble here but I can't remember the lsat time I > saw WebDNA crash .. not even under heavy load .. > The first thing I would look for here is some bad code .. code that > expects a cart to be passed, eg - what happens if you call the page > with no cart= ? > > I just don't buy the webDNA crashing line anymore .. > > Alain > > > On 14/05/2004, at 6:59 AM, Paul Uttermohlen wrote: > > > Blocking the ranges of IP's used by the MSNBot servers via the > > firewall did > > the trick.... BUT the client wants the sites indexed by MSN. > > > > Gotta let them in... Gotta keep webcat from crashing a thousand times > > an > > hour... > > > > Paul > > > > > > > > > > On 5/13/04 2:09 PM, "Gary Krockover" wrote: > > > >> Can you do a redirect for MSNBots, something like: > >> > >> [showif [browsername]^MSNBOT] > >> [redirect (to a static sitemap that has no links & doesn't carry the > >> [cart])] > >> [/showif] > >> > >> Not sure if that would rectify the problem, just thinking out loud.... > >> > >> GK > >> > >> At 11:37 AM 5/13/2004, you wrote: > >>> Scott, > >>> > >>> This looks like what I experienced last week. I tracked the cart > >>> problems > >>> down to MSNBot which was flooding the shopping cart folder with > >>> carts. Many > >>> of the carts had file names that where hundreds of characters long. > >>> That was > >>> partly due to the way I handle cart naming for repeat customers. > >>> > >>> MSNBot is beta. I think it is flawed. MSN insists that it's just > >>> sophisticated. And perhaps it is WebDNA that is flawed. > >>> > >>> There may be some truth to that. Large volumes of carts or long file > >>> names > >>> for carts should not cause Webcatalog to crash, but it does... And > >>> with > >>> great frequency and predictability. > >>> > >>> So Scott, if there is anything that you can do to stop this crashing > >>> behavior displayed when MSNBot floods our servers it would be greatly > >>> appreciated. > >>> > >>> Crashes occur on 4.5.1, 5.0.1 and 6 > >>> > >>> Thanks, Paul > >> > >> > >> ------------------------------------------------------------- > >> This message is sent to you because you are subscribed to > >> the mailing list . > >> To unsubscribe, E-mail to: > >> To switch to the DIGEST mode, E-mail to > >> > >> Web Archive of this list is at: http://webdna.smithmicro.com/ > > > > ________________________________________________________ > > Paul Uttermohlen > > http://www.Anoweb.com/ > > http://www.Uttermohlen.com/ > > Paul@Anoweb.com > > Columbus, Ohio 43026 > > 614-529-8963 > > _______________________________________________________ > > > > > > > > > > ------------------------------------------------------------- > > This message is sent to you because you are subscribed to > > the mailing list . > > To unsubscribe, E-mail to: > > To switch to the DIGEST mode, E-mail to > > > > Web Archive of this list is at: http://webdna.smithmicro.com/ > > > > > > ------------------------------------------------------------- > This message is sent to you because you are subscribed to > the mailing list . > To unsubscribe, E-mail to: > To switch to the DIGEST mode, E-mail to > Web Archive of this list is at: http://webdna.smithmicro.com/ ------------------------------------------------------------- This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: To switch to the DIGEST mode, E-mail to Web Archive of this list is at: http://webdna.smithmicro.com/
Associated Messages, from the most recent to the oldest:
How about 10,000 page requests from MSNBot in about 3 hours. I created robots.txt ---------------- # MSNBot Search User-agent: msnbot Disallow: / ------------- on all the root directory of all our sites just to stop webDNA from crashing. I will refine the file later. Right now w/out this file its like MSNBot is doing a DoS on us. Eduardo ----- Original Message ----- From: "Alain Russell" To: "WebDNA Talk" Sent: Thursday, May 13, 2004 12:13 PM Subject: Re: SOP for WebDNA talk - MSNBot Crashing > I don't mean to cause trouble here but I can't remember the lsat time I > saw WebDNA crash .. not even under heavy load .. > The first thing I would look for here is some bad code .. code that > expects a cart to be passed, eg - what happens if you call the page > with no cart= ? > > I just don't buy the webDNA crashing line anymore .. > > Alain > > > On 14/05/2004, at 6:59 AM, Paul Uttermohlen wrote: > > > Blocking the ranges of IP's used by the MSNBot servers via the > > firewall did > > the trick.... BUT the client wants the sites indexed by MSN. > > > > Gotta let them in... Gotta keep webcat from crashing a thousand times > > an > > hour... > > > > Paul > > > > > > > > > > On 5/13/04 2:09 PM, "Gary Krockover" wrote: > > > >> Can you do a redirect for MSNBots, something like: > >> > >> [showif [browsername]^MSNBOT] > >> [redirect (to a static sitemap that has no links & doesn't carry the > >> [cart])] > >> [/showif] > >> > >> Not sure if that would rectify the problem, just thinking out loud.... > >> > >> GK > >> > >> At 11:37 AM 5/13/2004, you wrote: > >>> Scott, > >>> > >>> This looks like what I experienced last week. I tracked the cart > >>> problems > >>> down to MSNBot which was flooding the shopping cart folder with > >>> carts. Many > >>> of the carts had file names that where hundreds of characters long. > >>> That was > >>> partly due to the way I handle cart naming for repeat customers. > >>> > >>> MSNBot is beta. I think it is flawed. MSN insists that it's just > >>> sophisticated. And perhaps it is WebDNA that is flawed. > >>> > >>> There may be some truth to that. Large volumes of carts or long file > >>> names > >>> for carts should not cause Webcatalog to crash, but it does... And > >>> with > >>> great frequency and predictability. > >>> > >>> So Scott, if there is anything that you can do to stop this crashing > >>> behavior displayed when MSNBot floods our servers it would be greatly > >>> appreciated. > >>> > >>> Crashes occur on 4.5.1, 5.0.1 and 6 > >>> > >>> Thanks, Paul > >> > >> > >> ------------------------------------------------------------- > >> This message is sent to you because you are subscribed to > >> the mailing list . > >> To unsubscribe, E-mail to: > >> To switch to the DIGEST mode, E-mail to > >> > >> Web Archive of this list is at: http://webdna.smithmicro.com/ > > > > ________________________________________________________ > > Paul Uttermohlen > > http://www.Anoweb.com/ > > http://www.Uttermohlen.com/ > > Paul@Anoweb.com > > Columbus, Ohio 43026 > > 614-529-8963 > > _______________________________________________________ > > > > > > > > > > ------------------------------------------------------------- > > This message is sent to you because you are subscribed to > > the mailing list . > > To unsubscribe, E-mail to: > > To switch to the DIGEST mode, E-mail to > > > > Web Archive of this list is at: http://webdna.smithmicro.com/ > > > > > > ------------------------------------------------------------- > This message is sent to you because you are subscribed to > the mailing list . > To unsubscribe, E-mail to: > To switch to the DIGEST mode, E-mail to > Web Archive of this list is at: http://webdna.smithmicro.com/ ------------------------------------------------------------- This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: To switch to the DIGEST mode, E-mail to Web Archive of this list is at: http://webdna.smithmicro.com/
DOWNLOAD WEBDNA NOW!
Top Articles:
Talk List
The WebDNA community talk-list is the best place to get some help: several hundred extremely proficient programmers with an excellent knowledge of WebDNA and an excellent spirit will deliver all the tips and tricks you can imagine...
Related Readings:
Download URL & access on the fly ? (1997)
download a file (2001)
Protect (1997)
Plugin causes type 2 error on WebStar startup (2000)
E-mail Attachments (1997)
NT [delete] (1998)
Passing a search through a form (2003)
multiple search commands (1997)
blank page from template (1997)
[TaxableTotal] - not working with AOL and IE (1997)
Yet another db manager (2004)
FEW QUESTIONS (1997)
Opinion: [input] should be called [output] ... (1997)
[WebDNA] RE: OT Oversees merchant account and processor to use in UK with Webdna (2008)
Limiting user access to .tmpl files (1997)
Add a field to the error log? (1997)
lineitem variables (1999)
Forumulas.db & Variables (2002)
Stats (2001)
[OT] Web Site Mapping Software (2000)