Re: WebCAT has the devil in it!
This WebDNA talk-list message is from 2003
It keeps the original formatting.
numero = 54059
interpreted = N
texte = Alex McCombie wrote: > The db in question gets updated thousands of times per day... As > demonstrated by the tmp files I believe. Not proof of anything of the sort. If the tmp files are normally not present there are only two things you can say with any certainty: 1) There were a lot of tmp files being created in this short period of time 2) WebCat has suddenly decided to stop deleting tmp files Unlike humans, computer code does the same thing repeatedly without variation. The only reason 2 can happen is if something has changed (like too many connections for WebCat to keep up). > > Since user interaction with the site causes these updates, I cant see a > spider being an issue, unless I missed the significance of that. Badly behaved spiders will hit your server much faster and harder than even lots of users can (hundreds/thousands of simultaneous requests for the same resource). > > My log files are about 3 gigs per month in size and as such would be a bit > of an issue trying to get through them. Get a decent grep program or one of the many log digesting programs. What you are looking for are many requests from the same client during the time period. I would recommend getting a copy of something like 'less' which will let you browse the log files without loading the entire log file into RAM. You can also use grep to display only records from the time period when the tmp files were being created. You should also be rotating your logs much more frequently if they are so large. If you can set to rotate by size (10MB?) or set the rotation frequency to every 12 hours. > This was occurring with some regularity from midday tues. till about 10 > minutes before I 'accidentally' noticed the files in the DIR. That is exactly the kind of behavior you might see if a spider was hitting your site. The fact that it stopped just before you noticed is, IMHO, coincidence. The fact that the tmp files started showing up at a specific time and stopped at a specific time certainly suggests that it was an outside agent... John -- John Peacock Director of Information Research and Technology Rowman & Littlefield Publishing Group 4501 Forbes Boulevard Suite H Lanham, MD 20706 301-459-3366 x.5010 fax 301-429-5748 ------------------------------------------------------------- This message is sent to you because you are subscribed to the mailing list
. To unsubscribe, E-mail to: To switch to the DIGEST mode, E-mail to Web Archive of this list is at: http://webdna.smithmicro.com/
Associated Messages, from the most recent to the oldest:
Alex McCombie wrote: > The db in question gets updated thousands of times per day... As > demonstrated by the tmp files I believe. Not proof of anything of the sort. If the tmp files are normally not present there are only two things you can say with any certainty: 1) There were a lot of tmp files being created in this short period of time 2) WebCat has suddenly decided to stop deleting tmp files Unlike humans, computer code does the same thing repeatedly without variation. The only reason 2 can happen is if something has changed (like too many connections for WebCat to keep up). > > Since user interaction with the site causes these updates, I cant see a > spider being an issue, unless I missed the significance of that. Badly behaved spiders will hit your server much faster and harder than even lots of users can (hundreds/thousands of simultaneous requests for the same resource). > > My log files are about 3 gigs per month in size and as such would be a bit > of an issue trying to get through them. Get a decent grep program or one of the many log digesting programs. What you are looking for are many requests from the same client during the time period. I would recommend getting a copy of something like 'less' which will let you browse the log files without loading the entire log file into RAM. You can also use grep to display only records from the time period when the tmp files were being created. You should also be rotating your logs much more frequently if they are so large. If you can set to rotate by size (10MB?) or set the rotation frequency to every 12 hours. > This was occurring with some regularity from midday tues. till about 10 > minutes before I 'accidentally' noticed the files in the DIR. That is exactly the kind of behavior you might see if a spider was hitting your site. The fact that it stopped just before you noticed is, IMHO, coincidence. The fact that the tmp files started showing up at a specific time and stopped at a specific time certainly suggests that it was an outside agent... John -- John Peacock Director of Information Research and Technology Rowman & Littlefield Publishing Group 4501 Forbes Boulevard Suite H Lanham, MD 20706 301-459-3366 x.5010 fax 301-429-5748 ------------------------------------------------------------- This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: To switch to the DIGEST mode, E-mail to Web Archive of this list is at: http://webdna.smithmicro.com/
John Peacock
DOWNLOAD WEBDNA NOW!
Top Articles:
Talk List
The WebDNA community talk-list is the best place to get some help: several hundred extremely proficient programmers with an excellent knowledge of WebDNA and an excellent spirit will deliver all the tips and tricks you can imagine...
Related Readings:
Emailer on NT CAN'T handle large email files (1997)
off topic - dna snipets (1997)
Permissions with 3.06 Linux (2000)
Major problem SOLVED! (1999)
A few questions. . . (1997)
Help formatting search results w/ table (1997)
can WC render sites out? (1997)
[WebDNA] Suddenly, lots of WebDNA problems with FastCGI (2013)
Sendmail Context (2003)
Gone fishin' (was The Robert Elliot Report?) (2006)
Weird [blank] interpretation (1999)
I am sorry for always being such an ass (2004)
Re:Signal Raised (1997)
Pithy questions on webcommerce & siteedit (1997)
Hierarchy of form/text/math variables (2000)
Bug or syntax error on my part? (1997)
Use of Back and Reload Buttons on ShoppingCart page? (1997)
WC2.0 Memory Requirements (1997)
Plain Language Manual? (1998)
WebCat2b15MacPlugin - [protect] (1997)