Re: WebCatalog as a ListServ
This WebDNA talk-list message is from 1998
It keeps the original formatting.
numero = 17866
interpreted = N
texte = >>> This is right, this is really a job for a listserv, and it will do the dup >>> checking for you. >> >>Agree. And, as Ken lined out, we cannot blame WebCat for that. I can't imagine a >>database application which handles such mass-mailing satisfiable on the same >>machine. Even if some program does it somehow - the task has to be performed >>anyway and it WILL bring your server down.Listservers generally are very friendly when it comes to sharing the CPU with other processes running on the same Mac. Listservers don't create 5000 files on the hard drive in order to send 5000 email messages, this stuff is all done internally in the listserver's RAM partition, so the MacOS is never invoked like it is 5000 times in a row when you make WebCatalog create 5000 new files.>As a RAM database, I don't think it's too much to ask for for it to be able >to write files to a disk; even if it's a lot of them. But it should be able >to do this in a multithreaded manner.I tried to explain this earlier but I guess I didn't make a very good explanation, so I will try again:The MacOS is invoked by WebCatalog each time WebCatalog is asked to create a SINGLE file.Why?Because one of the good things about the MacOS is that it keeps track of all the files available to it. That means whenever a new file is created, the MacOS instantly grabs the CPU -- and does not share the CPU -- for however long it takes the OS to properly catalog that new file in its internal database.By using [sendmail] inside your [founditems] context on a search that retrieves 5000 records, you are asking WebCatalog to create 5000 new files, all at once. That means the MacOS also must be invoked 5000 times in a row -- by the ONE WebCatalog thread that is creating those 5000 new files.Therefore ... even if all the other WebCatalog threads are free, the MacOS is NOT!When the MacOS is finished cataloguing all 5000 new files in its database, only then will the MacOS be able to give some CPU time to the web server ... and to other applications such as the WebCatalog.acgi.So as I mentioned before, the slowdown you're seeing is not the fault of WebCatalog at all -- it's a simple MacOS limitation combined by your WebDNA code which causes the OS to be invoked 5000 times in succession.Of course, the faster your Mac, the shorter length of time it's CPU will be monopolized by the MacOS. But if you want a responsive server 100% of the time, a faster Mac won't fix things either, you also need to change the way you deal with your listserver needs.>And that's my problem with the Mac WebCatalog CGI at the moment. I have a >customer who very well could be the next Land's End. I want a program that >won't choke when serving 5000 simultaneous web surfers all doing searches at >the same time. This will require heavy multithreading.WebCatalog can handle all your regular web traffic, provided you do not do things that tie up the MacOS for several minutes at a time ... :)>If WebTen is better, just speak up, and I'll switch.WebTen is much faster than WebSTAR, due in large part to the fact that its TCP stack is optimized for web serving while OT is not -- and WebTen has LOTS of built in features that you have to pay extra for (third-party applications) if you want the same features with WebSTAR, too. But WebTen still won't solve your server delay problem if you continue to use WebCatalog as a listserver instead of assigning that task to the appropriate software for the job.>For those of you still reading this tirade, I'd like to also spread some >information I have gleaned from attempting this. I had a list that had about >50 emails (out of 5000) that had a ~(tilde) somewhere in the email address >field. If I tried to send the whole list at once, WebCatalog would write >double (10000) files to the hard drive. I still can't figure out why. (try >emptying the trash of 10000 files in one directory...I don't know about the tilde problem, but emptying the trash is yet another situation in which the MacOS needs to monopolize the CPU. It simply needs to delete each file you're getting rid of from its internal database in order to function properly.Emptying the trash provides a very clear example of just how effective the MacOS is at grabbing the CPU and not letting anything else use it until it has performed the requested task. But fortunately, it takes *less* CPU time for the OS to catalog new files than it does to delete them ... :)>Finally, if anyone has any suggestions or comments regarding LetterRip or >Sims with Autoshare, please forward them to me, as I'm in the market for >some listServ software.As far as listservers are concerned, you might want to consider running ListSTAR *and* the Typhoon module that controls it. Typhoon's ListSTAR module will work with WebCatalog, possibly with no modification but for your specific situation, most likely with minor modification -- which you should be able to do in no time since you're an experienced WebDNA programmer.The reason for using the ListSTAR module is to integrate your existing email address database with the ListSTAR database, so you don't have to duplicate those addresses in both software products all the time.Once the SINGLE bulk email message is created by WebCatalog -- which will take a few ticks -- with no server slowdowns or delays whatsoever -- then that message can be delivered by Emailer to ListSTAR, and then ListSTAR will send it to your 4300 recipients. This wll solve your existing problems by eliminating your server delays, and it is a far better system for the long term as well ... :)Sincerely, Ken Grome 808-737-6499 WebDNA Solutions mailto:ken@webdna.net http://www.webdna.net
Associated Messages, from the most recent to the oldest:
>>> This is right, this is really a job for a listserv, and it will do the dup >>> checking for you. >> >>Agree. And, as Ken lined out, we cannot blame WebCat for that. I can't imagine a >>database application which handles such mass-mailing satisfiable on the same >>machine. Even if some program does it somehow - the task has to be performed >>anyway and it WILL bring your server down.Listservers generally are very friendly when it comes to sharing the CPU with other processes running on the same Mac. Listservers don't create 5000 files on the hard drive in order to send 5000 email messages, this stuff is all done internally in the listserver's RAM partition, so the MacOS is never invoked like it is 5000 times in a row when you make WebCatalog create 5000 new files.>As a RAM database, I don't think it's too much to ask for for it to be able >to write files to a disk; even if it's a lot of them. But it should be able >to do this in a multithreaded manner.I tried to explain this earlier but I guess I didn't make a very good explanation, so I will try again:The MacOS is invoked by WebCatalog each time WebCatalog is asked to create a SINGLE file.Why?Because one of the good things about the MacOS is that it keeps track of all the files available to it. That means whenever a new file is created, the MacOS instantly grabs the CPU -- and does not share the CPU -- for however long it takes the OS to properly catalog that new file in its internal database.By using
[sendmail] inside your
[founditems] context on a search that retrieves 5000 records, you are asking WebCatalog to create 5000 new files, all at once. That means the MacOS also must be invoked 5000 times in a row -- by the ONE WebCatalog thread that is creating those 5000 new files.Therefore ... even if all the other WebCatalog threads are free, the MacOS is NOT!When the MacOS is finished cataloguing all 5000 new files in its database, only then will the MacOS be able to give some CPU time to the web server ... and to other applications such as the WebCatalog.acgi.So as I mentioned before, the slowdown you're seeing is not the fault of WebCatalog at all -- it's a simple MacOS limitation combined by your WebDNA code which causes the OS to be invoked 5000 times in succession.Of course, the faster your Mac, the shorter length of time it's CPU will be monopolized by the MacOS. But if you want a responsive server 100% of the time, a faster Mac won't fix things either, you also need to change the way you deal with your listserver needs.>And that's my problem with the Mac WebCatalog CGI at the moment. I have a >customer who very well could be the next Land's End. I want a program that >won't choke when serving 5000 simultaneous web surfers all doing searches at >the same time. This will require heavy multithreading.WebCatalog can handle all your regular web traffic, provided you do not do things that tie up the MacOS for several minutes at a time ... :)>If WebTen is better, just speak up, and I'll switch.WebTen is much faster than WebSTAR, due in large part to the fact that its TCP stack is optimized for web serving while OT is not -- and WebTen has LOTS of built in features that you have to pay extra for (third-party applications) if you want the same features with WebSTAR, too. But WebTen still won't solve your server delay problem if you continue to use WebCatalog as a listserver instead of assigning that task to the appropriate software for the job.>For those of you still reading this tirade, I'd like to also spread some >information I have gleaned from attempting this. I had a list that had about >50 emails (out of 5000) that had a ~(tilde) somewhere in the email address >field. If I tried to send the whole list at once, WebCatalog would write >double (10000) files to the hard drive. I still can't figure out why. (try >emptying the trash of 10000 files in one directory...I don't know about the tilde problem, but emptying the trash is yet another situation in which the MacOS needs to monopolize the CPU. It simply needs to delete each file you're getting rid of from its internal database in order to function properly.Emptying the trash provides a very clear example of just how effective the MacOS is at grabbing the CPU and not letting anything else use it until it has performed the requested task. But fortunately, it takes *less* CPU time for the OS to catalog new files than it does to delete them ... :)>Finally, if anyone has any suggestions or comments regarding LetterRip or >Sims with Autoshare, please forward them to me, as I'm in the market for >some listServ software.As far as listservers are concerned, you might want to consider running ListSTAR *and* the Typhoon module that controls it. Typhoon's ListSTAR module will work with WebCatalog, possibly with no modification but for your specific situation, most likely with minor modification -- which you should be able to do in no time since you're an experienced WebDNA programmer.The reason for using the ListSTAR module is to integrate your existing email address database with the ListSTAR database, so you don't have to duplicate those addresses in both software products all the time.Once the SINGLE bulk email message is created by WebCatalog -- which will take a few ticks -- with no server slowdowns or delays whatsoever -- then that message can be delivered by Emailer to ListSTAR, and then ListSTAR will send it to your 4300 recipients. This wll solve your existing problems by eliminating your server delays, and it is a far better system for the long term as well ... :)Sincerely, Ken Grome 808-737-6499 WebDNA Solutions mailto:ken@webdna.net http://www.webdna.net
Kenneth Grome
DOWNLOAD WEBDNA NOW!
Top Articles:
Talk List
The WebDNA community talk-list is the best place to get some help: several hundred extremely proficient programmers with an excellent knowledge of WebDNA and an excellent spirit will deliver all the tips and tricks you can imagine...
Related Readings:
Cart file not being written (2000)
Word search (1997)
WebCat consulting $ (1998)
[TaxableTotal] - not working with AOL and IE (1997)
2.0Beta Command Ref (can't find this instruction) (1997)
WebCat2b13MacPlugIn - [include] (1997)
Nested tags count question (1997)
Occasional crashes with $remove (1997)
Protect and Serve (1999)
Bulletin Board (1998)
Limiting user access to .tmpl files (1997)
F3 crashing server (1997)
Nested tags count question (1997)
WC2.0 Memory Requirements (1997)
problems with 2 tags shakur (1997)
WCS Newbie question (1997)
[closeDataBase myDataBase.db] not working? (1997)
Header values are not accepted (1998)
[WebDNA] TLS 1.2 and [tcpconnect] (2018)
form data submission gets truncated (1997)