Re: [WebDNA] Append speed limits
This WebDNA talk-list message is from 2008
It keeps the original formatting.
numero = 101430
interpreted = N
texte = Ken, have a look there:http://www.euca.us/webdnatest/show_speed.htmlThere is a bunch of speed tests (though none of them considers the [append] stuff). These tests use [ELAPSEDTIME] (in 60th of second). I would first write a small speed test to check if the 60th of second rule applies to a multiprocessor server.chrisOn Nov 15, 2008, at 10:51, Kenneth Grome wrote:> Your suggestion would work if I didn't need to do live data> processing on the entire data set during the event, but I> do. This is why I'll need a custom program designed to> store and process the data on multiple cores.>> Erlang is designed for transparent multi-core data stuff but> not many people know it which is why I'll probably end up> with C for the first version. Unless I completely redesign> the system so that 20,000 hits a second is not necessary.>> Do you think webdna can handle 500 hits a second? There's> an "almost good enough" alternative I can use with this> requirement, and if I decide to try this as a 'first> version' maybe webdna can handle it.>> Sincerely,> Ken Grome>>>>>>>> 20,000 records per second is a lot. I think the RAM-based>> application is necessary (disk-based is by way too slow)>> but am not sure a quad processor would properly>> distribute the load over the CPUs. If the hits you expect>> come from several nodes, what is would do is a round>> robin between several WebDNA servers using a network load>> balancing system (Alteon, f5-networks..), then write to>> disk afterwards, then merge the data in a single file.>> Not easy anyway....>>>> - chris>>>> On Nov 15, 2008, at 1:25, Kenneth Grome wrote:>>> I've considered it but the business model requires that>>> the data itself is not maintianed by cloud services.>>>>>> Google has something similar BTW, I think they call it>>> "Big Table". When I read about these options the first>>> thing that occurred to me is how similar they may be to>>> webdna's internal structure.>>>>>> I don't know webdna's internal structure of course so>>> I'm just guessing, but from what I've learned in my own>>> experience, and also from the things Grant has>>> explained over the years, I think there are more than a>>> few similarities.>>>>>> Sincerely,>>> Ken Grome>>>>>>> If the dataset gets too large for RAM based db>>>> management, you could try>>>> http://aws.amazon.com/simpledb/.>>>>>> ------------------------------------------------------->>> -- This message is sent to you because you are>>> subscribed to the mailing list
.>>> To unsubscribe, E-mail to: >>> archives: http://mail.webdna.us/list/talk@webdna.us>>> old archives: http://dev.webdna.us/TalkListArchive/>>>> --------------------------------------------------------->> This message is sent to you because you are subscribed to>> the mailing list .>> To unsubscribe, E-mail to: >> archives: http://mail.webdna.us/list/talk@webdna.us>> old archives: http://dev.webdna.us/TalkListArchive/>>> ---------------------------------------------------------> This message is sent to you because you are subscribed to> the mailing list .> To unsubscribe, E-mail to: > archives: http://mail.webdna.us/list/talk@webdna.us> old archives: http://dev.webdna.us/TalkListArchive/
Associated Messages, from the most recent to the oldest:
Ken, have a look there:http://www.euca.us/webdnatest/show_speed.htmlThere is a bunch of speed tests (though none of them considers the [append] stuff). These tests use [elapsedtime] (in 60th of second). I would first write a small speed test to check if the 60th of second rule applies to a multiprocessor server.chrisOn Nov 15, 2008, at 10:51, Kenneth Grome wrote:> Your suggestion would work if I didn't need to do live data> processing on the entire data set during the event, but I> do. This is why I'll need a custom program designed to> store and process the data on multiple cores.>> Erlang is designed for transparent multi-core data stuff but> not many people know it which is why I'll probably end up> with C for the first version. Unless I completely redesign> the system so that 20,000 hits a second is not necessary.>> Do you think webdna can handle 500 hits a second? There's> an "almost good enough" alternative I can use with this> requirement, and if I decide to try this as a 'first> version' maybe webdna can handle it.>> Sincerely,> Ken Grome>>>>>>>> 20,000 records per second is a lot. I think the RAM-based>> application is necessary (disk-based is by way too slow)>> but am not sure a quad processor would properly>> distribute the load over the CPUs. If the hits you expect>> come from several nodes, what is would do is a round>> robin between several WebDNA servers using a network load>> balancing system (Alteon, f5-networks..), then write to>> disk afterwards, then merge the data in a single file.>> Not easy anyway....>>>> - chris>>>> On Nov 15, 2008, at 1:25, Kenneth Grome wrote:>>> I've considered it but the business model requires that>>> the data itself is not maintianed by cloud services.>>>>>> Google has something similar BTW, I think they call it>>> "Big Table". When I read about these options the first>>> thing that occurred to me is how similar they may be to>>> webdna's internal structure.>>>>>> I don't know webdna's internal structure of course so>>> I'm just guessing, but from what I've learned in my own>>> experience, and also from the things Grant has>>> explained over the years, I think there are more than a>>> few similarities.>>>>>> Sincerely,>>> Ken Grome>>>>>>> If the dataset gets too large for RAM based db>>>> management, you could try>>>> http://aws.amazon.com/simpledb/.>>>>>> ------------------------------------------------------->>> -- This message is sent to you because you are>>> subscribed to the mailing list .>>> To unsubscribe, E-mail to: >>> archives: http://mail.webdna.us/list/talk@webdna.us>>> old archives: http://dev.webdna.us/TalkListArchive/>>>> --------------------------------------------------------->> This message is sent to you because you are subscribed to>> the mailing list .>> To unsubscribe, E-mail to: >> archives: http://mail.webdna.us/list/talk@webdna.us>> old archives: http://dev.webdna.us/TalkListArchive/>>> ---------------------------------------------------------> This message is sent to you because you are subscribed to> the mailing list .> To unsubscribe, E-mail to: > archives: http://mail.webdna.us/list/talk@webdna.us> old archives: http://dev.webdna.us/TalkListArchive/
christophe.billiottet@webdna.us
DOWNLOAD WEBDNA NOW!
Top Articles:
Talk List
The WebDNA community talk-list is the best place to get some help: several hundred extremely proficient programmers with an excellent knowledge of WebDNA and an excellent spirit will deliver all the tips and tricks you can imagine...
Related Readings:
[item] appears after 'remove last item' ... (1997)
Variables (1999)
bug in [SendMail] (1997)
Tags in the header (2000)
encryption (2003)
[WebDNA] More wish-list requests implemented (2016)
Shhh... (2006)
Displaying photo attached to first record (1997)
corrupted jpgs (2003)
Re:Password Authentication - request example (1998)
problems with 2 tags (1997)
RE: combining strings (1997)
WebCat2b13MacPlugin - nested [xxx] contexts (1997)
RE: [WebDNA] Latest Windows Version (2010)
newbies to web, spaces in email address (1998)
[WebDNA] [OT] Email Server recommendations? (2014)
Q: writefile and uploads. (1998)
[WebDNA] To Flush, to commit or to...? (2014)
[WebDNA] i wish we could require one *amongst of a collection* of search comparisons/params (2011)
MacAuthorize hub, no modal password dialog? (1997)