Re: [WebDNA] Append speed limits
This WebDNA talk-list message is from 2008
It keeps the original formatting.
numero = 101430
interpreted = N
texte = Ken, have a look there: http://www.euca.us/webdnatest/show_speed.html There is a bunch of speed tests (though none of them considers the [append] stuff). These tests use [ELAPSEDTIME] (in 60th of second). I would first write a small speed test to check if the 60th of second rule applies to a multiprocessor server. chris On Nov 15, 2008, at 10:51, Kenneth Grome wrote: > Your suggestion would work if I didn't need to do live data > processing on the entire data set during the event, but I > do. This is why I'll need a custom program designed to > store and process the data on multiple cores. > > Erlang is designed for transparent multi-core data stuff but > not many people know it which is why I'll probably end up > with C for the first version. Unless I completely redesign > the system so that 20,000 hits a second is not necessary. > > Do you think webdna can handle 500 hits a second? There's > an "almost good enough" alternative I can use with this > requirement, and if I decide to try this as a 'first > version' maybe webdna can handle it. > > Sincerely, > Ken Grome > > > > > > >> 20,000 records per second is a lot. I think the RAM-based >> application is necessary (disk-based is by way too slow) >> but am not sure a quad processor would properly >> distribute the load over the CPUs. If the hits you expect >> come from several nodes, what is would do is a round >> robin between several WebDNA servers using a network load >> balancing system (Alteon, f5-networks..), then write to >> disk afterwards, then merge the data in a single file. >> Not easy anyway.... >> >> - chris >> >> On Nov 15, 2008, at 1:25, Kenneth Grome wrote: >>> I've considered it but the business model requires that >>> the data itself is not maintianed by cloud services. >>> >>> Google has something similar BTW, I think they call it >>> "Big Table". When I read about these options the first >>> thing that occurred to me is how similar they may be to >>> webdna's internal structure. >>> >>> I don't know webdna's internal structure of course so >>> I'm just guessing, but from what I've learned in my own >>> experience, and also from the things Grant has >>> explained over the years, I think there are more than a >>> few similarities. >>> >>> Sincerely, >>> Ken Grome >>> >>>> If the dataset gets too large for RAM based db >>>> management, you could try >>>> http://aws.amazon.com/simpledb/. >>> >>> ------------------------------------------------------- >>> -- This message is sent to you because you are >>> subscribed to the mailing list
. >>> To unsubscribe, E-mail to: >>> archives: http://mail.webdna.us/list/talk@webdna.us >>> old archives: http://dev.webdna.us/TalkListArchive/ >> >> --------------------------------------------------------- >> This message is sent to you because you are subscribed to >> the mailing list . >> To unsubscribe, E-mail to: >> archives: http://mail.webdna.us/list/talk@webdna.us >> old archives: http://dev.webdna.us/TalkListArchive/ > > > --------------------------------------------------------- > This message is sent to you because you are subscribed to > the mailing list . > To unsubscribe, E-mail to: > archives: http://mail.webdna.us/list/talk@webdna.us > old archives: http://dev.webdna.us/TalkListArchive/
Associated Messages, from the most recent to the oldest:
Ken, have a look there: http://www.euca.us/webdnatest/show_speed.html There is a bunch of speed tests (though none of them considers the [append] stuff). These tests use [elapsedtime] (in 60th of second). I would first write a small speed test to check if the 60th of second rule applies to a multiprocessor server. chris On Nov 15, 2008, at 10:51, Kenneth Grome wrote: > Your suggestion would work if I didn't need to do live data > processing on the entire data set during the event, but I > do. This is why I'll need a custom program designed to > store and process the data on multiple cores. > > Erlang is designed for transparent multi-core data stuff but > not many people know it which is why I'll probably end up > with C for the first version. Unless I completely redesign > the system so that 20,000 hits a second is not necessary. > > Do you think webdna can handle 500 hits a second? There's > an "almost good enough" alternative I can use with this > requirement, and if I decide to try this as a 'first > version' maybe webdna can handle it. > > Sincerely, > Ken Grome > > > > > > >> 20,000 records per second is a lot. I think the RAM-based >> application is necessary (disk-based is by way too slow) >> but am not sure a quad processor would properly >> distribute the load over the CPUs. If the hits you expect >> come from several nodes, what is would do is a round >> robin between several WebDNA servers using a network load >> balancing system (Alteon, f5-networks..), then write to >> disk afterwards, then merge the data in a single file. >> Not easy anyway.... >> >> - chris >> >> On Nov 15, 2008, at 1:25, Kenneth Grome wrote: >>> I've considered it but the business model requires that >>> the data itself is not maintianed by cloud services. >>> >>> Google has something similar BTW, I think they call it >>> "Big Table". When I read about these options the first >>> thing that occurred to me is how similar they may be to >>> webdna's internal structure. >>> >>> I don't know webdna's internal structure of course so >>> I'm just guessing, but from what I've learned in my own >>> experience, and also from the things Grant has >>> explained over the years, I think there are more than a >>> few similarities. >>> >>> Sincerely, >>> Ken Grome >>> >>>> If the dataset gets too large for RAM based db >>>> management, you could try >>>> http://aws.amazon.com/simpledb/. >>> >>> ------------------------------------------------------- >>> -- This message is sent to you because you are >>> subscribed to the mailing list . >>> To unsubscribe, E-mail to: >>> archives: http://mail.webdna.us/list/talk@webdna.us >>> old archives: http://dev.webdna.us/TalkListArchive/ >> >> --------------------------------------------------------- >> This message is sent to you because you are subscribed to >> the mailing list . >> To unsubscribe, E-mail to: >> archives: http://mail.webdna.us/list/talk@webdna.us >> old archives: http://dev.webdna.us/TalkListArchive/ > > > --------------------------------------------------------- > This message is sent to you because you are subscribed to > the mailing list . > To unsubscribe, E-mail to: > archives: http://mail.webdna.us/list/talk@webdna.us > old archives: http://dev.webdna.us/TalkListArchive/
christophe.billiottet@webdna.us
DOWNLOAD WEBDNA NOW!
Top Articles:
Talk List
The WebDNA community talk-list is the best place to get some help: several hundred extremely proficient programmers with an excellent knowledge of WebDNA and an excellent spirit will deliver all the tips and tricks you can imagine...
Related Readings:
Title not showing up in ViewOrder template (1998)
Standalone Testing (1998)
Is there a way to just get to the last record in the database? (1997)
Email Destination Folder Question (2004)
AppleScript: Tell application:app location? (1998)
passing variables to .asp (2000)
Adding content advice needed (1998)
webcat (2000)
[WebDNA] Credit card processing (2012)
Searching,Brain Dead! (1999)
[shell] (2002)
Looking for a Manual (1997)
Emailer setup (1997)
Updating fields from different platforms (1998)
BinaryBody problem ...... (2003)
Syntax of a URL for including ?cart=[cart] and #something? (1998)
Appending Record Prob (2001)
New Web Site Developers Database (1996)
[searchString] (1997)
[WebDNA] how can i grab the contents of a post? (2011)