RE: [WebDNA] Append speed limits
This WebDNA talk-list message is from 2008
It keeps the original formatting.
numero = 101426
interpreted = N
texte = If the dataset gets too large for RAM based db management, you could try http://aws.amazon.com/simpledb/. -----Original Message----- From: Kenneth Grome [mailto:kengrome@gmail.com] Sent: Friday, November 14, 2008 3:27 PM To: talk@webdna.us Subject: Re: [WebDNA] Append speed limits > There are tricks you can use to speed up the appends. For > one, you could close the DB and use APPENDFILE instead. Unless I'm missing something, it seems to me that append will be faster than appendfile because append is performed in RAM and appendfile must write to the hard drive. This depends upon the prefs of course, because if safewrite is on webdna is going to write the data to disk every time anyways -- so I will probably have to turn safewrite off to achieve the best performance. But in my situation, if I use webdna I must also run periodic sorts to determine the specific position of certain records in the sorted db, and in this case it seems that opening and closing the db frequently is 'undesirable'. I haven't run my tests yet because one of my computers has been unavailable for the last few days. When it's available again I'll run some tests, but based on what the folks in another forum have been saying, I'm probably going to need something a whole lot more powerful than webdna -- maybe a custom C application running on a high-performance quad-core machine using 2-3 cores for RAM data storage and one core for periodic snapshots. Others have suggested Erlang rather than C because it's designed specifically for high performance in a transparent manner over multiple core. The good thing is that I only need this capacity for about an hour at a time, then I can write the data to disk afterwards. This means no data gets written to disk during each hour-long 'event' -- it all gets stored in RAM, probably in a red-black tree to keep it sorted and provide the fastest insert/remove performance. I was initially attracted to webdna as a possible tool for this task because of its direct RAM access. But it's not written for multi-core processors and with 70+ million appends / replaces per hour I think I may need an app that's optimized for this specific task and takes full advantage of multiple cores. Sincerely, Ken Grome --------------------------------------------------------- This message is sent to you because you are subscribed to the mailing list
. To unsubscribe, E-mail to: archives: http://mail.webdna.us/list/talk@webdna.us old archives: http://dev.webdna.us/TalkListArchive/
Associated Messages, from the most recent to the oldest:
If the dataset gets too large for RAM based db management, you could try http://aws.amazon.com/simpledb/. -----Original Message----- From: Kenneth Grome [mailto:kengrome@gmail.com] Sent: Friday, November 14, 2008 3:27 PM To: talk@webdna.us Subject: Re: [WebDNA] Append speed limits > There are tricks you can use to speed up the appends. For > one, you could close the DB and use APPENDFILE instead. Unless I'm missing something, it seems to me that append will be faster than appendfile because append is performed in RAM and appendfile must write to the hard drive. This depends upon the prefs of course, because if safewrite is on webdna is going to write the data to disk every time anyways -- so I will probably have to turn safewrite off to achieve the best performance. But in my situation, if I use webdna I must also run periodic sorts to determine the specific position of certain records in the sorted db, and in this case it seems that opening and closing the db frequently is 'undesirable'. I haven't run my tests yet because one of my computers has been unavailable for the last few days. When it's available again I'll run some tests, but based on what the folks in another forum have been saying, I'm probably going to need something a whole lot more powerful than webdna -- maybe a custom C application running on a high-performance quad-core machine using 2-3 cores for RAM data storage and one core for periodic snapshots. Others have suggested Erlang rather than C because it's designed specifically for high performance in a transparent manner over multiple core. The good thing is that I only need this capacity for about an hour at a time, then I can write the data to disk afterwards. This means no data gets written to disk during each hour-long 'event' -- it all gets stored in RAM, probably in a red-black tree to keep it sorted and provide the fastest insert/remove performance. I was initially attracted to webdna as a possible tool for this task because of its direct RAM access. But it's not written for multi-core processors and with 70+ million appends / replaces per hour I think I may need an app that's optimized for this specific task and takes full advantage of multiple cores. Sincerely, Ken Grome --------------------------------------------------------- This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: archives: http://mail.webdna.us/list/talk@webdna.us old archives: http://dev.webdna.us/TalkListArchive/
"Olin Lagon"
DOWNLOAD WEBDNA NOW!
Top Articles:
Talk List
The WebDNA community talk-list is the best place to get some help: several hundred extremely proficient programmers with an excellent knowledge of WebDNA and an excellent spirit will deliver all the tips and tricks you can imagine...
Related Readings:
PSC recommends what date format yr 2000??? (1997)
[Reload mydata.db] (1998)
Problems with webcat 2.01 for NT (1997)
t or f (1997)
PIXO support (1997)
Database Structure? (1998)
[WebDNA] incomplete redirect tag (2009)
Credit card types (1997)
Simple way to create unique SKU (1997)
WC2b15 File Corruption (1997)
WebCatalog for guestbook ? (1997)
linetiems in thankyou.tpl (1997)
[WebDNA] Some code inside a large loops fail ... but why? (2010)
Counting records (2000)
More! on [ShowCart] and GET vs. POST (1997)
More on the email templates (1997)
WebDNA Partner edition .. (2002)
Attn: Bug in GeneralStore example b15 (1997)
date formatting for CC card expiration date check (1998)
New Beta Documentation (1997)