Re: [WebDNA] Re: Hard-coded db write delay when running certain code?

This WebDNA talk-list message is from

2011

It keeps the original formatting. numero = 106226
interpreted = N
texte = I will test your code as soon as i get it. It is not related to [append] =since a loop with 800 append (just before the 1000 limits that writes to =disk) takes less than 0.1s to execute[loop start=3D1&end=3D800&advance=3D1][append db=3Dstates.txt&autonumber=3Dref]state=3Dtest_multiple[/append][/loop]Just few questions to reproduce your environment: do you test your =script locally? (http://localhost/....). What OS are your running WebDNA =7.0 on?> but once again I will remind you that flushing to disk has nothing to =do with append times! =20yes, it has: with option "Automatically commit databases to disk after =modification", the [append] tags will be processed one at a time, and if =your db is something like 500MB and your disk writes at 100MB/s, then it =will take *at least* 5 seconds to write.To me, it sounds like a tcpconnect timeout (150 seconds)- chrisOn Jan 31, 2011, at 14:08, Kenneth Grome wrote:> Sorry for the repetition in my previous email, I just re-read it and =it seems I kept saying the same thing over and over again.> > WebDNA 7.0 will append to RAM and automatically flush to> > disk every 1000 records; otherwise, it writes to RAM and> > the db file should not show anything on disk, since this> > is the RAM copy which is updated.=20> Anything related to db flushing or writing to disk is irrelevant. =WebDNA is simply refusing to append until 140 seconds has passed since =the last append.> This prevents the server from delivering the page in a few seconds =like it should. Instead the page doesn't get sent from the server for =140 seconds because that's the soonest that WebDNA will append the data =to the db and finish processing the rest of the WebDNa code on the page.> > May be your script> > adds 1000 records every 2min20s?=20> No, my script adds only ONE record at a time -- and it and clearly it =should be adding this record far faster than 140 seconds since the last =request.> > Maybe you are appending to a huge database.> No, this behavior starts from the very beginning with a completely =empty db file, so it has nothing to do with the size of the db.> > ... or to a slow disk (if you checked the> > option flush to disk after each write) and it takes> > 2min20s to be written to disk?> I did not check the option to flush to disk every time, but once again =I will remind you that flushing to disk has nothing to do with append =times! =20> The data is not even being written to RAM until 140 seconds has passed =since the last write --and that's the problem here.> Something seems to be "hanging up the process" for exactly 140 seconds =every time it is run. Maybe it is the internal code related to =tcpconnect (or append?) ... but the data always gets written to the db =140 seconds after the last append, and this is certainly more than just =a coincidence.> > Could you provide your code and the coordinates the> > server you tcpconnect to and i will try this on my side.> I'll send a zip file by private email with instructions. =20> Sincerely,> Kenneth Grome> > I'm running a tcpconnect script to extract data from an> > online database on a non-webdna server, and I'm seeing> > EXACTLY 2 minutes and 20 seconds in between db writes.> >=20> > I'm using a 10 second delay in a meta-refresh tag so the> > page is only supposed to delay 10 seconds before> > reloading itself and sending the next tcpconnect. This> > meta-refresh delay works fine, but something is up with> > my db writes ...> >=20> > The results that get written to my results.db each time a> > new tcpconnect is processed is always exactly 2 minutes> > and 20 seconds later than the last db write. Just take> > a look at these timestamps and you'll see what I mean:> >=20> > 01/31/2011@06:30:37> > 01/31/2011@06:32:57> > 01/31/2011@06:35:17> > 01/31/2011@06:37:37> > 01/31/2011@06:39:57> > 01/31/2011@06:42:17> > 01/31/2011@06:44:37> > 01/31/2011@06:46:57> > 01/31/2011@06:49:17> > 01/31/2011@06:51:37> > 01/31/2011@06:53:57> > 01/31/2011@06:56:17> > 01/31/2011@06:58:37> > 01/31/2011@07:00:57> > 01/31/2011@07:03:17> > 01/31/2011@07:05:37> > 01/31/2011@07:07:57> > 01/31/2011@07:10:17> > 01/31/2011@07:12:37> >=20> > As you can see, every db write is exactly 2 minutes and> > 20 seconds later than the previous one. To me this> > suggests that WebDNA's has some kind of a "time delay"> > built into it, and that's why every one of these records> > is written to the db on exactly the same schedule.> >=20> > Does WebDNA 7.0 have some internal code that's preventing> > db writes more frequently than 2:20?> >=20> > I think there must be some code inside the WebDNA 7.0> > software that is delaying each write to the db until> > exactly 2:20 after the last write, or at the very least> > I believe that WebDNA has some internal code that is> > causing the results seen above.> >=20> > Because it seems virtually impossible that these db> > writes would occur on such a precise and predictable> > schedule otherwise, especially when the data being> > written is obtained via tcpconnect which naturally> > includes unpredictable response times from the remote> > server.> >=20> > Instead of seeing these 2:20 delays for every db write I> > SHOULD be seeing a variety of different times inbetween> > db writes -- unless something internal to WebDNA is> > causing this pattern.> >=20> > Here's my code structure:> >=20> > ---------------------------------------------------------> > ------------------------> >=20> > [search db=3Dsearch.db ... &max=3D1]> > [founditems]> >=20> > [text]content=3D> > [middle startafter=3Dxxx]> > [tcpconnect]> > [tcpsend]> > [/tcpsend]> > [/tcpconnect]> > [/middle]> > [/text]> >=20> > [append> > db=3Dresults.db]stamp=3D[date]@[time]&content=3D[content]...> > [/append]> >=20> > > >=20> > [/founditems]> > [/search]> >=20> > [flushdatabases]> >=20> > ---------------------------------------------------------> > ------------------------> >=20> > Something internal to WebDNA *MUST* be controlling the> > timing of db writes. That's my conclusion until someone> > tells me otherwise.> >=20> > By the way, when I use Firefox to visit the page> > specified in my tcpconnect I get a response in a couple> > of seconds, so I know for a fact that the same pages I'm> > retrieving via tcpconnect should not take anywhere near> > 2:20 to be returned.> >=20> > But that's how long it's taking until thedata is written> > to the db, and of course the response page cannot be> > returned until my WebDNA code has been fully processed.=20> > This is why I think the delay is the result of some> > internal WebDNA code that's preventing db writes until> > the "next interval" -- which appears to be 2:20 since> > the last db write, or perhaps on the next x seconds of> > time that ends in 17, 37 or 57?> >=20> > Please if you know something about this, let me know> > what's going on here. My script is working fine except> > for these frustrating and unexplained db write delays,> > and it seems to me that given the precise time interval> > of these db writes there must be a WebDNA explanation> > for this behavior.> >=20> > Thanks.> >=20> > Sincerely,> > Kenneth Grome> --------------------------------------------------------- This message =is sent to you because you are subscribed to the mailing list . To =unsubscribe, E-mail to: archives: =http://mail.webdna.us/list/talk@webdna.us Bug =Reporting:support@webdna.us Associated Messages, from the most recent to the oldest:

Re: [WebDNA] Hard-coded db write delay when running certain code? (christophe.billiottet@webdna.us 2011)
[WebDNA] Hard-coded db write delay when running certain code? (Kenneth Grome 2011)

I will test your code as soon as i get it. It is not related to [append] =since a loop with 800 append (just before the 1000 limits that writes to =disk) takes less than 0.1s to execute[loop start=3D1&end=3D800&advance=3D1][append db=3Dstates.txt&autonumber=3Dref]state=3Dtest_multiple[/append][/loop]Just few questions to reproduce your environment: do you test your =script locally? (http://localhost/....). What OS are your running WebDNA =7.0 on?> but once again I will remind you that flushing to disk has nothing to =do with append times! =20yes, it has: with option "Automatically commit databases to disk after =modification", the [append] tags will be processed one at a time, and if =your db is something like 500MB and your disk writes at 100MB/s, then it =will take *at least* 5 seconds to write.To me, it sounds like a tcpconnect timeout (150 seconds)- chrisOn Jan 31, 2011, at 14:08, Kenneth Grome wrote:> Sorry for the repetition in my previous email, I just re-read it and =it seems I kept saying the same thing over and over again.> > WebDNA 7.0 will append to RAM and automatically flush to> > disk every 1000 records; otherwise, it writes to RAM and> > the db file should not show anything on disk, since this> > is the RAM copy which is updated.=20> Anything related to db flushing or writing to disk is irrelevant. =WebDNA is simply refusing to append until 140 seconds has passed since =the last append.> This prevents the server from delivering the page in a few seconds =like it should. Instead the page doesn't get sent from the server for =140 seconds because that's the soonest that WebDNA will append the data =to the db and finish processing the rest of the WebDNa code on the page.> > May be your script> > adds 1000 records every 2min20s?=20> No, my script adds only ONE record at a time -- and it and clearly it =should be adding this record far faster than 140 seconds since the last =request.> > Maybe you are appending to a huge database.> No, this behavior starts from the very beginning with a completely =empty db file, so it has nothing to do with the size of the db.> > ... or to a slow disk (if you checked the> > option flush to disk after each write) and it takes> > 2min20s to be written to disk?> I did not check the option to flush to disk every time, but once again =I will remind you that flushing to disk has nothing to do with append =times! =20> The data is not even being written to RAM until 140 seconds has passed =since the last write --and that's the problem here.> Something seems to be "hanging up the process" for exactly 140 seconds =every time it is run. Maybe it is the internal code related to =tcpconnect (or append?) ... but the data always gets written to the db =140 seconds after the last append, and this is certainly more than just =a coincidence.> > Could you provide your code and the coordinates the> > server you tcpconnect to and i will try this on my side.> I'll send a zip file by private email with instructions. =20> Sincerely,> Kenneth Grome> > I'm running a tcpconnect script to extract data from an> > online database on a non-webdna server, and I'm seeing> > EXACTLY 2 minutes and 20 seconds in between db writes.> >=20> > I'm using a 10 second delay in a meta-refresh tag so the> > page is only supposed to delay 10 seconds before> > reloading itself and sending the next tcpconnect. This> > meta-refresh delay works fine, but something is up with> > my db writes ...> >=20> > The results that get written to my results.db each time a> > new tcpconnect is processed is always exactly 2 minutes> > and 20 seconds later than the last db write. Just take> > a look at these timestamps and you'll see what I mean:> >=20> > 01/31/2011@06:30:37> > 01/31/2011@06:32:57> > 01/31/2011@06:35:17> > 01/31/2011@06:37:37> > 01/31/2011@06:39:57> > 01/31/2011@06:42:17> > 01/31/2011@06:44:37> > 01/31/2011@06:46:57> > 01/31/2011@06:49:17> > 01/31/2011@06:51:37> > 01/31/2011@06:53:57> > 01/31/2011@06:56:17> > 01/31/2011@06:58:37> > 01/31/2011@07:00:57> > 01/31/2011@07:03:17> > 01/31/2011@07:05:37> > 01/31/2011@07:07:57> > 01/31/2011@07:10:17> > 01/31/2011@07:12:37> >=20> > As you can see, every db write is exactly 2 minutes and> > 20 seconds later than the previous one. To me this> > suggests that WebDNA's has some kind of a "time delay"> > built into it, and that's why every one of these records> > is written to the db on exactly the same schedule.> >=20> > Does WebDNA 7.0 have some internal code that's preventing> > db writes more frequently than 2:20?> >=20> > I think there must be some code inside the WebDNA 7.0> > software that is delaying each write to the db until> > exactly 2:20 after the last write, or at the very least> > I believe that WebDNA has some internal code that is> > causing the results seen above.> >=20> > Because it seems virtually impossible that these db> > writes would occur on such a precise and predictable> > schedule otherwise, especially when the data being> > written is obtained via tcpconnect which naturally> > includes unpredictable response times from the remote> > server.> >=20> > Instead of seeing these 2:20 delays for every db write I> > SHOULD be seeing a variety of different times inbetween> > db writes -- unless something internal to WebDNA is> > causing this pattern.> >=20> > Here's my code structure:> >=20> > ---------------------------------------------------------> > ------------------------> >=20> > [search db=3Dsearch.db ... &max=3D1]> > [founditems]> >=20> > [text]content=3D> > [middle startafter=3Dxxx]> > [tcpconnect]> > [tcpsend]> > [/tcpsend]> > [/tcpconnect]> > [/middle]> > [/text]> >=20> > [append> > db=3Dresults.db]stamp=3D[date]@[time]&content=3D[content]...> > [/append]> >=20> > [thisurl]">> >=20> > [/founditems]> > [/search]> >=20> > [flushdatabases]> >=20> > ---------------------------------------------------------> > ------------------------> >=20> > Something internal to WebDNA *MUST* be controlling the> > timing of db writes. That's my conclusion until someone> > tells me otherwise.> >=20> > By the way, when I use Firefox to visit the page> > specified in my tcpconnect I get a response in a couple> > of seconds, so I know for a fact that the same pages I'm> > retrieving via tcpconnect should not take anywhere near> > 2:20 to be returned.> >=20> > But that's how long it's taking until thedata is written> > to the db, and of course the response page cannot be> > returned until my WebDNA code has been fully processed.=20> > This is why I think the delay is the result of some> > internal WebDNA code that's preventing db writes until> > the "next interval" -- which appears to be 2:20 since> > the last db write, or perhaps on the next x seconds of> > time that ends in 17, 37 or 57?> >=20> > Please if you know something about this, let me know> > what's going on here. My script is working fine except> > for these frustrating and unexplained db write delays,> > and it seems to me that given the precise time interval> > of these db writes there must be a WebDNA explanation> > for this behavior.> >=20> > Thanks.> >=20> > Sincerely,> > Kenneth Grome> --------------------------------------------------------- This message =is sent to you because you are subscribed to the mailing list . To =unsubscribe, E-mail to: archives: =http://mail.webdna.us/list/talk@webdna.us Bug =Reporting:support@webdna.us christophe.billiottet@webdna.us

DOWNLOAD WEBDNA NOW!

Re: [WebDNA] Re: Hard-coded db write delay when running certain code?

2011

Top Articles:

Related Readings: