[WebDNA] Re: Hard-coded db write delay when running certain code?
This WebDNA talk-list message is from 2011
It keeps the original formatting.
numero = 106225
interpreted = N
texte = --Boundary-01=_55tRNNWr5Dx+y9T Content-Type: text/plain; charset="iso-8859-6" Content-Transfer-Encoding: 7bit Sorry for the repetition in my previous email, I just re- read it and it seems I kept saying the same thing over and over again. > WebDNA 7.0 will append to RAM and automatically flush to > disk every 1000 records; otherwise, it writes to RAM and > the db file should not show anything on disk, since this > is the RAM copy which is updated. Anything related to db flushing or writing to disk is irrelevant. WebDNA is simply refusing to append until 140 seconds has passed since the last append. This prevents the server from delivering the page in a few seconds like it should. Instead the page doesn't get sent from the server for 140 seconds because that's the soonest that WebDNA will append the data to the db and finish processing the rest of the WebDNa code on the page. > May be your script > adds 1000 records every 2min20s? No, my script adds only ONE record at a time -- and it and clearly it should be adding this record far faster than 140 seconds since the last request. > Maybe you are appending to a huge database. No, this behavior starts from the very beginning with a completely empty db file, so it has nothing to do with the size of the db. > ... or to a slow disk (if you checked the > option flush to disk after each write) and it takes > 2min20s to be written to disk? I did not check the option to flush to disk every time, but once again I will remind you that flushing to disk has nothing to do with append times! The data is not even being written to RAM until 140 seconds has passed since the last write --and that's the problem here. Something seems to be "hanging up the process" for exactly 140 seconds every time it is run. Maybe it is the internal code related to tcpconnect (or append?) ... but the data always gets written to the db 140 seconds after the last append, and this is certainly more than just a coincidence. > Could you provide your code and the coordinates the > server you tcpconnect to and i will try this on my side. I'll send a zip file by private email with instructions. Sincerely, Kenneth Grome > I'm running a tcpconnect script to extract data from an > online database on a non-webdna server, and I'm seeing > EXACTLY 2 minutes and 20 seconds in between db writes. > > I'm using a 10 second delay in a meta-refresh tag so the > page is only supposed to delay 10 seconds before > reloading itself and sending the next tcpconnect. This > meta-refresh delay works fine, but something is up with > my db writes ... > > The results that get written to my results.db each time a > new tcpconnect is processed is always exactly 2 minutes > and 20 seconds later than the last db write. Just take > a look at these timestamps and you'll see what I mean: > > 01/31/2011@06:30:37 > 01/31/2011@06:32:57 > 01/31/2011@06:35:17 > 01/31/2011@06:37:37 > 01/31/2011@06:39:57 > 01/31/2011@06:42:17 > 01/31/2011@06:44:37 > 01/31/2011@06:46:57 > 01/31/2011@06:49:17 > 01/31/2011@06:51:37 > 01/31/2011@06:53:57 > 01/31/2011@06:56:17 > 01/31/2011@06:58:37 > 01/31/2011@07:00:57 > 01/31/2011@07:03:17 > 01/31/2011@07:05:37 > 01/31/2011@07:07:57 > 01/31/2011@07:10:17 > 01/31/2011@07:12:37 > > As you can see, every db write is exactly 2 minutes and > 20 seconds later than the previous one. To me this > suggests that WebDNA's has some kind of a "time delay" > built into it, and that's why every one of these records > is written to the db on exactly the same schedule. > > Does WebDNA 7.0 have some internal code that's preventing > db writes more frequently than 2:20? > > I think there must be some code inside the WebDNA 7.0 > software that is delaying each write to the db until > exactly 2:20 after the last write, or at the very least > I believe that WebDNA has some internal code that is > causing the results seen above. > > Because it seems virtually impossible that these db > writes would occur on such a precise and predictable > schedule otherwise, especially when the data being > written is obtained via tcpconnect which naturally > includes unpredictable response times from the remote > server. > > Instead of seeing these 2:20 delays for every db write I > SHOULD be seeing a variety of different times inbetween > db writes -- unless something internal to WebDNA is > causing this pattern. > > Here's my code structure: > > --------------------------------------------------------- > ------------------------ > > [search db=search.db ... &max=1] > [founditems] > > [text]content= > [middle startafter=xxx] > [tcpconnect] > [tcpsend] > [/tcpsend] > [/tcpconnect] > [/middle] > [/text] > > [append > db=results.db]stamp=[date]@[time]&content=[content]... > [/append] > >
> > [/founditems] > [/search] > > [flushdatabases] > > --------------------------------------------------------- > ------------------------ > > Something internal to WebDNA *MUST* be controlling the > timing of db writes. That's my conclusion until someone > tells me otherwise. > > By the way, when I use Firefox to visit the page > specified in my tcpconnect I get a response in a couple > of seconds, so I know for a fact that the same pages I'm > retrieving via tcpconnect should not take anywhere near > 2:20 to be returned. > > But that's how long it's taking until thedata is written > to the db, and of course the response page cannot be > returned until my WebDNA code has been fully processed. > This is why I think the delay is the result of some > internal WebDNA code that's preventing db writes until > the "next interval" -- which appears to be 2:20 since > the last db write, or perhaps on the next x seconds of > time that ends in 17, 37 or 57? > > Please if you know something about this, let me know > what's going on here. My script is working fine except > for these frustrating and unexplained db write delays, > and it seems to me that given the precise time interval > of these db writes there must be a WebDNA explanation > for this behavior. > > Thanks. > > Sincerely, > Kenneth Grome --Boundary-01=_55tRNNWr5Dx+y9T Content-Type: text/html; charset="iso-8859-6" Content-Transfer-Encoding: 7bit
Sorry for the repetition in my previous email, I just re-read it and it seems I kept saying the same thing over and over again.
> WebDNA 7.0 will append to RAM and automatically flush to
> disk every 1000 records; otherwise, it writes to RAM and
> the db file should not show anything on disk, since this
> is the RAM copy which is updated.
Anything related to db flushing or writing to disk is irrelevant. WebDNA is simply refusing to append until 140 seconds has passed since the last append.
This prevents the server from delivering the page in a few seconds like it should. Instead the page doesn't get sent from the server for 140 seconds because that's the soonest that WebDNA will append the data to the db and finish processing the rest of the WebDNa code on the page.
> May be your script
> adds 1000 records every 2min20s?
No, my script adds only ONE record at a time -- and it and clearly it should be adding this record far faster than 140 seconds since the last request.
> Maybe you are appending to a huge database.
No, this behavior starts from the very beginning with a completely empty db file, so it has nothing to do with the size of the db.
> ... or to a slow disk (if you checked the
> option flush to disk after each write) and it takes
> 2min20s to be written to disk?
I did not check the option to flush to disk every time, but once again I will remind you that flushing to disk has nothing to do with append times!
The data is not even being written to RAM until 140 seconds has passed since the last write --and that's the problem here.
Something seems to be "hanging up the process" for exactly 140 seconds every time it is run. Maybe it is the internal code related to tcpconnect (or append?) ... but the data always gets written to the db 140 seconds after the last append, and this is certainly more than just a coincidence.
> Could you provide your code and the coordinates the
> server you tcpconnect to and i will try this on my side.
I'll send a zip file by private email with instructions.
Sincerely,
Kenneth Grome
> I'm running a tcpconnect script to extract data from an
> online database on a non-webdna server, and I'm seeing
> EXACTLY 2 minutes and 20 seconds in between db writes.
>
> I'm using a 10 second delay in a meta-refresh tag so the
> page is only supposed to delay 10 seconds before
> reloading itself and sending the next tcpconnect. This
> meta-refresh delay works fine, but something is up with
> my db writes ...
>
> The results that get written to my results.db each time a
> new tcpconnect is processed is always exactly 2 minutes
> and 20 seconds later than the last db write. Just take
> a look at these timestamps and you'll see what I mean:
>
> 01/31/2011@06:30:37
> 01/31/2011@06:32:57
> 01/31/2011@06:35:17
> 01/31/2011@06:37:37
> 01/31/2011@06:39:57
> 01/31/2011@06:42:17
> 01/31/2011@06:44:37
> 01/31/2011@06:46:57
> 01/31/2011@06:49:17
> 01/31/2011@06:51:37
> 01/31/2011@06:53:57
> 01/31/2011@06:56:17
> 01/31/2011@06:58:37
> 01/31/2011@07:00:57
> 01/31/2011@07:03:17
> 01/31/2011@07:05:37
> 01/31/2011@07:07:57
> 01/31/2011@07:10:17
> 01/31/2011@07:12:37
>
> As you can see, every db write is exactly 2 minutes and
> 20 seconds later than the previous one. To me this
> suggests that WebDNA's has some kind of a "time delay"
> built into it, and that's why every one of these records
> is written to the db on exactly the same schedule.
>
> Does WebDNA 7.0 have some internal code that's preventing
> db writes more frequently than 2:20?
>
> I think there must be some code inside the WebDNA 7.0
> software that is delaying each write to the db until
> exactly 2:20 after the last write, or at the very least
> I believe that WebDNA has some internal code that is
> causing the results seen above.
>
> Because it seems virtually impossible that these db
> writes would occur on such a precise and predictable
> schedule otherwise, especially when the data being
> written is obtained via tcpconnect which naturally
> includes unpredictable response times from the remote
> server.
>
> Instead of seeing these 2:20 delays for every db write I
> SHOULD be seeing a variety of different times inbetween
> db writes -- unless something internal to WebDNA is
> causing this pattern.
>
> Here's my code structure:
>
> ---------------------------------------------------------
> ------------------------
>
> [search db=search.db ... &max=1]
> [founditems]
>
> [text]content=
> [middle startafter=xxx]
> [tcpconnect]
> [tcpsend]
> [/tcpsend]
> [/tcpconnect]
> [/middle]
> [/text]
>
> [append
> db=results.db]stamp=[date]@[time]&content=[content]...
> [/append]
>
> <META HTTP-EQUIV="Refresh" CONTENT="10; URL=[thisurl]">
>
> [/founditems]
> [/search]
>
> [flushdatabases]
>
> ---------------------------------------------------------
> ------------------------
>
> Something internal to WebDNA *MUST* be controlling the
> timing of db writes. That's my conclusion until someone
> tells me otherwise.
>
> By the way, when I use Firefox to visit the page
> specified in my tcpconnect I get a response in a couple
> of seconds, so I know for a fact that the same pages I'm
> retrieving via tcpconnect should not take anywhere near
> 2:20 to be returned.
>
> But that's how long it's taking until thedata is written
> to the db, and of course the response page cannot be
> returned until my WebDNA code has been fully processed.
> This is why I think the delay is the result of some
> internal WebDNA code that's preventing db writes until
> the "next interval" -- which appears to be 2:20 since
> the last db write, or perhaps on the next x seconds of
> time that ends in 17, 37 or 57?
>
> Please if you know something about this, let me know
> what's going on here. My script is working fine except
> for these frustrating and unexplained db write delays,
> and it seems to me that given the precise time interval
> of these db writes there must be a WebDNA explanation
> for this behavior.
>
> Thanks.
>
> Sincerely,
> Kenneth Grome
--Boundary-01=_55tRNNWr5Dx+y9T--
Associated Messages, from the most recent to the oldest:
--Boundary-01=_55tRNNWr5Dx+y9T Content-Type: text/plain; charset="iso-8859-6" Content-Transfer-Encoding: 7bit Sorry for the repetition in my previous email, I just re- read it and it seems I kept saying the same thing over and over again. > WebDNA 7.0 will append to RAM and automatically flush to > disk every 1000 records; otherwise, it writes to RAM and > the db file should not show anything on disk, since this > is the RAM copy which is updated. Anything related to db flushing or writing to disk is irrelevant. WebDNA is simply refusing to append until 140 seconds has passed since the last append. This prevents the server from delivering the page in a few seconds like it should. Instead the page doesn't get sent from the server for 140 seconds because that's the soonest that WebDNA will append the data to the db and finish processing the rest of the WebDNa code on the page. > May be your script > adds 1000 records every 2min20s? No, my script adds only ONE record at a time -- and it and clearly it should be adding this record far faster than 140 seconds since the last request. > Maybe you are appending to a huge database. No, this behavior starts from the very beginning with a completely empty db file, so it has nothing to do with the size of the db. > ... or to a slow disk (if you checked the > option flush to disk after each write) and it takes > 2min20s to be written to disk? I did not check the option to flush to disk every time, but once again I will remind you that flushing to disk has nothing to do with append times! The data is not even being written to RAM until 140 seconds has passed since the last write --and that's the problem here. Something seems to be "hanging up the process" for exactly 140 seconds every time it is run. Maybe it is the internal code related to tcpconnect (or append?) ... but the data always gets written to the db 140 seconds after the last append, and this is certainly more than just a coincidence. > Could you provide your code and the coordinates the > server you tcpconnect to and i will try this on my side. I'll send a zip file by private email with instructions. Sincerely, Kenneth Grome > I'm running a tcpconnect script to extract data from an > online database on a non-webdna server, and I'm seeing > EXACTLY 2 minutes and 20 seconds in between db writes. > > I'm using a 10 second delay in a meta-refresh tag so the > page is only supposed to delay 10 seconds before > reloading itself and sending the next tcpconnect. This > meta-refresh delay works fine, but something is up with > my db writes ... > > The results that get written to my results.db each time a > new tcpconnect is processed is always exactly 2 minutes > and 20 seconds later than the last db write. Just take > a look at these timestamps and you'll see what I mean: > > 01/31/2011@06:30:37 > 01/31/2011@06:32:57 > 01/31/2011@06:35:17 > 01/31/2011@06:37:37 > 01/31/2011@06:39:57 > 01/31/2011@06:42:17 > 01/31/2011@06:44:37 > 01/31/2011@06:46:57 > 01/31/2011@06:49:17 > 01/31/2011@06:51:37 > 01/31/2011@06:53:57 > 01/31/2011@06:56:17 > 01/31/2011@06:58:37 > 01/31/2011@07:00:57 > 01/31/2011@07:03:17 > 01/31/2011@07:05:37 > 01/31/2011@07:07:57 > 01/31/2011@07:10:17 > 01/31/2011@07:12:37 > > As you can see, every db write is exactly 2 minutes and > 20 seconds later than the previous one. To me this > suggests that WebDNA's has some kind of a "time delay" > built into it, and that's why every one of these records > is written to the db on exactly the same schedule. > > Does WebDNA 7.0 have some internal code that's preventing > db writes more frequently than 2:20? > > I think there must be some code inside the WebDNA 7.0 > software that is delaying each write to the db until > exactly 2:20 after the last write, or at the very least > I believe that WebDNA has some internal code that is > causing the results seen above. > > Because it seems virtually impossible that these db > writes would occur on such a precise and predictable > schedule otherwise, especially when the data being > written is obtained via tcpconnect which naturally > includes unpredictable response times from the remote > server. > > Instead of seeing these 2:20 delays for every db write I > SHOULD be seeing a variety of different times inbetween > db writes -- unless something internal to WebDNA is > causing this pattern. > > Here's my code structure: > > --------------------------------------------------------- > ------------------------ > > [search db=search.db ... &max=1] >
[founditems] > >
[text]content= > [middle startafter=xxx] >
[tcpconnect] >
[tcpsend] > [/tcpsend] > [/tcpconnect] > [/middle] > [/text] > > [append > db=results.db]stamp=
[date]@
[time]&content=[content]... > [/append] > >
[thisurl]"> > > [/founditems] > [/search] > >
[flushdatabases] > > --------------------------------------------------------- > ------------------------ > > Something internal to WebDNA *MUST* be controlling the > timing of db writes. That's my conclusion until someone > tells me otherwise. > > By the way, when I use Firefox to visit the page > specified in my tcpconnect I get a response in a couple > of seconds, so I know for a fact that the same pages I'm > retrieving via tcpconnect should not take anywhere near > 2:20 to be returned. > > But that's how long it's taking until thedata is written > to the db, and of course the response page cannot be > returned until my WebDNA code has been fully processed. > This is why I think the delay is the result of some > internal WebDNA code that's preventing db writes until > the "next interval" -- which appears to be 2:20 since > the last db write, or perhaps on the next x seconds of > time that ends in 17, 37 or 57? > > Please if you know something about this, let me know > what's going on here. My script is working fine except > for these frustrating and unexplained db write delays, > and it seems to me that given the precise time interval > of these db writes there must be a WebDNA explanation > for this behavior. > > Thanks. > > Sincerely, > Kenneth Grome --Boundary-01=_55tRNNWr5Dx+y9T Content-Type: text/html; charset="iso-8859-6" Content-Transfer-Encoding: 7bit
Sorry for the repetition in my previous email, I just re-read it and it seems I kept saying the same thing over and over again.
> WebDNA 7.0 will append to RAM and automatically flush to
> disk every 1000 records; otherwise, it writes to RAM and
> the db file should not show anything on disk, since this
> is the RAM copy which is updated.
Anything related to db flushing or writing to disk is irrelevant. WebDNA is simply refusing to append until 140 seconds has passed since the last append.
This prevents the server from delivering the page in a few seconds like it should. Instead the page doesn't get sent from the server for 140 seconds because that's the soonest that WebDNA will append the data to the db and finish processing the rest of the WebDNa code on the page.
> May be your script
> adds 1000 records every 2min20s?
No, my script adds only ONE record at a time -- and it and clearly it should be adding this record far faster than 140 seconds since the last request.
> Maybe you are appending to a huge database.
No, this behavior starts from the very beginning with a completely empty db file, so it has nothing to do with the size of the db.
> ... or to a slow disk (if you checked the
> option flush to disk after each write) and it takes
> 2min20s to be written to disk?
I did not check the option to flush to disk every time, but once again I will remind you that flushing to disk has nothing to do with append times!
The data is not even being written to RAM until 140 seconds has passed since the last write --and that's the problem here.
Something seems to be "hanging up the process" for exactly 140 seconds every time it is run. Maybe it is the internal code related to tcpconnect (or append?) ... but the data always gets written to the db 140 seconds after the last append, and this is certainly more than just a coincidence.
> Could you provide your code and the coordinates the
> server you tcpconnect to and i will try this on my side.
I'll send a zip file by private email with instructions.
Sincerely,
Kenneth Grome
> I'm running a tcpconnect script to extract data from an
> online database on a non-webdna server, and I'm seeing
> EXACTLY 2 minutes and 20 seconds in between db writes.
>
> I'm using a 10 second delay in a meta-refresh tag so the
> page is only supposed to delay 10 seconds before
> reloading itself and sending the next tcpconnect. This
> meta-refresh delay works fine, but something is up with
> my db writes ...
>
> The results that get written to my results.db each time a
> new tcpconnect is processed is always exactly 2 minutes
> and 20 seconds later than the last db write. Just take
> a look at these timestamps and you'll see what I mean:
>
> 01/31/2011@06:30:37
> 01/31/2011@06:32:57
> 01/31/2011@06:35:17
> 01/31/2011@06:37:37
> 01/31/2011@06:39:57
> 01/31/2011@06:42:17
> 01/31/2011@06:44:37
> 01/31/2011@06:46:57
> 01/31/2011@06:49:17
> 01/31/2011@06:51:37
> 01/31/2011@06:53:57
> 01/31/2011@06:56:17
> 01/31/2011@06:58:37
> 01/31/2011@07:00:57
> 01/31/2011@07:03:17
> 01/31/2011@07:05:37
> 01/31/2011@07:07:57
> 01/31/2011@07:10:17
> 01/31/2011@07:12:37
>
> As you can see, every db write is exactly 2 minutes and
> 20 seconds later than the previous one. To me this
> suggests that WebDNA's has some kind of a "time delay"
> built into it, and that's why every one of these records
> is written to the db on exactly the same schedule.
>
> Does WebDNA 7.0 have some internal code that's preventing
> db writes more frequently than 2:20?
>
> I think there must be some code inside the WebDNA 7.0
> software that is delaying each write to the db until
> exactly 2:20 after the last write, or at the very least
> I believe that WebDNA has some internal code that is
> causing the results seen above.
>
> Because it seems virtually impossible that these db
> writes would occur on such a precise and predictable
> schedule otherwise, especially when the data being
> written is obtained via tcpconnect which naturally
> includes unpredictable response times from the remote
> server.
>
> Instead of seeing these 2:20 delays for every db write I
> SHOULD be seeing a variety of different times inbetween
> db writes -- unless something internal to WebDNA is
> causing this pattern.
>
> Here's my code structure:
>
> ---------------------------------------------------------
> ------------------------
>
> [search db=search.db ... &max=1]
> [founditems]
>
> [text]content=
> [middle startafter=xxx]
> [tcpconnect]
> [tcpsend]
> [/tcpsend]
> [/tcpconnect]
> [/middle]
> [/text]
>
> [append
> db=results.db]stamp=[date]@[time]&content=[content]...
> [/append]
>
> <META HTTP-EQUIV="Refresh" CONTENT="10; URL=[thisurl]">
>
> [/founditems]
> [/search]
>
> [flushdatabases]
>
> ---------------------------------------------------------
> ------------------------
>
> Something internal to WebDNA *MUST* be controlling the
> timing of db writes. That's my conclusion until someone
> tells me otherwise.
>
> By the way, when I use Firefox to visit the page
> specified in my tcpconnect I get a response in a couple
> of seconds, so I know for a fact that the same pages I'm
> retrieving via tcpconnect should not take anywhere near
> 2:20 to be returned.
>
> But that's how long it's taking until thedata is written
> to the db, and of course the response page cannot be
> returned until my WebDNA code has been fully processed.
> This is why I think the delay is the result of some
> internal WebDNA code that's preventing db writes until
> the "next interval" -- which appears to be 2:20 since
> the last db write, or perhaps on the next x seconds of
> time that ends in 17, 37 or 57?
>
> Please if you know something about this, let me know
> what's going on here. My script is working fine except
> for these frustrating and unexplained db write delays,
> and it seems to me that given the precise time interval
> of these db writes there must be a WebDNA explanation
> for this behavior.
>
> Thanks.
>
> Sincerely,
> Kenneth Grome
--Boundary-01=_55tRNNWr5Dx+y9T--
Kenneth Grome
DOWNLOAD WEBDNA NOW!
Top Articles:
Talk List
The WebDNA community talk-list is the best place to get some help: several hundred extremely proficient programmers with an excellent knowledge of WebDNA and an excellent spirit will deliver all the tips and tricks you can imagine...
Related Readings:
Multiple Merchant Accounts? (1997)
Hideif not working? (2001)
Location of Browser Info.txt file (1997)
WebCat2b12 Mac.acgi--[searchstring] bug (1997)
Date search - yes or no (1997)
Problem: 3.0 doesn't update carts (1997)
Mime-Version in email header (1997)
carriage returns in data (1997)
Attn: Bug in GeneralStore example b15 (1997)
Need relative path explanation (1997)
[WebDNA] cart stored as a session cookie (2008)
Stumpted Again (1997)
WebCat2b13MacPlugIn - [showif][search][/showif] (1997)
Can't find templaes (1997)
SMSI - MacWorld (2005)
'page impression' techniques for banner ads (1999)
Adding order weight (1999)
Nested tags count question (1997)
Format Thousands Looks Busted (2000)
[WebDNA] Search using CL (2017)