Re: [WebDNA] Some code inside a large loops fail ... but why?
This WebDNA talk-list message is from 2010
It keeps the original formatting.
numero = 104435
interpreted = N
texte = I have had similar situations with large iterations, but only inside of a [spawn]. Your example below looks similar to my experience in that you have a single text variable that is rewritten every time and then saved. In my own testing years ago I discovered with that my large db import would fail constantly between record 950 and 980 of the import. I came to the conclusion that there is a memory issue, perhaps a memory leak. I only discovered this, and pinpointed the issue, after creating a complex method to write out a status message to a text file after every iteration. Then another page meta refreshes every few seconds so we can watch the progress. Progress is reported in # records processed, # records to go, KB size of the import file as it shrinks, SKU of the item being imported. I've given my customers a 500 record limitation on imports because of this. But again, these were my findings within a [spawn]. I've successfully tested up to 7000 records import if running through the browser. But I have to walk away from the computer and let it process, and that used to take overnight to finish. Matt Kenneth Grome wrote: > When I run a loop that iterates 10 times it seems to work fine. Everything inside the loop is processed correctly, and the data gets written to the db as expected. But when I iterate 10000 times the replace context fails to record the data to the db: > > [loop start=1&end=10000] > > [text]rawData=[removehtml][tcpconnect host=www.website.com&port=80][tcpsend skipheader=T]GET /folder/file.asp?record=[index] HTTP/1.0[unurl]%0D%0A%0D%0A[/unurl][/tcpsend][/tcpconnect][/removehtml][/text] > > [replace append=T&db=database.db&eqsomefielddatarq=[index]]somefield=[index]&html=[url][rawData][/url][/replace] > > [/loop] > > Network activity continues while the large loop is running, and it looks just like the network activity that exists when the small loop is run, so I believe the tcpconnect is functioning properly. But for some reason no data gets written to the db in the large loop. > > Any idea why the replace fails when I set the number of iterations so high? > > ----------------------------- > > Another question which may lead to new insights here: > > When the tcpconnect fails to get a response, what happens? Does it prevent the rest of the loop from executing? If so, is there some kind of timeout built into tcpconnect that we can use to force the loop to continue after such a tcpconnect freeze/failure? > > Sincerely, > Kenneth Grome > www.KenGrome.com > > > > > > > --------------------------------------------------------- > This message is sent to you because you are subscribed to > the mailing list
. > To unsubscribe, E-mail to: > archives: http://mail.webdna.us/list/talk@webdna.us > old archives: http://dev.webdna.us/TalkListArchive/ > Bug Reporting: http://forum.webdna.us/eucabb.html?page=topics&category=288 > > > -- Matthew A Perosi JewelerWebsites.com ------------------------------by Psi Prime------- Senior Web Developer 323 Union Blvd. Totowa, NJ 07512 Pre-Sales: 888.872.0274 Service: 973.413.8213 Training: 973.413.8214 Fax: 973.413.8217 http://www.jewelerwebsites.com http://en.wikipedia.org/wiki/Psi_Prime%2C_Inc http://www.psiprime.com
Associated Messages, from the most recent to the oldest:
I have had similar situations with large iterations, but only inside of a [spawn]. Your example below looks similar to my experience in that you have a single text variable that is rewritten every time and then saved. In my own testing years ago I discovered with that my large db import would fail constantly between record 950 and 980 of the import. I came to the conclusion that there is a memory issue, perhaps a memory leak. I only discovered this, and pinpointed the issue, after creating a complex method to write out a status message to a text file after every iteration. Then another page meta refreshes every few seconds so we can watch the progress. Progress is reported in # records processed, # records to go, KB size of the import file as it shrinks, SKU of the item being imported. I've given my customers a 500 record limitation on imports because of this. But again, these were my findings within a [spawn]. I've successfully tested up to 7000 records import if running through the browser. But I have to walk away from the computer and let it process, and that used to take overnight to finish. Matt Kenneth Grome wrote: > When I run a loop that iterates 10 times it seems to work fine. Everything inside the loop is processed correctly, and the data gets written to the db as expected. But when I iterate 10000 times the replace context fails to record the data to the db: > > [loop start=1&end=10000] > > [text]rawData=[removehtml][tcpconnect host=www.website.com&port=80][tcpsend skipheader=T]GET /folder/file.asp?record=[index] HTTP/1.0[unurl]%0D%0A%0D%0A[/unurl][/tcpsend][/tcpconnect][/removehtml][/text] > > [replace append=T&db=database.db&eqsomefielddatarq=[index]]somefield=[index]&html=[url][rawData][/url][/replace] > > [/loop] > > Network activity continues while the large loop is running, and it looks just like the network activity that exists when the small loop is run, so I believe the tcpconnect is functioning properly. But for some reason no data gets written to the db in the large loop. > > Any idea why the replace fails when I set the number of iterations so high? > > ----------------------------- > > Another question which may lead to new insights here: > > When the tcpconnect fails to get a response, what happens? Does it prevent the rest of the loop from executing? If so, is there some kind of timeout built into tcpconnect that we can use to force the loop to continue after such a tcpconnect freeze/failure? > > Sincerely, > Kenneth Grome > www.KenGrome.com > > > > > > > --------------------------------------------------------- > This message is sent to you because you are subscribed to > the mailing list . > To unsubscribe, E-mail to: > archives: http://mail.webdna.us/list/talk@webdna.us > old archives: http://dev.webdna.us/TalkListArchive/ > Bug Reporting: http://forum.webdna.us/eucabb.html?page=topics&category=288 > > > -- Matthew A Perosi JewelerWebsites.com ------------------------------by Psi Prime------- Senior Web Developer 323 Union Blvd. Totowa, NJ 07512 Pre-Sales: 888.872.0274 Service: 973.413.8213 Training: 973.413.8214 Fax: 973.413.8217 http://www.jewelerwebsites.com http://en.wikipedia.org/wiki/Psi_Prime%2C_Inc http://www.psiprime.com
"Psi Prime Inc, Matthew A Perosi "
DOWNLOAD WEBDNA NOW!
Top Articles:
Talk List
The WebDNA community talk-list is the best place to get some help: several hundred extremely proficient programmers with an excellent knowledge of WebDNA and an excellent spirit will deliver all the tips and tricks you can imagine...
Related Readings:
Suffix Mapping (1998)
WebCat2 - Getting to the browser's username/password data (1997)
WebCatalog 2.0 b 15 mac (1997)
remotely creating and populating a stock inventory db -almost there! (1999)
Lambada for Macintosh is Here! (1996)
Sendmail html/text problem (2003)
Closing Databases (take 2) (1998)
Showing once on a founditems (1997)
[ShowIf] and empty fields (1997)
Reverse text string (2002)
Browser Resolution (1999)
[WebDNA] [listwords] and [index] (2009)
Shopping Cart Problem (1998)
Sending mail (2003)
WebCat2b15MacPlugIn - [authenticate] not [protect] (1997)
Math (1997)
Quest renew freware program for FunnelWeb! (2006)
3 card formulas! (1999)
New Beta Documentation (1997)
Looking for someone to double check my code... (1999)