Re: [WebDNA] Some code inside a large loops fail ... but why?
This WebDNA talk-list message is from 2010
It keeps the original formatting.
numero = 104433
interpreted = N
texte = Text variables are retained in the spawned thread, but containing = context (loop, founditems, etc) status is not. So, you can set a text = variable to the loop's index value before beginning your spawn. For = example: [writefile test.txt][date] [time] [/writefile] [loop start=3D1&end=3D10] [text]x=3D[index][/text] [spawn][appendfile test.txt][x] [/appendfile][/spawn] [/loop] results in the file test.txt containing: 01/12/2010 21:19:49 1 2 3 4 5 6 7 8 9 10 Brian Fries BrainScan Software On Jan 12, 2010, at 9:13 PM, Kenneth Grome wrote: >> Looking at the docs for [spawn] I see that you may have to=20 >> stagger (in time) those spawns since too many at once will=20 >> weigh down the server too much). >=20 > I tried spawn but the loop index was not passed into the spawn context = so the tcpconnect which needs that index fails. If it actually worked = the huge number of spawns would bring down my webdna server or the data = source server unle >=20 > The best solution is really the simplest, but only if the tcpconnect = has a timeout or never fails. It is very simple to hit the server = consecutively with the code I already posted, and if the tcpconnect = never failed I could get all 300k records from that simple little bit of = code. >=20 >=20 >> Could you spawn/divide the task into batches so that if one=20 >> tcpconnect fails then it will only spoil that one batch? =20 >=20 > Possibly, but first I would have to figure out how to pass the loop = index value into the spawn context, then I would have to figure out how = to create the groupings so that only a small number of tcpconnects were = spawned at one time. And then I would have to figure out how to control = the speed that each spawned thread is created so they are not all = spawned at once. As you can see this is very complicated compared with = the simple code I posted earlier. >=20 > For now I'm using a meta-refresh in my browser to hit that same code = with a loop of only 10 records at a time. If the browser fails I will = know it because the page won't be returned, instead an error will be = displayed. Then I can easily look in the db to find the last record = appended and continue from there. >=20 > The browser technique is possible only because I'm using it to = initially populate the db. It wouldn't be practical on a regular basis. = After the db gets populated initially, adding new records will be far = simpler. =20 >=20 > I was hoping to use tcpconnect to do the initial population but = without a timeout everything grinds to a halt with no warnings or = errors, and I don't think there is any way to create an error message = (or send an error email) once the tcpconnect has failed because as soon = as that happens no other code is processed. >=20 > Sincerely, > Kenneth Grome > www.KenGrome.com >=20 >=20 >=20 >=20 >=20 >=20 > --------------------------------------------------------- > This message is sent to you because you are subscribed to > the mailing list
. > To unsubscribe, E-mail to: > archives: http://mail.webdna.us/list/talk@webdna.us > old archives: http://dev.webdna.us/TalkListArchive/ > Bug Reporting: = http://forum.webdna.us/eucabb.html?page=3Dtopics&category=3D288
Associated Messages, from the most recent to the oldest:
Text variables are retained in the spawned thread, but containing = context (loop, founditems, etc) status is not. So, you can set a text = variable to the loop's index value before beginning your spawn. For = example: [writefile test.txt][date] [time] [/writefile] [loop start=3D1&end=3D10] [text]x=3D[index][/text] [spawn][appendfile test.txt][x] [/appendfile][/spawn] [/loop] results in the file test.txt containing: 01/12/2010 21:19:49 1 2 3 4 5 6 7 8 9 10 Brian Fries BrainScan Software On Jan 12, 2010, at 9:13 PM, Kenneth Grome wrote: >> Looking at the docs for [spawn] I see that you may have to=20 >> stagger (in time) those spawns since too many at once will=20 >> weigh down the server too much). >=20 > I tried spawn but the loop index was not passed into the spawn context = so the tcpconnect which needs that index fails. If it actually worked = the huge number of spawns would bring down my webdna server or the data = source server unle >=20 > The best solution is really the simplest, but only if the tcpconnect = has a timeout or never fails. It is very simple to hit the server = consecutively with the code I already posted, and if the tcpconnect = never failed I could get all 300k records from that simple little bit of = code. >=20 >=20 >> Could you spawn/divide the task into batches so that if one=20 >> tcpconnect fails then it will only spoil that one batch? =20 >=20 > Possibly, but first I would have to figure out how to pass the loop = index value into the spawn context, then I would have to figure out how = to create the groupings so that only a small number of tcpconnects were = spawned at one time. And then I would have to figure out how to control = the speed that each spawned thread is created so they are not all = spawned at once. As you can see this is very complicated compared with = the simple code I posted earlier. >=20 > For now I'm using a meta-refresh in my browser to hit that same code = with a loop of only 10 records at a time. If the browser fails I will = know it because the page won't be returned, instead an error will be = displayed. Then I can easily look in the db to find the last record = appended and continue from there. >=20 > The browser technique is possible only because I'm using it to = initially populate the db. It wouldn't be practical on a regular basis. = After the db gets populated initially, adding new records will be far = simpler. =20 >=20 > I was hoping to use tcpconnect to do the initial population but = without a timeout everything grinds to a halt with no warnings or = errors, and I don't think there is any way to create an error message = (or send an error email) once the tcpconnect has failed because as soon = as that happens no other code is processed. >=20 > Sincerely, > Kenneth Grome > www.KenGrome.com >=20 >=20 >=20 >=20 >=20 >=20 > --------------------------------------------------------- > This message is sent to you because you are subscribed to > the mailing list . > To unsubscribe, E-mail to: > archives: http://mail.webdna.us/list/talk@webdna.us > old archives: http://dev.webdna.us/TalkListArchive/ > Bug Reporting: = http://forum.webdna.us/eucabb.html?page=3Dtopics&category=3D288
Brian Fries
DOWNLOAD WEBDNA NOW!
Top Articles:
Talk List
The WebDNA community talk-list is the best place to get some help: several hundred extremely proficient programmers with an excellent knowledge of WebDNA and an excellent spirit will deliver all the tips and tricks you can imagine...
Related Readings:
Currency Conversion (2003)
Pre-flight public flag (1997)
recording the previous found record.. (2004)
Show items based on login Password (2003)
Sending E-mail (1997)
Moving files from one server to another... (2001)
WebCat2 as a chat server? (1997)
WebCat2b13MacPlugIn - [include] doesn't allow creator (1997)
Location of Browser Info.txt file (1997)
is there a [purchase]? (1998)
PC Hub and Web Merchant (1998)
WebCatalog for guestbook ? (1997)
SIMS Email Question (2003)
BUG REPORT -- Refusing connections! (1999)
creator code (1997)
WebCommerce: Folder organization ? (1997)
Ampersand (1997)
Q: how long for answers to the WebDNA-Talk list? (1997)
WSDL Web services design wizard (2005)
referrer and no caches (1997)