Re: Slow Code, how to speed up
This WebDNA talk-list message is from 2002
It keeps the original formatting.
numero = 41119
interpreted = N
texte = >Here's code that takes 1600 ticks to process on my server. How can I speed it up? It is opening a 700k file and then processing it, and the file is getting larger at all times, since it is a list of referrering web sites. > >[!] Search Temp Database [/!] >[search db=referer.txt&neREFERRERdatarq=findall&REFERRERsumm=t][founditems] > >[!] Append to Temp 2 Database [/!] >[append db=templog2.db]referrer=[referrer]&count=[search >db=referer.txt&eqREFERRERdatarq=[url][REFERRER][/url]&deDTsort=1&DTtype=date >&max=1][numfound]&dt=[founditems][dt][/founditems][/search][/append] >[/founditems][/search]That inner [append] is taking more time because it is opening and closing the database file once per time through your outer [founditems] loop, and all that file access is killing the speed.Instead, open templog2.db just once outside the loop, then write raw data to the end of templog2.db using [appendfile], and then flush templog2.db to force it to be read back in as a database. Here's pseudo-code for it. You have to be *very* careful about your linebreaks here, because you don't want extra carriage returns in the middle of your templog.db file to screw up the db format:close templog2.db [text]tab=%09[/text] [appendfile file=templog2.db][search db=referer.txt&blah][founditems][referrer][tab][your count stuff][tab][your dt stuff] [/founditems][/appendfile]The idea here is that with [appendfile] you are just opening the file once, and doing all the writing to it inside your loop. -- ______________________________________________________________________________ Grant Hulbert | Internet Software Inventor | http://www.hulbertfamily.com/------------------------------------------------------------- This message is sent to you because you are subscribed to the mailing list
. To unsubscribe, E-mail to: To switch to the DIGEST mode, E-mail to Web Archive of this list is at: http://search.smithmicro.com/
Associated Messages, from the most recent to the oldest:
>Here's code that takes 1600 ticks to process on my server. How can I speed it up? It is opening a 700k file and then processing it, and the file is getting larger at all times, since it is a list of referrering web sites. > >[!] Search Temp Database [/!] >[search db=referer.txt&neREFERRERdatarq=findall&REFERRERsumm=t][founditems] > >[!] Append to Temp 2 Database [/!] >[append db=templog2.db]referrer=[referrer]&count=[search >db=referer.txt&eqREFERRERdatarq=[url][referrer][/url]&deDTsort=1&DTtype=date >&max=1][numfound]&dt=[founditems][dt][/founditems][/search][/append] >[/founditems][/search]That inner [append] is taking more time because it is opening and closing the database file once per time through your outer [founditems] loop, and all that file access is killing the speed.Instead, open templog2.db just once outside the loop, then write raw data to the end of templog2.db using [appendfile], and then flush templog2.db to force it to be read back in as a database. Here's pseudo-code for it. You have to be *very* careful about your linebreaks here, because you don't want extra carriage returns in the middle of your templog.db file to screw up the db format:close templog2.db [text]tab=%09[/text] [appendfile file=templog2.db][search db=referer.txt&blah][founditems][referrer][tab][your count stuff][tab][your dt stuff] [/founditems][/appendfile]The idea here is that with [appendfile] you are just opening the file once, and doing all the writing to it inside your loop. -- ______________________________________________________________________________ Grant Hulbert | Internet Software Inventor | http://www.hulbertfamily.com/------------------------------------------------------------- This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: To switch to the DIGEST mode, E-mail to Web Archive of this list is at: http://search.smithmicro.com/
Grant Hulbert
DOWNLOAD WEBDNA NOW!
Top Articles:
Talk List
The WebDNA community talk-list is the best place to get some help: several hundred extremely proficient programmers with an excellent knowledge of WebDNA and an excellent spirit will deliver all the tips and tricks you can imagine...
Related Readings:
# fields limited? (1997)
Sending emails (went[ot] ) (2002)
Replacing a Word (1999)
Not really WebCat (1997)
Authorize.net, SIM, tcpconnect and applescript (2003)
RE: NewCart+Search with one click ? (1997)
Customizing WebDelivery.html (1998)
Mac: [ListFiles] bug alert (1997)
WebCat2b13MacPlugIn - [include] doesn't allow creator (1997)
WebCat2 - Getting to the browser's username/password data (1997)
no global [username] or [password] displayed ... (1997)
Protect (1997)
[WebDNA] PayPal IPN harvester (2013)
WebDNA 5 & iTools 7 (2003)
Summing fields (1997)
[COMMITDATABASE FileName] (2000)
Ampersand Character (&) (1997)
&fieldsdir=ra truely random?? (2000)
still facing truncated [texta] problem (2000)
error: Too many nested [xxx] contexts (1997)