UPDATE - WebDNA - UNIX Shell program request
This WebDNA talk-list message is from 2004
It keeps the original formatting.
numero = 59335
interpreted = N
texte = What I discovered after I mailed the request was that I could directly import a fixed length file after all; the SQL manual online alludes to it, but the book I am using to help decipher SQL is in error on the matter. A user pointed me in the right direction, and after some experimentation, I got it to work as promised.Using the LOAD DATA LOCAL INTO FILE command with appropriate parameters resulted in a file parse; time less than a second. If anyone wants this code, let me know.As usual, I get to blame the documentation for my own shortcomings!Thanks to all who replied.LarryOn Sep 9, 2004, at 9:11 AM, Larry Hewitt wrote:> I am looking for some UNIX experience.>> BUSINESS CASE:>> We run VoteWashington.Org, Washington State's county election > management service. OSX, WebCat 6, MySQL>> We receive an ascii file in fixed length format from a major county > directly out of their ballot counting equipment with precinct level > vote data; 28000 lines of data that needs to be parsed into a SQL > table. The client cannot reprogram the file output to include any > delimiters - limitations of the software in the elections equipment. > It s not acceptable to ask the client to intervene in the process to > do this manually on election night.>> They upload the file, and begin a process whereby the file is parsed > into usable form and read into tables. It is lengthy - according to > what I understand (a relative beginner with SQL), SQL cannot > automatically parse in a LOAD INTO FILE command a fixed-length file.>> I have a routine that does the parsing, but am looking for a more > efficient manner of processing the file.>> OPPORTUNITY:>> If anyone out there has experience with UNIX and OSX, I am wondering > if a shell script can be written and run from the WedDNA SHELL context > to parse the file at the OS level, inserting delimeters. The LOAD > INTO FILE could then be used without the following parsing routine.>> I am willing to pay for a successful implementation of this, or > something like it that resolves the issue of the long parsing routine.>> If anybody is interested in discussing this offline, please email me > directly. I can share the sample file with you, and give you some > more particulars about this issue.>> TIA>> LArry>> Larry Hewitt360.807.1100 OFFICE> HCO, LLC360.807.1103 FAX> 109 No. Tower360.880.4855 CELL> P.O. Box 1017larryh@hewittco.com> Centralia, WA 98531www.votewashington.org> www.hewittco.com>>> -------------------------------------------------------------> This message is sent to you because you are subscribed to> the mailing list
.> To unsubscribe, E-mail to: > To switch to the DIGEST mode, E-mail to > > Web Archive of this list is at: http://webdna.smithmicro.com/>>Larry Hewitt360.807.1100 OFFICEHCO, LLC360.807.1103 FAX109 No. Tower360.880.4855 CELLP.O. Box 1017larryh@hewittco.comCentralia, WA 98531www.votewashington.orgwww.hewittco.com-------------------------------------------------------------This message is sent to you because you are subscribed to the mailing list .To unsubscribe, E-mail to: To switch to the DIGEST mode, E-mail to Web Archive of this list is at: http://webdna.smithmicro.com/
Associated Messages, from the most recent to the oldest:
|
- UPDATE - WebDNA - UNIX Shell program request ( Larry Hewitt 2004)
|
What I discovered after I mailed the request was that I could directly import a fixed length file after all; the SQL manual online alludes to it, but the book I am using to help decipher SQL is in error on the matter. A user pointed me in the right direction, and after some experimentation, I got it to work as promised.Using the LOAD DATA LOCAL INTO FILE command with appropriate parameters resulted in a file parse; time less than a second. If anyone wants this code, let me know.As usual, I get to blame the documentation for my own shortcomings!Thanks to all who replied.LarryOn Sep 9, 2004, at 9:11 AM, Larry Hewitt wrote:> I am looking for some UNIX experience.>> BUSINESS CASE:>> We run VoteWashington.Org, Washington State's county election > management service. OSX, WebCat 6, MySQL>> We receive an ascii file in fixed length format from a major county > directly out of their ballot counting equipment with precinct level > vote data; 28000 lines of data that needs to be parsed into a SQL > table. The client cannot reprogram the file output to include any > delimiters - limitations of the software in the elections equipment. > It s not acceptable to ask the client to intervene in the process to > do this manually on election night.>> They upload the file, and begin a process whereby the file is parsed > into usable form and read into tables. It is lengthy - according to > what I understand (a relative beginner with SQL), SQL cannot > automatically parse in a LOAD INTO FILE command a fixed-length file.>> I have a routine that does the parsing, but am looking for a more > efficient manner of processing the file.>> OPPORTUNITY:>> If anyone out there has experience with UNIX and OSX, I am wondering > if a shell script can be written and run from the WedDNA SHELL context > to parse the file at the OS level, inserting delimeters. The LOAD > INTO FILE could then be used without the following parsing routine.>> I am willing to pay for a successful implementation of this, or > something like it that resolves the issue of the long parsing routine.>> If anybody is interested in discussing this offline, please email me > directly. I can share the sample file with you, and give you some > more particulars about this issue.>> TIA>> LArry>> Larry Hewitt360.807.1100 OFFICE> HCO, LLC360.807.1103 FAX> 109 No. Tower360.880.4855 CELL> P.O. Box 1017larryh@hewittco.com> Centralia, WA 98531www.votewashington.org> www.hewittco.com>>> -------------------------------------------------------------> This message is sent to you because you are subscribed to> the mailing list .> To unsubscribe, E-mail to: > To switch to the DIGEST mode, E-mail to > > Web Archive of this list is at: http://webdna.smithmicro.com/>>Larry Hewitt360.807.1100 OFFICEHCO, LLC360.807.1103 FAX109 No. Tower360.880.4855 CELLP.O. Box 1017larryh@hewittco.comCentralia, WA 98531www.votewashington.orgwww.hewittco.com-------------------------------------------------------------This message is sent to you because you are subscribed to the mailing list .To unsubscribe, E-mail to: To switch to the DIGEST mode, E-mail to Web Archive of this list is at: http://webdna.smithmicro.com/
Larry Hewitt
DOWNLOAD WEBDNA NOW!
Top Articles:
Talk List
The WebDNA community talk-list is the best place to get some help: several hundred extremely proficient programmers with an excellent knowledge of WebDNA and an excellent spirit will deliver all the tips and tricks you can imagine...
Related Readings:
Re Running Locally (2001)
Quitting WebMerchant ? (1997)
PCS Emailer's role ? (1997)
target=_blank and form variables (1997)
Show if time tags (1997)
More Applescript (1997)
Using [Include] Context (1999)
fresh eyes needed. Append won't work. (2000)
form crasehes server (1997)
System 8.5.1 question (1999)
Nested searches (1998)
Another question (1997)
[WebDNA] Hosting Needs for WebDNA (2014)
Ongoing group search problems ... (1997)
Projects & Contractors (1997)
A few questions... (1999)
Separate SSL Server (1997)
Can you identify the [body]? (1999)
RE: Signal Raised (1997)
WebCat2b13MacPlugin - [math][date][/math] problem (1997)