Re: SMSI: databases corrupted on crash and permission issuesonOSX

This WebDNA talk-list message is from

2002

It keeps the original formatting. numero = 43643
interpreted = N
texte = We will. At the very least any sort of a 'safe write' feature will becontroled by a preference so that it can easily be enabled or disabledthrough the admin.> -----Original Message-----> From: WebCatalog Talk> [mailto:WebDNA-Talk@talk.smithmicro.com]On Behalf> Of Aaron Lynch> Sent: Thursday, September 19, 2002 10:30 PM> To: WebCatalog Talk> Subject: Re: SMSI: databases corrupted on crash and permission issues> onOSX>>> Scott, just please test this extensively with big DBs.>> We have a 300 meg database, and doing certain deletes takes *forever*>>> On 9/19/02 7:49 PM, Scott Anderson mashed the following keys :>> >> In the meantime I have to ask: why doesn't webcat write> databases to> >> a temp file (i.e. .orders.db.tmp) before replacing the> original file?> >> If coded robustly, this change should virtually eliminate the> >> possibility of truncating db files.> > This is currently being considered for the upcoming patch> release. What I> > found interesting is that there is already code in the> engine to do this,> > but it was disabled by the original authors, I do not know why.> >> > The UNIX builds can't benefit from the level of exception> handling available> > on the MAC and PC platforms which is why when one thread> crashes the whole> > WebDNA process quits. We have worked very hard to improve> the level of> > robustness of the UNIX builds, and as a result, the 4.5> builds are a vast> > improvement over the 4.0 build. But even with the> improvements, I think it> > is certainly time to implement the above solution to help> maintain db> > integrity. In the mean time, the best thing to do, is> exactly what you are> > doing, using the debug and error logs to help track down> which template is> > causing the engine thread to quit.> >> >> >> -----Original Message-----> >> From: WebCatalog Talk> >> [mailto:WebDNA-Talk@talk.smithmicro.com]On Behalf> >> Of Dale LaFountain> >> Sent: Thursday, September 19, 2002 6:13 PM> >> To: WebCatalog Talk> >> Subject: SMSI: databases corrupted on crash and permission> >> issues on OSX> >>> >>> >> We just upgraded the back end of our store (www.tfaw.com) from a> >> G4-867/OS9/Webstar/Webcat3.07 to> >> Dual-1Ghz-DDR/OSX10.2/apache/Webcat4.5. Most things are> working well,> >> but we're having a database problem and lost of problems with file> >> permissions.> >>> >> Our sales database has been truncated on two separate occasions> >> today. Problems with this 40+ MB database coincide exactly with> >> WebDNAMonitor entries in system.log noting that webcat has> exited and> >> been restarted:> >> Sep 19 10:12:32 tfaw WebDNAMonitor[574]: WebDNA receives> >> signal SIGBUSVal> >> Sep 19 10:12:32 tfaw WebDNAMonitor[574]: WebDNA with pid> >> [4338] is alive> >> Sep 19 10:12:33 tfaw WebDNAMonitor[574]: WebDNA with pid> >> [4338] is dead> >> Sep 19 10:12:33 tfaw WebDNAMonitor[574]: WebDNA restarts with> >> pid [25411]> >> Sep 19 15:31:02 tfaw WebDNAMonitor[574]: WebDNA receives> >> signal SIGTERMal> >> Sep 19 15:31:02 tfaw WebDNAMonitor[574]: WebDNA with pid> >> [25411] is alive> >> Sep 19 15:31:03 tfaw WebDNAMonitor[574]: WebDNA with pid> >> [25411] is dead> >> Sep 19 15:31:03 tfaw WebDNAMonitor[574]: WebDNA restarts with> >> pid [10584]> >> Sep 19 15:31:04 tfaw WebDNAMonitor[574]: Thread receives> >> terminate signal> >> Sep 19 15:31:04 tfaw WebDNAMonitor[574]: WebDNA with pid> >> [10584] is dead> >> Sep 19 15:31:04 tfaw WebDNAMonitor[574]: WebDNA restarts with> >> pid [10585]> >>> >> I'm still running my webcatwatch.sh every minute from> root's crontab,> >> because I have found some situations where WebDNAMonitor> is not able> >> to recover gracefully from a crash (~1 time in 10). One> of our other> >> OSX servers still crashes 4-6 times per day, but I'm hoping to> >> partially fix that with a hardware upgrade.> >>> >> We're now watching (debug file enabled - could the filename be any> >> more cumbersome??) to see if a specific template is> causing the crash> >> in 4.x when it ran fine for 2 years in 3.x. We're also going to> >> archive some data from this db to speed access and saving> of the file.> >>> >> In the meantime I have to ask: why doesn't webcat write> databases to> >> a temp file (i.e. .orders.db.tmp) before replacing the> original file?> >> If coded robustly, this change should virtually eliminate the> >> possibility of truncating db files.> >>> >> I realize this situation has come up multiple times in the past on> >> this list, and manual backups of crucial databases hourly> or more was> >> the only real solution. This is fine except for the size> and number> >> of databases we use and the frequency of changes (lots of changes).> >> Yes, I could write a shell script that saves incremental copies of> >> each database and notifies me if the database is suddenly> truncated.> >> However, I believe that maintaining database integrity> should be the> >> responsibility of the application to the fullest extent> possible, and> >> not left as an exercise for each developer to implement their own> >> safety net. Life would be much simpler for everyone if> Webcat didn't> >> mangle our databases in the first place, or even better,> if it didn't> >> crash in the middle of saving a db...> >>> >> We have heard from several developers on this topic in the past. I> >> would really appreciate a response from SMSI this time around.> >>> >>> >> Permissions: Whenever webcat moves an orderfile, it changes the> >> ownership of that file from the shared developer owner> with mode 664> >> to owner www and the 644 permissions (no group write).> User-created> >> shoppingcarts are www/www and mode 664, which is fine until webcat> >> authorizes them, after which they are www/www and mode 644. Our> >> developers often need to manipulate these orderfiles manually for> >> various reasons. The developers are in the www group, and> the parent> >> directories (Orders, Completed, etc) all have 770 permission.> >>> >> A couple solutions come to mind:> >>> >> - set up a crontab to chmod all files in these directories every> >> minute (did this already, but it's a sledgehammer solution)> >> - swap out all the [movefile] contexts with [shell]mv oldpath> >> newpath[/shell] contexts to preserve the file settings> >> - make developers login as www (same user as webcat> process), but I'm> >> not too hot about the idea of giving user www a valid shell for> >> security reasons.> >> - report this as a potential bug to SMSI and hope they fix it in a> >> future release> >>> >>> >> On a semi-related note: OSX Jaguar client -> Jaguar client file> >> permissions are an equal pain in the a$$. The default> permission set> >> through personal file sharing are 600, which of course prevents> >> apache from serving the files unless you force all developers to> >> login as www (which, again, is not desirable). I haven't> tried 10.2> >> server yet, but that's an expensive solution considering> the fraction> >> of 10 Server features we would be using. I'm going to try NFS> >> instead of AFP since all our developers are running OSX.> >>> >> Any thoughts on the above?> >>> >> -Dale> >>> >> -------------------------------------------------------------> >> This message is sent to you because you are subscribed to> >> the mailing list .> >> To unsubscribe, E-mail to: > >> To switch to the DIGEST mode, E-mail to> >> > >> Web Archive of this list is at: http://search.smithmicro.com/> >> >> > -------------------------------------------------------------> > This message is sent to you because you are subscribed to> > the mailing list .> > To unsubscribe, E-mail to: > > To switch to the DIGEST mode, E-mail to> > > > Web Archive of this list is at: http://search.smithmicro.com/> >>>> -------------------------------------------------------------> This message is sent to you because you are subscribed to> the mailing list .> To unsubscribe, E-mail to: > To switch to the DIGEST mode, E-mail to> > Web Archive of this list is at: http://search.smithmicro.com/-------------------------------------------------------------This message is sent to you because you are subscribed to the mailing list .To unsubscribe, E-mail to: To switch to the DIGEST mode, E-mail to Web Archive of this list is at: http://search.smithmicro.com/ Associated Messages, from the most recent to the oldest:

Re: SMSI: databases corrupted on crash and permission issuesonOSX (Scott Anderson 2002)

We will. At the very least any sort of a 'safe write' feature will becontroled by a preference so that it can easily be enabled or disabledthrough the admin.> -----Original Message-----> From: WebCatalog Talk> [mailto:WebDNA-Talk@talk.smithmicro.com]On Behalf> Of Aaron Lynch> Sent: Thursday, September 19, 2002 10:30 PM> To: WebCatalog Talk> Subject: Re: SMSI: databases corrupted on crash and permission issues> onOSX>>> Scott, just please test this extensively with big DBs.>> We have a 300 meg database, and doing certain deletes takes *forever*>>> On 9/19/02 7:49 PM, Scott Anderson mashed the following keys :>> >> In the meantime I have to ask: why doesn't webcat write> databases to> >> a temp file (i.e. .orders.db.tmp) before replacing the> original file?> >> If coded robustly, this change should virtually eliminate the> >> possibility of truncating db files.> > This is currently being considered for the upcoming patch> release. What I> > found interesting is that there is already code in the> engine to do this,> > but it was disabled by the original authors, I do not know why.> >> > The UNIX builds can't benefit from the level of exception> handling available> > on the MAC and PC platforms which is why when one thread> crashes the whole> > WebDNA process quits. We have worked very hard to improve> the level of> > robustness of the UNIX builds, and as a result, the 4.5> builds are a vast> > improvement over the 4.0 build. But even with the> improvements, I think it> > is certainly time to implement the above solution to help> maintain db> > integrity. In the mean time, the best thing to do, is> exactly what you are> > doing, using the debug and error logs to help track down> which template is> > causing the engine thread to quit.> >> >> >> -----Original Message-----> >> From: WebCatalog Talk> >> [mailto:WebDNA-Talk@talk.smithmicro.com]On Behalf> >> Of Dale LaFountain> >> Sent: Thursday, September 19, 2002 6:13 PM> >> To: WebCatalog Talk> >> Subject: SMSI: databases corrupted on crash and permission> >> issues on OSX> >>> >>> >> We just upgraded the back end of our store (www.tfaw.com) from a> >> G4-867/OS9/Webstar/Webcat3.07 to> >> Dual-1Ghz-DDR/OSX10.2/apache/Webcat4.5. Most things are> working well,> >> but we're having a database problem and lost of problems with file> >> permissions.> >>> >> Our sales database has been truncated on two separate occasions> >> today. Problems with this 40+ MB database coincide exactly with> >> WebDNAMonitor entries in system.log noting that webcat has> exited and> >> been restarted:> >> Sep 19 10:12:32 tfaw WebDNAMonitor[574]: WebDNA receives> >> signal SIGBUSVal> >> Sep 19 10:12:32 tfaw WebDNAMonitor[574]: WebDNA with pid> >> [4338] is alive> >> Sep 19 10:12:33 tfaw WebDNAMonitor[574]: WebDNA with pid> >> [4338] is dead> >> Sep 19 10:12:33 tfaw WebDNAMonitor[574]: WebDNA restarts with> >> pid [25411]> >> Sep 19 15:31:02 tfaw WebDNAMonitor[574]: WebDNA receives> >> signal SIGTERMal> >> Sep 19 15:31:02 tfaw WebDNAMonitor[574]: WebDNA with pid> >> [25411] is alive> >> Sep 19 15:31:03 tfaw WebDNAMonitor[574]: WebDNA with pid> >> [25411] is dead> >> Sep 19 15:31:03 tfaw WebDNAMonitor[574]: WebDNA restarts with> >> pid [10584]> >> Sep 19 15:31:04 tfaw WebDNAMonitor[574]: Thread receives> >> terminate signal> >> Sep 19 15:31:04 tfaw WebDNAMonitor[574]: WebDNA with pid> >> [10584] is dead> >> Sep 19 15:31:04 tfaw WebDNAMonitor[574]: WebDNA restarts with> >> pid [10585]> >>> >> I'm still running my webcatwatch.sh every minute from> root's crontab,> >> because I have found some situations where WebDNAMonitor> is not able> >> to recover gracefully from a crash (~1 time in 10). One> of our other> >> OSX servers still crashes 4-6 times per day, but I'm hoping to> >> partially fix that with a hardware upgrade.> >>> >> We're now watching (debug file enabled - could the filename be any> >> more cumbersome??) to see if a specific template is> causing the crash> >> in 4.x when it ran fine for 2 years in 3.x. We're also going to> >> archive some data from this db to speed access and saving> of the file.> >>> >> In the meantime I have to ask: why doesn't webcat write> databases to> >> a temp file (i.e. .orders.db.tmp) before replacing the> original file?> >> If coded robustly, this change should virtually eliminate the> >> possibility of truncating db files.> >>> >> I realize this situation has come up multiple times in the past on> >> this list, and manual backups of crucial databases hourly> or more was> >> the only real solution. This is fine except for the size> and number> >> of databases we use and the frequency of changes (lots of changes).> >> Yes, I could write a shell script that saves incremental copies of> >> each database and notifies me if the database is suddenly> truncated.> >> However, I believe that maintaining database integrity> should be the> >> responsibility of the application to the fullest extent> possible, and> >> not left as an exercise for each developer to implement their own> >> safety net. Life would be much simpler for everyone if> Webcat didn't> >> mangle our databases in the first place, or even better,> if it didn't> >> crash in the middle of saving a db...> >>> >> We have heard from several developers on this topic in the past. I> >> would really appreciate a response from SMSI this time around.> >>> >>> >> Permissions: Whenever webcat moves an orderfile, it changes the> >> ownership of that file from the shared developer owner> with mode 664> >> to owner www and the 644 permissions (no group write).> User-created> >> shoppingcarts are www/www and mode 664, which is fine until webcat> >> authorizes them, after which they are www/www and mode 644. Our> >> developers often need to manipulate these orderfiles manually for> >> various reasons. The developers are in the www group, and> the parent> >> directories (Orders, Completed, etc) all have 770 permission.> >>> >> A couple solutions come to mind:> >>> >> - set up a crontab to chmod all files in these directories every> >> minute (did this already, but it's a sledgehammer solution)> >> - swap out all the [movefile] contexts with [shell]mv oldpath> >> newpath[/shell] contexts to preserve the file settings> >> - make developers login as www (same user as webcat> process), but I'm> >> not too hot about the idea of giving user www a valid shell for> >> security reasons.> >> - report this as a potential bug to SMSI and hope they fix it in a> >> future release> >>> >>> >> On a semi-related note: OSX Jaguar client -> Jaguar client file> >> permissions are an equal pain in the a$$. The default> permission set> >> through personal file sharing are 600, which of course prevents> >> apache from serving the files unless you force all developers to> >> login as www (which, again, is not desirable). I haven't> tried 10.2> >> server yet, but that's an expensive solution considering> the fraction> >> of 10 Server features we would be using. I'm going to try NFS> >> instead of AFP since all our developers are running OSX.> >>> >> Any thoughts on the above?> >>> >> -Dale> >>> >> -------------------------------------------------------------> >> This message is sent to you because you are subscribed to> >> the mailing list .> >> To unsubscribe, E-mail to: > >> To switch to the DIGEST mode, E-mail to> >> > >> Web Archive of this list is at: http://search.smithmicro.com/> >> >> > -------------------------------------------------------------> > This message is sent to you because you are subscribed to> > the mailing list .> > To unsubscribe, E-mail to: > > To switch to the DIGEST mode, E-mail to> > > > Web Archive of this list is at: http://search.smithmicro.com/> >>>> -------------------------------------------------------------> This message is sent to you because you are subscribed to> the mailing list .> To unsubscribe, E-mail to: > To switch to the DIGEST mode, E-mail to> > Web Archive of this list is at: http://search.smithmicro.com/-------------------------------------------------------------This message is sent to you because you are subscribed to the mailing list .To unsubscribe, E-mail to: To switch to the DIGEST mode, E-mail to Web Archive of this list is at: http://search.smithmicro.com/ Scott Anderson

DOWNLOAD WEBDNA NOW!

Re: SMSI: databases corrupted on crash and permission issuesonOSX

2002

Top Articles:

Related Readings: