Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > Miscellaneous > Announcements

Notices

Reply
 
Thread Tools Search this Thread
Old 12-29-2006, 01:03 PM   #1
Alexander Turcic
Fully Converged
Alexander Turcic ought to be getting tired of karma fortunes by now.Alexander Turcic ought to be getting tired of karma fortunes by now.Alexander Turcic ought to be getting tired of karma fortunes by now.Alexander Turcic ought to be getting tired of karma fortunes by now.Alexander Turcic ought to be getting tired of karma fortunes by now.Alexander Turcic ought to be getting tired of karma fortunes by now.Alexander Turcic ought to be getting tired of karma fortunes by now.Alexander Turcic ought to be getting tired of karma fortunes by now.Alexander Turcic ought to be getting tired of karma fortunes by now.Alexander Turcic ought to be getting tired of karma fortunes by now.Alexander Turcic ought to be getting tired of karma fortunes by now.
 
Alexander Turcic's Avatar
 
Posts: 18,163
Karma: 14021202
Join Date: Oct 2002
Location: Switzerland
Device: Too many to count here.
Apologies for the outage

Duh, we just had our first real outage. It's somehow related to Murphy's Law I think, because it was only the last two days when I was unable to access the Internet for other reasons.

I wish I could say who was to blame for the outage, like a squirrel which somehow squeezed its way into our server, but the truth is: I don't know. A note popped up in my e-mail saying that our Apache server went down on December 28, 2006 05:22:23 PST. A simple restart did the trick - alas with a day delay thanks to my absense. We'll make sure this is not going to happen again!

Thanks to everyone who informed us, including David @ TeleRead, Laurens, Simon, Roland, Daniel, and everyone else!

Anyways, I hope everyone of you saddled up and is ready to ride into a Happy New Year!
Alexander Turcic is offline   Reply With Quote
Old 12-29-2006, 01:34 PM   #2
Hadrien
Feedbooks.com Co-Founder
Hadrien understands the importance of being earnest.Hadrien understands the importance of being earnest.Hadrien understands the importance of being earnest.Hadrien understands the importance of being earnest.Hadrien understands the importance of being earnest.Hadrien understands the importance of being earnest.Hadrien understands the importance of being earnest.Hadrien understands the importance of being earnest.Hadrien understands the importance of being earnest.Hadrien understands the importance of being earnest.Hadrien understands the importance of being earnest.
 
Hadrien's Avatar
 
Posts: 2,263
Karma: 145123
Join Date: Nov 2006
Location: Paris, France
Device: Sony PRS-t-1/350/300/500/505/600/700, Nexus S, iPad
We use another machine to ping the server on Feedbooks. If the ping's not working it send us a texto on our cellphone, that's pretty usefull. You still need an Internet connection with SSH access to the server though to restart everything...
Hadrien is offline   Reply With Quote
Advert
Old 12-29-2006, 01:46 PM   #3
tribble
iLiad Maniac
tribble knows what time it istribble knows what time it istribble knows what time it istribble knows what time it istribble knows what time it istribble knows what time it istribble knows what time it istribble knows what time it istribble knows what time it istribble knows what time it istribble knows what time it is
 
tribble's Avatar
 
Posts: 1,382
Karma: 2369
Join Date: Apr 2006
Location: Germany
Device: Bookeen Opus (i love that thing) and iPad (what an irony)
If a simple apache restart did the trick, you might want to take a look at this: http://www.tildeslash.com/monit/

tribble is offline   Reply With Quote
Old 12-29-2006, 02:02 PM   #4
Lisana
GadgetGirl
Lisana began at the beginning.
 
Posts: 10
Karma: 10
Join Date: Dec 2006
Location: Southern California
Device: Sony Reader / eBookman (currently misplaced)
Welcome back!

Here I'd just found the forum a few days ago, and you were gone! I'm glad that all it took was a simple server restart.
Lisana is offline   Reply With Quote
Old 12-29-2006, 02:49 PM   #5
Alexander Turcic
Fully Converged
Alexander Turcic ought to be getting tired of karma fortunes by now.Alexander Turcic ought to be getting tired of karma fortunes by now.Alexander Turcic ought to be getting tired of karma fortunes by now.Alexander Turcic ought to be getting tired of karma fortunes by now.Alexander Turcic ought to be getting tired of karma fortunes by now.Alexander Turcic ought to be getting tired of karma fortunes by now.Alexander Turcic ought to be getting tired of karma fortunes by now.Alexander Turcic ought to be getting tired of karma fortunes by now.Alexander Turcic ought to be getting tired of karma fortunes by now.Alexander Turcic ought to be getting tired of karma fortunes by now.Alexander Turcic ought to be getting tired of karma fortunes by now.
 
Alexander Turcic's Avatar
 
Posts: 18,163
Karma: 14021202
Join Date: Oct 2002
Location: Switzerland
Device: Too many to count here.
Thanks for the tips guys! It's really my fault, because I didn't plan a backup solution for something like this (me not being around, server going down). It looks as if the Apache process didn't do a graceful restart after the logs had been rotated (which is being done every day).

It's just weird because we didn't change any settings.
Alexander Turcic is offline   Reply With Quote
Advert
Old 12-29-2006, 03:16 PM   #6
Alexander Turcic
Fully Converged
Alexander Turcic ought to be getting tired of karma fortunes by now.Alexander Turcic ought to be getting tired of karma fortunes by now.Alexander Turcic ought to be getting tired of karma fortunes by now.Alexander Turcic ought to be getting tired of karma fortunes by now.Alexander Turcic ought to be getting tired of karma fortunes by now.Alexander Turcic ought to be getting tired of karma fortunes by now.Alexander Turcic ought to be getting tired of karma fortunes by now.Alexander Turcic ought to be getting tired of karma fortunes by now.Alexander Turcic ought to be getting tired of karma fortunes by now.Alexander Turcic ought to be getting tired of karma fortunes by now.Alexander Turcic ought to be getting tired of karma fortunes by now.
 
Alexander Turcic's Avatar
 
Posts: 18,163
Karma: 14021202
Join Date: Oct 2002
Location: Switzerland
Device: Too many to count here.
OK, here is what happened: Every night, we rotate our log files. After rotation, the Apache server receives a "USR1" signal to be gracefully reloaded. Gracefully means that before Apache is reloaded, its children must wait to complete their request before dying. The problem is that on rare occasions, but especially during high system loads, some children may still be up waiting to finish their requests while the master process is already being reloaded. When this happens, Apache fails to restart.

Googling reveals this link:

Quote:
Sending the parent Apache process a USR1 signal will make it close the current log files, and re-open them, without loosing any connections currently in progress. This should be used instead of a HUP signal in any log rotation script. The script should first move the current log files to new names (the logs are still open at this stage). Then it should send a USR1 signal to the parent Apache process. The parent will tell the child process to die when they have finished processing their current request, and will open the log files for newly created children (since the old files have been renamed, the opened files will be newly created). As the old children finish their current requests they will close their handle to the (old) log files, and exit. When all the children are dead you can safely process the old log files (for example, by compressing it). Since you cannot know for definite when the old children have all died, the best way to do this is to make your log rotation script sleep for a while after sending the USR1 signal.
What it doesn't tell me is for how long we should make the rotation script sleep. The only other solution I can think of is to send the "HUP" signal instead, which would close all children and spawn new children instead. Alas, it would also mean that all connections that existed during the reload would be dropped.
Alexander Turcic is offline   Reply With Quote
Old 12-29-2006, 03:22 PM   #7
Leaping Gnome
Evangelist
Leaping Gnome once ate a cherry pie in a record 7 seconds.Leaping Gnome once ate a cherry pie in a record 7 seconds.Leaping Gnome once ate a cherry pie in a record 7 seconds.Leaping Gnome once ate a cherry pie in a record 7 seconds.Leaping Gnome once ate a cherry pie in a record 7 seconds.Leaping Gnome once ate a cherry pie in a record 7 seconds.Leaping Gnome once ate a cherry pie in a record 7 seconds.Leaping Gnome once ate a cherry pie in a record 7 seconds.Leaping Gnome once ate a cherry pie in a record 7 seconds.Leaping Gnome once ate a cherry pie in a record 7 seconds.Leaping Gnome once ate a cherry pie in a record 7 seconds.
 
Posts: 490
Karma: 1641
Join Date: Oct 2006
Location: Louisville
Device: Sony Reader PRS-500
Just tell it to sleep for 5 minutes, that should be plenty of time for anything it is doing.

Glad to see you guys are back up. I wondered what happened, after being down for over a day I was wondering if for some reason the site was taken down. I checked a few other eBook sites, but didn't see any notices or posts, so wasn't sure what was happening.
Leaping Gnome is offline   Reply With Quote
Old 12-29-2006, 04:59 PM   #8
tribble
iLiad Maniac
tribble knows what time it istribble knows what time it istribble knows what time it istribble knows what time it istribble knows what time it istribble knows what time it istribble knows what time it istribble knows what time it istribble knows what time it istribble knows what time it istribble knows what time it is
 
tribble's Avatar
 
Posts: 1,382
Karma: 2369
Join Date: Apr 2006
Location: Germany
Device: Bookeen Opus (i love that thing) and iPad (what an irony)
Alexander, try monit. It lets you monitor the apache and restart it, if it doesnt react on HTTP requests. You can make it send you an email, if the restart does not work. Its a pretty awesome too that monit thingy And you can monitor almost anything else on the machine.
tribble is offline   Reply With Quote
Old 12-31-2006, 07:33 AM   #9
Alexander Turcic
Fully Converged
Alexander Turcic ought to be getting tired of karma fortunes by now.Alexander Turcic ought to be getting tired of karma fortunes by now.Alexander Turcic ought to be getting tired of karma fortunes by now.Alexander Turcic ought to be getting tired of karma fortunes by now.Alexander Turcic ought to be getting tired of karma fortunes by now.Alexander Turcic ought to be getting tired of karma fortunes by now.Alexander Turcic ought to be getting tired of karma fortunes by now.Alexander Turcic ought to be getting tired of karma fortunes by now.Alexander Turcic ought to be getting tired of karma fortunes by now.Alexander Turcic ought to be getting tired of karma fortunes by now.Alexander Turcic ought to be getting tired of karma fortunes by now.
 
Alexander Turcic's Avatar
 
Posts: 18,163
Karma: 14021202
Join Date: Oct 2002
Location: Switzerland
Device: Too many to count here.
Thanks for the tips, guys!

We just installed monit and configured it to make sure that this kind of outage is not going to happen again.
Alexander Turcic is offline   Reply With Quote
Old 01-03-2007, 12:21 PM   #10
sUnShInE
Drama Queen
sUnShInE is a marvel to beholdsUnShInE is a marvel to beholdsUnShInE is a marvel to beholdsUnShInE is a marvel to beholdsUnShInE is a marvel to beholdsUnShInE is a marvel to beholdsUnShInE is a marvel to beholdsUnShInE is a marvel to beholdsUnShInE is a marvel to beholdsUnShInE is a marvel to beholdsUnShInE is a marvel to behold
 
sUnShInE's Avatar
 
Posts: 784
Karma: 11712
Join Date: Nov 2002
Location: United States
Device: Palm Tungsten T|T3
Happy New Year everyone.

Love,

Murphy's Law
sUnShInE is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Apologies in Advance for Dumb Question sherryk_us Sony Reader 9 06-26-2009 06:13 AM
Total N00B questions! (Apologies in advance). OUTATIME Sony Reader 34 02-09-2008 02:51 PM
Database outage Alexander Turcic Announcements 2 07-14-2005 04:44 PM
Gmail - first observed outage! doctorow Lounge 4 08-23-2004 08:04 PM


All times are GMT -4. The time now is 01:19 AM.


MobileRead.com is a privately owned, operated and funded community.