OK, here is what happened: Every night, we rotate our log files. After rotation, the Apache server receives a "USR1" signal to be
gracefully reloaded. Gracefully means that before Apache is reloaded, its children must wait to complete their request before dying. The problem is that on rare occasions, but especially during high system loads, some children may still be up waiting to finish their requests while the master process is already being reloaded. When this happens, Apache fails to restart.
Googling reveals
this link:
Quote:
|
Sending the parent Apache process a USR1 signal will make it close the current log files, and re-open them, without loosing any connections currently in progress. This should be used instead of a HUP signal in any log rotation script. The script should first move the current log files to new names (the logs are still open at this stage). Then it should send a USR1 signal to the parent Apache process. The parent will tell the child process to die when they have finished processing their current request, and will open the log files for newly created children (since the old files have been renamed, the opened files will be newly created). As the old children finish their current requests they will close their handle to the (old) log files, and exit. When all the children are dead you can safely process the old log files (for example, by compressing it). Since you cannot know for definite when the old children have all died, the best way to do this is to make your log rotation script sleep for a while after sending the USR1 signal.
|
What it doesn't tell me is
for how long we should make the rotation script sleep. The only other solution I can think of is to send the "HUP" signal instead, which would close all children and spawn new children instead. Alas, it would also mean that all connections that existed during the reload would be dropped.