Saturday's outage

[Paddy]

Many of you may have noticed that the site was inaccessible from about 4:30 EDT yesterday until about 1:30 a.m. today. Strange symptoms: Safari reported a dropped server connection and Firefox either offered to download a PHP page (forums) or served up a blank page (home page). Oddly, I was able to access the site using Internet Exploder 5.2.3, but couldn't post. (WHY IE could get to the site when none of the 4 other browsers I tried could, is beyond me...)

Turns out the problem was on ASO's end - the server tmp directory was full. It was emptied and we were back in business. A rare problem I was told. Well, the rare problem returned in a matter of minutes and we were elevated to level 2 support.

The final diagnosis:

"The /tmp directory was filling up due to complications with the gzip apache module. I have deactivated this module which should prevent this issue from recurring."

I don't know enough about Apache to know what exactly this means and don't have time to figure it out right now, but there you have it. (If anyone can serve up a plain English explanation, I'm all ears!)

So...given that this problem was new and it recurred quickly, ASO must have mucked about with the Apache configuration some time yesterday and somebody made a mistake somewhere.

[giantmike]

I'm pretty sure gzip is used to compress image files (maybe other types of files as well) before they are sent to the user's web browser. This increases the loading speed of the site. I imagine the /tmp directory is used to store the gzipped images while the person is browsing the site. Once the person leaves, it should auto-clean the directory of those compressed images. Alternatively, there should be a batch job that will run nightly (maybe more frequently for a heavily used site) to clear the folder in case of hanging files.

[Xairbusdriver]

giantmike, surely ASO uses something better than a hanging file cabinet!

An added problem could be that TS is not the only site on the server. One of the others could be one that has loads of graphics for viewing and/or downloading. Anyway, I'm glad we had someone in addition to Kriss to baby sit this problem while the rest of of slept!

[sandbox]

Paddy, it must have affected only your server because I manage domains on 3 other servers @ASO that had no issues.


L8r

[Paddy]

Yes, I agree - the other domain I help manage (Chris Kleeman's) was fine as well. Someone must have mucked around with the Ana Lucia server.

The first explanation (before the problem occurred yet again) was as follows:

"Unfortunately there is not much you can do.  Since you are on a shared server, the tmp directory gets filled by all of the users, not just you.  This can be filled by errors in scripts, or by a user dumping files into it.  It is very rare for it to become as full as it did, but it does happen."