Server outage

[Diana]

Hi All,

The server rolled over last night sometime around 2:00 am. It was never down in the sense that the power was off...it was always on...but, we were mailbombed to the extent that the server couldn't open any more connections.

My sheepish face peeks up here, but part of this was my fault. Last week I attempted to wrap sendmail in a program called TCPWrapper. This wrapper program intercepts connections it is set to listen for, then reads two special files..hosts.allow and hosts.deny to see if those connections are allowed. I created a HUGE hosts.deny file of spammer machines. This worked!...tons of spam never got into the machine.

This was working for several days...Yayy!!.. until last night

Turns out that sendmail in it's original config would only accept 12 simultaneous connections and after that gracefully ask all other sendmail connections to wait. This only affected sendmail and is a transparent thing to the end user cause it doesn't deny mail...only asks the sender to wait.

When I wrapped sendmail, it seems to have lost the ability to count. It started accepting ALL connections as asked. When the mailbomb started, sendmail opened so many connections that it used up all the file nodes for the whole machine...meaning no more processes at all could be run. This caused all server processes to halt.

We rebooted the machine and it came back up as it should. Within ten minutes the mailbombers found us again and it fell over again. This time I already had shell access so I was able to kill sendmail. This gave back all those processes and the server regained its feet without needing a second reboot. yippee!..(I'm so easy to please...if the spammers would just leave me alone.. )

So, I returned the sendmail configuration to its original settings, removing the wrapper process. I dumped all the sendmail connections that were hung and dumped the /tmp directory. By dumping the tmp directory, I may have yet again disrupted the board because PHP runs the board and it places stuff there as it needs..including session files. If you were uncerimoneously dropped or saw strange anomylies this morning, I'm sorry...but all is well now.

I'm researching new machines to replace this stalwart but aging horse. I thank you for your patience.

see ya,

[kelly]

Thanks for the explanation Diana.

You know you're much appreciated around here.

My ISP was down for a while this morning.

Don't know if they're related.  

[sandbox]

I had noticed a glitch in response about that time but wrote it off as a consequence of the approaching storm. Pulling up pages was stalled for a short time, without the error message that was presented a few days back. If it had continued to fail I would have sent a heads-up, but a new page brought the site back without a problem.  

[Highmac]

Thanks for the explanation, Diana. Explains why I got nothing this morning - just couldn't find the url  (around 8-9am UK time). But all is well now. Thanks also for getting it fixed . Much appreciated....  what a way to spend a Sunday morning, eh?