TS was down for over 12 hours

[Paddy]

Quite unrelated to the outages we had last week, A Small Orange had a massive problem on its hands early this morning. The computer of an employee working remotely from home was infected with a virus that took out almost half of ASO's servers. The malicious program deleted data - in some cases ALL the data on a few servers, not just site files and they are still working to get everything restored. Some sites also had a virus planted on their front page for about 30 minutes before ASO discovered it. We did not have that happen and luckily, we seem to have lost few if any posts in our forums; the ASO backup must have been very recent or our database was not one of the ones that was deleted. If there was something you posted late last night/early this morning and it's gone, please repost it.

While some of ASO's customers are understandably furious about all of this (some run businesses online and this means potential loss of sales etc.) I think ASO has done a good job in getting everything back up and running - at least for us. I have several other clients on ASO servers and none of their sites were affected, luckily.

If you're interested in reading more about all of this, there are a couple of threads in the ASO forums:

http://forums.asmallorange.com/index.php?showtopic=12908
http://forums.asmallorange.com/index.php?showtopic=12905

[sandbox]

Well Paddy, you may already know that i was hit pretty hard, as were my clients. It didn't start last night for me it was the night before and progressed. I still have clients with no FTP access which they became aware of at 4pm yesterday.

There are a few issues and I may have been one of the first with problems, and i think if you were in my shoes you may view this issue differently. I'm going rethink keeping so many domains on a single provider. Its impossible to handle the workload in situations like this.

[gunug]

Thanks Paddy!  At work I'm often not sure whether it's the outside website or something with the firewall; I think I've had to convince someone to allow Techsurvivors three times now.

[Paddy]

SB - do you think the virus that attacked the ASO servers last night was the cause of your problems too? ASO should know this if that's the case, since they believe they've traced it to the one employee's computer which was logged in remotely.

While I certainly understand that those running businesses are upset, this can happen no matter where you're hosted. A number of years ago, our North Andover school websites were hacked by that mad Hungarian (?) hacker who puts an animated GIF of the flag (and sometimes obscenities) on the index page of the site he's hacked. It took LunarPages a number of hours or maybe half a day (don't remember now) to sort it all out. Luckily it was on a Sunday and our traffic was typically fairly low that day.

In the case of TS, I have absolutely no complaints with the way ASO has handled things. We don't appear to have lost anything and given the magnitude of the attack (some 25 servers down) and the time of day (middle of the night) I think they've done a good job.

[sandbox]

Paddy, though TS is listed in the effected area A-D, I have clients on servers in the I's that have had no FTP access since yesterday afternoon, when they needed to upload. So I suspect the problem is broader than we are led to understand.
The problems both mine....yours...and ours are self generated.....by ASO

[krissel]

Quick note and then I'm off to the folks...


When I logged on to the ASO status page around 5 AM today, I noticed several servers that were totally out near the middle and bottom of the list. No way did this just affect the servers A-D.


Later...

[Paddy]

Hurley, Jack, Kate and Nadia are all servers that have also had problems today - but I don't know if they're the SAME problems. Kate's problems go back a couple of days from what I can tell - they tried to move everyone off it, but had issues and reverted back. It may be that they have now moved everyone and taken the server off line entirely. There have been no complaints from residents of that server for over 24 hours in the forums.

I've also found reference to people being moved off Hurley. It's possible that those servers are being taken completely off line - I know they've got a bunch of new ones. With everything else that is going on right now, I would imagine deleting defunct servers from the server status page would be a long way down the list of priorities. And I also know that one of my clients on the new server Frogurt was on that server before it was actually listed on the server status page. I sent in a ticket (low priority) asking about it and the response was "oops - we'll fix that!" and they did.

[Xairbusdriver]

But that is simply NOT satisfactory, Paddy! I want problems fixed before appear, not afterward! Anyone can fix them then!

[Mayo]

Wellll, I was this close to signing up two domains with A Small Orange, but this incident has caused me to reconsider that decision...

I don't know how common this kind of thing is among Web hosting companies, but this seems to be evidence of incredibly sloppy network security on the part of A Small Orange. An employee working from home manages to infect the company's servers with a virus? Give Me A Break! At the very least that employee should be Long Gone, and maybe his/her immediate supervisor too... Perhaps that is a bit harsh, but isn't this a serious breach?

This may not have been a major problem for Techsurvivors, but how much time and therefore money has this cost A Small Orange customers who do depend on the Internet for their business?

I'm willing to be convinced otherwise, so if anyone wants to give it a go... But for the time being I'm going to take another look at the other Web hosts on my list.

[Paddy]

Mayo, I've been happy with ASO until this incident. I know this incident is a bit disturbing - but this kind of thing DOES happen elsewhere too. I don't think there is such a thing as 100% uptime anywhere, unless you've got very deep pockets and fork out for dedicated servers with complete redundancy. As for the employee, I think it's a little difficult to judge from afar, especially as we don't know all the details. Frankly, I think ASO is unusual in that we probably know far MORE about the circumstances of the hacking than most hosts would ever reveal. (See the ASO forum thread, now 17 pages long that I linked to)

ASO has always been very responsive to me when I've had any issues and helped me in some cases above and beyond what you'd normally expect from a web host.

Our previous issues on Ana Lucia were solved with the move to a new server. (Interestingly, I found a blog post from someone who'd been unhappy with ASO and was on our old server, Ana Lucia, as well. Much happier once he was moved, though now complaining about today...) The problem we experienced earlier this week with the Russian Yandex bot was not something that ASO has any control over.

If you look up just about any host of any size you'll find "xxxxx sucks" - sometimes thousands of times. Every time I've gone looking, I've had a hard time finding a host that gets consistently good USER reviews - I'm not talking about those sham sites where the web hosts obviously pay to get good reviews or "ratings."

One site that has lots of user reviews: http://www.webhostingjury.com/

I can only speak of my own experiences. NONE of my other three sites on ASO have had any significant problems, and are not on any of the servers affected today. One has been here 2 years, one a year, and one just a month. I can say the same of sites I've hosted with Lunarpages (who get good ratings at webhostingjury BTW). And for that matter, the 4 sites I have at GoDaddy (5 years, 4 years and 2 for 3 years) have never had any problems either, though they're also my most straightforward sites. I would never try putting a Joomla site at GoDaddy for instance (too many horror stories) and I really loathe their control panel. Apart from the existing sites, I only use them for domain registration now.

[sandbox]

Mayo, I would be cautious if I were you. That said, there has been a recent discovery of sloppy hosting, whether it was by policy or by accident is unclear, I have email from the Top of ASO stating both, with apologies.

I was the person responsible for bringing TS to ASO and I feel somewhat responsible for any problems that may occur though, less as time goes on. I've given my opinion out here in public and in email warnings to the Admins of TS prior to this thread, so now I feel barely responsible going forward.

I liked Tim Dorr, the founder of ASO, but during this last week he has been unreachable. Discovering that has reduced my confidence and ability to justify my tolerance through associative slack, sort of speak. So I'm addressing this from a somewhat business and efficiency perspective.

ASO has been a good provider for TS compared to our last one and maybe the next will be even better, if it comes to that. Who knows? Not for me to say anymore.

On the upside: ASO will now be tight as a drum. They will certainly correct their mistakes and implement better security, and oversee the watcher of their remaining clientele. So if your willing to join ASO I think you'll be boarding a tight ship. They will certainly have less work, loosing a percentage of their domains as a result of this incident, which could make or break them.

In any case, I will rethink my willingness to put x % of domains in one basket and give myself sometime to see if leaving ASO entirely is a rational move.

Understand that my issue is more complicated than what TS has experienced, so I have other issues besides just a sloppy employee uploading a virus.

ASO has a good backup system, they have recovered most of my data so cudos to them for that. I don't know that everyone is as pleased as I am or TS is, because from others that I have spoken with and emailed, they have not been so lucky.

That said, I could have fully restored my sites and those that I'm responsible for because I have everything that is on the servers in my office and backed up twice. The only portion of the site that could not be restored is active email and recent logs.

[dboh]

Ironic that their servers are named for characters in the series "Lost."

[sandbox]

Ana Lucia was interesting though her server was troublesome. I've never watched a full episode of Lost but I've seen some of the actors around the web.

QUOTE(dboh @ Jul 10 2009, 08:40 AM) <{POST_SNAPBACK}>

Ironic that their servers are named for characters in the series "Lost."

[Xairbusdriver]

Apparently, even more 'disturbances are expected today from whoever has been attacking many government and commercial sites and hosts. I have no idea if the problems here are related to that attack, but it should be obvious that there are still too many ways to find 'chinks in the armor' when it comes to having anything in the "cloud." Just further discourages me from relying on such a method for anything approaching my personal or financial data. I remained convinced that one cannot trust anyone with that kind of data. Digital or real. Period. As far as I'm concerned, it is simply a fact of life and I have to deal with it as best I can.

"There is a reason baskets are fairly inexpensive. It makes it easier to have many of them." A. Nonymus

[Paddy]

Jim, from what I've read, the attacks on government web sites etc. are denial of service attacks (DNS attacks) not hackers and viruses. DNS attacks will take you out while they're underway, but they don't require you to reinstall all your sites/server software etc. like ASO had to do.

http://news.postbulletin.com/newsmanager/t...20&a=407315

And then again...maybe there is some relation:

http://tech.yahoo.com/news/nm/20090710/wr_...uth_internet_22