Tuesday, November 29, 2005

Server attacked... and repaired.

Early last week our biggest server was attacked and an intruder go into it [not physically, but basically took over control of it] and then started using it to attack other computers and servers in London and around the world. The people we connect to in London received complaints about this and disconnected it from the Internet.

When someone has got in that deeply the only thing you can do really is to wipe the hard drive and re-install everything. Fortuneately we had a backup that was only 8 hours old [we do daily backups]... but re-installing is definitely non-trivial. What made it worse was the following morning one of my colleagues went to Dubai and another was teaching in another country, leaving me to sort everything out. Although before he did leave we talked through the re-build and decided to do the security upgrade we were planning to do soon.

That security upgrade was pretty complex. I won't bore you with the details, but basically it took me about 18-20 hours per day for about 5 days to get things back to where they were. However, I must admit that the pain was worth it and now with my colleague back we can see the fruits of that labour.

What did surprise us was just how many people were using our facilities. We have about 250 users and over 400 email accounts on the server. Of course, that means that was the number of people disturbed by that one hacker.

No comments: