Back from Oz

Here is update on what is happening...

Peter and I are somewhat frustrated at the moment.

1) We are getting technical problems that interfere with going forward... Peter is trying to upgrade the security of all our systems, which involves putting in a new system called IPSEC. The server concerned keeps crashing - we suspect hardware even though its only 12 month old. Because he is testing for incoming connections, this stops incoming connections working, including our phone system!

2) This is only part of the security upgrade which we need to do this year. We had already planned this, but as a result of Peter going to ICCM in Holland earlier this year it has become a bigger priority. He met with folk from another similar group and in discussions with them security became a larger priority than it already was.

3) Coming back from Australia I am still feeling I am catching up. Australia has left me feeling I would be happy not to get on another plane for quite a few years - but expect I will have to - I really need to visit a couple of countries in North Africa soon. Australia was good - meeting with another group that is very similar to us and maybe have some projects together. However the main purpose was to meet with a group who have developed a replacement audience follow-up system which we need to replace pretty soon. It was good meeting with them, but concerning in that IF we partner with them and a group others here in the Middle East to roll out the system in Arabic it will mean committing one of our two programmers for a few years to the project.

4) The economic issues are affecting our programmers - one of our programmers is facing the challenge of trying to get a second job paying equal too or more than we pay to try to support his retired parents. There is no adequate pension in the Middle East! This means two things - firstly he will be less available to travel and secondly he aims to be working about 16 hours per day to get the money he needs. Its a nightmare really for people in this economic crisis.

5) I bought a new camera for the office in Brisbane [cheaper than UK or Hong Kong strangely] and have been getting all the extra bits needed [some on ebay] to make it do what we want since returning from Australia. I look forward to making the short pilot films we have planned for this year. I hope I don't get sidetracked by all this boring techie stuff... I am finding all the techie stuff wearing me down.

6) On a personal note, last week I sprained my arm moving the boat [I still feel 18 inside even though I'm slightly older than that] and didn't rest it and was climbing over the lighting grid at the youth theatre, and moving a table to the office guest flat and doing more lighting and then sanding down the seats for the boat... silly really... but left me with a VERY painful arm on Saturday - almost too much pain to sleep. It is getting better now, but I must rest it. Rest is not a word I really understand.

That's about it

Email... I hate it

I was recently chatting online with a colleague from Egypt and mentioned that I had spent the trip out to a series of meetings in Australia trying to catch up with my email. Well, I've been here 3 weeks and I have just spent the evening filing some of the emails that have arrived and been dealt with since I have been here. I am beginning to hate email... and flying but that's another story...

Anyhow, he mentioned that he couldn't find a suitable way to file his email into folders. I'm sure mine is far from perfect, but Mac Mail has a great search function. Anyhow he asked me what my system was - it is firstly person name for those I deal with quite a lot, then alternatively company/organisation name for those companies that I deal with partly anonymously and partly when I don't have relationship with a person so might forget their name and finally I have some general topic folders for emails that don't fit into either of the other two categories.

But it is so big and so tiring dealing with emails. OK, so yes the emails to and from my family when I'm traveling are not tiring to deal with I like them... but the rest. I currently have about 450 individual person folders for people I deal with fairly regularly, 180 company folders and 40 general folders.

One of the things we deal with at the office is spam... creating methods of reducing it. Currently over 95% of the emails we receive at our server is spam. We have various methods that we employ to reduce that, so we get almost no spam into our email boxes. However the quantity of spam is increasing and increasing... and it looked like it might get to the stage where the world could not cope with the quantity of spam circulating and email would cease to function. Sometimes in my heart of hearts I privately hope this might happen... then at least I would not have to deal with them all!

Out of the bubble...

Years ago I had a friend who as a psychologist who used to work among media people in Southern California. One of the phenomena he talked about which rang bells with me was the 'bubble effect' among media people. When they were working on a project they went into a bubble and were not like normal human beings, emerging as [relatively] normal people from the bubble when the project was over.

As a result of the attacks we had in October 2008 we decided to upgrade two of the servers. These servers would be three years old this January. They have a life span of approximately three years before needing an upgrade. Each time we upgrade, it's not just a case of new hardware and put in the CDROM and type install but we have to think through of the needs, particularly security needs, for the entire life of the server. So, Peter and I thought through what we felt we needed for the next three years.

The last time we did an upgrade it took Peter and I approx one month - ie two man months of labour. This time we then brought a colleague over from Egypt and between us we hoped that in one month we could get the new hardware and software working to replace the old. We expected that three man months would be about right... but it wasn't... it has taken approx nine man months of labour to do the upgrade!

Why so much more? We were adding many extra security features which proved very much more complex than we expected. In fact, the added security was somewhat frightening for me. I was thinking back over the steps to get here... from the original servers... to the upgraded servers... to these new servers. The complexity seems exponential. The upgraded servers were about twice as complex as the original, and the new ones about four times as complex as the upgraded ones. Hence I'm already getting edgy about what it will be like in time years time... sixteen times as complex as these new servers?

The main security issue is for each application within the server to be isolated from every other application, running in a virtual server with its own security. That way an attack on one part should not affect the whole. So in reality it's like going from two servers with eight primary applications to having eight servers with one major application each. But they cannot be totally isolated and we then had to work out secure ways that each application could talk to the others that they needed to. Yipes... yes, horribly complex and hence why I was concerned about the future.

Over Christmas we had another colleague and his wife over so that he could have extra training and to plan together the next step of the project he is working on. So... having been in the upgrade bubble and not completely out of it, it was straight into another bubble. Not that it was bad, but it did mean we didn't get a break at all.

We also had end of year calculations to do and create budgets and plans for 2009. Actually doing this took Pete and me out of the upgrade bubble for a while and did enable us to see the 'wood for the trees' which was helpful. But budget planning is not one of my favourite pastimes. Just before the end of the year Peter remarked, 'You know, I wouldn't do this if they paid me...' I had put into works exactly my thoughts!

Then we discovered that for various reasons we had to upgrade one of the other servers that is only one year old. We lease the hardware, so strangely enough because of the drop in price of hardware the new server will be under 2/3 the price of the old one! We had to wait for delivery of the hardware which was handed over to us yesterday. Yes, that means we are still in the upgrade bubble.

One Egyptian colleague is still working with us on the upgrade process, which we hope to complete by the end of February. It should be quicker now on the extra server as we know roughly what we want and can copy the two we already have working.

So does that mean its all straightforward for a while? No, not entirely... tomorrow Peter is off to Holland for five days to attend a conference and then next Wednesday I fly off to Australia for about a month, partly to attend a conference, partly to evaluate some new software and to see if a partnership with an Australian group will happen and partly to visit other organisations and... but... its not coming together easily...

A couple of days after I had finally booked my tickets I heard that there will be a delay on the new software, which means I shall probably have to go back to Australia sometime later in the year. It's both a huge expense and a huge cost in my time. I am not best pleased to put it mildly. There is only one light at the end of the tunnel as far as the trip is concerned. If everything works out I shall see my son for a couple of days on the way back through Manila.

So there we are... it almost seems like we cannot get out of the bubble, but the bubble is expanding to keep us inside it!

Most men, so I'm told, are glued to the TV when either the football or Olympics is being shown. Not me. I'm happily fairly oblivious to either. But now... it's the Volvo Ocean... and why do I mention it? Well, my team is currently in the lead. And, to make things even better, they have just beaten a world record.

Torben Grael and the crew of Ericsson 4 swept into the history books yesterday as the first monohull to breach the 600-mile barrier in 24 hours. They’ve been chased by men, machines and the elements in the last 48 hours – and nothing has touched them.

They had been lying fourth behind but battling it with the leaders - Green Dragon, Puma and Telephonica Black. But its pretty dreadful weather they are sailing through as Mark Chisnell puts it:

In their foaming, boiling, 25-knot wake the fleet lies scattered as the devil and the deep blue sea picked off the hindmost one by one – the cold front sweeping over them with a mix of murderous squalls and ugly waves in a pitch black night. We’re almost down to the last man standing.

If you're as gripped as I am you can follow the race online, even through a 3D virtual simulator, where the boat's instruments, when they are working, relay everything via satellite to your computer at home... almost in real time. But they don't always work. In fact, Ericsson 4 have equipment failure now.

So back to reality for me... over the past couple of weeks we have been battling murderous squalls on the technical front. Three weeks ago I wrote about the DDoS attack. One of the outcomes of reviewing this was a decision to upgrade two or more of the servers. They are three years old now and so replacing them is about due. But its not just a case of copy the files and off you go... it will take about three of us at least a month to move everything over and upgrade all the systems on the new servers. A very big job, which is why we only try to do it every three years!

Having decided to do this we brought Raed over from Egypt to help and then ordered the new hardware. We lease the servers rather than buy them, leaving the leasing company responsible for the hardware maintenance. On Monday they will pass them over to us, with a bare operating system on them and we will start the task of checking them and installing all the systems and moving the sites across.

In between all this the attacks have continued - like a cold front sweeping over us. We watch the attackers in real time, and have defense mechanisms set up to rebuff them. But trying to second guess their moves is difficult, so we have set up what is called a 'honey trap' to try and lure them in to showing their methods. This will give us some indication of how much they know about us and why certain sites are more attacked than others.

One of our partners - with a site for central Asia - was online chatting with me today and they want to increase the facilities, to start online broadcasting to their region. Another site - for the Middle East - will have new facilities and a new design before the new year. A further new site - also for the Middle East - should be live before the new year. So it feels like a 'foaming, boiling, 25-knot' race downwind barely in control of what is happening. I am looking forward to Christmas - which I hope will be the end of this leg of our race and the sites and new servers will all be behind me.

DOS attack

Most of day yesterday we suffered what was called a 'Distributed Denial of Service' or DDoS attack. This meant that web sites on one server were unavailable at times. The problem will have shown itself as either the server appearing to run slowly, or unavailable or problems within the website that looked like a MySQL problem.

So what is a DDoS attack? Well in our case all of these were caused by a whole load of computers sending invalid file requests many times per second - or at their slowest many many times per minute. What this did was to start extra instances of the web server to respond to these requests, till the server ran out of resources and failed to deliver. Normally the 'load of computers' are Windows computers with viruses [usually called a botnet] that allow them to be controlled from a master computer or robot system. All automated. Against us.

Peter eventually wrote a new rule into our automated response system to stop this happening by blocking users who try the same method of attack. Within seconds they were being blocked.

Fortunately it was a relatively minor attack. We recorded only 59 computers attacking us from the time we turned on the rule in the automated response system to block them. Today this has dropped to a trickle of 26 still attacking us in the first 8 hours of the day - all being blocked. Some botnets are huge - for instance, this August the Dutch police shut down a botnet of approximately 100,000 [Windows] computers infected and controlled by two people.

Oh, the the problem on Wednesday turned out to be a faulty cable. How come a faulty cable did all that? Well, the switch connecting to a workstation in the office, which, by the way, was turned off, sensed something strange on the cable and decided to keep trying to sort it out many thousands or millions of times per second. It also decided to tell the entire LAN about the problem [a broadcast message] again many thousands or millions of times per second. This broadcast message affected other switches and affected the server. Cable fixed, fault disappeared!

In case you're thinking that sounds rather like the DoS attack we suffered, it was. It was a type of DoS attack. The difference being that one is accidentaly, but from the evidence in the logs we can see the other was malicious.