Tuesday, February 03, 2009

Out of the bubble...

Years ago I had a friend who as a psychologist who used to work among media people in Southern California. One of the phenomena he talked about which rang bells with me was the 'bubble effect' among media people. When they were working on a project they went into a bubble and were not like normal human beings, emerging as [relatively] normal people from the bubble when the project was over.

As a result of the attacks we had in October 2008 we decided to upgrade two of the servers. These servers would be three years old this January. They have a life span of approximately three years before needing an upgrade. Each time we upgrade, it's not just a case of new hardware and put in the CDROM and type install but we have to think through of the needs, particularly security needs, for the entire life of the server. So, Peter and I thought through what we felt we needed for the next three years.

The last time we did an upgrade it took Peter and I approx one month - ie two man months of labour. This time we then brought a colleague over from Egypt and between us we hoped that in one month we could get the new hardware and software working to replace the old. We expected that three man months would be about right... but it wasn't... it has taken approx nine man months of labour to do the upgrade!

Why so much more? We were adding many extra security features which proved very much more complex than we expected. In fact, the added security was somewhat frightening for me. I was thinking back over the steps to get here... from the original servers... to the upgraded servers... to these new servers. The complexity seems exponential. The upgraded servers were about twice as complex as the original, and the new ones about four times as complex as the upgraded ones. Hence I'm already getting edgy about what it will be like in time years time... sixteen times as complex as these new servers?

The main security issue is for each application within the server to be isolated from every other application, running in a virtual server with its own security. That way an attack on one part should not affect the whole. So in reality it's like going from two servers with eight primary applications to having eight servers with one major application each. But they cannot be totally isolated and we then had to work out secure ways that each application could talk to the others that they needed to. Yipes... yes, horribly complex and hence why I was concerned about the future.

Over Christmas we had another colleague and his wife over so that he could have extra training and to plan together the next step of the project he is working on. So... having been in the upgrade bubble and not completely out of it, it was straight into another bubble. Not that it was bad, but it did mean we didn't get a break at all.

We also had end of year calculations to do and create budgets and plans for 2009. Actually doing this took Pete and me out of the upgrade bubble for a while and did enable us to see the 'wood for the trees' which was helpful. But budget planning is not one of my favourite pastimes. Just before the end of the year Peter remarked, 'You know, I wouldn't do this if they paid me...' I had put into works exactly my thoughts!

Then we discovered that for various reasons we had to upgrade one of the other servers that is only one year old. We lease the hardware, so strangely enough because of the drop in price of hardware the new server will be under 2/3 the price of the old one! We had to wait for delivery of the hardware which was handed over to us yesterday. Yes, that means we are still in the upgrade bubble.

One Egyptian colleague is still working with us on the upgrade process, which we hope to complete by the end of February. It should be quicker now on the extra server as we know roughly what we want and can copy the two we already have working.

So does that mean its all straightforward for a while? No, not entirely... tomorrow Peter is off to Holland for five days to attend a conference and then next Wednesday I fly off to Australia for about a month, partly to attend a conference, partly to evaluate some new software and to see if a partnership with an Australian group will happen and partly to visit other organisations and... but... its not coming together easily...

A couple of days after I had finally booked my tickets I heard that there will be a delay on the new software, which means I shall probably have to go back to Australia sometime later in the year. It's both a huge expense and a huge cost in my time. I am not best pleased to put it mildly. There is only one light at the end of the tunnel as far as the trip is concerned. If everything works out I shall see my son for a couple of days on the way back through Manila.

So there we are... it almost seems like we cannot get out of the bubble, but the bubble is expanding to keep us inside it!