Thursday, February 04, 2010

Frustrations

Every Monday the team gets together to review the previous week and plan the next. Dena, one of the key team members felt that there was a word from God: 'Hang in there' for this week. When we had come into work on Monday had found that there were notices on all the cars telling us there was going to be a power outage from 8:00 to 13:00 hours on Tuesday.

So because I was the duty person this week I came in early on Tuesday to turn off all the servers. The duty person is one of the three of us who receive SMS text messages telling us any problems with our servers. For the week we are on duty that person is the key person for all maintenance and the other two [hopefully] get an undisturbed week to work on other projects.

I started powering down all the servers at 7:45, and at 7:56 CLUNK... the power went off. Very unusual for Cyprus to be early. The final server was closing down and the battery backup system gave it enough power to close down neatly. Well, there was nothing to do so for the morning I went off to the boat to put a coat of paint on her.

I arrived back after lunch and the power was already on. So I powered up all the servers. One had a hard disk problem which I manually corrected, but the main Internet connection wasn't working. Diagnosed it with a colleague to be one of the Cisco routers that might or might not be working. This isn't like the router you have at home, its a much more complex rack mounted piece of equipment that when new was a couple of thousand dollars (we bought second hand through ebay). It's a complex unit - we have two of them, which we need to replace soon.

Cisco routers have a terminal connection that you can connect a 'dumb terminal' to and check their status. So tried that and found that the programs we used to emulate a dumb terminal didn't seem to work and so we were then scrabbling around to try and work out what was happening.

Peter arrived in an hour later and said it looked like that when the power had come back on there was a surge which had blown the power supply. We keep spares for some things and did have a spare Cisco power supply so we then had to take it out of the rack to change the power supply.

At that point we found a layout logic problem with the rack. All the cables from the switches to the patch panels went across the routers. This meant we had to remove all the cables in the bay before we could remove the router. This meant all the cables needed labeling so they went back in the right place.

Now detail is not my middle name so it took three attempts to label the cables correctly. We then unplugged them all, and changed the power supply and the router started up correctly. Sounds nice and easy... well... we also had to remove the rat droppings from a vermin attack last year that we had not seen before!

We put it back in the bay and some things started working. Well... most things started working. There had been about 40 cables to label and then plug back in the right place. I had misplugged some of them and mislabeled one of them! I told you detail was not my middle name.

Eventually between all three of us (me on duty and the other two who should be doing other things) we got the system back up and running and everything working again.

I say everything... come Wednesday and totally unconnected with this there was a problem with email on one of our servers in Germany and Peter had been working on this problem and thought it was a problem with the security certificate. So he ordered another one. Unfortunately he put a password into the certificate when there should be none, so when the certificate was authenticated and delivered it didn't work. So he tried (with me across it, as I am the authorized representative of the organisation) to get a replacement.

That failed, coming up with a 'security failure' - so then I am trying to phone South Africa to talk to the company that issues the security certificates and find out what is happening. No answer from the company and tried sending emails/response forms... still now answer. All the while this means our and other companies email is not working!

I'm frustrated. I hate computers. Peter and I ended up talking about the future. I realised I spend my time approximately the following way:

  • 80% Technical
  • 10% Writing proposals
  • 15% Organisational admin
  • 25% Partner interaction
  • 10% Media
Those of you astute will realise the total is 140%. Yes, that's right I spent more than a working week each week working! The bit I enjoy is the media. The bit I hate is the technical.  Of course, if I have too I shall continue this pattern, but I would really like to somehow get more time for media.

I try to get the technical to be a smooth operation - it's not smooth, we have more work than possible for the team - and that creates a burden for me.

So, why don't we just drop the technical? Or reduce it?

When I say technical, what I mean here is the system maintenance. Maintaining more than 10 servers as a platform to enable the media work to proceed. Many different organisations rely on these servers.

Pete and I looked at the different servers and realised that if we just cut back to our core media project we would still need to maintain almost all of the servers just for that one project. We would save very little time. And... the partner contributions from many different groups actually help us to run those servers which then creates the platform for the main media project we want to achieve. Sort of Catch 22.

Come Thursday and I am still 'Hanging in there', but frustrated.

I get an alert (I'm duty person remember) about the temperature in the server room. The air conditioning in the server room is not working. No problem... obviously didn't restart when the power went back on. Walk round to the server room and press the remote.

Nothing happens.

Oh, must be flat batteries on the remote, since we leave it on 24/7 and don't change it. Find another remote for the same type of air conditioner, test it to make sure it does work on the other air conditioner... take it to the server room and press the remote start.

Nothing happens.

Obviously something else blew up when the power went back on.

Frustrating.

When Dena (our administrator) is back in on Monday I shall have to get her to arrange an air conditioning technician to come and sort out the unit. This being Cyprus, who know how long that will take.

Anyway in all these frustrations we realised we had two needs:

  • A Finance Director
  • About size young people Media/IT literate 
With both those two needs met then I would be released to spend more time on the media. I'm not looking for 100%, coming up to 50% is what I would like. And I'd like to bring my working week down to about 120%.

    Thursday, October 22, 2009

    Mopping up operations

    No administrator means... when things go wrong, we fix it. Well... actually having an administrator means we fix many things anyway, but lots of things the administrator checks up on and sorts out before we get water everywhere.

    It's still pretty warm [34C] here in Cyprus and the water cooler is an important part of the office. So, when David came in and found it leaking everywhere it was not cool... well, the water was, but it was now all over the floor.

    But the good news is we have an administrator who will hopefully start next Monday. This is definitely and answer to prayer, and she has gifts in other areas too... she has worked professionally in photography and has an interest in art, so maybe there will be more pictures coming out of the office to show you what is happening.

    Tuesday, October 13, 2009

    Recession, web 2.0 and the way forward

    While I was in the UK I got the chance to visit the CEO of a very big group doing similar work to us, except their budget is approx 200 times bigger than ours. I like them a lot and there is a lot of synergy with them. Because we are in the non-profit area we act as partners rather than competitors.

    When I visited I was shown a new integrated social networking and media platform that they will launch in a few weeks. It's close to some of what we had talked about as a group early in the spring. They had the budget to execute it very well. Not that some of the projects we do aren't done well. There is a very creative project we are also about to launch within the next few weeks... but... overnight and today it set me thinking.

    This group are also involved with another very large project for audience interaction that I went to see in Australia at the beginning of the year. It's excellent, but very 'heavy weight'. We are working on a module to integrate SMS into that system right now. I have two programmers working on that for the next two weeks.

    Yesterday I had an email from the external partners co-ordinator for this project. He challenged us 'Given the rapid changes occurring in social media, this may need to be more a loosely aligned organism rather than a complex system to cope with these changes.' That is what really set me thinking. How and why are projects so difficult to achieve?

    We have not always succeeded by a long way, but I always try to be 'fast and light' rather than 'big and heavy' since the world is changing. My target is that if it is not achievable within 6 months then its probably too complex. The Desk Top Publishing [DTP] war is an example of what I mean:

    Originally there had been two DTP programs head to head: Aldus Pagemaker and QuarkXpress. By the mid 90s QuarkXpress had become the world leader with Aldus/Adobe Pagemaker a second runner. Both were big complex programs. By 1997 Quark was in version 4.0 and so slow was development it took till 2002 to release a 5.0 version.

    Adobe purchased Aldus and then released InDesign in 1999. InDesign was a small simple program with many, many plug-ins. So radical and stable was version 1.0 of InDesign Quark were forced to bring out a 4.1 version within months.

    I think that is the way of the future and allows rapid and evolving development. I think their model is right: a small simple core with plug-ins that can be upgraded easily and quickly to do the majority of the work.

    I'm looking at one of our projects. It has been excellent with very good audience feedback. It was that project that I mention that in the spring said we needed to integrate social networking into the core of the project. At the moment there is a large amount of interaction, but it still has a significant publishing aspect to the project.

    Globally we have two things happening simultaneously:
    • A recession leading to a post-recession
      I believe that early next year there will be a second dip, the upturn will be for the Christmas period only, and will then be approx 12 months before we see a real upturn from the recession.
    • Web 2.0
      Web 2.0 will permanently change the way we do things. Publishing is dead, long live self-publishing. Well, maybe that is a bit strong, but it is the direction of the future.
    One respected media consultant put it this way 'While traditional media isn't going away, you basically have two choices: Evolve, or disappear.'

    So that's the challenge I have been thinking about today. Chatting with Peter he mentioned that his brother, who is a software developer, said their company only does projects that are achievable within 4-6 weeks. That was exactly what I was thinking about: Light and fast. RAD. Rapid Application Development. And that's the model Adobe took with InDesign.

    We have to change - we have to develop a method of working that allows every project to be achieved in 4-6 weeks. Its a different way of thinking. RAD implies developing a prototype very quickly and then evolving it into the final product using customer feedback. So the specification is inevitably light, not tied down.

    This doesn't mean that it's flaky - let me quote the media consultant I have been reading again: 'Are People More Creative or Productive Working without Limits? No. Absolutely not. Boundaries matter. One of the biggest threats to not reaching your goal is working without limits.' For us the boundary needs to become time.

    When I worked for the BBC many, many years ago I used to work on TV news. The 6 o'clock news went out at 6 o'clock. Plus or minus no seconds! I remember well dubbing a film [yes, we shot film in those days] which was number three in the running order while number one in the running order was going out. At the same time I was listening to production talkback, and hit the rewind button on the telecine machine which would allow a few seconds grace by the time it had rewound ready for play out. The 6 o'clock news went out at 6 o'clock. Plus or minus no seconds!

    In so many ways we have lost the concept of deadlines. And computers have caused this. When I was dubbing that film I did the best job I could, then grabbed a few extra gramophone records with sound effects, ran to the studio and added the extra effects live as it went out. It was the best we could do and the audience would have believed all the sounds were real.

    But 'the best job I could' doesn't work for computer programs - they either work or they don't work. Looking nice but not working is useless. So we have become used to massive overruns and time slips to make it work.

    Somehow we need to change our approach and methods so that we can return to the concept of making something workable within a limited time frame and within a limited budget. But something that is nevertheless good enough that the audience accept it, enjoy it and interact with it. The combination of the recession and web 2.0 are forcing that upon us. We have to embrace it and 'evolve or disappear'.

    Wednesday, October 07, 2009

    October? Where has the summer gone?

    October already... where has the summer gone? It's difficult to catch up with a blog 2 months old, but I'll try.

    Normally we aim to spend approx 3 months doing security stuff for our servers per year. This involves upgrading the applications and looking at new ways of doing security better. Sometimes it might also involve rebuilding one or two servers completely. That we did over the new year, from November to February this year. It was more painful than we expected, but worthwhile.

    We were expecting a new colleague and so some of the extra things we hoped to do were put off... 'David can do that when he arrives...' Well, David has arrived and getting him sorted out has taken time too, and he is now taking some of the load, but... we had three security problems this year which meant that we didn't actually get much break from security and we are still doing it 10 months later!

    That's not good news for me as the technical stuff I don't really like. I'm hoping that this month will be the turning point from the technical and I can get back to some media work. We'll see. The big thing we are doing is integrating 'dual factor authentication' to our systems. Dual factor authentication is where you have two 'factors' to get into the system - this is something you know and something you have. Banks use this: Often you log in to an online bank with a username and password [something you know] and a pin number generated by a little keyfob device [something you have]. Our system is similar, but its a really huge job doing it.

    In between all this I have been working on the scripts and getting equipment for a pilot for a serious of short films we hope to make next year. If all goes well we will shoot the pilot this November. I'm really looking forward to this as media production is much more what I enjoy and get a kick out of.

    September, we went back to the UK for 3 weeks. It coincided with my son's 21st birthday and also allowed us both to meet friends and have some business meetings in the UK. It was good but tiring.

    Now, less than a week back we have two programmers out from the UK, working on new code to upgrade our SMS [mobile phone text message] system. One of the programmers has been out before - he developed the original version of this system - and with him is another programmer working on 'web services' to integrate the system.

    Web services is a strange, but very helpful new concept where servers go to and read web pages themselves to communicate with each other. In this case it will allow servers to send and receive text messages. Servers sending text messages themselves? Whatever for? Well... one use is for a server to send us a text message to tell us it is having a problem and please could we fix it!

    I still have an unbelievable amount of email to catch up with... I HATE EMAIL... well... no I don't really but it seems to be that the volume of communication has increased exponentially and I am having problems keeping up with it.

    So that's about it. Not the most exciting summer [you'll have to see my other blogs about other activities which were more interesting] but worthwhile and hopefully moving.

    Saturday, May 30, 2009

    Work expands to fill the time available?

    We now have an extra person working with us. He's a professional System Administrator from the USA. He's with another group, but on placement with us for two years, spending approx 80% of his time with us and 20% with the other group. His character is laid back and fits in pretty well with Pete and me. So far he has just started... but...

    We have had many attacks against our systems recently so have been involved in upgrading the security systems for all our servers. One of the primary ways of doing this is through what is called 'dual factor authentication'. OK, that's a horrid techie phrase, but what does it mean? Here's the problem:

    Normally we use a username and password to log into the systems, be that email or website admin or whatever. But if there is someone listening across what we are doing they can capture our user name and password and impersonate us and use it themselves. We have had this happen twice from one of our partner groups in Egypt.

    The solution is simple - change the password every time you log in! Sounds simple but complex to administrate in a way that works for normal people. So what we actually do is have the normal user name and password as an identifier plus a password that changes every time. That changing password is either displayed on a little dongle that the user caries or is something you plug into your computer to send the password automatically. The dongle you remove when you are not using it. This latter solution is what we have chosen.

    But now we have to get it all to work. This means integrating with all our various systems on 9 live servers. So although we have extra help, we think we need extra extra help. In fact, looking at what is happening worldwide we know that cyber-security will only get more complex. Indeed yesterday Barack Obama launched his cyber security plan. Acts of terror today, he said, could come "not only from a few extremists in suicide vests, but from a few key strokes of a computer - a weapon of mass disruption."

    Alongside this we are planning a pilot series filming some illustrative stories almost allegorical, but not quite. At the end of last year we had some funds available to purchase hardware, so have invested in a new camera system. We've not bought all the equipment needed yet for the series, but will have enough for filming a couple of pilots which we hope to do this summer.

    In between this, our administrator has left, so there is more admin to cope with. And I HATE COMPUTERS. Yesterday I tried to update our accounts package. It took ages and was complex and the problem I thought would be fixed with the upgrade [some of the statements were refusing to print] are still there. So I have to talk to technical support about that.