Critical but very boring techie stuff

This week has been filled with very boring but critical technical stuff: working with Peter on the roll-out for the new email upgrade and getting a version control system working. Plus of course all the normal stuff of going through accounts with David. Plus of course working with a programmer from Egypt who is over here for a month working on a project. Plus of course all the normal maintenance of the system. Plus communicating with our partners about what is going on.

We never get closer than about three months away from the top of our to-do list and mostly we are way further behind than that. One of our partners today suggested that we should stop doing anything else and just focus on our to-do list till its cleared.

This is a great idea in theory, but in practise would be a disaster. The reason is this: Although Peter and I can do all this boring techie stuff it doesn't fulfil us and we see it as boring techie stuff too. So, if we just focussed on that and never did anything interesting we would get so bored we would give up. So we have to try to balance boring with some interesting to keep us motivated. Which means the boring keeps piling up. And since we lost John last Autumn the to-d0 list is increasing rather than diminishing.

So what is this boring techie stuff: Email is still an essential tool for our people and for communicating with people who want to know more about Jesus. But... and here's the but... spam is killing it off. As I mentioned in a previous post up to 96% of all email is now spam. So, because we handle about 50,000 real emails a month we have to also deal with up to one-and-a-quarter million spam emails per month. Yes, you did read correctly that is about 1,250,000 junk emails to deal with! Per month. Actually is not quite that number because the 'real email' count includes a few junk emails that slip through the net.

The aim is to get rid of most of them before the end user sees them. I get approximately 1,300 emails per month, so if we didn't have a spam filter running on the spam percentages above I could get more than 16,000 emails per month. Say it took me 5 seconds per email to open it and check if it was spam and then delete it because it was spam, that would take me about 80,000 seconds or 22 hours of clicking, reading and deleting. I would get almost nothing done at all! As the filtering works fairly well I can spend less than 30 minutes per week clicking, reading and deleting spam. [As an aside our top user of the email system received over 6,700 emails this month - most users receive low hundreds of emails per month.]

Any improvement we can make gives me and the 300 other people more time to do something useful. The effective cost is horrendous. 300 people all wasting time. Amongst us we are probably losing the equivalent of about 50 man hours per week in clicking, reading and deleting spam. If someone were paid the minimum wage in the UK of £5.50 this would be costing over £14,000 per year. Alongside this is the cost of Peter, Alex and I implementing tools to try to reduce this to as low a level as possible. Spam is costing the world economy millions of millions of pounds per year. It's a total waste.

Anyhow... the upgrade we have implemented is called 'greylisting'. It's a clever idea based on the fact that almost every spammer is running a special program to deliver all their junk around the world as fast as possible and is not running a proper MTA. MTA stands for 'Mail Transfer Agent' and it's the program that runs on a server to handle email between users. MTAs are designed so that if they cannot deliver on first attempt they keep trying for a few days... but spammer programs don't do this.

So what greylisting does is this: When someone new tries to deliver to us our MTA responds with 'We have a temporary problem please try back in a few minutes' and we log the message as attempted delivery. A spammer program goes away and doesn't come back because its only aim is to deliver as many as possible as fast as possible... a few failures doesn't matter. An MTA on the other hand does try back in a few minutes and when it does so, we match it with the previous attempt and this time our MTA says 'OK, we'll take it this time' and we also log the MTA as valid so that next time it tries with another email from [hopefully] a valid user then we will accept it immediately. [There is also a special website all about greylisting if you are interested].

The greylisting system isn't perfect, but it does kill most spam and the only impact is to slightly delay email from valid users the very first time they try to send it. Alongside this there were two other upgrades called SPF and Domain Keys. Read the links if you want to know more. They make greylisting look positively simple to understand.

The second techie thing I was involved with is setting up a version control system. Version control systems allow us to have a repository for all the programs we or others write and all the configuration files we use. Every time we make a change to a file or program we update the version control system. It stores the changes and because it knows of the changes we make each time we can rewind what we have done and other members of staff can see what the changes were that were made between different versions. All sounds clever and is something we should have implemented a year or two ago, but have been too over-run with work to do so.

The one we will be using is called Subversion or SVN for short. Subversion is a pun on a tree of versions... The internet is full of horrid puns, which I suppose is the techie way of staying sane. For instance there is a robot program to use with Subversion called CIA. Why CIA? Because CIA monitors and informs on subversion .

That same partner [from paragraph 2] and I wondered about spam... we've never seen anything remotely helpful in any spam message so cannot understand the mentality of people who send them. But then my wife was explaining the other night about the forums for support for Google AdSense, where people were saying they had bought a site and put Google AdSense on it and couldn't understand why they were not making any money [hmmm... maybe because you have no content on the site?] . Now we know why people send spam, it's these same people who are clicking on the spam and expecting miraculous growth in certain parts of their anatomy. Pity its not growth in their brain cells.

One problem is that although there have been laws in existence for the last 5 years to protect us from spam [EC Directive 2002/58/EC], almost nobody manages to sue the spammer. This month only the second company in the UK was successfully sued for damages resulting from sending spam. And the amounts in both cases were piteously small: the first award was £300 and the second £1350. What we need is a significant number of spammers behind bars for many years to deter people from trying it on.

Ok, so light relief time: Found this wonderful spam cartoon on http://www.ChristianLinksExchange.com

Week of mixtures

Been a mixed week. We had meetings at the beginning of the week with one of our partners. They have appointed a new Internet manager and he spent two days with us going over things. It clarified in my mind how what we are doing is important, but we have maybe 20% of the manpower and resources needed to do the stuff. And its running us dry [Peter and I] trying to do it. One question came up "If you stop doing this, who would you recommend we go to?" The answer is that there is nobody who can take over what we are doing in some areas at all.

Alongside this we had one of the programmers we work with over from Egypt at the start of a one month visit. He is developing a second website with us and starting on a third. If these are as effective as the first one this will be tremendous. I wish I could describe what these sites will be about, but... hmmm.... sorry, cannot!

And alongside this Peter was working on the new upgrade to the email system and I have been helping with some of the database functions for that. We hope to make the upgrade live next week.

Other think I did was build an autocue unit. This is fully described with pictures in my equipment notes blog so I won't repeat it here.

Work and play...

The last week seems to have flown by. I completed the preview copy of the dub from Arabic to Farsi and the film is now being reviewed by one of the satellite TV stations to be put out over the air. Another friend working in Iran is very keen to have a copy as soon as possible.

In between completing the dub there has been a plethera of little things to do, discussing with Peter about the new upgrade to the email system, the SMS phone system and the failover system... [which was failing over when it shouldn't have been!] Actually the one that was most painful to sort out was the failover system.

So what is the 'failover system' I hear you ask. Basically because our systems have to work 24 hours a day 365 days a year we have two servers for most of the tasks that we need only one for and we have some clever software that says 'Use this server if you can, but it you cannot use this other one'. Sounds great
in practise but configuring this system is a tricky business and for some oddball reason it was switching over to the backup when it shouldn't. I looked through all the configurations and checked everything and it was just being 'ornery' as the Americans put it, which means it was not behaving as we wished.

The configurations were difficult to wade through as I needed to relate the configuration files on four separate servers. Eventually I decided to re-write all the configuration files and [in my view] make them more logical and easy to read. And... yes, so far it worked the failover system is now working 'as advertised'.

The SMS phone system has also been causing us grief. Ever since Christmas it has been getting increasingly unreliable. By that I mean we have had to restart one or more phones every 12 hours at least. We consulted with the manufacturers who said [like they all do] 'We don't have these sorts of problems normally...' and then in passing mentioned there was new firmware for the phones that 'might fix it'. We installed the new firmware and it does appear [so far] to have fixed it.

The email system upgrade is a major undertaking. There are two parts to it: firstly making our system more compatible with other systems and secondly reducing the spam coming in to the system. Peter has been working on it for nearly a month now. He has completed the first part and working on the second part which is far from trivial.

I spent most of Friday talking through with him and looking at the configuration for a system called 'greylisting'. Blacklisting is where you say 'never ever send me email from this person'. Whitelisting is where you say 'always let me have email from this person'. Greylisting is where you say 'I'm not sure about this person, check out that its a real person and not a spam robot sending it'. Greylisting works because real email servers will try again and again to deliver but spam robots try once and give up, so basically for greylisting to work you log the email and then say to the incoming server 'please try again later'. If they do its a real email ad we accept it, if they don't then we're sure its spam!

They say 'all work and no play makes Jack a dull boy' and in between all this work I have been over the last two to three weeks stripping the varnish off the boom, mast, gaff, rudder and centreboard of my little mirror dinghy and then revarnishing them and repairing parts of the mast where a friend had a little accident with the dinghy last year. Now I'm longing to sail her and hopefully next weekend if the weather is good I will be able to have a day sailing.

This afternoon I went for a wander to my favourite place in Larnaca... the marina, where I can drool over the larger boats and dream of the one I would love to buy. The marina is somewhat empty at the moment as many of the boats are 'on the hard' [ie dry land] being repaired by their owners. There is one thing that is certain about all types of boat... you are constantly repairing them.

One boat that I have been watching for the last 9 years is called Windsong registered in Littlehampton in the UK. Its a 44 foot long wood hull ketch and has been very sadly deteriorating steadily over the last decade, although the owner has done a small amount of repair work during that period. It looks a boat in desparate need of TLC. Her masts are at the state where you don't need to rub off the varnish, the sun and wind has done that already. The deck and cabin too have been at the sharp end of the elements.

So what makes Windsong so special? She is the only surviving David Hillyard sailing yacht to take part in 'Operation Dynamo' - and one of the very few pure sailboats. Operation Dynamo was the name given to the flotilla of 'little boats' that evacuated the British troops from Dunkirk. Which, of course, gives her the right to fly both red and white ensigns... and there is not another little boat up this end of the Mediterranean who has that right!

Built in 1931 it is interesting the sailing technology of the day. We don't see an autopilot at the stern to control the rudder but a wind vane. This keeps the boat at a constant direction relative to the wind rather than a constant direction. Of course the modern autopilots can do both and even tack the boat for you, but it's facinating to see how the old mechanical systems used to work. The mast has steps up it to make climbing easier rather than using a bosuns chair and a halyard. I would love to see inside and see how the cabin is laid out and the navigation was done.

I just hope this dear old boat is not allowed to deteriorate further and is restored to her former glory and actually enjoyed as a sailing boat again.

Pictures and sound

The ongoing saga of video editing... I am now trying to complete a dub of a film from Arabic to Farsi. We were getting on well and then the transfer from Windows to mac took place. Should be easy mac can read Windows files. Hmmm... never expect anything to be easy. In the process I found out that the Windows machine had been making corrupt files. Although this explains a lot [like why we were getting problems on the Windows machine] it means that I have had to start from the beginning with the picture edit for the dub.

'Picture edit? I thought this was a sound dub...' I hear you mutter. Yes it is, but the opening and closing captions have to be replaced with Farsi rather than Arabic, and the original Arabic pictures don't match the new audio. So I have to re-edit that.

Peter has been involved with upgrading the email system, which is taking a lot longer than he hoped. Its pretty complex keeping up with all the changes to email to try to defeat spam.

Alex has been relaying a big prayer conference this week. We had over 4,400 listeners connecting over 38,000 times over the 4 days to listen to it live and nearly 3,000 viewers connecting over 18,000 times to watch it [either live or pre-recorded].

Flew last week, flu this week!

It seems like yesterday I was back from traveling, but its actually been over a week!

Much of the time has been taken up developing a newsletter to send out to people so that they can read about what we do. I know there's this blog and one of the things I did [well actually a very dear friend, called my wife, did it for me] was to cut down a couple of extracts of the blog as a way of encouraging people to read it!

I went to the launch of a new streaming studio too... they also moved all the control for one of the 24/7 radio stations we had been handling to their own facilities. We will still be handling the streaming to listeners on the internet, on their behalf.In many ways the new studio is actually our success as we planted, watered, nurtured and are now seeing a plant in full bloom so to speak.

When I first suggested the idea of internet radio to them just over three years ago they were not interested. Within a few months they changed that to a cautious first step into internet radio with us developing all the systems needed and running the station 24 hours a day from our own centre here in Cyprus... and it developed and developed till last year when the person in charge admitted that he had been seeing the traditional radio audience decreasing , while the internet audience was rapidly increasing.

Last Autumn they decided to start a regular weekly live broadcast from our studio. The number of people wanting to chat with staff using MSN or Skype was so great, the computer handling them slowed to almost a complete stop.

There's a limit to how much we can achieve alone. Currently we are working beyond that limit. That's why it was so exciting this week to see one of our partners open their own studio for live Internet radio instead of us doing it all.

In between all this we are rebuilding the edit suite... or I should say John is rebuilding it. He has taken out all the old woodwork for the old tube based screens which were build in and created a working surface and panel for the audio meters etc needed. Its nearly complete now. Pete and I were remarking on how with people around helping it really makes a difference to what we can do and we are really really grateful for what John has done.

From Friday I had a terrible cold or a light dose of flu, or something in between. Anyhow I didn't feel great at all. I think I caught it while travelling, or maybe I had been working so hard my body was giving up. Anyhow I wasn't well over the weekend.

Today I was still not 100% so I kind of did a couple of things and put the equipment into the new edit suite. All the audio wiring still has to be done. We need to but a new computer for this [a macpro which will costs between £2500 and £3500 depending on the options we get]. Currently we're using a macmini but this isn't really powerful enough for the purpose, however its vastly more reliable than the old Windows machine was. Two more smaller parts are needed: a Behringer BCF2000 control surface and a Contour ShuttlePro. The last edit suite has lasted 5 or 6 years [with some modifications on the way] so this is the biggest rebuild and hopefully when we have everything will last us another 5-6 years.