When 4th best is good enough?

I can't believe its so long since I wrote on this blog, I have posted a couple of items on my other blogs, but basically not had time to blog for a couple of weeks or more. Pete and I are reviewing what we are doing. We are running about 220 component projects for various people and having problems maintaining it all. Running Internet servers grows more complex by the day.

Last week one of our partners from Spain flew over and we had a couple of days with him and one day with another partner too thinking through what we do. One phrase struck me that he said, 'Many times I am doing my fourth best option...' More and more we are faced with the least bad option rather than the best. Obviously choosing between a good and a bad option is a no-brainer. Choosing between two equally good options is easy, but we find choosing between two equally bad options very difficult. There is a phrase I remember from when I grew up 'Heads you win, tails I lose'. When its like that it just feels... well... strange.

Because we are so short of staff we are faced with making cuts in some of those 220 component projects. Which ones? Any cut will affect somebody seriously. Some people may not hear the message if we do cut. Another thing this guy from Spain said was that on the questionnaires he has to complete for any project he does within his organization there is a significant question that he has to answer for his bosses: 'What will happen if this project is not done?' I puzzled and puzzled over this.

Something about the question concerned me. Was it because we were faced with a similar one 'What will happen if this project is cut?' or something deeper than that. The more I puzzled the more I could not see it till the Friday evening. On Friday evening a bunch of folk who all follow Jesus come round to our house. We have a meal together, pray for each other and then look at the Bible together.

Anyhow, on that Friday evening we were looking at the book of Acts and about Peter and a Roman Centurian. The question came up "What if Peter had said 'No I'm not going to visit the Centurian'?" The answer we came to is that God would have found another way to communicate His love to the Centurian. Ahh... here's why I was so perturbed. The re is one answer to the question 'What will happen if this project is not done?' and its the same in every case. The answer is 'God will find another way to communicate His love to those He wants to'.

But that doesn't actually make the decision about what to cut any easier. It could mean 'Does anything I do matter if God will find another way to communicate?' Pete and I came back to what we feel God is calling us to. This guy from Spain again... he said we need to come back to our mission statement. We knew and could easily state what our 'unique selling points' [don't you just love marketeer speak?] are, but a mission statement?

Every company and organization has their mission statement today. McDonalds mission statements starts off 'McDonald's vision is to be the world's best quick service restaurant experience. Being the best means providing outstanding quality, service, cleanliness and value, so that we make every customer in every restaurant smile...' or how about 'Establish Starbucks as the premier purveyor of the finest coffee in the world while maintaining our uncompromising principles while we grow...' Trouble is that reading too many mission statements makes me want to vomit.

Anyhow, we did eventually get a mission statement and core values. We have also created a questionnaire that will help us evaluate all the projects we are involved with. Now all we have to do is persuade our partners to fill in the questionnaire... maybe we should offer a prize. Any suggestions on what the prize should be? Leave a comment...

Email working and spam being killed

At last, we sigh with relief the main parts of the email upgrade are working and spam seems to have been reduced dramatically. A very boring week of testing, checking, compiling programs and configuring.

I was getting about 40-50% spam and now in the last two days [since we switched on the extra spam checking] I have had only one spam email. It seems so strange to me, why people are bothering sending spam, it really is a constant battle and more and more people are killing more and more spam emails... and so they have to send more and more and more and more...

Today I spent most of the day writing a couple of small [but complex] scripts to create all the SPF and Domain Key records for our DNS system. Time was when DNS was simple. Now it grows in complexity and reading the files gets more complicated. And looking at things I can see it getting more complicated by the month. Oh well...

Just looking at part of DNS that we are not using yet: Service calls. Instead of typing http://www.blogger.com your system would ask blogger.com what http services it supports and sent you to the right machine. Complex but useful for the way the internet will be growing in the future.

Our Egyptian programmer is approximately half way through his time here and the messaging system is working functionally, but with some glitches. Hopefully we will go into testing mode next week. This is the core of a new website which will be about community interaction.

What makes the messaging system different is that it works like a threaded discussion forum, but with the primary key being the person you are interacting with. This is because the focus is on relationship building.

This weekend is Easter and I will take a couple of days off and hopefully go sailing.

Email gets more complicated by the minute

OK, so we have the email upgrade working but... we do a lot of email forwarding and the new SPF and Domain Keys don't handle forwarding well. It appears that I have to violate one RFC [RFC's are the Internet 'rules for how things work'] in order to get other RFCs to work.

Oh well, lets watch it over the next few days and see what happens. It's now past midnight and I kept working to try and get everything sorted so email wasn't breaking.

Blogging from my phone

OK, so its possible to blog from a mobile phone, but does it work for me? If this post is on my blog then it does!

Critical but very boring techie stuff

This week has been filled with very boring but critical technical stuff: working with Peter on the roll-out for the new email upgrade and getting a version control system working. Plus of course all the normal stuff of going through accounts with David. Plus of course working with a programmer from Egypt who is over here for a month working on a project. Plus of course all the normal maintenance of the system. Plus communicating with our partners about what is going on.

We never get closer than about three months away from the top of our to-do list and mostly we are way further behind than that. One of our partners today suggested that we should stop doing anything else and just focus on our to-do list till its cleared.

This is a great idea in theory, but in practise would be a disaster. The reason is this: Although Peter and I can do all this boring techie stuff it doesn't fulfil us and we see it as boring techie stuff too. So, if we just focussed on that and never did anything interesting we would get so bored we would give up. So we have to try to balance boring with some interesting to keep us motivated. Which means the boring keeps piling up. And since we lost John last Autumn the to-d0 list is increasing rather than diminishing.

So what is this boring techie stuff: Email is still an essential tool for our people and for communicating with people who want to know more about Jesus. But... and here's the but... spam is killing it off. As I mentioned in a previous post up to 96% of all email is now spam. So, because we handle about 50,000 real emails a month we have to also deal with up to one-and-a-quarter million spam emails per month. Yes, you did read correctly that is about 1,250,000 junk emails to deal with! Per month. Actually is not quite that number because the 'real email' count includes a few junk emails that slip through the net.

The aim is to get rid of most of them before the end user sees them. I get approximately 1,300 emails per month, so if we didn't have a spam filter running on the spam percentages above I could get more than 16,000 emails per month. Say it took me 5 seconds per email to open it and check if it was spam and then delete it because it was spam, that would take me about 80,000 seconds or 22 hours of clicking, reading and deleting. I would get almost nothing done at all! As the filtering works fairly well I can spend less than 30 minutes per week clicking, reading and deleting spam. [As an aside our top user of the email system received over 6,700 emails this month - most users receive low hundreds of emails per month.]

Any improvement we can make gives me and the 300 other people more time to do something useful. The effective cost is horrendous. 300 people all wasting time. Amongst us we are probably losing the equivalent of about 50 man hours per week in clicking, reading and deleting spam. If someone were paid the minimum wage in the UK of £5.50 this would be costing over £14,000 per year. Alongside this is the cost of Peter, Alex and I implementing tools to try to reduce this to as low a level as possible. Spam is costing the world economy millions of millions of pounds per year. It's a total waste.

Anyhow... the upgrade we have implemented is called 'greylisting'. It's a clever idea based on the fact that almost every spammer is running a special program to deliver all their junk around the world as fast as possible and is not running a proper MTA. MTA stands for 'Mail Transfer Agent' and it's the program that runs on a server to handle email between users. MTAs are designed so that if they cannot deliver on first attempt they keep trying for a few days... but spammer programs don't do this.

So what greylisting does is this: When someone new tries to deliver to us our MTA responds with 'We have a temporary problem please try back in a few minutes' and we log the message as attempted delivery. A spammer program goes away and doesn't come back because its only aim is to deliver as many as possible as fast as possible... a few failures doesn't matter. An MTA on the other hand does try back in a few minutes and when it does so, we match it with the previous attempt and this time our MTA says 'OK, we'll take it this time' and we also log the MTA as valid so that next time it tries with another email from [hopefully] a valid user then we will accept it immediately. [There is also a special website all about greylisting if you are interested].

The greylisting system isn't perfect, but it does kill most spam and the only impact is to slightly delay email from valid users the very first time they try to send it. Alongside this there were two other upgrades called SPF and Domain Keys. Read the links if you want to know more. They make greylisting look positively simple to understand.

The second techie thing I was involved with is setting up a version control system. Version control systems allow us to have a repository for all the programs we or others write and all the configuration files we use. Every time we make a change to a file or program we update the version control system. It stores the changes and because it knows of the changes we make each time we can rewind what we have done and other members of staff can see what the changes were that were made between different versions. All sounds clever and is something we should have implemented a year or two ago, but have been too over-run with work to do so.

The one we will be using is called Subversion or SVN for short. Subversion is a pun on a tree of versions... The internet is full of horrid puns, which I suppose is the techie way of staying sane. For instance there is a robot program to use with Subversion called CIA. Why CIA? Because CIA monitors and informs on subversion .

That same partner [from paragraph 2] and I wondered about spam... we've never seen anything remotely helpful in any spam message so cannot understand the mentality of people who send them. But then my wife was explaining the other night about the forums for support for Google AdSense, where people were saying they had bought a site and put Google AdSense on it and couldn't understand why they were not making any money [hmmm... maybe because you have no content on the site?] . Now we know why people send spam, it's these same people who are clicking on the spam and expecting miraculous growth in certain parts of their anatomy. Pity its not growth in their brain cells.

One problem is that although there have been laws in existence for the last 5 years to protect us from spam [EC Directive 2002/58/EC], almost nobody manages to sue the spammer. This month only the second company in the UK was successfully sued for damages resulting from sending spam. And the amounts in both cases were piteously small: the first award was £300 and the second £1350. What we need is a significant number of spammers behind bars for many years to deter people from trying it on.

Ok, so light relief time: Found this wonderful spam cartoon on http://www.ChristianLinksExchange.com