16 July 2005 Mine is a sad and familiar story. I was drowning in a deluge of spam (a.k.a. junk email), and it had become such a problem that email was fast becoming useless for me. Having to sort through and delete hundreds of spam emails per day was bad enough. Worse was the increasing frequency with which I was accidentally deleting legitimate email along with the spam.

There are various ways to fight this deluge of spam. The most common is to use a filter that tries to recognise and delete the spam (or, more usually, move it to a spam box for later perusal). This is quite effective. A small amount of spam will not be recognised as such (false-negatives), and will end up in your inbox anyway. But the amount of spam will usually be cut down to a manageable amount, rendering email usable again.

The problem with the filtering approach is the false-positives: legitimate mail that gets mis-identified as spam. Even if it's moved to a spam box rather than deleted, when you're searching through hundreds of spam emails you're almost certain to miss the one or two legitimate mails hiding amongst them, and you'll delete them along with the spam. (At least, that's what I found myself doing.)

A second approach is to use techniques such as domain blocking, real-time blacklists, and other methods of blocking whole groups of addresses known to send spam. But this is really just a variation on filtering (filters usually take the sender's address into account when deciding whether an email is spam or not, as well as the body).

I didn't want to run the risk of someone sending me an email, me deleting it accidentally, and them never knowing that I didn't received it. So I chose to use a third approach: white-list plus challenge/response (plus a number of other features of the impressive TMDA system).

The first part of this system is a `white-list' of email addresses belonging to people I know, or have exchanged email with in the past. Any email from an address on the white-list is delivered directly. The challenge/response part comes in when someone I don't know (maybe you) sends me an email. Your message is temporarily held, and you are automatically sent a `challenge' email, asking you to confirm your address. By simply replying to this challenge, your original message is released and delivered to me. (It really is that simple: you just have to hit "Reply" and "Send" in your email software.) What's more, your address is then added to the white-list, so forever afterwards you'll be able to send me email directly, without needing to confirm your address again.

Of course, if I'm the one that emailed you, it would be rude to challenge your reply! The TMDA system arranges for replies to bypass the challenge/response system, and be delivered directly.

Why does this work? For two reasons. Firstly, spammers almost never use a real email address. Email was invented in a more innocent age, and takes on trust whatever email address the sender supplies. So it's very easy for a spammer to lie about their email address. And they have every reason to, given that what they're doing is illegal in many countries. If a spammer spoofs their address, the challenge never reaches them, and the spam is never delivered.

Secondly, even on the rare occasions when a spammer does provide a real address^{11 1}In my experience, the only spammers who use a real address are the occasional `Nigerian 419 fraudster'. , they are sending out thousands of junk emails per day, many of them to addresses that don't actually exist. It would take far too much effort to reply to each challenge individually - or indeed, to notice the challenge emails amongst all the bounce-messages they'll be getting from non-existent addresses. Responding to challenges would defeat the whole raison d'être of spam: its cost-effectiveness.

Some people object to the challenge/response approach, saying they're not prepared to jump through hoops to send someone an email. My feeling is that the white-list plus challenge/response approach is the lesser of many evils when it comes to defeating spam. I think it's politer to at least let someone know their email won't be read (if they refuse to reply to the challenge), than to risk accidentally deleting it and leaving them unsure as to whether it has been read or not. If they're so annoyed by the challenge that they change their mind about emailing me, that's their prerogative. Remember, the only people who receive a challenge will be people who I've never heard of, and who have never emailed me before. Would you let a stranger into your house without first asking them to confirm who they are? Why should we expect to be able to put email into a stranger's private email inbox without first being asked to confirm that we're not a spammer?

Another criticism is that spammers frequently send email from forged addresses - real addresses that don't belong to them (remember, an email sender can supply whatever address they like). Challenges will then be sent to people who never sent me an email in the first place. This is a more serious criticism in my view. TMDA provides a number of features (too numerous to detail here - see the TMDA web site if you're interested) to help reduce the number of challenges that are sent out.

In particular, I do in fact use spam filters. Not so much to filter out all spam, but to help decide when to send a challenge and when not. Messages that are clearly identified as spam are not challenged. Challenges are only sent to messages that look like legitimate mail (but are from an address that's not in my white-list).

A few challenges will no doubt still be sent to forged addresses. But many mail server delivery failure messages get sent to the wrong address too, for the same reason. All things considered, I feel that a well designed challenge/response system is the best choice amongst all the many, imperfect anti-spam options available.