The average user gives little thought to email. It is the most common application of the internet and we just expect it to work. It’s simple really, type in a email address and quick note, click send and your message is magically delivered. Unfortunately the truth is radically different. Email is one of the most cumbersome, most abused, and increasingly complex technologies that the internet has ever produced. Email in its current state is hopelessly broken and yet it is the 800lb Gorilla that everyone pretends isn’t in the room. It’s time to reexamine our folly and to discuss how to solve this monstrosity.
Email is still very much the frontline in an decade old game of cat and mouse. Back in July I explained that the inherent problem with email is that it is based on a protocol (SMTP) that was developed in the stone age of the internet. (Email – A House of Cards). That post covers the history and problems with SMTP so I won’t revisit that aspect of the problem. However, I want address the abuse of email known as Spam and the retrofit countermeasures that we continue to rely on.
Countermeasures – The tangled web we weave.
Initially, we simply configured our mail servers to not relay other organizations email. Then we had to implement Antispam software that used metrics and monitoring systems to score email as to the probability it was spam and take actions based on the message’s resulting score. Then we began to subscribe to “Real-Time Blacklists” (RBLs) which were maintained by third-party organizations that blacklisted certain IP addresses as being spammers. In the past few years we’ve taken a stricter approach to DNS configuration and specifically implementing Reverse DNS so that recipient mail servers could verify the originating party. The most recent machination involves Sender Policy Framework (SPF) records to specify which IP addresses are authorized to send mail on our behalf.
Each approach offered a better quality of service but in turn creates an additional maze that must be navigated to send a simple email.
As an example of the circus that has been created by email you only have to look at the industry that has been built to “solve” the email issue. There are literally hundreds of Antispam software solutions and just as many appliance solutions. Not to mention the myriad of email scrubbing services and email hosting services. Name any major technology vendor and I guarantee they offer an “Email Security” or Antispam product, but here is the nasty little truth, they don’t want to solve the email problem they want to help you live with it. There is simply too much money to be lost if they actually solved the problem.
Hello, my name is ______?
What is the problem? Identity Verification and Authentication. That ultimate problem of an electronic communication. How do I know that you are who you say you are? How do I prove to you that I am who I say I am?
This is not a new issue. The best minds of the internet ecosystem have been wrestling with this problem with years. We’ve see technologies like PGP encryption and OpenID begin to tackle small bytes of this issue but email is its own animal mostly due to that archaic underlying protocol of SMTP.
How do we solve it?
Think back to this past summers amazing DNS hack and aftermath. (For the uninformed or short-term memory loss sufferers check our Wired’s “Secret Geek A-Team Hacks Back, Defends Worldwide Web” article or Dan Kaminsky’s post “An Astonishing Collaboration”). To over simplify, Dan found a major DNS flaw that was consistent among almost all vendors. To solve the issue he reached out and was able to bring all the parties to the table to discuss the problem and cooperate on a fix. It was truly a watershed event as they all worked to patch the problem and release those patches in the same day.
This event lingered in the back of my mind until today when I had a minor epiphany. Why can’t we do the same with SMTP?
I wasn’t able to find current statistics of email server market share so I’m going to make some assumptions. I’d wagre that Microsoft Exchange and Lotus Notes comprise the lion’s share of corporate email systems while many other smaller organizations rely on an Open Source Linux solution for their email. Therefore, why can’t Microsoft, IBM, and a handful of other relevant parties collaborate to create a replacement to the SMTP protocol.
The main obstacle to an overhaul of SMTP has always been the issue of adoption. In laymen terms SMTP is simply used everywhere, how would you ever replace it. Here is the roadmap.
The major vendors and relevant parties form a consortium to create a protocol to replace SMTP. Then once the protocol has been co developed and ratified they begin to rollout their latest software with support for the new protocol while still supporting SMTP. Within 3-5 years many organizations would have servers that supported the new protocol and then you could begin to phase out SMTP. Realistically it would take 10 years to accomplish but Rome was not built in a day. (Look at the timeframe for IPv6 adoption.)
Imagine the coup for Microsoft and IBM to jointly announce such an endeavor and to showcase their latest software and the benefits of the new email protocol.I refuse to believe that this could not be made reality.
So the answer to the initial question, “Is there room for innovation in email?”. Absolutely. Even more so the community demands it.
