The engineering challenge behind the history of email

01The scaling problem nobody planned for

When SMTP was designed in 1982, the ARPANET had a few thousand users. The protocol was built to handle a few hundred messages per day between a few dozen servers. Nobody anticipated that email would grow to 320 billion messages per day across millions of servers. The scaling problem was not just about volume. It was about the number of participants, the diversity of systems, and the range of use cases. The original SMTP had no authentication, no encryption, no spam filtering, and no mechanism for handling multimedia content. Every one of these features was added later, bolted onto a protocol that was never designed to carry the load it now carries.

The scaling problem manifested in several ways. Server queues that were designed to hold a few messages now hold millions. DNS lookups that were once fast and reliable are now a bottleneck. Message stores that were designed for text now hold gigabytes of attachments. And the spam problem, which was not anticipated at all, consumes the majority of the engineering effort in modern email systems. The remarkable thing is not that email has problems at scale; it is that it works at all. The protocol has been stretched, extended, and patched for over forty years, and it still functions. This is a testament to the original design principle: keep the core simple, and let everything else be added as extensions.

02Authentication: proving who you are

The original SMTP had no authentication. Any server could connect to any other server and claim to be sending mail from anyone. This was fine in 1982, when the ARPANET was a small, trusted community of researchers. It became a catastrophe when the internet grew to include billions of users and millions of servers. Spammers exploited the lack of authentication to forge sender addresses, making it impossible to trace malicious email back to its source. The email community spent over a decade building authentication systems on top of SMTP: SPF (Sender Policy Framework) to verify that the sending server was authorized to send mail for a domain, DKIM (DomainKeys Identified Mail) to cryptographically sign messages, and DMARC (Domain-based Message Authentication, Reporting, and Conformance) to tell receiving servers what to do when authentication fails.

Each of these systems was added after the fact, as an extension to a protocol that was never designed to support authentication. SPF was published in 2006, DKIM in 2007, and DMARC in 2012, thirty years after SMTP was defined. The delay between the problem and the solution meant that forged email was a problem for over twenty years before it was addressed. The engineering challenge was not just technical; it was political. Adding authentication to an open system required consensus among thousands of email providers, and it required backward compatibility with millions of existing servers. The solution had to be opt-in, not mandatory, because forcing authentication would break email for users whose servers had not yet implemented it. The result is a layered authentication system that is not mandatory, not universal, and not perfect, but that nonetheless catches the majority of forged email.

03The MIME extension: when text was not enough

For the first ten years of its existence, email could only carry text. Specifically, it could only carry ASCII text: the 128 characters that make up the basic English character set. No accents, no other alphabets, no images, no attachments. If you wanted to send a file, you had to encode it as text using a tool like uuencode, paste the encoded text into the message body, and hope the recipient could decode it. This was cumbersome, error-prone, and limited to small files. MIME, the Multipurpose Internet Mail Extensions, solved this. Published in 1992 as RFC 2045 through 2049, MIME defined a way to include non-text content in email messages: images, audio, video, application files, and text in any character encoding.

MIME is one of the most important extensions in the history of email, and it is almost invisible. Every email with an attachment uses MIME. Every email with HTML formatting uses MIME. Every email with non-ASCII characters uses MIME. The engineering challenge MIME solved was encoding binary data in a text-only protocol. SMTP was designed to carry text, and it could not be changed without breaking every email server in the world. MIME worked within this constraint by encoding binary data as text using base64, a scheme that converts binary bytes into ASCII characters that SMTP could carry. The solution was not elegant, but it was backward-compatible, and backward compatibility is the highest law of internet engineering. MIME succeeded because it extended SMTP without breaking it.

04The spam arms race

The spam problem is the single largest engineering challenge in the history of email. At its peak in 2008, spam accounted for over 90 percent of all email traffic. The spam arms race has been going on for over thirty years, and it shows no sign of ending. Spammers develop new techniques; engineers develop new filters; spammers adapt. The first filters were simple keyword matches. Spammers learned to vary their wording. The next filters used statistical analysis of word frequencies. Spammers learned to obfuscate their text. The current generation of filters uses machine learning and AI to classify messages based on content, sender behavior, network patterns, and user feedback. Spammers respond with AI-generated content that is designed to evade AI filters.

The engineering effort required to fight spam is enormous. Google, Microsoft, and Yahoo each employ hundreds of engineers whose sole job is to improve spam detection. The filters process billions of messages per day, classify them in milliseconds, and update their models continuously. The cost of this effort is invisible to users, who simply do not see the spam that is filtered. But the cost is real, and it is growing. The spam arms race is an example of a problem that cannot be solved by engineering alone. It requires policy, legal action, and international cooperation. The engineers who built email did not anticipate that their system would become a battlefield, but that is what it has become, and the battle consumes more resources than the system itself.

05Encryption: the gap between design and deployment

Email was designed without encryption. Messages were sent in plaintext, readable by any server that handled them. This was not a problem in 1982, when the network was small and trusted. It became a problem when email started carrying sensitive information: passwords, financial data, personal correspondence, business contracts. TLS encryption was added to SMTP in 1999 as RFC 2487, allowing servers to encrypt messages in transit. But TLS was optional, not mandatory, and many servers did not implement it. As of 2025, over 90 percent of SMTP connections are encrypted, but the remaining unencrypted connections are a vulnerability.

The gap between the availability of encryption and its universal deployment illustrates a fundamental engineering challenge: you cannot mandate changes in a decentralized system. There is no central authority that can require all email servers to encrypt. Each server operator makes their own decision, and the system can only encourage, not enforce. End-to-end encryption, which would protect messages from being read by the servers that handle them, has never been widely deployed in email. PGP and S/MIME provide end-to-end encryption, but they require users to manage keys, and most users do not. The result is that email is encrypted in transit between servers but not end-to-end. Your email provider can read your email. This is not a technical limitation; it is a deployment challenge. The technology exists. The challenge is getting it adopted by billions of users across millions of servers.

06The challenge of backward compatibility

Every engineering decision in email is constrained by backward compatibility. There are billions of email clients and millions of email servers in the world, and they all speak SMTP. Any change to the protocol must work with existing clients and servers, or it will break email for some users. This constraint has shaped every extension to email. MIME was designed to work with existing SMTP servers. TLS was designed to fall back to plaintext if the other server did not support encryption. SPF, DKIM, and DMARC were designed to be optional, so that servers that did not implement them would not break. Every new feature has to coexist with every existing system, and this coexistence requirement limits what can be changed.

Backward compatibility is both the strength and the weakness of email. It is the strength because it means email works everywhere. A twenty-year-old email client can still send mail to a brand-new server. A brand-new client can still receive mail from a twenty-year-old server. No other communication technology has this level of compatibility. It is the weakness because it means email cannot be fundamentally redesigned. The protocol that was designed in 1982 is still the protocol we use today, with extensions bolted on. If someone wanted to design email from scratch today, they would do many things differently. But they cannot, because the existing system has billions of users and millions of servers, and you cannot redesign a system that large without breaking it. The engineering challenge of email is the challenge of evolving a system that cannot be replaced.

07The unfinished system

Email is an unfinished system. It has been under continuous development for over fifty years, and it will never be finished. New challenges emerge constantly: new types of spam, new security vulnerabilities, new user expectations, new regulatory requirements. The engineers who maintain email systems are engaged in a never-ending process of adaptation. The protocol they are adapting was designed for a world that no longer exists, but it is the protocol they have, and they must make it work in the world that does. The engineering challenge of email is not a problem that will be solved. It is a process that will continue for as long as email exists.

The lesson of this engineering history is that infrastructure is never done. The systems that connect people, whether email, the web, or the telephone network, are living systems that must adapt continuously to changing conditions. The engineers who build these systems do not solve problems once and for all. They solve them for now, knowing that new problems will emerge, and that the solutions they develop today will become the constraints of tomorrow. Email is the greatest example of this principle. It is a system that was designed in one era, extended in another, and is still being adapted in a third. It works not because it was designed perfectly, but because it was designed to be adaptable. And adaptability, not perfection, is what makes infrastructure last.