Drop #210 (2023-03-01): You've Got Mail

JMAP; Stalwart; mox

I reckon most of us likely do not give much thought to a service we all use every single day. We are content…—No. Scratch that. We are resolved to the fact that we have become vassals to an endless stimulus–response loop. It all begins with a rounded corner, rectangular notification and an increment of a tiny numerical bubble indicator letting us know we have yet-another email to inspect, address or ignore.

Today, the HTTP protocol rules the day, but in times long past, email protocols were the hotness.

Jonathan Postel's SMTP (Simple Mail Transfer Protocol) RFC dates back to 1982. J. K. Reynolds' POP (Post Office Protocol) followed closely after in 1984. And, IMAP (Internet Mail Access Protocol) burst on to the scene in 1986 (though did not receive a proper RFC until version 2 of the protocol in 1988).

Since then, rather than truly address all the shortcomings — including confidentiality, authenticity, integrity, and complexity — technologists have just layered on cruft, after cruft, after cruft til we arrive in the state we are in today.

If you don't believe my take that email is just one big hot mess, head over to your Google Mail or Microsoft 365 inbox. (Sadly, we've centralized this service as daftly as we have so many others.), Then, open up any given message and “Show Original”. One message I picked at random had 54 lines of email header content (the bits that have metadata about the message, including how it got to you) before the real content even started.

And, if you question my “centralization” posit, consider that there were 1.5M MX (mail exchange) servers discovered in 2008 and only just over 650K today. Sure, some of that reduction is due to load balancers and beefier single machines, but the truth of the matter is that a handful of email service providers dominate the space.

We can and should be better than this.

So, today we take a look at how we might break these email chains. We’ll start with a review of modern protocols for email, a popular implementation of those protocols, and a way to channel the energy of the fediverse and get back to our self-hosting roots.

---

JMAP

Photo by Jac Alexandru on Unsplash

There was a time when one needed at least passing knowledge of the aforementioned protocols to use email. Unless they’re setting up an alternative client to a self-hosted or less popular/more traditional mail server, it is unlikely most humans today have never seen the ALL CAPS acronyms that power the way we send, receive, and consume email messages.

SMTP is how traditional email servers talk to each other, and POP/IMAP are common client-server protocols in traditional email setups. If you use the official Google or Microsoft mail clients for their respective platforms, that is not necessarily the case (when you lock folks in to your ecosystem, you get to change the rules and idioms).

POP and IMAP are horrible protocols. They're semi-structured, chatty, difficult to parse, require the use of shims to support interoperability with related services such as calendar-ing; and, they are some of the biggest reasons you likely despise your modern email client experience (b/c form follows function).

JMAP (JSON Meta Application Protocol) hopes to change all that, but you've likely never heard of it, either. Fastmail introduced JMAP back in 2014, but it did not become an IETF RFC until a few years later.

Since this is Fastmail’s pride-and-joy, I shall let them explain:

JMAP is friendly to servers. A stateless protocol, there’s no need for the server to maintain a version of the mailbox view that’s out of sync with the current state, as IMAP does, just so that clients can use integer offsets to refer to messages.

JMAP is friendly to the network. Unlike IMAP which can send an unbounded number of unsolicited updates, in JMAP you explicitly ask for the fields you are interested in, and the update command can contain a limit — if there are too many changes, return an error and let the client load the view from scratch. If you’re not caching the entire mailbox locally, then re-fetching a few pages of index listing is better than getting 100,000 EXPUNGED lines down the wire.

JMAP is friendly to multiple clients. In IMAP, if you rename a folder or move messages, then the client which initiated the action can update their local cache, but every other client has to re-fetch everything – there is no unique message ID which follows the message. JMAP uses globally unique IDs for messages, so immutable data can be cached forever.

JMAP does everything. Instead of separate protocols with different syntax for sending email than for receiving it (and separate protocols again for contacts and calendars) JMAP combines them all under one protocol. Best of all, the push notification includes all tokens for each service, so you can synchronise changes to your entire mail account plus your contacts and your calendar in a single round trip — up to date immediately, the way it should be.

So, yes, yet again, we're abusing HTTP with one more purpose-built RPC rather than think a bit more out of the box, but at least JMAP takes a more holistic approach to overhauling the email client/server paradigm.

You can explore everything about JMAP in a very friendly and accessible way at jmap.io (branding is everything these days).

If a “hands-on” approach is more your liking, then make your way to the middle section.

Stalwart

Photo by pineapple L on Unsplash

Stalwart [GH] is a “scalable, secure and robust open-source mail server software designed for the 21st century”. Their tools are designed around JMAP, is highly scalable, and is built on Rust and RocksDB, so it has some safety and speed advantages over offerings. Best of all, it's open source, and has tons of batteries included:

• JMAP full compliance:

  • JMAP Core (RFC 8620)

  • JMAP Mail (RFC 8621)

  • JMAP over WebSocket (RFC 8887)

  • JMAP for Sieve Scripts (DRAFT-SIEVE-12).

• IMAP4 full compliance + extensions:

  • IMAP4rev2 (RFC 9051)

  • IMAP4rev1 (RFC 3501)

• Flexible and robust message storage:

  • Sieve Script message filtering with support for all extensions.

  • Full-text search support available in 17 languages.

  • Local Mail Transfer Protocol (LMTP) message ingestion.

• Secure:

  • OAuth 2.0 authorization code and device authorization flows.

  • Domain Keys Identified Mail (DKIM) message signing.

  • Access Control Lists (ACLs).

  • Rate limiting.

• Scalable and fault-tolerant:

  • Node autodiscovery and failure detection over gossip protocol.

  • Replication and cluster consensus over the Raft protocol.

  • Read-only replicas.

  • No third-party replication or cluster coordination software required.

They do a fantastic job explaining things, but you can also dive in head first and poke at their client and server implementations.

mox

Photo by Davide Baraldi on Unsplash

If you're more of a Golang fan, then you might be interested in Mechiel Lukkien's mox, a “modern full-featured open source secure mail server for low-maintenance self-hosted email”. (Why do so many of us, including me, feel the need to add the word “modern” to every description?)

It, too, has a number of batteries included:

• Quick and easy to start/maintain mail server, for your domain(s).

• SMTP (with extensions) for receiving and submitting email.

• IMAP4 (with extensions) for giving email clients access to email.

• Automatic TLS with ACME, for use with Let's Encrypt and other CA's.

• SPF, verifying that a remote host is allowed to sent email for a domain.

• DKIM, verifying that a message is signed by the claimed sender domain, and for signing emails sent by mox for others to verify.

• DMARC, for enforcing SPF/DKIM policies set by domains. Incoming DMARC aggregate reports are analyzed.

• Reputation tracking, learning (per user) host- and domain-based reputation from (Non-)Junk email.

• Bayesian spam filtering that learns (per user) from (Non-)Junk email.

• Slowing down senders with no/low reputation or questionable email content (similar to greylisting). Rejected emails are stored in a mailbox called Rejects for a short period, helping with misclassified legitimate synchronous signup/login/transactional emails.

• Internationalized email, with unicode names in domains and usernames (“localparts”).

• TLSRPT, parsing reports about TLS usage and issues.

• MTA-STS, for ensuring TLS is used whenever it is required. Both serving of policies, and tracking and applying policies of remote servers.

• Web admin interface that helps you set up your domains and accounts (instructions to create DNS records, configure SPF/DKIM/DMARC/TLSRPT/MTA-STS), for status information, managing accounts/domains, and modifying the configuration file.

• Autodiscovery (with SRV records, Microsoft-style and Thunderbird-style) for easy account setup (though not many clients support it).

• Prometheus metrics and structured logging for operational insight.

You might find mox easier to get up and running, too. Just fire up a VM or VPS dedicated for email, name it host.domain (e.g., mail.example.com), create a mox user+homedir, download the single binary to that homedir and run:

./mox quickstart you@example.com

This creates an account, generates a password and configuration files, prints the DNS records you need to manually create and prints commands to start mox and optionally install mox as a service.

mox's spam filtering gives it a slight edge over Stalwart (IMO), but there are other options to explore for self-hosting your email that we'll cover in future editions.

FIN

Running an email server may sound daunting, but the more we can all do to crawl back from centralized services, will help make the internet more robust and resilient. ☮