Writing Mails from Rust (1/3): Mail in general

A short view into some of the inner workings of mail.

This is the first part in a three part blog post about (e-)mails and how to create, encode and send them using the mail crate (a library).


How a mail is sent

At a high level there are six steps:

  1. A user creates the content they want to send.
    • Normally done by the user’s mail program, e.g. using some web rich text editor or Thunderbird
  2. Content is converted to a mail
    • Usually by the aformentioned mail program
  3. The mail is sent to a Message Submission Agent (MSA).
    • The MSA is normally identified by a domain name, e.g., smtp.gmail.com.
    • Do not confuse this with the MX entry in the domain name registry.
    • This likely will use the Simple Mail Transfer Protocol (SMTP).
    • The MSA might slightly modify the mail, such as adding some signature to make it verifiable that the mail is actually sent by you.
  4. The mail is transferred to the receiver's Mail Exchanger (MX) which passes it to a Message Delivery Agent (MDA).
    • This might be done by a separate Mail Transfer Agent (MTA) which gets the mail from the MSA you sent it to.
    • The transfer from MTA to MX might also be done by SMTP, although there are other protocols.
    • Note: when sending a mail from two mailboxes of the same provider, such as a@1aim.com to b@1aim.com, this step may not happen in the classical sense.
    • In the past the mail might have gone through multiple hops, but that isn't really a thing anymore so it's not covered here.
  5. The user retrieves the mail from the MDA.
    • This is often done by IMAP or POP3 if you use a mail program.
  6. The mail program displays the mail.
    • This is anything but simple as many parts of mail are underspecified as to how to exactly display certain parts or even just semantically interpret them.

The mail crate is mainly focused on creating and encoding mails so that they then can be sent, through it also provides bindings to new-tokio-smtp to make it simple to send mails to an MSA. In the future it might also support parsing mails, but functionality such as displaying them or retrieving them from an MDA using IMAP/POP3 is outside of the scope of the crate.


The mail standard(s)

TL;DR: There are many interconnected standards, standards replacing standards and standards updating standards making it easy to overlook some parts or misinterpret others.

Mail is pretty old and there are a large number of standards which have to be considered when implementing a program to create and send mails. Many of the standards also have one or multiple new standards which "replace"/obsolete the previous standard.

For example mail was first specified by the IETF in RFC 822, which on itself actually replaces RFC 733 which is a standard for some pre-mail text messages. The problem of RFC 822 (and IMHO many mail related standards) is that many parts where either rather vague or allowed many more possibilities then originally intended. For example RFC 822 allows using control characters like XON, XOFF in text in (some) headers. Many of the standards which then replaced or obsoleted RFC 822 did clarify and further restrict parts by deprecating them, i.e. the grammars for mail are now split into a "normal" part and a "obsolete" part which you still have to be able to parse due to backward compatibility but should never generate.

While often it's enough to use the latest standard in a chain of standards obsoleting each other, it's often not as simple as there are many other standards extending the existing standards. These standards always refer to the standard which had been the "newest" when they were released, but were not updated when that standard had been obsoleted and replaced by a newer one. Furthermore they still apply to the newer standard, which is always meant to be backwards compatible with the older standard (with respect to mail at least). This might require you to trace some feature or grammatical construct through the standards to find out what it actually means in context of the newest standard.

Below is a list of may relevant standards for creating mails. Note that for most RFCs in the list there are one ore more additional standards updating it. This is especially true for the MIME (Multipurpose Internet Mail Extensions) related RFCs:

RFC Description
5322 Internet Message Format (aka. mail)
6532 Internationalized Mail Headers
2045 MIME Part One: Format of Internet Message Bodies
2046 MIME Part Two: Media Types
2047 MIME Part Three: Message Header Extensions for Non-ASCII Text
4289 MIME Part Four: Registration Procedures
6838 Media Type Specifications and Registration Procedures
2049 MIME Part Five: Conformance Criteria and Examples
2183 Extends MIME, adds the Content-Disposition header
2231 Extends MIME, adds Encoded Words to support non-US-ASCII text in headers

The next table is a list of some RFCs related to sending mails using the Simple Mail Transfer Protocol (SMTP).

RFC Description
5321 Simple Mail Transfer Protocol
6531 SMTP Extension for Internationalized Mails
6152 SMTP extension, adding 8BITMIME
3207 SMTP extension, add transport layer encryption with STARTTLS command


A simple mail

A mail based only on the mail standard consists of a number of headers followed by a blank line followed by a single body. Both headers and the body can only contain US-ASCII characters (i.e., 7-bit ASCII) and have a soft line length limit of 78 characters and a hard line length limit of 998 characters excluding the end of line sequence which is specified to be CRLF ("\r\n"). So neither attachments, HTML mails, embedded images nor non US-ASCII characters are part of the core standard.

Most of these features get added through the MIME standard(s) which adds support for having multiple mail bodies (or whole mails) inside the main mail body. Additionally it adds headers like Content-Type which allows specifying what kind of data the body contains (e.g., text/html) and Content-Transfer-Encoding which allows to encode the bodies with base64/quoted-printable encoding allowing to include arbitrary non-US-ASCII data like e.g. images. Additionally MIME defines encoded words which can be used to have non us-ascii characters in mail headers, e.g., in the Subject header. Lastly there are the standards around internationalisation which allow you to directly use UTF-8 in most places but is not necessary always supported by the receivers mail provider.

The problem with this is that you often have many ways how to handle certain things (like UTF-8 in headers) but all of them tend to have some drawback. For example with SMTP servers normally supporting the 8BITMIME extension you can directly use non-US-ASCII bodies without needing transfer encoding, but as the line length limit still applies and lines are still broken with \r\n this doesn't really work for binary data and can even be a problem for UTF-8 data if your mail program cannot simply insert \r\n. E.g. if you have UTF-8 encoded JSON data you can only insert \r\n between fields and in some languages it's not trivial to detect word boundaries. This leaves a situation where it's almost always better to use base64. The quoted-printable encoding can also be a good option, but is a bad idea if the send text is not mostly US-ASCII characters as it can increase the length of a mail body by threefold in the worst case scenarios, which for example, someone writing in Arabic script would be the common case!

Below is a simple example mail, followed by an explanation of it:

MIME-Version: 1.0
From: <person1@example.com>
To: Person Two <person2@example.com>
Subject: Happy New Year =?UTF-8?B?8J+OiQ==?=
Reply-To: No Reply <no-reply@example.com>
Date: Tue,  8 Jan 2019 16:26:50 +0000
Message-Id: <worldunique1@example.com>
Content-Type: multipart/mixed; boundary="=_^0"

--=_^0
Content-Id: <worldunique2@example.com>
Content-Type: text/plain; charset=utf-8

Hy there, it's the image.
--=_^0
Content-Disposition: attachment; filename=the-image.png;
 modification-date="Mon,  7 Jan 2019 15:14:16 +0000";
 read-date="Tue,  8 Jan 2019 16:26:40 +0000"
Content-Id: <worldunique3@example.com>
Content-Transfer-Encoding: base64
Content-Type: image/png

TG9yZW0gaXBzdW0gZG9sb3Igc2l0IGFtZXQsIGNvbnNlY3RldHVyIGFkaXBpc2NpbmcgZWxpdC4gRn
[...]
VzY2UgdmFyaXVzIGxvYm9ydGlzIGludGVyZHVtLiBEb25lYyB1bHRyaWNpZXMgc2VtcGVyIGxlY3R1
--=_^0--

The mail above is relatively simple and contains no strange syntax. But a few things should be taken note of: The From/To headers can contain multiple mailboxes. Each consists of an optional display name and a mail address surrounded by <>. The display name is a phrase which consists of one or more words which might be whole quoted strings, or just normal words. Additionally if it's not an internationalised mail and the display name contains a non-US-ASCII UTF-8 character it needs to put the character (or the whole word it's in) into an encoded word. Furthermore like most headers mailboxes allow placing comments in many places and comments can by themselves contain comments.

Both of following headers are semantically the same:

In the second variant Max was quoted (unnecessarily), then a comment was added which contained another comment, following by using a quoted-string like encoding of the UTF-8 text Musterman. Lastly another comment is added into the mail address (foobar).

While this parts of the grammar are less relevant for generating mails as there is normally no need to produce such mails some parts like quoting and encoding need to be handled as the user will normally provide the display name as a simple UTF-8 string and the library has to check if it needs to quote or encode words, and if so, which ranges of characters from the input it should choose to quote/encode. E.g. encoding a whole word should be preferred over encoding of a single character. On the other hand it still should be displayed as one text in the end so it doesn't necessarily matter.

Media Types

Media types, also sometimes know as Mime-Types, are the things apearing as values in Content-Type headers of a MIME body in a mail, or even in HTTP. It just happened that while the specifications of media types for mail and HTTP is mostly the same it has some small differences.

Common media types include text/plain; charset=utf-8 or image/png. They specify what kind of data it is and some parameters which need to be known to display the data (e.g. which encoding (charset) is used for text).

Here it should be noted that these media types just specify what kind of data it is but not how to handle it. E.g. text/html; charset=utf-8 defines that it is an HTML document but it doesn't mean that your mail program will display it as such, or if it does, that it will display it correctly or support all possible HTML element. Technically, it's possible to send a complete website/web-game including non-inline JS and CSS with a mail, but likely it won't be displayed "correctly" by your mail program. Some of these restrictions are pure technical others are to prevent social engineering, protect from viruses or protect your privacy.

RFC 6838 does a good job at specifying a fairly constrained grammar for media types and a how certain parts of it should semantically be interpreted. Sadly this RFC is useless when writing your programs as it only specify what newly registered media types should comply to. But even when just looking at registered media types they don't necessarily comply with the semantic constraints and when using a program "in the wild" having to handle non-registered media types is possible. RFC 6838 also doesn't constrain the problems/annoyances some of the things which can be done with media types parameters in mail can cause.

For example RFC 2231 extends media types to allow UTF-8 text in parameter values outside of internationalised mails (by percent encoding them) but also adds a way to split any parameter into multiple parts which can also be encoded but do not all need to be encoded but if ... Let's cut it short here it's basically a mess to be able to comply with the line length limit for long parameters like e.g. file names and it became worse through a likely unintended interaction with RFC 6532 (International Mail Headers).

Multipart bodies: Attachments & Embeddings

Like mentioned before MIME allows placing multiple bodies inside of a body of a mail. Each of this bodies (including the "container" bodies) do have their own headers and mainly differ in that they have a media type as Content-Type which starts with multipart/ as well as a boundary parameter in that media type.

As a body inside a body can further contain additional bodies in a recursive manner this creates some form of tree of bodies. By combining different kinds of multipart media types different effects can be achieved like e.g. attachments.

Commonly used multipart media types are:

So if you want to create a mail which contains an embedded image, HTML text, alternate plain text and a PDF file as attachment you would have a MIME tree roughly like:

multipart/mixed
╠═multipart/alternate
║ ╠═text/plain
║ ╚═multipart/related
║   ╠═text/html
║   ╚═image/png
╚═application/pdf

Note that you should set the Content-Disposition: attachment header in the application/pdf body and the Content-Disposition: inline in the image/png body. Content-Disposition was added with RFC 2183 and helps the mail program to display the mail in the way it was intended to be displayed. It additionally contains a number of parameters including file-name and read-date which is especially useful for attachments. Ironically the grammar of the parameters are defined to be "the same as media type parameters". Which means they can have all the annoyances like inconsistent UTF-8 encoding.

While the above tree of bodies is the "common" way to handle embedded content and attachments there are little constraints in place about how bodies can be combined. It is just that the more unusual the structure of the mail is the less likely it will be displayed in a "nice" way (there is no correct way anyway). E.g. you could place the multipart/alternate as the outer most body and then place the multipart/mixed where above the text/plain is placed and the text/plain and application/pdf bodies inside of it. Which theoretically would create a mail which has an attachment if you view it as plain text but not if you view it as HTML, though it's unlikely to be displayed in this way by any mail program.


SMTP

The Simple Mail Transfer Protocol (SMTP) is a common way to send a mail to a server. I will not go deeply into SMTP but there are a few things which should be noted: