Writing Mails from Rust (3/3): Example usage with explanations

Using the mail crate to create mails from handlebars templates

After the previous two posts this post will go step by step through how the mail crate can be used to create mails based on a handlebars template and send them to a Mail Submission Agent (MSA).

The previous blog posts can be found here:

The full example can be found here. This blog post not only explains what happens there but why it's that way and sometimes how you can do it differently. In following it will frequently have code snippets from that example repository.

The Context

TL;DR: Use mail_core::default_impl::simple_context::new.

Main functions in the mail crate are generic over a context. This context provides some general functionality not specific to the mail you currently send, but to all mails. Theoretically it would have been possible to create a shared global or thread local context, but this makes unit tests hard and can cause problems in less common setups (but if you don't care you still can simply put it into a lazy_static so that you don't have to carry it around).

At the point of writing the context has the following structure:

pub trait Context: Debug + Clone + Send + Sync + 'static {

    fn generate_message_id(&self) -> MessageId;
    fn generate_content_id(&self) -> ContentId;

    fn offload<F>(&self, fut: F) -> SendBoxFuture<F::Item, F::Error>
        where F: Future + Send + 'static,
              F::Item: Send + 'static,
              F::Error: Send + 'static;

    fn offload_fn<FN, I>(&self, func: FN ) -> SendBoxFuture<I::Item, I::Error>
        where FN: FnOnce() -> I + Send + 'static,
              I: IntoFuture + 'static,
              I::Future: Send + 'static,
              I::Item: Send + 'static,
              I::Error: Send + 'static
    {
        self.offload(future::lazy(func))
    }

    fn load_resource(&self, source: &Source)
        -> SendBoxFuture<MaybeEncData, ResourceLoadingError>;

    fn load_transfer_encoded_resource(&self, resource: &Resource)
        -> SendBoxFuture<EncData, ResourceLoadingError>
    {
        default_impl_for_load_transfer_encoded_resource(self, resource)
    }

}

The trait requires Debug + Clone + Send + Sync + 'static for usability as a context should be "cheap" to clone and easy to pass around, i. e. it should be something like a Arc<InnerContext> internally.

Logically it can be split in three parts:

  1. generate_message_id/generate_message_id: These functions are used to generate IDs, the problem is that such IDs need to be world unique and as such the implementation of ID generators is slightly more tricky (more about this below).

  2. offload/offload_fn: These functions provide access to a thread pool, which will be used to offload computation intensive work from the main tasks, so that the tokio event loop is not blocked. They don't really matter if you don't use it with tokio but instead use Future::wait(), but if you want to send them using the provided SMTP functionality it does. This might go away when we switch to a newer version of the futures crate.

  3. load_resource/load_transfer_encoded_resource: These functions are used to load a resource (e.g. an embedded image or attachment) and potentially transfer encode it. The default implementation for Context only supports loading resources from disk, but other implementations can provides ways to load resources from other sources, e.g. a document database and might add better caching schemes. For example a very often embedded logo.png could be stored/loaded already base64 encoded (though this would be a performance optimization irrelevant for many simple setups).

    The special thing here is that Resource can provide a resource not only as "data" or transfer encoded "data" but also as a Source which is an IRI (roughly a URI with UTF-8 support). The is especially useful for mail templates which can specify resources through a relative path (using a not so standard path: IRI scheme, e.g. path:./misc/logo.png) and allows custom schemas (e.g. my_doc_db:pictures/logo.png) at least with custom Context implementations.

Many simple use-cases won't need any of this, so a default context implementation is provided, also all parts used to create that default implementation are exported so that if you write a custom implementation you don't have to rewrite these parts.

A instance of the default implementation can be created using the simple_context::new function:

pub fn new(
    domain: Domain,
    unique_part: SoftAsciiString
) -> Result<Context, ContextSetupError>

This will crate a context using a simple thread pool, handling path: schemes for resources which (in contrast to file:) can be relative to the current working dir. The required parameters are used for the message/content ID generation: Firstly the ID is namespaced by a domain name. This is not specific to the implementation but common practice for mails. Normally a domain you control is used which makes sure "no one else" generates IDs which can conflict with your IDs. Which means we still have to make sure we don't produce conflicts ourself. For a single instance we can also easily make sure that there is no conflict by having a shared monotonic increasing atomic counter. Now to make sure multiple instances of the software can't generate conflicts you have to pass some "unique" part/identifier or similar to them (which isn't reused, the generated IDs should be world unique for past, present and future). If that is too troublesome for you, you could instead pass in a "random enough" ASCII string. For example a UUIDv4 would be more then enough to make a collision unlikely enough that it can be ignored. Just be warned that iff there is a collision with mail or content IDs it can totally mess up some mail programs which rely on it one way or another. Lastly for privacy reason we hash the non-domain part as else people could guess way to easy how many mail we did send.

So a possible setup could be:

extern crate uuid; // features=["v4"]
extern crate mail;

use uuid::Uuid;
use mail::{
    // Note that that context is the concrete context type
    // returned by `simple_context::new`, not the context trait.
    default_impl::simple_context::{self, Context}
};

fn partial_random_context() -> Context {
    // This should be retrieved from a configuration e.g. passed in
    // as a command line argument
    let domain = "mail.crate.example.com".parse().unwrap();

    // You could use `rand` and generate a long enough ascii string that way.
    // Or better have a actually guaranteed unique id for each instance using this crate.
    let unique_part = Uuid::new_v4().to_string().parse().unwrap();

    // This can only fail if:
    // - The domain name has some problems with `"Puny"` encoding (only matters for non us-ascii domain names).
    // - There is no current working directory (e.g. because the dir the program was started in was deleted since then).
    simple_context::new(domain, unique_part).unwrap()
}

Handlebars templates

Template configuration

Mail handlebars templates can include multiple handlebars templates (as a mail can have multiple alternate bodies) as well as a number of embedded or attached resources. So we first need to create a TOML config which specifies this.

For example a template.toml:

name = "email_hallo_world"
subject = "Greeting of {target}."

[[bodies]]
path = "plain_body.hbs"
media_type = "text/plain; charset=utf-8"

[[bodies]]
path = "html_body.hbs"
media_type = "text/html; charset=utf-8"

  [bodies.embeddings]
  logo = "./logo.png"

This specifies a template with the name "email_hallo_world". The name is used to refer to it later on in the program. Then there is a subject field which is interpreted as a small handlebars template. Following this we have a bodies array (the start of each element is marked with [[bodies]] which basically means "now a new element for the bodies array).

Each body needs a way to specify its template source, which can be done with the path field which is a path relative to a set base dir. Which the functions used for loading a template will automatically set to be relative to the directory the template file is in. Additionally you specify the media type (currently this can only be omitted for bodies, which path ends in "html" or "txt" and will always default to charset=utf-8).

In the above example there is an additional embedded image, note that it's grouped with the first second alternate bodies, by using [embeddings] instead of [bodies.embeddings] you can group it with "all" bodies, too. But if you only want to use the image in one of the bodies grouping it with the body is better style, and less likely to cause problems with mail programs if one of the alternate body types is not know by the mail program. Note that the order of the alternate bodies is the same as in the mail, i.e. the body you want to display most is the last body and the one you want to display least (i.e. the fallback) is the first body.

Embedding tables always map names to a way to specify the embedding (here logo to ./logo.png). The name can then later on be used to look up the right content ID in the body template (see below). Note that the media type for embeddings and attachments is autodetected based on file ending, though this will get some improvements in the future. The values of entries in an embeddings table map to a Resource and can be specified in three ways:

  1. As string: In which case it's interpreted as a path IRI, relative to the configured base path, which defaults to the dir the template TOML file is in.

  2. As a serialized Source: In the above case this would be logo = { iri="path:./logo.png" }.

    This can also handle the case where you want to specify a specific media type (instead of the crate trying to auto detect it). For this you need to do something like: logo = { iri="path:./logo.png" , use_media_type={ Default="image/png" } }. (There is an issue to make this a bit less verbose but this is not implemented yet). Note that this is a fallback for the case that the thing your context implementation loads the resource from doesn't know the correct media type (like it is the case when you load something from disk). So if you have e.g. logo = { iri="my_doc_db:pictures/logo", use_media_type={ Default="image/png" } } and a custom Context implementation and your document database stores media types and you upload a JPEG instead of a PNG it will not use the image/png but the image/jpeg from the database.

  3. As a fully serialized Resource: In which case you would do something like logo = { "Source": { "iri": "path:./logo.png" }}. This isn't very ergonomic, but the above cases get converted to this case, so it's possible anyway.

The templates

Plain text body

In ./plain_body.hbs we have:

Hy {data.name},

the new greeting for {data.target} is now "Hallo World {data.target}".

Note the data. prefix, we will come to it in the next section.

HTML body

Note, that the following is not a complete HTML mail body. It's missing the whole <html>... ceremony which you should add, though interestingly most mail programs handle it well if you just provide a body.

In "./html_body.hbs" we have:

<img src="cid:{{cids.logo}}" alt="logo"/>
<h1>Hy {data.name}</h1>

the new greeting for {data.target} is now "Hallo World {data.target}".

As you can see we use the cid (Content-Id) schema to refer to the image by the Content-Id field of the mail/mime body which contains the actual image data. cids will be a map of all available embeddings mapping from their name to their actual content id. To prevent name collisions with the cids field we put the actual data into a data field.

Including the templates

Note: This requires the handlebars feature of the mail crate.

Now that we have a template description (template.toml) and the actual templates (html_body.hbs,plain_body.hbs) we can load the template so that we can use it with the mail crate.

Each template is at runtime represented by a Template instance. It will contain all necessary parts for using it. Through that means it will have loaded instances of all embedding attachments.

If this can cause a problem you can simple discard and reload Template instances. On the other hand it allow you to also do thinks like having a LRU cache for them. Initially the library had ways to "just" unload loaded resource but not handlebars templates or to load embedding/attachments only on demand and then discard them. But all of this approaches had some drawbacks, which is why we now only have caching on the whole Template level. Note that this does not include embeddings and attachments passed in through the template data, only attachments and embeddings which are hard coded for a template are affected.

In the end loading a handlebars template is as simple as:

use futures::Future;
use mail::template::{
    load_toml_template_from_path,
    handlebars::Handlebars
};

let template = load_toml_template_from_path(
    Handlebars::new(),
    template_path,
    context
).wait()?;

Note that Handlebars is not the handlebars::Handlebars instance but the mail::template::handlebars::Handlebars. We need to pass this in as the mail crate is not limited to Handlebars (through it currently only provides default impl for handlebars). Bindings to similar template engines can be created easily and bindings to template engines which work a bit different (like Askama) can also be done, but might be a bit more tricky to do.

The template_path is the path to the template.toml file and context is a reference to anything which implements mail::Context. While it returns a Future it is a future waiting for a task in the offload thread-pool to finish, as such it does not need to be run with tokio and you can simply call Future::wait() if you call it in synchronous code.

Using a template

If we have a template it's easy to use it to create a mail::Mail instance:

use mail::template::TemplateExt;

let mut mail = template.render(data.into(), context)?;

What data can be accepted depends on the used template engine (here Handlebars). Using a small trick it was possible to make it accept any data implementing Serialize (if the template engine also accepts any data which implements Serialize and the engine implements TemplateExt for any data implementing Serialize, which Handlebars does).

For now let's ignore the .into() behind the data, which hides a mechanism to provide additional embeddings and attachments through the template input data (we'll come back to this later).

While we now have a mail instance it doesn't yet contain all information we might want to have, like you might already have noticed we never gave it any information about sender, receiver and similar. Which is fine, as the render should just render the templates, i.e. create a mail/mime multipart body structure which appropriately contains all bodies and embeddings and headers specific to that part. But it should not handle any additional thinks, like adding body unrelated mail headers.

Because Mail is built to be modifiable and intentionally only validates some constraints when encoding the mail (like e.g. that there is a From header) it is fairly easy to add all headers we need.

use mail::headers::{_From, _To};

mail.insert_headers(headers! {
    _From: [("Hy Example World", "hy@mail.crate.example.com"),],
    _To: ["random@example.com",],
}?);

Here we insert a From header with a display name and a mail address, as well as a To header with just a single mail address. The _ in _From/_To is needed to prevent name collisions with rust std traits (std::from::From). Note that the [ ] are added as both From and To are a list of mailboxes (mail address and an optional display name). Be aware, that above code will error if any mail address is malformed.

While we used strings in aboves snippet in the real example this are variables e.g. _To: [to_mail_addres,].

Also be aware that the subject is a part of the mail template (and a handlebars template on itself) so a Subject header was already inserted.

Now let's create some example data and put some of the above snippets together. We could use a HashMap or similar for the data, but it's Rust so let's do it in a more type safe way:

// omitted imports

#[derive(Debug, Serialize)]
pub struct HelloWorldData {
    pub name: String,
    pub target: String
}

pub struct HelloWorldTemplate(Template<Handlebars>);

impl HelloWorldTemplate {

    const PATH: &'static str = "./templates/hallo_world/template.toml";

    /// Load this template.
    ///
    /// Note this template has a hard coded path, quite convenient for a
    /// example, but not at all required.
    pub fn load(ctx: &impl Context) -> Result<Self, Error> {
        let template = load_toml_template_from_path(
            Handlebars::new(),
            Self::PATH.into(),
            ctx
        ).wait()?;

        Ok(HelloWorldTemplate(template))
    }

    /// Create a mail using the template.
    ///
    /// Note that passing in a invalid mail address through `from`
    /// to `to` an error is returned.
    ///
    /// Note that we could support additional input data types,
    /// but we don't want that here, so we only have `HelloWorldData`.
    pub fn create_mail(
        &self,
        from: &str,
        to: &str,
        data: HelloWorldData,
        ctx: &impl Context
    ) -> Result<Mail, Error> {
        let mut mail = self.0.render(data.into(), ctx)?;
        // This could also contain headers like `Reply-To`.
        mail.insert_headers(headers! {
            _From: [from,],
            _To: [to,]
        }?);
        Ok(mail)
    }
}

and if we want to use it we can have something like:

let ctx = context::partial_random_context();
let template = templates::HelloWorldTemplate::load(&ctx)?;
let mail = template.create_mail(
    "from@example.com",
    "to@example.com",
    HelloWorldData {
        name: "Lucy".to_owned(),
        target: "Tom".to_owned()
    },
    &ctx
)?;

Printing a mail to stdout

Now that we have a mail it would be nice to print it to stdout. For this we need to encode the mail into a buffer which we then can print to stdout.

Encoding involves a number of steps including:

  1. Loading and transfer encoding all Resource instances which are not yet loaded/transfer encoded.

  2. Checking the mail for validity. This tests thinks like check if there is a From header. If it has multiple mailboxes check if there is a Sender header.

  3. Add some auto-generated headers if needed (like Date).

  4. Add/Override some of the headers which are generated from the context. Mainly all Content-Type and Content-Transfer-Encoding headers.

  5. Turn all of this into a buffer of bytes, or in this case a string as we want to print it to stdout.

While that sounds like much for the API user it's simply:

let encodable = mail.into_encodable_mail(ctx.clone()).wait()?;

Turns a Mail into a EncodableMail.

let encoded = encodable.encode_into_bytes(MailType::Ascii)?;

Here we have to pass in the mail type. Which can either be Ascii or Internationalized. Internationalized is only required if a mail address is used which has a non-us-ascii character in the local part. In all other cases it is possible to use MailType::Ascii which then will use all the different mechanisms to encoded non-us-ascii chars.

Lastly we get a string from it and print it.

let mail_str = str::from_utf8(&encoded).unwrap();
println!("{}", mail_str);

Note that there is a good reason why we encode into a byte buffer (besides it being easier to implement). Which is that mail can have non-UTF-8, non-ASCII bytes under some circumstances. It's just that the things needed to cause the circumstances are currently not supported. Future versions of the crate might decide to not support that parts at all and in turn write into a string buffer.

Sending a mail via SMTP

Note: This requires the smtp feature of the mail crate and an MSA to which you can send the mail

The mail crate comes with optional binding to new-tokio-smtp, a small tokio-based SMTP client library. By setting the smtp feature for the mail dependency you can enable the bindings.

Given that we can already create a "raw" mail (and print it to stdout) sending a mail over SMTP should be straightforward, at least if we have an open SMTP connection and know which SMTP command we have to send for the recipient(s) and which for sender. Which is what the bindings are for. They can get the SMTP recipient(s)/sender out of a mail and know wether or not the mail has to be send over SMTPUTF8 (i.e. internationalized mail). It also adds some error conversion and helper methods.

In the end we can do something like:

use mail::smtp;

let fut = smtp::send(a_mail.into(), con_config, ctx)?;
run_with_tokio(fut)?;

This will get some SMTP relevant information out of the Mail instance and encode it. It will use the con_config (type ConnectionConfig) to open a new connection to a SMTP server, including STARTTLS and authentication. Then it will send the encoded mail to the server after which it closes the connection. In the whole process it will automatically choose the sender and recipients and whether or not it has to use SMTPUTF8.

Alternatively to smtp::send there is smtp::send_batch which sends a batch of any number of mails. I should note here that the SMTP bindings currently do not support connection pooling (which for a lot of setups is less of a problem then it might seem). The underlying new-tokio-smtp allows the implementation of many forms of connection pooling by external libraries, but currently there exists no connection pool implementation. It also should be noted, once there is a a connection pool including it is likely very easy. Lastly you can enable the mail-smtp crates extended-api feature to get access to some of the functions used to create send/send_batch which in turn would allow anyone to create their own connection pool at any time.

At this point I should add a note about how to create a ConnectionConfig which is a (re-exported) type from the new-tokio-smtp crate.

The simplest way is to create a non-secured connection to a local mail server, e.g. for testing. Which can be do in following way:

let con_config = ConnectionConfig::builder_local_unencrypted()
    //.auth, .client_id, .port
    .build();

Note that this should only be used for testing.

To create a config for a typical connection using STARTTLS AUTH PLAIN something similar to following can be used:

use mail::smtp::auth::Plain;

let host = "mail.example.com".parse().unwrap();
let auth = Plain::from_username("username", "password")
    .expect("\\0 is not allowed in auth plain in username/password");
let config = ConnectionConfig::builder(host)
    .expect("could not resolve host name")
    .auth(auth)
    .build();

Which will create a connection config which will connect to the first IP address resolved from the domain name on port 587. Then it uses STARTTLS to make sure the connection is encrypted and follows this by authenticating using AUTH PLAIN using the username both as authorization and authentication identity. At the moment if any step fails then it is seen as a connection failure and no further steps are made. E.g. if the server doesn't support STARTTLS it will quit the connection, if auth fails it also will quit the connection. And so on.

Current supported authentication commands include Login, Plain. Although others can be created by any one using the new-tokio-smtp library without requiring a pull-request or similar (though if you do please still make a pull request, so that the work doesn't need to be done again).

Handling attachments in template input (e.g. an Avatar picture)

The last section is about how you can have an input to a template which contains an attachment or even an embedded image. As an example we will add a template which expects and enforces user input containing a image (as the users avatar).

In ./templates/avatar/template.toml we now have:

name = "email_avatar"
subject = "Your new avatar"

[[bodies]]
path = "html_body.hbs"
media_type = "text/html; charset=utf-8"

Our single body template is defined as:

<h1>Hy you have a new Avatar.</h1>
<img src="cid:{{cids.avatar}}" alt="new avatar"/>

Again we omitted the whole <html>... overhead. Note the cids.avatar, which requires a mapping from "avatar" to the content ID. As you can see we didn't provide an avatar embedding in the template description, so this will fail when rendering the template (consider using the handlebars strict mode to make these errors less silent - sadly it also makes some forms of handlebars templates hard to write).

In the end we will add an embedding with the content ID avatar through the user input and make sure this is enforced at compile time. For this we first have some code representing the template like we had above (note that you don't need to have a type per template, it's just a useful pattern, alongside having a type per-group of similar templates).

pub struct AvatarTemplate(Template<Handlebars>);

impl AvatarTemplate {

    const PATH: &'static str = "./templates/avatar/template.toml";

    pub fn load(ctx: &impl Context) -> Result<Self, Error> {
        let template = load_toml_template_from_path(
            Handlebars::new(),
            Self::PATH.into(),
            ctx
        ).wait()?;

        Ok(AvatarTemplate(template))
    }

    pub fn create_mail(
        &self,
        from: &str,
        to: &str,
        avatar_iri: &IRI,
        ctx: &impl Context
    ) -> Result<Mail, Error> {
        let data = self.prepare_data(avatar_iri);
        let mut mail = self.0.render(data, ctx)?;
        // This could also contain headers like `Reply-To`.
        mail.insert_headers(headers! {
            _From: [from,],
            _To: [to,]
        }?);
        Ok(mail)
    }
}

Mostly same as before but we now pass in an IRI to specify the avatar image. Also we have a (not yet shown) function prepare_data which takes the input data (here just an IRI, but it could be far more complex) and returns some data which we directly pass to render.

Now lets take a look at prepare_data:

// We don't have any additional data.
#[derive(Debug, Serialize)]
pub struct AvatarData;

impl AvatarTemplate {

    pub fn prepare_data(iri: IRI, ctx: &impl Context) -> Result<LoadedTemplateData<AvatarData>, Error> {
        let mut embeddings = HashMap::new();
        embeddings.insert("avatar".to_owned(), Resource::Source(Source {
            iri: iri,
            use_media_type: Default::default(),
            use_file_name: Default::default()
        }));

        let template_data = TemplateData {
            data: AvatarData.into(),
            attachments: Vec::new(),
            inline_embeddings: embeddings
        };

        template_data.load(ctx).wait()
    }
}

This function can be split in two parts (so you probably would have split it into two functions if this wouldn't be an example where we want to keep things side by side):

  1. One part where it converts an IRI into a TemplateData<AvatarData> instance, which also contains a list of additional embeddings and attachments.

  2. One part where it loads the TemplateData which means all contained resources are loaded (and potentially also transfer encoded depending on Context implementation). We need to do this here as only a loaded resource has a content id and content IDs are not only meant to be world unique, but if you have two instances of the same resource you should, if possible, give them the same content ID. I.e. before loading you don't know the content ID, and as such can't use it in a handlebars body.

And that's it. With this the template engine will see a mapping from avatar to the avatar images content ID.