AWS Knowledge Series: Simple Email Service

AWS Simple email service (SES) is Amazon’s answer to services like Mailchimp that allows you to efficiently manage your the promotional and transactional email traffic for your business. In this article we will look at many aspects of SES including the following:

• Prerequisite for using SES in production
• Sending rich templated emails
• Design approach for sending large number of emails using AWS Lambda
• Handling unsubscribe and bounces / complaints

Prerequisite for using SES in production

The most basic prerequisite for using SES in production is a SES verified domain that you will be using for your “FROM” email address. The simplest way to go about doing this is to buy the domain from Amazon on Route53. If you have domain registered already with another provider (e.g. GoDaddy), you must obtain administrator privileges that allow you to modify the DNS records of the domain.

To verify the domain with SES, login to AWS console and select the SES service. Click the “Verify a new Domain” button to bring up the following dialog.

Enter your domain name and check the “Generate DKIM Settings” check box. Click the “Verify This Domain” button to initiate the process. If your domain is registered on Route53, it will give you an option to add the required DNS records (TXT records and DKIM records if DKIM was checked) to your hosted zone on Route53. If your domain is registered on a different provider, download the DNS records file so that you can copy paste the DNS entries on your provider’s DNS management console.

Once the DNS records are configured all you have to do is wait. The SES service will look up the DNS records in the background and within 15 mins to an hour your domain will be in verified state. If the domain is not verified — Most probably the DNS records are wrongly configured. So check again and confirm that they are configured exactly as specified on the SES

The second steps is to enable “production” access for SES. By default, your SES account has only sandbox access. This means that you can only send emails to “SES verified” email addresses. Also your email sending rate is capped at 1 email / second.

Click the “Sending statistics” in your SES console to see the current status of your SES account as shown above. Click the “Edit your account details” to bring up this dialog.

Fill up the required details as mentioned and submit your request for review by AWS. This will create a support case for enabling production access to your SES account. It will take upto 24 hours for AWS to review your case and enable production access. Once production access is enabled, you are all set to send emails using SES. By default you will be able to send 50000 emails in a 24 hour period and your email send rate is limited to 14 emails / seconds. If your use case involves sending more emails and sending emails at higher rate, you will have to raise additional support cases to increase these limits. Again typically AWS will review and approve the increase quota request in 24 hours.

Sending rich templated email

Now you have setup your SES service for sending email, it is time to discuss how to send one. As with all AWS services, one can use the SES APIs to send email. The following code show how to use the AWS Python SDK to send a simple text email.

The code is kind of self explanatory but here is what is happening:

• Obtain the SES service client object for the region from where you want to send out the email. Remember this has to be the same region where you have a verified domain registered and production access for your SES service enabled.
• Use the “send_emai” API to send email to your user.
• > Source — This is the from e-mail address. This has to be from the domain that is verified. You do not need to have a mailbox for this address.
• > Destination — This is the email address of the user whom you want to send the email.
• > ReplyToAddress — This is the email address that will be receiving the email if user hits reply on their email client. If you do not specify this, it will default to the source address you have specified.
• > Message — This is the content of the email that you want to send out to the user. You can provide both the text and html variant.

There is also a provision to specify CC and BCC list while sending out e-mail messages. However this is not a very “rich” email. In order to send out nicely formatted, rich and responsive emails, you have to use templates. In my opinion this is where AWS SES lags behind other services like Mailchimp which offers WYSIWYG editors for authoring templates. The good news however is that SES fully supports the HandlebarJS (https://handlebarsjs.com/) templating language. So the entire process of using templates with SES looks like the following:

• Author your email template as a responsive HTML
• Use handlebarjs placeholders to personalise the email per — receiver
• Upload the template file to SES
• Use the send_templated_email API to send rich, responsive and personalised emails to your users

The following image shows how a HTML template using handlebarjs placeholder looks like.

As you can see it uses multiple handlebarjs placeholders (contained within “{{}}”) that can be used to personalise email for each user. Once you have the template file, the next step is to make it available to SES. The way to do that is to call the create_template API. The following shows a sample code to do the same.

The above script uses developer credentials (profile = default) to create a template in SES. Here is what is happening in the code snippet above.

• Read the HTML template file from disk.
• Invoke the delete_template API to ensure that any old template is deleted.
• Invoke the create_template API to create the template.

The above code creates a template named “welcome-email” in your SES account. Now that email template is created, here is how you can use it to send rich personalised emails.

Here is how you send email using template:

• Create the JSON to provide the data that will be used to personalise the email content. The keys of the JSON has to match the handlebarjs placeholders of the template. In a real production system, this data will be fetched from database or an API.
• Use the send_templated_email API to send the email using the template we created previously.

That's it — You are now ready to send out welcome e-mail to new users registering on your system. You can plugin the code above to appropriate Cognito user pool trigger such that once user’s are registered, a lambda function containing a code similar to above called and it send out personalised welcome email. See “Workflow triggers” in my article on Cognito User Pool to know more about it.

AWS Knowledge Series: Cognito User Pool

Sending large number of emails using AWS Lambda

One of the most common use case that you will come across when developing any consumer solution is the ability to send periodic emails to your entire user population. Following are some of the challenges that you will face when implementing this using server-less technologies like AWS Lambda:

• SES send rate has a limit — You can increase it by raising a support ticket. It is recommended to not increase it beyond a certain value.
• There is a limit on number of concurrent Lambda execution. Again you can increase it by raising a support ticket. It is recommended to not increase it beyond a certain value.
• Each Lambda function invocation can last only 15 minutes at max. Hence you need to have a good batching strategy that processes your user population in batches that ensures that sending emails for each users of the batch does not take more than 15 minutes and no two Lambda invocation targets same set of users.
• AWS Cloud Watch rules (scheduler) has limits on the total number of jobs that you can schedule.

Here is how you can approach to solve this problem:

• Since each Lambda execution can not last beyond 15 minutes there will be an upper limit on the number of users it can process in a single execution. For example, let us say processing email for one user takes 200ms. Assuming that this remains constant for all users, at most a single Lambda execution can handle only 4500 users (not considering the Lambda VM start-up duration here). In real life this limit will depend on factors like the amount of time it takes for you to fetch user specific data to personalise the email, Lambda run-time startup time, network latency of other services invoked during processing. You have to take the worst case scenario and decide the processing time for a single user. If your worst worst case scenario for processing a single user email is 300ms, you should settle for a batch size of 1500 to 2000. I will discuss why this can not be 3000 (=15 mins / 300 ms) in subsequent points.
• Since we will be processing users in batch, we need to ensure that each Lambda execution is processing mutually exclusive set of users. So find a way to order your user population. For example you can use user’s registration time or email address to have a predictable “order”. This will allow us to create distinct batches of users that can then be handed over to Lambda for sending emails.
• As mentioned previously, SES has an email sending limit. You can not send emails at a rate more than what is configured for your SES account. You can always raise a support ticket to increase the email sending rate limit however increasing it beyond a certain threshold is not recommended for a simple reason that a rogue / wrongly coded function can just end up sending large number of emails costing you money and reputation. So there are two things to be taken care here:
• > Your email sending rate must not exceed the one configured for your SES account. There is no easy way to control the rate at which we send emails. Hence it is mandatory that the email sending operation catches the rate exceeded exception and implement retry with exponential backoff while sending email. Keeping the batch size small enough as mentioned in point 1 above must allows us enough headroom that even with retries, we should be able to send out emails to all the users being processed by the current execution. For this reason as mentioned in point 1, even if your worst case scenario is 300ms / user email, you should settle for a batch size of 1500 to 2000 and not 3000.
• > Your Lambda concurrent execution must not exceed the concurrency limit configured for your account. Let us say that for your account the concurrent execution limit for Lambda is 1000 and your peak Lambda concurrency for non-email processing is 750. What this means is that you can only have 250 concurrent execution of Lambda processing your email. Again this is not something that can be easily controlled. To ensure that you do not cross the concurrent execution limits of Lambda, you have to segment your users and then process each segment at different times. One way to do this is to use consistent hashing with a unique user attribute like user id or user email to segment the users. This segmentation must happen as the users are registering on your system. So for example, if you have segmented your users in 5 different segments, you can schedule the first segment at 6 am, next one at 7 am and so on and so forth. Within a segment, the users are processed using the order defined in point two above. Segmenting and ordering users ensures that you never go beyond a certain concurrency of Lambda execution while sending emails.
• The user specific data being fetched for personalising the email must have consistent predictable latency. Ensure that fetching such data takes same amount of time for users who have large amount of data as well as for user who have less amount of data. Once way to do that is to pre-fetch / pre-compute and cache the data before starting the email send process.

Handling unsubscribe and bounces / complaints

As mentioned in the AWS best practices for SES, you must provide a way for users to unsubscribe from promotional emails if they wish to do so. Secondly in order to maintain your “reputation” as a sender you must also handle bounce and complains else your mail could end up in the spam folder, your domain could be blacklisted, and SES could suspend your service.

The recommended way to offer unsubscribe capability to the end users is by adding the “list-unsubscribe” header in the email as it allows end users to unsubscribe from within their favourite email clients. As of now, the send_templated_email or send_email API do not offer a way to add this header. The send_raw_email API allows you to specify the required e-mail headers but you can not use templates with send_raw_email API. I am hoping that AWS will add the capability to specify the e-mail headers in future release of the SDK.

There are two ways to use the “list-unsubscribe” header — (1) One way is to specify an email address in the header. When user clicks unsubscribe in the email client, the email client creates a new message to the email address specified in the “list-unsubscribe” header. You need to setup a receive rule set for this email address and then write handler code to remove the sender from the said e-mail list. The way you will do is to setup a email receiving rules and direct the payload to an SNS topic. This SNS topic will have a Lambda function as subscriber and with in the Lambda function you need to parse the SNS payload to get the sender email and then update your “unsubscribed users” list with that email address. (2) The second a more modern way of doing this is to provide a link to a web page where user can specify his / her email address and click unsubscribe to remove that email address from the list. This does involve more code and setup. Again this is where SES lags behind providers like mailchimp. Mailchimp templating language allows you to embed the unsubscribe link / header with just a single line directive. Once users click on the link or hit unsubscribe in mail client, mailchimp invokes a web hook and provides you with all the details required to unsubscribe the user.

A round about way to offer unsubscribe is by having a user profile section within your app / web site where you can provide options for users to subscribe / unsubscribe from various emails that you may be sending.

In order to handle the bounces and complaints you need to first enable the SES to send your SNS notifications when such events occur. The way to do that is as follows: Go to the SES console and click the verified domain that is being used for sending email. Under the “Notifications” header you can configure the SNS topics where SES will notify you about bounces and complaints.

Click the “Edit configuration” button to bring up the following dialog. Fill up the required information and you are all set to receive information whenever an email bounces or user marks the email as spam.

If you have an SNS topic configured for this, you can use it or else create a new one and use the newly created SNS topic as destination for bounces and complaints. Do check the “Include original headers” check box and save the configuration.

Now you can author a Lambda function which can be configured as subscriber of the SNS topic. Whenever SES sends a notification to topic about bounce / complaint, Lambda function will be invoked. The Lambda event payload contains a “stringified” JSON in the “Message” attribute. This JSON will contain the email address. Now you can store this email address in the “bounce / complaints” email list.

While sending emails using SES, you have to check if the user’s email exists in the “unsubscribed users” or “bounce / complaints” list or not. If you do find the email in any of the list, you must not send the email to that address.