AWS Knowledge Series : RESTful API using API gateway + Lambda + DynamoDB + Terraform

In this article we will look at writing RESTful service using API gateway and Lambda. We will use DynamoDB as a backend store. We will use terraform to deploy all the AWS resources. We will write following APIs:

Product DynamoDB Table

Let us design the DynamoDB table for the product. Read my article about DynamoDB table design here:

AWS Knowledge Series: DynamoDB — Part 1

Following are the access patterns that we want to implement for our product table:

• Insert, Update, Delete and Fetch a single product
• Fetch products belonging to a category ordered by product rating

The following terraform definition shows the table schema definition:

provider "aws" {
  profile = "default"
  region  = "ap-south-1"
}
resource "aws_dynamodb_table" "product_table" {
  name         = "PRODUCT"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "product_id"
  attribute {
    name = "product_id"
    type = "S"
  }
  attribute {
    name = "category"
    type = "S"
  }
  attribute {
    name = "product_rating"
    type = "N"
  }
  global_secondary_index {
    name            = "ProductCategoryRatingIndex"
    hash_key        = "category"
    range_key       = "product_rating"
    projection_type = "ALL"
  }
}

The primary key with product_id as hash key will allow us to access a single product record for CRUD operation. The global secondary index uses category as hash key and product rating attribute as range. This secondary index allows us to fetch products belonging to a given category sorted by their ratings.

The provider definition at the top tells terraform to use “default” configuration and create all the resources in the “ap-south-1” region. In order to create the above DynamoDb table in AWS, you will have to do the following:

• Sign up for an AWS account (https://aws.amazon.com/premiumsupport/knowledge-center/create-and-activate-aws-account/).
• Create a user and provide programatic admin access for the user (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html and https://docs.aws.amazon.com/IAM/latest/UserGuide/getting-started_create-admin-group.html)
• Install AWS CLI on your system (https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html)
• Configure AWS CLI on your system (https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html). By default it will create a profile named “default”. You can also create a named profile (https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html) if you want to use different AWS accounts on your system.
• Download and configure Terraform on your system (https://www.terraform.io/downloads.html)

For this article I suggest that you create following folders on your system:

product-api

• > lambda
• > terraform

The “lambda” folder will store code for the Lambda function.

The “terraform” folder will store the terraform configuration

Once the above setup is complete, create a file called “product.tf” and copy the terraform configuration shown above. Open a console window and change directory to terraform folder. Execute the following commands:

terraform init

This will download the files related to the provider configured in your terraform file which is AWS.

Now execute the following command to see that the terraform script is valid.

terraform validate

This will look at all the terraform files in the current folder and check if they are valid.

Now to deploy the DynamoDB table to your AWS environment — use the following command:

terraform apply

Note: I am using the default profile I created using AWS CLI hence my terraform provider block has default as the name of the profile. If you are using named profile change it accordingly. I am deploying the resources in ap-south-1 region. You can deploy it in any region of your choice. AWS console uses us-east-1 as the default region.

Once terraform executes, it will compute what infrastructure components have to be deployed. In this case, it will indicate that one DynamoDB table needs to be created. Enter “yes” when prompted to create the PRODUCT DynamoDB table. You can verify that the table has been created by login into AWS console and navigating to the DynamoDB service. Make sure that you have selected the same region in the AWS console as specified in the terraform file.

API Gateway

API gateway is a service that allows developers to create, publish, maintain, monitor, and secure APIs at any scale. In traditional client server architecture, it is equivalent to the load balancer / web server component.

The above diagram shows what we will build today. The API gateway is the HTTP endpoint that the client applications will call to invoke our APIs. It will invoke a Lambda function which will perform the required CRUD operation on the DynamoDB table we created above. While the API gateway offers numerous configurations using which we can tune how it can scale, what caching strategy it can use, what authentication mechanism it will use etc. For this article, we will setup a simple API gateway with default configuration parameter. To do this we can add the following terraform configuration to the product.tf file we created above.

resource "aws_api_gateway_rest_api" "product_apigw" {
  name        = "product_apigw"
  description = "Product API Gateway"
  endpoint_configuration {
    types = ["REGIONAL"]
  }
}
resource "aws_api_gateway_resource" "product" {
  rest_api_id = aws_api_gateway_rest_api.product_apigw.id
  parent_id   = aws_api_gateway_rest_api.product_apigw.root_resource_id
  path_part   = "product"
}
resource "aws_api_gateway_method" "createproduct" {
  rest_api_id   = aws_api_gateway_rest_api.product_apigw.id
  resource_id   = aws_api_gateway_resource.product.id
  http_method   = "POST"
  authorization = "NONE"
}

As you can see we are creating three resources:

aws_api_gateway_rest_api : product_apigw

• This the the API gateway resource. Once API gateway is created, it will provide an internet facing HTTP endpoint of the form:

https://<<API GW ID>>.execute-api.<<Region>>.amazonaws.com/<<Stage>>

• API gateway id is a randomly generated host name.
• Region refers to the region where the API gateway is created. In out case as per the terraform above this will be “ap-south-1”.
• Stage refers to a named deployed instance of the API gateway. We will talk about it in section that explains how to connect the API gateway with Lambda.

aws_api_gateway_resource: product

• This creates the “/product” path of the RESTful API.

aws_api_gateway_method: createproduct

• This creates a method stub that is mapped to the HTTP POST verb. We will map this method with the Lambda function in the next section. In this sample we are not covering authentication / authorisation hence we specify the authorisation attribute as NONE. This will mean that any one can call the create product API. I will talk about securing API gateway resources with Cognito in a future article.

Run the terraform apply command as shown before to deploy the resources. The output will indicate that it will add the API gateway resource. Type “yes” when prompted to create the three resources.

Lambda Function

Lambda function are equivalent to your application server. This is where you will write your business logic that will work on your data. For our sample we will have to write a lambda function that creates a product record in the DynamoDB table. In AWS all the resources and the actions that you can perform on them are protected by IAM roles and policies. So in order to access DynamoDB table from Lambda, we will have to create appropriate role and policy for the Lambda function such that it can access the DynamoDB table.

So we have to create following resources:

• Role — A role that can be assigned to the Lambda function we are writing
• Policy — A policy that allows us to perform the putItem operation on the PRODUCT table that we created above
• Role — Policy Mapping — Assigning policy to the role

First let’s create a policy definition document. Create a file called policy.json under the terraform folder and copy the following content:

{
    "Version": "2012-10-17",
    "Statement": [
      {
          "Effect": "Allow",
          "Action": [
              "logs:CreateLogStream",
              "logs:CreateLogGroup",
              "logs:PutLogEvents"
          ],
          "Resource": "arn:aws:logs:*:*:*"
      },
      {
          "Effect": "Allow",
          "Action": [
              "dynamodb:PutItem"
          ],
          "Resource": "arn:aws:dynamodb:*:*:table/PRODUCT"
      }
    ]
}

The policy allows for creation of cloud watch logs streams so whatever we log in the Lambda function will shown up in the cloud watch logs. Secondly the policy allows operation “PutItem” on the “PRODUCT” DynamoDB table.

The following terraform script creates the role / policy and role-policy mapping resources:

resource "aws_iam_role" "ProductLambdaRole" {
  name               = "ProductLambdaRole"
  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Effect": "Allow",
      "Sid": ""
    }
  ]
}
EOF
}
data "template_file" "productlambdapolicy" {
  template = "${file("${path.module}/policy.json")}"
}
resource "aws_iam_policy" "ProductLambdaPolicy" {
  name        = "ProductLambdaPolicy"
  path        = "/"
  description = "IAM policy for Product lambda functions"
  policy      = data.template_file.productlambdapolicy.rendered
}
resource "aws_iam_role_policy_attachment" "ProductLambdaRolePolicy" {
  role       = aws_iam_role.ProductLambdaRole.name
  policy_arn = aws_iam_policy.ProductLambdaPolicy.arn
}

Append the above content in product.tf folder and execute the terraform apply command to create these resources in AWS.

Let us implement the create product lambda function. I am using Python runtime for the Lambda but you can write it in any runtime that is supported by Lambda. The implementation and APIs of AWS services are identical so implementing this in any other language should not be much of a challenge.

Create a file named createproduct.py and copy the following content in it and save it to the lambda folder.

import logging
import boto3
import json
import os
session = boto3.Session(region_name=os.environ['REGION'])
dynamodb_client = session.client('dynamodb')
def lambda_handler(event, context):
    try:
        print("event ->" + str(event))
        payload = json.loads(event["body"])
        print("payload ->" + str(payload))
        dynamodb_response = dynamodb_client.put_item(
            TableName=os.environ["PRODUCT_TABLE"],
            Item={
                "product_id": {
                    "S": payload["productId"]
                },
                "category": {
                    "S": payload["category"]
                },
                "product_rating": {
                    "N": str(payload["productRating"])
                },
                "product_name": {
                    "S": payload["productName"]
                },
                "product_price": {
                    "N": str(payload["productPrice"])
                },
                "product_description": {
                    "S": payload["productDescription"]
                }
            }
        )
        print(dynamodb_response)
        return {
            'statusCode': 201,
           'body': '{"status":"Product created"}'
        }
    except Exception as e:
        logging.error(e)
        return {
            'statusCode': 500,
           'body': '{"status":"Server error"}'
        }

The implementation is very straightforward. We are using the AWS boto3 python SDK to access AWS services — in this case the DynamoDB service. We are creating the dynamodb client outside the lambda handler function so that as long as the Lambda VM is alive, we can reused the same client instance.

We then parse the payload sent by the client application and invoke the put item API of DynamoDb service to upsert the product item.

Now to deploy the lambda function, we first have to package the python code in a ZIP archive. Create a batch / shell script that bundles up the python files and creates a ZIP out of it.

On MacOS, you can do it as follows:

zip ./product_lambda.zip *.py

Open a terminal window navigate to the lambda folder and execute the above commands to package the python code in a zip file.

Append he following terraform script to the product.tf file.

resource "aws_lambda_function" "CreateProductHandler" {
  function_name = "CreateProductHandler"
  filename = “../lambda/product_lambda.zip”
  handler = "createproduct.lambda_handler"
  runtime = "python3.8"
  environment {
    variables = {
      REGION        = “ap-south-1”
      PRODUCT_TABLE = aws_dynamodb_table.product_table.name
   }
  }
  source_code_hash = filebase64sha256(“../lambda/product_lambda.zip”)
  role = aws_iam_role.ProductLambdaRole.arn
  timeout     = "5"
  memory_size = "128"
}

Here is what is happening in the above terraform definition of the Lambda resource:

• We are creating a Lambda function named CreateProductHandler
• The source code for the lambda function is located in the zip file that we created above
• We are defining the file name and the function where the Lambda function entry point is located. In our case the file name is called createproduct.py and the lambda entry point function is called lambd_handler hence we specify “createproduct.lambda_handler”
• We use the python 3.8 runtime for the lambda function
• We define two environment variables for the Lambda function — which we are referring in out Lambda code
• REGION — The AWS region where the function is deployed
• PRODUCT_TABLE — The name of the DynamoDB table where product items are stored. Notice how we are referring the previously defined DynamoDb table resource.
• The source_code_hash is an attribute that is used by terraform to decide if the function has to be deployed or not. While deploying the Lambda function terraform compares the hash of the ZIP file specified and compares it with the hash of the deployed function. If hash are the same then it does not deploy it — if the hash are different it overwrites the code of the Lambda function.
• As explained before we assign the role we created earlier which allows Lambda to access DynamoDb service and execute the put item API.
• The timeout of the function is set to 5 seconds and the VM memory limit is set to 128 MB.

You can execute the terraform apply to deploy the Lambda function to AWS.

Connecting API Gateway to Lambda Function

This is the last step in making the RESTful API functional. Here is what we want to achieve:

• When we receive an HTTP POST request on the “/product” resource, we want to invoke the CreateProductHandler lambda function.
• We want to pass the payload of the HTTP request to the Lambda function.
• Once lambda function executes — we want the API gateway to pass the response back to client.

Append the following terraform script to the product.tf:

resource "aws_api_gateway_integration" "createproduct-lambda" {
rest_api_id = aws_api_gateway_rest_api.product_apigw.id
  resource_id = aws_api_gateway_method.createproduct.resource_id
  http_method = aws_api_gateway_method.createproduct.http_method
  integration_http_method = "POST"
  type                    = "AWS_PROXY"
uri = aws_lambda_function.CreateProductHandler.invoke_arn
}
resource "aws_lambda_permission" "apigw-CreateProductHandler" {
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.CreateProductHandler.function_name
  principal     = "apigateway.amazonaws.com"
  source_arn = "${aws_api_gateway_rest_api.product_apigw.execution_arn}/*/POST/product"
}
resource "aws_api_gateway_deployment" "productapistageprod" {
  depends_on = [
    aws_api_gateway_integration.createproduct-lambda
  ]
  rest_api_id = aws_api_gateway_rest_api.product_apigw.id
  stage_name  = “prod”
}

We are creating three resources:

aws_api_gateway_integration: createproduct-lambda

• Here we are mapping the API gateway resource (“/product”) and the associated HTTP method (POST) with the Lambda function we created. The integration method is set to POST and the integration type is set to AWS_PROXY. The API gateway then transforms the incoming HTTP request to JSON format and delivers as the “event” object to the Lambda function. Once Lambda returns, it transforms the lambda response to HTTP response. For this kind of integration, Lambda function is supposed to return two attributes mandatorily — status — this translates to the JTTP response code returned to the client and body — this will be the response body received by the client. Status has to be numeric and body has to be string type.

aws_lambda_permission: apigw-CreateProductHandler

• This grants permission to the API gateway service to invoke our lambda function.

aws_api_gateway_deployment: productapistageprod

• This deploys the API to a “stage”. As mentioned previously, stage is a named reference of API gateway deployment, which is a snapshot of the API. Stage is used to manage and optimize a particular deployment. For example, you can configure stage settings to enable caching, customize request throttling, configure logging, define stage variables.

Deploy the resources by running terraform apply command.

Once everything is deployed — It is time to test! You will need to find out the base URL of the API Gateway stage. Login to the AWS console and navigate to the API gateway service. Click the API gateway you created above and navigate to the Stage from left menu. The right side pane will show the URL as shown below:

Execute the CURL command as shown below:

curl -X POST "https://gxr92mgspa.execute-api.ap-south-1.amazonaws.com/prod/product" -H 'Content-Type: application/json' -d'
{
  "productId": "1",
  "category": "Category",
  "productName": "<<Product Name>>",
  "productPrice": 12,
  "productDescription": "Product description",
  "productRating": 2
}
‘

The following response will be returned:

{“status”:”Product created”}

Now you know how to use API gateway, Lambda and DynamoDB to create restful API. Now to destroy all the AWS resources that you created execute the following command:

terraform destroy

This will remove all the resources that you created above.

Find the complete source at https://github.com/santhedan/apigwsample