Space Vatican

Ramblings of a curious coder

Creating a Custom CloudFormation NAT Gateway Resource

Update: You can now provision NAT gateways with CloudFormation.

Amazon recently announced their NAT gateway service which allows you to attach an AWS managed NAT gateway to you private subnets instead of managing NAT instances yourself. There is a full rundown of the differences, but for me the big win is eliminating the single point of failure of a single NAT instance without the complexity of a failover setup. It’s also nice not to worry about picking the right size of NAT instance.

We manage our VPC infrastructure with CloudFormation, which doesn’t yet have support for NAT gateways. However CloudFormation has custom lambda resources that can do pretty much anything. Even when CloudFormation does gain support for NAT gateways, hopefully this provides another example of how to create custom resources.

Lambda Backed Custom Resources

When you use a Lambda backed resource, CloudFormation invokes the specified Lambda function asynchronously. It passes some metadata such as the id of the stack, the logical id of the resource (the key of the resource in your template) and of course the resource properties. When you’re done you upload a JSON document describing what you’ve done to a presigned S3 url it provides.

You ALWAYS need to write to this presigned url, even in the case of failure, or CloudFormation will just keep waiting (and eventually timeout). AWS provides the cfn-response snippet that makes this a little easier: you just need to pass the event, context and response data and it will craft the JSON document, upload it and call context.done for you.

For a NAT gateway the inputs we need are the elastic IP allocation id and the subnet id, so your template looks something like this:

1
2
3
4
5
6
7
8
9
10

{
  "MyGateway": {
    "Type": "Custom::NatGateway",
    "Properties": {
      "ServiceToken": "arn:aws:lambda:eu-west-1:123456789012:function:CreateNatGateway",
      "SubnetId": {"Ref": "NatSubnet"},
      "AllocationId": {"Fn::GetAtt": ["NatIP", "AllocationId"]}
    }
  }
}

In my template NatIP is an elastic ip created by the template and NatSubnet is a subnet created by the template, but how you create those is obviously up to you. CloudFormation doesn’t care about the type (Custom::NatGateway) beyond the Custom:: prefix. It allows both making the template a little more readable and allows a single Lambda function to handle multiple resource types.

Handling requests

There are 3 types of requests: create, update and delete. The data passed to your function varies slightly.

Upon receiving a create request you should create your resource from the data in event.ResourceProperties and your response should include the PhysicalResourceId. This needs to uniquely identify what your Lambda function created, and will be passed back to you in subsequent update or delete requests for the resource. It’s also what {"Ref": "MyGateway"} will evaluate to in the remainder of your template. The NAT gateway id is the obvious thing to use in this case.

For update requests CloudFormation also passes an OldResourceProperties attribute that describes the resource properties as they were specified prior to this update. If you can update the resource in-place, then return the same PhysicalResourceID. If not then create a new resource, return a different PhysicalResourceId and CloudFormation will follow up with a delete request on the old id. NAT gateways can’t be changed after they are created so we handle create and update requests identically.

Lastly, delete requests should destroy the resource.

The lambda function entry point just dispatches on the request type:

1
2
3
4
5
6
7
8

exports.handler = function(event, context) {
  if (event.RequestType === 'Delete') {
    deleteGateway(event, context);
  }
  else if (event.RequestType === 'Update' || event.RequestType === 'Create') {
    createGateway(event, context);
  }
};

The createGateway method creates a gateway and sends the response back to CloudFormation. Any properties you set in responseData will become available via Fn::GetAtt. For my usage there aren’t really any properties of the gateway that I wanted to expose.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

var createGateway = function(event, context){
  var responseData = {};
  var subnetId = event.ResourceProperties.SubnetId;
  var allocationId = event.ResourceProperties.AllocationId;

  if(subnetId && allocationId){
    ec2.createNatGateway({AllocationId: allocationId, SubnetId: subnetId}, function(err, data){
      if(err){
        responseData = {Error: "create gateway failed " + err },
        response.send(event, context, response.FAILED, responseData);
      }
      else{
        response.send(event, context, response.SUCCESS, responseData, data.NatGateway.NatGatewayId);
      }
    });
  }
};

The deleteGateway method just needs to delete the gateway:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

var deleteGateway = function(event, context){
  var responseData = {};
  if(event.PhysicalResourceId.match(/^nat-/)) {
    ec2.deleteNatGateway({NatGatewayId: event.PhysicalResourceId}, function(err, data){
      if(err){
        responseData = {Error: "delete gateway failed " + err};
        response.send(event, context, response.FAILED, responseData);
      }
      else{
        response.send(event, context, response.SUCCESS, responseData);
      }
    });
  }
  else{
    console.log("No valid physical resource id passed to destroy - ignoring " + event.PhysicalResourceId);
    response.send(event, context, response.SUCCESS, responseData);
  }
};

The validation of the physical resource id is to deal with a slight edge case. CloudFormation autoassigns a physical resource id to your resource, which is then overwritten when the response comes back. However if you try and rollback the stack update before that has happened, then it will issue the delete request with this dud physical resource id. In these cases we want the rollback to do nothing, rather than fail because we’ve passed a bogus id to deleteNatGateway.

Using the gateway

A NAT gateway is no use until you’ve added it to your route table, but unfortunately the AWS::EC2::Route resource doesn’t support NAT gateways yet, so we need another custom resource to handle the route. I’ve handled this within the same lambda function, so the top level entry point no needs to dispatch on the resource type:

1
2
3
4
5
6
7

exports.handler = function(event, context) {
  if (event.ResourceType === 'Custom::NatGateway') {
    handleGateway(event, context); //calls the code described previously
  } else if (event.ResourceType === 'Custom::NatGatewayRoute') {
    handleRoute(event, context);
  }
};

The aws-sdk provides updateRoute, deleteRoute and replaceRoute apis, which map naturally onto the request types but replaceRoute can only be used if the destination CIDR and route table id haven’t changed, so we check event.OldResourceProperties to see if we can use replaceRoute.

1
2
3
4
5
6
7
8
9
10
11
12
13
14

var handleRoute = function(event, context) {
  if (event.RequestType === 'Delete') {
    deleteRoute(event, context);
  } else if (event.RequestType === 'Create') {
    createRoute(event, context);
  } else if (event.RequestType === 'Update') {
    if (event.ResourceProperties.DestinationCIDRBlock === event.OldResourceProperties.DestinationCIDRBlock &&
        event.ResourceProperties.RouteTableId === event.OldResourceProperties.RouteTableId) {
      replaceRoute(event, context);
    } else {
      createRoute(event, context);
    }
  }
};

There is no ID that is returned by the createRoute api: a route is identified by the route table and the cidrblock, so we concatenate these to create a physical identifier. Other than that the createRoute, replaceRoute and deleteRoute functions just call through to the corresponding EC2 api.

You can then put something like this in your template:

1
2
3
4
5
6
7
8
9
10
11
12

{
  "NatRoute": {
    "Type": "Custom::NatGatewayRoute",
    "DependsOn": ["NatGateway"],
    "Properties": {
      "ServiceToken": "arn:aws:lambda:eu-west-1:123456789012:function:CreateNatGateway",
      "RouteTableId": {"Ref": "PrivateSubnetsRouteTable"},
      "DestinationCidrBlock": "0.0.0.0/0",
      "NatGatewayId": {"Ref": "MyGateway"}
    }
  }
}

Timing

Some resources (such as routes) create instantly, but a NAT gateway takes a minute or two to become available and you may run into issues if you create routes before it is ready. I haven’t found any better way of dealing with this than have the function poll the DescribeNATGateways until its state changes. Luckily this happens fast enough to be well clear of a lambda function maximum execution time (5 minutes), although you are of course paying for execution time while your function is in fact just waiting.

You want CloudFormation to get the ID of the gateway as soon as possible rather than waiting for this, to avoid the case where it can’t destroy the gateway because it doesn’t know it’s id. I’ve handled this by responding to CloudFormation as soon as createNatGateway is complete and then used aWaitCondition in the template so that CloudFormation waits before creating the route. The lambda function polls DescribeNATGateways and signals the wait condition when the state changes to available (This required changed the response.send function from cfn-response so that it doesn’t terminate the Lambda function.)

I’m not aware of a way of making this self contained.

Packaging up

One last niggle is that at time of writing, the version of the aws-sdk included in the Lambda environment doesn’t include the NAT gateway related functions (it includes 2.2.12 but we require 2.2.24). This means that you need to upload a package containing your source and the required version of aws-sdk. Assuming you have saved your script as nategateway.js and that you have npm installed you would do this:

1
2

npm install aws-sdk
zip -r nat_gateway.zip nat_gateway.js  node_modules/

And then upload nat_gateway.zip. In the Lambda config for your function you need to set the handler to nat_gateway/index.

TLDR;

The full script for a lambda function that handles both the gateway and its route, along with some of the error handling that I omitted here for reasons of brevity, is here along with a CloudFormation template that uses it.

The template creates a VPC, 2 subnets (a public one that the NAT gateway lives in and a private one that uses the gateway for external traffic), the NAT gateway and routes. It also creates the Lambda function and role needed for the custom resources.

The template uses a zip file I have uploaded to S3 in eu-west-1, so you will have to reupload it to an s3 bucket in your region (& update the template) to use it in other regions. I also highly recommend that you check the zip file rather than just blindly running code I’ve written, trustworthy as I may be. This stack will cost a small amount of money to run, since NAT gateways are not free.

Please note that this does have drawbacks compared to “native” cloudformation support for nat gateways, particularly if you were to use this to update an existing stack. Cloudformation doesn’t realize that your Custom::NatGatewayRoute and AWS::EC2::Route are the same underlying resource, so it won’t use ReplaceRoute to update the route. In fact in my experiments I had to do 2 stack updates because it wasn’t deleting the AWS::EC2::Route before adding the custom route. You would likely face a similar task if you wanted to switch from this to native cloudformation resources in the future. Caveat lector etc.