Skip to content
NAT Gateway is the most expensive line you don't see
← ← Back to Thinking Cloud

NAT Gateway is the most expensive line you don't see

It isn't on the bill. Not as "NAT Gateway", anyway. It's under EC2-Other, a usage type called NatGateway-Hours next to another called NatGateway-Bytes, and in Cost Explorer's default view it's folded into the EC2 line, which is why a team that runs no EC2 instances can look at a $100 EC2 charge and assume it's a mistake.

It's not a mistake. It's $0.045 an hour, every hour, for each NAT gateway you have, before a single byte goes through it, plus $0.045 for every gigabyte that does. On the three-account bill we published it was 13 % of the total, more than Lambda, DynamoDB and WAF combined, for a component that does nothing except let private things reach the internet. This article is why you have it, and how we went from three to one.

Why you have a NAT gateway

Because something is in a private subnet. Private subnets have no route to the internet gateway, on purpose: nothing on the internet can reach them. The price is that they can't reach the internet either, and "the internet" includes the AWS API endpoints, so a Lambda in a private subnet can't call S3, Secrets Manager or SQS without a way out. The way out is a NAT gateway in a public subnet, with a route from the private subnets to it.

Our reason was Aurora. The database is in a private subnet, correctly. The Lambdas that talk to it have to be in the VPC to reach it. Once they're in the VPC, they need the NAT for everything else they do: fetching a secret, putting a message on a queue, calling a third-party API. One requirement, database access, pulled a dozen functions into the VPC and a NAT into every account.

VPC private subnet Lambda Aurora gateway endpointS3 · DynamoDB · free interface endpointSecrets Manager · $7/mo public subnet NAT gateway$0.045 / h + $0.045 / GB everything else, by default internetgateway S3 · SQSSecrets Mgr3rd-party APIsback into AWS

What it actually cost us

Three accounts, one NAT each, one availability zone each (we'd deliberately not done one per AZ, which would have tripled it):

Hours Bytes Monthly
prod $33 ~60 GB × $0.045 = $3 $36
staging $33 ~5 GB $33
dev $33 ~10 GB $33
Total $102

Notice the shape. It's almost entirely the hourly charge. The bytes are nothing, because most of what goes through the NAT is small API calls. So the usual advice, "reduce your NAT data processing", would have saved us three dollars. The lever is the count.

Five ways to need fewer of them

1. Gateway endpoints for S3 and DynamoDB. Free. A gateway endpoint is a route-table entry that sends traffic for S3 or DynamoDB through AWS's network instead of out and back. It costs nothing, it's two lines of CDK, and it should be in every VPC on day one regardless of NAT. It removed about half of our NAT bytes, which was worth $1.50. Do it anyway; the reason is latency and not paying egress for S3 traffic, not the NAT.

vpc.addGatewayEndpoint('S3', { service: ec2.GatewayVpcEndpointAwsService.S3 });
vpc.addGatewayEndpoint('DynamoDB', { service: ec2.GatewayVpcEndpointAwsService.DYNAMODB });

2. Interface endpoints for the services you actually call. $7.30 a month each. An interface endpoint is a network interface in your subnet with a private IP for one AWS service: Secrets Manager, SQS, ECR, CloudWatch Logs, STS. Each costs about $7.30 a month plus $0.01 a gigabyte. The arithmetic: an endpoint pays for itself only if it lets you remove a NAT, or if you push more than about 700 GB a month through it. Five endpoints to cover everything our Lambdas call is $37, which is more than the NAT they'd replace. So we use them in exactly one place, below.

3. Get out of the VPC. This is the big one. A Lambda only needs to be in the VPC if it talks to something that's only reachable there. We audited the dozen functions: seven talked to Aurora, five didn't. The five were in the VPC because the CDK construct had vpc: set as a default on our function factory. Removing them from the VPC took one flag and removed them from the NAT's client list entirely. A Lambda outside the VPC reaches AWS APIs directly, for free, and starts faster.

For the seven that need Aurora, there's a further option we're evaluating: RDS Data API, which lets a function outside the VPC query an Aurora cluster over HTTPS. It changes the driver, so it's not a flag, but it would empty the VPC of Lambdas entirely.

4. Don't put one in dev. The dev environment's NAT existed so that dev Lambdas could reach the internet. After step 3, the only things in dev's private subnets that needed the internet were the seven database functions fetching secrets and posting to queues. Two interface endpoints, Secrets Manager and SQS, at $15 a month, replaced a $33 NAT, and dev's third-party API calls (there aren't any in dev; they're stubbed) stopped needing a path at all. Staging got the same treatment.

5. A NAT instance where you truly need cheap egress. If dev had needed general internet access, the answer is a NAT instance rather than a NAT gateway: a t4g.nano running the open-source fck-nat image costs about $3 a month, handles a couple of hundred megabits, and is a fine trade for a non-production environment where a single point of failure is acceptable. We didn't need it, but it's the right tool for the "I need a NAT but not a $33 NAT" case. Never in production.

There's a sixth that only applies if you can go IPv6-only: an egress-only internet gateway is free and does for IPv6 what a NAT does for IPv4. Most third-party APIs aren't reachable over IPv6 yet, so it wasn't an option for us.

Where we ended up

Before After Monthly
prod 1 NAT gateway 1 NAT gateway + S3/DynamoDB gateway endpoints $36 → $36
staging 1 NAT gateway Secrets Manager + SQS interface endpoints, no NAT $33 → $15
dev 1 NAT gateway Secrets Manager + SQS interface endpoints, no NAT $33 → $15
Lambdas in VPC 12 7
Total $102 $66

Production keeps its NAT: it calls third-party APIs, it needs a real path, and $36 for a managed, multi-AZ-capable component with no maintenance is fair. The saving is $36 a month, which sounds small until you notice it's 5 % of the whole bill, it took an afternoon, and the same afternoon made five Lambdas start faster.

NAT and endpoint cost, before and after before prod NAT $36 staging NAT $33 dev NAT $33 $102 after prod NAT $36 staging $15 dev $15 $66 · −$36 / month Lambdas in the VPC: 12 → 7 · gateway endpoints added everywhere · one afternoon The lever was the count of NATs, not the bytes through them.

The rule for new things

Before a new function or service goes into a private subnet, it has to answer one question in the pull request: what in the private subnet does it talk to? If the answer is "nothing", it doesn't go in. If the answer is "the database", it goes in and it gets the endpoints it needs listed. If the answer is "it needs the internet", then it's production or it's a design conversation.

The NAT gateway isn't a bad product. It's a product that's easy to have three of without deciding to. Count yours.

If your EC2-Other line is bigger than your EC2 line, we can tell you why in an afternoon.