Creating AWS VPC Gateway Endpoint for DynamoDB Breaks ECS Task

When I create a DynamoDB VPC Gateway Endpoint it breaks an ECS task that connects to DynamoDB. The ECS task/service/cluster is in the same VPC in the same region as the gateway endpoint on a private subnet and its security group allows outbound access to DynamoDB. The VPC has DNS resolution and DNS hostnames Enabled.

The task is running aws-sdk for C++ to connect to a DynamoDB table and returns the following error when we try to use the gateway endpoint:

ERROR: Failed to get item: Invalid DNS Label found in URI host

As soon as I remove the endpoint the task stops failing.

I tried creating a VPC Gateway Endpoint for DynamoDB to alleviate some traffic on our NAT gateway and I expected the ECS task to still be able to connect to DynamoDB through the gateway endpoint.

I would suggest going over the considerations when using a vpc endpoint. Usually it's linked to custom DNS issue.

• A gateway endpoint is available only in the Region where you created it. Be sure to create your gateway endpoint in the same Region as your DynamoDB tables.

• If you're using the Amazon DNS servers, you must enable both DNS hostnames and DNS resolution for your VPC. If you're using your own DNS server, ensure that requests to DynamoDB resolve correctly to the IP addresses maintained by AWS.

• The outbound rules for the security group for instances that access DynamoDB through the gateway endpoint must allow traffic to DynamoDB. You can use the prefix list ID for DynamoDB as the destination in the outbound rule.

• DynamoDB does not support resource-based policies (for example, on tables). Access to DynamoDB is controlled through the endpoint policy and policies for individual users and roles.

• If you use AWS CloudTrail to log DynamoDB operations, the log files contain the private IP addresses of the EC2 instances in the service consumer VPC and the ID of the gateway endpoint for any requests performed through the endpoint.

• Gateway endpoints support only IPv4 traffic.

• The source IPv4 addresses from instances in your affected subnets change from public IPv4 addresses to private IPv4 addresses from your VPC. An endpoint switches network routes and disconnects open TCP connections. The previous connections that used public IPv4 addresses are not resumed. We recommend that you do not have any critical tasks running when you create or modify a gateway endpoint. Alternatively, test to ensure that your software can automatically reconnect to DynamoDB if a connection breaks.

• Endpoint connections cannot be extended out of a VPC. Resources on the other side of a VPN connection, VPC peering connection, transit gateway, or AWS Direct Connect connection in your VPC cannot use a gateway endpoint to communicate with DynamoDB.

• Your account has a default quota of 20 gateway endpoints per Region, which is adjustable. There is also a limit of 255 gateway endpoints per VPC.

It turned out that the container was missing a necessary permission in its role:

> Error message: User:
> arn:aws:sts::XXX:assumed-role/dataSaverContainerRole/XXX is not
> authorized to perform: dynamodb:DescribeEndpoints on resource: *
> because no identity-based policy allows the dynamodb:DescribeEndpoints
> action

Once I added the permission it started working.