Home / AWS

AWSTroubleshootingAdvanced

EC2 packet drops with no obvious cause: ENA bandwidth, PPS, conntrack and link-local allowance exceeded

Retransmits, DNS timeouts or dropped connections on busy EC2 instances while CPU and bandwidth graphs look fine. Read the ENA allowance counters, find which limit you hit and fix it with sizing, connection tracking changes or caching.

Deepak Bhardwaj4 min read

Applies toAmazon EC2 Nitro instances with the ENA driver (Linux and Windows), CloudWatch agent, security groups connection tracking

Symptoms

  • TCP retransmits, latency spikes or random connection failures on a busy instance (proxy, NAT instance, load generator, database).
  • DNS lookups occasionally time out on that instance only.
  • CloudWatch NetworkIn/NetworkOut are below the instance's advertised bandwidth.

Every EC2 instance has allowances beyond headline bandwidth: packets per second, number of tracked connections, and link-local traffic (DNS, IMDS, NTP). When one is exceeded, packets are queued or dropped silently. The ENA driver counts each case.

Step 1: read the counters

Linux:

ethtool -S eth0 | grep -E "allowance_exceeded|conntrack_allowance_available"

bw_in_allowance_exceeded: 0
bw_out_allowance_exceeded: 18234
pps_allowance_exceeded: 0
conntrack_allowance_exceeded: 4211
linklocal_allowance_exceeded: 960
conntrack_allowance_available: 120

Windows (PowerShell, ENA driver 2.2+ exposes them as adapter statistics; see the ENA Windows driver docs for your version):

Get-NetAdapterStatistics -Name "Ethernet" | Format-List *

The counters are cumulative since driver load. Take two readings a minute apart; what matters is whether they increase during the problem.

Counter Limit exceeded
bw_in_allowance_exceeded / bw_out_allowance_exceeded Aggregate bandwidth (including bursts above the baseline for burstable network sizes)
pps_allowance_exceeded Packets per second; small packets hit this before bandwidth
conntrack_allowance_exceeded Number of tracked connections for the instance size
linklocal_allowance_exceeded PPS to link-local services: Route 53 Resolver, IMDS, Amazon Time Sync
conntrack_allowance_available Remaining tracked-connection capacity (newer drivers)

Step 2: collect them in CloudWatch

CloudWatch agent config (Linux):

{
  "metrics": {
    "append_dimensions": { "InstanceId": "${aws:InstanceId}" },
    "metrics_collected": {
      "ethtool": {
        "interface_include": ["eth0"],
        "metrics_include": [
          "bw_in_allowance_exceeded", "bw_out_allowance_exceeded",
          "pps_allowance_exceeded", "conntrack_allowance_exceeded",
          "linklocal_allowance_exceeded", "conntrack_allowance_available"
        ]
      }
    }
  }
}

Alarm on the rate of change (a metric math RATE() or the difference between periods), not the absolute value.

Step 3: fix by counter

Bandwidth (bw_*)

  • Instances "up to X Gbps" have a baseline and burst using credits. Sustained traffic above baseline gets shaped. Check the baseline for your type in the EC2 network performance docs.
  • Fix: larger size, a network-optimised family (the n variants), or spread across instances.
  • Single-flow limits apply too (a single TCP flow is capped lower than aggregate outside a cluster placement group); use parallel streams for bulk transfers.

Packets per second

  • Typical for DNS servers, VoIP, game servers, small-message proxies.
  • Fix: larger instance, enable multiple queues (default with ENA), larger messages/batching, or ENA Express where supported.

Connection tracking

Security groups track connections so replies are allowed back. Each instance size can track a limited number.

  • Connections are untracked (and don't count) when the security group rules allow the traffic in both directions without restriction, for example inbound 0.0.0.0/0 on the port and outbound 0.0.0.0/0 all traffic. Some traffic is always tracked (ICMP, and flows matched by restrictive rules).
  • Many idle or half-closed connections fill the table. Reduce idle timeouts per ENI:
aws ec2 modify-network-interface-attribute --network-interface-id eni-0abc \
  --connection-tracking-specification TcpEstablishedTimeout=600,UdpStreamTimeout=60,UdpTimeout=30
  • On the OS: close idle connections in the app, enable keepalive, and tune connection pools.
  • For high-connection front ends (proxies, NAT instances), larger instances or a load balancer in front.

Warning

Changing security group rules to "allow all" to make connections untracked is a security decision, not just a performance tweak. Only do it where an upstream control (NLB, firewall, NACLs) restricts traffic.

The Amazon-provided DNS resolver, IMDS and Time Sync share a per-ENI packet rate limit (documented as 1024 packets per second per ENI for these services). Busy proxies or crawlers that resolve a name per request exceed it.

  • Run a local caching resolver (systemd-resolved, dnsmasq, unbound) and point the OS at it.
  • Cache IMDS credentials (SDKs already do; custom scripts often don't).
  • Reuse connections in the application.

Step 4: confirm the fix

Re-run ethtool -S eth0 | grep allowance during peak load. The counter you fixed should stop increasing. Watch application latency and TCP retransmits (nstat -az TcpRetransSegs).

Prevent it next time

  • Collect the ENA allowance counters on every production instance by default.
  • Load test with realistic connection counts and packet sizes, not just throughput.
  • Size network-heavy roles (proxies, NAT, DNS) by PPS and connections, not CPU.
  • Use local DNS caching on every instance that makes many outbound calls.

Related fixes