Azure Private Link Service troubleshooting: connections stuck Pending, NAT IP pressure and PROXY protocol breaking the backend

#Symptom

You publish an internal service through a Private Link Service (PLS) behind a Standard internal load balancer. A consumer in another VNet, subscription or tenant creates a private endpoint against it, and one of these happens:

• The endpoint shows Pending and never connects.
• TCP connects but the backend resets it straight away, or health probes start failing after a change.
• It works under light load. At peak, new connections fail or stall while existing ones carry on.
• Long-lived connections (database sessions, message brokers) drop after about five minutes of idle time.
• The backend logs show every client arriving from the same few IPs in your subnet, so per-client rate limits and allow-lists behave badly.

All of these come from how PLS works. Consumer traffic is source-NATed to NAT IPs taken from a subnet in your VNet, then sent to the load balancer frontend. The provider never sees the consumer's real IP unless PROXY protocol is turned on.

#Quick map

#Check 1: connection state and visibility

List the PLS and its endpoint connections:

RG=rg-provider
PLS=pls-orders-api

az network private-link-service show -g $RG -n $PLS \
  --query "{alias:alias, visibility:visibility.subscriptions, autoApproval:autoApproval.subscriptions, proxy:enableProxyProtocol}" -o jsonc

az network private-link-service show -g $RG -n $PLS \
  --query "privateEndpointConnections[].{name:name, state:privateLinkServiceConnectionState.status, desc:privateLinkServiceConnectionState.description, linkId:linkIdentifier}" -o table

If the state is Pending, approve it. Take the connection name from the output above:

az network private-link-service connection update -g $RG --service-name $PLS \
  -n "<endpoint-connection-name>" \
  --connection-status Approved \
  --description "CHG-12345 approved for orders consumer"

--connection-status accepts Approved, Rejected and Removed. A rejected or removed connection doesn't come back on its own. The consumer has to delete the private endpoint and create a new one.

For a known partner, pre-approve their subscription so later endpoints don't sit in the queue:

az network private-link-service update -g $RG -n $PLS \
  --visibility <consumer-sub-id> \
  --auto-approval <consumer-sub-id>

Warning

--visibility and --auto-approval replace the whole list. Pass every subscription that should stay on it, or you'll hide the service from existing consumers. Existing approved connections keep working, but those consumers can't create new endpoints until you fix the list.

The consumer side should show Approved too:

az network private-endpoint show -g rg-consumer -n pe-orders \
  --query "manualPrivateLinkServiceConnections[0].privateLinkServiceConnectionState.status || privateLinkServiceConnections[0].privateLinkServiceConnectionState.status"

#Check 2: the NAT subnet

PLS NAT IPs come from a subnet you choose. That subnet must have Private Link service network policies disabled. This setting is separate from the private endpoint network policies setting:

az network vnet subnet show -g $RG --vnet-name vnet-provider -n snet-pls-nat \
  --query "{plsPolicies:privateLinkServiceNetworkPolicies, prefix:addressPrefix}"

az network vnet subnet update -g $RG --vnet-name vnet-provider -n snet-pls-nat \
  --private-link-service-network-policies Disabled

Older CLI builds used --disable-private-link-service-network-policies true. Names vary by version, so check az network vnet subnet update --help.

Keep the NAT subnet small and dedicated. Don't share it with VMs that have their own NSG and UDR expectations. NAT IPs must be able to reach the LB frontend and the backends, and backend NSGs must allow the NAT subnet's prefix as a source. The AzureLoadBalancer service tag alone isn't enough here.

#Check 3: NAT IP capacity at peak

Each NAT IP provides a pool of source ports toward the backend. Every consumer connection uses one. A busy consumer with connection-per-request clients, or many consumers behind one PLS, can use up those ports. The pattern is the same as SNAT exhaustion elsewhere: new connections fail, existing ones are fine.

Microsoft allows up to eight NAT IPs per PLS and recommends keeping at least eight free addresses in the NAT subnet. Check what you have:

az network private-link-service show -g $RG -n $PLS \
  --query "ipConfigurations[].{name:name, ip:privateIPAddress, primary:primary, subnet:subnet.id}" -o table

Add one:

az network private-link-service update -g $RG -n $PLS \
  --ip-configurations '[
    {"name":"nat-1","primary":true,"subnet":{"id":"<nat-subnet-id>"},"privateIPAllocationMethod":"Dynamic"},
    {"name":"nat-2","primary":false,"subnet":{"id":"<nat-subnet-id>"},"privateIPAllocationMethod":"Dynamic"}
  ]'

Include the existing configurations in the array. This is a full replace. You must always keep at least one NAT IP.

The PLS resource exposes metrics in Azure Monitor. Metric names vary by API version, so list them before you build an alert:

PLS_ID=$(az network private-link-service show -g $RG -n $PLS --query id -o tsv)
az monitor metrics list-definitions --resource $PLS_ID --query "[].name.value" -o tsv

Chart the NAT port usage metric (if your build exposes one) and the bytes in/out at the time of the failures. If NAT ports peak when connections fail, the fix is more NAT IPs plus connection pooling on the consumer side. Adding backends doesn't help.

#Check 4: the idle timeout

PLS has an idle timeout of about 300 seconds. Clients that hold a TCP session open without traffic, such as connection pools, AMQP links and replication streams, get dropped silently. The next write then fails or hangs until retransmits give up.

Set TCP keepalive below 300 seconds on whichever end you control. On a Linux backend:

sudo sysctl -w net.ipv4.tcp_keepalive_time=240
sudo sysctl -w net.ipv4.tcp_keepalive_intvl=30
echo -e "net.ipv4.tcp_keepalive_time=240\nnet.ipv4.tcp_keepalive_intvl=30" | sudo tee /etc/sysctl.d/90-pls-keepalive.conf

Kernel keepalive only applies to sockets that set SO_KEEPALIVE. Many drivers need a setting of their own, such as a JDBC tcpKeepAlive=true or a pool's validation interval. Check the client library.

#Check 5: PROXY protocol v2

With enableProxyProtocol set, PLS adds a PROXY protocol v2 header to every new TCP connection, health probes included. The header carries the consumer's source address and an Azure TLV (type 0xEE, subtype 0x01) with the private endpoint's linkIdentifier as a 4-byte little-endian integer. That's how you tell consumers apart.

The setting has to match the backend exactly:

• Enabled on the PLS but the backend doesn't expect it: the backend reads the header as garbage, and TLS or HTTP parsing fails at once.
• Backend expects it but the PLS doesn't send it: the backend waits for a header that never arrives.
• Several PLS resources share one load balancer or backend pool: PROXY protocol must be set the same way on all of them.

Caution

Turning PROXY protocol on or off for a live PLS breaks every connection, and the health probes, until the backend config changes to match. Do it in a maintenance window with the backend change staged. Stand up a separate PLS and LB for the cut-over if you can.

NGINX backend example:

server {
    listen 8443 ssl proxy_protocol;
    set_real_ip_from 10.20.30.0/28;   # PLS NAT subnet
    real_ip_header proxy_protocol;
    # ...
}

To map a linkIdentifier back to a consumer, take it from the privateEndpointConnections output in Check 1. Log it at the backend so you can attribute traffic and apply per-consumer limits.

#Check 6: load balancer constraints

PLS works only with a Standard Load Balancer. The backend pool must be NIC-based, not IP-based. PLS is IPv4 only, carries TCP and UDP only, and must be in the same region as the LB and VNet. If you use an inbound NAT rule against a backend pool, a load-balancing rule must exist too. When the consumer sees nothing at all, check that the LB rule's probe is healthy on its own before you suspect PLS:

az network lb show -g $RG -n ilb-orders \
  --query "{sku:sku.name, pools:backendAddressPools[].name, rules:loadBalancingRules[].{fe:frontendIPConfiguration.id, port:frontendPort, probe:probe.id}}" -o jsonc

#Prevent it next time

• Manage visibility and autoApproval in IaC. Never edit them by hand with partial lists.
• Start with at least two NAT IPs, and alert on NAT port usage before it reaches capacity.
• Write the roughly 300-second idle timeout into the consumer onboarding doc, along with the keepalive settings.
• Decide on PROXY protocol at design time, use the same setting on every PLS that shares an LB, and log the linkIdentifier.
• Keep a runbook entry that maps each linkIdentifier and connection name to an owner, so a reject or remove is never a guess.