Packet capture on Azure VMs with Network Watcher: prove where a connection fails

#When to reach for packet capture

Use it when the configuration says the traffic should work but it doesn't:

• Connections time out intermittently, and NSG flow logs show the flow as allowed.
• Clients see connection reset and the app team says "it's the network".
• TLS handshakes fail between two VMs, or through an NVA or Azure Firewall.
• A vendor asks for a capture to prove packets left your side.

A capture answers three questions that config checks can't: did the SYN leave the client, did it reach the server, and who sent the RST or stopped answering.

Run the quick checks first. If one of them already explains the failure, you don't need a capture:

RG=rg-prod-app
VM=vm-app-01

# Can the VM reach the destination at all, and where does it stop?
az network watcher test-connectivity -g $RG --source-resource $VM \
  --dest-address 10.20.1.4 --dest-port 443

# Where is the traffic routed?
az network watcher show-next-hop -g $RG --vm $VM \
  --source-ip 10.10.1.4 --dest-ip 10.20.1.4

#Prerequisites

Install the extension. Use NetworkWatcherAgentWindows for Windows VMs:

az vm extension set -g $RG --vm-name $VM \
  --publisher Microsoft.Azure.NetworkWatcher \
  --name NetworkWatcherAgentLinux

Tip

If the storage account has a firewall or only allows private endpoints, the capture upload from the VM can fail even though the capture itself ran. Test with a storage account the VM can definitely reach, or write to a local file with --file-path and pull it off the VM afterwards. On Linux the path must start with /var/captures.

#Step 1: Start a filtered capture on both ends

If you capture on one side only, you learn half the story. Start a session on the client VM and on the server VM at the same time, with the same filter.

SA=stnetcapprod01           # storage account in the same region
LOC=westeurope

az network watcher packet-capture create -g $RG -n cap-client-$(date +%H%M) \
  --vm vm-app-01 --storage-account $SA \
  --time-limit 900 --capture-size 256 \
  --filters '[{"protocol":"TCP","remoteIPAddress":"10.20.1.4","remotePort":"443"}]'

az network watcher packet-capture create -g rg-prod-data -n cap-server-$(date +%H%M) \
  --vm vm-db-01 --storage-account $SA \
  --time-limit 900 --capture-size 256 \
  --filters '[{"protocol":"TCP","remoteIPAddress":"10.10.1.4","localPort":"443"}]'

Key parameters:

For a scale set, add --target-type AzureVMSS --target <vmss-name> and limit the instances with --include "0" "1".

Caution

Captures can hold credentials, tokens and customer data from unencrypted protocols. Truncate packets with --capture-size, keep the storage account private, and delete the files when the incident is closed.

#Step 2: Reproduce, then stop

Reproduce the failure while both sessions run, and write down the exact client-side timestamp of a failed attempt. Then stop both sessions so the files are written:

az network watcher packet-capture list -l $LOC -o table

az network watcher packet-capture show-status -l $LOC -n cap-client-1412 \
  --query "{status:packetCaptureStatus, reason:stopReason}"

az network watcher packet-capture stop -l $LOC -n cap-client-1412
az network watcher packet-capture stop -l $LOC -n cap-server-1412

#Step 3: Download the files

Unless you set --storage-path, files go to the network-watcher-logs container:

az storage blob list --account-name $SA -c network-watcher-logs \
  --auth-mode login --query "[].name" -o tsv | grep -i vm-app-01

az storage blob download --account-name $SA -c network-watcher-logs \
  --auth-mode login -n "<blob name from the list>" -f ./client.cap

#Step 4: Read the capture

Open both files in Wireshark, or use tshark for a quick scan:

# All connection attempts (SYN without ACK)
tshark -r client.cap -Y "tcp.flags.syn==1 && tcp.flags.ack==0" \
  -T fields -e frame.time -e ip.src -e ip.dst -e tcp.dstport

# Who sends resets?
tshark -r client.cap -Y "tcp.flags.reset==1" -T fields -e frame.time -e ip.src -e ip.dst

# Retransmissions and zero windows
tshark -r server.cap -Y "tcp.analysis.retransmission || tcp.analysis.zero_window"

Compare the two sides for the failed timestamp:

The SYN-without-a-reply and reset rows are where you get proof: the same packet appears in one file and not the other, so you know which hop dropped or answered it.

#Quick alternative: tcpdump through Run Command

When you can't install the extension (for example, a change freeze on extensions), a short tcpdump through Run Command can still show whether packets arrive. Run Command has an execution time limit, so keep the capture short:

az vm run-command invoke -g $RG -n vm-db-01 --command-id RunShellScript \
  --scripts "timeout 60 tcpdump -i eth0 -nn -c 200 'host 10.10.1.4 and port 443'"

The output comes back as text, which is enough to see whether SYNs arrive. Use the Network Watcher capture when you need a file to analyse.

#Clean up

az network watcher packet-capture delete -l $LOC -n cap-client-1412
az network watcher packet-capture delete -l $LOC -n cap-server-1412

Deleting the session doesn't delete the capture file. Remove the blobs yourself, or let a lifecycle rule delete them.

#Prevent it next time

• Pre-install the Network Watcher agent in your VM baseline (image or IaC), so a capture during an incident doesn't wait on an extension install.
• Keep one capture storage account per region with no public access, RBAC-only access and a lifecycle rule that deletes network-watcher-logs blobs after 7–14 days.
• Turn on flow logs (VNet flow logs) for production VNets. They show whether a flow was allowed or denied before you need a capture.
• Document the paths for each critical flow (client, NVA or firewall, server) in the runbook, so you know which VMs to capture on at 3 a.m.
• Concurrent capture sessions per region are limited. Check the current Network Watcher limits before you script captures across a whole fleet.