Symptom
The connection opens, then stalls as soon as a large packet is sent:
sshlogs in, butscp,git cloneorrsyncfreezes after a few KB.- TLS clients hang at "Client Hello" or after the server certificate is sent.
SELECT 1works over the VPN, but a query returning a large result set never returns.- The same workload is fine between two instances in one VCN and breaks to on-prem or the internet.
This is almost always a path MTU problem. Small packets get through and full-size packets with the Don't Fragment (DF) bit set are dropped somewhere along the path. The "Fragmentation Needed" message that should tell the sender to shrink them either never gets sent or gets filtered.
Why OCI hits this more than most clouds
Most OCI platform images set the primary VNIC MTU to 9000, and traffic inside a VCN (and between peered VCNs in the same region) carries jumbo frames. Paths leaving the VCN are smaller:
| Path | Effective MTU | Notes |
|---|---|---|
| Inside a VCN, instance to instance | 9000 | Default on most images; check ip link |
| Internet Gateway / NAT Gateway | 1500 | Packets over 1500 sent toward the internet are dropped |
| Site-to-Site VPN (IPSec) | Less than 1500 | ESP and tunnel headers take space; the exact value depends on cipher and NAT-T |
| FastConnect | 1500 by default | Some virtual circuit setups support jumbo MTU; check the virtual circuit and provider settings |
| On-prem firewalls and SD-WAN | Often 1400 or less | GRE, overlay or PPPoE overhead |
TCP handles this through Path MTU Discovery (PMTUD): a router that can't forward a DF packet replies with ICMP type 3 code 4, and the sender lowers its path MTU. If anything blocks that ICMP message (a security list, an NSG, an on-prem firewall or a CPE), the sender keeps retransmitting 9000-byte or 1500-byte segments that never arrive. That is a PMTUD black hole.
Tip
The default security list that OCI creates with a VCN includes an ingress rule for ICMP type 3 code 4 from
0.0.0.0/0. Custom security lists, NSG-only designs and landing-zone templates often leave it out. Check what is actually attached to the subnet instead of assuming the default.
Checks
1. Confirm the instance MTU
ip link show | grep -E "^[0-9]+:|mtu"
ip route get 10.50.1.20 # shows the interface and any per-route mtu
On Windows:
netsh interface ipv4 show subinterfaces
2. Probe the path with DF set
On Linux, the ICMP payload is MTU minus 28 bytes (20-byte IP header and 8-byte ICMP header).
# Inside the VCN: should succeed at 9000
ping -M do -s 8972 -c 3 10.0.2.15
# Toward on-prem or the internet: walk down until it works
for s in 8972 1472 1440 1400 1372 1360; do
echo -n "$s: "; ping -M do -s $s -c 1 -W 2 10.50.1.20 >/dev/null && echo ok || echo fail
done
# tracepath shows the hop where the PMTU drops (if ICMP gets back)
tracepath -n 10.50.1.20
Windows equivalent: ping -f -l 1472 10.50.1.20.
Read the results like this:
| Result | Meaning |
|---|---|
Large ping fails, Frag needed and DF set (mtu = N) returned | PMTUD works; the sender should adapt. Look elsewhere if TCP still hangs |
| Large ping fails silently, small succeeds | Black hole: ICMP 3/4 is blocked or never generated |
tracepath shows pmtu 1500 then no reply | The drop is past the hop where PMTU was last reported |
3. Look at the TCP MSS actually in use
ss -tin dst 10.50.1.20 | grep -oE "mss:[0-9]+|pmtu:[0-9]+"
An MSS of 8948 on a flow to on-prem means the instance never learned a smaller path MTU.
4. Check security rules for ICMP type 3
oci network security-list get --security-list-id $SL \
--query "data.\"ingress-security-rules\"[?protocol=='1']"
# protocol is a string: "1" = ICMP, "all" = every protocol (which also covers ICMP)
oci network nsg rules list --nsg-id $NSG --all \
--query "data[?protocol=='1' || protocol=='all'].{dir:direction,src:source,icmp:\"icmp-options\"}"
Also check the on-prem side: the CPE, perimeter firewall and any host firewall must allow ICMP type 3 code 4 (and ICMPv6 type 2 for IPv6) toward the OCI CIDRs.
5. Capture it
sudo tcpdump -ni ens3 'icmp[icmptype]==3 or (tcp and host 10.50.1.20 and greater 1400)'
Repeated retransmits of the same large segment with no ICMP coming back confirm the black hole.
Fixes
Use them in this order. The first fixes the cause; the others contain it.
Allow ICMP type 3 code 4
Add it to the NSG (adding NSG rules is incremental):
oci network nsg rules add --nsg-id $NSG --security-rules '[{
"direction":"INGRESS","protocol":"1","source":"0.0.0.0/0","sourceType":"CIDR_BLOCK",
"isStateless":false,"icmpOptions":{"type":3,"code":4},
"description":"PMTUD - fragmentation needed"}]'
Warning
oci network security-list update --ingress-security-rulesreplaces the whole rule list. Export the current rules withsecurity-list get, add the ICMP rule to that JSON and pass the full list back. If you pass only the new rule, every other ingress rule is deleted and traffic stops.
Clamp MSS on the VPN or edge device
For IPSec, clamping TCP MSS on the CPE (or on the OCI-side firewall appliance in a hub VCN) is the most reliable fix, because it doesn't depend on ICMP. Use the value your CPE vendor recommends for IPSec with your cipher. A common approach is to clamp to the path MTU on Linux-based routers:
# On a Linux router or firewall VM forwarding traffic to the tunnel
sudo iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN \
-j TCPMSS --clamp-mss-to-pmtu
Lower MTU per route, not per interface
Dropping the interface to 1500 also costs jumbo-frame throughput inside the VCN (database replication, backup and storage traffic). Set a lower MTU only on routes that leave the VCN:
# On-prem range over VPN: lock MTU at 1400
sudo ip route add 10.50.0.0/16 via 10.0.1.1 dev ens3 mtu lock 1400
Make it persistent with NetworkManager:
sudo nmcli connection modify "System ens3" +ipv4.routes "10.50.0.0/16 10.0.1.1 mtu=1400"
sudo nmcli connection up "System ens3"
The subnet's default gateway is the first usable address in the subnet (for example 10.0.1.1). Connection names vary by image; check nmcli connection show.
Lower the interface MTU (last resort)
For a host that only talks to the internet or on-prem (a NAT'd proxy, for example), set the whole interface to 1500:
sudo nmcli connection modify "System ens3" 802-3-ethernet.mtu 1500
sudo nmcli connection up "System ens3"
On Windows: netsh interface ipv4 set subinterface "Ethernet" mtu=1500 store=persistent (the interface name varies).
Tricky cases
- UDP and fragmented packets. PMTUD doesn't help UDP applications that send large datagrams (some DNS-over-UDP responses, syslog, VXLAN overlays). Fragments sent toward the internet over 1500 bytes are dropped. Make the application send smaller datagrams or use TCP.
- Kubernetes overlays. Flannel on OKE adds VXLAN overhead on top of the node MTU; VCN-native pod networking does not. If pods on Flannel clusters hang to on-prem but nodes don't, compare pod and node MTU.
- Network Firewall or third-party NVA in the hub. The appliance becomes the device that must generate ICMP 3/4 or clamp MSS. Check its MTU and clamping settings after every policy or version change.
- Asymmetric paths. If FastConnect is primary and VPN is backup, PMTU learned on one path is wrong after failover. MSS clamping on both paths avoids surprises.
Prevent it next time
- Put ICMP type 3 code 4 in every security list and NSG baseline, in Terraform, so new subnets get it.
- Add a scripted probe (
ping -M do -s 1372and-s 1472to on-prem targets) to your post-change network checks for DRG, VPN, FastConnect and firewall changes. - Clamp MSS on every IPSec edge and document the value next to the tunnel configuration.
- Keep 9000 MTU inside the VCN for throughput, and use per-route MTU for paths that leave it.