OCI instance network throughput below the shape's bandwidth: testing with iperf3, SR-IOV vs paravirtualized and multi-flow tuning

#Goal

A replication stream, backup copy or Spark shuffle runs far below the "Max network bandwidth" listed for the shape. You want to know whether the shortfall comes from the platform, the guest OS or the way you are testing, and how to fix it.

OCI publishes aggregate bandwidth per shape, usually expressed per OCPU with a cap. It does not publish a single-flow limit. One TCP stream almost never fills a large VM, so check with parallel flows before you open a ticket.

#Step 1: Find the bandwidth you are entitled to

Flexible shapes scale bandwidth with OCPUs. A 2-OCPU VM gets much less than the shape's maximum.

oci compute instance get --instance-id <instance-ocid> \
  --query 'data.{shape:shape,ocpus:"shape-config".ocpus,gbps:"shape-config"."networking-bandwidth-in-gbps",vnics:"shape-config"."max-vnic-attachments"}' \
  --output table

oci compute shape list --compartment-id <compartment-ocid> \
  --query 'data[?shape==`VM.Standard.E5.Flex`]."networking-bandwidth-options"' 

The networking-bandwidth-in-gbps value in shape-config is the number to test against. Field names vary slightly between CLI versions. If a key comes back empty, run the command without --query and read the raw output.

Tip

Burstable instances (baseline below 100%) and very small shapes have low bandwidth by design. Resize to more OCPUs before you tune anything else. Bandwidth scales with OCPUs, not with memory.

#Step 2: Test correctly

Use two instances of the same shape and size, in the same VCN. Run the test over private IPs, without a firewall, NAT or load balancer in the path, because they add their own limits.

# both nodes
sudo dnf install -y iperf3
sudo firewall-cmd --add-port=5201-5210/tcp     # temporary, runtime only

# receiver
for p in $(seq 5201 5204); do iperf3 -s -p $p -D; done

# sender: one flow, then many
iperf3 -c <receiver-private-ip> -p 5201 -t 30
iperf3 -c <receiver-private-ip> -p 5201 -t 30 -P 8
# 4 processes x 8 streams, so one iperf3 process isn't the CPU bottleneck
for p in $(seq 5201 5204); do iperf3 -c <receiver-private-ip> -p $p -t 30 -P 8 > /tmp/ip$p.log & done; wait
grep SUM /tmp/ip520*.log | grep receiver

How to read the results:

On a single-OCPU VM, iperf3 itself may be the bottleneck. Watch mpstat -P ALL 1 on both ends while the test runs.

#Step 3: Check and change the networking type

Most VM images launch with hardware-assisted networking (SR-IOV, shown as VFIO in the API). Some imported images, older images and Windows configurations use paravirtualized networking. Paravirtualized networking is more portable and supports live migration better, but it costs CPU on every packet.

oci compute instance get --instance-id <instance-ocid> \
  --query 'data."launch-options".{network:"network-type",boot:"boot-volume-type"}'

# inside the guest
ethtool -i ens3 | grep driver      # mlx5_core = SR-IOV VF, virtio_net = paravirtualized

To switch to hardware-assisted networking:

oci compute instance update --instance-id <instance-ocid> \
  --launch-options '{"networkType":"VFIO"}' \
  --update-operation-constraint ALLOW_DOWNTIME

Warning

Changing the networking type reboots the instance. The guest also needs the VF driver for the NIC (Mellanox mlx5, which recent Oracle Linux kernels ship), or it comes back up with no network. Take a boot volume backup first, and make sure you can reach the serial console. Not every shape and image combination supports both types. Check the shape and image documentation.

#Step 4: Spread receive processing across CPUs

With SR-IOV, the VF exposes several queues. If interrupts for all of them land on one CPU, throughput stalls well below the shape's rated figure.

ethtool -l ens3                   # current vs maximum combined channels
sudo ethtool -L ens3 combined 8   # up to the "Pre-set maximums" value
ethtool -g ens3                   # ring sizes
sudo ethtool -G ens3 rx 4096 tx 4096   # if the max allows; brief link reset
systemctl status irqbalance       # should be active
grep mlx5 /proc/interrupts | head

Make the ethtool settings persistent with a NetworkManager dispatcher script or a udev rule. Test each change with the multi-process iperf3 run from step 2, one change at a time. Driver names and channel maximums vary by shape and kernel, so check your own output rather than copying numbers from another system.

#Step 5: Tune TCP for high bandwidth-delay paths

Within an AD the round-trip time is small, and the default autotuning is usually enough. For cross-region copies over RPC or FastConnect, the window has to cover bandwidth × RTT:

ping -c 20 <remote-ip> | tail -1          # get RTT
sudo tee /etc/sysctl.d/90-net-tuning.conf <<'EOF'
net.core.rmem_max = 134217728
net.core.wmem_max = 134217728
net.ipv4.tcp_rmem = 4096 87380 134217728
net.ipv4.tcp_wmem = 4096 65536 134217728
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr
EOF
sudo sysctl --system
sysctl net.ipv4.tcp_available_congestion_control   # confirm bbr is available on your kernel

Use a 128 MB maximum as a starting point, not a rule. Measure before and after.

#Step 6: Fix the application, not just the host

• Parallelise: use rclone --transfers, oci os object bulk-upload --parallel-upload-count, multiple RMAN channels, or multiple Data Guard or replication streams.
• Add VNICs only when you've hit a per-VNIC limit: the shape's bandwidth is shared across its VNICs, so a secondary VNIC doesn't raise the total.
• Bare metal gives you the whole NIC, and is usually the next step if a large VM still falls short.

#Prevent it next time

• Record the networking type, OCPUs and rated bandwidth in your instance build checklist, and run a baseline iperf3 test when you build a new image.
• Put ethtool and sysctl settings in cloud-init or configuration management, not in a manual runbook.
• Alarm on the oci_vcn VNIC metrics (VnicToNetworkBytes, VnicFromNetworkBytes, and drop metrics where your region publishes them) to spot saturation early. Metric names vary by namespace version, so check the metric explorer.
• When an instance is resized down, note in the change record that its bandwidth drops too.