OCI secondary VNIC or secondary private IP not reachable: configuring the Linux side with policy routing

#Symptom

You attached a secondary VNIC or assigned a secondary private IP. The Console shows it as ATTACHED, and the IP is listed on the VNIC. But:

• ip addr shows the new interface with no IPv4 address, or the secondary IP is missing.
• Pings to the new IP fail from other hosts in the subnet.
• Traffic arrives on the secondary VNIC, but replies leave through the primary VNIC and are dropped.

OCI configures the primary VNIC's primary IP through DHCP. Secondary VNICs and secondary private IPs are not configured in the OS automatically. You have to configure them yourself, and with more than one VNIC you also need policy routing.

#Check the OCI side first

Confirm that the attachment and IPs exist and see which NIC index they map to:

C=ocid1.compartment.oc1..xxxx
I=ocid1.instance.oc1..xxxx

oci compute vnic-attachment list --compartment-id $C --instance-id $I \
  --query 'data[].{vnic:"vnic-id", nic:"nic-index", state:"lifecycle-state"}' --output table

oci network private-ip list --vnic-id <vnic-ocid> \
  --query 'data[].{ip:"ip-address", primary:"is-primary"}' --output table

oci network vnic get --vnic-id <vnic-ocid> \
  --query 'data.{mac:"mac-address", nsgs:"nsg-ids", skipSrcDst:"skip-source-dest-check", subnet:"subnet-id"}'

Then, from inside the instance, read the same data from the instance metadata service (v2). It is the most reliable way to match MAC addresses to IPs and gateways:

curl -s -H "Authorization: Bearer Oracle" http://169.254.169.254/opc/v2/vnics/ | jq
# Per VNIC: macAddr, privateIp, subnetCidrBlock, virtualRouterIp, vnicId, nicIndex

ip -br link

Match each macAddr to an interface in ip -br link. Interface names (ens3, ens5, enp0s5, ...) vary by shape and image. Always match by MAC, never by name.

#Oracle Linux: oci-network-config and ocid

Oracle Linux images on OCI include oci-utils. As root:

sudo oci-network-config show       # VNICs known to OCI vs. configured in the OS
sudo oci-network-config configure  # configure missing VNICs and secondary IPs

configure sets the addresses and the per-VNIC routing tables for you. If show fails with an error about getting an API session, the tool can't call the OCI API. Give it an OCI CLI config for root, or give the instance an instance principal: a dynamic group plus a read-only policy on the instance and VNICs in its compartment. See instance principal and dynamic group fixes.

To have this happen automatically when VNICs are attached or detached later, run the ocid daemon:

sudo systemctl enable --now ocid.service
sudo systemctl status ocid.service

Subcommands and options vary between oci-utils versions. Check sudo oci-network-config --help on your build.

#Manual configuration with NetworkManager (Oracle Linux 8/9, RHEL)

Use this when oci-utils isn't available or you want the configuration managed by your own tooling. Example: the secondary VNIC is ens5 with IP 10.0.3.5/27, and the subnet's virtual router (virtualRouterIp) is 10.0.3.1.

sudo nmcli connection add type ethernet ifname ens5 con-name vnic1 \
  ipv4.method manual ipv4.addresses 10.0.3.5/27 \
  ipv4.never-default yes \
  ipv4.route-table 200 \
  ipv4.routes "0.0.0.0/0 10.0.3.1" \
  ipv4.routing-rules "priority 200 from 10.0.3.5 table 200" \
  ipv6.method disabled

sudo nmcli connection up vnic1

• never-default keeps the main default route on the primary VNIC.
• route-table 200 plus the rule from 10.0.3.5 sends replies from this IP out through ens5, to that subnet's gateway.
• Use a different table number for each extra VNIC.

Verify:

ip -4 addr show ens5
ip rule show
ip route show table 200
ping -I 10.0.3.5 -c 3 10.0.3.1

#Secondary IP on the primary VNIC

Assign the IP in OCI, then add it to the existing DHCP connection. The extra address is static, and the primary address stays on DHCP:

oci network vnic assign-private-ip --vnic-id <primary-vnic-ocid> --ip-address 10.0.0.25

nmcli -g NAME,DEVICE connection show     # find the connection for the primary interface
sudo nmcli connection modify "System ens3" +ipv4.addresses 10.0.0.25/24
sudo nmcli connection up "System ens3"

Use the prefix length of the subnet (subnetCidrBlock in the metadata). The connection name varies by image, so check the connection show output.

Warning

nmcli connection up on the primary interface briefly reapplies its configuration. On a production host, run it from the serial console or during a maintenance window. If you make a mistake on the primary interface, you can lock yourself out of SSH. See recovering with the serial console.

#Ubuntu: netplan

Cloud-init owns /etc/netplan/50-cloud-init.yaml, so put the secondary VNIC in its own file, /etc/netplan/60-vnic1.yaml:

network:
  version: 2
  ethernets:
    vnic1:
      match:
        macaddress: "02:00:17:00:61:01"
      set-name: ens5
      addresses: [10.0.3.5/27]
      routes:
        - to: 0.0.0.0/0
          via: 10.0.3.1
          table: 200
      routing-policy:
        - from: 10.0.3.5
          table: 200
          priority: 200

sudo chmod 600 /etc/netplan/60-vnic1.yaml
sudo netplan try      # rolls back automatically if you lose the session
ip rule show && ip route show table 200

#Moving a secondary IP between instances (floating IP)

For a simple active/passive pair, move the IP to the standby's VNIC, then add it in the standby's OS:

oci network vnic assign-private-ip --vnic-id <standby-vnic-ocid> \
  --ip-address 10.0.0.25 --unassign-if-already-assigned

Caution

The IP moves away from the current holder immediately. Fence the old primary first (stop the service or remove the address in its OS) to avoid both nodes answering for it. Clients may keep stale ARP entries for a short time, so send a gratuitous ARP from the new holder if your cluster software doesn't (arping -U -I ens3 10.0.0.25).

#Prevent it next time

• On Oracle Linux, enable ocid in your base image so attached VNICs and secondary IPs are configured automatically.
• On other images, ship the netplan or nmcli configuration with your configuration management, matched by MAC, and include policy routing from day one.
• Add a check to monitoring: compare /opc/v2/vnics/ with the addresses in ip -4 addr and alert on differences.
• Remember that each VNIC has its own NSGs. Attach the right NSGs when you create the VNIC, not afterwards.
• Check the shape's VNIC limit before planning more interfaces. It scales with OCPU count and varies by shape, so read the current shape documentation.