FastConnect with IPSec VPN as backup: making both sides prefer the same path and avoiding asymmetric routing

#Symptoms

• After adding an IPSec VPN as a backup for FastConnect, some on-premises users can't reach OCI, or sessions drop at random.
• On-premises firewall logs show "no session" / "out of state" drops for OCI traffic.
• Everything works while one of the two links is down.

#How each side chooses

OCI → on-premises. When the DRG learns the same on-premises prefix over FastConnect and IPSec, OCI chooses by its documented route preference:

1. Longest prefix match.
2. For equal prefixes from different connection types, OCI prefers FastConnect over IPSec VPN (see "Routing details for connections to your on-premises network" in the OCI Networking docs for the full order).
3. Among the same type, BGP attributes such as AS path length are considered.

On-premises → OCI. Your router sees the OCI VCN prefixes over both BGP sessions. Unless you configure preference, it may pick the VPN (for example because of how the router breaks ties, or because the VPN's BGP session came up first).

If OCI sends replies over FastConnect while your router sends requests over the VPN, every flow is asymmetric. Stateful firewalls in between, which see only one direction on each path, drop the traffic.

#Step 1: see what OCI learned and prefers

DRG=ocid1.drg.oc1.eu-frankfurt-1.aaaa...

oci network drg-attachment list --drg-id $DRG \
  --query 'data[].{name:"display-name",type:"network-details".type,rt:"drg-route-table-id"}' --output table

# Dynamic routes in the DRG route table used by your VCN attachment
oci network drg-route-table list-drg-route-rules --drg-route-table-id <vcn-attachment-drg-rt> \
  --route-type DYNAMIC --all \
  --query 'data[].{dest:destination,via:"next-hop-drg-attachment-id",conflict:"is-conflict",blackhole:"is-blackhole"}' --output table

For each on-premises prefix, the next hop attachment shows which path OCI uses. Rows flagged as conflicts mean two paths offered the same prefix and one was chosen over the other.

Check link and BGP status:

oci network virtual-circuit get --virtual-circuit-id <vc-ocid> \
  --query 'data.{state:"lifecycle-state",bgp:"bgp-session-state",prefixes:"customer-bgp-peering-ip"}'

oci network ip-sec-tunnel list --ipsc-id <ipsec-ocid> \
  --query 'data[].{id:id,status:status,bgp:"bgp-session-info"."bgp-state",routing:routing}' --output table

#Step 2: see what on-premises prefers

On the router (Cisco IOS-style example; use your platform's equivalent):

show ip bgp 10.50.0.0/16
show ip route 10.50.0.0

If the best path points to the VPN tunnel interface while FastConnect is up, you've found the asymmetry.

#Step 3: make on-premises prefer FastConnect

On the router, raise local preference for routes learned from the FastConnect neighbour:

route-map FROM-OCI-FC permit 10
 set local-preference 200
!
router bgp 65010
 neighbor 169.254.10.1 route-map FROM-OCI-FC in      ! FastConnect BGP peer

Local preference is the first BGP decision step for routes learned from outside the AS, so this reliably selects FastConnect for on-prem → OCI traffic.

#Step 4: make OCI prefer FastConnect for on-premises prefixes

Normally the connection-type preference already does this. Make it explicit and robust:

• Advertise the same prefixes over both links. If you advertise more specific prefixes over VPN than over FastConnect, longest-prefix match sends that traffic over the VPN regardless of type preference.
• AS path prepending on the VPN advertisements (prepend your ASN two or three times towards the IPSec BGP peers) makes the VPN path less attractive where AS path is evaluated.

route-map TO-OCI-VPN permit 10
 set as-path prepend 65010 65010 65010
!
router bgp 65010
 neighbor 169.254.20.1 route-map TO-OCI-VPN out      ! IPSec tunnel BGP peer

#Step 5: test failover both ways

1. With both links up: trace from on-prem to an OCI VM and back (traceroute, mtr), confirm both directions use FastConnect.
2. Shut the FastConnect BGP session (maintenance window). Confirm both directions move to the VPN within your BGP timers.
3. Bring it back. Confirm both directions move back.

Warning

Failover tests drop in-flight sessions. Do them in a planned window and tell application owners.

#Common pitfalls

#Prevent it next time

• Write down the intended path for each direction and the BGP knob that enforces it.
• Keep identical prefix advertisements on primary and backup links.
• Test failover at least twice a year and after router changes.
• Alert on BGP session state for both the virtual circuit and IPSec tunnels.