Sev1: traffic stopped after an OCI security list or NSG change — find and revert it fast

#Symptoms

• Apps lose connectivity to databases, load balancers mark backends critical, or SSH stops working, right after a network change.
• Only some instances are affected (those in one subnet or one NSG).

#1. What changed?

oci audit event list --compartment-id $C --start-time $(date -u -d '-6 hours' +%FT%TZ) --end-time $(date -u +%FT%TZ) --all \
  --query 'data[?contains("data"."event-name", `SecurityList`) || contains("data"."event-name", `NetworkSecurityGroup`) || contains("data"."event-name", `RouteTable`) || contains("data"."event-name", `SecurityRules`)].{t:"event-time", e:"data"."event-name", who:"data"."identity"."principal-name", res:"data"."resource-name"}' \
  --output table

UpdateSecurityList replaces the whole rule list — a script or Terraform run with a stale rule set can silently drop rules.

#2. How the rules combine

• A packet is allowed if any security list on the subnet or any NSG on the VNIC allows it (union).
• Both are stateful by default; stateless rules need explicit return-traffic rules.
• Route tables are separate: a missing route looks like a firewall drop.

#3. Inspect current rules

oci network security-list get --security-list-id <sl-ocid> \
  --query 'data.{"in":"ingress-security-rules"[].{src:source, proto:protocol, port:"tcp-options"."destination-port-range"}, "out":"egress-security-rules"[].{dst:destination, proto:protocol}}'

oci network nsg rules list --nsg-id <nsg-ocid> \
  --query 'data[].{dir:direction, src:source, dst:destination, proto:protocol, ports:"tcp-options"."destination-port-range", stateless:"is-stateless"}' --output table

#4. Revert

The fastest correct revert is your IaC: re-apply the last known-good Terraform for that VCN. If the change was manual, add the missing rule back:

oci network nsg rules add --nsg-id <nsg-ocid> --security-rules '[{
  "direction":"INGRESS","protocol":"6","source":"10.0.2.0/24","sourceType":"CIDR_BLOCK",
  "tcpOptions":{"destinationPortRange":{"min":1521,"max":1521}},"description":"app subnet to DB"}]'

Warning

oci network security-list update --ingress-security-rules replaces every ingress rule with what you pass. Export the current rules to JSON, edit, and pass the full list — or prefer NSG rules, which you can add and remove individually.

#5. Verify the path

• Network Path Analyzer (Networking → Network Command Center) tests a source/destination/port across security lists, NSGs and routes, and tells you which element blocks it.
• VCN flow logs show REJECT records for the blocked flow:

search "<compartment>/<log-group>/<flow-log>" | where data.action = 'REJECT' and data.destinationPort = 1521 | sort by datetime desc

#6. Load balancer specifics

LB health checks come from the LB subnet. Backends must allow the health check port from the LB subnet CIDR or the LB's NSG. After fixing, health status may take a few check intervals to turn OK.

#Prevent it next time

• Manage security lists and NSGs only through IaC; deny manage security-lists / manage network-security-groups to operators in production.
• Prefer NSGs (per-resource, additive rules) over large shared security lists.
• Events rule on network rule changes → Notifications channel for the network team.