Home / OCI

OCIRunbookAdvanced

Sev1: Autonomous Database wallet was rotated and every application connection fails

Right after a wallet rotation, applications fail with ORA-29024, ORA-28860 or SSL handshake errors. Confirm the rotation, distribute the new wallet fast, or switch suitable clients to TLS without a wallet, then make rotation a planned change.

Deepak Bhardwaj4 min read

Applies toOracle Autonomous Database Serverless with mTLS connections; JDBC thin, ODP.NET, python-oracledb, SQL*Plus clients

Symptoms

  • Applications that worked a minute ago fail with ORA-29024: Certificate validation failure, ORA-28860: Fatal SSL error, ORA-12506, or Java SSLHandshakeException.
  • All clients that use a downloaded wallet fail at once; the database itself is up.
  • Someone rotated the instance wallet or regional wallet (console: Database connection > Rotate wallet), possibly as a response to a suspected leak.

Step 1: confirm the rotation (2 minutes)

oci db autonomous-database get --autonomous-database-id $ADB \
  --query 'data.{state:"lifecycle-state", mtls:"is-mtls-connection-required"}'

oci db autonomous-database-wallet get --autonomous-database-id $ADB
# regional wallet: console > Autonomous Database > Database connection > Regional wallet (or the CLI regional wallet command for your version)

The wallet details include the rotation state and time. Check the Audit service for RotateAutonomousDatabaseWallet or RotateAutonomousDatabaseRegionalWallet events to see who and when.

Warning

Rotation can be immediate or have a grace period (the console offers a choice on current versions). With immediate rotation, old wallets stop working right away. A regional wallet rotation affects every Autonomous Database in the region that clients reach with the regional wallet.

Step 2: if the rotation was a mistake

You cannot undo a rotation. The only way forward is to distribute the new wallet (step 3) or move clients to TLS (step 4).

Step 3: download and distribute the new wallet

oci db autonomous-database generate-wallet --autonomous-database-id $ADB \
  --password "$WALLET_PW" --generate-type SINGLE --file /secure/wallet_ORDERS.zip

Then, for each application:

Client Where the wallet lives What to do
JDBC thin TNS_ADMIN dir or oracle.net.wallet_location Replace files (cwallet.sso, ewallet.p12, tnsnames.ora, sqlnet.ora, truststore.jks, keystore.jks), restart the app
python-oracledb thin wallet_location / config_dir Replace ewallet.pem and tnsnames.ora, restart
ODP.NET managed WALLET_LOCATION in sqlnet.ora or connection string Replace files, recycle app pool
Containers / Kubernetes Secret mounted as files Update the secret, roll the deployment
OCI Vault-stored wallet Secret version Upload the new version, restart consumers

Kubernetes example:

unzip -o /secure/wallet_ORDERS.zip -d /tmp/wallet
kubectl -n orders create secret generic adb-wallet --from-file=/tmp/wallet \
  --dry-run=client -o yaml | kubectl apply -f -
kubectl -n orders rollout restart deploy/orders-api
rm -rf /tmp/wallet

Test from one host before the fleet:

export TNS_ADMIN=/opt/app/wallet
sqlplus -L admin@orders_high <<< "select sysdate from dual;"

Caution

The wallet zip and its password are credentials. Transfer them through your secrets tooling, not email or chat, and delete local copies after use.

Step 4: move suitable clients to TLS (no wallet)

Autonomous Database Serverless supports one-way TLS for clients that connect through a private endpoint or from an allowed IP/VCN in the access control list. Then no wallet is needed, so wallet rotation cannot break them.

oci db autonomous-database update --autonomous-database-id $ADB --is-mtls-connection-required false

This requires an access control list or private endpoint to be configured. Get the TLS connection string from Database connection > TLS and use it with a modern client (JDBC thin 19.x+ / python-oracledb thin), which trusts the server certificate through the system or JDK CA store.

Step 5: confirm

  • Application health checks green; connection pool errors gone.
  • Database metric CurrentLogons / sessions back to normal.
  • No remaining ORA-29024 or ORA-28860 in application logs.
oci monitoring metric-data summarize-metrics-data --compartment-id $COMP --namespace oci_autonomous_database \
  --query-text 'Sessions[1m]{resourceId = "'"$ADB"'"}.mean()'

Prevent it next time

  • Treat wallet rotation as a change: schedule it, use the grace-period option, and pre-stage new wallets.
  • Store wallets in OCI Vault or your secrets manager and have applications load them at start, so rotation is a secret update plus restart.
  • Prefer TLS with private endpoints for applications inside OCI; keep mTLS for clients that need it.
  • Use instance wallets per database rather than one regional wallet everywhere, to limit the blast radius.

Related fixes