Home / Tracks / Critical problems
Sev-1 incidents: outages, data-loss risks and security events, with triage, fix and prevention. 148 posts.
26 posts
Pods are evicted in waves with "The node was low on resource: ephemeral-storage" and nodes show DiskPressure. Free space on the affected nodes, find the workload filling the disk, and set ephemeral-storage limits and log rotation so it cannot recur.
Outbound calls from App Service to databases or APIs start timing out at peak, then recover when load drops. Confirm SNAT port exhaustion, stop the bleeding by scaling out and fixing connection reuse, then move egress to a NAT gateway or private endpoints.
Writes fail with error 40544 (database has reached its size quota) while reads still work. Confirm which limit you hit, raise max size or the tier within minutes, then reclaim space and stop it happening again.
Requests fail or slow down with 429 (TooManyRequests) and retry storms make it worse. Tell a real RU shortage from a hot partition, add headroom fast, and fix the partition key, indexing or query that caused it.
Producers get QuotaExceededException or ServerBusyException and orders stop flowing. Find whether the entity is full or the namespace is throttled, restore sending, drain the backlog safely and right-size the namespace.
Restore applications, pipelines and Terraform runs that fail because an app registration's client secret or certificate expired. Find every expiring credential, rotate without a second outage and move to managed identities.
Restore kubectl access to an AKS cluster when the API server times out or rejects connections. Covers authorized IP ranges, private cluster DNS, expired cluster certificates and az aks command invoke as a lifeline.
Recover an AKS cluster or node pool left in a Failed provisioning state after an upgrade. Find the blocking PodDisruptionBudget, quota or subnet issue, reconcile the cluster and finish the upgrade safely.
Triage a backup outage where many VM backup jobs fail at once. Group failures by error code, fix the common causes (VM agent, extension, snapshot limits, Key Vault access, policy changes) and confirm you have a restorable point again.
Fix an outage where Azure Load Balancer stops sending traffic because all backends fail the health probe. Read the probe and data path metrics, check NSGs for the AzureLoadBalancer tag, and fix the probe endpoint.
Bring back a dropped Azure SQL database, or recover from a bad UPDATE or DELETE, using deleted-database restore, point-in-time restore and long-term retention backups, then swap the copy in safely.
Recover deleted or overwritten blobs, a deleted container or a deleted storage account using soft delete, versioning and point-in-time restore, in the right order and without making things worse.
Start a VM protected by Azure Disk Encryption (ADE) that fails because its Key Vault, secret, key or access settings changed. Recover the vault objects, re-enable disk encryption access and plan the move to encryption at host.
Work through an ExpressRoute outage layer by layer: circuit and provider state, BGP sessions, ARP and route tables, the virtual network gateway and the connection, with VPN failover as the fallback.
Recover a deleted Azure Key Vault, secret, key or certificate from soft delete, find who deleted it, and turn on purge protection so the next mistake is a five-minute fix.
Restore log ingestion when a Log Analytics workspace hits its daily cap and alerts go silent. Confirm the cap, find the table or resource that spiked, raise the cap safely and stop the noisy source.
Diagnose and fix intermittent outbound failures caused by Azure Firewall running out of SNAT ports. Read the SNAT utilisation metric, find the noisy workload, add capacity with more public IPs or a NAT gateway, and reduce connection churn.
Restore name resolution when apps that use private endpoints start failing with 403s or timeouts because a private DNS zone, VNet link or A record was removed or changed.
Roll back a bad release on Azure App Service by swapping slots back, understand why sticky settings and warm-up cause post-swap failures, and make swaps safe with preview and health checks.
A realistic recovery plan after a resource group deletion. What Azure can bring back per resource type, the order to rebuild in, how to find who did it, and the locks and policies that stop it happening again.
Respond to a leaked storage account key or SAS token. Rotate keys in the right order, revoke user delegation SAS, check what was accessed, and move apps to Entra ID authentication so keys stop mattering.
Recover when a Conditional Access policy blocks all administrators from the Azure portal and Entra admin center. Uses break-glass accounts, explains what to do if they fail, and how to prevent lockouts with report-only mode and exclusions.
Fix deployments and scale-outs that fail because an Azure subnet ran out of IP addresses. Covers finding the consumer, quick relief options, adding address space without downtime, and moving AKS to an overlay network model.
Recover a production Azure VM stuck at boot using boot diagnostics, the serial console and az vm repair. Covers Windows BCD and pending-update loops, Linux GRUB and fstab errors, and a full OS disk, with a safe rollback path.
Restore service quickly when browsers show certificate errors because an Application Gateway listener certificate expired or Key Vault renewal didn't reach the gateway. Includes the emergency fix, root causes and the controls that stop it happening again.
Respond to a sudden flood of requests that is slowing or breaking a public site. Covers telling attack from legitimate surge, rate limiting, geo and bot rules in Front Door WAF, protecting origins, scaling, and the follow-up hardening.
20 posts
After a bucket policy change, every request (even from AdministratorAccess) returns AccessDenied and the app is down. Use the account root user or the Organizations privileged root action to delete the policy, then rebuild it safely with tests.
Pods cannot resolve service or external names and the whole platform fails at once. Check CoreDNS pods, scheduling and security groups, fix VPC resolver throttling and ndots amplification, and scale DNS so a node loss cannot take it out.
A full root filesystem stops logging, databases and even SSH logins. Get in (or around the OS with a rescue instance), free space safely, grow the EBS volume online, and add alarms so it never fills silently again.
The Application Load Balancer returns 503 because no target passes health checks. Read the health reason codes, separate a broken app from a broken health check or network change, restore traffic, and harden checks so one bad dependency cannot take the site down.
The app fails with "FATAL: sorry, too many clients already" or MySQL error 1040. Get an admin session, kill idle and runaway connections, stop the source of the storm, and put RDS Proxy or proper pooling in front so it does not recur.
Stop an EC2 Auto Scaling group that keeps replacing instances. Read scaling activities, find whether ELB health checks, a short grace period, a bad launch template or user data is the cause, freeze replacements and roll back.
Respond to leaked IAM access keys or a compromised role. Deactivate keys, revoke active sessions, find what the attacker did with CloudTrail, clean up created resources and stop long-lived keys from coming back.
Treat a sudden AWS cost spike as an incident. Find the service, account and usage type with Cost Explorer from the CLI, check the usual suspects (NAT data processing, logs, GPU instances, loops), stop the spend and add anomaly detection and budget actions.
Troubleshoot a Direct Connect outage from the physical connection up through the virtual interface, BGP session, advertised prefixes and the Direct Connect or transit gateway, and confirm VPN failover works.
Contain a compromised EC2 instance without destroying evidence. Isolate with a forensic security group, detach from Auto Scaling and load balancers, revoke the instance role's credentials, snapshot volumes and decide when to rebuild.
Stop DynamoDB throttling that is failing user requests. Tell table-level from partition-level throttling, find hot keys with Contributor Insights, apply quick relief (on-demand, capacity, GSI fixes) and redesign the key so it doesn't come back.
Fix applications that freeze on Amazon EFS. Check burst credit balance and PercentIOLimit, switch throughput mode, verify mount targets and security groups, and recover hung NFS mounts without rebooting everything.
Restore service when unrelated Lambda functions throttle because one function consumed the account's concurrency. Find the culprit, cap it with reserved concurrency, protect critical functions and fix the trigger that caused the spike.
Restore HTTPS on ALB, CloudFront or API Gateway when an ACM certificate expires. Understand why managed renewal failed (DNS validation records, imported certificates, CAA), issue a replacement fast and alert before it happens again.
Restore access to EBS, S3, RDS and Secrets Manager data when a customer managed KMS key is disabled or pending deletion. Cancel the deletion, re-enable the key, find who did it and stop it happening again.
Get back into an EC2 instance when the SSH key is lost, sshd is misconfigured or the Windows Administrator password is unknown. Try Session Manager, EC2 Instance Connect and the serial console first, then the rescue-volume method and SSM automation runbooks.
Regain kubectl access to Amazon EKS after a bad aws-auth ConfigMap edit, a deleted admin role or a creator who left. Use EKS access entries to grant admin from IAM, fix aws-auth, and avoid the single point of failure next time.
Recover an RDS instance or Aurora cluster after deletion or a destructive change. Restore to a point in time, from retained automated backups or a final snapshot, use Aurora Backtrack where enabled, and swap endpoints with minimal app change.
Recover S3 data after an accidental delete or bad overwrite. Remove delete markers in bulk, restore earlier versions, fall back to replicas or AWS Backup, and stop the next incident with Object Lock and MFA delete.
Handle an SQS incident where messages pile up and land in the DLQ. Read the queue metrics, separate poison messages from a broken consumer, fix visibility timeout and batch settings, and redrive the DLQ without duplicating work.
20 posts
A bucket was switched to public or a long-lived pre-authenticated request was shared. Make it private and revoke PARs within minutes, use audit and access logs to learn what was read and by whom, and close the gaps that allowed it.
Right after a wallet rotation, applications fail with ORA-29024, ORA-28860 or SSL handshake errors. Confirm the rotation, distribute the new wallet fast, or switch suitable clients to TLS without a wallet, then make rotation a planned change.
After a database or Grid Infrastructure patch, applications get ORA-12514 (listener does not know of service) or ORA-12541. Check the database and listener, re-register services, start application services that did not come back, and verify with srvctl.
After patching or a reboot, a production instance never comes back and the serial console shows emergency mode. Get a console session, fix /etc/fstab or the missing block volume, boot, and add nofail and tests so one bad mount cannot take a server down.
A node pool cycle or upgrade stops part-way. Old nodes are cordoned and cannot drain, new nodes do not join, and capacity drops below what the workload needs. Unblock drains, fix the new nodes, restore capacity and finish the upgrade safely.
Fix workloads that use instance or resource principals and start failing with 404 NotAuthorizedOrNotFound. Check the dynamic group matching rule, the policy statement and its domain prefix, compartment moves, and the token on the instance.
Restore service when Autonomous Database hits its session limit or runs at 100% CPU. Find the sessions and SQL responsible, kill safely, scale ECPUs or enable auto scaling, and fix connection pools and service choices.
Replace an expired TLS certificate on an OCI flexible load balancer, whether it uses a load balancer certificate bundle or the OCI Certificates service. Update the listener without dropping other settings, then automate renewal and expiry alerts.
Respond to leaked OCI credentials. Identify the credential type, delete it, block the user if needed, search the Audit log for what was done with it, clean up, and move workloads to instance and resource principals.
Recover an Oracle database on an OCI VM DB system that stops because DATA or RECO storage is full. Confirm which area is full, free space safely, scale the DB system storage online and set alarms so it doesn't happen again.
Fix applications that start with an empty data directory because an iSCSI-attached OCI block volume didn't reconnect after a reboot. Reconnect the target, make it automatic, harden fstab, and consider paravirtualized attachments.
Restore name resolution after a public zone record, private zone, private view or VCN resolver change in OCI DNS. Check the record, the resolver's attached views and forwarding rules, revert from Audit details, and lower TTLs for the next change.
Troubleshoot an OCI FastConnect outage from the provider and cross-connect state through the virtual circuit, BGP session, advertised routes and DRG route tables, and confirm IPSec VPN failover takes over.
Restore access after IAM policies were deleted or a policy update replaced its statements. Confirm what's missing with Audit, rebuild statements from IaC or evidence, handle identity domain admin lockout, and keep policies in version control.
Recover an OCI compute instance that fails to boot after a bad fstab entry, kernel update, full disk or broken network config. Detach the boot volume, attach it to a rescue instance, fix it and put it back, with a backup first.
Get deleted or overwritten objects back in OCI Object Storage. Stop the deleter, list object versions, restore previous versions, fall back to a replication target or backups, and use retention rules to block the next accident.
Restore access to block volumes, Object Storage buckets and databases encrypted with a customer-managed OCI Vault key that was disabled or scheduled for deletion. Cancel deletion, re-enable the key, fix IAM, find who did it.
Restore kubectl access to an Oracle Kubernetes Engine cluster. Check the cluster state, public vs. private endpoint, the endpoint NSG and route, and the OCI CLI token that the kubeconfig uses for authentication.
Recover after an OCI compute instance is terminated. Check whether the boot volume was preserved, relaunch from it or from a backup, reattach block volumes, restore the private IP and DNS, and protect instances from the next accidental terminate.
Recover from an outage caused by a security list, network security group or route table edit. Use Audit to find the change, understand how security lists and NSGs combine, revert safely and verify with VCN flow logs and Network Path Analyzer.
20 posts
An UPDATE without the right WHERE clause or a DELETE wiped production data. Find the query ID, clone the table as it was just before that statement, compare, and restore only the damaged rows (or swap the whole table) before the Time Travel window closes.
After a producer release, consumers fail with SerializationException or schema-not-found errors and lag explodes. Identify the bad schema version, stop the producer, get consumers past the bad records, restore compatibility settings and prevent unchecked schema changes.
Someone ran kafka-topics --delete on a live topic. Producers fail or auto-create an empty topic, consumers reset, and data is gone from the brokers. Freeze auto-creation, recreate the topic with the right config, replay from mirrors, sinks or sources, and fix consumer offsets.
After activating or editing an account-level network policy, logins fail with "IP is not allowed to access Snowflake" for users, BI tools and pipelines. Find a path back in, unset or fix the policy, then roll network rules out safely with user-level testing.
A release changed shared code, a requirement or a provider version, and now many DAGs show import errors, vanish from the UI or never get scheduled. Find the broken import, roll back or hotfix, catch up missed runs safely, and add the CI checks that stop it recurring.
Recover an incremental dbt model that lost years of history because someone ran --full-refresh against a source that only keeps recent data. Restore from warehouse time travel, protect the model with full_refresh=false and fix the run process.
Restore an Airflow deployment where no tasks are scheduled because the scheduler is crashed, hung or blocked by its metadata database. Check heartbeats, database health, connections and locks, clean up old metadata safely and bring the backlog through.
Find out why scheduled ADF pipelines stopped running with no failures to alert on. Check trigger runtime state after CI/CD deployments, restart triggers, backfill missed tumbling windows and add a 'no runs' alert.
Get a failing production Spark job through tonight's run. Read the failure in the Spark UI, separate driver OOM, executor OOM, container kills and shuffle fetch failures, apply the quick fix and plan the real one.
When the morning data is late or missing, find which upstream step stalled, communicate, and backfill the missed intervals in Airflow in the right order without overloading sources or creating duplicates.
Recover a Debezium CDC connector that fails because the source purged the binlog or WAL it needed, or its offsets were lost. Choose between incremental snapshots, a filtered re-snapshot or a new connector, and keep consumers consistent.
Revenue doubled overnight because a failed load was re-run on top of a partial one. Find the duplicates, remove them safely in Delta Lake, Snowflake or SQL, and redesign the load with MERGE or partition overwrite so re-runs are always safe.
Fix Event Hubs consumers that keep disconnecting each other, stall on partitions or restart from the wrong position. Check consumer groups, partition ownership in the checkpoint store, client versions and retention.
Restore ADF and Synapse pipelines that fail together because a linked service secret, SAS token, service principal secret or SHIR credential expired. Find the shared credential, rotate it through Key Vault and move to managed identity.
Get a Kafka broker or KRaft controller quorum back after a crash. Read the startup errors, handle long log recovery, corrupted index or segment files, wrong cluster IDs and lost controller quorum without making data loss worse.
Recover when a consumer group suddenly starts from the beginning (duplicates) or from the end (missed data). Find why offsets were lost, stop consumers, reset to the right position by time or saved offsets, and protect committed offsets.
Restore a Kafka cluster where partitions are offline or under-replicated after one or more brokers failed. Find affected partitions, bring brokers back, decide on unclean leader election only as a last resort, and rebalance safely.
Recover Snowflake data after a DROP, TRUNCATE, bad UPDATE or a pipeline that overwrote a table. Use UNDROP, Time Travel AT/BEFORE queries and zero-copy clones, then swap safely and understand Fail-safe limits.
Clear a Synapse dedicated SQL pool where queries pile up in the queue. Find what holds concurrency slots, kill or isolate runaway loads, use workload groups and classifiers to protect BI, and scale if you must.
Recover when a source system renames, retypes or drops columns and pipelines fail or, worse, load nulls silently. Detect the change, quarantine bad batches, adapt the load safely and put schema checks and contracts in place.
20 posts
A misconfigured job or notebook used the production catalog instead of dev, overwriting or appending test data to live tables. Stop writers, find every table it touched from audit and Delta history, restore versions or remove the bad rows, and add catalog isolation so dev compute cannot reach prod.
After an IP access list update, users and integrations get "Source IP address is blocked by Databricks IP ACL". Regain access from an allowed address or through the account console, fix or disable the list, and roll changes out with a tested allow list next time.
The workspace URL times out or shows a blank page for everyone, while jobs may still run. Check name resolution for the workspace and browser-authentication endpoints, fix private DNS zone links and records, and restore front-end connectivity without opening public access permanently.
Overnight jobs fail with INVALID_PARAMETER_VALUE or "validation failed for ... policy" errors after a platform team tightened a cluster policy. Find the failing rule, roll the policy back or relax it, repair runs, and introduce a policy change process that tests existing jobs first.
Queries on managed and external tables fail across all workspaces with 403 AuthorizationPermissionMismatch or "Failed to get credentials". Check the access connector, its managed identity and storage role assignments, restore them, and update the storage credential if the identity changed.
Fix an Auto Loader ingestion that stopped picking up files or fails on every restart. Diagnose schema evolution stops, broken or moved checkpoints, Event Grid notification setup and backlog, and recover without double-loading files.
Fix clusters that terminate during start because they can't reach the Databricks control plane (secure cluster connectivity relay) or Azure dependencies after a firewall, UDR or DNS change. Check required egress, Private Link DNS and NSG rules.
Fix job and cluster launch failures in a VNet-injected Azure Databricks workspace when the host or container subnet has no free IPs. Confirm it from cluster events, free capacity now and resize or redesign the subnets properly.
Recover when job clusters can't get VMs from Azure: quota exceeded for a VM family, regional capacity stockouts, or spot evictions. Read the termination reason, switch node types or capacity type fast, and request quota before the next peak.
Two runs of the same job overlapped, causing duplicate rows or ConcurrentAppendException. Find where the extra run came from (ADF retry, manual run, schedule overlap), clean the data and enforce single-run behaviour with queueing and idempotent writes.
Restore ADF, CI/CD and external tools that call Azure Databricks after an Entra ID client secret, Databricks OAuth secret or PAT expired. Identify which credential, rotate it, update every consumer and move to managed identity or federation.
Fix job and cluster failures caused by library installation: egress to PyPI or Maven blocked, a new dependency version breaking the job, a yanked package or a private feed outage. Pin, mirror and package dependencies so a public repo can't take down production.
Respond to a Databricks PAT found in Git, a notebook output or a log. Revoke the token, find what it was used for in audit system tables, check data access and exfiltration paths, and move people and automation to OAuth.
Treat a sudden Azure Databricks cost jump as an incident. Use system.billing tables to find the workspace, SKU, cluster or job behind it, stop the waste safely and put budgets, policies and auto-termination in place.
Restore BI when a Databricks SQL warehouse fails to start or queues every query. Check warehouse state and events, quota and capacity for classic/pro, scaling limits, runaway queries and Power BI connection storms.
Catch up a Structured Streaming job reading from Azure Event Hubs when input rate exceeds processing rate. Read stream progress, fix partition parallelism, batch sizing, slow sinks and state growth, and catch up safely before retention deletes events.
Fix Delta tables that throw FileNotFoundException because data files referenced by the transaction log were removed by an aggressive VACUUM, a lifecycle rule or a manual delete. Repair the table, recover files if possible and set safe retention.
Fix clusters and jobs that terminate with an init script failure. Find the script's log, check common causes (moved location, blocked downloads, apt locks, bad line endings), hotfix the script and move scripts to Unity Catalog volumes or workspace files.
Regain administration of an Azure Databricks account and Unity Catalog metastore after the only admin left, an admin group was deleted in Entra ID or SCIM removed members. Use Entra Global Administrator bootstrap rights, restore groups and set up resilient admin ownership.
Recover a dropped managed table in Unity Catalog with UNDROP, re-register external tables, restore overwritten data with Delta time travel, and stop cleanup scripts from dropping production objects again.
21 posts
DROP TABLE or TRUNCATE ran on production. Find the exact time from logs, restore a copy of the cluster to just before it with PITR (pgBackRest, base backup plus WAL, or managed-service restore), then dump the table from the copy and load it into production.
The AG failed over and the new primary is healthy, but apps time out or reach the old node. Check listener resource state, DNS records and TTL, MultiSubnetFailover, cluster permissions and cloud load balancer probes, then restore connections quickly.
Secondaries fell behind longer than the oplog window and now report RS102 "too stale to catch up", leaving the primary without redundancy. Grow the oplog, resync the stale members by initial sync or from a snapshot, and protect the window from heavy writes.
After a failover (manual or by Orchestrator, MHA or a script), the old primary came back writable and some clients kept writing to it. Fence the old primary, measure the divergence with GTIDs, reconcile the errant transactions and rebuild it as a replica.
New connections and parses fail with ORA-04031 in the shared pool or large pool. Buy time safely, find what is consuming the pool (literal SQL, high version counts, a leak, large pool users), resize SGA components and fix the application pattern behind it.
Applications get NotWritablePrimary or "no primary found", and rs.status() shows only SECONDARY members. Work out whether a voting majority is reachable, fix network or member problems, and use a forced reconfiguration only when the majority is truly gone.
The replica SQL thread stopped on a duplicate key (1062) or a missing row (1032). Find the failing transaction, decide whether skipping is safe, skip it the GTID way, check for errant transactions that will break the next failover, and verify data with checksums.
mysqld crashes during start-up with InnoDB assertion or corruption errors. Save the data directory first, read the error log, raise innodb_force_recovery one level at a time until it starts, dump the data, and rebuild a clean instance. Or restore from backup if that's faster.
Every insert and update hangs, the error log says MySQL is waiting for disk space, and the data or binlog volume is at 100%. Free space safely by purging only binary logs that all replicas and CDC tools have read, then fix retention so it doesn't happen again.
Applications fail with "no listener", "listener does not currently know of service" or "SID not known". Check the listener is running and listening on the right address, and that the database has registered its services. Then fix tnsnames, LOCAL_LISTENER and firewall problems, with commands for each step.
The only sysadmin login was removed, sa is disabled and its password lost, or the DBA group was deleted from AD. A local Windows administrator can start SQL Server in single-user mode, connect as sysadmin and add a proper admin back. Here are the steps, the traps and the clean-up.
The instance crashes or won't mount because a controlfile copy or an online redo log member is missing or corrupt. What you do depends on which file is gone and whether a good copy exists. The options run from a simple copy to RMAN autobackup restore, CLEAR LOGFILE, or incomplete recovery.
The database mounts but won't open because a datafile needs recovery, is missing or can't be locked. Find the affected file, decide between recover, restore-and-recover or taking the file offline, and use RMAN to bring the database back with no data loss where possible.
patronictl list shows no Leader, every node is a replica or stopped, and applications can't write. The usual causes are a lost etcd quorum, no candidate passing the failover rules, or a paused cluster. Diagnose each one and get a leader back without risking split-brain.
The primary server is down, without Patroni or another manager to fail over. Check the standby's lag, fence the old primary so you don't get two primaries, promote with pg_promote, repoint applications, and rebuild the old primary as a standby with pg_rewind.
PostgreSQL warns that a database must be vacuumed within N transactions, or stops accepting writes altogether. Find what is holding back freezing, such as long transactions, abandoned replication slots or orphaned prepared transactions, remove it, and run a targeted VACUUM to get write access back.
The Redis master is unreachable, clients get connection errors or READONLY, and Sentinel hasn't promoted a replica. Check Sentinel quorum and majority, replica eligibility and announced addresses, then force a failover and fix clients that bypass Sentinel.
A TDE-encrypted database won't restore or attach on another server (error 33111), or the original server was rebuilt. Find the certificate backup, recreate the master key and certificate, restore, and set up certificate backups so this never depends on luck again.
A database shows RECOVERY_PENDING or SUSPECT after a restart or storage problem. Work out which it is and why from the error log. Then bring it online by fixing files and permissions, restoring from backup, or, only as a last resort, using emergency mode repair.
The SQL Server service starts and stops at once, or won't start at all. Find the real error in ERRORLOG, then fix the common causes. Covers a missing tempdb path, damaged or missing master files, account and permission problems, port conflicts and bad startup parameters, using minimal-configuration and single-user starts.
Confirm the damage with DBCC CHECKDB, find out what is corrupt, and recover with an index rebuild or a page restore before you ever reach for REPAIR_ALLOW_DATA_LOSS.
21 posts
Users report the assistant suddenly giving dangerous commands, phishing links or wrong policy answers. Trace answers back to the chunks that caused them, purge the bad documents from every index and cache, close the ingestion path that let them in, and add provenance checks.
A secret-scanning alert or a sudden token spike shows an Azure OpenAI, OpenAI or Bedrock credential is exposed. Revoke and rotate in minutes without a long outage, confirm abuse from usage logs, contain cost, purge the secret from Git history and move to keyless auth.
All AI features fail with 404 DeploymentNotFound after someone deleted, renamed or recreated a model deployment, or an IaC run replaced it. Recreate the deployment with the same name, model version and capacity, restore quota, and stop drift with locks and code.
A wave of spot or preemptible evictions removed most GPU nodes, model pods are Pending and replacements cannot get capacity. Fail traffic over, get any GPU capacity you can (other sizes, zones, on-demand), and redesign so spot can never take the whole service down.
After a config change, model upgrade or new GPU node type, vLLM fails at startup or under load with torch.OutOfMemoryError or "No available memory for the cache blocks". Restore service with known-good settings, find what changed in the memory budget, and stop it recurring.
Someone got your assistant to reveal its system prompt, ignore its rules, leak data from retrieved documents or call tools it shouldn't. Contain it, find every conversation affected, judge what was exposed or done, then close the specific hole and the class of problem.
An autonomous agent is deleting records, sending emails or changing systems it shouldn't. Stop new runs, halt runs in progress, cut its tool credentials, contain the damage, then work out what happened from traces before anything is switched back on.
After public network access was disabled, a hub-spoke change or a DNS server move, calls to Azure OpenAI fail with 403 "public access is disabled", timeouts or name resolution errors. Check what the name resolves to from the app's network, fix the private DNS zone links or forwarders, and test from each spoke.
Indexer runs fail with storage or quota errors, vector uploads are rejected, and the RAG assistant stops learning new content. Confirm which limit you hit (total storage, vector index size or document count), buy time, then add partitions, shrink the vector footprint or move to a bigger tier.
Streaming worked in development, but in production users see nothing for 30 seconds and then the whole answer, or the stream stops halfway with a timeout. Find the hop that buffers or times out (NGINX, Application Gateway, APIM, Front Door, corporate proxy) and fix buffering and idle timeouts.
Inference demand rises but new GPU VMs fail to start because the region or zone has no capacity for that size, or you've hit quota. Get capacity from other zones, sizes or regions now, cut demand per request, and set up reservations so baseline capacity is guaranteed.
Inference pods crash, CUDA calls fail, and nvidia-smi errors or hangs. Read the Xid codes in the kernel log, tell software faults from hardware ones, drain and reset or replace the node, and keep the rest of the fleet serving.
New model replicas never start and the scheduler says no node has GPUs, even though GPU nodes exist. Check whether GPUs are advertised, whether taints and selectors match, whether the device plugin or GPU operator is broken, and whether the node pool can scale at all.
Production calls suddenly return deployment or model-not-found errors, or behave differently overnight. The model version reached its retirement date or was auto-upgraded. Restore service on a supported version fast, then put lifecycle tracking in place so it doesn't surprise you again.
Requests to a custom model endpoint fail with 408/504 or upstream timeouts in the morning or during traffic spikes, then recover. New instances are loading a large model while traffic is already routed to them, or the endpoint scaled in too far. Fix minimum capacity, timeouts, readiness checks and autoscale rules.
Someone notices API keys, passwords or customer personal data in prompt logs, tracing tools or evaluation datasets. Stop new capture, rotate exposed secrets, find and purge every copy (logs, traces, caches, evaluation sets, provider-side storage), and fix the pipeline so sensitive data is redacted before it's written.
After a re-index, pipeline change or data refresh, the assistant says "I don't know" far more often or cites the wrong documents. Compare retrieval before and after, find the change (chunking, embeddings, filters, missing documents), roll back with an index alias, and add a retrieval gate to the pipeline.
A pipeline that parses model output (ticket classification, extraction, routing) starts failing on JSON parse or schema validation errors. Find whether it's truncation, model change, prompt drift or input size, switch to schema-enforced output, and add validation, retries and a dead-letter path.
Users wait seconds for the first word, the request queue grows, and GPU utilisation looks fine. Separate queueing from prefill cost from KV cache pressure using vLLM metrics, then apply the fix that matches, whether that's scaling out, limiting long prompts, chunked prefill or prefix caching.
Qdrant, Milvus or Weaviate pods restart with OOMKilled after the collection grew or a bulk load ran. Get search back by adding memory or moving vectors to disk, slow the ingestion down, then size memory properly with quantization and on-disk indexes.
Search throws dimension errors, or still works but returns irrelevant results, after someone changed the embedding deployment, model version or dimensions setting. Find which side changed, restore matching query and document embeddings, and lock the embedding config to the index.