Home / Azure

AzureTroubleshootingAdvanced

Sev1: Azure Cache for Redis timeouts take the app down — server load, memory pressure and client fixes

The app is throwing RedisTimeoutException and pages fail. Separate server-side pressure (CPU, memory, big keys) from client-side causes (thread pool, bandwidth) and stabilise quickly.

Deepak Bhardwaj3 min read

Applies toAzure Cache for Redis (Standard/Premium), StackExchange.Redis 2.x, redis-cli 6+

Symptoms

  • Logs full of RedisTimeoutException or RedisConnectionException; response times climb and requests fail.
  • Cache-aside fallback floods the database, which then also slows down.

1. Read the timeout message

StackExchange.Redis timeouts include useful counters:

Timeout performing GET (5000ms), inst: 0, qu: 0, qs: 120, aw: False, rs: ReadAsync,
ws: Idle, in: 65536, serverEndpoint: ..., mc: 1/1/0, mgr: 10 of 10 available,
IOCP: (Busy=0,Free=1000,Min=8,Max=1000), WORKER: (Busy=180,Free=32587,Min=8,Max=32767)
Counter What it tells you
qs high Requests sent and waiting for the server — server slow or network
in high Data sitting in the socket not read — client too busy
WORKER Busy > Min .NET thread pool starvation on the client
Large values in the operation Big keys hogging the connection

2. Check the server side

az monitor metrics list --resource <redis-resource-id> \
  --metric "serverLoad" "usedmemorypercentage" "connectedclients" "evictedkeys" "cacheRead" \
  --interval PT1M --offset 1h -o table
  • Server load above ~80%: Redis is single-threaded per shard for commands; expensive commands (KEYS *, big SMEMBERS, Lua scripts) block everything.
  • Used memory near 100% with evictions: fragmentation and eviction work slow the server.
  • Cache read bandwidth at the tier limit: large values saturate the network.

Find slow commands and big keys:

redis-cli -h <name>.redis.cache.windows.net -p 6380 --tls -a '<key>' SLOWLOG GET 20
redis-cli -h <name>.redis.cache.windows.net -p 6380 --tls -a '<key>' --bigkeys
redis-cli -h <name>.redis.cache.windows.net -p 6380 --tls -a '<key>' INFO memory

Warning

--bigkeys scans the keyspace. It uses SCAN and is generally safe, but on an already overloaded cache run it from a replica-aware window or after the immediate fire is out.

3. Immediate relief

  • Kill the bad caller: if SLOWLOG shows KEYS * or a huge range query, disable the feature or job that issues it.
  • Free memory: set TTLs on keys without one, delete obvious large junk keys (UNLINK is non-blocking).
  • Scale: move to a larger size or add shards (Premium clustering). Scaling is online but adds load briefly.
az redis update -g rg-cache -n redis-prod --sku Premium --vm-size P2
  • Reboot only if the cache is wedged; you lose data on Standard unless persistence is on, and every client reconnects at once.

4. Client-side fixes

  • Raise the thread pool minimum on .NET to absorb bursts:
ThreadPool.SetMinThreads(workerThreads: 200, completionPortThreads: 200);
  • Reuse one ConnectionMultiplexer per process. Creating one per request exhausts connections.
  • Keep values small (under ~100 KB). Split or compress large payloads.
  • Use async APIs and avoid sync-over-async (.Result) in request paths.
  • Set connectTimeout and syncTimeout explicitly and add retry with jitter.

5. Protect the database during recovery

When the cache is unhealthy, every miss hits the database. Add a short circuit breaker or serve stale data where acceptable, and rate-limit cache rebuilds so a cold cache does not start a stampede.

Prevent it next time

  • Alert on server load > 70%, used memory > 80% and evicted keys > 0.
  • Set maxmemory-reserved and maxfragmentationmemory-reserved so Redis has headroom.
  • Block dangerous commands in code reviews; never call KEYS in production.
  • Load test with production-like value sizes before releases.

Related fixes