Redis latency spikes: finding the cause with SLOWLOG, LATENCY DOCTOR and INFO

#The symptom

The application reports Redis timeouts or a p99 that has jumped from under a millisecond to tens or hundreds of milliseconds. Average latency often looks fine. CPU on the host may look low, because Redis executes commands on one main thread: a single slow command blocks every other client while it runs.

Work through the checks in order. Most incidents end at step 2 or 3.

#1. Measure from the client side first

Confirm the latency is real and where it starts. Run these from an application host, not from the Redis server itself.

# Continuous PING round trips, in milliseconds
redis-cli -h redis01 -p 6379 --latency

# Same, but a new line of min/max/avg every 15 seconds - good for spotting spikes
redis-cli -h redis01 -p 6379 --latency-history -i 15

Then, on the Redis host, check how much latency the machine itself adds (virtualisation, noisy neighbours):

# Runs for 60 seconds; do not run on a host that is already CPU-bound
redis-cli --intrinsic-latency 60

If intrinsic latency alone is several milliseconds, the problem is the VM or host, not Redis. If --latency from the app host is high but from the Redis host is low, look at the network, TLS, or connection pools.

#2. Read the slow log

The slow log records commands whose execution time (not network time) exceeded a threshold.

redis-cli CONFIG GET slowlog-log-slower-than   # microseconds; default 10000 (10 ms)
redis-cli CONFIG GET slowlog-max-len           # entries kept in memory; default 128

redis-cli SLOWLOG LEN
redis-cli SLOWLOG GET 20

Each entry shows an ID, Unix timestamp, duration in microseconds, the command with its arguments and (Redis 4+) the client address and name. During an investigation, lower the threshold and keep more entries:

redis-cli CONFIG SET slowlog-log-slower-than 5000
redis-cli CONFIG SET slowlog-max-len 1024

Typical offenders:

To see which commands cost the most overall, not just the slowest single calls:

redis-cli INFO commandstats | sort -t= -k4 -rn | head
# cmdstat_hgetall:calls=...,usec=...,usec_per_call=...

Redis 7 adds per-command percentiles:

redis-cli INFO latencystats

Warning

A single KEYS * on a 50-million-key instance can block for seconds and trigger Sentinel or cluster failover, because the node stops answering health checks. Rename or disable KEYS for application users with ACLs: ACL SETUSER app -keys.

#3. Turn on the latency monitor

The slow log only sees commands. Forks, AOF fsyncs, expiry and eviction cycles block the server too, and the latency monitor records them.

# Record any event that takes longer than 10 ms (default 0 = disabled)
redis-cli CONFIG SET latency-monitor-threshold 10

# After the next spike:
redis-cli LATENCY LATEST
redis-cli LATENCY DOCTOR
redis-cli LATENCY HISTORY fork

LATENCY LATEST lists each event name with the latest and maximum latency. LATENCY DOCTOR gives a plain-English analysis with advice. Event names vary by version; common ones are below.

Reset the counters after you change something so you can see the effect: redis-cli LATENCY RESET.

#4. Fork latency from RDB and AOF rewrite

Redis forks a child for BGSAVE and BGREWRITEAOF. The fork copies page tables, and the main thread is blocked while it happens. Large datasets on some VMs can take hundreds of milliseconds.

redis-cli INFO stats       | grep latest_fork_usec
redis-cli INFO persistence | egrep "rdb_bgsave_in_progress|aof_rewrite_in_progress|rdb_last_bgsave_time_sec"

Fixes:

• Disable transparent huge pages. With THP on, copy-on-write after the fork copies 2 MB pages instead of 4 KB pages, which causes long stalls and memory growth. Redis logs a warning at startup when THP is enabled.

  cat /sys/kernel/mm/transparent_hugepage/enabled
  echo never | sudo tee /sys/kernel/mm/transparent_hugepage/enabled
  

  Make it persistent with your distribution's method (tuned profile, systemd unit or kernel boot parameter transparent_hugepage=never).

• Schedule snapshots less often, or take them only on a replica.

• Split a very large instance into shards so each fork is smaller.

On ElastiCache and Azure the fork happens during backups and replica syncs. Move the backup window to a quiet period and keep memory headroom (reserved-memory-percent on ElastiCache, maxmemory-reserved on Azure).

#5. AOF fsync on slow disks

redis-cli CONFIG GET appendonly
redis-cli CONFIG GET appendfsync
redis-cli INFO persistence | grep aof_delayed_fsync

aof_delayed_fsync increasing means the background fsync could not keep up and the main thread had to wait. Options:

• appendfsync everysec instead of always (up to about one second of writes at risk on a crash).
• no-appendfsync-on-rewrite yes to skip fsync while a rewrite is running (more data at risk during the rewrite).
• Put the AOF on faster storage, away from other heavy I/O.

Caution

Relaxing fsync settings trades durability for latency. Only do it if the data can be rebuilt or the RPO agreed for this instance allows it.

#6. Mass expiry, eviction and big deletes

Thousands of keys created with the same TTL expire together, and the active expiry cycle works harder. Large keys freed during eviction or DEL also block.

redis-cli INFO stats | egrep "expired_keys|evicted_keys|lazyfree_pending_objects"
redis-cli CONFIG GET "lazyfree-*"

• Add jitter to TTLs in the application (for example base TTL plus a random 0–10%).

• Turn on lazy freeing so large values are freed on a background thread:

  redis-cli CONFIG SET lazyfree-lazy-eviction yes
  redis-cli CONFIG SET lazyfree-lazy-expire yes
  redis-cli CONFIG SET lazyfree-lazy-server-del yes
  redis-cli CONFIG SET lazyfree-lazy-user-del yes   # Redis 6+: DEL behaves like UNLINK
  

• Find and split big keys: redis-cli --bigkeys, then MEMORY USAGE <key>.

#7. Clients and connections

Connection storms and huge replies also show up as latency.

redis-cli INFO clients    # connected_clients, blocked_clients
redis-cli INFO stats | egrep "total_connections_received|rejected_connections"
# Clients with the largest output buffers (omem, bytes) first
redis-cli CLIENT LIST | awk '{o=0; for(i=1;i<=NF;i++){ if($i ~ /^omem=/){split($i,a,"="); o=a[2]} } print o, $0}' | sort -rn | head

High total_connections_received growth means clients connect per request. Use connection pooling and TLS session reuse. A large omem (output buffer memory) points at a client reading a huge reply or a slow Pub/Sub subscriber.

#Managed services

On managed services SLOWLOG usually works. Some CONFIG and LATENCY settings are set through the parameter group or advanced settings instead of CONFIG SET; check what your service allows.

#Prevent it next time

• Keep slowlog-log-slower-than at 10 ms or lower and ship SLOWLOG GET output to your logging tool on a schedule.
• Leave latency-monitor-threshold set (for example 50 ms) in production; the overhead is small.
• Block KEYS, FLUSHALL and FLUSHDB for application users with ACLs.
• Disable THP on every self-managed Redis host as part of the build.
• Alert on p99 latency from the client, EngineCPUUtilization (or Server Load) above about 80%, and rising evicted_keys.
• Review --bigkeys output monthly and set a size limit for collections in the application.