App Service health check: instances flapping out of rotation, or a bad instance never replaced

Two opposite complaints land on the same feature:

• One instance is clearly broken (high error rate, stuck thread pool, full local disk) and keeps receiving traffic for an hour or more. Health check is enabled but nothing seems to happen.
• Instances drop in and out of the load balancer during a dependency blip, capacity halves and the remaining instances fall over.

Both come from how health check decides what "unhealthy" means and the limits it applies before acting. Fix the endpoint design first, then tune the thresholds.

#How health check actually behaves

These values come from the current documentation. Check the docs if your plan behaves differently, because the limits have changed before.

Warning

Enabling health check or changing its path restarts the app. The setting is not slot-specific, so a slot swap carries the configuration across. Change it in a staging slot or a maintenance window.

#Symptom → cause → fix

#Check the current configuration

RG=rg-web-prod
APP=app-orders-prod

# Health check path and the app's own worker count
az webapp config show -g $RG -n $APP \
  --query "{path:healthCheckPath, workers:numberOfWorkers}"

az webapp show -g $RG -n $APP --query "{httpsOnly:httpsOnly, plan:appServicePlanId}" -o json

# Threshold overrides, if any
az webapp config appsettings list -g $RG -n $APP \
  --query "[?starts_with(name, 'WEBSITE_HEALTHCHECK')]" -o table

# Instances currently running the app
az webapp list-instances -g $RG -n $APP --query "[].name" -o tsv

Find the health metric name on your resource rather than guessing it:

APP_ID=$(az webapp show -g $RG -n $APP --query id -o tsv)
az monitor metrics list-definitions --resource $APP_ID \
  --query "[?contains(name.value, 'Health')].name.value" -o tsv

Then chart it split by instance. A single instance sitting well below the others is the one being failed:

az monitor metrics list --resource $APP_ID --metric HealthCheckStatus \
  --interval PT5M --aggregation Average --filter "Instance eq '*'" -o table

If HealthCheckStatus is not in your list-definitions output, use the name it does return.

#Design the endpoint for what you want it to do

The key decision is what a 500 from this path means. Health check acts per instance, so it should report on per-instance failures.

If you must include a shared dependency, only fail on errors that are local to the instance, such as a connection pool on that instance that has stopped recovering. Keep the endpoint well under the one-minute timeout. Give each dependency call a timeout of a few seconds.

#Securing the endpoint

The path must allow anonymous access if you use your own authentication. App Service sends an x-ms-auth-internal-token header that you can validate: it is the Base64-encoded SHA-256 hash of the WEBSITE_AUTH_ENCRYPTION_KEY environment variable. Apply IP restrictions or a VNet first, and treat this header check as an extra layer on top.

# Flask example: answer probes from App Service only, and keep it instance-local
import base64, hashlib, hmac, os, shutil
from flask import Flask, request, abort

app = Flask(__name__)
EXPECTED = base64.b64encode(
    hashlib.sha256(os.environ["WEBSITE_AUTH_ENCRYPTION_KEY"].encode()).digest()
).decode()

@app.get("/healthz")
def healthz():
    token = request.headers.get("x-ms-auth-internal-token", "")
    if not hmac.compare_digest(token, EXPECTED):
        abort(403)
    free = shutil.disk_usage("/tmp").free
    if free < 500 * 1024 * 1024:          # local temp disk nearly full
        return "low disk", 503
    if not app.config.get("WARMED_UP"):    # set at the end of startup
        return "warming", 503
    return "ok", 200

Log failures from inside the endpoint. The probe requests are missing from HTTP logs, so your own log line is the only record of why a probe failed.

#Configure and tune

# Set the path (generic configuration works across CLI versions)
az webapp config set -g $RG -n $APP \
  --generic-configurations '{"healthCheckPath": "/healthz"}'

# Remove a failing instance faster (5 minutes instead of 10)
# and never take out more than a third of the fleet
az webapp config appsettings set -g $RG -n $APP --settings \
  WEBSITE_HEALTHCHECK_MAXPINGFAILURES=5 \
  WEBSITE_HEALTHCHECK_MAXUNHEALTHYWORKERPERCENT=34

# HTTPS Only, so probes go over HTTPS and are not redirected
az webapp update -g $RG -n $APP --https-only true

Trade-offs to settle before changing the defaults:

• Lower MAXPINGFAILURES removes bad instances faster but reacts to short GC pauses and cold dependencies. At 2 failures, a two-minute stall takes an instance out.
• Lower MAXUNHEALTHYWORKERPERCENT protects capacity during wide failures but leaves more bad instances serving traffic when several fail for real.
• Instance count matters more than either setting. With two instances and the default cap, one instance can be removed and the survivor takes all the load. Size the plan so N-1 instances can handle peak load.

Caution

Do not point health check at an endpoint that fails when the database is down. During a database incident every instance fails together. None are removed because of the all-unhealthy rule, and after an hour the platform starts replacing instances. You get cold restarts in the middle of the outage and no gain.

#When an instance is removed but never replaced

Replacement is rate-limited per plan and per scale unit, so in a bad week a broken instance can stay out of rotation and you run with less capacity. Recover by hand:

1. Identify the instance from the metric split above.
2. Restart that specific instance with Diagnose and solve problems > Advanced Application Restart in the portal. That restarts one instance at a time instead of the whole app.
3. If the same instance keeps failing, scale out by one (az appservice plan update --ids <plan-id> --number-of-workers <n+1>) to restore capacity while you investigate. Scaling back in does not let you choose which instance goes.

#Prevent it next time

• Treat the health endpoint as code with its own tests: instance-local checks, a warm-up gate, a hard internal timeout.
• Run at least two instances (three for zone-redundant plans) and size for losing one.
• Alert on the health metric split by instance, not on the app average, which hides a single bad worker.
• Keep apps that share a plan on compatible health check designs; one healthy neighbour can block replacement.
• Review WEBSITE_HEALTHCHECK_* overrides in code review, and roll out path changes through a staging slot.