Hugging Face model download fails: 401, 403 gated repos, timeouts and offline mode

#Symptom

An inference pod or VM that serves a Hugging Face model fails on startup or restart. The log ends with one of these:

• 401 Client Error ... Repository Not Found or RepositoryNotFoundError
• 403 Client Error ... Access to model ... is restricted or GatedRepoError
• ReadTimeout / TimeoutError while fetching config.json or a .safetensors shard
• LocalEntryNotFoundError or "Cannot find the requested files in the disk cache and outgoing traffic has been disabled"
• No space left on device halfway through a multi-shard download

The model worked yesterday, which usually means a new node, an empty cache, a rotated token or a new network path.

#Quick checks

Run these from the same pod or VM, with the same environment, as the failing process.

# Which identity is being used, if any
hf auth whoami

# Where the cache lives and what is in it
echo "HF_HOME=$HF_HOME HF_HUB_CACHE=$HF_HUB_CACHE HF_HUB_OFFLINE=$HF_HUB_OFFLINE"
hf cache ls

# Can this host reach the Hub at all
curl -sS -o /dev/null -w "%{http_code}\n" https://huggingface.co/api/models/<org>/<model>

# Free space where the cache lives
df -h "${HF_HUB_CACHE:-$HOME/.cache/huggingface/hub}"

Tip

CLI names vary by version. Current huggingface_hub releases ship the hf command (hf auth login, hf download, hf cache ls). Older ones use huggingface-cli login, huggingface-cli download and huggingface-cli scan-cache. Run hf version or pip show huggingface_hub to see what you have.

For Kubernetes, check the environment the container actually received, not the manifest you think was applied:

kubectl -n inference exec deploy/llm-server -- env | grep -E '^HF_|_PROXY'
kubectl -n inference logs deploy/llm-server --previous | grep -iE 'gated|401|403|timeout|offline|space'

#Symptom → cause → fix

#Fix authentication

The token is read from HF_TOKEN first, then from the token file under HF_HOME. A stale exported HF_TOKEN silently overrides a fresh hf auth login.

# Test a token without touching the stored login
HF_TOKEN=hf_xxx hf auth whoami

# Test access to a specific gated or private repo
HF_TOKEN=hf_xxx hf download <org>/<model> config.json --dry-run

In Kubernetes, keep the token in a Secret and inject it as HF_TOKEN:

kubectl -n inference create secret generic hf-token --from-literal=HF_TOKEN=hf_xxx \
  --dry-run=client -o yaml | kubectl apply -f -
kubectl -n inference set env deploy/llm-server --from=secret/hf-token

Warning

HUGGING_FACE_HUB_TOKEN is deprecated in favour of HF_TOKEN. If both are set with different values, HF_TOKEN wins. Remove the old variable so nobody rotates the wrong one.

Gated access is granted per account, not per token. If the service account that owns the token was created after someone accepted the licence on their personal account, the service account still gets 403 until it requests and receives access itself.

#Fix network and timeouts

Most "Hub is down" reports are actually egress rules or proxies.

export HTTPS_PROXY=http://proxy.corp.local:3128
export NO_PROXY=localhost,127.0.0.1,.svc,.cluster.local
export REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt   # if TLS inspection re-signs traffic
export HF_HUB_DOWNLOAD_TIMEOUT=60
export HF_HUB_ETAG_TIMEOUT=30

Model files are served from Hugging Face storage and CDN hostnames, not only huggingface.co. If metadata loads but shards time out, your firewall probably allows only the main domain. Check the current hostnames with your network team rather than hard-coding a list from a blog.

#Fix offline mode and the cache

HF_HUB_OFFLINE=1 stops all HTTP calls and only reads the local cache. It is the right setting for production, but only after the cache is populated on the volume the pod actually mounts.

# Populate a shared cache once, from a host with Hub access
export HF_HUB_CACHE=/models/hf-cache
hf download <org>/<model> --revision <commit-sha> --include "*.json" --include "*.safetensors" --include "tokenizer*"

# Confirm what is there
hf cache ls --cache-dir /models/hf-cache --revisions

Then run the server with the same cache path and offline mode on:

HF_HUB_CACHE=/models/hf-cache HF_HUB_OFFLINE=1 \
  vllm serve <org>/<model> --revision <commit-sha>

Clean up old revisions when the disk fills:

hf cache prune                          # removes unreferenced revisions and partial downloads
hf cache rm model/<org>/<model> --dry-run

Caution

Don't prune or delete a shared cache volume while pods are reading from it. Running replicas that lazily open shards can crash, and with offline mode on they cannot re-download. Drain or scale down the consumers first.

#Prevent it next time

• Pin a commit hash. --revision main means a different model can load after any restart. Use the commit SHA from the model page and change it through your normal release process.
• Pre-stage weights. Download once into a PVC, data disk or object storage bucket, and mount it read-only. Use an init container or a pipeline job, not the serving container, to fetch.
• Run production offline. Set HF_HUB_OFFLINE=1 on serving pods so a Hub outage, token expiry or rate limit can't stop a restart.
• Use a service account token. Create a read-only token owned by a team or service account, request gated access with that account, store it in your secret manager and put its expiry on the rotation calendar.
• Size the cache. A model's total shard size is on its "Files" tab. Give the cache volume room for at least two revisions so a rollout doesn't fail on disk space.
• Alert on startup. Alert when a model server restarts more than once in a short window and log the first HF_-related error, so a failing pull is noticed before every replica is down.