Home / AI & GenAI

AI & GenAITroubleshootingAdvanced

Serving many LoRA adapters on one vLLM server: adapter load errors, rank limits, latency spikes and memory sizing

One base model with dozens of fine-tuned LoRA adapters saves GPUs, until adapters fail to load, requests slow down or memory runs out. Fix rank and module mismatches, size max-loras and CPU cache, and load adapters dynamically and safely.

Deepak Bhardwaj4 min read

Applies tovLLM 0.6+ OpenAI-compatible server with LoRA (multi-LoRA), PEFT-trained adapters, Kubernetes deployments

Why multi-LoRA

Instead of one GPU deployment per fine-tuned model, you serve one base model and many small LoRA adapters. Each request names the adapter as its model. The server batches requests for different adapters together.

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --enable-lora \
  --lora-modules support-bot=/models/lora/support claims=/models/lora/claims \
  --max-loras 8 --max-lora-rank 64 --max-cpu-loras 32

curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" \
  -d '{"model":"claims","messages":[{"role":"user","content":"Summarise claim 1234"}]}'

Problem 1: adapter won't load

Error / symptom Cause Fix
LoRA rank 128 is greater than max_lora_rank 64 Adapter trained with a higher rank than the server allows Raise --max-lora-rank (more memory) or retrain with a lower rank
Target module errors / unexpected keys Adapter trained on a different base model or targets modules vLLM doesn't support for this architecture Use the exact base model (same family, size and revision); check supported target modules
Model claims not found (404) Adapter name not registered Check GET /v1/models; register via --lora-modules or dynamic loading
Garbage output from adapter Base model mismatch (e.g. trained on base, served on instruct) Serve the same base the adapter was trained on

Check the adapter's config:

cat /models/lora/claims/adapter_config.json | python -m json.tool | grep -E '"r"|base_model|target_modules'

r is the rank; base_model_name_or_path must match what you serve.

Problem 2: latency rises with many adapters

  • --max-loras is how many adapters can be active in one batch on the GPU. Requests for adapter number max-loras + 1 wait until a slot frees, which shows up as queueing.
  • --max-cpu-loras is how many adapters are cached in CPU memory; others are loaded from disk on demand (slow first request).
  • Higher --max-lora-rank and more active adapters add compute per token.

Watch the server metrics (/metrics):

curl -s localhost:8000/metrics | grep -E "vllm:num_requests_waiting|vllm:lora_requests_info|time_to_first_token"

vllm:lora_requests_info (name may vary by version) shows running and waiting adapters.

Tuning:

Pattern Setting
Many adapters, each with steady traffic Raise --max-loras (costs GPU memory)
Long tail of rarely used adapters Larger --max-cpu-loras, local NVMe for adapter files
A few heavy adapters dominate Give them their own replicas; keep the long tail on a shared pool

Problem 3: GPU memory

LoRA slots are pre-allocated from GPU memory: roughly max_loras × max_lora_rank × (sum of target module sizes). That memory is taken from the KV cache budget, so more/larger LoRA slots means less KV cache and fewer concurrent sequences.

  • Set --max-lora-rank to the largest rank you actually use, not the maximum supported.
  • Standardise adapter ranks across teams (e.g. 16 or 32).
  • Watch KV cache usage after changing LoRA settings; if gpu_cache_usage hits 100% more often, you traded concurrency for adapters.

Problem 4: adding adapters without restarts

vLLM supports dynamic loading when explicitly enabled:

export VLLM_ALLOW_RUNTIME_LORA_UPDATING=True
vllm serve ... --enable-lora --max-loras 8 --max-lora-rank 32

curl -X POST localhost:8000/v1/load_lora_adapter -H "Content-Type: application/json" \
  -d '{"lora_name": "hr-policy", "lora_path": "/models/lora/hr-policy"}'
curl -X POST localhost:8000/v1/unload_lora_adapter -H "Content-Type: application/json" \
  -d '{"lora_name": "hr-policy"}'

Warning

Runtime adapter loading lets any caller who reaches these endpoints load files from paths the server can read. Never expose them publicly; keep them behind an internal admin network or gateway rule, and allow only a controlled adapter directory.

With several replicas, each replica must load the adapter. Use a controller/job that calls every pod, or keep --lora-modules in config and roll the deployment.

Problem 5: routing in front of replicas

If replicas have different adapter sets, the gateway must route by model name. If every replica has every adapter, any load balancer works, but cold adapters on each replica add first-request latency. Prefix-aware or adapter-aware routing (sticky by adapter) keeps adapters warm.

Checklist

Item Check
Base model Exactly the one adapters were trained on
Rank max-lora-rank ≥ max adapter rank, not more than needed
Active slots max-loras covers adapters used concurrently
CPU cache max-cpu-loras covers the commonly used set
Security Dynamic load endpoints not exposed
Capacity KV cache headroom after LoRA allocation

Prevent it next time

  • Publish an adapter contract: base model revision, allowed ranks, target modules.
  • Validate adapters in CI by loading them into a test server and running a smoke prompt.
  • Monitor queueing per adapter and promote heavy adapters to dedicated replicas.
  • Treat adapter files as versioned artefacts with checksums.

Related fixes