Home / AI & GenAI

AI & GenAIHow-toIntermediate

Standalone LLM server on one GPU with vLLM: install, serve, secure and measure

Stand up a production-minded single-GPU inference server with vLLM. Covers choosing a model that fits, installing drivers and the container, the OpenAI-compatible server, memory flags, a reverse proxy with authentication, systemd, monitoring and load testing.

Deepak Bhardwaj3 min read

Applies tovLLM 0.x on Ubuntu 22.04/24.04 with one NVIDIA GPU (L4, L40S, A100, H100); models of 7–32B parameters

1. Pick a model that fits

GPU memory Fits comfortably (with KV cache room)
24 GB (L4, RTX 4090) 7–8B in BF16, 14B in FP8/AWQ
48 GB (L40S) 14B in BF16, 32B in FP8/AWQ
80 GB (A100/H100) 32B in BF16, 70B in INT4/AWQ (tight)

Leave 20–40% of memory for the KV cache. That memory is what lets the server handle many concurrent requests.

2. Drivers and container runtime

sudo ubuntu-drivers install            # or install the NVIDIA datacenter driver
sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker && sudo systemctl restart docker
nvidia-smi

3. Run vLLM (OpenAI-compatible)

docker run -d --name vllm --gpus all --ipc=host -p 127.0.0.1:8000:8000 \
  -v /srv/hf-cache:/root/.cache/huggingface \
  -e HF_TOKEN=$HF_TOKEN \
  --restart unless-stopped \
  vllm/vllm-openai:latest \
  --model Qwen/Qwen2.5-14B-Instruct-AWQ \
  --served-model-name chat-default \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.90 \
  --max-num-seqs 64 \
  --enable-prefix-caching \
  --api-key "$VLLM_API_KEY"

Pin a specific image tag in production instead of latest.

Flag Why
--max-model-len Caps context. Lower values leave more KV cache for concurrent requests
--gpu-memory-utilization Share of GPU memory vLLM may use
--max-num-seqs Upper bound on concurrent sequences
--enable-prefix-caching Reuses shared prompt prefixes (system prompts, RAG templates)
--served-model-name A stable name for clients, so changing the model doesn't break them

4. Test

curl -s http://127.0.0.1:8000/v1/chat/completions \
  -H "Authorization: Bearer $VLLM_API_KEY" -H "Content-Type: application/json" \
  -d '{"model":"chat-default","messages":[{"role":"user","content":"Say hello"}],"max_tokens":20}'

5. Expose it safely

vLLM listens only on localhost. Put Nginx or Caddy in front, with TLS, and restrict it to the app subnet:

server {
  listen 443 ssl;
  server_name llm01.corp.local;
  ssl_certificate     /etc/ssl/llm01.crt;
  ssl_certificate_key /etc/ssl/llm01.key;
  allow 10.20.0.0/16; deny all;
  location /v1/ {
    proxy_pass http://127.0.0.1:8000;
    proxy_read_timeout 300s;
    proxy_buffering off;          # streaming
  }
}

Better still: route apps through your AI gateway, which handles keys and per-team limits.

6. Monitoring

vLLM exposes Prometheus metrics at /metrics: vllm:num_requests_running, vllm:num_requests_waiting, vllm:kv_cache_usage_perc (named gpu_cache_usage_perc in older versions), time to first token and generation throughput. Add the DCGM exporter for GPU temperature, memory and errors.

7. Load test before go-live

vllm bench serve --backend openai-chat --base-url http://127.0.0.1:8000 \
  --endpoint /v1/chat/completions --model chat-default \
  --dataset-name random --random-input-len 2000 --random-output-len 300 \
  --num-prompts 300 --max-concurrency 32

(Older versions use benchmarks/benchmark_serving.py.) Record tokens per second and p95 time to first token at your expected concurrency.

8. What "standalone" means here

One GPU, one server: a restart or GPU fault means downtime. Fine for internal tools and pilots. For production SLOs, see the HA self-hosted inference post.

Tip

If requests queue up (num_requests_waiting > 0 for a long time) while GPU memory is full, reduce --max-model-len or use a smaller quantisation before buying a bigger GPU.

Related fixes