GPU basics for LLM inference: VRAM, model size math, precision and which GPU you actually need

"Will this model fit on our GPU?" is the first question of every self-hosted LLM project. You can answer it roughly with arithmetic before you rent anything.

#The two things that use VRAM

1. Model weights: parameters x bytes per parameter.
2. KV cache: memory for the attention keys and values of every token in every active request. Grows with context length and concurrent users.

Plus some overhead for the runtime, CUDA kernels and activations (often a few GB).

#Weights: the quick formula

weights_GB ≈ parameters_in_billions × bytes_per_parameter

Inference is normally done in BF16/FP16 or lower; FP32 is rarely used.

#KV cache: why context and users matter

Rough per-token KV cache size:

kv_bytes_per_token ≈ 2 × layers × kv_heads × head_dim × bytes_per_value

Example, a typical 8B model with grouped-query attention (32 layers, 8 KV heads, head_dim 128, FP16):

2 × 32 × 8 × 128 × 2 bytes ≈ 131 KB per token
8,192-token context  ≈ 1 GB per active request
16 concurrent requests at 8k context ≈ 16 GB

The model's config.json (on its model card) has num_hidden_layers, num_key_value_heads and head_dim (or hidden_size / num_attention_heads). Serving engines like vLLM allocate KV cache in blocks and share prefixes, so real usage is often lower than this worst case, but plan with it.

Tip

If a model fits but you get out-of-memory errors under load, it is usually the KV cache. Reduce the maximum context length or the number of concurrent sequences before buying a bigger GPU.

#Putting it together

8B model, FP16, 16 users at 8k context:

weights  16 GB
KV cache 16 GB
overhead  ~3 GB
total    ~35 GB  -> fits a 48 GB GPU, tight on 40 GB, not on 24 GB

Same model at 4-bit: weights ~5 GB, total ~24 GB, so a 24 GB GPU may work with smaller context or fewer users.

70B model, FP16: weights alone ~140 GB, so you need several GPUs (tensor parallelism), or 8-bit/4-bit quantization on fewer large-memory GPUs.

#Precision and quantization trade-offs

Always evaluate a quantized model on your own tasks; the drop depends on the model and the use case.

#Memory bandwidth matters for speed

Generating each output token reads the weights from GPU memory. So token speed for a single user is roughly limited by memory bandwidth / weights size. A GPU with twice the bandwidth generates tokens roughly twice as fast for the same model, and a 4-bit model generates faster than FP16 on the same GPU. Compute (TFLOPS) matters more for the prompt-processing phase and for high concurrency.

#Read nvidia-smi

nvidia-smi
nvidia-smi --query-gpu=name,memory.total,memory.used,utilization.gpu,temperature.gpu,power.draw --format=csv -l 5
nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv

#Choosing a GPU class (rule of thumb)

Check cloud availability and quota in your region early; large GPUs are often constrained.

#Prevent it next time

• Do the weights + KV cache arithmetic before choosing hardware, and write it in the design.
• Load-test with realistic prompt lengths and concurrency, and watch memory.
• Set maximum context length deliberately on the serving engine.