Home / AI & GenAI

AI & GenAICheat sheetIntermediate

Advisory: choosing a self-hosted LLM serving stack - vLLM, SGLang, TensorRT-LLM/Triton, TGI, Ollama or llama.cpp

Serving engines differ in throughput, hardware support, model coverage and operational fit. Match the engine to the environment (data-center GPUs, laptops, CPUs, edge), model formats and features you need (OpenAI-compatible API, structured outputs, LoRA adapters), and benchmark with your traffic before standardising.

Deepak Bhardwaj2 min read

Applies toSelf-hosted LLM inference on NVIDIA/AMD GPUs, CPUs and edge devices; vLLM, SGLang, NVIDIA TensorRT-LLM with Triton/NIM, Hugging Face TGI, Ollama, llama.cpp

At a glance

Engine Strengths Typical use
vLLM High throughput (paged KV cache, continuous batching), wide model support, OpenAI-compatible server, LoRA, quantization General production GPU serving
SGLang Fast serving with strong prefix/structured-generation optimisations High-throughput serving, complex prompting patterns
TensorRT-LLM (with Triton or NVIDIA NIM) Maximum performance on NVIDIA GPUs after model compilation Latency/cost-critical NVIDIA deployments
Hugging Face TGI Simple deployment of HF models, good integrations Teams standardised on HF ecosystem (check current maintenance status)
Ollama Very easy local setup, model library Laptops, dev, small PoCs (see the single-VM PoC post)
llama.cpp CPU and Apple Silicon, GGUF quantized models, edge Offline/edge, CPU-only servers

Projects move quickly; check current feature support and release cadence.

Decision questions

  1. Hardware: data-center NVIDIA, AMD, CPUs, Apple Silicon, edge?
  2. Scale: single user/dev vs many concurrent users?
  3. Models and formats: Hugging Face weights, GGUF, FP8/INT4 checkpoints?
  4. Features: OpenAI-compatible API, tool calling, structured output, multi-LoRA serving, vision, speculative decoding?
  5. Operations: container images, Kubernetes operators/Helm charts, metrics, upgrades?
  6. Support: community only, or a vendor-supported distribution?

Typical fits

Situation Choice
Production chat/RAG on NVIDIA GPUs vLLM or SGLang
Squeezing maximum tokens/sec per GPU at large scale TensorRT-LLM (compare against vLLM with your models)
Developer laptops and quick experiments Ollama
CPU-only or edge devices llama.cpp

Benchmark your way

  • Same model, same precision, same GPU.
  • Your prompt/output length distribution and concurrency.
  • Measure TTFT, inter-token latency, throughput, memory use, and startup time.
  • Include operational tests: rolling upgrade, scale-out, failure recovery (see the HA self-hosted inference post).

Tip

Standardise on one primary engine for production with an OpenAI-compatible API. Applications then don't care which engine runs underneath, and you can switch after re-benchmarking.

Checklist

  • Hardware and scale defined
  • Required features listed
  • Candidates benchmarked with real traffic shape
  • Operational tests done
  • One production standard chosen behind a stable API

Related fixes

AI & GenAI / How-to

Standalone LLM server on one GPU with vLLM: install, serve, secure and measure

Stand up a production-minded single-GPU inference server with vLLM. Covers choosing a model that fits, installing drivers and the container, the OpenAI-compatible server, memory flags, a reverse proxy with authentication, systemd, monitoring and load testing.

3 min readIntermediate