Home / AI & GenAI

AI & GenAIHow-toAdvanced

Transformation: when and how to move from a hosted LLM API to self-hosted open-weight models

Self-hosting open-weight models can cut cost at high volume and meet strict data or latency needs, but it adds GPU operations. Decide with numbers, pick a model with an evaluation, size GPUs, serve with vLLM behind an OpenAI-compatible API, and migrate workloads one at a time.

Deepak Bhardwaj3 min read

Applies toOpen-weight models (Llama, Mistral, Qwen, Gemma families and similar; check each licence), vLLM or TGI on Kubernetes/VMs, GPUs on Azure/AWS/OCI or on-premises

Decide with numbers, not enthusiasm

Reason to self-host Check
Cost at high, steady volume Tokens/day × API price vs. GPU cost at realistic utilisation (include staff time)
Data must not leave your environment Can a private/regional managed offering meet it instead?
Latency / on-prem / disconnected sites Needs local inference
Customisation (fine-tuned small model for a narrow task) Often the strongest case

Reasons it disappoints: low or spiky volume (idle GPUs), needing frontier-level quality, no team to run GPUs 24×7.

Rough cost check:

API cost/month      = (input tokens × in-price + output tokens × out-price) per month
Self-host cost/month = GPU instances × hours × price + storage/network + operations effort
Break-even utilisation = when self-host throughput at your SLA covers the token volume

Measure throughput on your own prompts (tokens/s per GPU at your latency target). Vendor benchmarks rarely match your traffic.

1. Pick candidate models

  • Licence: commercial use terms, user thresholds, attribution, acceptable-use policy.
  • Size vs task: 7–14B models handle classification, extraction and focused RAG well; larger models for complex reasoning.
  • Context length, language support, tool-calling ability.

2. Evaluate against the current API model

Run your evaluation set (task accuracy, groundedness, format validity, safety refusals) on 2–3 candidates and the current hosted model. Accept a model only if it meets the bar for this workload. Different workloads may end on different models.

3. Size and serve

vllm serve /models/qwen2.5-14b-instruct \
  --tensor-parallel-size 1 --max-model-len 16384 \
  --gpu-memory-utilization 0.90 --enable-prefix-caching \
  --api-key "$VLLM_KEY" --port 8000

vLLM exposes an OpenAI-compatible API (/v1/chat/completions), so client changes are small:

from openai import OpenAI
client = OpenAI(base_url="https://llm.internal.example/v1", api_key=os.environ["LLM_KEY"])

Sizing rules of thumb (verify by load test): weights at FP16 ≈ 2 bytes × parameters, plus KV cache for concurrent sequences. Quantization (FP8/INT4) cuts memory and often raises throughput (see the quantization post).

4. Production platform

  • Kubernetes with GPU node pools, at least two replicas across zones, autoscaling on queue length.
  • Gateway in front: auth, rate limits, routing between self-hosted and API models, fallback to the API on outage.
  • Monitoring: TTFT, tokens/s, queue length, KV cache usage, GPU health (DCGM).
  • Model artefacts in a registry with versioning; reproducible container images.

5. Migrate workload by workload

  1. Start with a high-volume, narrow task (classification, summarisation) where a smaller model passes evaluation.
  2. Route a percentage through the gateway; compare quality metrics and cost.
  3. Move the rest of that workload; keep API fallback.
  4. Repeat for the next workload. Keep complex tasks on the hosted API if no open model meets the bar.

Warning

Self-hosting moves safety work to you: content filtering, prompt injection shields and abuse monitoring that a hosted API may provide must be added around your own endpoint.

Checklist

  • Cost model with measured throughput
  • Licence reviewed per model
  • Evaluation vs current API model per workload
  • Serving platform with HA, gateway and monitoring
  • Gradual migration with API fallback
  • Safety controls replaced

Related fixes

AI & GenAI / How-to

Transformation: moving self-hosted LLM inference from FP16 to FP8 or INT4 quantized models

Serving open-weight models at FP16/BF16 uses the most GPU memory per request. Quantizing weights (and optionally the KV cache) to FP8 or INT4 can fit a model on fewer GPUs or serve more concurrent users. Pick the scheme your GPUs support, quantize or use a pre-quantized checkpoint, evaluate quality, load test and roll out behind the gateway.

3 min readAdvanced

AI & GenAI / How-to

Transformation: from long prompts to a LoRA fine-tuned small model for a narrow task

When a high-volume task needs a long prompt full of rules and examples on a large model, a small model fine-tuned with LoRA can match quality at lower latency and cost. Decide if fine-tuning is the right fix, build a training set from logged outputs, train with PEFT, evaluate against the prompt baseline and serve adapters with vLLM.

3 min readAdvanced