Home / AI & GenAI

AI & GenAIHow-toIntermediate

LiteLLM proxy virtual keys: per-team budgets, model access and rate limits

Give each team its own LiteLLM virtual key with an allowed model list, a monthly budget and TPM/RPM limits, then check spend, block a key and read the budget errors.

Deepak Bhardwaj6 min read

Applies toLiteLLM proxy (open source) with PostgreSQL; field names and error text vary by release, check your version's API reference

The goal: every team that calls an LLM through your LiteLLM proxy gets its own key. That key only reaches the models the team is approved for, stops when it hits its monthly budget, and cannot flood a shared upstream deployment. When a team asks "why are we getting 400s?" or finance asks "who spent what?", you can answer from the proxy instead of guessing.

This post covers the setup and the day-to-day admin calls. It assumes the proxy is already running and routing to at least one upstream model.

Prerequisites

Virtual keys, budgets and spend tracking need a database. Without one the proxy answers key and budget calls with No connected db. and budgets are not enforced.

Requirement Where it goes
PostgreSQL database DATABASE_URL env var or general_settings.database_url
Master key starting with sk- LITELLM_MASTER_KEY env var or general_settings.master_key
Model list model_list in config.yaml

A minimal config.yaml:

model_list:
  - model_name: chat-standard
    litellm_params:
      model: azure/gpt-4.1-mini-prod
      api_base: os.environ/AZURE_API_BASE
      api_key: os.environ/AZURE_API_KEY
  - model_name: chat-premium
    litellm_params:
      model: azure/gpt-4.1-prod
      api_base: os.environ/AZURE_API_BASE
      api_key: os.environ/AZURE_API_KEY

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/DATABASE_URL

The model_name values are what teams put in their requests and in key allow-lists. Keep them stable and generic (chat-standard), so you can change the upstream deployment without touching every team's key.

Warning

The master key can create, read and delete every key. Store it in a secret store, never hand it to an application, and rotate it if it appears in a log or a chat message.

Set two shell variables for the examples below:

PROXY=https://llm-gw.internal.example.com
MK=$LITELLM_MASTER_KEY

Create a team with a budget

Put the budget and limits on the team, not on each key. A key that belongs to a team is checked against the team budget (and team member budgets), not the key owner's personal budget. The details of this changed in newer releases, so check the notes for your version.

curl -sS "$PROXY/team/new" \
  -H "Authorization: Bearer $MK" -H "Content-Type: application/json" \
  -d '{
    "team_alias": "service-desk-bot",
    "models": ["chat-standard"],
    "max_budget": 500,
    "budget_duration": "30d",
    "tpm_limit": 200000,
    "rpm_limit": 300
  }'
  • max_budget is in the currency of your model cost map, usually USD. Spend is calculated by the proxy from token counts and its price table, so it is an estimate and not your invoice.
  • budget_duration resets spend on a schedule. Values like 30s, 30m, 30h and 30d are accepted. If you leave it out, the budget never resets.
  • tpm_limit and rpm_limit are enforced by the proxy. If you run several proxy replicas, configure Redis so the counters are shared. Otherwise each replica enforces the limit on its own.

Save the team_id from the response.

Issue a key for the team

curl -sS "$PROXY/key/generate" \
  -H "Authorization: Bearer $MK" -H "Content-Type: application/json" \
  -d '{
    "team_id": "<team_id>",
    "key_alias": "service-desk-bot-prod",
    "models": ["chat-standard"],
    "metadata": {"owner": "servicedesk-platform", "env": "prod", "ticket": "CHG-4812"}
  }'

The response contains the key (sk-...). It is only shown once, so put it straight into the team's secret store. Always set key_alias and metadata. When a key shows up in an error message months later, these fields are how you trace it back to an owner.

Test the key with an OpenAI-compatible call:

curl -sS "$PROXY/v1/chat/completions" \
  -H "Authorization: Bearer sk-team-key" -H "Content-Type: application/json" \
  -d '{"model":"chat-standard","messages":[{"role":"user","content":"Reply with OK"}],"max_tokens":5}'

The same key used with "model":"chat-premium" should fail, because that model is not on the key's or the team's list.

Day-to-day checks

Spend and limits for a key or team

curl -sS "$PROXY/key/info?key=sk-team-key" -H "Authorization: Bearer $MK"
curl -sS "$PROXY/team/info?team_id=<team_id>" -H "Authorization: Bearer $MK"

Compare spend with max_budget and look at budget_reset_at. A team near its limit in week one usually has a batch job or a retry loop. Ask them for the call pattern before you raise the budget.

Raise a budget or change models

curl -sS "$PROXY/team/update" \
  -H "Authorization: Bearer $MK" -H "Content-Type: application/json" \
  -d '{"team_id": "<team_id>", "max_budget": 750, "models": ["chat-standard", "chat-premium"]}'

Record the change ticket in the team's metadata so the reason is kept next to the setting.

Block a key without deleting it

# Stop the key immediately (reversible)
curl -sS -X POST "$PROXY/key/block" \
  -H "Authorization: Bearer $MK" -H "Content-Type: application/json" \
  -d '{"key": "sk-team-key"}'

# Restore it
curl -sS -X POST "$PROXY/key/unblock" \
  -H "Authorization: Bearer $MK" -H "Content-Type: application/json" \
  -d '{"key": "sk-team-key"}'

Blocking keeps the spend history and settings. Use it for a leaked key or a runaway client while you investigate. Delete the key only after a replacement is in place.

Caution

POST /key/delete removes the key at once and every client using it starts failing. Block first, confirm in the logs that traffic has moved to the new key, then delete.

Reading budget and limit errors

What the client sees Cause Fix
type: budget_exceeded, message names the virtual key or key_alias Key or team spend reached max_budget Check /team/info. Wait for the reset, or raise the budget with an approval
ExceededBudget: Crossed spend within team A team member's own budget inside the team is used up Raise that member's budget or move the workload to its own key
429 with a rate limit message Key or team tpm_limit/rpm_limit reached Spread the load, or raise the limit if upstream quota allows it
401, key not valid or blocked Key blocked, deleted, mistyped or expired /key/info; check blocked and expires
Error saying the key can't access the model Model not on the key's or team's models list Add it with /team/update or /key/update
No connected db. Proxy started without a database Set DATABASE_URL and restart

Exact message text and status codes (400 vs 429) for budget errors differ between releases. Alert on the type field rather than on the full message.

Prevent it next time

  • Create teams and keys from a script or pipeline, not by hand, and always include key_alias, owner and ticket metadata.
  • Give every key an expiry (duration on /key/generate, for example "90d") and a rotation reminder.
  • Set team TPM limits below the upstream deployment's quota, so one team cannot use up capacity that others share.
  • Run Redis with multiple proxy replicas so rate limits and budgets are counted across the whole fleet.
  • Compare proxy spend with the provider's billing once a month. Large gaps usually mean the price table is out of date or traffic is bypassing the gateway.
  • Back up the proxy's PostgreSQL database. It holds every key, team and spend record. If you lose it, every team needs new keys.

Related fixes