LLM API basics: your first call, messages and roles, tokens, temperature and max tokens

Every LLM feature, from a chatbot to an agent, is built on one simple call: send a list of messages, get a message back. Learn that call well and the rest is plumbing.

#Messages and roles

The model has no memory between calls. A conversation is just you resending earlier messages each time.

#First call: Azure OpenAI

pip install openai azure-identity

from openai import AzureOpenAI
from azure.identity import DefaultAzureCredential, get_bearer_token_provider

token = get_bearer_token_provider(DefaultAzureCredential(), "https://cognitiveservices.azure.com/.default")
client = AzureOpenAI(
    azure_endpoint="https://my-aoai.openai.azure.com",
    azure_ad_token_provider=token,
    api_version="2024-10-21",            # use a current GA version for your resource
)

resp = client.chat.completions.create(
    model="gpt-4o-mini-prod",            # your DEPLOYMENT name, not the model family
    messages=[
        {"role": "system", "content": "You are a concise assistant for IT support engineers."},
        {"role": "user", "content": "Explain what a 503 error from a load balancer usually means."},
    ],
    temperature=0.2,
    max_tokens=300,
)
print(resp.choices[0].message.content)
print(resp.usage)                        # prompt_tokens, completion_tokens, total_tokens

Using Entra ID (DefaultAzureCredential) avoids API keys; your identity needs the Cognitive Services OpenAI User role.

#First call: Amazon Bedrock

import boto3
br = boto3.client("bedrock-runtime", region_name="eu-west-1")

resp = br.converse(
    modelId="anthropic.claude-3-5-haiku-20241022-v1:0",   # or an inference profile ID; check model access
    system=[{"text": "You are a concise assistant for IT support engineers."}],
    messages=[{"role": "user", "content": [{"text": "Explain what a 503 error from a load balancer usually means."}]}],
    inferenceConfig={"temperature": 0.2, "maxTokens": 300},
)
print(resp["output"]["message"]["content"][0]["text"])
print(resp["usage"])

Model IDs and availability differ by region; enable access to the model in the Bedrock console first.

#Tokens

Models read and write tokens, chunks of text. In English a token is roughly three-quarters of a word on average, but it varies by language and content (code and non-English text often use more).

Tokens matter for three reasons:

Count before sending (OpenAI-family models):

import tiktoken
enc = tiktoken.get_encoding("o200k_base")     # encoding used by recent OpenAI models
print(len(enc.encode("Explain what a 503 error from a load balancer usually means.")))

#The parameters to set first

Some reasoning models ignore or reject temperature and use a different output limit parameter (for example max_completion_tokens). Check the model's documentation.

Tip

If answers are cut off mid-sentence, check finish_reason (length means you hit max_tokens). If they are too long, ask for brevity in the system message and lower max_tokens.

#A multi-turn conversation

history = [{"role": "system", "content": "You are a concise assistant for IT support engineers."}]

def ask(q):
    history.append({"role": "user", "content": q})
    r = client.chat.completions.create(model="gpt-4o-mini-prod", messages=history, temperature=0.2, max_tokens=300)
    answer = r.choices[0].message.content
    history.append({"role": "assistant", "content": answer})
    return answer

print(ask("What causes a 503 from Azure Application Gateway?"))
print(ask("How do I check backend health from the CLI?"))

The history grows with every turn, so cost and latency grow too. Real apps trim or summarise old turns.

#Errors you will meet early

#Prevent it next time

• Always set max_tokens and log usage for every call.
• Keep API keys out of code; use Entra ID or IAM roles.
• Put the system prompt in configuration, versioned, not scattered in code.