Azure OpenAI fine-tuning cheat sheet: training files, jobs, events, checkpoints, metrics and deployments

#Before you start

• Fine-tunable models and regions change often. Check the current fine-tuning model table for your region before you plan a job.
• Hosting a fine-tuned deployment has an hourly charge whether or not you call it, separate from token charges. Check the current pricing page.
• A fine-tuned deployment that receives no completion or chat calls for 15 consecutive days is deleted. The model itself survives and can be redeployed.

All Python examples use Entra ID and the v1 endpoint:

from azure.identity import DefaultAzureCredential, get_bearer_token_provider
from openai import OpenAI

token = get_bearer_token_provider(DefaultAzureCredential(),
                                  "https://cognitiveservices.azure.com/.default")
client = OpenAI(base_url="https://<resource>.openai.azure.com/openai/v1/",
                api_key=token)

If your SDK version doesn't accept a token provider as api_key, use AzureOpenAI(azure_endpoint=..., azure_ad_token_provider=token, api_version=...) with a dated api-version; the method calls below stay the same.

#Training data

Chat models take JSONL, one conversation per line:

{"messages": [{"role": "system", "content": "Classify the incident."}, {"role": "user", "content": "Disk 98% on sql-prod-03"}, {"role": "assistant", "content": "{\"category\": \"storage\", \"severity\": 2}"}]}

Validate locally before you upload. Upload failures and validating_files errors cost a queue slot and time.

import json, sys, tiktoken
enc = tiktoken.get_encoding("o200k_base")   # approximate; the service count is authoritative
bad, tokens = 0, []
for n, line in enumerate(open(sys.argv[1], encoding="utf-8"), 1):
    try:
        msgs = json.loads(line)["messages"]
        assert msgs[-1]["role"] == "assistant", "last message must be assistant"
        assert all(m["role"] in ("system", "user", "assistant", "tool") for m in msgs)
        tokens.append(sum(len(enc.encode(m.get("content") or "")) for m in msgs))
    except Exception as e:
        bad += 1; print(f"line {n}: {e}")
print(f"rows={len(tokens)} bad={bad} max_tokens={max(tokens)} total={sum(tokens)}")

#Files

f = client.files.create(file=open("train.jsonl", "rb"), purpose="fine-tune")
v = client.files.create(file=open("valid.jsonl", "rb"), purpose="fine-tune")
print(f.id, f.status)

client.files.retrieve(f.id).status          # wait for "processed"
for x in client.files.list(purpose="fine-tune"):
    print(x.id, x.filename, x.status, x.created_at)
client.files.delete("file-abc123")

Don't start a job until both files show processed.

#Jobs

job = client.fine_tuning.jobs.create(
    model="gpt-4.1-mini-2025-04-14",        # base model name and version from your region's table
    training_file=f.id,
    validation_file=v.id,
    suffix="triage-v3",
    seed=42,
    method={"type": "supervised",
            "supervised": {"hyperparameters": {"n_epochs": 3,
                                               "batch_size": 8,
                                               "learning_rate_multiplier": 1.0}}},
)
print(job.id, job.status)

Older SDKs and api-versions use a top-level hyperparameters={...} instead of method. Leave hyperparameters out on the first run to get the service defaults, then change one at a time.

client.fine_tuning.jobs.retrieve(job.id).status
for j in client.fine_tuning.jobs.list(limit=20):
    print(j.id, j.model, j.status, j.fine_tuned_model, j.trained_tokens)
client.fine_tuning.jobs.cancel(job.id)

#Events: why a job failed

for e in client.fine_tuning.jobs.list_events(job.id, limit=50):
    print(e.created_at, e.level, e.message)

print(client.fine_tuning.jobs.retrieve(job.id).error)

REST equivalent:

TOKEN=$(az account get-access-token --resource https://cognitiveservices.azure.com --query accessToken -o tsv)
curl -s -H "Authorization: Bearer $TOKEN" \
  "https://<resource>.openai.azure.com/openai/v1/fine_tuning/jobs/<job-id>/events?limit=50"

#Checkpoints and metrics

Checkpoints are created at the end of each successful epoch. Each one is a deployable model, which matters when the last epoch overfits.

for c in client.fine_tuning.jobs.checkpoints.list(job.id):
    print(c.step_number, c.fine_tuned_model_checkpoint, c.metrics)

The result file is a CSV of training and validation metrics per step:

import io, pandas as pd
j = client.fine_tuning.jobs.retrieve(job.id)
csv = client.files.content(j.result_files[0]).read()
df = pd.read_csv(io.BytesIO(csv))
print(df.columns.tolist())
print(df[["step", "train_loss", "valid_loss"]].dropna().tail(10))

Column names vary by version; print them first.

Tip

Loss doesn't measure quality. Run the same offline evaluation set against the base model, every checkpoint you might ship and the current production model. Ship only if the fine-tune wins on your metric, not on loss.

#Deploy

az cognitiveservices account deployment create \
  --resource-group rg-ai --name aoai-prod \
  --deployment-name triage-ft-v3 \
  --model-name "gpt-4.1-mini-2025-04-14.ft-<job-suffix-id>" \
  --model-version "1" --model-format OpenAI \
  --sku-name Standard --sku-capacity 1

az cognitiveservices account deployment show -g rg-ai -n aoai-prod \
  --deployment-name triage-ft-v3 --query "properties.provisioningState"

Fine-tuned deployments can take a while to reach Succeeded. Available SKUs for fine-tuned models (Standard, Global, Developer, provisioned) vary by model and region. Developer deployments are for evaluation and are removed after a fixed lifetime, so don't put production traffic on them.

Test call:

r = client.chat.completions.create(model="triage-ft-v3",
        messages=[{"role": "user", "content": "Disk 98% on sql-prod-03"}])
print(r.choices[0].message.content)

#Housekeeping

az cognitiveservices account deployment list -g rg-ai -n aoai-prod \
  --query "[?contains(properties.model.name, '.ft-')].{name:name, model:properties.model.name, sku:sku.name}" -o table

# Fine-tuned models: list via jobs, delete the model when retired
for j in client.fine_tuning.jobs.list(limit=100):
    if j.status == "succeeded":
        print(j.fine_tuned_model, j.created_at)

Caution

Deleting a fine-tuned model is permanent; you'd have to rerun the job. Delete the deployment first, keep the training files and job ID in your model registry, and only then delete the model.

#Prevent it next time

• Keep the validator and the evaluation set in the same repo as the training data, and run both in CI.
• Record job ID, base model version, file IDs, hyperparameters and chosen checkpoint per release.
• Send a synthetic call to every low-traffic fine-tuned deployment at least weekly so it doesn't hit the 15-day inactivity deletion, or accept that it will be deleted and automate redeployment.
• Track base model retirement dates. When the base retires, the fine-tune has to be retrained on a newer base.