Home / AI & GenAI

AI & GenAIHow-toIntermediate

Tracing LLM applications with OpenTelemetry: see every prompt, retrieval and tool call

Instrument an LLM app with OpenTelemetry spans for retrieval, model calls and tools, record GenAI semantic convention attributes like tokens and model, and handle prompt privacy.

Deepak Bhardwaj2 min read

Applies toPython LLM apps exporting to Azure Monitor / Application Insights, AWS X-Ray / CloudWatch, OCI APM, Grafana Tempo, Jaeger

When a customer says "the bot was slow and wrong at 10:42", you need one trace showing: the query rewrite, the search (and what it returned), each model call (model, tokens, latency), each tool call, and the final answer.

1. Set up the SDK

pip install opentelemetry-sdk opentelemetry-exporter-otlp

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource

provider = TracerProvider(resource=Resource.create({"service.name": "support-bot"}))
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))   # OTEL_EXPORTER_OTLP_ENDPOINT
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("support-bot")

For Application Insights, azure-monitor-opentelemetry (configure_azure_monitor()) sets this up in one call.

2. Spans for each step

def answer(question):
    with tracer.start_as_current_span("rag.request") as root:
        root.set_attribute("app.user_tier", user.tier)

        with tracer.start_as_current_span("retrieval.search") as s:
            hits = search(question, k=5)
            s.set_attribute("retrieval.top_k", 5)
            s.set_attribute("retrieval.top_score", hits[0].score if hits else 0)
            s.set_attribute("retrieval.doc_ids", [h.id for h in hits])

        with tracer.start_as_current_span("chat " + MODEL) as s:
            s.set_attribute("gen_ai.operation.name", "chat")
            s.set_attribute("gen_ai.request.model", MODEL)
            s.set_attribute("gen_ai.request.temperature", 0.2)
            resp = llm.chat(build_prompt(question, hits))
            s.set_attribute("gen_ai.response.model", resp.model)
            s.set_attribute("gen_ai.usage.input_tokens", resp.usage.prompt_tokens)
            s.set_attribute("gen_ai.usage.output_tokens", resp.usage.completion_tokens)
            s.set_attribute("gen_ai.response.finish_reasons", [resp.choices[0].finish_reason])
        return resp

The gen_ai.* names follow the OpenTelemetry GenAI semantic conventions (still evolving; check the current spec). Libraries such as OpenLLMetry or the OpenTelemetry instrumentations for OpenAI SDKs can create model-call spans automatically.

3. Prompts and answers: privacy first

Prompt and response text often contain personal data. Options:

  • Don't record content by default; record ids, token counts and scores.
  • Record content only for sampled traces, to a store with short retention and restricted access.
  • Redact emails, phone numbers and IDs before recording.

4. Useful dashboards

  • p50/p95 latency per step (retrieval vs model vs tools).
  • Tokens and cost per request, per tenant.
  • Error rate by error.type (429, content filter, timeout).
  • Low retrieval scores over time (content gaps).

5. Connect to user feedback

Return the trace id with every answer and store it with thumbs-up/down feedback. A thumbs-down then links straight to the full trace.

Tip

Propagate trace context into tool calls and downstream APIs (W3C traceparent header). Then a slow answer caused by a slow CRM API shows up in the same trace instead of in a different team's dashboard.

Related fixes