Applies toLLM tracing with OpenTelemetry, Azure Monitor/Application Insights, CloudWatch, Langfuse/Phoenix-style LLM observability tools, gateway logging (APIM, Bedrock invocation logs)
What each signal costs
| Signal | Volume | Keep for |
|---|---|---|
| Metrics (tokens, latency, errors per feature/model) | Small | All requests, long retention |
| Trace spans without payloads | Medium | Sampled; full for errors |
| Prompt/response payloads | Large (KB per call, more for RAG context) | Sampled, redacted, short retention |
| Retrieved chunks | Very large | IDs only, rarely full text |
Sampling rules
| Request type | Rule |
|---|---|
| Errors, timeouts, guardrail blocks | 100% with payload (redacted) |
| User thumbs-down / flagged | 100% with payload |
| Normal traffic | 1–10% with payload; 100% metrics |
| New feature or model in rollout | Higher sampling temporarily |
With OpenTelemetry, combine a head sampler for normal traffic with tail-based sampling in the collector to keep errors:
# OpenTelemetry Collector tail sampling (excerpt)
processors:
tail_sampling:
decision_wait: 10s
policies:
- name: errors
type: status_code
status_code: { status_codes: [ERROR] }
- name: sample-rest
type: probabilistic
probabilistic: { sampling_percentage: 5 }
Limit payload size
- Truncate prompt and completion attributes to a maximum length.
- Log retrieved chunk IDs and scores, not chunk text.
- Store large payloads in cheap object storage with a reference in the trace, if needed at all.
- Redact PII before export (see the Presidio post).
Retention tiers
| Data | Hot (queryable) | Archive |
|---|---|---|
| Metrics | 13 months | — |
| Sampled traces with payloads | 7–30 days | Optional, compressed |
| Audit-relevant records (who used what, policy blocks) | Per policy | Per policy |
Use log table plans or tiers designed for verbose data where your platform offers them (see the Log Analytics cost post).
Check the bill
- Cost of telemetry per 1,000 LLM calls, tracked monthly.
- Top tables/streams by ingestion volume.
Warning
Full prompt logging often captures personal data and secrets users paste in. Less logging is both cheaper and safer; keep full payloads only where you have a clear need and retention limit.
Checklist
- Metrics for 100% of calls
- Payload sampling with errors and flagged kept
- Payload size caps and redaction
- Chunk IDs instead of chunk text
- Retention tiers set
- Telemetry cost per 1,000 calls tracked