Home / AWS

AWSHow-toIntermediate

Advisory: an observability strategy on AWS - CloudWatch, managed Prometheus and Grafana, OpenTelemetry

A design guide for metrics, logs and traces across AWS accounts. Decide what CloudWatch covers natively, where Amazon Managed Service for Prometheus and Grafana fit, standardise instrumentation on OpenTelemetry, centralise across accounts, and set retention, cost and ownership rules from the start.

Deepak Bhardwaj3 min read

Applies toAmazon CloudWatch (metrics, logs, Application Signals, cross-account observability), Amazon Managed Service for Prometheus, Amazon Managed Grafana, AWS Distro for OpenTelemetry (ADOT), AWS X-Ray

What each piece is good at

Tool Strength
CloudWatch metrics AWS service metrics out of the box, alarms, anomaly detection
CloudWatch Logs + Logs Insights Central log store for AWS services and apps, ad-hoc queries
CloudWatch Application Signals Service-level metrics (latency, errors, SLOs) from OpenTelemetry instrumentation
Managed Prometheus (AMP) Kubernetes and app metrics in Prometheus format, PromQL, high cardinality
Managed Grafana (AMG) Dashboards over CloudWatch, AMP, X-Ray, OpenSearch and others
X-Ray / OTel traces Distributed tracing across services 
Instrumentation:  OpenTelemetry SDKs / auto-instrumentation in apps
Collection:       ADOT collector or CloudWatch agent (OTel-compatible) per node/cluster
Metrics:          AWS services → CloudWatch; Kubernetes/app metrics → AMP (or CloudWatch for small estates)
Logs:             CloudWatch Logs (bulk vended logs → S3)
Traces:           OTel → X-Ray / Application Signals
Dashboards:       Grafana for cross-source views; CloudWatch dashboards for service teams
Alerting:         CloudWatch alarms for AWS resources; Prometheus rules for cluster/app metrics; one paging route

Standardising on OpenTelemetry for instrumentation keeps you free to change backends later.

Multi-account design

  • CloudWatch cross-account observability: a monitoring account sees metrics, logs and traces from source accounts (linked via OAM sinks/links).
  • AMP workspaces per environment or per platform; remote write from clusters in many accounts with IAM roles.
  • Central Grafana with data sources per account/workspace; SSO via IAM Identity Center.

Decide the rules early

Topic Rule example
Naming Metric and log group naming conventions (/org/app/env/component)
Labels/dimensions Required: service, env, team; banned: user IDs, request IDs as metric labels
Retention App logs 30 days, audit 1 year in S3; metrics per backend defaults
Cost Budgets per account for CloudWatch; cardinality limits in AMP
Ownership Every dashboard and alarm has a team owner
SLOs Defined for user-facing services, tracked in Application Signals or Grafana

Kubernetes specifics

  • Container Insights (enhanced observability) for quick cluster visibility.
  • AMP with Prometheus scrapers (managed collector or ADOT) for detailed metrics; recording rules to reduce query cost.
  • Logs via Fluent Bit to CloudWatch, filtering noisy namespaces.

Common mistakes

Mistake Effect
Every team picks its own agent and backend No cross-service view, duplicate costs
High-cardinality labels Exploding AMP/CloudWatch custom metric costs
Logging everything at DEBUG CloudWatch Logs becomes the biggest bill line
Dashboards without alarms (or the reverse) Either nobody notices, or nobody understands
No trace context propagation Traces stop at the first hop

Tip

Start with one golden-path service: OTel instrumentation, standard dashboards, SLO and alarms. Make it a template, then roll out. Teams copy what works much faster than they follow a standards document.

Checklist

  • Role of each tool agreed
  • OpenTelemetry as the instrumentation standard
  • Cross-account observability configured
  • Naming, labels, retention and cost rules published
  • SLOs for user-facing services
  • Golden-path template rolled out

Related fixes