Prometheus Monitoring & Alerting for Observable, Resilient Systems

Prometheus is open-source and self-hosted; Datadog is a managed SaaS. Prometheus gives you full control over data retention, cardinality and cost — there is no per-host or per-metric billing. Datadog reduces operational overhead and adds APM, log management and synthetic monitoring in one product. We recommend Prometheus when you need cost predictability at scale, strict data-residency for GDPR, or deep Kubernetes-native integration without vendor dependency.

Cardinality control starts at metric design: every label value must come from a bounded set. We audit existing exporters, enforce label conventions in CI with promtool, drop high-cardinality labels at the relabelling stage before ingestion, and monitor time-series count per job with an alert on unexpected growth. For existing high-cardinality metrics, we introduce recording rules that aggregate before storage.

When you need more than two to four weeks of retention, cross-cluster query federation or true HA with deduplication. Thanos adds a sidecar that ships TSDB blocks to object storage (S3, GCS) and a query layer that deduplicates Prometheus replicas. Mimir is the horizontally scalable alternative with a single-binary deployment option. Both extend Prometheus to months or years of retention at object-storage cost, which is an order of magnitude cheaper than local disk.

Threshold-based alerts generate excessive noise. We implement multi-window, multi-burn-rate SLO alerts: a fast burn window (five to sixty minutes) catches sudden outages; a slow burn window (six to twenty-four hours) catches gradual degradation. Alertmanager routes by severity, applies inhibition rules (suppress warning if critical is firing), deduplicates across HA replicas and delivers resolved notifications. Every alert ships with a runbook link.

Prometheus is the metrics store and query engine; Grafana is the visualisation and dashboard layer. Grafana queries Prometheus (or Thanos/Mimir) via the PromQL data source. We provision dashboards as JSON or Jsonnet committed to the repository — no manual click-through required. Recording rules pre-compute expensive aggregations so dashboard load times remain fast regardless of time range or cardinality.

We use sharded Prometheus instances — each shard scrapes a subset of targets using consistent hashing relabelling — and Thanos or Mimir as the global query layer. kube-prometheus-stack manages ServiceMonitor and PodMonitor resources so new workloads are discovered automatically. Recording rules offload aggregation from query time. Horizontal Pod Autoscaler metrics are served from a separate kube-state-metrics instance to avoid scrape latency impact on the main stack.

Prometheus uses a pull model: it scrapes HTTP endpoints at a configured interval. This makes target inventory explicit and simplifies debugging. The Pushgateway exists for the narrow case of short-lived batch jobs that complete before a scrape interval. We avoid Pushgateway for long-running services — it creates stale metric problems and removes the self-healing property of pull. For serverless or ephemeral workloads, we use remote-write to push directly to Mimir.