OpenTelemetry Instrumentation for Vendor-Neutral Distributed Observability

OpenTelemetry is a CNCF project that defines a vendor-neutral API, SDK and wire protocol (OTLP) for traces, metrics and logs. Unlike proprietary agents — Datadog APM, New Relic, Dynatrace — it embeds no vendor-specific code in your application. You instrument once using the open OTel API and route signals to any compliant backend via the Collector. Switching backends requires only Collector exporter configuration, not application code changes.

Traces record the end-to-end journey of a single request across services — each operation is a span with timing, attributes and status. Metrics are numeric aggregations over time (request rate, error rate, latency percentiles) suited for dashboards and alerting. Logs are timestamped text or structured events from individual components. OTel unifies all three under one SDK and wire protocol, and exemplars link metric data points directly to the traces that produced them.

OTel itself does not redact PII — that is the responsibility of the pipeline. We configure the OTel Collector's attribute processor and transform processor to drop, hash or mask span attributes and log fields that may contain regulated data (emails, user IDs, card numbers, SSNs) before signals reach any backend. This keeps telemetry compliant with GDPR and similar frameworks without requiring application-level instrumentation changes.

Head-based sampling decides at the start of a trace whether to record it — fast and memory-efficient, but it discards rare error traces with the same probability as routine ones. Tail-based sampling buffers the complete trace in the Collector before deciding, allowing policies such as "always keep traces with errors or latency above 1 s". We configure tail-based sampling for production systems where missing error traces is more costly than the additional Collector memory and CPU.

The OTel Collector is a vendor-agnostic agent that receives OTLP (or other formats), processes signals — batching, filtering, attribute transformation, PII redaction, tail-based sampling — and exports to one or more backends simultaneously. You can export directly from SDKs to a backend, but the Collector decouples instrumentation from backend choice, centralises sensitive data handling (PII redaction) and enables fan-out to multiple backends without application changes. We recommend it for any production deployment.

Overhead depends on sampling rate and cardinality. With head-based sampling at 10 % and well-tuned span attributes (no high-cardinality labels like user ID per span), CPU overhead is typically under 2 % and memory impact is minimal. Auto-instrumentation agents add library-loading cost at startup. Tail-based sampling in the Collector adds memory proportional to the buffer window. We profile instrumented services before and after deployment and adjust sampling policies to keep overhead within agreed SLOs.

Migration is incremental. We start by running the OTel Collector alongside the existing agent, routing a duplicate OTLP stream to a trial OTel-compatible backend while the proprietary agent continues in production. Once signal parity is confirmed — trace coverage, metric cardinality, alert fidelity — we remove auto-instrumentation from the proprietary agent, leaving only OTel SDKs in the application. The Collector can still export to Datadog via its OTLP exporter if Datadog is retained as a backend, so there is no hard cutover.