Navigating OpenTelemetry Metric Cardinality: Balancing Performance and Data Accuracy

The OpenTelemetry blog recently published a practical guide detailing the intricacies of understanding and managing metric cardinality limits within the OpenTelemetry SDK. Authored by Cijo Thomas from Microsoft, the post serves to clarify the often-misunderstood behavior of these limits, which are fundamentally designed to protect applications from excessive memory consumption that can arise from high-cardinality attributes in metrics. The key takeaway is that when a metric stream exceeds its configured cardinality limit—which defaults to 2000 unique attribute combinations per metric stream—the SDK aggregates the overflowing data into a single 'overflow' data point. While the overall total of the metric remains accurate, any queries that filter or group by the attributes of an overflowed measurement will inevitably undercount, as the individual attribute breakdowns are lost. For platform engineers, Site Reliability Engineers (SREs), and any technical professional who relies on OpenTelemetry metrics for operational insights, this guidance is critically important. The subtle nature of how these cardinality limits affect collected data means that crucial observability artifacts—such as dashboards, automated alerts, and Service Level Objectives (SLOs)—could be silently misrepresenting the true state of systems. Misinterpreting these metrics can lead to delayed incident response, incorrect capacity planning decisions, or foster a false sense of security regarding application health and performance. The article directly addresses this frequently unexpected consequence: while aggregate totals are preserved, the granular attribute data essential for effective troubleshooting and deep analytical dives can be compromised, directly impacting the reliability of foundational observability data. This challenge of metric cardinality is not a new phenomenon; it represents a long-standing issue in the realm of distributed systems monitoring. Traditional monitoring tools have historically struggled with the explosion of unique time series generated by modern microservices architectures, containerized workloads, and dynamic cloud environments, often leading to prohibitive storage costs and severe query performance degradation. OpenTelemetry, positioned as the de facto standard for observability instrumentation, aims to provide a vendor-agnostic and robust solution. Its SDKs incorporate cardinality limits as a crucial defensive mechanism, ensuring that the act of instrumentation itself does not become a source of instability by consuming excessive memory within the application process. This design choice aligns perfectly with OpenTelemetry's broader goal of being production-safe and performant by default. The project continuously refines its specifications and implementations to strike a delicate balance between comprehensive data collection and operational stability, a commitment evident in ongoing developments across the OpenTelemetry Collector and various language SDKs. In practice, this means that practitioners should proactively review their existing OpenTelemetry metric configurations, paying particular attention to metrics that incorporate potentially high-cardinality attributes, such as `tenant_id` or `session_id`. The guide specifically advises choosing appropriate limits based on whether metrics utilize delta or cumulative temporality, noting that cumulative metrics are inherently more susceptible to overflow when dealing with high-cardinality dimensions over extended periods. It is imperative to actively monitor for cardinality overflow by leveraging internal metrics exposed by the SDKs and to establish robust alerting mechanisms. Upon detection of an overflow, teams must be prepared to either adjust the cardinality limit, remove the problematic high-cardinality attribute if its operational necessity is low, or, where appropriate, switch to a delta temporality for specific metrics. This also underscores the critical need for robust schema governance practices to prevent the accidental introduction of unbounded attributes into production metrics, thereby ensuring that observability data remains both comprehensive for analysis and reliable for operational decision-making.