Customers over control: how we measure On-call reliability

In a thought-provoking article published on May 28, 2026, Incident.io makes a compelling case for a customer-driven approach to measuring on-call reliability, moving beyond the conventional technical metrics that often dominate engineering discussions. Titled "Customers over control: how we measure On-call reliability," the post argues that true reliability is not merely about internal system uptime or component health, but fundamentally about the end-user's experience. This philosophy is deeply embedded in Incident.io's DNA, and the article articulates how this perspective leads to superior outcomes across the board. The core of Incident.io's On-call product encompasses a range of features, including configuring escalation paths, managing rotas and schedules, and facilitating cover requests. However, when assessing its reliability, the company deliberately narrows its focus to two paramount pieces of functionality: the successful ingestion of alerts and the timely delivery of notifications to on-call responders. An alert, in this context, is defined as a message received from a customer's observability tooling, signaling a potential issue that requires attention. The prompt delivery of these notifications is equally crucial, ensuring that responders can investigate and act without delay. To quantify these critical functions, Incident.io employs a set of well-defined Service Level Indicators (SLIs). For alert ingestion, they utilize a classic HTTP API availability SLI, measuring the proportion of requests successfully processed. For notification delivery, the SLI focuses on latency, specifically the proportion of notifications dispatched within a five-minute window. These SLIs are not arbitrary; they are meticulously chosen to reflect the customer's perspective of reliability. The goal is to measure what customers actually feel and experience, rather than solely focusing on internal system metrics that might not directly correlate with user satisfaction. The article further elaborates on how this customer-centric mindset influences their approach to third-party dependencies. Instead of dismissing issues caused by external services as 'not our fault,' Incident.io proactively designs its systems to anticipate and cope with such failures. This means assuming that third-party dependencies *will* experience degradation or outages and building resilience mechanisms to mitigate their impact on the customer experience. This proactive stance ensures that even when external factors are at play, the core reliability of their On-call product remains robust from the customer's viewpoint. Incident.io's methodology for measuring alert ingestion, for example, involves slicing each month into minute-long intervals and measuring the error rate for each minute. A 'good' minute is defined as one with an error rate below 10%. This granular approach allows them to identify and rectify issues that might be 'outside' their immediate stack but still affect alert ingestion. By framing reliability in this manner, they avoid including 'noise' in their signal and focus on the true availability of the service from the customer's perspective. This commitment to customer experience over internal control not only enhances the reliability of their On-call product but also fosters a culture of continuous improvement, where every aspect of the service is evaluated through the lens of its impact on the end-user. The post underscores that reliability is, at its heart, a customer experience problem, and addressing it effectively requires a fundamental shift in measurement and design philosophy.