Splunk Report: Observability Leaders Restore Predictability in the Agentic AI Era

Cisco recently released its sixth annual Splunk State of Observability report, titled "Restoring Predictability in the AI Era." This year's research, which surveyed 2,050 ITOps and engineering professionals across ten countries, reveals a significant challenge: the rapid adoption of AI agents is outpacing organizations' ability to effectively monitor them. The report highlights that while AI agents offer immense potential, they also introduce unpredictable outcomes and volatile costs. A key finding is that 83% of respondents are concerned about the unintended consequences of AI agents, and over half (55%) struggle to accurately control AI-related costs. This matters deeply to practitioners because the proliferation of AI agents fundamentally changes the observability landscape. Traditional monitoring approaches, designed for more static and predictable systems, are proving inadequate for the dynamic and often opaque nature of agentic AI. The report identifies a group of "leaders" who are successfully navigating this new terrain. These leaders are 2.4 times more likely to exceed their observability ROI expectations and 54% report at least $5 million in annual business value from their observability practices. Their success stems from closely monitoring AI agent behavior, quality, cost, usage, security, and privacy, and integrating observability into strategic business planning. This trend aligns with the broader industry movement towards "agent observability" – a specialized form of observability focused on understanding and governing the behavior of autonomous AI systems. As AI agents gain access to entire tech stacks and make independent decisions, the need for robust guardrails and transparent monitoring becomes critical. This is driving increased collaboration between ITOps, engineering, and security teams, with 94% of leaders reporting improved collaboration due to monitoring AI. The report also subtly points to a shift in cost concerns; while AI infrastructure might seem like the obvious culprit for rising costs, the report indicates that verbose logs from years of "just in case" data collection are often the primary driver, rather than AI infrastructure itself. In practice, this means practitioners should prioritize developing and implementing observability strategies specifically tailored for AI agents. This includes investing in tools and practices that provide visibility into model quality, cost attribution, and the decision-making processes of AI agents. Organizations should move beyond simply collecting telemetry to actively using it to govern AI behavior and connect AI spend to tangible business value. The report suggests that aggressive data sampling and routing towards high-value signals are key practices for leaders. Furthermore, fostering closer collaboration between engineering, operations, and security teams is crucial for building trust in AI systems and ensuring their responsible deployment. Ignoring agent observability risks not only financial unpredictability but also potential operational failures and security vulnerabilities as AI agents become more deeply embedded in critical workflows.