Amazon Bedrock AgentCore Evaluations: OpenTelemetry for AI Agent Observability

The emergence of AI agents introduces novel challenges for observability that traditional software monitoring tools cannot adequately address. Unlike predictable applications, AI agents can experience "quiet degradations" where they continue to function without errors but fail to perform their intended tasks correctly. This could stem from subtle changes in prompts, model updates, or tool descriptions, leading to issues like incorrect tool usage or fabricating responses. Amazon Bedrock AgentCore Evaluations, a recent AWS capability, tackles this by integrating deeply with OpenTelemetry (OTel) for robust AI agent observability. A core architectural principle of AgentCore Evaluations is its reliance on telemetry rather than live request/response payloads. AI agents instrumented for AgentCore Observability are designed to emit OpenTelemetry spans, which detail every individual step of an agent's interaction. These spans meticulously record critical events such as user prompts, invocations of underlying models, calls to external tools along with their parameters, the results returned by these tools, and the agent's final responses. These rich OTel spans are then collected and stored in Amazon CloudWatch. The AgentCore Evaluations system subsequently reads these spans from CloudWatch, or can receive them directly via an API. This data is then used to reconstruct the entire interaction flow of the AI agent, allowing for comprehensive scoring and evaluation. This design offers significant advantages, particularly framework independence, meaning that any agent emitting spans in the supported OpenTelemetry format can be evaluated. The system currently supports frameworks like Strands Agents and LangGraph, with automatic instrumentation for agents hosted on AgentCore Runtime. A key benefit is the parity between production and test environments; the same span format is used for both, ensuring consistent evaluation. Setting up this evaluation system requires Amazon Bedrock access, model invocation permissions for LLM-as-a-judge evaluators, and Transaction Search enabled in CloudWatch. Crucially, the AI agent itself must be configured to emit OTel spans, either automatically via AgentCore Runtime or through the AWS Distro for OpenTelemetry (ADOT) with appropriate instrumentation.