PagerDuty for AI: How the SRE Agent Triages AI Incidents · PagerDuty · 2026-07-21
PagerDuty Engineering describes triage after an AI incident fires, not a demonstrated fully automated repair. A relevance drop can stem from prompts, retrieval, or the model, but an upstream service failure can also affect output; LLM-judge alerts themselves can be noisy.
The example concerns degraded tool selection in the internal Insights Agent. Through connectors, the SRE Agent reads Arize monitor state and relevant traces, looks for patterns in evaluation explanations, and checks GitHub code changes. The team encodes existing runbooks as skills to guide investigation. In this case the agent identifies a minor degradation and recommends closing the incident with follow-up improvements. A code-suggestion skill then constrains proposed system-prompt changes, which still need replay against failed traces and regression testing.
The useful operational idea is to combine AI-quality signals with conventional service context instead of treating every low score as an emergency. The article reports a practical example without a controlled study or quantified incident-response improvement. Automatic rechecking of noisy scores, routing by failure type, and further automation are presented as future directions, not validated capabilities.