AI does not yet outperform engineers in managing IT outages

AI tools are rapidly advancing in monitoring IT systems. However, a new study conducted by Datadog and Carnegie Mellon University shows that engineers maintain a significant lead in managing complex incidents. Based on real outages observed in production, this test compares several advanced models to human specialists. The results mainly reveal the current limitations of models when facing critical and unforeseen situations.

AI is advancing but remains limited against complex incidents

Tech companies now present AI agents capable of automatically analyzing production incidents, despite recent progress made by these models. These systems are meant to help teams detect anomalies and identify the root causes of outages. However, the ARFBench benchmark shows this automation is still imperfect. The project relies on real incidents observed during emergency situations, with manually validated data to avoid artificial scenarios.

The study is based notably on several key figures:

• 63 real incidents analyzed from Slack exchanges in emergency situations.
• 750 questions created around the studied incidents.
• 142 monitoring indicators used in the benchmark.
• More than 5 million data points were examined.

The tests evaluate both anomaly detection and the models’ ability to understand complex relationships between multiple metrics. GPT-5 achieves an F1 score of 47.5% on the most difficult questions while maintaining an overall accuracy of 62.7%. Researchers also recall that trillions of dollars are lost annually due to system outages, which underscores the strategic importance of AI tools in modern digital infrastructures.

Engineers maintain a clear lead over current models

Against the models’ results, human engineers retain better overall accuracy. Domain experts scored 72.7%, well above the best models tested. Even Datadog non-experts reached 69.7%, more than the automated system.

These results indicate that engineers still interpret the overall context of an incident better. They can more easily understand interactions between several technical signals and the unusual behaviors of infrastructures.

No AI model has surpassed the benchmark human performance. However, some specialized systems are gradually narrowing the gap. The hybrid model Toto-1.0-QA-Experimental, developed by Datadog, achieves an accuracy of 63.9%. This system combines an internal forecasting model with Qwen3-VL 32B.

In anomaly detection, Toto even obtains an F1 score at least 8.8 points higher than competing models. This result confirms that a model specialized in observability data can better meet a specific technical task than a generalist system.

Despite these advances, engineers remain essential during critical incidents. Models sometimes lose business context, ignore certain metadata, or misinterpret multiple indicators simultaneously.

Collaboration between AI and humans becomes the most credible scenario

The study mainly highlights that the errors of humans and models differ. AI systems detect some anomalies quickly, while humans better understand ambiguous situations and operational constraints.

Researchers explain these differences create complementary skills. Models sometimes miss contextual details, whereas humans make more mistakes on precise timestamps or complex instructions.

To measure this potential, researchers imagined an “expert oracle” capable of systematically choosing the best answer between a human and an AI. In this theoretical scenario, accuracy climbs to 87.2% with an F1 score of 82.8%.

This result does not yet represent a concrete product. However, it shows that collaboration between artificial intelligence and engineers could greatly improve IT incident management in the coming years. Automated systems thus seem destined to assist human teams rather than completely replace them in the short term.