2 years with an AI engineer on shift: what it fixed, what it almost broke, and the rules we wrote afterwards

We run a managed NOC for ISPs, data center operators and hosting providers worldwide. About two years ago we noticed that our engineers were doing the same investigation on every shift and on every customer network, regardless of vendor or topology: interface state, optics, neighbor tables, routing adjacencies, prefix reachability, logs, then a manual correlation across four or five terminals before anyone could say what broke and who was affected. So we built an internal AI NetOps assistant that uses the same tools our engineers already use, runs them in parallel across the stack, correlates the output and hands the on-call engineer a root cause hypothesis with the evidence attached, together with an assessment of which services are affected.

This talk is the operational report after two years of running it inside a real NOC, not a product pitch and not a lab demo.

We will cover the trust ladder we enforced and why each step existed. The agent started with no CLI access at all, reading only from monitoring, logs, flow data, BGP telemetry and the source of truth. It was then allowed a whitelist of operational show commands. Only later, and only for a narrow set of actions, was it permitted to propose changes, each gated by explicit human approval and a recorded audit trail. We will be honest about where that ladder stalled and which rungs we deliberately never climbed.
We will then walk through the kind of production alerts every NOC sees in its NMS and show how the agent actually behaves on them: a BGP session that will not establish, an IGP topology change, an OSPF adjacency dropping, a Layer 2 circuit going down. These are real cases from networks we operate, with the agent's investigation shown step by step: which data it pulled, how it correlated it, what root cause it proposed, and how it assessed impact at the same time. Among the cases we have already presented: a BGP failure the agent traced to a malformed attribute list while confirming in parallel that redundant paths carried the traffic; an IGP link failure it reasoned about from BGP-LS data without polling a single router; an OSPF adjacency loss that came down to mismatched BFD timers; a PE-CE link failure where the system separated a transceiver fault from a fiber cut and mapped the affected services within minutes. The NOC keeps running, so the final selection will favor the most interesting cases we see between now and the conference. Across these workflows, median time from alert to identified root cause dropped from 22 minutes to under 4.

We will also cover the cases where it was wrong. Some of them are the kind of mistake any engineer would catch instantly and the model did not, and those are the most instructive, because each one turned into a rule, a data fix or a guardrail. We will close with the numbers across [N] production infrastructures of different types: [X] alerts processed, [Y] percent resolved without escalation, plus screenshots of real investigations so the audience can judge the output for themselves.