I spent part of the evening of October 21, 2026, at an IT leadership dinner built around an unusually useful premise. Before discussing where AI was working, everyone was asked to acknowledge where it was not. The event was called The AI Nightmare, and while the framing was intentionally provocative, the discussion underneath it raised a serious question for IT service delivery leaders: How do we know whether AI is actually improving service delivery if we do not yet agree on what good looks like? (The AI Nightmare, 2026).
There is no shortage of AI metrics. People are discussing ticket deflection, resolution time, technician productivity, automation rates, cost per ticket, LLM spend, and knowledge utilization. The problem is that many organizations are beginning to set targets for these measures before reliable benchmarks exist. That became one of the most interesting themes of the evening (The AI Nightmare, 2026).
Starting With What Is Not Working
The event opened by asking attendees to anonymously share an "AI nightmare": something they had experienced, heard about, or were concerned could happen. It was a small design choice, but it changed the discussion. Technology events often begin with successful implementations, polished case studies, and carefully selected performance numbers. Starting with failure created a different environment and permitted people to discuss where their own AI initiatives were still uncertain (The AI Nightmare, 2026).
That became particularly important once the live polling started. Instead of comparing polished success stories, people were comparing assumptions, and some of those assumptions were much less settled than the wider AI conversation might suggest.
The Ticket Deflection Benchmark Does Not Yet Feel Mature
One of the first live poll questions asked attendees what ticket deflection rate they were targeting twelve months from now. The options ranged from below 25 percent to above 80 percent, with another option for organizations that were not targeting a specific rate at all (The AI Nightmare, 2026).
What interested me was not which percentage received the most votes, but the range itself. When a metric is mature, organizations operating in similar environments usually develop some shared understanding of what reasonable performance looks like. The discussion suggested that AI-based ticket deflection has not reached that level of maturity yet (The AI Nightmare, 2026).
Different organizations have different service models, customer populations, knowledge quality, automation maturity, ticket categories, and definitions of what counts as "deflected." A 40 percent result in one environment may therefore not be directly comparable with 40 percent somewhere else. That creates a risk for technology leaders because a target can look precise while still being based on a weak comparison. Before asking whether your deflection target is competitive, it may be worth asking a more fundamental question:
Competitive against what?
Maybe We Are Measuring the Wrong Thing
The conversation became more interesting when one participant challenged ticket closure as a measure of customer value. From an internal service perspective, faster closure sounds positive, but customers rarely want a ticket closed quickly. They would usually prefer not to have needed the ticket at all (The AI Nightmare, 2026).
That shifts the discussion from tickets resolved to tickets prevented.
One attendee described an example where email-based ticket submission was removed for a particular customer and requests were directed through a self-service portal supported by an improved knowledge base. The reported result was a 34 percent deflection rate (The AI Nightmare, 2026).
That example should not be treated as an industry benchmark. It was one account, one environment, and one intervention. But it demonstrates something useful. AI and self-service create value when they remove unnecessary demand from the service process, not simply when they help technicians process that demand faster.
That distinction matters because a service desk can become highly efficient at resolving tickets without necessarily improving the underlying customer experience.
The Technician Time Nobody Measures
Another live poll asked how much technician time attendees believed was being lost to manual tracking, context switching, and documentation gaps. During the poll, the highest response category, more than 20 percent, moved ahead of the other options (The AI Nightmare, 2026).
The organizers also acknowledged an important limitation: they did not have a credible external mid-market benchmark against which to compare that result. The event therefore treated the room's responses as an initial data point rather than a comparison against an established standard (The AI Nightmare, 2026).
That distinction is important.
Context switching, repeated information gathering, incomplete documentation, ticket updates, and manual coordination are familiar problems in service operations. But familiarity does not automatically make them well measured.
If an organization believes technicians lose 15 or 20 percent of their time to this work, the first response should not necessarily be:
"AI can recover 20 percent."
The better response is:
"Can we measure where the time is actually going?"
That creates a stronger starting point for automation. AI should not be asked to solve a productivity problem that has never been baselined.
The Hardest ROI Question: What Happens to the Time You Save?
The discussion about AI return on investment exposed an even more difficult issue. Several participants described being able to see operational improvements from AI without being able to translate those improvements cleanly into financial returns (The AI Nightmare, 2026).
A technician may save an hour, a workflow may require fewer manual steps, documentation may be completed faster, and a support analyst may handle more work. Those are real improvements.
But from a finance perspective, the question becomes:
What happened to the capacity that was created?
Participants noted that if headcount remains unchanged, labor expenditure remains flat, and revenue does not increase, the organization may have created capacity without yet realizing a direct financial return from it (The AI Nightmare, 2026).
That does not mean the AI investment failed. It means efficiency and realized value are not the same thing. One useful framing raised during the discussion was to ask whether an initiative does at least one of three things:
Reduce cost. Create or extract value. Mitigate risk.
That creates a more disciplined connection between technical improvement and outcomes the wider business already understands (The AI Nightmare, 2026).
For example, if AI gives a technician back five hours per week, leadership still needs to decide what those five hours will be used for. More customers? Preventive work? Higher value engineering? Faster project delivery? Reduced overtime? Avoided hiring?
Until that connection is made, "hours saved" remains an operational metric rather than a complete ROI argument.
The Cost That Appears After the AI Project Starts
The final poll focused on unplanned AI implementation costs. The options included knowledge base cleanup, change management and retraining, and higher than expected LLM licensing costs. Knowledge base cleanup emerged as the strongest concern among the attendees participating in the poll (The AI Nightmare, 2026).
That result made sense in the context of the discussion. Many AI projects begin with an assumption that organizational knowledge already exists and simply needs to be connected to a model. Implementation often reveals something different.
Documentation may be outdated. Two articles may describe the same process differently. Information can sit in individual mailboxes or chat conversations. Ownership may be unclear. Categories may be inconsistent. The current procedure may be difficult to distinguish from an obsolete one. In some cases, important knowledge was never documented in the first place.
At that point, the AI project becomes a data and knowledge management project.
One attendee described discovering exactly that after connecting AI to an existing knowledge base. The model itself was not the main constraint. Considerable work was required to make the underlying information usable and trustworthy (The AI Nightmare, 2026).
That is an important budgeting lesson:
The cost of enterprise AI is not only the model, platform, or license. It is also the cost of making organizational knowledge fit for machine use.
Governance Is Lagging Behind Ambition
The discussion eventually moved from local experience to broader industry research. Deloitte's 2026 research provides useful external context. Its survey covered more than 3,200 business and IT leaders across 24 countries and found that only 21 percent of respondents said their organizations had a mature governance model for agentic AI. At the same time, 74 percent expected their organizations to be using AI agents to at least a moderate degree by 2027 (Deloitte, 2026).
Those figures were also highlighted in secondary coverage of the Deloitte findings, including reporting from GRC Report and LetsDataScience, both of which emphasized the same underlying tension: enterprise adoption of AI agents is moving faster than governance maturity (GRC Report, 2026; LetsDataScience, 2026).
That difference is significant. Organizations appear to be building toward widespread agent adoption faster than they are building the governance needed to manage that adoption.
Deloitte identifies capabilities such as defining which decisions AI agents can make independently, establishing human approval boundaries, monitoring agent behavior, and maintaining audit trails of agent actions as important parts of mature governance (Deloitte, 2026). The secondary coverage reinforces the same point by framing governance as the area struggling to keep pace with scaling and deployment (GRC Report, 2026; LetsDataScience, 2026).
The phrase used during the evening was:
Governance is the gap, not capability (The AI Nightmare, 2026).
I think that captures the situation well. Most organizations are no longer asking whether an AI agent can perform a task. Increasingly, the difficult questions are: Should it? Under what conditions? Using which information? With whose authority? How do we know what it did? What happens when it gets something wrong?
Those are operating model questions, not simply model capability questions.
What IT Leaders Should Measure Differently
The evening reinforced something I have increasingly seen across enterprise AI discussions. Organizations often begin with the technology and then search for a measurement framework afterward. The sequence should probably be reversed.
Before implementing AI into a service workflow, understand the current process well enough to establish a baseline. How many tickets are avoidable? Why are they being created? How much technician time is spent gathering information? How much is spent documenting work? How often does context switching occur? Which requests could genuinely be resolved without a technician? What does poor knowledge quality currently cost?
Only then does it become possible to tell whether AI changed the operation. Otherwise, organizations risk comparing an AI-enabled future against an operational past they never measured properly.
A Practical Checklist for IT and Service Leaders
- Confirm That the Benchmark Actually Exists Before putting an "industry standard" into a business case, find the source. Check how the metric was defined, which organizations were measured, what population was surveyed, and whether the comparison is genuinely relevant to your operating environment.
The discussion at The AI Nightmare demonstrated how quickly organizations can end up setting targets where no commonly accepted external benchmark is available (The AI Nightmare, 2026).
If the benchmark cannot be sourced, treat it as an assumption.
- Establish Your Own Baseline Before Selecting the AI Target Measure the process before automating it. For ticket deflection, understand current ticket volume, avoidable demand, self-service usage, escalation patterns, and knowledge quality. For productivity, understand where technician time is actually going.
Your own baseline may ultimately be more useful than a generic industry average.
- Measure Demand Prevented, Not Only Work Completed Resolution time and ticket closure still matter, but add measures that capture whether users avoided needing support in the first place.
The discussion around ticket prevention versus closure highlighted the value of successful self-service, recurring issue elimination, automated remediation, stronger knowledge effectiveness, and reductions in repeat demand (The AI Nightmare, 2026).
- Separate Capacity Creation From Financial ROI If AI saves 1,000 technician hours, identify what happens to those hours.
Do they avoid hiring?
Reduce overtime?
Increase customer capacity?
Accelerate projects?
Improve preventive maintenance?
The ROI discussion at the event highlighted the gap between operational capacity creation and financial value realization (The AI Nightmare, 2026).
Without that second step, hours saved are evidence of productivity improvement, not necessarily realized financial return.
- Budget for Knowledge Preparation Do not assume the existing knowledge base is AI-ready.
Include discovery, deduplication, ownership, taxonomy, validation, access controls, lifecycle management, and ongoing maintenance in the implementation plan.
Knowledge base cleanup emerged as the leading unplanned cost concern in the event's live polling (The AI Nightmare, 2026).
The information layer may require more work than the model layer.
- Treat Adoption and Governance as Separate Maturity Curves An organization can deploy agents quickly while remaining immature in governance.
Deloitte's finding that 74 percent expect moderate or greater agent adoption by 2027, while only 21 percent report mature agentic AI governance, illustrates the difference between these two maturity curves (Deloitte, 2026). GRC Report and LetsDataScience also highlighted this same divergence in their coverage of the Deloitte findings (GRC Report, 2026; LetsDataScience, 2026).
Measure both.
Track how broadly AI is being used, but also track approval boundaries, access controls, monitoring, auditability, exception handling, human escalation, and accountability.
- Build Measurement Into the Workflow From the Beginning Do not wait until an executive asks for ROI six months after deployment.
Define before implementation what should improve, what the baseline is, how improvement will be measured, who owns the metric, how frequently it will be reviewed, and what outcome would cause the initiative to change or stop.
AI measurement should be part of the operating model, not an afterthought.
Closing Thought
What stayed with me from the evening was not a particular AI product or a spectacular automation result. It was how many experienced IT leaders were willing to acknowledge that some of the numbers they are being asked to manage toward are still immature.
Ticket deflection targets varied widely. Technician productivity losses were often being estimated rather than measured. Operational efficiency did not automatically translate into financial return. Knowledge preparation appeared as a high hidden cost. Those were themes that emerged directly from the evening's polling and discussion (The AI Nightmare, 2026).
At the same time, Deloitte's external research shows a similar maturity gap at a much broader level: agent adoption is moving ahead quickly while mature governance remains far less common (Deloitte, 2026). Secondary reporting from GRC Report and LetsDataScience reinforces that same interpretation, emphasizing that governance is struggling to keep pace with enterprise agent adoption (GRC Report, 2026; LetsDataScience, 2026).
None of that means enterprise AI is failing.
It means the discipline around measuring and governing it is still developing.
That may be the more important stage of maturity. Every established operational discipline eventually develops common definitions, baselines, controls, and benchmarks. AI-enabled IT service delivery is still building many of those.
Until stronger benchmarks emerge, technology leaders may be better served by doing something less exciting but more defensible: measure the starting point accurately, define the business outcome before deploying AI, track what actually changes, and be willing to say when the benchmark does not yet exist.
Because the question is no longer simply:
"Can AI save us time?"
The better question is:
"What changed in the business because of the time it saved?"
References
Deloitte (2026) Business and IT leaders report AI agents are scaling faster than their guardrails. Deloitte Insights, 24 April 2026. Available at: https://www.deloitte.com/us/en/insights/topics/emerging-technologies/ai-agents-scaling-faster.html
GRC Report (2026): Deloitte survey finds enterprises charging ahead with AI agents while governance struggles to keep up. Available at: https://www.grcreport.com/post/deloitte-survey-finds-enterprises-charging-ahead-with-ai-agents-while-governance-struggles-to-keep-up
LetsDataScience (2026) Deloitte finds agentic AI governance lags adoption. Available at: https://letsdatascience.com/news/deloitte-finds-agentic-ai-governance-lags-adoption-db8826b6
The AI Nightmare (2026): Live polling and roundtable discussion on AI in IT service delivery. IT leadership dinner, 21 October 2026. Unpublished event discussion.
Top comments (0)