🦄 Making great presentations more accessible.
This project aims to enhances multilingual accessibility and discoverability while maintaining the integrity of original content. Detailed transcriptions and keyframes preserve the nuances and technical insights that make each session compelling.
Overview
📖 AWS re:Invent 2025 - Building AI That Customers Actually Trust (AIM2203)
In this video, Samantha Snetselaar from Qualtrics presents their approach to AI agent management, focusing on maintaining human connection in customer experience. She introduces two key concepts: agent efficacy for measuring deployed AI agents, and experience agents as Qualtrics' solution. The presentation highlights that AI in customer service has the highest failure rate among AI applications, with 73% of consumers using AI daily but finding it underperforming in customer service. Qualtrics addresses this through their intelligence layer that measures AI agents against traditional support channels using their CSAP model, examining customer satisfaction, sentiment, effort, and resolution rates without LLM bias. Their methodology includes conversational analytics surveys and benchmarking across multiple agent types. Qualtrics positions experience agents as an orchestration layer that transforms experience management from reactive measurement to proactive activation, integrating operational data with experience insights to resolve issues automatically and increase customer loyalty while decreasing attrition.
; This article is entirely auto-generated while preserving the original presentation content as much as possible. Please note that there may be typos or inaccuracies.
Main Part
The Experience Gap: Why AI Agents Are Failing to Deliver Human Connection
Hi everyone, my name is Samantha Snetselaar. I am with Qualtrics. Qualtrics is an experience management organization, and I have been at Qualtrics for a little over nine years now. One thing that continually keeps me at Qualtrics is knowing that our mission is all about human connection. At Qualtrics, we see that there are really four key pillars of an organization. You have your brand, which is that promise of what you are putting out into the market about who you are and what you stand for. It is embodied in the products and services that you deliver, which is shaped by your employees, who ultimately deliver the customer experience. That customer experience then correlates back to what your brand is.
Organizations today really need to think about how they can integrate all those different key pillars into one experience layer to really understand how each is affecting the other. At Qualtrics, what we look at is how we can help organizations deepen the relationships and build that bridge with their customers and with their employees. We took that same methodology and that same philosophy and we applied it to our AI and then more specifically our agents. In today's conversation, I am going to walk you through two components of Qualtrics methodology. The first is how we are going to help you with your AI agents that are already deployed today, which we call agent efficacy. So how can we make sure that those agents are really performing in the right way to build that human connection and that empathy with your customers and with your employees? The second part I will be able to share a little bit more about how Qualtrics thinks about experienced agents and where and when are the right places to deploy them.
When looking at Qualtrics and understanding our perspective, we are really seeing a huge focus on the rush to automate. Organizations are really losing that human connection. You have organizations that are buying all these AI tools and technologies, but it is falling short. You are sending customers down dead ends and employees are so inundated with all of these new tools and technology they have to use that it is really causing employee burnout. People do not just want faster responses and faster resolution. They want to feel seen and heard.
Do not just take our word for it. We had our Execum Institute who recently did this 2026 customer experience trends report. They went all over the globe and surveyed consumers trying to get a better understanding of what is going on in the customer experience world. What is exciting is that when you look at consumer comfort with AI, it has rebounded. Seventy-three percent of us are using it for our daily tasks. However, when you drill down specifically into customer service, AI is failing. It has the highest failure rate of any AI application. In addition to that, it really underperforms on specific metrics when you are looking at ease of use, whether it saved you time, whether you found the ideas useful, and most importantly, whether you received any benefits from that AI or that agent interaction.
We talked with a lot of our customers and we are starting to see this is especially true when it comes to your customer service departments and customer experience parts of your organization. They are feeling the pain of this. There is so much pressure to use AI, so much pressure to be more productive, to be more efficient, to dig deeper into the insights. But AI technology is a tool and experience is that outcome. So AI does not equal experience. The question becomes how can you test and validate that your AI agents are really performing that great customer experience versus just really thinking about operational efficiencies and things like deflecting calls?
We start to see all of our customers come to us with these problems because we have always been focusing on how we can help organizations deliver that best in class customer experience. We have really diagnosed what problems we can help our customers solve. The first one is really focusing on customer experience blind spots. How can we help our customers who were starting to see that they could not easily answer the questions of whether their AI agents were improving that customer experience or not? There was damage to brand loyalty because they were not providing that great interaction with their customers. Second, there was no consistent benchmarking, so they did not have the ability to understand how their human agent in the call center is performing as it relates to their AI agent that they have on their website. What is the difference in that customer experience, and also when can they understand when they can throttle down that human agent and throttle up that AI agent because they are providing a just as good or better customer experience? There is also ineffective optimization. This is more so when thinking about what is that root cause of when that AI agent is deployed, why did it fail?
Organizations say to use thumbs up or thumbs down, but that's all the information they have available to them. Additionally, you have siloed reporting, so you have all these insights from different vendors and different tools. You might have your own homegrown agent, you might be deploying an AWS agent, a Salesforce agent, or a ServiceNow agent, but how do you know how those are all performing holistically and being able to benchmark to truly understand what that conversation and connection is like with your customers and employees.
Agent Efficacy: Qualtrics' Intelligence Layer for Measuring and Improving AI Agent Performance
That's where Qualtrics comes into play. We've provided an intelligence layer to ensure that your AI agents are not just efficient but are also driving positive CX outcomes. How can we focus on top line revenue growth and understand that there are cross-sell and upsell opportunities? We're improving brand loyalty and solving customer problems while also helping you with bottom line growth and operational efficiencies. We offer a best practice methodology to measure and improve AI agent experience while ensuring that AI automation is a positive customer experience for your customers and employees.
What we look to do is improve the business impact of AI agents, ensure positive customer experience, and help avoid AI agent risk. The way we do that is to measure your AI channels against all of your traditional support channels. You're going to have your human in the contact center, your legacy chatbot, and your homegrown agent that you might be using, plus all the different agents that you have deployed that are interacting with your customers. From there, we measure all agent interactions against each other without the bias of an LLM as a judge or proprietary measurement.
We report beyond just customer satisfaction and thumbs up or thumbs down to really understand sentiment, emotion, intent, effort, and what that interaction is like with your agent. This results in faster adoption of new AI channels by being able to indicate when they're going to start to outperform and reduce that risk and build trust of those agents within your organization. We're also going to help determine which AI channels are providing the best customer experience from that holistic perspective and then optimize the agents.
Once we're able to diagnose and get a full understanding of what that agent is doing and whether it's providing a great customer experience or a poor one, we can feed that information back into coaching the author and when you're building the agents in your studio. We can continually feed that information back to improve that agent again and again to keep delivering a better customer experience. We can also feed all that sentiment data into the reference layer for the LLM.
When looking at how we measure AI agent effectiveness, there are several components. One is CX and outcome metrics, which is Qualtrics' bread and butter. We look at customer satisfaction, sentiment, effort, and resolution rate. We also bring in risk, security, and quality assurance. How can we look at manual quality assurance? How can we look at toxicity that's coming up or biases within your agent conversations? How can we think about customer data leakage indicators?
Taking it further, we look at operational efficiency. What's the containment rate? How's the average handling time? Then we move further down into agent quality and performance, percentage of errors, agent confusion, and customer behavior and engagement. We're able to pull all that information from both the conversation between your agent and your customer. At the end of that conversation, we bring up a Qualtrics survey.
Within that survey, we've applied our own methodology to ensure that you're getting the most information on how to improve and where the gaps are as they relate to your customer experience. We're focusing on customer satisfaction and our AP3 model, and then we're looking at operational metrics. When you think about the CSAP perspective, we're going to be asking questions about support topics team by team and against operational data. How was that overall satisfaction when you were having this conversation with agents? Then we're going to look at agent performance with feedback specifically about the agent that handled that interaction and resolution. Did we solve your problem? Tell us why. And then we look at unstructured feedback.
The great thing about Qualtrics with our surveys is that we provide a generative AI layer for conversational analytics. When you take a survey, you might answer a couple of questions and then there's an open-ended text box for you to fill out based on your response. Based on the response you give in that open-ended text box, our survey platform will start to ask additional questions based on your response to provide more information. It will now tell us more about the specific topic to allow you as an organization to gather as much information as you can from that respondent when they are giving you feedback.
When they're giving you feedback, we also look at operational metrics within the study. We consider things like time expired, asking how long they actually spoke with the agent, looking at things like channel used—which channels did you use before you interacted with this agent—and also understanding the number of contacts. Is this your first time reaching out to the agent? And then lastly, thinking about resolution: did we resolve your issue? We're gathering all of that feedback and also taking in different benchmarks of understanding, looking at friendliness, knowledge, and understanding.
From a friendliness perspective, I want to know these as separate coachable skills. That's why we're going to ask separate questions for each of these components and then bring them into the broader survey. Perception of friendliness is obviously a huge key thing. You expect your customer service agents to be friendly right off the bat. Knowledge is about whether they actually understand and were not only able to answer your question, but whether they truly understand the root cause of your problem and help you get to resolution. And then lastly with understanding, it's the perception of how well the agent understood the issue rather than just the agent's knowledge.
From Measurement to Activation: Experience Agents as an Orchestration System
That's a little bit about how we're thinking through our methodology and our approach for what we call agent efficacy. I'd be happy to talk to you more about how you can think about where your AI agents are deployed today and whether they are interacting with your customer or you can take the same methodology and apply it to internal agents that are interacting with your employees to continually help build that customer experience. What I also want to touch on is how Qualtrics has taken this methodology and applied it to when we're building our own experience agents and where we see the market going when it comes to experience management.
Qualtrics has always been focused on helping organizations listen to their customers when they want to get feedback and understand that information by bringing it into the platform for analysis, looking at key drivers and things of that nature to really understand macro trends. The organization can see what you need to do to improve that customer experience, but also being able to act and optimize. Thinking through things like that micro perspective, how can I close the loop with an individual when they do have a problem? Where we're seeing modern experience management programs go, and where Qualtrics really is at the frontier of helping organizations think about experience management, is thinking about how we can move from this measurement system to an activation engine.
How can we listen across every signal, not just direct feedback when our customers are answering a survey, but more so thinking about where we can go out and pull feedback whether that be within social, on digital, and bringing in operational data sets or even looking at things like chats and agents? How can we move from really being reactive—here's what you need to do—to being more proactive? Understanding what's next, how can we spot risks that are at the early stage? How can we forecast churn before it begins? And then how can we test what-if scenarios and strategies for our organization before we commit?
When thinking about that activation and optimization at scale, how can we resolve issues automatically, route to the right owners, and close the loop as fast as possible? When you think about the power of experience agents, we are already deployed in a large number of organizations and we are that touch point that your customers have with you. How can we think about providing these agents and providing this empathetic and authentic conversation as well as the action behind it and being able to close the loop as quickly as possible? How can we act as a system of orchestration, not just really thinking about dashboards, but how can we route information to the right people at the right time so they can act on it within the organization?
How can we help resolve issues before they escalate and provide outcomes that you can prove and are delivered fast and at scale? With Qualtrics agents, we're looking to increase that resolution time, we're looking to increase loyalty with your customers, and we're also looking to decrease that attrition. When you think about where experience agents orchestrate in your technology stack, think of it as the experience layer. You have all this operational data that sits in relation to your customers. You've got your CRM, your customer support, your call center, your CDP, and so on. What Qualtrics does is that's all operational. Qualtrics is then the experience layer of understanding not what, which is what operational data gives you, but why. What is that experience human connection element that's connected to all of that different data?
We're already at the place where we're having that interaction with your customers whether it's through feedback, through surveys, or looking at digital and site intercepts and bringing in all that behavioral data of what your customers are doing on your website and apps and within your call center and how those conversations are going in your human call center today and then bringing in social. With our ability to take all that information and then provide the right analysis and action, that's really what sets our experience agents apart from other agents out there today.
My call to action for all of you is this: if you do have Qualtrics appointed in your organization or another survey provider, think about it like this. Let's think about if there's one survey that your organization is leveraging where they're already closing the loop. What I mean by that is if a survey pops up, someone asks a question, and then it goes to someone in the customer service center to answer that question or to fix that problem. What if we can leverage our experience agent to have that conversation already and either route to the human to review a response before it sends or automatically have that resolution already sent in real time? That's a little bit about the Qualtrics agents. I'd be happy to discuss more, but thank you for your time. I really appreciate it.
; This article is entirely auto-generated using Amazon Bedrock.
Top comments (0)