Abstract
Executive expectations for AI deployment continue to rise, and stakeholders increasingly want clear proof of return on investment. Yet many organizations face a measurement gap: traditional KPIs often fail to capture the full value of AI-enabled service delivery, especially when performance improvements extend beyond speed and efficiency.
In this Customer Engagement Leadership Council session, Pamela Fusting presented a real-world case study of AI deployment in a resident service center, exploring how one early adopter realized the need to re-baseline traditional speed and efficiency KPIs. Instead, a new barometer, Value on Investment (VOI) became a critical benchmark for assessing the deployment’s success.
Key Action Items
- Identify and prioritize high-impact AI use cases within existing service workflows
- Define ROI and VOI metrics that capture efficiency, complexity, cost avoidance, and service quality
- Establish guardrails and operational measures to integrate AI while maintaining service levels
Introduction
Customer Experience leader Pam Fustig introduced herself and briefly described her work at the Tennessee Department of Human Services, which serves 2–2.5M Tennesseans across 95 counties. She outlined the agency’s scope of services and the population served, establishing the contact-center context for the case study.
Pam shared highlights of a 12-month AI deployment process, discussing the initiative and results. She framed the topic as AI deployments in contact centers and referenced industry research showing broad interest in AI and a desire for quick return on investment (ROI).
Case Study Takeaways
- Timeline: multi-year program with a 12-month AI push
- Abandonment rate spiked from 2.3% to 43.4% initially – a misleading statistic that needed to be put in context
- Average handle time increased after launch – in part because many calls were quickly resolved by automation and the more complex calls that remained required longer handle time
- Containment and self-service obscured failure signals
- Sustained ~25% reduction in after-call wrap-up time
- Insight: AI measurement is a different reality, no standard industry benchmarks currently exist
- Recommendation: shift ROI to a “Value on Investment” metric set – move from a “feature first” mentality to an “outcome based” mindset
KEY DISCUSSION POINTS
Objectives and AI tools
The specific AI tools and objectives deployed over the past year were outlined. They included virtual agents, generative call summarization, authenticated chat, proactive SMS, voice agents, and extended hours for customer service. Routing, agent assist, and auto-summarization were all part of the new configurations. Desired outcomes included faster resolution, improved metrics, and repurposing agents toward casework to demonstrate dollar savings.
Deployment outcomes and metrics paradox
Outcomes after deployment: traditional KPIs moved unfavorably—average handle time rose, abandonment spiked from 2.3% to 43.4%, and inbound volume initially increased—yet much of that reflected call containment where callers completed self-service or received texts and disconnected, prompting a need to re-baseline measurement approaches. A key takeaway: containment must be measured alongside abandonment to avoid misleading conclusions.
Lessons learned, must-dos, and pilot strategy
Pam advocated for measuring “Value on Investment” (financial and non-financial outcomes) and detailed lessons learned: search for performance impact, re-baseline KPIs, establish guardrails and secure executive sponsorship. She recommended prioritizing pilots and building agent capacity to monitor and refine AI. She reported a sustained ~25% reduction in after-call wrap-up time after fine-tuning. She also explained that AI exposes knowledge gaps, shifts caller behavior, and must be examined and/or tweaked before expected ROI is realized. AI tools need months of monitoring and fine-tuning after deployment.
Q&A and closing
When the floor was opened for questioning, participants affirmed the value-on-investment framing and raised practical questions. When a member asked why early quarters sometimes regress, these initial negative metrics were attributed to a mix of agent onboarding differences and AI behaviors that required more change management, training, and prompt adjustments. A misleading prompt-appeals example was shared –AI language misled (and confused) callers – and the corrective actions needed to remedy it. An important final question was posed: Is AI making it easier for your customers?
