Finally, hard data on a real-world AI business use case: It’s huge for customer service
Researchers measured gains and trade-offs from human customer service reps using an AI assistant. One big win? Fewer angry customers.
The tech giants are spending hundreds of billions to build out massive infrastructure needed for an imminent economy where AI supercharges productivity and boosts economic growth. That’s the theory, anyway. But how much will AI actually boost productivity in business? It’s very much an open question, with a paucity of data available.
But in the current issue of the Quarterly Journal of Economics, a research paper presents some hard data on the topic. Titled “Generative AI at Work,” the paper shares research by Erik Brynjolfsson, Danielle Li, and Lindsey Raymond, which measured productivity of real workers in an industry prime for AI-powered improvements: customer service. Their findings offer an important, early look at the value of incorporating AI in business.
The researchers found that when human customer service agents used an AI assistant, the company saw a 15% boost in productivity on average. But the gains were not evenly distributed. Less experienced agents saw the biggest boosts in productivity and speed, while more experienced agents saw smaller gains in speed and a slight decrease in quality. AI really helped when handling “moderately rare problems,” where the human agent might lack the knowledge to resolve the issue.
One of the other benefits of the use of AI assistants was that it helped international workers improve their fluency in English. After AI was introduced, the data showed big jumps in scores for “comprehensibility” and “native fluency” (i.e. the person seemed like a native American English speaker).
Most of us have had a frustrating customer service experience and the accompanying rage that it may induce. The study notes, “We see regular instances of swearing, verbal abuse, and ‘yelling’ (typing in all caps).” This can lead to attrition among customer service workers, which adds to costs.
In what might be the most promising (and surprising) finding in the study, the authors found that AI-assisted customer service resulted in customers being more polite and less likely to ask to speak to a manager.
“We find access to AI assistance significantly improves how customers treat agents of all skill and experience levels, with the largest effects for agents in the lower to lower-middle range of both the skill and tenure distributions,” the authors wrote.
The study followed 5,172 customer support agents at a Fortune 500 firm that sells business software. They staggered the introduction of an AI chatbot assistant that suggests responses to customer queries over eight months starting in late 2020 through 2021.
It’s important to note that in the fast-moving world of AI, this experiment took place a relatively long time ago. The chatbot used in this study was built using GPT-3, the model that preceded ChatGPT’s launch in November 2022. It was trained using successful customer service calls from top-performing agents.
As tech giants like Microsoft, Google, and Salesforce race to squeeze AI into the software we use, perhaps the greatest benefit will be that it not only helps us solve our problems faster, but it makes humans treat each other better.
