Turbocharging Voice of Customer Analytics Using Generative AI

Author: Duane Lyons

Generative AI has the potential to greatly improve and simplify the process of analyzing and understanding customer feedback. Leveraging text-centric models trained on massive amounts of data combined with a discipline defined as ‘prompt engineering’ will also allow organizations of all sizes to understand customer feedback much more easily. 

 

Prompt engineering is the process of structuring text that can be interpreted and understood by a generative AI model. Our first-hand experience has taught us that the combination of efficient prompt engineering integrated into LLM-enabled applications can have dramatic implications and deliver massive benefits.  

 

One application for this type of LLM-enabled analysis is Wavicle’s own ActiveInsights tool, a voice of customer (VoC) analytics application that is available in the AWS Marketplace and has been successfully implemented at multiple Wavicle customers. ActiveInsights is an end-to-end application that includes the ability to integrate with internal sources (like CSAT and employee feedback), third-party review sites such as Google Maps, and social media platforms to obtain location-level attributes, ratings, and reviews for each restaurant location (and competitor locations) as well as data visualizations and notifications for users. In addition, one of the platform’s core capabilities is to leverage sophisticated text analytics/natural language processing (NLP) algorithms to analyze each piece of customer feedback provided in textual format. This is a prime example of an application for LLMs where a well-engineered prompt can help organizations rapidly process and understand feedback.

 

Simplifying the process 

As I mentioned above, a core capability of ActiveInsights is to analyze each piece of customer feedback. Each piece of feedback is systematically analyzed in the following manner:

 

• Feedback is broken into sub-comments where each sub-comment represents an independent idea or thought. This is critical when analyzing customer feedback because customers will frequently provide feedback that touches on multiple aspects of a visit, experience, or product purchase. Without this critical pre-processing step, analyzing customer feedback becomes massively less insightful.   
• Each sub-comment is assigned to a specific topic (from a list of pre-defined topics) and the sentiment of the sub-comment is identified. 
• Lastly, the menu items or products that are being discussed are identified for each sub-comment. 

 

The ActiveInsights application does all of the above and does it extremely well. While the previous version of our solution required thousands of lines of custom coding to perform the analysis, now, an LLM can be used to enable this analysis more rapidly using customized prompts.

 

Analysis in action

With efficient prompt engineering, the new version of ActiveInsights has retired thousands of lines of custom code and leverages generative AI to analyze each piece of customer feedback and do the same analysis as above.

 

Let’s use the following review comment left on Google as an example: 

“Me and my granddaughters were out for a day of shopping and we decided to stop and get something to eat. I got a Double cheese burger, fries with No salt and a Caramel iced coffee. Double cheeseburger was very dry, fries were cold and caramel iced coffee was really cold. One granddaughter had a breakfast sandwich (bacon and eggs) combo with OJ. She loved it and she ate all of it. The other got a kid’s meal and it was sic! Much better than Dunkin” 

 

Leveraging and prompt engineering generative AI, the new version of ActiveInsights yields the following output: 

 

Sub-comment	Sentiment	Topic	Menu item
Me and my granddaughters were out for a day of shopping and we decided to stop and get something to eat.	Neutral	Customer experience	None
I got a Double cheeseburger, fries with No salt and a Caramel iced coffee.	Neutral	Order accuracy	Double cheeseburger, fries, caramel iced coffee
Double cheeseburger was very dry,	Negative	Food quality	Double cheeseburger
fries were cold	Negative	Food temperature	Fries
and caramel iced coffee was really cold.	Positive	Food temperature	Caramel iced coffee
One granddaughter had a breakfast sandwich (bacon and eggs) combo with OJ.	Neutral	Order accuracy	Breakfast sandwich (bacon and eggs) combo with OJ
She loved it and she ate all of it.	Positive	Food quality	Breakfast sandwich (bacon and eggs) combo with OJ
The other got a kid’s meal and it was sic!	Positive	Food quality	Kid’s meal
Much better than Dunkin	Positive	Competitor mention	None

 

So how did the LLM do? To answer that, I want to focus on format and accuracy. 

 

First, is the LLM providing the output in the correct format? The answer to that is YES, since it breaks the original comment into sub-comments, determines the sentiment, assigns each sub-comment to a topic, and identifies mentions of menu items.

 

Second, how accurate is the output? I have few thoughts here:

 

• At first glance, breaking out comments into sub-comments seems extremely accurate and perhaps better than what is part of our existing solution.  
• The assignment of sentiment is better as well. Specifically, it handles a very common edge case found in feedback for restaurant visits. Customers frequently provide a comment that includes a menu item and its temperature without additional context. Using the above as an example, “fries were cold” is intuitive to us as humans. However, commercially available sentiment engines do not do this well. From the above example, you see that the LLM handled this, as well as the other sub-comment “and caramel iced coffee was really cold,” correctly. 
• For the sub-comment “She loved it and she ate all of it,” it is worth noting that the LLM was able to identify that the menu item being discussed was a breakfast sandwich (although the menu item was not explicitly mentioned). In the NLP space, this is referred to as co-referencing. I’m highlighting this because the prompt engineering we constructed did NOT ask for this to be done, yet it still accomplished this! 

Why is this important? 

Thinking narrowly about our solution, ActiveInsights, as well as our existing customers, the impact is meaningful in several ways. Specifically:

 

• Accuracy: Improving the accuracy of the analysis process (such as breaking comments into sub-comments, calculating sentiment, and identifying which menu items are being discussed) allows greater insights to be gleaned from the data. 
• Simplification: Replacing thousands of lines of custom code with an efficient example of prompt engineering that can be understood by business users allows Wavicle and our client IT and data science personnel to retire thousands of lines code that are being maintained.  
• Cost reduction: The above example has been tested with ChatGPT4 and Claude 2. Assuming ChatGPT 4 and a volume of 30k+ comments to be analyzed daily (which is a lot!), the total monthly costs for enabling an LLM to do this analysis of customer feedback is only a few thousand dollars. Furthermore, this cost is likely to go down further.  

 

Thinking more broadly, how does the ability to leverage LLMs to easily perform complex analysis of customer feedback impact the enterprise software vendors in the space such as SMG, Medallia, and Qualtrics? While these platforms have a host of features, their ability to analyze customer feedback is at their core. If companies can now do this on their own with a simple, LLM-enabled application such as ActiveInsights, will they continue to spend what they spend on packaged applications for commoditized services such as the ability to execute surveys?

 

Taking the next steps 

Are you interested in seeing if your organization can leverage the power of generative AI to measurably improve your abilities to do VoC analytics? Want to eliminate your dependency on third-party SaaS solutions that overcharge for commoditized capabilities and keep your data locked within their silos? If so, contact us to see if you qualify for a no-cost proof-of-concept that includes an analysis of your customer feedback using ActiveInsights.

 

 Get in touch with us to learn more about how we can help you achieve this integration.