Analyze This: In-Depth Multi-Categorical Analysis of Competitors' Customer Reviews with ChatGPT API in Record Time

I want to share details of the application of ChatGPT API for parsing, analyzing, and summarizing competitors' customer reviews.

My friend Anna Dormeneva works in a marketing research company, called HINTS. One of their clients, a prominent ed-tech company offering "English as a Foreign Professional Language'' courses, wanted to know what people thought about their competitors and how they could get ahead. Among other things, Anna, the research team leader on the project, decided to analyze customer reviews from various review aggregators.

As someone who enjoys experimenting with OpenAI and Huggingface APIs in my downtime, I sensed an opportunity when I heard about the challenge of finding someone capable of analyzing thousands of reviews both quickly and consistently. And just like that, I knew what my long Easter weekend would be about.

Here's how you do reviews analysis manually:

1. Decide what review aggregators you're going to parse.
2. Write parsers for each review aggregator website. Libraries like Scrapy, beautifulsoup4, or Selenium will help you a lot, but you still have to figure out XPaths on your own.
3. Come up with some aspects of analysis as well as go through a bunch of reviews to see what people mention to decide what categories your analysis is going to use (like, "mobile app availability", or "refund policy", etc)
4. MOST EXCITING PART: go through hundreds of reviews and try to analyze each according to your chosen categories. Try to keep up the quality of your analysis.
5. Aggregate the results in a spreadsheet and draw graphs and conclusions.

Every step in this process can be delegated to an LLM. In my case, it was the ChatGPT API. If I'm not mistaken, it was still a gpt-3.5-turbo model.

Here's the overview of the AI-powered process:

As you can see, there are still humans in the picture. They are there to make conclusions and to check whether the analysis aspects the machine suggests are actually relevant to us. I guess further development might help in automating that as well.

I wanted to just run an app, type in the company name and get thousands of parsed and analyzed reviews on that company service. The prototype I was building was concentrating on solving one particular problem of analyzing customer reviews on courses. The approach to different industries and business domains also can be included in further research and development.

When I type in a company name, the app uses Google to search for "${companyName} reviews". Then we need to solve a problem of parsing reviews from HTML we never saw and therefore don't have a parser ready. Here's how ChatGPT might help:

I take HTML and divide it into a few pieces, each of which must fit into the model context window and is likely to contain what I'm looking for - customer reviews. Then I send those pieces to ChatGPT with a prompt asking it to construct an XPath expression for retrieving reviews. Having a number of candidates for the XPath expression I use them to try to get reviews from the HTML I have and with some statistical magic, I usually get the right one and all the reviews from the page.

Of course, it is not always that simple, because some review aggregators only show "review previews" and a button "Read full review", they change HTML selectors on the fly, and try to come up with unparsable pagination. I don't have perfect solutions for those problems, but it's only a prototype, of course, there is a lot of work ahead!

After we get all the reviews we can take a few samples and ask ChatGPT to come up with categories of analysis. This is a semi-automatic process since we want a person to decide on the final list of aspects. In this case, it was:

1. Teaching approach
2. Platform convenience
3. Course program
4. Mobile app availability
5. Price
6. Teacher quality
7. Cancellation/rescheduling policy
8. Studying materials quality
9. Customer service
10. Course effectiveness
11. Course duration
12. Course schedule
13. Marketing
14. Speaking practice
15. Homework

When we are happy with the categories list we analyze every review based on the list.

Example of a review:

Its analysis result:

{
   "review": "review text...",
   "analysis": {
       "teaching approach": {
           "sentiment": "0",
           "context": ""
       },
       "platform convenience": {
           "sentiment": "1",
           "context": "I really like the vocabulary-trainer, diverse lessons"
       },
       "course program": {
           "sentiment": "0",
           "context": ""
       },
       "mobile app availability": {
           "sentiment": "0",
           "context": ""
       },
       "price": {
           "sentiment": "-1",
           "context": "Cannot reschedule lessons on a more budget-friendly plan"
       },
       "teacher quality": {
           "sentiment": "1",
           "context": "I like the teacher, she also has a cat ^_^"
       },
       "cancellation/rescheduling policy": {
           "sentiment": "0",
           "context": ""
       },
       "studying materials quality": {
           "sentiment": "1",
           "context": "There's also a great course on tenses that I bought separately. It changed my understanding of how tenses are used and how they work in the language."
       },
       "customer service": {
           "sentiment": "0",
           "context": ""
       },
       "course effectiveness": {
           "sentiment": "1",
           "context": "The lessons are a great distraction from my routine and work because I have to talk a lot, so my brain gets a workout"
       },
       "course duration": {
           "sentiment": "0",
           "context": ""
       },
       "course schedule": {
           "sentiment": "1",
           "context": "It's also great that the lessons are at a convenient time for me. That was one of the main criteria for me, to find a schedule that works for me."
       },
       "marketing": {
           "sentiment": "0",
           "context": ""
       },
       "speaking practice": {
           "sentiment": "1",
           "context": "The lessons are a great distraction from my routine and work because I have to talk a lot"
       },
       "homework": {
           "sentiment": "1",
           "context": "the variety of homework assignments (there are listening exercises, grammar tasks, listening without transcription, and then the teacher opens it up)"
       }
   }
}

Based on all results we can take one category, take all the context related to that category and ask ChatGPT to summarize the analysis of it.

The prompt I used looked like this:

[
  {
    "role": "system", 
    "content": (
       f"You analyze customer reviews of an online language course. "
       f"You will be given an aspect of analysis, and a list of results, consisting of sentiment and context. "
       f"You will be asked to summarize the results."
    )
  },{
    "role": "user", 
    "content": f"Point of analysis is: {point_name}. Results:\n{points}"
  }
]

The result looks something like this:

The analysis of the reviews shows that the majority of users have positive feedback about the homework in the online course. They find it convenient to do homework in the app, which it can be done on any device, and to learn new words. Many users note that homework can be done anywhere and at any convenient time. Users also appreciate the variety of tasks and materials, and the ability to choose the volume of homework and complete it according to their level of knowledge. Some users express dissatisfaction with certain aspects, such as unclear audio texts or an underdeveloped mobile application. However, overall, the homework in the online course has received positive reviews.

In the end, we end up with a high-quality analysis of hundreds of customer reviews in just a few hours at most, whereas the manual process would take us around 120 hours of very inconsistent and likely less-quality results.

I exported everything in Excel format so Anna could create some graphs for use in a presentation for a client. Combined with other things Anna's team did, the project was a resounding success!