DEV Community: Evgenii Basmov

Analyze This: In-Depth Multi-Categorical Analysis of Competitors' Customer Reviews with ChatGPT API in Record Time

Evgenii Basmov — Tue, 02 May 2023 09:51:38 +0000

I want to share details of the application of ChatGPT API for parsing, analyzing, and summarizing competitors' customer reviews.

My friend Anna Dormeneva works in a marketing research company, called HINTS. One of their clients, a prominent ed-tech company offering "English as a Foreign Professional Language'' courses, wanted to know what people thought about their competitors and how they could get ahead. Among other things, Anna, the research team leader on the project, decided to analyze customer reviews from various review aggregators.

As someone who enjoys experimenting with OpenAI and Huggingface APIs in my downtime, I sensed an opportunity when I heard about the challenge of finding someone capable of analyzing thousands of reviews both quickly and consistently. And just like that, I knew what my long Easter weekend would be about.

Here's how you do reviews analysis manually:

Decide what review aggregators you're going to parse.
Write parsers for each review aggregator website. Libraries like Scrapy, beautifulsoup4, or Selenium will help you a lot, but you still have to figure out XPaths on your own.
Come up with some aspects of analysis as well as go through a bunch of reviews to see what people mention to decide what categories your analysis is going to use (like, "mobile app availability", or "refund policy", etc)
MOST EXCITING PART: go through hundreds of reviews and try to analyze each according to your chosen categories. Try to keep up the quality of your analysis.
Aggregate the results in a spreadsheet and draw graphs and conclusions.

Every step in this process can be delegated to an LLM. In my case, it was the ChatGPT API. If I'm not mistaken, it was still a gpt-3.5-turbo model.

Here's the overview of the AI-powered process:

As you can see, there are still humans in the picture. They are there to make conclusions and to check whether the analysis aspects the machine suggests are actually relevant to us. I guess further development might help in automating that as well.

I wanted to just run an app, type in the company name and get thousands of parsed and analyzed reviews on that company service. The prototype I was building was concentrating on solving one particular problem of analyzing customer reviews on courses. The approach to different industries and business domains also can be included in further research and development.

When I type in a company name, the app uses Google to search for "${companyName} reviews". Then we need to solve a problem of parsing reviews from HTML we never saw and therefore don't have a parser ready. Here's how ChatGPT might help:

I take HTML and divide it into a few pieces, each of which must fit into the model context window and is likely to contain what I'm looking for - customer reviews. Then I send those pieces to ChatGPT with a prompt asking it to construct an XPath expression for retrieving reviews. Having a number of candidates for the XPath expression I use them to try to get reviews from the HTML I have and with some statistical magic, I usually get the right one and all the reviews from the page.

Of course, it is not always that simple, because some review aggregators only show "review previews" and a button "Read full review", they change HTML selectors on the fly, and try to come up with unparsable pagination. I don't have perfect solutions for those problems, but it's only a prototype, of course, there is a lot of work ahead!

After we get all the reviews we can take a few samples and ask ChatGPT to come up with categories of analysis. This is a semi-automatic process since we want a person to decide on the final list of aspects. In this case, it was:

Teaching approach
Platform convenience
Course program
Mobile app availability
Price
Teacher quality
Cancellation/rescheduling policy
Studying materials quality
Customer service
Course effectiveness
Course duration
Course schedule
Marketing
Speaking practice
Homework

When we are happy with the categories list we analyze every review based on the list.

Example of a review:

Its analysis result:

{
   "review": "review text...",
   "analysis": {
       "teaching approach": {
           "sentiment": "0",
           "context": ""
       },
       "platform convenience": {
           "sentiment": "1",
           "context": "I really like the vocabulary-trainer, diverse lessons"
       },
       "course program": {
           "sentiment": "0",
           "context": ""
       },
       "mobile app availability": {
           "sentiment": "0",
           "context": ""
       },
       "price": {
           "sentiment": "-1",
           "context": "Cannot reschedule lessons on a more budget-friendly plan"
       },
       "teacher quality": {
           "sentiment": "1",
           "context": "I like the teacher, she also has a cat ^_^"
       },
       "cancellation/rescheduling policy": {
           "sentiment": "0",
           "context": ""
       },
       "studying materials quality": {
           "sentiment": "1",
           "context": "There's also a great course on tenses that I bought separately. It changed my understanding of how tenses are used and how they work in the language."
       },
       "customer service": {
           "sentiment": "0",
           "context": ""
       },
       "course effectiveness": {
           "sentiment": "1",
           "context": "The lessons are a great distraction from my routine and work because I have to talk a lot, so my brain gets a workout"
       },
       "course duration": {
           "sentiment": "0",
           "context": ""
       },
       "course schedule": {
           "sentiment": "1",
           "context": "It's also great that the lessons are at a convenient time for me. That was one of the main criteria for me, to find a schedule that works for me."
       },
       "marketing": {
           "sentiment": "0",
           "context": ""
       },
       "speaking practice": {
           "sentiment": "1",
           "context": "The lessons are a great distraction from my routine and work because I have to talk a lot"
       },
       "homework": {
           "sentiment": "1",
           "context": "the variety of homework assignments (there are listening exercises, grammar tasks, listening without transcription, and then the teacher opens it up)"
       }
   }
}

Based on all results we can take one category, take all the context related to that category and ask ChatGPT to summarize the analysis of it.

The prompt I used looked like this:

[
  {
    "role": "system", 
    "content": (
       f"You analyze customer reviews of an online language course. "
       f"You will be given an aspect of analysis, and a list of results, consisting of sentiment and context. "
       f"You will be asked to summarize the results."
    )
  },{
    "role": "user", 
    "content": f"Point of analysis is: {point_name}. Results:\n{points}"
  }
]

The result looks something like this:

The analysis of the reviews shows that the majority of users have positive feedback about the homework in the online course. They find it convenient to do homework in the app, which it can be done on any device, and to learn new words. Many users note that homework can be done anywhere and at any convenient time. Users also appreciate the variety of tasks and materials, and the ability to choose the volume of homework and complete it according to their level of knowledge. Some users express dissatisfaction with certain aspects, such as unclear audio texts or an underdeveloped mobile application. However, overall, the homework in the online course has received positive reviews.

In the end, we end up with a high-quality analysis of hundreds of customer reviews in just a few hours at most, whereas the manual process would take us around 120 hours of very inconsistent and likely less-quality results.

I exported everything in Excel format so Anna could create some graphs for use in a presentation for a client. Combined with other things Anna's team did, the project was a resounding success!

Dancing with Spaghetti: Managing Real-World Complexity for Coding Wizards

Evgenii Basmov — Tue, 18 Apr 2023 08:15:49 +0000

Arguing about programming paradigms is like debating the best way to eat spaghetti – everyone has their preferred method, but in the end, we're all just trying to untangle a messy, saucy problem.

I believe all those arguments about coding paradigms, ideologies, and styles are fundamentally based on people's misconception that their code can somehow decrease the complexity of the problem they're trying to solve.

In one of many YouTube videos on the topic of why OOP is bad, there's a video with millions of views where the author (Brian Will) poses a question towards the end:

Are we actually decreasing the total complexity of our program by splitting code into many small methods and separate classes, or are we just displacing the complexity, merely spreading it around?

He's right to notice that structuring a program with objects doesn't help to decrease the total complexity, but I would argue that it's impossible regardless of the way you code.
Don't get me wrong; you could easily increase the complexity if you're not careful, but that doesn't have anything to do with your paradigm of choice. I'm sure people have already come up with all kinds of FizzBuzzEnterpriseEdition using different styles and approaches.

I think Brian and most people who keep bringing up these arguments don't understand one simple thing: our profession is not about complexity minimization; it's complexity management.

I like to think about programming as “Complexity Bending”
We take a problem and shift and displace its complexity in a way that is useful for us. We can use libraries and tools like databases, message queues, and schedulers to delegate some aspects of a problem to them, but the complexity that, for example, a database helps us deal with doesn't go anywhere. The database developers had to manage the complexity of data storage, accessing, availability, and so on.

So, if OOP or any other approach helps you and your team better understand the problem and manage its complexity, good for you. But if you hope to reduce the problem's complexity, not just manage it, then you're going to work in a very frustrating environment, because what you want to achieve is simply not possible.

Instead of devoting yourself to some "school of thought," you probably should:

Break down problems into smaller, more manageable pieces. This is very trivial, yet when this idea gets named the "Single Responsibility Principle" and "Dependency Injection," some people grab their pitchforks. My theory is that this happens because people are introduced to these ideas before they have enough real-world experience to connect them to the base idea of breaking down problems. They think it is some complication "gatekeepers" came up with just to complicate their CS exams this semester. As with many things in life, if deep down you don't feel like you need it, don't bother. Don't use SRP or DI until you need them. Just keep them on your radar. And when you eventually feel you need them, don't be shy to rely on them.
While some might see microservices, modularity, and serverless code as a sneaky way to make developers replaceable, the truth is, there's a lot more to it than just fulfilling management's dark fantasies. Sure, it might help with that too. But, it also means you can more easily hop between jobs or projects without getting stuck in the mire of "job security" (a.k.a. spaghetti code). Let's face it, who wouldn't prefer working with code that's a breeze to understand and maintain? It might not be as thrilling as untangling a complex web of interdependent modules, but it sure beats needing a degree in archaeology just to figure out how the darn thing works.
Update your README. Ah, documentation – the forgotten stepchild of programming, locked away in the attic. We've evolved as a society enough to only approve merge requests with unit tests, but when it comes to documentation, it's often "oh, yeah, we don't keep it up-to-date." Yikes! What was I thinking, attempting to follow the README instructions to deploy the app locally? Let's be real: getting to know a codebase is like navigating a foreign city. Sure, you can find the nearest Starbucks, but try doing it with a map from the 1800s.
Subpar software artifacts often result from stubbornly sticking to convoluted algorithms, whereas embracing the KISS principle usually leads to a decrease in cognitive load and a corresponding boost in system sturdiness. In simpler terms: if you care about something, don't overcomplicate it. I know it's tempting to craft a one-liner using just a smattering of symbols, but let's fight that urge, shall we?

In summary, throw yourself a complexity party! Transform into a programming wizard who conjures up a "do it" button for those who thrive in other realms, like business, medical research, finance, etc. You'll enrich their lives by harnessing your passion for technical solutions and mastery over complexity.

So, stop the endless fights over programming paradigms and embark on an all-you-can-learn buffet, sampling knowledge from various sources that can prove helpful in different circumstances. Remember, it's not about reducing complexity – it's about embracing it and taming the wild beast to create a better world for you and others!

And, as you set off on this journey, never forget the immortal words of Albert Einstein: "Everything should be made as simple as possible, but not simpler." In the world of programming, that means accepting that complexity will always be a part of the job, but we can choose to dance with it rather than fight against it. Now, go out there and show that complexity who's boss!