DEV Community: ozmasg

Food Project: Data source

ozmasg — Wed, 13 Apr 2022 16:30:37 +0000

Brief

In this blog, we'll discuss the data we need, how we can get it, and how we might want to format it.

What data do we need?

As discussed in the last post, the data we need is the ratings received for each restaurant per minimum interval. For the sake of brevity, we'll just assume for now that the minimum interval is a month.

The biggest problem that comes to mind is that we need this data not only for the present, but also at each interval in the past.

For example, we can easily get the current rating data at the time of writing, which is April 2022. But we also need the state of reviews at March 2022, February 2022, January 2022.

Where could we get this data?

I was hoping to use Google's own API as they have a generous monthly free package, but unfortunately it is quite limited for our purposes. It only offers current review data, and only a very limited view of a location's review history.

This means we'll have to lean on an alternative data source. After looking through a few options on my own, I've landed on DataForSEO, which provides us with a way to get scraped data from Google's search engine results.

How will we get this data

Our input options are limited to whatever Google has decided to make available on their SERPs (Search Engine Result Pages), because that's what DataForSEO operates on.

We'll get the data with the following steps:

Get a list of different neighbourhoods in Singapore, provided by the Google Locations API on DataForSEO.
Request 'food nearby' search results at each of these locations using DataForSEO's SERP API.
For each unique result, we will request available reviews using DataForSEO's Business API, using the Google Place ID's returned in the previous step.

This will give us - based on Google's discretion - up to 5300 locations, some of which may be duplicates, or non-food locations, which we will have to filter out.

After doing some initial testing, I've found that out of the 4985 locations returned by the SERP API, 4005 were duplicates, and 127 were non-food locations, leaving us with only 890 unique results.

It may be worth toying with increasing the number of results per location, or trying to look into the coverage of the areas provided by step 1, but this seems sufficient for our initial proof-of-concept.

Obstacles

What obstacles do we have?

Price

The first obstacle is pricing. The SERP API is very affordable, but the Business API is much more expensive, priced at $0.00075 per 10 reviews.

The aforementioned 890 unique locations add up to a total of 660,750 reviews (rounded up to the nearest 10 for each location). That would cost us approximately ~USD 55 to request all the results for all of the 890 locations. While that's not prohibitively expensive, it's not something I'd like to spend before ensuring that all of this will pan out the way I'm imagining.

I don't believe this to be a major obstacle - if the concept proves to be useful, $50 is not a large price to pay, even if it is not paid back. However, it may be prudent to rely instead on mock data, at least for the testing phase.

Limitations

The second obstacle is a limitation of DataForSEO's API. Their Business API also only returns up to 4490 results per location, which is not enough to cover the full depth of reviews of some more popular locations.

Although it would be ideal to cover each location's entire history, I believe it is not an essential part of our website, given that anything beyond 4490 results is probably reaching into the mid-2010s.

This may be a time to refine our requirements to include a target timeframe - perhaps 'recent trends over the last two years' or something of the sort, which should be covered by 4490 reviews.

Where are we at now?

We've laid out the steps for how we can get the data from DataForSEO, as well as discussed the major obstacles in dealing with their API, as well as how we can 'deal' with them.

I believe in the next step, we can finally start looking at how we will approach the design of our software.

Food Project: What do we need?

ozmasg — Mon, 11 Apr 2022 06:28:04 +0000

Brief

We'll go through the expected features, lay out some initial requirements, and then figure out what data we need to even begin thinking about implementing our requirements.

What is a requirement?

A requirement is a target feature of the end product. These depend on where the requirements come from, but some examples are:

Technical requirements (ie. to abide by internal standards)
Functional requirements (usually for the end user)

What will be our requirements?

To keep things simple, we'll define our requirements based on what we know, which is the functionality we want to achieve:

View a restaurant's average rating derived from recent reviews over a user-specified time period
Observe trends in restaurant popularity based on the number of reviews it has received within user-specified time intervals
Observe trends in restaurant quality based on the average rating it has received within user-specified time intervals

What data do we need to achieve these requirements?

Let's start from what data we need to be able to achieve these requirements. Once we know what data we need, we can think about where to get it from, how to store it, how to retrieve it, and how we deliver it to our users.

Thankfully, it seems the data we need is just the ratings received for each restaurant per minimum interval.

If we assume the minimum interval is per-week, that data may look something like this (assumed for one restaurant):

timestamp	01-July-2022	08-July-2022	15-July-2022
1-star ratings	4	0	3
2-star ratings	16	12	11
...
5-star ratings	2	5	1

And to achieve those requirements:

Calculate average of data with rating history over last X months
Calculate # of reviews per interval with all rating history
Calculate average rating per interval with all rating history

It should be noted that we only need rating history, and have no need for anything else that's contained from the Google reviews.

What does the picture look like right now?

It seems we'll need:

A data source for Google reviews data
A database to store Google reviews data
A service to pull data from the data source, parse it, and store it in the database
Frontend which will display the data, which is retrieved from...
Backend which hosts an API, providing calculated & formatted data which is read from

Food Project: Starting from the concept

ozmasg — Fri, 08 Apr 2022 08:20:27 +0000

Disclaimer: This is really more of a personal blog. If things go well and I start to share these documents, perhaps I will come back to clean these up.

Introduction

This is my first personal project and dev log!

The goal here is to document my thought processes as I work through the project, not only as a way to reflect on decisions I made later down the line, but also to be able to use some of these documents as a guiding light if needed.

Problem Statement

Ratings on Google Maps - as far as I'm aware - are not guided by any sort of recency.

This means even if a restaurant has decreased drastically in quality recently, it may still appear to be highly rated based on its past ratings.

Project Concept

To provide the functionality of isolating the ratings of restaurants within a certain time period rather than over the course of its entire history.

This will allow users to judge a restaurant based on its 'recent' rating, where the acceptable time period is determined by user discretion.

Targeted Core Features:

View a restaurant's rating based on reviews over a specified time period

Targeted Extra Features:

Observe trends in restaurant popularity based on the number of reviews it has received over a specified time period
Observe trends in restaurant quality based on the increases or decreases of rating over a specified time period

Limitations:

Scope of data will be limited to Singapore.
Initial source of data will be limited to Google reviews, and will not include other review sources.
Only observe up to the N most popular restaurants in specific locales around Singapore.
1. This is mostly due to how Google search and the SERP API works, preventing us from being able to just get data for all restaurants in Singapore.

Concerns:

Scope should be limited so that the initial product can be completed quickly.
Expenditure should be minimized as far as possible, as there is no plan to profit off this project.