DEV Community: Emirhan Akdeniz

Introducing SerpApi's MCP Server

Emirhan Akdeniz — Fri, 05 Dec 2025 17:26:27 +0000

As the AI Agents ecosystem and usage grow, we at SerpApi are looking to support that effort and simplify how AI developers can get fast and reliable access to the latest Google and other Search web data sources. Today, we are releasing an open-source Model Context Protocol (MCP) server that exposes our web search API to AI agents and developers through a familiar and common interface.

This server lets any MCP-compatible AI model client call SerpApi’s search tools without writing any custom code or dealing with complicated SDKs. The agent can ask the server what tools it offers and then invoke them as needed to achieve the desired outcome. SerpApi’s MCP server includes a unified search tool so that LLM (Large Language Models) agents can retrieve real-time search results through simple function calls. The search tool is flexible and supports all schemas and parameters that are supported in our REST API.

The server supports multiple engines (Google, Bing, Yahoo, DuckDuckGo, Yandex, Baidu, YouTube, eBay, Walmart, and more) and specialised queries such as weather or stock lookups, returning clean JSON results optimised for AI application consumption. You may register for SerpApi to claim free credits.
What is the Model Context Protocol (MCP)?

What is the Model Context Protocol (MCP)?

The Model Context Protocol (MCP) is an open protocol created by Antrophic to let AI models operate in richer, more context-aware ways inside real-world development environments and applications. Instead of relying solely on static training data, MCP lets models access live information—such as local files in a file system, project metadata, or connected external systems—all through a unified, standardised protocol. It works over simple transports like stdio, sse, or even USB-C (when implemented over serial-style connections), making it easy to embed into many platforms. To support broad interoperability, MCP also clarifies how transport methods such as server-sent events or process pipes can carry messages between tools and clients. MCP is defined through a clear protocol specification built on JSON-RPC, and many implementations explicitly align with JSON-RPC 2.0, ensuring that tools and clients can communicate reliably without vendor lock-in. As adoption grows, MCP aims to function as a truly standardized protocol that any AI system can use—whether it’s ChatGPT from OpenAI, Claude from Anthropic, Gemini, or any other model—so long as they connect through a compatible endpoint. These foundations also help address security risks by enabling structured access control, controlled sandboxing, and predictable interactions between models and tools.

With MCP, developers can build powerful MCP tools that expose datasets, APIs, services, or an entire workspace to AI assistants and chatbots. This enables models to read docs, run development tools, inspect code, or even debug problems interactively using natural language while still following strict structured messaging. Because everything follows the same protocol, tools become reusable across editors, terminals, and other apps. This growing MCP integration ecosystem means that not only OpenAI models but also third-party AI systems can participate in shared workflows. MCP supports emerging agentic AI patterns, where models autonomously sequence many small tool calls to complete complex use cases, while still respecting boundaries around sensitive data and maintaining scalability as environments grow. With these enhancements, MCP turns AI from a passive text generator into an active participant that can interact with your environment in a safe, structured way—unlocking smarter automation, deeper integrations, and more capable AI-assisted workflows.

Key Features

Multi-Engine Search: A single search tool covers dozens of engines that are providers of external data sources—Google (full and light), Bing, Yahoo, DuckDuckGo, Yandex, Baidu, YouTube, eBay, Walmart, and others.
Structured Results: The server autogenerates structured JSON block templates (external tools such as answer boxes, knowledge graph data, calculations, and more) along with organic, news, image, and shopping results. AI Assistant Agents get exactly what they need without manual parsing.
Raw and Clean Output: Developers can request raw JSON or a clean, normalised version with fields like title, snippet, link, etc.
Robust Reliability: Error handling and retry logic are included by default. Authentication failures, rate limits, and network timeouts return helpful messages that agents can use to respond gracefully.
Open Source: We at SerpApi believe that open source is an integral part of the development of AI Tools. Anyone with some Python knowledge should be able to contribute to Artificial Intelligence tooling and general AI Systems. You may head to our GitHub Repository to access an open-sourced version of the MCP Server.

Getting Started

You can start by using our remotely hosted MCP Server at https://mcp.serpapi.com/SERPAPI_API_KEY/mcp. The MCP server supports requests from any origin.

You can also self-host the MCP server locally to have full control of the deployment. You can start with it by cloning the repository and installing the dependencies:

git clone https://github.com/serpapi/serpapi-mcp.git
cd serpapi-mcp/
uv sync

Set your SerpApi API key in a .env file:

SERPAPI_API_KEY=your_key_here

Then start the server:

uv run src/server.py

Configure your MCP-client to connect to the server. Once connected, the AI-powered agent will detect SerpApi’s search tool and any additional resources the server provides in automation.

You can also use the provided Docker configuration to deploy the service as a container.
Interacting with the server

You can use the MCP inspector to interact and explore the MCP service. You can start by installing the package:

npm install -g  @modelcontextprotocol/inspector

Then start the inspector:

npx @modelcontextprotocol/inspector

The service will become available on localhost:6274. You can try listing the available tools, see their description and parameters.

You can also test the MCP server inside Microsoft’s VS Code IDE. You can add the MCP server config to the .vscode/mcp.json file and test it inside the VS Code Copilot.

{
  "servers": {
    "serpapi-mcp": {
      "type": "http",
      "url": "https://mcp.serpapi.com/<API_KEY>/mcp"
    }
  }
}

Accelerating AI Integration

By adopting the MCP standard, SerpApi makes web search a native capability for AI agents. Instead of bespoke integration code with complicated connectors, agents discover and use SerpApi’s tools automatically. This reduces hallucinations and accelerates the development of agentic applications.

SerpApi releases this server to give AI developers fast, reliable access to live search data. We’re excited to support the growing ecosystem of AI agents and provide the infrastructure that helps them understand the real world with open standards and with minimal permissions needed. The MCP server is open-source, and we invite developers to explore it, contribute, and integrate it into their own AI workflows.

Try out the SerpApi MCP server and see how it fits into your agent stack.

Spiking Neural Networks

Emirhan Akdeniz — Wed, 12 Jun 2024 09:40:00 +0000

In this blog post, we will discuss the differences between spiking neural networks, and non-spiking neural networks, potential use cases of these algorithms, and open source a simple example script to compare a simple SNN model to an ANN model.

At SerpApi Blog, we discuss different topics around web scraping:

What is a spiking neural network?

A spiking neural network (SNN) is a type of artificial neural network that more closely mimics the behavior of biological neurons. Unlike traditional artificial neural networks (ANNs), which use continuous activation functions, SNNs use discrete events called spikes. These spikes represent the times at which neurons fire and enable the network to process information in a way that is more similar to synapses of the brain. This enabling mechanism is what makes SNNs superior to ANNs in inference times for temporal (in-time steps) data.

What is the difference between spiking and non-spiking neural networks?

The primary difference between spiking and non-spiking neural networks lies in how they handle information processing:

Spiking neural networks: Use spikes to represent information, with firing neurons only when their membrane potential reaches a certain threshold. These neuronal modelings are often associated with neuromorphic computing and use learning rules such as spike-timing-dependent plasticity (STDP) instead of backpropagation.
Non-spiking neural networks (e.g. ANNs): Use continuous activation functions like ReLU or sigmoid to process information and typically use backpropagation for learning.

How Does Spiking Neural Network Work?

SNNs work by simulating the behavior of biological neurons. When a neuron’s membrane potential exceeds a certain threshold, it generates a spike that propagates to other spiking neurons. These spikes can modify synaptic weights through learning rules such as spike-timing-dependent plasticity (STDP), enabling the network to learn from temporal patterns in the data.

What are the most effective ways to use neural networks for pattern recognition?

For pattern recognition, deep learning models like convolutional neural network models (CNNs) are highly effective. However, SNNs are gaining attention for their ability to recognize spatiotemporal (belonging to both space and time or to space-time) patterns with high precision and low power consumption.

What could it mean for the future of scraping?

In my humble opinion, SNNs could hold a place in finding patterns within changing and evolving HTML structures. Instead of classifying items and parsing them, SNNs may be useful for identifying where specific parts of the HTML are within the overall body. This could reduce human interaction and pave the way for the future of fully automated parsers with higher precision and lower inference times.

SNN vs ANN Comparison

The following is a simple demonstration script for comparing SNN and ANN models under the same conditions. I have to give a disclaimer that this is for demonstration purposes, and not proving purposes, or definite benchmarking. As I repeat time and again, I am not an expert in machine learning, I am just an enthusiast.

Let's import the libraries. We will be using PyTorch for the framework, sklearn for simple dataset-splitting tasks, and snntorch for creating SNN models in PyTorch:


import numpy as np

import torch

import torch.nn as nn

import torch.optim as optim

import snntorch as snn

from sklearn.model_selection import train_test_split

import time

Let's create a function that simulates motion data (a kind of temporal data):


def generate_motion_data(num_samples, event_length, num_events, noise_level):
    X = []
    y = []
    for _ in range(num_samples):
        motion_indices = np.random.randint(0, event_length, size=num_events)
        event_data = np.zeros(event_length)
        event_data[motion_indices] = 1
        noise = np.random.normal(0, noise_level, size=event_length)
        event_data += noise

        # Introduce variability in the patterns
        if np.random.rand() < 0.5:
            event_data = np.roll(event_data, np.random.randint(1, event_length))

        X.append(event_data)
        y.append(1 if np.sum(event_data) > 0 else 0)
    return np.array(X), np.array(y)

The output of this encoding will be a binary where 1 representing motion and 0 representing no motion. We also introduce a Gaussian noise with standard deviation to make the data more consistent with real-world data. Alongside noise, we introduce some random variability patterns to make the task harder. The model should be able to take into consideration all of these factors and predict the motion output within the series.

Let's create our data:


# Parameters

num_samples = 1000

event_length = 100

num_events = 100

noise_level = 0.1

# Generate data

X, y = generate_motion_data(num_samples, event_length, num_events, noise_level)

# Convert to PyTorch tensors

X = torch.tensor(X, dtype=torch.float32)

y = torch.tensor(y, dtype=torch.float32)

# Split into training and validation sets

X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)

Let's define an SNN model and train it:


# Define SNN model
class SpikingNN(nn.Module):
    def __init__(self, input_dim, hidden_dim, output_dim):
        super(SpikingNN, self).__init__()
        self.fc1 = nn.Linear(input_dim, hidden_dim)
        self.lif1 = snn.Leaky(beta=0.9)
        self.fc2 = nn.Linear(hidden_dim, output_dim)

    def forward(self, x):
        x = self.fc1(x)
        mem1, spk1 = self.lif1(x)
        x = self.fc2(spk1)
        return x

# Model, loss function, and optimizer for SNN
input_dim = event_length
hidden_dim = 64
output_dim = 1  # Binary classification

snn_model = SpikingNN(input_dim, hidden_dim, output_dim)
criterion = nn.BCEWithLogitsLoss()  # Binary cross-entropy loss
optimizer = optim.Adam(snn_model.parameters(), lr=0.001)

# Training loop for SNN
num_epochs = 100
snn_training_start = time.time()

for epoch in range(num_epochs):
    snn_model.train()
    optimizer.zero_grad()
    outputs = snn_model(X_train)
    loss = criterion(outputs.squeeze(), y_train)
    loss.backward()
    optimizer.step()

    # Calculate training loss
    train_loss = loss.item()

    # Validation
    snn_model.eval()
    with torch.no_grad():
        val_outputs = snn_model(X_val)
        val_loss = criterion(val_outputs.squeeze(), y_val)
        val_loss = val_loss.item()

    print(f'SNN Epoch {epoch+1}/{num_epochs}, Loss: {train_loss:.4f}, Validation Loss: {val_loss:.4f}')

snn_training_time = time.time() - snn_training_start
print(f"SNN Training Time: {snn_training_time:.4f} seconds")

Let's create an ANN model and train it for comparison:


# Define ANN model

class ANN(nn.Module):
    def __init__(self, input_dim, hidden_dim, output_dim):
        super(ANN, self).__init__()
        self.fc1 = nn.Linear(input_dim, hidden_dim)
        self.relu = nn.ReLU()
        self.fc2 = nn.Linear(hidden_dim, output_dim)

    def forward(self, x):
        x = self.fc1(x)
        x = self.relu(x)
        x = self.fc2(x)
        return x

# Model, loss function, and optimizer for ANN
ann_model = ANN(input_dim, hidden_dim, output_dim)
criterion = nn.BCEWithLogitsLoss()  # Binary cross-entropy loss
optimizer = optim.Adam(ann_model.parameters(), lr=0.001)

# Training loop for ANN
ann_training_start = time.time()

for epoch in range(num_epochs):
    ann_model.train()
    optimizer.zero_grad()
    outputs = ann_model(X_train)
    loss = criterion(outputs.squeeze(), y_train)
    loss.backward()
    optimizer.step()

    # Calculate training loss
    train_loss = loss.item()

    # Validation
    ann_model.eval()
    with torch.no_grad():
        val_outputs = ann_model(X_val)
        val_loss = criterion(val_outputs.squeeze(), y_val)
        val_loss = val_loss.item()

    print(f'ANN Epoch {epoch+1}/{num_epochs}, Loss: {train_loss:.4f}, Validation Loss: {val_loss:.4f}')

ann_training_time = time.time() - ann_training_start
print(f"ANN Training Time: {ann_training_time:.4f} seconds")

Let's define a function to run predictions, calculate the inference time, and compare the two models:


# Function to predict and measure inference time

def predict_and_measure_time(model, new_data):
    start_time = time.time()
    model.eval()
    with torch.no_grad():
        new_data_tensor = torch.tensor(new_data, dtype=torch.float32)
        outputs = model(new_data_tensor)
    inference_time = time.time() - start_time
    return outputs, inference_time

# Generate new test data
X_test, y_test = generate_motion_data(5, event_length, num_events, noise_level)

# Predictions with SNN
snn_outputs, snn_inference_time = predict_and_measure_time(snn_model, X_test)
snn_predictions = torch.round(torch.sigmoid(snn_outputs)).squeeze().numpy()
print("SNN Predictions:", snn_predictions)
print(f"SNN Inference Time: {snn_inference_time:.4f} seconds")

# Predictions with ANN
ann_outputs, ann_inference_time = predict_and_measure_time(ann_model, X_test)
ann_predictions = torch.round(torch.sigmoid(ann_outputs)).squeeze().numpy()
print("ANN Predictions:", ann_predictions)
print(f"ANN Inference Time: {ann_inference_time:.4f} seconds")

# Comparison Summary
print(f"Comparison Summary:")
print(f"SNN Training Time: {snn_training_time:.4f} seconds")
print(f"ANN Training Time: {ann_training_time:.4f} seconds")
print(f"SNN Inference Time: {snn_inference_time:.4f} seconds")
print(f"ANN Inference Time: {ann_inference_time:.4f} seconds")

# Final validation accuracies (from the last epoch)
snn_model.eval()
with torch.no_grad():
    snn_val_outputs = snn_model(X_val)
    snn_val_accuracy = ((torch.sigmoid(snn_val_outputs) > 0.5).squeeze().float() == y_val).float().mean().item()

ann_model.eval()
with torch.no_grad():
    ann_val_outputs = ann_model(X_val)
    ann_val_accuracy = ((torch.sigmoid(ann_val_outputs) > 0.5).squeeze().float() == y_val).float().mean().item()

print(f"Final SNN Validation Accuracy: {snn_val_accuracy:.4f}")
print(f"Final ANN Validation Accuracy: {ann_val_accuracy:.4f}")

Following numbers are not definitive as the task is simple. But it should give a clear idea as to how SNN can beat ANN:

Comparison Summary:

SNN Training Time: 0.6785 seconds

ANN Training Time: 0.3952 seconds

SNN Inference Time: 0.0007 seconds

ANN Inference Time: 0.0017 seconds

Final SNN Validation Accuracy: 1.0000

Final ANN Validation Accuracy: 1.0000

It took more time to train SNN due to framework and the architecture not in favor of training the model. However in the future we may see different approaches to optimize training time of SNN models. For inference time, SNN was faster than ANN model. This is where the energy efficiency is taking place. Because SNN is much easier to execute under same accuracy, it is consuming less CPU power.

Are spiking neural networks the future?

Spiking neural networks (SNNs) are increasingly being considered as a promising frontier in the future of artificial intelligence, particularly due to their closer resemblance to the neural computation seen in biological brains. Leveraging principles from neuroscience, SNNs process information through spikes, or action potentials, which offers a unique form of temporal coding and rate coding.

In contrast to traditional deep neural networks, which rely on gradient descent and back propagation, SNNs utilize spike-based learning algorithms and synaptic plasticity, making them more efficient in certain types of neural computation. The initialization of these spiking networks involves setting up multi-layer architectures capable of handling the temporal dynamics and correlations within spike trains.

One of the significant advantages of SNNs is their potential for lower energy consumption, especially when implemented on neuromorphic hardware. These processors mimic the brain’s architecture and function, enabling real-time processing with minimal latency. This is particularly beneficial in applications like robotics, computer vision, and large-scale network models, where real-time and efficient computations are crucial.

SNNs also offer improved interpretability compared to traditional deep-learning models. Each single neuron in an SNN can be examined for its specific role in the network, which aids in understanding how neural computations propagate through the system. Feedforward and recurrent neural networks can both be implemented within the SNN framework, providing versatility in handling different types of data and tasks.

Despite these advantages, SNNs face challenges in terms of learning algorithms and network models. The nonlinear nature of spike-based communication and the need for precise temporal synchronization complicate the development of effective supervised learning techniques. Additionally, the number of spikes and their timing (latency) play a crucial role in the plausibility and performance of SNNs.

Recent advances in state-of-the-art neuromorphic processors and spiking neuron models show promise for overcoming these hurdles. As research in neuroscience and artificial intelligence continues to converge, SNNs may become more viable for practical applications, enhancing the capabilities of both AI and computational neuroscience.

In summary, spiking neural networks hold significant potential for the future of AI, particularly in areas requiring efficient, real-time processing with low energy consumption. Their biologically inspired approach offers a plausible and powerful alternative to traditional deep learning, potentially revolutionizing fields such as robotics, computer vision, and beyond.

Conclusion

I’m grateful to the reader for their attention. I hope this blog post will help people understand the potential of spiking neural networks and their use cases. The reader may find the full comparison script below:

Full script:


!pip install torch snntorch scikit-learn

import numpy as np
import torch
import torch.nn as nn
import torch.optim as optim
import snntorch as snn
from sklearn.model_selection import train_test_split  
import time

# Generate synthetic event-based motion data
def generate_motion_data(num_samples, event_length, num_events, noise_level):
    X = []
    y = []
    for _ in range(num_samples):
        motion_indices = np.random.randint(0, event_length, size=num_events)
        event_data = np.zeros(event_length)
        event_data[motion_indices] = 1
        noise = np.random.normal(0, noise_level, size=event_length)
        event_data += noise

        # Introduce variability in the patterns
        if np.random.rand() < 0.5:
            event_data = np.roll(event_data, np.random.randint(1, event_length))

        X.append(event_data)
        y.append(1 if np.sum(event_data) > 0 else 0)
    return np.array(X), np.array(y)

# Parameters
num_samples = 1000
event_length = 100
num_events = 100
noise_level = 0.1

# Generate data
X, y = generate_motion_data(num_samples, event_length, num_events, noise_level)

# Convert to PyTorch tensors
X = torch.tensor(X, dtype=torch.float32)
y = torch.tensor(y, dtype=torch.float32)

# Split into training and validation sets
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)

# Define SNN model
class SpikingNN(nn.Module):
    def __init__(self, input_dim, hidden_dim, output_dim):
        super(SpikingNN, self).__init__()
        self.fc1 = nn.Linear(input_dim, hidden_dim)
        self.lif1 = snn.Leaky(beta=0.9)
        self.fc2 = nn.Linear(hidden_dim, output_dim)

    def forward(self, x):
        x = self.fc1(x)
        mem1, spk1 = self.lif1(x)
        x = self.fc2(spk1)
        return x

# Model, loss function, and optimizer for SNN
input_dim = event_length
hidden_dim = 64
output_dim = 1  # Binary classification

snn_model = SpikingNN(input_dim, hidden_dim, output_dim)
criterion = nn.BCEWithLogitsLoss()  # Binary cross-entropy loss
optimizer = optim.Adam(snn_model.parameters(), lr=0.001)

# Training loop for SNN
num_epochs = 100
snn_training_start = time.time()

for epoch in range(num_epochs):
    snn_model.train()
    optimizer.zero_grad()
    outputs = snn_model(X_train)
    loss = criterion(outputs.squeeze(), y_train)
    loss.backward()
    optimizer.step()

    # Calculate training loss
    train_loss = loss.item()

    # Validation
    snn_model.eval()
    with torch.no_grad():
        val_outputs = snn_model(X_val)
        val_loss = criterion(val_outputs.squeeze(), y_val)
        val_loss = val_loss.item()

    print(f'SNN Epoch {epoch+1}/{num_epochs}, Loss: {train_loss:.4f}, Validation Loss: {val_loss:.4f}')

snn_training_time = time.time() - snn_training_start
print(f"SNN Training Time: {snn_training_time:.4f} seconds")

# Define ANN model
class ANN(nn.Module):
    def __init__(self, input_dim, hidden_dim, output_dim):
        super(ANN, self).__init__()
        self.fc1 = nn.Linear(input_dim, hidden_dim)
        self.relu = nn.ReLU()
        self.fc2 = nn.Linear(hidden_dim, output_dim)

    def forward(self, x):
        x = self.fc1(x)
        x = self.relu(x)
        x = self.fc2(x)
        return x

# Model, loss function, and optimizer for ANN
ann_model = ANN(input_dim, hidden_dim, output_dim)
criterion = nn.BCEWithLogitsLoss()  # Binary cross-entropy loss
optimizer = optim.Adam(ann_model.parameters(), lr=0.001)

# Training loop for ANN
ann_training_start = time.time()

for epoch in range(num_epochs):
    ann_model.train()
    optimizer.zero_grad()
    outputs = ann_model(X_train)
    loss = criterion(outputs.squeeze(), y_train)
    loss.backward()
    optimizer.step()

    # Calculate training loss
    train_loss = loss.item()

    # Validation
    ann_model.eval()
    with torch.no_grad():
        val_outputs = ann_model(X_val)
        val_loss = criterion(val_outputs.squeeze(), y_val)
        val_loss = val_loss.item()

    print(f'ANN Epoch {epoch+1}/{num_epochs}, Loss: {train_loss:.4f}, Validation Loss: {val_loss:.4f}')

ann_training_time = time.time() - ann_training_start
print(f"ANN Training Time: {ann_training_time:.4f} seconds")

# Function to predict and measure inference time
def predict_and_measure_time(model, new_data):
    start_time = time.time()
    model.eval()
    with torch.no_grad():
        new_data_tensor = torch.tensor(new_data, dtype=torch.float32)
        outputs = model(new_data_tensor)
    inference_time = time.time() - start_time
    return outputs, inference_time

# Generate new test data
X_test, y_test = generate_motion_data(5, event_length, num_events, noise_level)

# Predictions with SNN
snn_outputs, snn_inference_time = predict_and_measure_time(snn_model, X_test)
snn_predictions = torch.round(torch.sigmoid(snn_outputs)).squeeze().numpy()
print("SNN Predictions:", snn_predictions)
print(f"SNN Inference Time: {snn_inference_time:.4f} seconds")

# Predictions with ANN
ann_outputs, ann_inference_time = predict_and_measure_time(ann_model, X_test)
ann_predictions = torch.round(torch.sigmoid(ann_outputs)).squeeze().numpy()
print("ANN Predictions:", ann_predictions)
print(f"ANN Inference Time: {ann_inference_time:.4f} seconds")

# Comparison Summary
print(f"Comparison Summary:")
print(f"SNN Training Time: {snn_training_time:.4f} seconds")
print(f"ANN Training Time: {ann_training_time:.4f} seconds")
print(f"SNN Inference Time: {snn_inference_time:.4f} seconds")
print(f"ANN Inference Time: {ann_inference_time:.4f} seconds")

# Final validation accuracies (from the last epoch)
snn_model.eval()
with torch.no_grad():
    snn_val_outputs = snn_model(X_val)
    snn_val_accuracy = ((torch.sigmoid(snn_val_outputs) > 0.5).squeeze().float() == y_val).float().mean().item()

ann_model.eval()
with torch.no_grad():
    ann_val_outputs = ann_model(X_val)
    ann_val_accuracy = ((torch.sigmoid(ann_val_outputs) > 0.5).squeeze().float() == y_val).float().mean().item()

print(f"Final SNN Validation Accuracy: {snn_val_accuracy:.4f}")
print(f"Final ANN Validation Accuracy: {ann_val_accuracy:.4f}")

Google Search Parameters (2024 Guide)

Emirhan Akdeniz — Tue, 23 Apr 2024 02:25:21 +0000

The Google Search URL parameters are important to understand whether you are maximizing the conversion rate in your ad groups and optimizing your cost per click(CPC) rates in Google Analytics for your ad campaigns, improving your SEO(Search Engine Optimization) metrics for your e-commerce business, or collecting data for your social media project. Using custom parameters for your search will affect the Search Engine Result Page (SERP) data significantly in your favor. SerpApi unlocks the power of SERP data to you with minimal effort, the fastest response times, and in the most complete form.

In this blog post, we will explore some of the Google search parameters with their corresponding attributions in SerpApi Documentation. You can head to SerpApi Playground to test all of the information you will read below.

What are Google Search parameters?

Google Search parameters are like special codes you can add to the web address (URL) when you search on Google. They help you control and customize your search results. For example, you can use them to specify the language or location for your search.

Here's how they work: When you type a search query into Google, it creates a URL with your search terms. You can then add parameters to this URL using an ampersand (&) followed by the parameter name and value. These parameters, often called "params" for short, tell Google how you want your search results to be customized.

For instance, if you want to search for "cats" in Spanish, you could add "hl=es" to the end of the URL. This tells Google to show results in Spanish. Once you've added all the parameters you want, you have your final URL ready for tailoring your tracking template. When you open this URL in your web browser, Google will show you search results based on the parameters you've specified.

The q parameter

It represents the search query submitted to Google. For example, in "q=Coffee", "Cofee" is the query you're searching for. It can also be used with special commands for filtering specific file types, filtering keywords in URLs and bodies, etc. giving you more exact matches. Some of these advanced search operators include filetype, inurl, intitle, intext, inanchor, allinurl, allinanchor, allintitle, allintext. You can check out Ultimate Guide to Google Search Operators for detailed, and up-to-date information on how to use Google search operators.

Reference in SerpApi Documentation

The ‘oq‘ parameter of Google represents the query typed into the Google search box.

The hl parameter

It stands for a host language, it determines the language used for the search. There are different versions of the ‘hl‘ parameters you can pass to a Google Search. The two-letter version is for all languages supported by Google. For example "hl=en" searches in the English language. There are also two letters followed by a hyphen and two letters or three-digit versions representing dialects. Examples of this kind are "hl=en-gb" representing British English, or "hl=es-419" representing Latin American Spanish.

Reference in SerpApi Documentation

The gl parameter

It represents geolocation, which can be used to personalize search results based on the region. It is a two-letter country code. For example "gl=us" will serve results for the United States.

Reference in SerpApi Documentation

The lr parameter

It stands for language restriction, it's used to limit your search results to webpages in a specific language or languages. It is a two-letter language code proceeded by the word lang and an underscore. Multiple languages can be passed using a bar symbol. For example "lr=lang_en|lang_fr" will restrict the results to websites with only English and French languages.

Reference in SerpApi Documentation

The cr parameter

It represents country restriction, it refines results based on preferred region. It is a two-letter uppercase country code proceeded by the word country. Multiple countries can be passed using a bar symbol. For example "cr=countryUS|countryFR" will restrict the results pages to the United States and France.

Reference in SerpApi Documentation

The tbs parameter

It stands for to be searched, it can be used for various purposes, for example, to filter results by a specific date or a date range (e.g., past week, past month). For example, "tbs=qdr:h" will serve only the results from the last hour.

Reference in SerpApi Documentation

The source parameter

It indicates the source of your search, such as the Google homepage or a referral from another website. For example "source=hp" means that the page was accessed through the homepage.

The ei parameter

It possibly stands for engine identifier, a unique identifier assigned to each search, likely used for internal tracking by Google. Its decoded form consists of four numbers. The first two are a Unix timestamp and microseconds of that timestamp respectively. The last two are undisclosed and unknown.

The ved parameter

It is used for internal tracking of links and elements. Apart from the link itself, the element index and type can be gathered by decoding the ved value.

Open Source Repository for Decoding the ved values

The gs_lp parameter

Not much is disclosed about this parameter. It contains the search query and search client.

The iflsig parameter

This parameter isn't disclosed. It is speculated to be about internal redirects. Other assumptions include relations to parsing.

The ibp parameter

It is responsible for rendering elements differently than their usual format. For example, when used with the ludocid parameter (unique id for a Google Business listing) in a mobile view, "ibp=gwp;0,7&ludocid=1663467585083384531", the 'ibp' parameter can give full page views of the listing.

Reference in SerpApi Documentation

The kgmid parameter

The parameter is responsible for serving elements saved by Google including knowledge graph. It can override the ‘q‘ (query) parameter. For example "kgmid=/m/07gyp7" will get results for McDonald's with a knowledge graph regardless of the query parameter. The parameter is likely acting as a cached version of the Google advanced search parameters.

Reference in SerpApi Documentation

The si parameter

It is used to serve knowledge graph tabs alongside other tasks. For example, in the example below, the 'si' parameter is responsible for serving the Google search results with the 'Music Groups' tab of the knowledge graph for Jimi Hendrix.

Reference in SerpApi Documentation

The sxsrf parameter

It is possibly an internal token for security. The 'sxsrf' consists of an encoded token probably for security purposes, and a unix timestamp.

The uule parameter

It is used to specify a location for a search. The 'uule' parameter has two types. The first type is an encoded form of the canonical name of a place in Google Ads API Geo Targets. The second type is an encoded form of the GPS coordinates alongside other data.

Reference in SerpApi Documentation

A Blog Post about Different Types of UULE

The sclient parameter

It stands for search client. For example "sclient=gws-wiz" means Google Web Search Wizard is used for the search.

The tbm parameter

It stands for to be matched. This parameter specifies the type of content that the user is searching for, allowing Google to tailor the search results accordingly. For instance, "tbm=isch" indicates that the search is intended for image search, while "tbm=nws" directs Google to prioritize results from Google News. By utilizing this parameter, users can fine-tune their searches to precisely start finding pages, sites, or specific content types using Google's sophisticated algorithm. Additionally, users can specify whether they want to search within meta information, anchor text, or other elements through the appropriate parameters. This level of customization empowers users to efficiently navigate the vast expanse of information accessible through Google's search bar.

Reference in SerpApi Documentation

Conclusion

I would like to thank the reader for their attention. I hope this writing could shed some light on how Google search parameters work. For more up-to-date information about webscraping, SERP data, and more related topics you may visit the SerpApi Blog.

Comparing Types of Databases: A Real-World Benchmark Analysis

Emirhan Akdeniz — Thu, 09 Nov 2023 19:48:01 +0000

Intro

Choosing between common database solutions can be tough. In this writing, I will be sharing how to choose which database to use for a specific real-world case, give introductory information on how different databases store data, open source a benchmarking script, discuss important factors to take in decision-making, and share benchmark reports in charts.

The Problem

SerpApi is an API for scraping Google and other search engines with fast, easy, and complete solutions. Our team is tackling a challenge within our operational database where we house a Locations collection featuring predefined data structures. We aim to offer these locations to clients reliably and allow them to utilize these in their searches with SerpApi's Google Search API, demonstrating the interconnected functionality of various types of databases. You may register to claim free credits to try out our products

The Present Solution and Its Shortcomings

We're operating a robust MongoDB server—an example of one of the types of NoSQL database, more specifically a document-oriented database—which stores Locations collection objects. This enables us to efficiently store data, manage complex data structures, and enhance the Location model's scalability. Its adeptness at performing complete searches, fuzzy matches, and geospatial evaluations between nodes stand out. The Location collection is one of the primary keys of the search object and is crucial for a big portion of our searches.

However, this approach has a scalability issue concerning portability. The firm coupling with a centralized database alongside other collections with parent-child relations limits our agility in customizing the schema and affects the playroom for diverse operational settings. Altering a small collection by adding or deleting new entries in such a hierarchical database carries inherent risks due to its monolithic nature.

Additionally, constants loaded during server initialization, which could reside in a cloud database with superior indexing, instead consume significant amounts of memory. For this analysis, we'll focus on the Location model within this DBMS.

Proposed Solutions

I have devised three prototypes to surmount the portability hurdle. These databases store and serve in various ways. I graph database benchmarks in order to provide different types of data for each prototype.

Using a Static Sqlite3 File

Deploying a static Sqlite3 file, akin to a personal database on each server, offers remarkable portability, allows for server-level data management, and provides the advantages of a relational database management system (RDBMS). This is reflective of the distributed database paradigm, where local updates can be methodically managed and validated through unit testing. Integration with our MongoDB documents is seamless, thanks to the BSON-serialization of Ruby hashes. SQLite is a popular database software choice for utilizing the advantages of relational models.

Streaming from a Static JSON File

Streaming from a static JSON file, which is a form of semi-structured data, is another solution, maintaining control at the server level for safe, localized development. Even updating key-value structures is relatively easy. This represents a move towards a microservices architecture, often hosted on cloud computing platforms. This acts as a kind of database for information management.

External Standalone MongoDB Server

A standalone MongoDB server represents a non-relational database alternative for storing child records of the search object, more specifically the Location collection. This option demands a deep understanding of database technology, offering real-time adjustments through a secured public IP, maintaining data security, and harnessing MongoDB 2dsphere for indexing data models for spherical approximations. It is a popular database to store information with wide community support.

Performance Benchmarks

The benchmarks involve various queries and GPS coordinates, providing an average of real-time performance over multiple iterations. These parameters act as a data set to compare different performance metrics of the prototypes.

I have applied the benchmark on JSON Streaming prototype only up to 5 concurrency. It wasn't efficient enough to benchmark all the way through in my local environment and was facing difficult bottlenecks.

There are important details to consider in this benchmark. The current solution we have with a Monolithic Mongodb database, and the Standalone Mongodb server is executing some commands externally on a remote server. Therefore the data is dependent on the connection time latency. Unlike these two other solutions, the Sqlite3 prototype, and JSON Streaming Prototype are contained within an individual server and are only affected by system load and throughput of hardware. his benchmark is storing different data types when it comes to time. For the purposes of simplifying things, we will be using average real-time taken as our indicator.

You may find the complete benchmark below:

require 'benchmark'
require 'thread'
require 'csv'

SAMPLE_QUERIES = ['USA', 'FalseInput', 'Canada', 'Austin', 'Ankara']
SAMPLE_COORDINATES = [[37.0902, -95.7129], [0, 0], [56.1304, -106.3468], [30.2672, -97.7431], [39.9334, 32.8597]]
TIMES = 5
CONCURRENCY_LEVELS = (0..50).to_a

def perform_searches(model, method, query, latitude = nil, longitude = nil)
  if method == :find_nearest_location
    model.send(method, latitude, longitude)
  else
    model.send(method, query)
  end
end

def concurrent_benchmark(model, method, concurrency_level, query, latitude = nil, longitude = nil)
  threads = []
  concurrency_level.times do
    threads << Thread.new do
      perform_searches(model, method, query, latitude, longitude)
    end
  end
  threads.each(&:join)
end

def benchmark_methods(model)
  results = {}

  CONCURRENCY_LEVELS.each do |concurrency|
    results[concurrency] = {}
    SAMPLE_QUERIES.each do |query|
      results[concurrency][query] = { search: 0.0, search_for_completion: 0.0, find_nearest_location: 0.0 }

      times = Array.new(TIMES) do
        Benchmark.measure do
          concurrent_benchmark(model, :search, concurrency, query)
        end
      end
      results[concurrency][query][:search] = {
        real: times.sum(&:real) / TIMES,
        cpu: times.sum(&:total) / TIMES,
        system: times.sum(&:stime) / TIMES,
        user: times.sum(&:utime) / TIMES
      }

      times = Array.new(TIMES) do
        Benchmark.measure do
          concurrent_benchmark(model, :search_for_completion, concurrency, query)
        end
      end
      results[concurrency][query][:search_for_completion] = {
        real: times.sum(&:real) / TIMES,
        cpu: times.sum(&:total) / TIMES,
        system: times.sum(&:stime) / TIMES,
        user: times.sum(&:utime) / TIMES
      }

      coord_index = SAMPLE_QUERIES.index(query)
      latitude, longitude = SAMPLE_COORDINATES[coord_index]
      times = Array.new(TIMES) do
        Benchmark.measure do
          concurrent_benchmark(model, :find_nearest_location, concurrency, query, latitude, longitude)
        end
      end
      results[concurrency][query][:find_nearest_location] = {
        real: times.sum(&:real) / TIMES,
        cpu: times.sum(&:total) / TIMES,
        system: times.sum(&:stime) / TIMES,
        user: times.sum(&:utime) / TIMES
      }
    end
  end

  results
end

def print_results(results)
  results.each do |concurrency, concurrency_results|
    puts "Concurrency Level: #{concurrency}"
    concurrency_results.each do |query, methods|
      puts "Query: #{query}"
      methods.each do |method, times|
        puts "  #{method}:"
        puts "    Average real time over #{TIMES} runs: #{times[:real].round(5)}s"
        puts "    Average CPU time over #{TIMES} runs: #{times[:cpu].round(5)}s"
        puts "    Average system time over #{TIMES} runs: #{times[:system].round(5)}s"
        puts "    Average user time over #{TIMES} runs: #{times[:user].round(5)}s"
      end
    end
    puts "-" * 30
  end
end
results = benchmark_methods(Location)
print_results(results)

def export_to_csv(results, filename="benchmark_results.csv")
  CSV.open(filename, "wb") do |csv|
    csv << ["Concurrency Level", "Query", "Method", "Average Real Time", "Average CPU Time", "Average System Time", "Average User Time"]
    results.each do |concurrency, concurrency_results|
      concurrency_results.each do |query, methods|
        methods.each do |method, times|
          csv << [concurrency, query, method, times[:real], times[:cpu], times[:system], times[:user]]
        end
      end
    end
  end
end

export_to_csv(results, filename="benchmark_results.csv")

Overall Benchmark Results

The overarching benchmarks, visualized in the graph above reveal that the JSON Streaming model is non-viable for our high-performance standards, primarily due to the demanding optimization it requires for handling big data. It is also easier to read minute details for data points within the charts without its data. It is not a fit prototype to be used in industry-standard methodologies.

Benchmark Results for Proximity Estimation

As depicted above, our prototypes, including the current SQL database solution and the standalone MongoDB server, demonstrate diverse performance levels in geospatial proximity estimation. The MongoDB model showcases the durability of its geospatial indexing capabilities.

While MongoDB has 2dsphere, for Sqlite3 I have used the simplification below:

 where(Sequel.~(gps_lat: nil)).where(Sequel.~(gps_lng: nil)).order(Sequel.lit("ABS(gps_lat - ?) + ABS(gps_lng - ?)", latitude, longitude)).first

The Haversine formula is much more effective in giving more accurate approximations for dense maps. But for the purposes of this task, using absolute difference as an indicator is enough. However, MongoDB is excelling at speed even though it is using the Haversine formula.

Also, the current monolithic solution and the standalone prototype give roughly the same numbers under different concurrencies. Let's point out the fact that we can improve the standalone server's hardware easily.

Benchmark Results for Fuzzy Search Capability

The above chart shows the results for making a fuzzy search, a feature critical in web applications, to find a result with the common words the user is providing. Although the difference is not that big, the Sqlite3 prototype is clearly ahead in terms of speed. Also, the margin between the standalone prototype and the current monolithic server is growing. There is one point that needs attention though. The Sqlite3 prototype uses a LIKE clause in SQL to make a fuzzy search. An equivalent or similar solution for enhancing MongoDB's similar functionality with SQL-like structured query language constructs. would result in faster search completions for a standalone MongoDB prototype.

Direct Search Method Benchmark Results

This is the method responsible for the overwhelming majority of the searches SerpApi is processing. So any kind of sign in here is crucial to the entire comparison. Although the Sqlite3 prototype is superior to the Standalone MongoDB prototype in lower concurrencies, the latter is keeping consistency over high concurrency and winning against the former which is pivotal for large amounts of data and complex data management scenarios. The difference between the current Monolithic MongoDB Solution, and the Standalone MongoDB Prototype is not that big in terms of milliseconds. The standalone database can also be configured to have higher resources to compete.

Conclusion

Discarding the JSON Streaming model due to its concurrency shortcomings in big data storage, we are left to decide between the Sqlite3 and MongoDB standalone prototypes. We are left with several factors between different types of databases, and the architecture we use them in.

Any kind of lack of efficiency in the Sqlite3 Prototype should be dealt with by upgrading the resources of each individual server in the cluster. This is definitely one of the bad signs for a high-load task, threatening maintenance of atomicity, and data integrity. The expectation to dig deep into SQL is also a downside although it is not as tough as learning a new programming language. In addition to these, the lack of efficiency in the search method alone is a good enough reason not to go forward with the database model. However, any way of delivering the efficiency of the search for completion method in the Sqlite3 prototype to a Standalone MongoDB server would be beneficial. Anything that works with the same logic of the LIKE clause of SQL in MongoDB could benefit SerpApi's Playground for the end user. Also, the standalone MongoDB server should be optimized with additional resources to get as close to the current solution as possible.

In conclusion, the standalone MongoDB database structure should be used for storing secondary constants and collections for our use case. I would like to thank the reader for their attention. I hope this writing could shed some light on how to benchmark different prototype solutions for database examples to tackle one problem.

Originally published at https://serpapi.com on November 8, 2023.

Cold Email Marketing with Open-Source Email Extractor

Emirhan Akdeniz — Fri, 07 Jul 2023 19:38:51 +0000

In this blog post, we will deep-dive into the fascinating world of Cold Email Marketing, exploring its multifaceted strategies, unraveling the do's and don'ts, and introducing you to an exceptional Open-Source tool that simplifies the process of collecting mass-targeted emails.

Related Sources

Clauneck: An Open-Source Command Line Tool to Scrape and Extract Emails and Social Network Handles with Targeted SERP Data
SerpApi's Google Search API: An Easy-to-use API that provides Real-Time Google Search Results in JSON format

What Cold Email Means?

Cold emailing, a pivotal strategy in digital marketing, stands as the lifeline of many B2B sales campaigns. It's a technique that involves sending unsolicited but well-crafted emails to potential clients or potential customers, primarily for sales, promotional, or networking purposes. Unlike its infamous cousin spam, a good cold email isn't an unsolicited barrage but a carefully personalized note sent to specific individuals or companies. The targets are meticulously chosen based on certain criteria, such as their industry, job title, or interest level in your product/service.

As with crafting a compelling sales pitch, building an effective cold email requires meticulous planning. From devising engaging subject lines, and curating appealing email body content, to ensuring that your email successfully avoids the recipient's spam folder, each step is essential. The best cold email practices are all about addressing a pain point, providing a solution, and adding value to the recipient's life or business, rather than being a pesky interruption.

Is Cold Emailing Illegal?

One of the biggest myths surrounding cold email outreach is the idea that it's illegal. However, in reality, the legality of cold emails hinges on certain key factors and how you execute your outreach email strategy. In most jurisdictions, it's entirely legal to send cold emails for B2B purposes, as long as the email includes an accurate email signature containing contact information, comprising the sender's physical address and phone number. Moreover, the email must offer a clear and conspicuous unsubscribe option, enabling recipients to opt out of future communications with just a click. It is also recommended to use an email checker for verifying valid email addresses to reduce the bounce rate of emails in order to prevent issues with your ESP.

However, things may get a little murky when it comes to scraping emails from social media accounts like LinkedIn or websites. Although it's generally considered legal if these addresses are publicly listed and meant for communication purposes, it's crucial to tread carefully. Every email marketer must vigilantly avoid violating privacy laws or regulations such as the CAN-SPAM Act in the U.S. or GDPR in Europe. Thankfully, CRM systems offer an integrated solution for managing this legal maze, helping to ensure compliance and improve email deliverability.

Does Cold Email Actually Work?

The efficacy of cold outreach campaigns is a hot topic in digital marketing circles, and the simple answer is yes, it can work phenomenally, but with a few caveats with the right kind of automation. The success of a cold email campaign depends heavily on multiple factors. These factors include the quality of your bulk email list, the recipient’s name or company’s name in addressing, a well-crafted opening line giving an intro to your service, the relevance of your message to the recipient, your cold email subject lines, a call to action for a better conversion rate, and the robustness of your follow-up email strategy. Metrics for a cold email strategy such as open rate, reply rate, and conversion rates stand as the cardinal indicators to measure the effectiveness of a cold email campaign.

When implemented right, cold emails can transform into a potent tool for lead generation, enabling salespeople to directly reach the decision-makers, bypassing the usual gatekeepers. To optimize your approach, consider A/B testing different cold email templates, subject lines, and CTAs. For instance, integrating a case study in the email body demonstrating how your product/service resolved a similar problem for another company can bolster your value proposition and amplify your response rate. Also, later in this blog post, we will talk about open-source email extractor software that can increase the precision of your email list to reach a more nuanced audience.

What is Email Scraping?

Email scraping or email extraction is a high-tech process of deploying an email scraper tool or bot to find email addresses and mine them from various online sources. These sources can span from social media platforms, LinkedIn profiles, and websites, to even text files. An efficient email scraper dives deep into HTML and JavaScript codes, scans web pages, and leverages APIs to collate email addresses.

The harvested data is usually stored in a structured format like a CSV or Excel file for effortless access and analysis. However, it's vital to note that while scraping public email addresses for communication or lead-generation purposes can be legal, there are ethical considerations and regulations to be mindful of. Upholding respect for user privacy and adherence to data protection laws should always be paramount when engaging in email scraping activities.

Is it Legal to Scrape Emails?

The legality of email scraping may seem dubious, but it primarily depends on the intent behind the act and the method of extraction. If the email addresses are publicly accessible and are collected for legitimate purposes like business prospecting, then it's typically considered legal. Email addresses displayed on websites or public social media accounts are often meant for communication purposes. Hence, employing an email scraper tool to compile these addresses aligns with legal practices. However, adhering to the respective data protection regulations of each country and obtaining consent where necessary is crucial to avoid any potential legal complications.

Why targeting is important in Scraping Emails?

The importance of targeting in email scraping lies in its potency to offer both efficiency and relevance. Efficiency stems from the need to minimize the number of irrelevant or uninterested recipients who are unlikely to engage with your content. By concentrating on a specific target audience or demographic, the volume of emails you scrape may decrease, but the quality of those emails in terms of potential engagement and response enhances dramatically.

Moreover, targeting aids in reducing the likelihood of being classified as spammers. When emails are scraped indiscriminately, there's a higher chance that your sales emails will be dispatched to individuals who have no interest in your content or offerings. These individuals are more likely to mark your emails as spam, activating spam filters which can negatively impact your email deliverability rates and sender reputation. By ensuring that the emails you scrape belong to people likely interested in your offerings, you reduce the risk of your emails being labeled as spam, thereby securing your sender's reputation. This leads to higher open rates, better engagement, and overall improved effectiveness of your email marketing efforts.

How can SerpApi's Google Search API help with targeting?

Although SerpApi's Google Search API is well known for SEO purposes, it can be an influential tool that enhances the process of email scraping by delivering targeted and refined search results. This API allows you to programmatically interact with the Google Search engine, extracting invaluable data from the search results page that can be used to inform and refine your business email scraping strategy.

One way SerpApi's Google Search API can aid with targeted email scraping is by creating a more refined list of websites or companies that align with your business or campaign objectives. For example, if you're running a cold email campaign targeting tech startups, you can use the Google Search API to generate a list of such companies. This list can then be input into your email scraper tool, which will extract the email addresses from these specific websites, ensuring your extracted email list is relevant and targeted.

An Open-Source Email Extractor Tool

Clauneck is a Ruby gem, and a user-friendly command line tool designed to scrape specific information from a series of URLs, either directly provided or fetched from Google search results via SerpApi's Google Search API. Its key feature is to extract email addresses and much more information like social media handles from web pages and to store the results in a CSV file ordered by their domain name.

What Clauneck provides?

The tool looks for high-quality sequences it can find in the targeted website such as email sequences, and then extracts data to a local file in CSV format. If it cannot find any one of the information on a website, it will label it as null. For unknown errors happening in-between (connection errors, encoding errors, etc.) the fields will be filled with as error.

Unlike Google Chrome extensions that need you to visit webpages one by one, Clauneck excels in bringing the list of websites to you by leveraging SerpApi’s Google Search API.

Below is a part of the landing page of SerpApi. You can observe the targeted information in red boxes:

Here's the end result:

Website	Information	Type of Information
serpapi.com	contact@serpapi.com	Email
serpapi.com	serpapicom	Instagram
serpapi.com	serpapicom	Facebook
serpapi.com	serp_api	Twitter
serpapi.com	null	Tiktok
serpapi.com	channel/UCUgIHlYBOD3yA3yDIRhg_mg	Youtube
serpapi.com	serpapi	Github
serpapi.com	serpapi	Medium

Prerequisites

Since SerpApi offers free credits that renew every month, and the user can access a list of free public proxies online, in terms of pricing, this tool is technically a free email finder tool. You may extract data from approximately 10,000 pages (100 results in 1 page, and up to 100 pages) with a free account from SerpApi.

For collecting URLs to scrape, one of the following is required:
SerpApi API Key: You may Register to Claim Free Credits
List of URLs in a text document (The URLs should be Google web cache links that start with https://webcache.googleusercontent.com)
For scraping URLs, one of the following is required:
List of Proxies in a text document (You may use public proxies. Only HTTP proxies are accepted.)
Rotating Proxy IP

Installation

You will need to install Ruby on your device. You may take a look at Installing Ruby Documentation.

Then you need to install it from your command line:


$ gem install clauneck

Basic Usage

You can use Clauneck as a command line tool or within your Ruby scripts. Here's the command line usage below:


clauneck --api_key YOUR_SERPAPI_KEY --output "results.csv" --q "site:*.ai AND inurl:/contact OR inurl:/contact-us" --proxy "proxies.txt"

The search query in the tutorial: "site:*.ai AND inurl:/contact OR inurl:/contact-us" will search for websites ending with .ai and at /contact or /contact-us paths. You may use more intricate search commands to target more specific groups in alignment with your needs.

Conclusion

I am grateful to the reader for their attention. I hope this blog post and the open-source email address extractor tool can bring ease to your marketing campaigns. Feel free to use it in your integrations, or plugins. If you want to know more about the tool's capabilities or would like to participate in its improvement, you may visit https://github.com/serpapi/clauneck.

Real World Example of AI Powered Parsing

Emirhan Akdeniz — Thu, 22 Jun 2023 18:41:09 +0000

At SerpApi, we're constantly exploring innovative advancements to enhance our services and provide our users with the most cutting-edge solutions. That's why we're excited to introduce a potential new AI feature called AI Powered Parsing, designed to revolutionize how we extract valuable insights from Real Time Search Engine Results using AI Tools.

Disclaimer

The inclusion of this AI feature in SerpApi's offerings is contingent upon official confirmation. The plans outlined here are based on the assumption of approval.

For starters, we plan to utilize AI Powered Parsing for Local Results in a Google Search served by SerpApi's Google Local Pack API, and SerpApi's Google Local API. AI Powered Parsers take advantage of an open-source model called bert-base-local-results, and an open-source ruby gem called google-local-results-ai-parser. We believe in transparency, which is why both the model and the gem composing the core of this AI technology are publicly available for anyone to examine.

To be at the forefront of leveraging gathering data from Search Engine Results Pages, You may register to claim free credits.

Open Sourced Materials in this Blog Post:

What is AI Powered Parsing?

AI-powered parsing for HTML refers to the use of artificial intelligence techniques to extract and understand the content and structure of HTML documents. HTML (Hypertext Markup Language) is the standard language for creating web pages, and parsing refers to the process of analyzing the HTML code to extract meaningful information from it.

Traditionally, HTML parsing has been performed using rule-based algorithms or template-based approaches, where specific patterns or rules are defined to identify and extract desired elements from the HTML. However, these methods can be limited in their ability to handle complex or inconsistent HTML structures, as they rely on predefined rules that may not adapt well to variations in coding styles or new web page designs, resulting in a disruption of the customer experience.

AI-powered parsing offers several advantages over traditional parsing approaches. It can handle a wider range of HTML structures, adapt to different coding styles and variations, and improve accuracy and robustness. Furthermore, AI models can learn from large datasets, which enables them to be continuously improved for their parsing capabilities as they are exposed to more diverse HTML documents.

How is it useful to the user?

One of the key advantages of AI-Powered Parsing is the ability to trade a little extra processing time for greater precision in extracting data from search results. This means you can obtain more accurate and reliable information, resulting in improved decision-making and insights.

We understand that search engines frequently evolve, and keeping up with these changes can be challenging. That's why we continuously update our standard parsers to ensure they remain compatible. However, in the rare event that a deprecation goes unnoticed, our cutting-edge AI Powered Parsers serve as a reliable backup solution, ensuring that you're always covered and can access the data you need.

Accessing this game-changing feature is as simple as including a single parameter in your API requests. We've designed it to be user-friendly and hassle-free, so you can start benefiting from AI-Powered Parsing without any complex setup or configuration.

At SerpApi, we value our users' needs and strive to provide the most reliable and up-to-date solutions to improve user experience. While we continually update our parsers, we understand the importance of offering a backup solution using the power of AI systems for those rare cases when changes occur unexpectedly.

How does it work?

Below, you'll find a general flowchart illustrating the basic functioning. Further sections will provide more specific information.

BERT-Based Classification Model for Google Local Listings

We are excited to present the BERT-Based Classification Model for Google Local Listings, an open-source model available at Huggingface. This powerful model, developed using the Hugging Face library, and a dataset gathered by our own APIs is a key component of our AI-Powered Parsing feature.

The BERT-based model excels in differentiating difficult semantic similarities, handling partial texts that can be combined later, and effortlessly handling vocabulary from diverse areas. It also returns an Assurance Score for after-correction and demonstrates strong performance against grammatical mistakes.

Additionally, we want to emphasize that the model's flaws and limitations are systematically documented in the model card. While the model offers robust performance, it does have certain constraints, and potential misclassifications in specific scenarios. These limitations are diligently addressed to provide users with a transparent understanding of the model's capabilities.

You can play with the model using Free Inference API at the Repository:

Google Local Results AI Server

We're thrilled to announce that we have also open-sourced a simple server code for deploying the bert-base-local-results model. The server code can be found in SerpApi's Github Repository.

The repository contains the code for a server that mimics the Inference API endpoints provided by Hugging Face. The server offers a straightforward interface to perform text classification using BERT-based models. It has been specifically designed by SerpApi to cater to heavy-load prototyping and production tasks, enabling the implementation of the google-local-results-ai-parser gem, which utilizes the serpapi/bert-base-local-results model.

By open-sourcing this server code, we aim to provide developers with a convenient and efficient way to deploy the bert-base-local-results model in their own environments. It offers flexibility and control, allowing you to customize the deployment according to your specific requirements.

Feel free to explore the repository, leverage the server code, and adapt it to suit your needs. We're excited to see how you integrate the bert-base-local-results model into your projects using this server code.

Google Local Results AI Parser

The google-local-results-ai-parser is a gem developed by SerpApi that allows you to extract and categorize structured data from Google Local Search Results using natural language techniques. It uses the serpapi/bert-base-local-results transformer model to parse Google Local Results Listings in English and extract important information, categorizing it into different sections such as ratings, reviews, descriptions, etc.

You may visit SerpApi's Github Repository for an in-depth look at its capabilities. I'll provide a basic explanation of the usage and functions of this next-generation parser.

Let's say you want to parse the following Google Local Result:

You can use the following simple code:

require 'google-local-results-ai-parser'
require 'nokolexbor'
require 'http'
require 'parallel'

html = "HTML of an Individual Local Result"
bearer_token = 'Huggingface Token or Private Server Key'
result = GoogleLocalResultsAiParser.parse(html: html, bearer_token: bearer_token)

The output will be:


{

  "address" => "Nicosia",

  "description" => "Iconic Seattle-based coffeehouse chain",

  "price" => "€€",

  "reviews" => "418",

  "rating" => "4.0",

  "type" => "Coffee shop"

}

You can also utilize it for multiple results:

require 'google-local-results-ai-parser'
require 'nokolexbor'
require 'http'
require 'parallel'

html_parts = [

  "HTML of an Individual Local Result",

  "HTML of another Individual Local Result",

  ...

]
bearer_token = 'Huggingface Token or Private Server Key'
results = GoogleLocalResultsAiParser.parse_multiple(html_parts: html_parts, bearer_token: bearer_token)

In this case, the output will be an array of dictionaries:


[

  {

    "address" => "Nicosia",

    "description" => "Iconic Seattle-based coffeehouse chain",

    "price" => "€€",

    "reviews" => "418",

    "rating" => "4.0",

    "type" => "Coffee shop"

  },

  ...

]

The AI-Powered Parser is able to do this by calling the model, and then doing a rule-based after correction for flaws of the model. This way, maximum precision can be provided. There are also other advanced parameters to cope with unexpected cases. You may refer to the Documentation for these capabilities.

SerpApi's Potential New Parameter

The AI Powered Parser Search operates by simply incorporating a single parameter named ai_parser. By integrating this AI-powered solution, the search results are not only enhanced but also become more precise. While the conventional parsing methods will still function as usual, the AI Powered Parser supercharges the parsing process, effectively refining and streamlining the precision of the results.

We favor solutions that are "no-brainers" - easy to implement, efficient, and notably improving outcomes. This AI parser fits right into that category, offering an innovative and straightforward way to enhance data extraction from Google Local Search Results or any other part that would be supported.

Imagine the following search:

https://serpapi.com/playground?q=Coffee

This is searching for the query Coffee on Google. Imagine the Google Local Pack feature is messing up and giving the address for the price, the price for the number of reviews, etc.

You can open an issue at SerpApi's Public Roadmap and or contact our Customer Success Engineers to notify us about the deprecation. While the issue is being handled by our engineers, you may use the following query:

https://serpapi.com/playground?q=Coffee&ai_parser=true

This will be searching for the query Coffee with our AI Powered Parsers, giving precise results until the issue is fixed. By trading extra time for superior precision, you can maintain your production capabilities as if the issue never happened. We desire to secure your channel of the data supply chain from us as much as we can.

Here's an example image showcasing the use of the parameter in the playground:

Unit Tests for Maintenance

At SerpApi, we value test-driven development. In order to ensure the quality and integrity of our AI Powered Parser Search feature, we have undertaken comprehensive unit testing.

I sifted through all the previous unit test examples available in the stack and enriched the test cases with some recent examples as well. To facilitate the unit testing process, I crafted a Rake task designed to automatically save resulting JSONs as unit test examples in their respective parts.

As a part of the validation, I manually examined these JSONs for any abnormalities or inconsistencies. Once this manual check was completed, I employed Large Language Model (LLM) prompts for a more in-depth verification.

Here's an example context prompt:


Do you think there are any keys that may contain falsely parsed information in the following array at the following possible keys:

"rating", "type", "years_in_business", "service_options", "hours", "reviews_original", "reviews", "address", "description", "price", "phone"

[

  {

    "position": 1,

    "title": "xxx xxxx Ltd",

    "place_id": "9149508700925112934",

    "place_id_search": "http://localhost:3000/search.json?ai_parser=true&device=desktop&engine=google_local&gl=uk&google_domain=google.com&hl=en&location=United+Kingdom&ludocid=9149508700925112934&q=xxx+xxxxx.Ltd",

    "lsig": "AB86z5W38iEx_9mjnRFzmp68DR6h",

    "gps_coordinates": {

      "latitude": 53.9177131,

      "longitude": -2.1785891

    },

    "links": {

      "website": "http://xxx-xxxxxxx.com/"

    },

    "reviews_original": "No reviews",

    "reviews": 0,

    "address": "No ReviewsBarnoldswick"

  }
]

The Answer: "address" key possibly contains a residue. The correct value is probably "Barnoldswick". The residue is probably "No Reviews"

Wait for my instructions.

Followed up by the query on the actual JSON:


Do you think there are any keys that may contain falsely parsed information in the following array at the following possible keys:

"rating", "type", "years_in_business", "service_options", "hours", "reviews_original", "reviews", "address", "description", "price", "phone"

[JSON I want to check out for oddities]

These models, given their generative capabilities, provide helpful feedback about the accuracy of parsed data in keys such as "rating", "type", "years_in_business", "service_options", "hours", "reviews_original", "reviews", "address", "description", "price", "phone", etc.

Given their generative nature, Generative AI language models like OpenAI’s GPT, chatbots like ChatGPT, or Google AI Labs Project in beta version Bard might produce insufficient information in their answers, necessitating multiple follow-up prompts or some extra interpretation of their output. Despite this, it proved to be an efficient alternative to manual inspection, facilitating the process of identifying potential errors or inconsistencies in our JSONs.

After this preliminary check with the JSONs from various older examples that were already available in the stack, we leveraged these in the unit tests. Anytime there was a modification in the responsible parts, these vetted examples proved extremely helpful for comparison.

The overarching objective of these unit tests, as with any other unit tests, was to scrutinize different behaviors in different fixes. The fixes could be localized within the google-local-results-ai-parser gem or be more broadly based within the stack. By implementing this rigorous testing procedure, we ensure our feature's robustness and reliability, enabling it to consistently deliver enhanced and precise results.

Dedicated Pages and Documentation Examples

At SerpApi, we firmly believe in the importance of clear and effective communication with our users. We recognize that the ease of understanding our features and APIs is fundamental to their optimal utilization. Therefore, we've dedicated ourselves to improving our user-facing documentation to facilitate a seamless integration experience.

We have designed two new dedicated pages to serve this purpose. The first page provides a comprehensive overview of our new feature – the AI Powered Parser Search. It outlines the feature's capabilities, details its operation, and presents potential use cases. We believe this will allow our users to grasp the feature's value and envision how it can best fit into their unique workflows. Here are some visual examples:

Disclaimer

These pages contain a work in progress. The final version of these pages might be different than showcased here.

The second page is a complete table of all AI Powered Parsers. It serves as a handy reference for users to understand the different parsers available and the unique functionalities they offer. With this table, users can quickly locate the most suitable AI Powered Parser for their specific needs and applications.

Furthermore, we have augmented our documentation with new examples in two of our key APIs: SerpApi's Google Local Pack Results API, and SerpApi's Google Local Results API to help our users understand how to leverage these APIs effectively.

In conclusion, we continually strive to keep our documentation up-to-date and user-friendly, knowing that it empowers our users to leverage our offerings to their full potential. Whether you are new to SerpApi or an experienced user, these resources will provide valuable guidance in your journey of leveraging AI Powered Parser Search or any of our extensive array of APIs in general.

Brief Overview of the Previous Attempt

In my previous try to tackle this problem, I attempted to develop a local model in Rails for text classification. I followed a similar method as the one mentioned here. However, I encountered some unexpected challenges along the way. Here are the key lessons I learned from that experience.

You can find some of the old blogs from series around the problem at:

Key Lessons from Past Mistakes

Externalize the Model

One of the main problems with my previous approach was to train the model locally. I had to take into account a ton of optimization problems to be resolved. At first sight, it might seem that containing a situation within the stack is an excellent way to keep the problem at bay. It might give you access to better benchmarking and debugging capabilities. But it just turned out to be wrong.

I had to thrive for a more time and performance-efficient model since it was local. This limitation alone is enough to create more confusion in the development of the model. For example, since it was using an RNN model, when I increased the dataset size, it became slower. When I tried to cover the cases in the expanded dataset with less performance, I lost precision. It was a vicious cycle. Not to mention the lack of documentation on Machine Learning in Rails.

In this solution, I have externalized the model to a server, giving more room to play with different models at different capacities. This was an important detail that made this approach into a working solution.

Don't Reinvent the Wheel

In the previous approach, I have tried to train the model from scratch. Play with different structures, different hyperparameters, different datasets, etc. This would seem as if I had more tools in my hand.

But in reality, it gave me more things to complicate the situation. I had to do many grid-type attempts with different structures and hyperparameters which may or may not bear good results. Many times I was lost in details and was limited with my computing power to make subtle observations and make deductions on model types.

In this approach, I used a transformers model called bert-base-uncased. It was already trained on a relatively large English corpus, and all I had to do was to finetune it well to our needs. This has significantly boosted not only the simplicity of training the model but also the performance on precision. The resulting model bert-base-local-results was more capable, easy to implement, and easier to understand.

In conclusion, using solutions that are already good at some part of the solution to your problem, that are easily distributable, and applicable are a better choice for most cases.

Anticipate Conflicts with Other Parts

One other problem I faced with the previous solution was the amount of parallel progress in the reviewing process. These parts ended up creating more trouble than a solution.

For example, I attempted to employ the Rails logic of DRY (Don't Repeat Yourself), and I separated the extraction of the common parts with the old parsers into a separate function to be used by both parsers. By the time I relooked at the previous solution, even the location of the files was updated, and it was blocking any kind of attempt to provide how it is performing.

In this solution, I have decoupled necessary parts even if it means repeating the same code in two files. This way, I can ensure that the changes made on other parts won't affect the reviewing process or the integration of the code.

In conclusion, sometimes repeating yourself is actually good for maintenance. It is also good for keeping maintenance problems at bay. Knowing that the AI Parser problem is most likely caused by a definite part of the whole code is an absolute time saver.

Improve the Product Instead of Replacing it

I made the mistake of thinking I can replace the whole parsing process with an AI solution. At first sight, it could sound great. But the unknown is always fearsome, thus this meant more reviewing time, more details to be careful on the review, harder to explain concepts since they are replacing a known method, etc.

In this solution, I have used it as a parallel backup solution. Being able to switch from a traditional parser to an AI Powered Parser meant less complexity with better control. I safeguarded the code so that if there is any unknown problem, the AI Powered Parsers would be bypassed and the API would resort to traditional parsers.

This way, I was able to provide the same feature without giving up on the old method. Instead of measuring how much performance or precision I have added with this feature, I was able to serve it as a definite bonus.

Refrain from Automation of Every Aspect

I have attempted to create a completely automatic solution from ground zero in the previous solution. From gathering data with SerpApi from within the Rails stack, feeding it directly into the training dataset loader, training process, creating a table for the training debugging values such as loss function and success rates, etc.

At first sight, it might seem like an over-encompassing solution. But the amount of complexity it brought to someone who isn't informed on the process was huge. I had to take unconventional approaches in many parts.

To exemplify, I didn't want us to download the Torch library into each server. So I devised a system in which you could develop a model in Torch with PT format, and then convert that to an ONNX file. Rails had some light options to run ONNX models. But since I wasn't well versed in writing code for transforming PT to ONNX, I had to resort to a Python script within a Rails Stack.

In this solution, I only attempted to fully automate the parsing process with google-local-results-ai-parser gem.

I have also offered a solution to automate serving the model with google-local-results-ai-server, but this is optional. People can still deploy their own solutions at Microsoft Azure, Amazon Sagemaker, or a new startup they think is better for them.

For training the model, people are free to choose which dataset to use or which framework to train if they wish to improve or replicate bert-base-local-results.

The essential take from this is to give people more flexibility in how they want to resolve the problem while providing at least one working path. Automating every aspect with limited built-in options will possibly cause confusion in the long run.

Handle Flaws of the Model Systematically

In my previous approach, because of other complexities, I wasn't able to do a full systematic flaw breakdown. I have only provided the necessary fixes for the flaws. Solutions to these flaws created secondary flaws in the results, and things became out of control.

In addition to that, I also employed some hardcoded regex to do after-corrections. This solution while improving the overall precision, blocked me from implementing better solutions that would encompass a wide range of flaws.

Also, I didn't separate the information fed by the model, and the after-corrections into separate parts. This made it harder to observe the flaw in model behavior versus the after corrections.

In this solution, I have employed a fully systematic breakdown of the known flaws of the model. Also instead of using dense regexes, I've handled after corrections mostly in relation to the position of the elements in the HTML part.

Furthermore, since the model was hosted on a different server, I was easily able to check little entries, and their classifications done by the model. Even using Huggingface's Free Inference API on the model page was extremely helpful in debugging different flaws.

Ensuring that the solution could be easily transferred to other languages, and other search results was a crucial part of this attempt. Until we create a new model for a new language or a new part, we cannot know for sure if it was effective. However, few details here are likely to improve the transferability of the solution.

Compartmentalize Different Parts and Open Source Most of Them

This is much related to refraining from automating every aspect, but subtly different topics. In my previous attempt, I tried to solve the problem from one place because I thought it would be easier to employ.

In this solution, I have broken it into 4 parts as I have lightly mentioned above:

The code to handle the gem, and serve enhanced results in Rails
The AI Powered Parser Gem at google-local-results-ai-parser
The Server Solution to serve the model which I have provided an alternative at google-local-results-ai-server
The Hosted Model bert-base-local-results

I could've easily gone for a server solution that also handles the parsing on a remote server, or used a custom hosting solution to combine multiple parts, etc., and have fewer steps to achieve the same result. This approach would ultimately fail in the long run.

To exemplify this with a relatively similar experience from the old project, I wasn't able to open-source most of the work because there were intended for our codebase, and I mitigated the risk of sharing bits of code to explain the solution. This has led me to create more blog posts to explain the idea.

Although I am grateful to all the readers who read my previous work, this situation caused a secondary problem; mitigating the propagating of the solution versus doing the actual work. Many times when I had to rephrase myself in other blog posts, keeping me from developing.

In this approach, different components are open-sourced, and only a minor code is intended for the stack. This way, I can express the intricate details of how to use google-local-results-ai-parser in a more advanced way inside its documentation, detailed flaws of the bert-base-local-results in the model card, etc. This is still creating a bigger corpus for the audience. But it is also helping me mitigate the risks at bay while having better control at different parts.

Also in the event that this work is not integrated, most of the materials will be out there for public use instead of bits and pieces that interested people must compile themselves.

In conclusion, open-sourcing the parts of your projects, especially ones that are not certain to be accepted, is a good way of mitigating the propagation of the idea versus its actual work. Also, a good compartmentalization of the open-source parts will reduce the risk of exposing delicate parts of your codebase to the public.

Conclusion

I'm grateful for the reader's time and attention. I am also grateful for any potential attention to open-sourced materials used in making it possible. I hope this solution communicates the idea of AI Powered Parsing to people who are interested, and also hope that key takes on my previous mistakes would benefit people in the most potent way. I would be more grateful open sourced materials that would provide use to any projects.

Open Source Review Analyzer Chrome Extension for ChatGPT

Emirhan Akdeniz — Fri, 28 Apr 2023 10:38:19 +0000

Introduction

Online reviews are instrumental in influencing consumer decisions. As a result, effective and efficient review analysis is essential for businesses. SerpApi has developed a practical tool that streamlines this process by harnessing both the capabilities of ChatGPT, and SerpApi: SerpApi's ChatGPT Review Analyzer Chrome Extension.

How SerpApi's ChatGPT Review Analyzer Works

The role of ChatGPT in review analysis

SerpApi's ChatGPT Review Analyzer employs OpenAI's ChatGPT, an advanced language model chatbot, and SerpApi's Review Results from Search Engines to process and analyze vast amounts of review data swiftly. By using artificial intelligence, natural language processing (NLP) techniques, and SERP data, the extension can extract valuable insights from reviews, generating concise summaries, sentiment analysis, and keyword extraction.

References

The ChatGPT Chrome Extension

This powerful tool is designed as a Chrome extension, allowing for smooth integration with your web browser. The extension is ready to use and usable with ChatGPT on Chrome and Brave browsers, offering valuable insights directly within your browsing experience.

You need two accounts for this extension to work. One of which is obviously the ChatGPT account. Another is the SerpApi Account. You may register to claim free credits recurring each month.

Alongside Google Search Results in an easy-to-understand JSON format without worrying about proxies and parsers, SerpApi also provides many search result features in different engines like Bing, Duckduckgo, or shopping engines like Walmart, eBay, etc.

References

Features of SerpApi's ChatGPT Review Analyzer

An example Output of the extension

    Quality of coffee (70% positive)
    Service (40% positive, 30% negative)
    Ambiance/decor (60% positive)
    Availability of snacks/food options (40% positive, 20% negative)
    Price (20% negative)
    Seating/WiFi (60% positive)
    Milk options (30% positive, 10% negative)
    Accessibility (10% positive)
    Cleanliness (20% positive)
    Location (10% positive)

Real-time review analysis

SerpApi offers a variety of engines that offer review results. Most of templates are covered in this extension. You can get real-time review results using the vast ecosystem of SERP Engines such as SerpApi’s Google Reviews Results API, Google Maps Reviews API, Walmart Product Reviews API, Apple App Store Reviews API, and much more. You can find links to some of their documentation and tutorials in the references.

The Google Chrome extension examines reviews instantly, providing valuable insights with the ChatGPT response. This real-time capability enables you to make informed decisions quickly without manually sorting through numerous reviews.

SerpApi also provides an add-on called Ludicrous Speed which provides up to 2.5x speeds. If you would like to improve the speed at which you gather your results, you may take a look at Ludicrous Speed Page.

References

Sentiment analysis

SerpApi's ChatGPT Review Analyzer can ascertain the overall sentiment of a review, classifying it as positive, negative, or neutral. This feature allows users to gauge general opinions about a product or service without reading every review in depth.

References

https://aws.amazon.com/what-is/sentiment-analysis/

Review summarization

The extension employs ChatGPT to produce concise summaries of extensive reviews, emphasizing the most critical points. This feature lets users grasp the crux of a review without dedicating time to reading the entire text.

References

https://huggingface.co/tasks/summarization

Keyword extraction and Topic Modeling

The tool can identify and extract relevant keywords from reviews, offering users a snapshot of the most frequently mentioned topics. This feature assists users in recognizing patterns and trends in customer feedback and opinions, allowing them to concentrate on specific areas of interest or concern.

This is actually also a bit of my personal area of interest. We have a variety of transformer models for on-demand labeled classification. But when it comes to generating labels for these, we haven't got many approaches. In one of my previous blog posts, I covered Latent Dirichlet Allocation (LDA) algorithm for label generation. LLMs prove to be useful in doing the same thing for the batches. A chain of batches, and selection of the most common labels between results has the possibility to provide label generation. This function could be used to classify a larger corpus of text for a variety of purposes beyond reviews.

References

Benefits of SerpApi's ChatGPT Review Analyzer

Time-saving

By automating the review analysis process, SerpApi's ChatGPT Review Analyzer saves users considerable time. This frees businesses and individuals to focus on other essential tasks, enhancing overall productivity without worrying about how to collect their reviews, or how to get insights from them.

Improved decision-making

With SerpApi's capacity to supply real-time insights, users can make better-informed decisions based on customer feedback. This can result in more effective marketing strategies, product improvements, and overall business growth.

Enhanced user experience

SerpApi's ChatGPT Review Analyzer Chrome extension streamlines the review analysis process, creating a more enjoyable and efficient browsing experience. Users can quickly access valuable insights by copying the results SerpApi already scraped for them without the need for opening a new tab to scrape each different review.

Competitive edge

By leveraging ChatGPT and SerpApi, businesses can gain a competitive advantage by staying abreast of trends and comprehending customer feedback better than their competitors. This can lead to improved customer satisfaction and loyalty, as well as increased market share.

How do I use ChatGPT extension in Chrome?

Installation process

The extension isn’t published under Chrome Web Store yet, so the only web page you can access is the GitHub Repository for the moment. It doesn’t have support for Firefox yet. You will need a Chrome or a Brave Browser to use it.

Visit the GitHub repository at https://github.com/serpapi/review-analyzer.
Download the repository as a ZIP file and extract it, or just clone it.
In your Chrome or Brave browser, open the Extensions page (chrome://extensions for Chrome Browser, brave://extensions for Brave Browser).
Enable "Developer mode" in the top right corner of the page.
Click "Load unpacked" and select the extracted folder from your computer.

References

Usage

Navigate to ChatGPT webpage.
Navigate to SerpApi Playground to filter the review results you desire.
Click on the SerpApi's ChatGPT Review Analyzer icon in your browser.
A popup window will open. Copy the review results you gathered from SerpApi in JSON format. (Do not try to trim the unnecessary parts.)
Click on the Analyze Button.
The extension will automatically process and display insights from the reviews, providing you with the 10 most common labels and their percentages based on occurrence rate. It will also indicate whether a label is positive or negative.

Here is an example of Google Maps Reviews Results for a Cafe in New York using SerpApi’s Google Maps Reviews API and GPT-4:

Coffee Quality (70% positive): 7 out of 10 reviews mentioned the quality of the coffee as a positive aspect.
Staff Attitude (30% positive, 60% negative): 6 out of 10 reviews mentioned staff attitude, with 4 of them being negative and 2 being positive.
Pastries and Baked Goods (50% positive, 30% negative): 5 out of 10 reviews mentioned the pastries and baked goods, with 3 being positive and 2 being negative.
Ambiance and Decor (70% positive): 7 out of 10 reviews mentioned the ambiance and decor positively.
Cleanliness (50% negative): 2 out of 10 reviews mentioned cleanliness negatively.
Seating Options (50% positive): 3 out of 10 reviews mentioned seating options as a positive aspect.
Service Speed (50% positive): 2 out of 10 reviews mentioned the speed of service positively.
Location (80% positive): 4 out of 10 reviews mentioned the location as a positive aspect.
Crockery (50% negative): 1 out of 10 reviews mentioned the lack of crockery negatively.
Restroom Availability (100% negative): 1 out of 10 reviews mentioned the lack of restroom negatively.

References

Alternative Input Method

Aside from using the extension with SerpApi's Engines like Google Maps Reviews API or Apple Reviews API, users can alternatively input reviews in an array format. This option offers flexibility in handling reviews from various sources such as tweets, social media posts, youtube video comments, etc., ensuring that valuable insights are not missed. SerpApi also serves these results in its engines. But it is not supported yet for this extension.

Here is an example of reviews from different sources about SerpApi:
Input:

[
"I wanted to try out events search API and found SERP API to be more manageable and versatile with plenty of other things to integrate! Pretty straightforward integration and support are on time with less than 15 mins TAT.",
"As the CEO of ReviewsJet, I have witnessed firsthand the immense value Serp API has brought to our platform. By integrating this powerful real-time API into our review software, we have revolutionized how our users collect and analyze customer feedback from sources such as Google, Yelp, Bing, Yahoo, eBay, and many more.

Before incorporating Serp API, we faced numerous challenges in aggregating reviews from various platforms. Our users often had to spend hours manually searching and compiling data, which led to inefficiencies and inaccuracies. Serp API has completely changed the game for us by automating this process and providing a seamless, comprehensive, and up-to-date source of information.

The real-time capabilities of Serp API have greatly benefited our clients. They can now access the most relevant and current data without worrying about outdated information. This has allowed them to make timely and well-informed decisions, ultimately leading to better products and services for their customers.

At ReviewsJet, we pride ourselves on delivering an exceptional user experience, and Serp API's seamless integration within our platform has been a significant contributing factor to this. Our clients can now easily access and analyze data from multiple sources in a unified interface, resulting in a more efficient and enjoyable workflow.

Furthermore, Serp API's extensive coverage across search engines and platforms has given our users a holistic view of their customer's experiences and opinions. This comprehensive understanding has empowered them to identify areas for improvement and capitalize on their strengths, leading to happier customers and more successful businesses.",
"I used the google scholar service, it was pretty smooth and nice. I spend couple of day to figure a way to do it myself. Then I decided to use SerpApi and it took like 10 minutes to implement and run my good."
...
]

Output:

Ease of Use (Positive) - 23%
Fast and Responsive API (Positive) - 20%
Excellent Customer Support (Positive) - 18%
Comprehensive and Versatile (Positive) - 15%
Pricing Concerns (Negative) - 10%
Documentation Quality (Mixed) - 7% (Positive: 5%, Negative: 2%)
Integration Issues (Negative) - 3%
Wide Range of Features (Positive) - 2%
Real-time Results (Positive) - 1%
Wish for More Free Credits (Negative) - 1%

References

Conclusion

I'm grateful for the reader's time and attention. SerpApi's ChatGPT Review Analyzer Chrome extension offers users a streamlined and efficient way to analyze reviews using SerpAPI results. By harnessing the power of ChatGPT, the extension provides valuable insights, such as sentiment analysis, review summarization, and label extraction. Although it is not an officially supported SerpApi tool or official plugin OpenAI APIs, the extension is an invaluable resource for businesses and individuals seeking to make informed decisions based on customer feedback. I hope it demonstrates to the reader that SerpApi is a great source for developing ChatGPT-powered AI tools.

Global SEO: Strategies for International Success

Emirhan Akdeniz — Thu, 20 Apr 2023 18:46:24 +0000

Introduction to Global SEO

In today's interconnected world, businesses must expand their online presence beyond local borders to reach a global audience, and to have a presence in global markets, especially in sectors like the e-commerce sector. Global SEO is the key to success in this endeavor. It involves optimizing your website for search engines, considering multiple languages, cultures, target countries, and technical elements to improve your visibility in international search results.

One way to achieve better international search results, and expand your market share is by utilizing SerpApi, an API to scrape SERP(an acronym for Search Engine Results Page) that can help you gain insights and make data-driven decisions. I will discuss about different tools SerpApi can provide to help your marketing strategy whether you are an individual or international SEO agency, and provide the details you need to consider in your endeavor.

References:

Importance of Global SEO

Global SEO is essential for businesses aiming to tap into international markets. It helps you:

Gain visibility in international search terms
Increase organic traffic from target regions
Build brand recognition in foreign markets
Improve user experience for global audiences
Boost sales and revenue from international customers
Enhance conversions by catering to a diverse audience

What is the difference between global and local SEO?

Local SEO focuses on optimizing a website to rank higher for a specific local audience, while global SEO targets a broader, international audience. While both share similarities, global SEO requires additional considerations such as language, culture, and regional search engine preferences whereas details such as an address, phone number, or local rating are more important aspects for local SEO.

How do you create a global SEO strategy?

To develop a successful global SEO strategy, consider the following components:

Language and Localization

Tailor your content to your target audience by translating and localizing it into different languages such as English, Spanish, and Japanese to have a better experience for native speakers by doing multilingual SEO with language targeting. Localization involves adapting your content to suit the cultural nuances and preferences of your target market, such as adapting pricing for specific regions such as the Middle East, North America, etc., or specific countries such as Australia, etc. Work with native translators, cultural consultants, native SEO experts, or native SEO companies in the targeted group to ensure accuracy and relevance. SerpApi can help you gather data on language and localization by providing insights into the search results of different countries and languages.

Here's an example of a search from Australia, with the English Language, in a Desktop Device from SerpApi's Google Search Results API:

References:

Technical SEO

Implement technical SEO aspects like international site structure, hreflang tags, and canonical tags to ensure that search engines correctly index your multilingual content. Also, you can improve your site's loading speed, add alt descriptions for images, and mobile-friendliness, etc. for a seamless user experience. Try to create case studies from the metrics such as the PPC (Pay-Per-Click) of your ads, click-through rate, bounce rate, conversion rate, and whatever information you can do with up-to-date on-page SEO practices for your web pages. Make competitor analysis before and after your optimisations. SerpApi can assist in tracking technical SEO factors by monitoring how your site appears in SERPs across different regions. It reduces the need for proxies, HTML parsers, calculating pagerank, which page content had been cached by the search engine's crawlers, and acts as an easy-to-understand API for your SEO needs.

Here's an example with SerpApi's Google About This Results API from google.com.tr domain:

References:

Content Strategy and Content Marketing

Create content that resonates with your target audience. Research their interests, pain points, and preferences to develop a content plan that addresses their needs. Utilize content marketing tactics to promote your brand in new markets, such as Brazil and Spain. Also, optimize your content with relevant keywords for each region. SerpApi can aid in keyword research by providing indicators for your search volume, which can inform your content strategy. You can then combine it with your SEO audit to craft your SEO marketing strategy. It is also vital to be in line with the regulations of the search engine you want to optimize your rank for.

References:

Link Building

Build high-quality backlinks from authoritative websites in your target regions. This will help improve your site's credibility and visibility in local search results. SerpApi can support your link-building efforts by helping you identify potential link opportunities in the target markets by giving you the tools to calculate the best opportunities from organic search data.

Here is another example from SerpApi's Google Search API that checks for different media outlets in the United Kingdom, for the English Language Results, in Mobile Devices:

Social Media and International Outreach

Leverage social media platforms popular in your target markets, such as Yahoo Japan and Naver in South Korea. Engage with influencers and local communities to increase brand awareness and drive traffic to your website. SerpApi can provide data on social media trends and popular platforms in different countries, assisting in your international outreach efforts.

Here's an example from SerpApi's Google Trends API that compares interest between different social media outlets on a worldwide scale:

References

https://serpapi.com/google-trends-api

What are global SEO strategies?

Global SEO strategies involve a combination of the following components:

Language and localization
Technical SEO
Content strategy and content marketing
Link building
Social media and international outreach

What makes strong international SEO?

Strong international SEO involves understanding your target audience, optimizing content for local search engines, and implementing effective technical SEO strategies. Additionally, creating a content strategy that resonates with your audience, building high-quality backlinks, and leveraging social media in your target markets contribute to a robust international SEO strategy.

Different Privacy Rules throughout the World

As you expand your global SEO efforts, it's crucial to be aware of different privacy rules throughout the world. For instance, the European Union's General Data Protection Regulation (GDPR) has strict requirements for data protection and privacy, while the California Consumer Privacy Act (CCPA) governs data privacy in the United States. Brazil has the Lei Geral de Proteção de Dados (LGPD), and the Personal Information Protection and Electronic Documents Act (PIPEDA) is in place in Canada. Ensuring compliance with these privacy laws is essential to avoid potential fines and maintain the trust of your international audience. Working with a professional who specializes in the field, and has the know how of these laws, as well as someone who can deliver the information to the SEO team in a healthy manner will come in handy.

References

International SEO Best Practices

Follow these international SEO best practices for optimal results:

Understanding Your Target Market

Conduct thorough market research to understand your target audience's demographics, preferences, and online behavior. This information will guide your global SEO strategy. Utilize digital marketing techniques and tools to reach your target market effectively.

Domain Structure and Targeting

Choose a domain structure that best suits your global SEO needs. Options include country code top-level domains (ccTLDs), subdomains, and subdirectories. Consider factors like user experience, brand perception, and search engine preferences when making your decision. You may also consider generic top-level domains (gTLDs) for a more global approach. Pay attention to the URL structure and ensure it is easily navigable and understandable by both searchers and search engines.

References:

https://developer.mozilla.org/en-US/docs/Learn/Common_questions/Web_mechanics/What_is_a_URL

Implementing Hreflang Tags

Use hreflang tags to indicate the language and regional targeting of your pages. This helps search engines serve the correct version of your content to users based on their location and language preferences.

References:

https://developer.mozilla.org/en-US/docs/Web/HTML/Element/a

Optimizing Content for Local Search Engines

While Google and Bing dominate the global search market, some regions have their preferred search engines, such as Baidu in China or Yandex in Russia. Optimize your content for local search engines by understanding their unique algorithms and ranking factors. Ensure that your content is also optimized for the local language, as it plays a significant role in search engine optimization. SerpApi allows different APIs so you can track your website's rank through different search engines.

Here's an example from SerpApi's Baidu Search API:

References

Tracking and Analyzing Your Global SEO Efforts

Monitor your global SEO performance using analytics tools like Google Analytics, Google Search Console, and other region-specific tools. Analyze your data to identify areas for improvement and refine your strategy accordingly. SerpApi can complement these tools by providing additional insights into SERP data across different countries and languages.

Common Global SEO Challenges

Some challenges you may face during your global SEO journey include:

Language barriers and translation inaccuracies
Adapting to cultural differences and preferences
Managing multiple domain structures
Navigating local search engine algorithms
Balancing global brand consistency with localization

By understanding and addressing these challenges, you can create a more effective global SEO strategy.

Conclusion

Global SEO is crucial for businesses aiming to expand their reach beyond local markets. By focusing on key components like language and localization, technical SEO, content strategy, link building, and international outreach, you can create a successful global SEO strategy. Additionally, follow international SEO best practices and be prepared to overcome common challenges in the process. SerpApi can be a valuable resource in your global SEO efforts, providing data-driven insights to help you make informed decisions and improve your international search engine visibility. You can register to claim free credits. I am grateful for the attention of the reader.

Real World Example of GPS Targeted Local SERP

Emirhan Akdeniz — Wed, 12 Apr 2023 13:31:05 +0000

Introduction

Local SEO is crucial for businesses looking to improve their visibility and conversions in their specific locations. In this blog post, we will delve into the concept of local SERP, the importance of local SEO, and the use of UULE parameter for accurate local search results. We will also explore an example grid search code for better search engine results precision using GPS coordinates of a location. The example search term we use will be Cafe, and we will gather only the results on the landing page of the search. You may scroll to the bottom of the page to get the full code.

We will use the following to scale the search volume with precision:

What is Local SERP?

A Local Search Engine Results Page (SERP) refers to the search results displayed by search engines when a user conducts a search query with local intent. These results are tailored to the user's geographical location and often include local business listings, maps, and reviews. Local SERPs help users find relevant, nearby businesses or services while enabling businesses to reach potential customers effectively.

What is meant by Local SEO?

Local SEO, or Local Search Engine Optimization, is the practice of optimizing a website, its content, and its online presence to improve visibility in local search results. Local SEO targets potential customers in a specific geographical area by focusing on local search signals such as business listings, reviews, and citations. The goal of local SEO is to increase organic traffic, conversions, and online visibility for businesses with a physical location or service area. SerpApi offers a variety of Local SEO tools including but not limited to Google Local Results API, Google Local Pack Results API, Google Maps API, Google Local Services API, etc.

References:

What are the benefits of Local SERP?

Local SERPs offer numerous benefits for businesses, including:

1) Improved Visibility: Local SERPs provide targeted exposure for businesses by displaying their information to users in the relevant geographical area.

2) Increased Conversions: By targeting users with location-specific intent, local SERPs can drive higher conversion rates.

3) Enhanced Reputation: Local search results often include reviews and ratings, which can help businesses build trust and credibility with potential customers.

4) Competitive Edge: Optimizing for local SERPs can give businesses a competitive advantage over competitors who neglect local SEO.

5) Better User Experience: Local SERPs provide users with relevant, location-specific information, resulting in a more satisfying search experience.

What is the definition of the word "uule"?

UULE (Universal URL Encoded Location) is a URL parameter utilized by Google to pinpoint a user's specific location during a search. This parameter is encoded with a geolocation identifier, known as a canonical name, which represents a particular geographical area, or by GPS coordinates. By incorporating the UULE parameter in search queries, businesses can obtain precise local search results tailored to their target audience's location.

There are two different UULE parameter versions:

Version 1: w+CAIQICI…

This version uses a "canonical name," typically used by Google AdWords to geotarget ads. An example of a canonical name is "West New York, New Jersey, United States," which includes the name of the place, region, and country. Decoding this version requires handling the entire string as a URI component, decoding it, splitting the string by the "plus," and base64-decoding the second part of the string (starting with "CAIQICI").

Version 2: a+cm9…

This version can be found in the cookie written by Google after obtaining the location via the Geolocation API on the website. This version requires no guessing of fields and uses text instead of protocol buffers. After URL-decoding the string and base64-decoding it, you will get ASCII text containing information such as role, producer, provenance, timestamp, latitude, longitude, and radius.

You can employ the uule parameter to get precise results using most of SerpApi’s Google Serp Engines to get real-time organic results. We have developed a ruby gem called uule_converter to convert GPS Coordinates into uule Version 2 as described above. The advantage of this uule is that you don’t have to know the Canonical Names of the locations you are gathering data from. You can employ a grid search with a checker for your local rankings to filter them by their SERP features such as an address, or GPS coordinate of a local result, or their metrics such as rating, number of reviews, etc.

References:

Example Serp Checker Code for a GPS Grid

Possible Integrations

The example code given here is using SerpApi’s Google Search Results Ruby Gem. However, you can use this gem to gather. You may use this gem, or the method therein to gather UULE parameters for a grid search, and then use them using SerpApi’s integrations in Different Languages. You may also make real-time organic searches for mobile devices to expand your keyword research in a specified location and enrich your rank tracker with pagination.

References:

Required Gems and Variables

require 'uule_converter'
require 'geocoder'
require 'json'
require 'fileutils'
require 'concurrent'
require 'rest-client'
require 'google_search_results'

We will be needing the above gems to construct our searcher.

# Set your API key
api_key = 'Your API Key'

We’ll also need the SerpApi API Key. You may register to claim free credits.

# Set the Country of the Place you want to search
address = "Liechtenstein"

Let’s set the address to a small country in Europe called Liechtenstein. I chose this country because of its size. It is surrounded by Austria and Switzerland. Later we will exclude results from those countries, and use only the cafes in Liechtenstein in our results.

# Set the Search Language
search_language = "en"

Search Language has no effect on the ordering of results for this case. We will set it to English for better display.

# Set the preference for grid search
grid_search = false

I added a Boolean variable here just in case you just want to filter your results with the example code.

# Get geocoded address object
address_geocoded = Geocoder.search(address).first

# Define the country code based on the geocoded address
desired_country = address_geocoded.country_code

We will use the Geocoder gem to get relevant data from the address we have given. Distinguishing the country will be important for filtering neighboring countries and making the search more precise in the future.

Grid Search

if grid_search
  # Define address's latitude and longitude boundaries
  bounds = address_geocoded.data['boundingbox'].map(&:to_f)
  south, north, west, east = bounds[0].round(6), bounds[1].round(6), bounds[2].round(6), bounds[3].round(6)

  # Grid step size (approximately 1 km)
  step_size = 0.009

Define the GPS boundaries of a location to create a grid layout. In our case, it will be the farthest point of each cardinal direction of the country Liechtenstein. The step_size variable defines how much distance we must step at each interval in order to cover an area. The uule we will use will give all the cafes in an approximately 1 km radius circle of the GPS coordinates we locked.

  # Create 'searches' directory if it doesn't exist
  FileUtils.mkdir_p('searches')

We will save each search inside the searches directory to be reunited later. This way, long grid searches will have a chance to be recovered in case of abruptions in your system.

  # Create a thread pool
  thread_pool = Concurrent::FixedThreadPool.new(10)

  # Create a concurrent counter for request numbering
  request_counter = Concurrent::AtomicFixnum.new(0)

  # Grid search
  puts "Starting grid search..."
  (south..north).step(step_size).each do |latitude|
    (west..east).step(step_size).each do |longitude|
      thread_pool.post do
        uule_encoded = UuleConverter.encode(latitude, longitude)

        #gl is set to the country code based on the geocoded address
        gl_parameter = desired_country.downcase
        #hl=en, Language parameter fixed for English
        hl_parameter = search_language

        #no_cache=true to get live results

        # Here, I have added a domain for extra precision
        # It is optional for the most part.
        google_domain = "google.li"

        # SerpApi request
        search = GoogleSearch.new(engine: "google_local", google_domain: google_domain, q: "Cafe", gl: gl_parameter, hl: hl_parameter, uule: uule_encoded, serp_api_key: api_key)
        hash_results = search.get_hash

        # You can get deeper results by following the `serpapi_pagination` -> `next_page`
        # To paginate into deeper results. Some results will correlate with the others.
        # It is later filtered out.

        # Save response to a JSON file
        request_number = request_counter.increment
        File.open("searches/response_#{request_number}.json", 'w') { |f| f.write(JSON.pretty_generate(hash_results)) }
      end
    end
  end

  # Shutdown the thread pool and wait for the tasks to complete
  thread_pool.shutdown
  thread_pool.wait_for_termination
end

We will iterate through each GPS coordinate, make a search using google_search_results gem, and then save it into a local file inside searches folder.

One thing to note here is that I have used only the first page results for demonstration purposes. Some of the results have second and more pages. You may refactor this code to implement pagination results as well.

Another detail here is that I have implemented the gl parameter to be the same with Liechtenstein, which is li. This ensures the locales of the search are in order. For extra precision, I have also implemented the google_domain parameter to be the same with the country.

Moreover, you can see that I have made parallel searches. This is a fast way to gather responses from SerpApi. There is also another way to make searches that is even faster, async searches.

You can find detailed instructions on how to use them in the following references. Although they are in Python, it is quite easy to implement them into any language.

References:

Filtering Results

# If you want to skip the grid search and filter results directly
if grid_search == false
  folder_path = "searches"
  file_pattern = File.join(folder_path, "*") # get all files in folder
  file_count = Dir.glob(file_pattern).length
  request_counter = file_count
else
  request_counter = request_counter.value
end

Let’s define the number of files we want to go through to filter results.

# Read searches from local storage
puts "Reading searches from local storage..."
cafes = []
collected_place_ids = []
(1..request_counter).each do |i|
  search_data = JSON.parse(File.read("searches/response_#{i}.json")) rescue nil
  next unless search_data

  if search_data['local_results']
    search_data['local_results'].each do |result|
      unless collected_place_ids.include?(result['place_id'])
        cafes.push(result)
        collected_place_ids.push(result['place_id'])
      end
    end
  end
end

The next step is to collect the unique local results. Since we target a circle with a 1 km in radius, we need to ensure that the excerpt areas in between must be scanned by the grid as well. For that reason, we will always have common results between searches.

# Find the neighbouring countries
class CountryNeighbours
  def self.get_neighbours(country_name)
    country_name = country_name.split(" ").join("%20")
    response = RestClient.get("https://restcountries.com/v3.1/name/#{country_name}?fullText=true")
    parsed_response = JSON.parse(response.body)

    if parsed_response.empty?
      return "Could not find country: #{country_name}"
    end

    neighbours = parsed_response[0]['borders']

    if neighbours.empty?
      return "#{parsed_response[0]['name']['official']} has no neighbors."
    end

    neighbour_names = []

    neighbours.each do |neighbour_code|
      neighbour_response = RestClient.get("https://restcountries.com/v3.1/alpha/#{neighbour_code}")
      parsed_neighbour_response = JSON.parse(neighbour_response.body)
      neighbour_names << parsed_neighbour_response[0]['name']['common']
    end

    puts "#{parsed_response[0]['name']['official']}'s neighbors: #{neighbour_names.join(", ")}"
    neighbour_names
  end
end

Let's define a class to find neighbouring countries.

# Create a thread pool
thread_pool = Concurrent::FixedThreadPool.new(10)

# Create an empty array to store filtered cafes
filtered_cafes = Concurrent::Array.new

# Neigbouring countries
neighbors = CountryNeighbours.get_neighbours(address_geocoded.data["address"]["country"])

# Filter cafes in neighbouring countries in their address
puts "Filtering cafes in neighbouring countries in their address..."
cafes.each do |cafe|
  thread_pool.post do
    if cafe['address']
      neighbouring_country = false
      neighbors.each do |country|
        neighbouring_country = true if cafe['address'].include?(", #{country}")
      end
      next if neighbouring_country

      filtered_cafes.push(cafe)
      puts "Added cafe: #{cafe['title']} at #{cafe['address']}"
      # You may use this part to further filter your data using GPS Coordinates
      #if cafe.key?('gps_coordinates') && cafe['gps_coordinates'].key?('latitude') && cafe['gps_coordinates'].key?('longitude')
      #  coordinates = [cafe['gps_coordinates']['latitude'], cafe['gps_coordinates']['longitude']]
      #  country = Geocoder.search(coordinates).first.country_code
      #  if country == desired_country
      #    filtered_cafes.push(cafe)
      #    puts "Added cafe: #{cafe['title']} at #{cafe['address']}"
      #  end
      #end
    end
  end
end

# Shutdown the thread pool and wait for the tasks to complete
thread_pool.shutdown
thread_pool.wait_for_termination

Since Liechtenstein is small in size, and surrounded by Austria, and Switzerland, we will have lots of results from these countries as well. We can filter them out from their address field. Also, this script could be improved by checking the GPS coordinates of places individually for their country.

# Reindex positions
filtered_cafes = filtered_cafes.each_with_index { |cafe, index|  cafe["position"] = index + 1 }

# Save filtered cafes to a JSON file
File.open('filtered_results.json', 'w') { |f| f.write(JSON.pretty_generate(filtered_cafes)) }

Finally, save the filtered results into a resulting JSON file.

Results

[
  {
    "position": 2,
    "title": "Georg Brot & Kaffee",
    "rating": 4.5,
    "reviews_original": "(86)",
    "reviews": 86,
    "price": "$$",
    "type": "Cafe",
    "address": "Eschen",
    "place_id": "15376372137427994719",
    "place_id_search": "https://serpapi.com/search.json?device=desktop&engine=google_local&gl=li&google_domain=google.li&hl=en&ludocid=15376372137427994719&q=Cafe&uule=a%2Bcm9sZToxCnByb2R1Y2VyOjEyCnByb3ZlbmFuY2U6MAp0aW1lc3RhbXA6MTY4MTIxNjI0MTIwNjExMgpsYXRsbmd7CmxhdGl0dWRlX2U3OjQ3MDQ4NDI5MApsb25naXR1ZGVfZTc6OTQ5ODY3NDAKfQpyYWRpdXM6LTEK",
    "lsig": "AB86z5UWdQN7B3xNWKgYU4wVHFey",
    "thumbnail": "https://serpapi.com/searches/643552f21e803f1d09648e64/images/f98bc385e043920a6d2147a9e986a289c30fda633fea7e587c4423723afebe06.jpeg",
    "gps_coordinates": {
      "latitude": 47.2124433,
      "longitude": 9.523155500000001
    },
    "service_options": {
      "dine_in": true,
      "takeaway": true
    }
  },
  ...
]

With a simple grid search, we can gather all the cafes that show up on the first page of Google Local Results in a small country. It has detailed fields for different uses to make inferences as to what ranks a place better than others. Moreover, if you have a local service that a cafe can use, you can gather all the relevant data about cafes around you. There are many use cases for local SEO as I mentioned above.

I am grateful for the attention of the reader. I hope this blog post will give a good idea about how you can effectively utilize Local SERP data.

Full Code

require 'uule_converter'
require 'geocoder'
require 'json'
require 'fileutils'
require 'concurrent'
require 'rest-client'
require 'google_search_results'

# Set your API key
api_key = 'Your API Key'

# Set the Country of the Place you want to search
address = "Liechtenstein"

# Set the Search Language
search_language = "en"

# Set the preference for grid search
grid_search = false

# Get geocoded address object
address_geocoded = Geocoder.search(address).first

# Define the country code based on the geocoded address
desired_country = address_geocoded.country_code

if grid_search
  # Define address's latitude and longitude boundaries
  bounds = address_geocoded.data['boundingbox'].map(&:to_f)
  south, north, west, east = bounds[0].round(6), bounds[1].round(6), bounds[2].round(6), bounds[3].round(6)

  # Grid step size (approximately 1 km)
  step_size = 0.009

  # Create 'searches' directory if it doesn't exist
  FileUtils.mkdir_p('searches')

  # Create a thread pool
  thread_pool = Concurrent::FixedThreadPool.new(10)

  # Create a concurrent counter for request numbering
  request_counter = Concurrent::AtomicFixnum.new(0)

  # Grid search
  puts "Starting grid search..."
  (south..north).step(step_size).each do |latitude|
    (west..east).step(step_size).each do |longitude|
      thread_pool.post do
        uule_encoded = UuleConverter.encode(latitude, longitude)

        #gl is set to the country code based on the geocoded address
        gl_parameter = desired_country.downcase
        #hl=en, Language parameter fixed for English
        hl_parameter = search_language

        #no_cache=true to get live results

        # Here, I have added a domain for extra precision
        # It is optional for the most part.
        google_domain = "google.li"

        # SerpApi request
        search = GoogleSearch.new(engine: "google_local", google_domain: google_domain, q: "Cafe", gl: gl_parameter, hl: hl_parameter, uule: uule_encoded, serp_api_key: api_key)
        hash_results = search.get_hash

        # You can get deeper results by following the `serpapi_pagination` -> `next_page`
        # To paginate into deeper results. Some results will correlate with the others.
        # It is later filtered out.

        # Save response to a JSON file
        request_number = request_counter.increment
        File.open("searches/response_#{request_number}.json", 'w') { |f| f.write(JSON.pretty_generate(hash_results)) }
      end
    end
  end

  # Shutdown the thread pool and wait for the tasks to complete
  thread_pool.shutdown
  thread_pool.wait_for_termination
end

# Find the neighbouring countries
class CountryNeighbours
  def self.get_neighbours(country_name)
    country_name = country_name.split(" ").join("%20")
    response = RestClient.get("https://restcountries.com/v3.1/name/#{country_name}?fullText=true")
    parsed_response = JSON.parse(response.body)

    if parsed_response.empty?
      return "Could not find country: #{country_name}"
    end

    neighbours = parsed_response[0]['borders']

    if neighbours.empty?
      return "#{parsed_response[0]['name']['official']} has no neighbors."
    end

    neighbour_names = []

    neighbours.each do |neighbour_code|
      neighbour_response = RestClient.get("https://restcountries.com/v3.1/alpha/#{neighbour_code}")
      parsed_neighbour_response = JSON.parse(neighbour_response.body)
      neighbour_names << parsed_neighbour_response[0]['name']['common']
    end

    puts "#{parsed_response[0]['name']['official']}'s neighbors: #{neighbour_names.join(", ")}"
    neighbour_names
  end
end

# If you want to skip the grid search and filter results directly
if grid_search == false
  folder_path = "searches"
  file_pattern = File.join(folder_path, "*") # get all files in folder
  file_count = Dir.glob(file_pattern).length
  request_counter = file_count
else
  request_counter = request_counter.value
end


# Read searches from local storage
puts "Reading searches from local storage..."
cafes = []
collected_place_ids = []
(1..request_counter).each do |i|
  search_data = JSON.parse(File.read("searches/response_#{i}.json")) rescue nil
  next unless search_data

  if search_data['local_results']
    search_data['local_results'].each do |result|
      unless collected_place_ids.include?(result['place_id'])
        cafes.push(result)
        collected_place_ids.push(result['place_id'])
      end
    end
  end
end

# Create a thread pool
thread_pool = Concurrent::FixedThreadPool.new(10)

# Create an empty array to store filtered cafes
filtered_cafes = Concurrent::Array.new

# Neigbouring countries
neighbors = CountryNeighbours.get_neighbours(address_geocoded.data["address"]["country"])

# Filter cafes in neighbouring countries in their address
puts "Filtering cafes in neighbouring countries in their address..."
cafes.each do |cafe|
  thread_pool.post do
    if cafe['address']
      neighbouring_country = false
      neighbors.each do |country|
        neighbouring_country = true if cafe['address'].include?(", #{country}")
      end
      next if neighbouring_country

      filtered_cafes.push(cafe)
      puts "Added cafe: #{cafe['title']} at #{cafe['address']}"
      # You may use this part to further filter your data using GPS Coordinates
      #if cafe.key?('gps_coordinates') && cafe['gps_coordinates'].key?('latitude') && cafe['gps_coordinates'].key?('longitude')
      #  coordinates = [cafe['gps_coordinates']['latitude'], cafe['gps_coordinates']['longitude']]
      #  country = Geocoder.search(coordinates).first.country_code
      #  if country == desired_country
      #    filtered_cafes.push(cafe)
      #    puts "Added cafe: #{cafe['title']} at #{cafe['address']}"
      #  end
      #end
    end
  end
end

# Shutdown the thread pool and wait for the tasks to complete
thread_pool.shutdown
thread_pool.wait_for_termination

# Reindex positions
filtered_cafes = filtered_cafes.each_with_index { |cafe, index|  cafe["position"] = index + 1 }

# Save filtered cafes to a JSON file
File.open('filtered_results.json', 'w') { |f| f.write(JSON.pretty_generate(filtered_cafes)) }

LLMs vs SerpApi: A Comparative Analysis of the Webscraping Capabilities

Emirhan Akdeniz — Thu, 26 Jan 2023 18:25:56 +0000

In this writing, we are going to compare current Open Source and Paid Large Language Models, ChatGPT and SerpApi for a technical task that requires a precise and fast solution. We will make a comparison by extracting different parts of the visible text from Google Local Results.

For more in-depth information on what Google Local Results look like, and how they might be served in a structured order, you can visit SerpApi's Google Local Results API Documentation.

For visual confirmation, here is how the feature looks like this:

Prompt to Test out

Here is the structure of the prompt we will be testing throughout different models:

Example 1:
Lineified text of Google Local Result

Classifications of different parts of the text.

Example 2:
...

Example in Question:
Lineified text of Google Local Result

Follow the rules:
- Rule 1
- Rule 2

classification:

This prompt consists of examples to tighten and empower the classification process, some rules for LLM not to generate unwanted words, and to point to the example in question, and the example in question. The prompt ends with a classification to complete, which will hopefully be followed by other classifications.

Below is the example prompt engineered to do text classification of Google Local Results. I have deducted the phone numbers for privacy purposes. But know that I have used the actual phone numbers when prompting to eliminate bias in phone numbers being picked up by only consisting of zeros.

Example 1:
Houndstooth Coffee 4.6(824) · $$ · Coffee shop 401 Congress Ave. #100c · In Frost Bank Tower Closed ⋅ Opens 7AM Cozy hangout for carefully sourced brews

title: Houndstooth Coffee
rating: 4.6
number of reviews: 824
expensiveness: $$
type: Coffee shop
address: 401 Congress Ave. #100c  · In Frost Bank Tower
hours information: Opens 7AM
description: Cozy hangout for carefully sourced brews
service options: -
phone number: -
years in business: -

Example 2:
A.D.A. Auto Repair Center 4.8(26) · Auto repair shop 30+ years in business · Nicosia · 00 000000 Closes soon ⋅ 3PM "I strongly recommend this repair shop."

title: A.D.A. Auto Repair Center
rating: 4.8
number of reviews: 26
expensiveness: -
type: Auto repair shop
address: Nicosia
hours information: Closes soon ⋅ 3PM
years in business: 30+ years in business
description: "I strongly recommend this repair shop."
phone number: 00 000000
years in business: -

Example 3:
A to M MARKET 5.0(2) · General store Nicosia · Near Macro Süpermarket Open ⋅ Closes 2AM In-store shopping

title: A to M MARKET
rating:  5.0
number of reviews: 2
expensiveness: -
type: General store
address: Nicosia · Near Macro Süpermarket
hours information: Open ⋅ Closes 2AM
description: -
service options: In-store shopping
phone number: -
years in business: -

Example 4:
Expeditionary 4x4 5.0(1) · Auto parts store (000) 000-0000 Open 24 hours

title: Expeditionary 4x4
rating:  5.0
number of reviews: 1
expensiveness: -
type: Auto parts store
address: -
hours information: Open 24 hours
description: -
service options: -
phone number: (000) 000-0000
years in business: -

Example 5:

Hibbett Sports 4.2(51) · Sporting goods store 4.2 (51) Independence, KS Closed ⋅ Opens 10 AM · (000) 000-0000 Closed ⋅ Opens 10 AM Closed Athletic shoes & activewear

Follow the rules:
    - Give the answer from only after the text `Example 5:`.
    - Do not manipulate the text from Example 5.
    - Make sure to check before and after text for addendum texts.
    - Do not give any reason for your answer.
    - Do not give any regex example.
    - Do not give any code example.
    - Give only classifications.


title:

Open Source LLM Models

GPT-2

Context: GPT-2 has 124M trainable parameters.

Answer:

title: Hibbett Sports 6.4(817) · Fitness Club 1.0 (5) 911-4

Conclusion: The answer is flawed and contains extra and made-up information. Cannot be used for parsing that requires multiple classifications from the getgo.

GPT-2 Large

Context: GPT-2 Large has 774M trainable parameters.

Answer:

title: Hibbett Sports 4.3

Conclusion: The answer is flawed and contains extra and made-up information. Cannot be used for parsing that requires multiple classifications from the getgo.

GPT-2 XL

Context: GPT-2 Large has 1.5B trainable parameters.

Answer:

title: Hibbett Sports 4.0

Conclusion: The answer is flawed and contains extra and made-up information. Cannot be used for parsing that requires multiple classifications from the getgo.

GPT-J 6B

Context: GPT-J 6B has 6B trainable parameters.

Answer:

title: Hibbett Sports

GPT-J 6B is not producing anything other than the requested classification. So I had to ask multiple classes to be sure:

Second Answer:

address: 4.2 (51) Independence, KS

Conclusion: Although the model provided the correct answer for the title, it wasn't able to classify the address. Not suitable for multiple classifications out of the box.

I must confess I was surprised by the speed at which the answer was returned. With proper finetuning, this model has the potential to be used for technical purposes for a relatively cheap price, and fast response times.

BLOOM

Context: Bloom has 176B trainable parameters. This model is a behemoth when it comes to open-source models. It has support for 46 languages and 13 programming languages. A switch between a greedy mode for technical purposes, and a sampling mode for creative purposes.

Answer:

title: Hibbett Sports
rating: 4.2
number of reviews: 51
expensiveness: -
type: Sporting goods store
address: Independence, KS
hours information: Closed ⋅ Opens 10 AM
description: Athletic shoes & activewear
service options: -
phone number: (000) 000-0000
years in business: Closed ⋅ Opens 10 AM
years in business: -

Conclusion: All the answers are correct. This is definitely suitable for technical tasks of engineering.
I must add my further observations about BLOOM. The downside is that it is big in size. As of now, around 330 GB. You also need at least 16 GB of RAM and some real free time in your hand to test this behemoth.
They were kind enough to set up a managed endpoint for testing purposes. But testing for specific purposes of technical nature will most likely take a hard manual prompt creation and a big decision to set it up.
Another thing I have noticed about BLOOM is that if it creates noisy data at the end, you can always use multiple prompts. Somehow this prompt lead the model to show always the right answer on the first line of generated text.
Also, I have tested cross-language prompts to see how it behaves. Sometimes it gives good results for English examples but Arabic classification. But it would be wise to use the same language prompts for the same language classifications just to be on the safe side.
I have to admit though, I am still surprised this is an open-source model. It is truly a behemoth at the standards of the industry.

Paid Models

GPT-3 Text-Ada-001

Context: This model is the lowest fee text model of OpenAI, and costs $0.0004 for 1000 tokens. Tokens are recognized parts of strings and letters by models. This model accepts up to 2048 tokens. The prompt we provided is 784 tokens (cost is affected by response token size as well).

Answer:

title:  Hibbett Sports
rating: 4.2
number of reviews: 51
expensiveness: Sporting goods store 4.2 (51)
type: Sporting goods store
address: 4.2 (51) Independence, KS
hours information: Open 10am-9pm
hours information: (000) 000-0000
description: (000) 000-0000
service options: -
phone number: (000) 000-0000
years in business: -

Conclusion: This model, even though the generation temperature is quite low (good for consistent classification and minimum made-up text), has failed to produce a healthy classification. I am not sure if finetuning would put it in shape.

GPT-3 Text-Babbage-001

Context: This model is a low-fee text model that is better than Text-Ada-001 of OpenAI, and costs $0.0.0005 for 1000 tokens. This model also accepts up to 2048 tokens.

Answer:

title:  Hibbett Sports
rating: 4.2
number of reviews: 51
expensiveness: $$
type: Sporting goods store
address: Independence, KS
hours information: Open 10 AM
description: (000) 000-0000 Closed ⋅ Opens 10 AM Closed
service options: -
phone number: -
years in business: -

Conclusion: This model missed many parts of the text that is crucial. However, a finetune of this model has the potential to be a good candidate for multiple classifications. However, finetuning costs $0.0006 for 1000 tokens, and using the finetuned version costs $0.0024. This is not really feasible considering the next paid model's base cost is lower than this.

GPT-3 Curie

Context: This model is a text model that is better than Text-Babbage-001 of OpenAI, and costs $0.0.0020 for 1000 tokens. This model also accepts up to 2048 tokens. This model carries the name of one of the most bada** scientists in history, the undisputed star of the Flying Universities.

Answer:

title:  Hibbett Sports
rating: 4.2
number of reviews: 51
expensiveness: -
type: Sporting goods store
address: Independence, KS
hours information: Closed ⋅ Opens 10 AM
description: Athletic shoes & activewear
service options: -
phone number: (000) 000-0000
years in business: -

Conclusion: All correct. Without a doubt, this is one of the good candidates for industrial scale. The downside is the token size. You have to carry the context and rules at each call, costing you more. The price for this model is $0.0020 per 100 tokens which is kind of limiting for big scales. Also, the concurrency at OpenAI is 3000 requests per minute for paying users. This is another limitation since the limit of tokens is also increasing the number of requests you must make concurrently.
I would suggest using this model for testing what you already have in place for it. Its precision could help improve your existing solution and provide feedback of its state.

ChatGPT

Context: I don't think this model even needs an introduction. Possibilities using this model seem quite endless. However, it is not open for commercial use now, and we don't really know what the pricing or concurrency limits will be for certain.

title:
Hibbett Sports
rating: 4.2
number of reviews: 51
expensiveness: -
type: Sporting goods store
address: Independence, KS
hours information: Closed ⋅ Opens 10 AM
description: Closed Athletic shoes & activewear
service options: -
phone number: (000) 000-000
years in business: -

Conclusion: It can do basic classification tasks with ease. Moreover, being able to carry out the conversation brings the possibility of using the contextual prompt to be reused with ease. I would have only one concern for this model though, and that is persistence. It would be good to have a temperature option for ChatGPT as well. Since definite pricing and limitations weren't announced, I cannot say something certain about its industrial use.

GPT-3 Text-Da-Vinci-003

Context: This is the best text model of OpenAI so far, and costs $0.0.0200 for 1000 tokens. Time will tell if naming it DaVinci was a good choice or if they should've waited for a superior model for that name. But for sure this is the most advanced text model out there. Its precision in understanding the prompt is absolutely beating the 4000 token limit. For many technical tasks, you don't even need finetuning.

Answer:

title: Hibbett Sports
rating: 4.2
number of reviews: 51
expensiveness: -
type: Sporting goods store
address: Independence, KS
hours information: Closed ⋅ Opens 10 AM
phone number: (000) 000-0000
years in business: -
description: Athletic shoes & activewear

Conclusion: This is without a doubt a killer product. However, when you need to correlate between different parts of the HTML to gather deeper info about the feature you want to extract, the 4000 token limit is playing a major blockage.
Another blockage is the price. Google allows 20 local results per page. This means the estimated cost for one page of visible elements will be around $0.16 per page. Now, this is just the price for parsing the features. You may add a proxy and server maintenance to it. When you scale this number, an in-house solution using Text-Da-Vinci-003 only becomes viable when you have a derivative product using the Google Local Results feature that is more valuable than the raw data. This is only possible if the solution is cutting down the maintenance fee, and the error rates are significantly less to a point where you don't have to hire new staff to take care of it.
But I have to point out that this is a very suitable model for testing existing models at this time. Its semantic power in understanding the rules and order in the generation is very suitable for testing the efficiency of technical tasks at this point.

SerpApi Google Local Results Parser

Context: SerpApi is a real-time API to access Google search results. It handles proxies, solves captchas, and parses all rich structured data for you. So any kind of preprocessing and parsing you have to do with an LLM model is actually handled by SerpApi. Apart from that, SerpApi can easily parse data invisible to the eye as well like Place ID or GPS coordinates.

Answer:

    {
      "position": 1,
      "title": "Hibbett Sports",
      "place_id": "5028461961943251508",
      "place_id_search": "https://serpapi.com/search.json?device=desktop&engine=google&gl=us&google_domain=google.com&hl=en&ludocid=5028461961943251508&q=Sports+Shop&tbm=lcl",
      "lsig": "AB86z5USlyxNPnHhPC2QbT2VYbMc",
      "rating": 4.2,
      "reviews_original": "(51)",
      "reviews": 51,
      "type": "Sporting goods store",
      "address": "Independence, KS",
      "phone": "(000) 000-0000",
      "hours": "Open ⋅ Closes 8 PM",
      "description": "Athletic shoes & activewear",
      "thumbnail": "https://serpapi.com/searches/63d036577bea270d77944e2d/images/581525d33cb22a3571e8ea251d12f5f818ab21466a5be2cfc24aa6e2784524e7.jpeg",
      "gps_coordinates": {
        "latitude": 37.2243156,
        "longitude": -95.7405395
      }
    },

Conclusion: SerpApi's biggest power comes from its dedication to its mission. Because it has a team dedicated to fixing errors and adding new features, you are getting the most out of what you expect. Moreover, not worrying about proxies, server maintenance, and many other things is another bonus. It beats the pricing point by a good margin, and it is fast. You can see the Pricing Page to make a comparison in pricing, and API Status Page to get more information on speed. It also guarantees up to 20% of your plan searches per hour as a throughput. You can also Register to Claim Free Credits.
All of this information is enough to see that some technical tasks are still yet not feasible to achieve by LLMs.

Final Thoughts

I can see that the companies that have a tradition of implementing AI have a better advantage in the future. the capabilities of such tools as shown above are truly impressive. Considering these are just baby steps of AI, one has to be ready for the point at which AI can beat traditional methods. This is only possible via an existing culture around AI.

If we go from an example, in my previous blog posts, I have parsed the inline version of Google Local Results with a CNN structured Hybrid AI and compared the results. That was a definite win in the long run in terms of AI. But considering there is a dedicated page for local results, and it is completely parsable via traditional methods, it nullifies the advantages of AI until it is faster and cheaper. I know that in the future, prices will be more accessible for the industry, and the speed will be much more efficient. It will make the entire process more interesting and efficient.

I am grateful to the reader for their time and attention. The contents of this writing only contain a portion of my observations. In my opinion, some fields, especially fields requiring technical precision, are not viable for LLMs at this point.

Analyzing the Distribution of Problems in Competition Reviews

Emirhan Akdeniz — Mon, 23 Jan 2023 12:23:48 +0000

Analyzing the Distribution of Problems in Competition's Reviews

In the highly competitive world of mobile app development, it is essential to stay ahead of the curve in terms of optimization and marketing strategies using best app store optimization tools. One key area of focus is understanding the common problems and complaints of competitors' app reviews, in order to improve upon and address those issues in your own app. With the use of ASO tools like SerpApi's Apple App Store Search Scraper API and Apple App Store Reviews Scraper API, as well as Natural Language Processing models such as facebook/bart-large-mnli and bert-large-uncased-whole-word-masking-finetuned-squad, we can analyze the distribution of problems in competition's reviews and gain valuable insights for our own app optimization which could potentially increase download an install rate, the conversion rate from your competition, give key ideas on your app marketing, boost your app visibility, and many more KPI objectives you can achieve with app analytics. You can find the full code at the end of this page.

Collecting Product IDs from the Apple App Store Search Scraper API

SerpApi’s Apple App Store Search Scraper API is a tool used for gathering in-depth metrics, keyword rankings, the number of app downloads, and much more details of an app in the iOS App Store, or Mac App Store. It is an essential tool for keyword tracking and gives you a good insight into how App Store’s ranking algorithm works. The first step we will use in creating this code is the process is to gather the product IDs of apps related to our search term, using the Apple App Store Search Scraper API. This can be done with the following code snippet:

def call_app_store_api(search_term, api_key):
  app_store_url = f"https://serpapi.com/search.json?api_key={api_key}&engine=apple_app_store&term={search_term}"
  response = requests.get(app_store_url)
  product_ids = [organic_result["id"] for organic_result in response.json()["organic_results"]]

  return product_ids

This function takes in a search term and API key and returns a list of product IDs for apps related to that search term. You may register to claim free credits and use your API Key in the tool.

Collecting Reviews from the Apple App Store Reviews Scraper API

Once we have the product IDs of the apps related to our search term, we can use the Apple App Store Reviews Scraper API to gather the reviews of each app with automation. This can be done with the following code snippet:

async def call_apple_reviews_api(product_id, api_key):
  apple_reviews_url = f"https://serpapi.com/search.json?api_key={api_key}&engine=apple_reviews&product_id={product_id}&sort=mostcritical"
  response = requests.get(apple_reviews_url)
  reviews = [review["text"] for review in response.json()["reviews"]]

  return reviews

This function takes in a product ID of an App Store Listing and SerpApi API key, and returns a list of reviews for that app, sorted from most negative to least negative with the help of sort=mostcritical parameter.

Collecting All Reviews for a Search Term

To gather all reviews for apps related to our search term, we can use the following function in the code snippet:

async def get_product_reviews(product_ids, api_key):
  reviews = await asyncio.gather(*[call_apple_reviews_api(product_id, api_key) for product_id in product_ids])
  reviews = list(chain(*reviews))

  return reviews

This function takes in a list of product IDs and SerpApi API key, and returns a list of all reviews for those apps. Notice that it doesn’t store which reviews belong to which app name. It will serve all the reviews of top apps ranked in a relevant keyword to your app.

Classifying Reviews as Positive or Negative

Sometimes, an app could have a little number of reviews, or a user might give a 1-star but a positive review. These kinds of events might prevent you from seeing genuine negative reviews of your competition. With the use of natural language processing models, we can classify the reviews as positive or negative, and return only the absolutely negative reviews in the tool. The following code snippet demonstrates this process:

def get_absolutely_negative_product_reviews(classifier, reviews):
  absolutely_negative_reviews = []
  candidate_labels = ['negative', 'positive']
  for review in reviews:
    result = classifier(review, candidate_labels)
    index = result["scores"].index(max(result["scores"]))
    classification = result["labels"][index]
    if classification == "negative":
      absolutely_negative_reviews.append(review)

  return absolutely_negative_reviews

This function takes in a classifier and a list of reviews and returns a list of only the absolutely negative reviews.

Summarizing Absolutely Negative Reviews

To gain a clearer understanding of the specific problems mentioned in the absolutely negative reviews, we can use a question-answering pipeline to summarize the reviews. This is important to find the main concern of the user in a long review, or in a mixed review with positive and negative aspects. Imagine the following review:

Screenshots are nice. But they don’t represent the app.

What we want is the concern for the honesty of the developer from this review which is the latter sentence. So we want the question-answer pipeline to give us keyword suggestions into what the main problem is in the review. The following code snippet demonstrates this extraction process:

def get_summaries(qa_pipeline, absolutely_negative_reviews):
  summaries = []
  for context in absolutely_negative_reviews:
    result = qa_pipeline({
      'context': context,
      'question': 'What is the problem?'
    })["answer"]
    summaries.append(result)

  return summaries

This function takes in a question-answering pipeline and a list of absolutely negative reviews and returns a list of summaries for those reviews.

Classifying Summaries and Returning Distribution of Problems

Understanding the ratio of problems in your competition’s reviews is a crucial part of mechanics in app ranking, user acquisition, and overall store intelligence. To understand the distribution of problems mentioned in the summaries, we can classify the summaries with a preset of potential problems and return the distribution of problems. The following code snippet demonstrates this process:

def get_problems_distribution(classifier, possible_problems, summaries):
  summary_dicts = []
  for summary in summaries:
    result = classifier(summary, possible_problems)
    index = result["scores"].index(max(result["scores"]))
    classification = result["labels"][index]
    summmary_dict = {"summary": summary, "classification": classification}
    summary_dicts.append(summmary_dict)

  classifications = [d['classification'] for d in summary_dicts]
  counter = Counter(classifications)
  total = sum(counter.values())
  ratios = {problem: counter[problem]/total for problem in possible_problems}
  ratios = dict(sorted(ratios.items(), key=lambda item: item[1], reverse=True))

  return ratios

This function takes in a classifier, a list of possible problems, and a list of summaries, and returns the distribution of problems mentioned in the summaries.

Showing Results

Finally, the following code snippet can be used to print the results of the problem distribution:

print("-----")
print(f"Summary of problems for search term: {term}")
for ratio in ratios:
  print(f"{ratio}: {ratios[ratio]*100:.2f}%")

Here is an example output:

Summary of problems for search term: Coffee
unenjoyable: 29.28%
incompleteness: 27.03%
expensiveness: 12.61%
dishonesty: 9.01%
too many advertisements: 8.11%
inefficiency: 8.11%
bugs: 5.86%

How is App Store Optimization done?

I will explain different aspects of a possible ASO strategy using this tool. With the results of our analysis, we can see that the top problems among the competition's negative reviews for the search term "Coffee" are unenjoyable experiences, incompleteness, and expensiveness. Here are some specific strategies to address these issues:

Unenjoyable experiences

To address this problem, it is crucial to gather customer feedback and gain a deep understanding of what is causing the negative experiences. This could involve conducting surveys, focus groups, or user testing to gather qualitative data. Additionally, using tools such as sentiment analysis on customer reviews can provide valuable insights into whether the root causes are from technical reasons such as app performance, or from design issues. Once the root causes of the negative experiences have been identified, it is important to develop and implement solutions to improve the overall enjoyment that is supported with app intelligence.

Incompleteness

In order to address this problem, it is essential to have a clear understanding of the features and functionality that customers expect from the app. This could involve conducting market research, surveying customers, localization demands, notifications for bugs or gathering data on the features of competitors' apps. Once the desired features have been identified, it is important to prioritize and implement them in the app, while also continuously gathering feedback to ensure they are meeting the needs of the customers.

Expensiveness

To address this issue, it is important to review and optimize pricing strategies regularly. This could involve conducting market research to understand the prices of similar apps and identifying opportunities to offer competitive pricing. Additionally, implementing in-app purchase options or subscription models can provide alternative revenue streams and make the app more accessible to a wider range of customers.

Additionally, it is also essential to address other problems such as dishonesty, too many advertisements, inefficiency, and bugs, as these can also negatively impact the customer's experience. To address these issues, it is crucial to have a robust testing and quality assurance process in place, as well as regular monitoring of customer feedback to identify and address any problems in a timely manner.

Overall, by gathering and analyzing customer feedback, understanding the competition, and implementing solutions to address the identified problems, it is possible to improve the overall performance of the app and outrank the competition in the search results.

Conclusion

In conclusion, outranking the competition in the search results requires a comprehensive approach that involves gathering and analyzing customer feedback, understanding the competition, and implementing solutions to address the identified problems. By using natural language processing models and other tools such as SerpApi's Apple Search Scraper API, and SerpApi's Apple App Store Reviews Scraper API, sentiment analysis, question-answering pipelines, and A/B testing, it is possible to gain valuable insights into customer needs and preferences, and to make data-driven decisions to improve the overall performance of the app, and decide according to your App Store Optimization Strategy. Additionally, it is important to stay up to date with the latest trends and technologies, and to continuously gather and analyze customer feedback to identify new areas for improvement such as finding the best keywords to compete in via keyword research, keyword optimization for metadata of your app description, Search Engine Optimization(SEO) for your app page on App Store or Google Play Store (for Android Apps), effective social media campaigns and overall market intelligence, etc. By following these strategies, it is possible to improve the overall performance of the app and outrank the competition in the search results or find the best app title, app keywords to use in the app description, optimize your app for the most organic downloads, and organic installs, gather crucial information on the weak points of the competitive app developers in top apps in a keyword, etc. from the getgo before entering the competition as a startup. I am grateful to the reader for their time and attention. You can find the full code below.

Full Code:

from collections import Counter
from transformers import pipeline
from itertools import chain
import requests
import asyncio

# Collect product_ids from Apple App Store Search Scraper API
def call_app_store_api(search_term, api_key):
  print("-----")
  print(f"Gathering product ids from SerpApi's Apple App Store Scraper API for search term: {search_term}")
  app_store_url = f"https://serpapi.com/search.json?api_key={api_key}&engine=apple_app_store&term={search_term}"
  response = requests.get(app_store_url)
  product_ids = [organic_result["id"] for organic_result in response.json()["organic_results"]]

  return product_ids

# Collect reviews of an individual app from most negative to least negative from Apple App Store Reviews Scraper API
async def call_apple_reviews_api(product_id, api_key):
  print("-----")
  print(f"Gathering Reviews from Apple App Store Reviews Scraper API for product id: {product_id}")
  apple_reviews_url = f"https://serpapi.com/search.json?api_key={api_key}&engine=apple_reviews&product_id={product_id}&sort=mostcritical"
  response = requests.get(apple_reviews_url)
  reviews = [review["text"] for review in response.json()["reviews"]]

  return reviews

# Collect all reviews of apps in a search term from Apple App Store Reviews Scraper API
async def get_product_reviews(product_ids, api_key):
  reviews = await asyncio.gather(*[call_apple_reviews_api(product_id, api_key) for product_id in product_ids])
  reviews = list(chain(*reviews))

  return reviews

# Classify reviews as positive or negative and return absolutely negative reviews
def get_absolutely_negative_product_reviews(classifier, reviews):
  absolutely_negative_reviews = []
  candidate_labels = ['negative', 'positive']
  for review in reviews:
    print("-----")
    print(f"Classifying review: {review}")
    result = classifier(review, candidate_labels)
    index = result["scores"].index(max(result["scores"]))
    classification = result["labels"][index]
    print(f"Classification: {classification}")
    if classification == "negative":
      absolutely_negative_reviews.append(review)

  return absolutely_negative_reviews

# Summarize absolutely negative reviews
def get_summaries(qa_pipeline, absolutely_negative_reviews):
  summaries = []
  for context in absolutely_negative_reviews:
    print("-----")
    print(f"Summarizing review: {context}")
    result = qa_pipeline({
      'context': context,
      'question': 'What is the problem?'
    })["answer"]
    print(f"Summary: {result}")
    summaries.append(result)

  return summaries

# Classify summaries and return distribution of problems
def get_problems_distribution(classifier, possible_problems, summaries):
  summary_dicts = []
  for summary in summaries:
    print("-----")
    print(f"Classifying summary: {summary}")
    result = classifier(summary, possible_problems)
    index = result["scores"].index(max(result["scores"]))
    classification = result["labels"][index]
    print(f"Classification: {classification}")
    summmary_dict = {"summary": summary, "classification": classification}
    summary_dicts.append(summmary_dict)

  classifications = [d['classification'] for d in summary_dicts]
  counter = Counter(classifications)
  total = sum(counter.values())
  ratios = {problem: counter[problem]/total for problem in possible_problems}
  ratios = dict(sorted(ratios.items(), key=lambda item: item[1], reverse=True))

  return ratios

api_key = "<SerpApi Key>"
term = "Coffee"
possible_problems = [
  "bugs",
  "too many advertisements",
  "inefficiency",
  "dishonesty",
  "unenjoyable",
  "incompleteness",
  "expensiveness"
]
classifier = pipeline(
  "zero-shot-classification",
  model="facebook/bart-large-mnli"
)
qa_pipeline = pipeline(
  "question-answering",
  model="bert-large-uncased-whole-word-masking-finetuned-squad"
)

product_ids = call_app_store_api(term, api_key)
reviews = asyncio.run(get_product_reviews(product_ids, api_key))
absolutely_negative_reviews = get_absolutely_negative_product_reviews(classifier, reviews)
summaries = get_summaries(qa_pipeline, absolutely_negative_reviews)
ratios = get_problems_distribution(classifier, possible_problems, summaries)

# Print results
print("-----")
print(f"Summary of problems for search term: {term}")
for ratio in ratios:
  print(f"{ratio}: {ratios[ratio]*100:.2f}%")

Open Source AI Image Classifier with Automatic Dataset Creator

Emirhan Akdeniz — Fri, 06 Jan 2023 19:08:18 +0000

Serapis AI Image Classifier is a program that allows you to automatically create image datasets using SerpApi's Google Images Scraper API, finetune a ResNet50 model, and classify images using the trained model.

Installation

You can install these dependencies using the following command:

pip install requirements.txt

Usage

You can use Serapis AI Image Classifier in one of the following three modes:

Create a dataset and train a new model from scratch

To create a dataset and train a new model from scratch, you will need to provide a list of labels and an image to use as a reference for the scraping process.
SerpApi API Key is necessary for this mode for the program to automatically scrape images you will use in your database using SerpApi's Google Images Scraper API.

You can register to SerpApi to claim free credits.

python serapis.py --train --labels eagle, bull, lion, man --image-path lionimage.jpg --api-key <SerpApi-API-KEY>

Use old scraped images and train a new model

To use old scraped images and train a new model, you will need to provide a list of labels and specify that you want to use old images with the --use-old-images flag.
You can also put your images in the images/ folder and also add enter them in images/catalogue.csv to manually train models using your own dataset.

python serapis.py --train --labels eagle, bull, lion, man --use-old-images --image-path lionimage.jpg

Use a previously trained model

To use a previously trained model, you will need to provide the path to the trained model and an image to classify.

python serapis.py --model-path models/1.pth --image-path lionimage.jpg

Dialogue Mode

You can also navigate the program by not providing any arguments and using the dialogue mode:

python serapis.py

The Output

The output will give you the answer:

The image contains Lion

Optional Arguments:

-h, --help                    Help to Nagigate
--train                       Whether to train a new model
--model-path MODEL_PATH       Pretrained Model path you want to use
--dialogue                    Whether to use dialogue to navigate through the program
--use-old-images              Whether to use old images you have downloaded to train a new model
--api-key API_KEY             SerpApi API Key
--limit LIMIT                 Number of images you want to scrape at most for each label
--labels LABELS [LABELS ...]  Labels you want to use to train a new model
--image-path IMAGE_PATH       Path to the image you want to classify

I am grateful to readers for their time and attention. I hope this repository helps in your tasks, or helps you grasp the possibilities you have when you have SERP data at your disposal.