A Review Aggregation Pipeline: Multi-Platform Collection and Sentiment Preprocessing

Every product team I've worked with eventually asks the same question: "what do people actually think about us?" And every data person building the answer discovers the same three problems in sequence. One: reviews about the same product live on half a dozen platforms with completely different shapes. Two: the text is multilingual, emoji-dense, sarcastic garbage that off-the-shelf sentiment models choke on. Three: the same reviews show up scraped and re-scraped across aggregator sites until your dataset quietly doubles.

This post is a field guide to building a review aggregation pipeline — collection, normalization, deduplication, and the preprocessing layer that makes sentiment analysis actually work. I'll use Python throughout.

Layer 1: Collection, or Why One Scraper Never Survives

A typical brand has reviews on its own Google Business listing, Amazon (if physical product), Trustpilot or G2 (if B2B), the App Store and Play Store (if mobile), and a long tail of niche verticals. Each has a different anti-bot posture:

• Official APIs first, always. Google Places, App Store Connect, Play Developer API, and Trustpilot's business API all expose reviews with permission. Use them where you can; they're stable and legal-risk-light.
• App Store/Play public feeds are JSON endpoints — trivially fetchable, no proxies required.
• Marketplace and review-platform pages vary from easy to Akamai-hard. For these, geo-matched residential proxies with sticky sessions per platform (each platform's review section is often localized, and your collection IP should look like it belongs to the market you're collecting) plus polite crawl delays have been reliable for me. Thordata's session-suffix sticky IPs keep the cookie jar and exit IP aligned per platform worker.

One design decision pays for itself immediately: collect raw, normalize later. Store each platform's payload as-received (JSON blob per review), keyed by source. Platform schemas change; your downstream analysis shouldn't break when they do.

from dataclasses import dataclass, field
from datetime import datetime

@dataclass
class RawReview:
    source: str            # "amazon", "g2", "google_places", ...
    external_id: str       # platform's review id, if any
    product_id: str        # YOUR canonical product id, mapped upstream
    raw: dict              # untouched payload
    fetched_at: datetime

# Normalized view derived on the fly:
@dataclass
class Review:
    source: str
    product_id: str
    rating: float | None       # 1-5 or 1-10, see normalization
    rating_scale: int
    text: str
    language: str
    author_hash: str           # hashed, salted author identifier
    verified_purchase: bool | None
    published_at: datetime | None

Layer 2: Normalization — The Boring Part That Decides Everything

Three normalization problems account for most of the pain.

Rating scales. Amazon and Google use 1–5; G2 uses 1–10; some enterprise platforms use 1–100. Never store a bare number. Store rating and scale, and compute rating_pct = rating / scale_max for cross-platform comparisons. Also normalize semantics: on a 1–10 scale, people treat 7 as mediocre; on a 1–5 scale, 3.5 is mediocre. If you're averaging, either bucket first (see sentiment preprocessing below) or fit per-platform calibration once you have distribution data.

Identity mapping. The same product has different IDs everywhere — ASIN, Google Place ID, G2 slug, app IDs per country store. Build a product_alias table: canonical product → (source, source_product_id, market). Reviews join through it. This is also where market-awareness matters: the US App Store and the German App Store are different product surfaces with different review populations; collect them separately and route each through a proxy exit in the right country, because some storefronts gate or degrade content by request geography.

Deduplication. Reviews get copied across sites (syndicated review networks, "reviews from around the web" widgets, scraper-run aggregator sites). Dedup on three signals in order of cost: exact source ID, then (author_hash, rating, published_date) tuple match, then fuzzy — shingled hash of the normalized text. A practical recipe:

import hashlib, re

def text_shingles(text: str, k: int = 5) -> set[int]:
    words = re.sub(r"[^\w\s]", "", text.lower()).split()
    return {hash(frozenset(words[i:i+k])) for i in range(max(1, len(words) - k + 1))}

def jaccard(a: set, b: set) -> float:
    return len(a & b) / len(a | b) if a | b else 1.0

def is_duplicate(r1: Review, r2: Review) -> bool:
    if r1.source == r2.source and r1.rating == r2.rating:
        return jaccard(text_shingles(r1.text), text_shingles(r2.text)) > 0.8
    # cross-source: near-identical text with same rating = syndicated review
    return r1.rating == r2.rating and jaccard(
        text_shingles(r1.text), text_shingles(r2.text)) > 0.7

Flag duplicates rather than dropping them — keeping one canonical review plus a syndicated_from pointer preserves both clean aggregates and the (interesting!) signal of how far a review has traveled.

Layer 3: Sentiment Preprocessing — Where Models Get Fed Garbage

Here's the thing most tutorials skip: you should not feed raw review text to a sentiment model. Reviews are a distinct dialect — short, emoji-heavy, sarcasm-prone, and full of domain-specific negation patterns ("this is not bad" is 3 stars, "this is NOT good" is 1 star, and both contain "not good"-adjacent tokens). Preprocessing for reviews, in the order I apply it:

1. Language detection and routing. Multi-platform global products collect reviews in dozens of languages. Detect first (fasttext's compressed lid model is accurate and instant), and either route to per-language models or translate. Never run one multilingual model blindly — per-language routing consistently beats a single multilingual model at the same budget.

import fasttext  # pip install fasttext-wheel; model: lid.176.ftz

_lang_model = fasttext.load_model("lid.176.ftz")

def detect_language(text: str) -> str:
    text = text.replace("\n", " ").strip()
    if not text:
        return "und"
    label, _ = _lang_model.predict(text, k=1)
    return label[0].replace("__label__", "")

2. Emoji and symbol handling. Emojis in reviews are signals, not noise. Convert with the emoji library into :thumbs_up:-style tokens so the model can see them, and separately retain a small feature: emoji polarity sum.

3. Negation scope marking. Light, deterministic, and worth it: mark tokens after a negator until punctuation, so "not good, not great" becomes "not good_NEG not great_NEG". Most transformer models handle raw negation better than BOW-era models did, but explicit marking still helps on short texts.

4. Rating-text consistency filtering. The single highest-value preprocessing step for reviews specifically: a large fraction of review text carries no opinion at all ("works as described", "arrived on time", "need to use longer before reviewing") — and a meaningful minority is inverted (1-star reviews saying "great product, terrible delivery"). Filter no-opinion reviews before aggregation, and flag rating-text disagreement for separate treatment, because that's often the most actionable content in the dataset: a 5-star review complaining about shipping is a logistics problem, not a product problem.

NO_OPINION_PATTERNS = [
    r"^(ok|okay|good|fine|nice)\.?$",
    r"works as (described|expected|advertised)",
    r"(haven'?t|have not|yet to) (used|tried|tested)",
    r"^.{0,5}$",                      # "..." / "👍" alone with no text
]

import re
def has_opinion(text: str) -> bool:
    t = text.strip().lower()
    return not any(re.search(p, t) for p in NO_OPINION_PATTERNS)

5. Aspect slicing before aggregation. The aggregate question "are reviews positive?" is nearly useless. The useful question is "what specifically do people praise and complain about?" Slice by product aspect — extract aspect mentions with simple patterns or an LLM pass — and compute sentiment per aspect. "BATTERY: negative, SCREEN: positive, SUPPORT: very negative" is a report a product team can act on; an average of 4.1 is not.

Putting It Together

The full pipeline, end to end: per-platform collectors (API-first, geo-matched sticky proxies where scraping is unavoidable) writing raw payloads → product alias mapping → normalization to a common schema → three-tier dedup → language detection and routing → opinion filtering and negation marking → per-language sentiment → aspect slicing → aggregation tables.

Run it weekly for most platforms — reviews accumulate slowly enough that daily collection is wasted requests — with daily cadence only for high-volume marketplaces where a spike of negative reviews is worth catching fast. And a compliance note that's specific to reviews: review text is user-generated content with platform terms attached. Internal analysis is one thing; republishing review text is quite another. Keep the pipeline's output as aggregates and statistics, and check each platform's terms before you display anything verbatim.

Disclosure: I use Thordata's residential proxies for the geo-matched review collection described in this post. If you want to try them, they're at thordata.com, and the code **thor020* gets you 10% off.*