Building a RAG Pipeline for Greenwashing Detection in Oil & Gas

Emmanuel Boakye — Sat, 30 May 2026 15:56:03 +0000

Automated Greenwashing Detection for Oil & Gas Sustainability Reporting

Stack: Python · React · Vercel | Live: claimify-esg.vercel.app

Stat	Value
Claims scored	2,203 across 10 oil & gas majors
Eval accuracy	86.7% (52/60 hand-labelled set)
Historical pledges tracked	710 from 2021 sustainability reports
Evidence corpus	290 chunks across 7 NGO sources + Guardian

Abstract

Corporate sustainability reports are structurally difficult to audit. Claims range from specific quantified commitments ("reduced Scope 1 emissions by 37% against our 2019 baseline") to vague aspiration ("we support the energy transition"). Claimify is a RAG (Retrieval-Augmented Generation) pipeline that scores each claim against a curated NGO evidence corpus rather than relying on the LLM's training weights alone. Retrieval gives the model citable, up-to-date source material for each claim; generation produces a structured verdict grounded in that material, not in generalities about the sector.

The pipeline has five stages: PDF ingestion, NLP filtering (ClimateBERT + GPT-4o structured extraction), two-stage retrieval (SBERT shortlisting + cross-encoder reranking), LLM scoring (a structured two-step prompt), and materiality adjustment (category-specific multipliers). A separate Commitment Tracker layer reuses the same retrieval stack to score 2021 pledges against 2023–2025 NGO evidence, measuring how much follow-through actually happened.

The pipeline runs entirely offline. No API calls happen at read time. The frontend reads two pre-generated JSON files, served as static files on Vercel.

Pipeline stages:

PDF download → pdfminer parse → ClimateBERT filter → GPT-4o extract → SBERT + rerank → GPT-4o-mini score → Materiality weight → React / Vercel

§01 — Background and Design Decisions

Oil and gas sustainability reports are written for multiple audiences at once: shareholder, regulator, and general public. The same company can, on the same page, disclose a precise Scope 1 reduction figure and an unfalsifiable aspiration in the next paragraph. Distinguishing between these is not a keyword problem or a sentiment problem. It requires knowing whether a claim makes a falsifiable factual assertion and, separately, whether available evidence refutes it.

Existing approaches had a shared gap: they either flagged anything containing "net-zero" or "carbon neutral" (keyword matching) or relied on journalists manually cross-referencing company statements with NGO reports. Neither approach scales, and neither produces a traceable reasoning chain. This pipeline is the automation of that intermediate layer.

The choice to use RAG rather than a prompt-only classifier is deliberate. LLM training weights are frozen at a cutoff date and do not contain the specific Carbon Tracker reports, Reclaim Finance assessments, or InfluenceMap briefings that contradict individual company claims. Giving the model that material at inference time means every verdict can be traced back to specific retrieved documents, not to generalised knowledge about the sector.

Architectural decisions and their reasons

Decision	Why we made it	What we accepted in exchange
Offline batch, not live API	All scoring runs once per ingestion cycle. The frontend reads a static JSON file. No cost per page-view, no failure modes at read time.	New reports need a full pipeline re-run. There is no live update between cycles.
Two-step LLM scoring	A single-step prompt that asked the model to weigh evidence and classify simultaneously inflated the contradicted rate whenever NGO evidence was critical, even for well-quantified claims. Separating the two decisions fixed it.	Two LLM calls per claim rather than one.
ClimateBERT before GPT-4o	A 310-page PDF produces thousands of sentences. Running GPT-4o over all of them is expensive and introduces noise. ClimateBERT drops roughly 65% of sentences before extraction, cheaply and locally.	ClimateBERT misclassifies a small fraction of climate sentences as off-topic and drops them permanently.
Company-scoped retrieval	Each corpus chunk carries an `applies_to` field listing relevant company IDs. Retrieval filters by this before cosine search so BP evidence never surfaces for a Shell claim.	Cross-company sector patterns must be duplicated into each company's scope list manually.
Materiality multipliers	A net-zero claim and a biodiversity mention with the same raw LLM score carry different reputational and legal risk. The multipliers encode this as an explicit, version-controlled domain judgement rather than leaving it implicit.	The multiplier values are not empirically calibrated. They are expert judgement and should be revisited.
gpt-4o-mini, not gpt-4o	Roughly 10x cheaper per token at comparable accuracy on structured JSON classification. The prompt examples are the primary quality driver; the model choice is a cost trade-off once the eval bar is met.	Higher error rate on edge cases in the well/weakly boundary.

Platform note: The Commitment Tracker extracts historical claims in a background subprocess. On Windows, PyTorch DLL loading fails in that context. The Tracker substitutes a keyword regex filter for ClimateBERT. The main ingestion pipeline, which runs in the foreground, keeps ClimateBERT. This is a platform limitation, not a design choice.

§02 — System Architecture

Two input streams meet at the retrieval stage.

Claims stream: Sustainability PDFs → pdfminer.six → ClimateBERT filter → GPT-4o extract → claims.jsonl + SBERT embeddings

Evidence stream: NGO sources + Guardian API → HTTP/PDF scraping with applies_to tagging → 350-word chunks / 50-word overlap → SBERT corpus embeddings → corpus_vectors.npy

Both streams converge at:

SBERT ANN search (company-scoped, k=20) — cosine similarity over applies_to-filtered corpus subset
Cross-encoder rerank (ms-marco-MiniLM-L-6-v2) — pairwise (claim, evidence) attention scoring → top-5
GPT-4o-mini scorer — CLAIM + RETRIEVED EVIDENCE in prompt → verdict grounded in docs
Materiality weight → rationales.json → React / Vite / Vercel static

Commitment Tracker branch (separate): 2021 PDFs → keyword filter → 710 pledges → same SBERT + cross-encoder retrieval → GPT-4o-mini gap scoring

§03 — NLP Pipeline: Filtering and Extraction

Before retrieval can happen, the pipeline needs claims. A 300-page corporate sustainability report typically contains thousands of sentences. Most of them are financial tables, legal disclaimers, headers, and general corporate narrative that has nothing to do with climate commitments. Running GPT-4o over all of them would be slow, expensive, and would pull in a lot of noise. Three steps handle this: sentence splitting, ClimateBERT relevance filtering, and structured claim extraction.

ClimateBERT is the gating step. It runs locally, no API call required, and processes sentences in batches of 32. In practice it drops 60–70% of a typical report. Only what passes goes to GPT-4o for extraction.

ClimateBERT relevance filter


python
# nlp/relevance_filter.py

MODEL_NAME = (
    "climatebert/"
    "distilroberta-base-climate-detector"
)
BATCH_SIZE = 32

def predict_batch(sentences: list[str]) -> list[bool]:
    tokenizer, model = get_model()
    inputs = tokenizer(
        sentences,
        padding=True,
        truncation=True,
        max_length=256,
        return_tensors="pt"
    )
    with torch.no_grad():
        outputs = model(**inputs)
    preds = torch.argmax(outputs.logits, dim=-1).tolist()
    # label 1 = climate-relevant
    return [p == 1 for p in preds]

DEV Community: Emmanuel Boakye