TL;DR:
- Remove unnecessary personal fields before storing scraped records.
- Detection and redaction are separate operations with different failure modes.
- A redaction middleware reduces data exposure; it does not establish GDPR compliance by itself.
A scraping pipeline can collect contact details even when its business task only needs organization names or product descriptions. Those details may enter logs, storage, embeddings, and model prompts before anyone notices them. The useful place to reduce exposure is immediately after extraction and before those downstream systems receive the record.
This tutorial implements a small middleware with Presidio. It uses synthetic contact information, a narrow pattern-based detector, and an explicit storage allowlist. The example is intentionally limited so you can see which entities it recognizes and which responsibilities remain outside the code.
Pipeline at a Glance
The pipeline runs acquisition, extraction, detection, redaction, validation, and storage in that order. Acquisition returns page content. Extraction selects business fields. The privacy stage then examines the selected text and removes recognized contact details before the storage stage receives anything.
Keep this boundary close to extraction. If raw HTML is logged upstream, a later redaction function cannot remove the copy already written to your logs. Treat raw captures as restricted diagnostic material when they may contain personal data, and give them a separate retention policy.
The Scrapeless Web Unlocker guide describes a URL acquisition path. Your middleware remains responsible for the field-level decisions after acquisition.
Identify Fields You Need to Keep
Define the allowed business fields before selecting a detector. If your task needs an organization name, category, and public description, do not persist contact emails merely because the page includes them. An allowlist makes that decision visible in code.
Also distinguish redaction from pseudonymization and anonymization. Removing a matched substring is a text transformation. Replacing a value with a stable token may still let records be linked. An anonymous result requires a broader assessment of whether a person can be identified from the remaining data and available context.
The pseudonymization discussion in the EDPB consultation guidelines explains why replacing an identifier alone does not settle whether a record can be attributed to a person. Whether a particular workflow has an appropriate legal basis, notice, retention policy, or transfer arrangement depends on its circumstances. The middleware below does not answer those questions.
Detect PII with Presidio
Presidio separates recognition from text transformation. Its regular-expression tutorial shows how PatternRecognizer accepts patterns with entity types and scores.
Install the analyzer and anonymizer:
python -m pip install presidio-analyzer presidio-anonymizer
This starter uses pattern recognizers directly. It does not load a language model or claim to identify names, addresses, or all phone formats. Add suitable recognizers and evaluate them against your actual language and data distribution before expanding the scope.
Redact, Tokenize, or Quarantine Records
The following program processes synthetic data, redacts two supported patterns, and writes only approved fields. The example.com email address and phone format are demonstration inputs, not scraped personal information.
import json
from pathlib import Path
from presidio_analyzer import Pattern, PatternRecognizer
from presidio_anonymizer import AnonymizerEngine
from presidio_anonymizer.entities import OperatorConfig
recognizers = [
PatternRecognizer(
supported_entity="EMAIL_ADDRESS",
patterns=[Pattern(
name="email",
regex=r"[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}",
score=0.8,
)],
),
PatternRecognizer(
supported_entity="PHONE_NUMBER",
patterns=[Pattern(
name="demo_us_phone",
regex=r"\b\d{3}-\d{3}-\d{4}\b",
score=0.7,
)],
),
]
engine = AnonymizerEngine()
allowed_fields = {"organization", "description", "source_url"}
def detect(text):
found = []
for recognizer in recognizers:
found.extend(recognizer.analyze(
text=text, entities=recognizer.supported_entities
))
return found
def clean_record(record):
cleaned = {}
for key, value in record.items():
if key not in allowed_fields:
continue
if not isinstance(value, str):
raise TypeError("This schema permits string fields only")
cleaned[key] = engine.anonymize(
text=value,
analyzer_results=detect(value),
operators={"DEFAULT": OperatorConfig("replace", {
"new_value": "<REDACTED>"
})},
).text
if any(detect(value) for value in cleaned.values()):
raise ValueError("Recognized contact data remains")
return cleaned
synthetic = {
"organization": "Example Research Lab",
"description": "Contact sample.person@example.com or 202-555-0143.",
"source_url": "https://example.com/",
"raw_contact": "sample.person@example.com",
}
result = clean_record(synthetic)
assert "raw_contact" not in result
assert "sample.person@example.com" not in result["description"]
assert "202-555-0143" not in result["description"]
Path("clean-record.json").write_text(
json.dumps(result, indent=2), encoding="utf-8"
)
print(result["description"])
The synthetic demonstration prints:
Contact <REDACTED> or <REDACTED>.
The Presidio anonymizer supports different transformations. Choose them deliberately. A reversible transformation needs key management and access rules; a stable replacement can preserve relationships that you may not want to retain.
If a required business field becomes unusable after redaction, quarantine the record for review instead of restoring the removed personal value automatically. Store a reason code without copying the sensitive substring into that reason.
Apply the Middleware to Scrapeless Output
Run the middleware after your parser maps the acquisition response into the declared record schema. Pass only the extracted fields to clean_record, rather than treating every HTML attribute as an approved business field.
For a page fetched through Scrapeless Web Unlocker, the sequence is: check the API envelope, verify page identity, parse the selected content, clean each record, and then export accepted records. A browser session follows the same storage boundary after its extraction step.
Some personal information lives outside body text. Query parameters, document names, images, and free-form metadata need their own treatment. If a URL can identify a person, do not assume it is safe merely because your current contact regex does not match it.
Validate Missed Entities and Retention Rules
A detector needs both positive and negative test cases. Positive cases check that supported personal values are removed. Negative cases check that normal product identifiers, dates, and technical strings are preserved. Include nonstandard formatting and the languages present in your real data.
The final detector call in the code confirms that its own recognized patterns are absent. It cannot detect an entity type the recognizer never supported. Document that scope so the check is not presented as a complete privacy assessment.
Keep raw captures out of routine logs. Record counts, entity types, reason codes, and schema outcomes instead of raw snippets. Define who can review quarantined records and when diagnostic captures are deleted. Apply the same controls to embeddings and model inputs, because those are additional downstream copies.
Conclusion
A practical privacy boundary starts with collecting and storing fewer fields. Add detection and redaction for the fields you still need, test the recognizers on realistic examples, and keep raw diagnostics separate. The software can enforce those decisions consistently once the decisions have been made.
FAQ
Does replacing matched emails make a dataset anonymous?
No. Other fields and contextual information may still identify a person. Anonymity depends on the complete dataset and use context.
Does this code recognize names and addresses?
No. It recognizes its configured email pattern and one demonstration phone format. Additional entity types need their own recognizers and evaluation.
Why use an allowlist as well as a detector?
An allowlist prevents unnecessary fields from reaching storage. Detection handles supported personal values inside fields that remain allowed.
Is a privacy middleware enough for GDPR compliance?
No. The workflow also needs decisions about its purpose, legal basis, transparency, retention, security, and other applicable obligations.
Only collect data you are authorized to access, and review applicable terms and privacy requirements. This article provides technical information, not legal advice.
Top comments (0)