Abdelrhafar Naouri

Posted on Sep 8 Originally published at soumit.ma

Structuring $30B in Government Procurement: Open Data & AI Architecture for Moroccan Public Tenders

#ai #python #data #opensource

Public procurement in emerging economies represents massive economic volume. In the Kingdom of Morocco alone, government tenders (Marchés Publics) exceed 300 Billion MAD (~$30B USD) every year, spanning high-speed rail (LGV), healthcare infrastructure, solar mega-projects (Noor), and digital transformation.

Yet, accessing and qualifying this data has historically been a major bottleneck for Small and Medium Enterprises (TPMEs) and researchers due to unstructured PDF archives, mixed bilingual documents (French & Arabic), and complex regulatory requirements.

To solve this, we built and open-sourced the Morocco Public Procurement Open Benchmark and launched the live intelligence engine at Soumit.ma.

Here is an architectural breakdown of how we extract, structure, and serve public procurement signals at scale.

🏗️ 1. The Real-World Data Challenges

Extracting government procurement data is drastically different from standard web scraping. Here is why:

┌─────────────────────────────────────────────────────────────┐
│              Raw Moroccan Procurement Stream                │
│                 (marchespublics.gov.ma)                     │
└──────────────────────────────┬──────────────────────────────┘
                               │
            ┌──────────────────┴──────────────────┐
            ▼                                     ▼
┌───────────────────────┐             ┌───────────────────────┐
│ Heterogeneous DCE ZIP │             │ Bilingual Complexity  │
│ Scanned PDFs, BPU, RC │             │ French ↔ Arabic Texts │
└───────────────────────┘             └───────────────────────┘

Challenge A: The "DCE Bundle" Problem

Tender dossiers (Dossiers de Consultation des Entreprises - DCE) are packaged as ZIP archives containing:

Règlement de Consultation (RC): Rules, deadlines, and qualification scoring.
Cahier des Prescriptions Spéciales (CPS): Technical specifications and service levels.
Bordereau des Prix / DQE: Unstructured price schedules.
Scanned raster blueprints and signed annexes: Often low-DPI scans.

Challenge B: Bilingual Arabic & French Context

Morocco's public administration operates bilingually:

Administrative and legal announcements (طلب عروض, ضمان مؤقت, دفتر التحملات) are often drafted in formal Arabic.
Technical specifications, bills of quantities, and engineering clauses are primarily in French.

Challenge C: The Regulatory Pivot (Décret n° 2-22-431)

Enacted in late 2023, Morocco's new Public Procurement Decree made electronic submission mandatory and introduced a mandatory 30% quota reservation for TPMEs (Very Small, Small, and Medium Enterprises). Extracting and validating this compliance metadata is critical for local bidders.

⚙️ 2. The Extraction & Processing Pipeline

To transform thousands of raw tenders into actionable structured records without hallucination, we adopted a multi-stage pipeline:

┌──────────────┐     ┌──────────────┐     ┌──────────────┐     ┌──────────────┐
│ Ingestion &  │ ──> │ Role-Based   │ ──> │ Deterministic│ ──> │ Edge Serving │
│ Deduplication│     │ DCE Routing  │     │ Extraction   │     │ (Cloudflare) │
└──────────────┘     └──────────────┘     └──────────────┘     └──────────────┘

Ingestion Layer: Ingests public notices from the official portal (marchespublics.gov.ma), extracting baseline metadata (reference number, contracting authority, estimated budget, submission deadline).
Role-Based DCE Classifier: Instead of feeding entire 200-page ZIPs into an LLM, a classifier identifies document roles (RC vs. CPS vs. Acte d'engagement) to target the exact clauses containing financial guarantees (cautionnement provisoire) and selection criteria.
Deterministic Financial Extraction: Critical numeric data (provisional bonds, payment terms, delay penalties) are parsed using rule-guided regular expressions paired with targeted LLM fallback validation.
Edge Delivery: Cached and served with sub-50ms TTFB across edge workers, powering live search and analytics.

📊 3. Open Datasets: Hugging Face & Zenodo (CERN)

In line with Morocco's Open Government Partnership (OGP) commitments, we released the structured open dataset to the global research community across two permanent hubs:

🤗 Hugging Face Hub: xoniques/morocco-public-tenders-open-index
🏛️ Zenodo (CERN - DOI: 10.5281/zenodo.22654978): zenodo.org/records/22654978
🗺️ Moroccan Public Buyers Directory: xoniques/morocco-public-buyers-directory
🚀 Interactive Explorer Space: hf.space/xoniques/morocco-tender-ai-explorer

Loading the Open Data in Python:

from datasets import load_dataset
import pandas as pd

# Load open dataset directly from Hugging Face
dataset = load_dataset("xoniques/morocco-public-tenders-open-index", data_files="tenders.parquet")
df = dataset["train"].to_pandas()

# Filter active healthcare tenders exceeding 10M MAD
medical_tenders = df[
    (df["sector"] == "Santé & Matériel Médical") & 
    (df["estimated_budget_mad"] >= 10_000_000)
]

print(medical_tenders[["reference_no", "buyer_name", "estimated_budget_mad", "submission_deadline"]])

Direct Analytics with DuckDB:

INSTALL httpfs;
LOAD httpfs;

SELECT 
    sector, 
    COUNT(*) AS total_tenders, 
    ROUND(SUM(estimated_budget_mad) / 1e6, 2) AS total_budget_million_mad
FROM 'https://huggingface.co/datasets/xoniques/morocco-public-tenders-open-index/resolve/main/tenders.parquet'
GROUP BY sector
ORDER BY total_budget_million_mad DESC;

🌐 4. Live Platform & Next Horizons

Structuring public procurement data is only step one. The future lies in:

Automated Bid Preparation: Generating proposal-ready checklists and compliance matrices for Moroccan TPMEs.
Predictive Pricing Intelligence: Understanding market discount rates (rabais moyen) across geographic regions and buyers.
Macro-Economic Barometers: Tracking national investment trends in real time.

🚀 Explore the live platform: Soumit.ma
📈 Read the Quarterly Procurement Barometer: Soumit.ma/tendances

What techniques have you used for structuring complex government PDF archives in bilingual environments? Let's discuss in the comments below!

DEV Community