DEV Community: Pablo Martinez

Simplificando la extracción de datos de licitaciones europeas (TED): Un enfoque práctico

Pablo Martinez — Sat, 31 Oct 2026 21:17:40 +0000

Trabajar con datos de contratación pública europea a través de la plataforma TED (Tenders Electronic Daily) es fundamental para muchos proyectos cívicos y empresariales. Sin embargo, quienes han lidiado con la API oficial o los volcados masivos de datos saben que la curva de aprendizaje puede ser empinada. La complejidad de los esquemas XML/JSON y la estructura de las consultas a menudo ralentizan la fase de desarrollo y prototipado.

El desafío habitual

Supongamos un escenario común: necesitas extraer diariamente las nuevas licitaciones relacionadas con "desarrollo de software" en España y estructurarlas en tu base de datos. Configurar las llamadas precisas a la API oficial, gestionar la paginación y, sobre todo, parsear los metadatos de los contratos requiere escribir bastante código repetitivo (boilerplate).

Una alternativa rápida para estructurar datos

Para proyectos que necesitan iterar rápido y prefieren delegar la complejidad de la extracción y el parseo inicial, utilizar extractores específicos puede ser una gran ventaja.

En lugar de construir el flujo desde cero contra la API oficial, puedes utilizar herramientas que ya entregan el dato limpio. Por ejemplo, puedes echar un vistazo a este EU TED Tenders Scraper disponible como actor en Apify.

Este enfoque te permite configurar parámetros de búsqueda básicos (país, sector, fechas) y obtener directamente un JSON o CSV estructurado con la información clave de cada licitación (presupuesto, adjudicador, plazos, enlaces).

Ejemplo conceptual de consumo

Si utilizas una solución automatizada como la mencionada, la integración en tu backend (por ejemplo, usando Node.js) se simplifica enormemente, pasando de gestionar endpoints complejos a simplemente consumir un webhook o un dataset estructurado:

Scraping 50k Spanish Public Tenders (PLACSP) into Clean JSON in <26s with Apify

Pablo Martinez — Sat, 31 Oct 2026 10:36:49 +0000

Scraping 50k Spanish Public Tenders (PLACSP) into Clean JSON in <26s with Apify

Public procurement in Spain represents over €100 billion annually. The Spanish central government, 17 autonomous regions, and thousands of municipal councils publish every tender, contract award, and minor RFP on the official procurement platform: PLACSP (Plataforma de Contratación del Sector Público).

Under Spanish Law 37/2007 (Reutilización de la Información del Sector Público / RISP), this data is legally public and open. However, if you have ever tried to consume PLACSP data programmatically, you already know the engineering horror behind the curtain.

In this technical showcase, we break down why traditional web scraping fails against PLACSP, how we engineered an Apify Actor that bypasses slow government XML feeds in sub-26 seconds using a dual-tier cache strategy, and how you can query active tenders in Python or cURL for $0.02 per result.

The Engineering Problem: Why Scraping PLACSP is a Nightmare

PLACSP provides an open syndication feed (licitacionesPerfilesContratanteCompleto3.atom). On paper, it sounds simple: download Atom feeds, parse entries, load to your database. In production, five major bottlenecks break standard pipelines:

1. Legacy CODICE XML & Deep Namespaces

PLACSP feeds do not return simple REST payloads. Each entry is wrapped in complex European CODICE XML standards:

<entry xmlns="http://www.w3.org/2005/Atom"
       xmlns:cbc="urn:dgpe:names:draft:codice:schema:xsd:CommonBasicComponents-2"
       xmlns:cac="urn:dgpe:names:draft:codice:schema:xsd:CommonAggregateComponents-2"
       xmlns:cbc-place-ext="urn:dgpe:names:draft:codice-place-ext:schema:xsd:CommonBasicComponents-2">
  <cac-place-ext:ContractFolderStatus>
    <cbc:ContractFolderID>2026/00142</cbc:ContractFolderID>
    <cbc-place-ext:ContractFolderStatusCode>PUB</cbc-place-ext:ContractFolderStatusCode>
    <cac:ProcurementProject>
      <cbc:Name>Mantenimiento de infraestructura cloud</cbc:Name>
      <cac:RequiredCommodityClassification>
        <cbc:ItemClassificationCode listURI="http://simap.europa.eu/cpv">72200000</cbc:ItemClassificationCode>
      </cac:RequiredCommodityClassification>
    </cac:ProcurementProject>
  </cac-place-ext:ContractFolderStatus>
</entry>

Parsing these deep, polymorphic trees requires strict namespace management, fallback logic across multiple schema versions (cbc vs cbc-place-ext), and safe extraction of split date/time fields.

2. High Latency & HTTP 504 Timeouts

The official servers at contrataciondelestado.es are notoriously fragile under load. A single paginated Atom document containing 500 records often takes 15 to 30 seconds to transfer. Traversing 10 to 15 pages in real-time will trigger scraper timeouts or gateway errors (504 Gateway Timeout).

3. State Tracking and Elapsed Deadlines

More than 70% of entries in raw feeds are status changes (EV for Evaluation, RES for Awarded, ADJ for Adjudicated) or tenders whose bid deadline elapsed weeks ago. To get actionable leads, you must:

Filter exclusively for active status (ContractFolderStatusCode: PUB).
Parse and validate submission deadlines against UTC now().
Filter out tenders that close before your team can act.

4. GDPR & Data Sanitization (Spanish Organic Law 3/2018)

Public procurement tenders include names and personal tax IDs (DNI/NIE) of individual natural persons (sole traders and physical person representatives) inside contact tags. Publishing or processing these without explicit consent violates GDPR. A production-grade extractor must apply automated regex filters to suppress personal identifiers while preserving corporate tax IDs (CIF) and public contracting body names.

Architecture: How the Apify Actor Achieves Sub-26s Runs

To solve the speed and timeout constraints, the PLACSP Spain Public Tenders Scraper implements a dual-tier caching architecture:

[ User Request / API Call ]
         │
         ▼
[ Apify Actor: placsp-spain-tenders-scraper ]
         │
         ├──► 1. Fetch Fast Hourly Cache (< 1.5s)
         │       (50k+ normalized active dossiers from CDN / tack.devetty.es)
         │
         ├──► 2. Live Feed Head Check (< 15s)
         │       (Pulls page 1 of PLACSP Atom feed to capture tenders published in the last 60 min)
         │
         ├──► 3. In-Memory Filter Engine (< 0.2s)
         │       - CPV Division match (e.g., '72' IT, '45' Construction, '50' Maintenance)
         │       - NUTS code & Territorial matcher (e.g., 'ES30' Madrid, 'ES51' Cataluña)
         │       - Budget ceiling without VAT (e.g., < 60.000 € for minor/simplified contracts)
         │       - Deadline validation (status == 'PUB' && deadline > now())
         │       - GDPR sanitizer (strips DNI/NIE)
         │
         ▼
[ Output: Clean JSON Dataset & Apify Key-Value Store ]

Why this works:

Zero compute waste: Instead of crawling 15 legacy XML pages from government servers on every run, the Actor loads an hourly pre-crawled and normalized dataset in compressed gzip format.
Real-time accuracy: It queries only the head of the live Atom stream (page 1) to ingest contracts published minutes ago.
Execution time: Complete extraction and filtering finishes in under 26 seconds, well below Apify's default timeout limits.

Consuming the Actor: Code Examples

The Actor is deployed as a Pay-Per-Event (PPE) actor on the Apify Store at $0.02 per matching tender. You only pay for exact tenders matching your criteria—never for compute time, pagination skips, or empty results.

1. Quick cURL Execution

You can trigger a synchronous run and get JSON results directly via the Apify API:

curl -X POST "https://api.apify.com/v2/acts/devetty~placsp-spain-tenders-scraper/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "cpv_prefix": "72",
    "province_or_nuts": "Madrid",
    "nuts_prefix": "ES30",
    "max_budget": 100000,
    "status_filter": "PUB",
    "only_future_deadlines": true,
    "max_results": 20
  }'

2. Production Python Integration (Apify Client)

Here is a full Python script to stream active IT and software contracts (CPV: 72) under €60,000 in Madrid and notify your CRM or messaging webhook:

import os
from apify_client import ApifyClient

# Initialize the Apify client with your API token
client = ApifyClient(os.getenv("APIFY_TOKEN", "YOUR_APIFY_TOKEN"))

# Define search parameters for Spanish public contracts
run_input = {
    # CPV 72: IT services: consulting, software development, Internet and support
    "cpv_prefix": "72",
    # Target region: Madrid (NUTS ES30)
    "province_or_nuts": "Madrid",
    "nuts_prefix": "ES30",
    # SME focus: Contracts under €60,000 (Simplified open procedure / Art. 159.6 LCSP)
    "max_budget": 60000,
    # Only open tenders currently accepting bids
    "status_filter": "PUB",
    "only_future_deadlines": True,
    "max_results": 25,
}

print("[*] Running PLACSP scraper on Apify...")
run = client.actor("devetty/placsp-spain-tenders-scraper").call(run_input=run_input)

# Fetch results from the run's default dataset
dataset_items = client.dataset(run["defaultDatasetId"]).list_items().items
print(f"[*] Retrieved {len(dataset_items)} active tenders matching criteria:\n")

for tender in dataset_items:
    print(f"📄 Dossier ID:   {tender['expediente_id']}")
    print(f"🏢 Authority:    {tender['organo_contratacion']}")
    print(f"📌 Title:        {tender['titulo']}")
    print(f"💰 Net Budget:   {tender['presupuesto_sin_iva']:,.2f} € (Total: {tender['importe_total_con_iva']:,.2f} €)")
    print(f"⏰ Bid Deadline: {tender['fecha_limite']}")
    print(f"🔗 Direct Link:  {tender['enlace_oficial']}")
    print(f"⚖️ Attribution:  {tender['cita_legal']}")
    print("-" * 80)

Clean JSON Output Schema

Every tender returned by the Actor conforms to a predictable schema ready for ingestion:

{
  "expediente_id": "2026/AR21U/00001560E",
  "organo_contratacion": "Subdirección General de Gestión Económica y Asuntos Generales",
  "titulo": "Mantenimiento y conservación de redes de comunicaciones y servidores",
  "presupuesto_sin_iva": 33057.85,
  "importe_total_con_iva": 40000.00,
  "fecha_limite": "2026-11-18 14:00:00",
  "estado": "PUB",
  "enlace_oficial": "https://contrataciondelestado.es/wps/poc?uri=deeplink:detalle_licitacion&idEvl=w1K9...",
  "cpvs": ["72500000", "72220000"],
  "provincias": ["Madrid"],
  "nuts": ["ES300"],
  "ciudades": ["Madrid"],
  "codigos_postales": ["28071"],
  "cita_legal": "Origen de los datos: Plataforma de Contratación del Sector Público, Ministerio de Hacienda"
}

Key Fields:

presupuesto_sin_iva & importe_total_con_iva: Clean floating-point values for budget threshold alerts.
enlace_oficial: Verified deep-link resolving straight to the bidding dossier on contrataciondelestado.es.
cita_legal: Mandatory attribution notice required by Spanish Law 37/2007 (RISP) pre-populated on every record for compliance.

Common Use Cases

GovTech SaaS & Alert Bots: Build daily Slack/Telegram alerts for B2B contractors without operating your own XML infrastructure.
SME & Freelance Bidding (Art. 159.6 LCSP): Monitor simplified procedures (Procedimiento Abierto Supersimplificado) under €60k where public bodies do not require provisional guarantees or heavy financial auditing.
Market Intelligence & Price Benchmarking: Track public procurement pricing trends and monitor competitor bids across specific CPV codes and autonomous regions.

Get Started

The scraper is live and maintained on the Apify Store:

👉 Try the PLACSP Spain Public Tenders Scraper on Apify Store

Pricing: $0.02 / result event (Pay-Per-Event). Zero infrastructure overhead.

Appendix: Ready-to-Publish Community Posts

1. LinkedIn B2B Engineering Post

🚀 How we automated 50k Spanish Public Procurement Tenders (PLACSP) into clean JSON in <26s.

Public procurement in Spain exceeds €100B/year, but scraping the official Plataforma de Contratación del Sector Público (PLACSP) is notoriously painful:
- Legacy CODICE XML schemas with 5+ nested namespaces
- Slow Atom servers with frequent 504 timeouts (15-30s per page)
- 70% outdated or closed tenders
- Strict GDPR restrictions on personal DNIs

We built and published the official PLACSP scraper on Apify Store using a dual-tier cache architecture:
1. Hourly pre-crawled normalized cache (<1.5s response)
2. Live feed head check for tenders published in the last 60 minutes
3. Instant CPV (e.g. 72 IT, 50 Maintenance), NUTS territory, and budget filters (<60k€ for SMEs)
4. Full GDPR and Law 37/2007 Open Data compliance

Now available as a Pay-Per-Event Actor ($0.02/result) with direct JSON/CSV export and API integration:
https://apify.com/devetty/placsp-spain-tenders-scraper?utm_source=linkedin&utm_medium=post&utm_campaign=placsp_showcase

Full technical breakdown and Python integration code on Dev.to.

#GovTech #OpenData #WebScraping #Python #Apify #B2B

2. Twitter / X Thread

1/4 Scraping Spanish Public Tenders (PLACSP) is an engineering nightmare: massive CODICE XML feeds, 504 server timeouts, and nested namespaces. 🇪🇸💻

Here is how we solved it to extract 50k+ active contracts in <26s with Python & @apify 👇

2/4 The bottleneck: PLACSP's syndication Atom feed serves 500 records/page, taking 20s+ per page. 
Our solution: Dual-tier caching. An hourly pre-crawled normalized cache + a 1-page live diff for the last 60 mins. Response time drops from 5 minutes to 24 seconds.

3/4 Filter directly by CPV (72 IT, 45 Construction, 50 Maintenance), budget (<60k€ for simplified SME contracts), and territorial NUTS (ES30 Madrid, ES51 BCN) with 100% GDPR sanitization.

4/4 Try the Actor on Apify Store (Pay-Per-Event, $0.02/tender):
https://apify.com/devetty/placsp-spain-tenders-scraper?utm_source=twitter&utm_medium=post&utm_campaign=placsp_showcase

Code examples & schema in our Dev.to showcase! #webscraping #opendata #python

3. Tech Forum / Indie Hackers Post

Title: Building a sub-26s GovTech Scraper for Spanish Public Tenders ($0.02 Pay-Per-Event)

Hey hackers,

If you've ever explored European GovTech, you know public procurement is huge, but government APIs are stuck in 2008.

In Spain, all public contracts are published on PLACSP via Atom syndication feeds using CODICE XML schemas. We wanted to build automated alerts for SMEs targeting simplified contracts (<60k€ under Art. 159.6 LCSP), but hitting government servers live was causing constant timeouts.

We deployed an Apify Actor with a dual-cache model:
- Daily/hourly bulk crawl cached as normalized JSON.gz
- Live diff against page 1 of the Atom feed during actor startup
- Built-in filters for CPV categories, NUTS regions, and active bid deadlines
- Pay-Per-Event pricing on Apify Store ($0.02 per matching dossier)

Check out the actor here:
https://apify.com/devetty/placsp-spain-tenders-scraper?utm_source=indiehackers&utm_medium=community&utm_campaign=placsp_showcase

Would love feedback on our dual-cache approach from anyone else scraping heavy public data feeds!

How to Price UGC Deliverables: The Formula for Usage Rights, Whitelisting, and Exclusivity

Pablo Martinez — Sat, 31 Oct 2026 10:36:14 +0000

When software engineers transition into freelancing or building developer tooling, one of the earliest lessons is that you never bill intellectual property the same way you bill hourly maintenance.

In the creator economy—specifically User-Generated Content (UGC) and short-form product marketing—an astonishing number of creators make the exact opposite mistake. They quote a flat fee for producing a video (say, $150 or €150) without separating production labor from commercial distribution rights.

Two weeks later, they discover the client put $40,000 in Meta or TikTok ad spend behind their clip, running it as a dark ad across four countries while locking them out of pitching competitors in the same niche.

If you don't unbundle your deliverables, you are essentially gifting brands free media distribution rights. Here is the operational pricing framework and mathematical formula used by professional creators and production studios to price UGC deals fairly, protect cash flow, and avoid leaving revenue on the table.

1. The Core Components of a Commercial Creator Deal

A professional UGC quote consists of three distinct layers:

Base Production Rate (Asset Creation): This covers concept ideation, scripting, product handling, staging/lighting, filming, editing, color-grading, and standard captioning. This reflects pure creative labor and deliverable ownership for organic posting.
Paid Ad Usage Rights (Commercial Distribution): Organic reach is free; paid ads generate revenue. When a brand runs your asset as a Spark Ad, Dark Post, or TikTok ad campaign, they extract compounding leverage from your likeness and creative angle. Ad usage is traditionally priced as a monthly percentage of your base production rate (industry standard: 30% to 50% per 30-day window).
Exclusivity & Category Lockout: If a fintech app requires that you do not partner with or create content for competing financial products for 60 days, you are incurring a direct opportunity cost. Exclusivity must always carry a premium—typically 30% to 100% of the base fee per month depending on how narrow the niche is.
Creative Variations (Hook Testing): Performance marketing teams rarely run a single video in isolation. They need multiple hook variations (the first 3 seconds) and call-to-action (CTA) variations to test ad retention. Hooks require marginal filming time but offer massive testing utility. They should be priced at 20% to 25% of the base video rate.

2. The Pricing Formula

To keep your quoting deterministic, transparent, and defensible in client negotiations, use this formula:

$$\text{Total Quote} = (\text{Base Rate} \times V) + (H \times 0.25 \times \text{Base Rate}) + (M \times U \times \text{Base Deliverables}) + E$$

Where:

$V$ = Number of finished video deliverables
$H$ = Number of additional hook / CTA variations
$M$ = Months of paid ad usage rights
$U$ = Ad usage percentage per month (default: 30%)
$E$ = Exclusivity premium (0 if non-exclusive)

3. A Step-by-Step Example with Real Numbers

Let’s walk through a common agency brief:

Deliverable: 2 organic vertical videos (9:16) for a SaaS product
Variations: 2 additional hook variations for split testing
Usage: 30 days (1 month) of paid ad / whitelisting rights
Exclusivity: 30 days of category exclusivity in direct software competitors

The Math:

Base Production:
$$2 \text{ videos} \times €150 = €300$$
Hook Variations:
$$2 \text{ extra hooks} \times (25\% \text{ of } €150) = 2 \times €37.50 = €75$$
Subtotal Assets: €375
Paid Ad Usage (30 days):
$$1 \text{ month} \times (30\% \text{ of } €300 \text{ base deliverables}) = €90$$
Category Exclusivity (30 days):
$$30\% \text{ exclusivity premium on base deliverables} = €90$$
Final Commercial Quote:
$$€300 + €75 + €90 + €90 = €555$$

If this creator had quoted a naive flat rate of €150 per video, they would have delivered €555 worth of commercial utility for €300. By structuring the quote line by line, the client understands exactly what they are paying for: base creative labor, testing assets, media rights, and market protection.

If you want to simulate variations or test different duration windows without doing manual spreadsheets, you can test the open UGC Rate & Usage Rights Calculator to calculate quotes and upfront deposits in seconds.

4. Payment Terms: Protecting Cash Flow

Closing the deal is only half the battle. In B2B marketing, Net-30 or Net-60 invoice terms often mean waiting two months for payout, while clients occasionally default on usage expiration dates.

To protect your business:

50% Upfront Deposit: Never film or ship finals without an initial 50% deposit cleared in your bank account. In our example above, invoice €277.50 before touching a camera.
Watermarked Drafts: Deliver review cuts with a clear, semi-transparent timecode watermark. Release raw or clean files only upon balance settlement.
Usage Expiration Tracking: Include strict start and end dates for paid ad rights directly in the contract. Once day 31 hits, the brand must either pull the ad from Ads Manager or renew usage for another monthly fee.

Treating creator production as an engineering workflow—with clean formulas, structured deliverables, and strict licensing terms—turns irregular creative gigs into predictable, profitable client relationships.