DEV Community

Cover image for Scraping Bayut Dubai into a Fixed 91-Column Schema (Zero Proxies, 90-Day DLD Medians & MCP)
Independent Builder
Independent Builder

Posted on

Scraping Bayut Dubai into a Fixed 91-Column Schema (Zero Proxies, 90-Day DLD Medians & MCP)

Pulling 3,771 active Dubai apartment listings every morning exposes how messy raw UAE portal feeds actually are: the same unit routinely appears 3 to 5 times under different brokerages at asking prices 15% apart, usually paired with "8%+ gross ROI" headlines that ignore 22 AED/sqft service charges and 4% DLD transfer friction.

For the Bayut pipeline, running headless browsers and burning residential proxies on individual HTML detail pages turned out to be wasted compute. Bayut's search index returns the full listing record—direct agent mobile numbers, TruCheck verification timestamps, off-plan payment milestones, and GPS coordinates—in a single sub-second HTTPS pass.

I published that extractor on Apify Store as kotaro001/bayut-com-scraper: zero residential proxies, a locked 91-column flat schema, uncapped free-tier runs, and 90-day Dubai Land Department (DLD) registered sold medians joined directly onto each row.

Flat Schema & Measured Fill Rates (50-Row Dubai Marina Benchmark)

Raw portal JSON tends to drift across buy, rent, commercial-buy, and commercial-rent verticals. Every run here outputs the exact same 91 flat columns so downstream SQL tables, pandas pipelines, and LLM tool calls never break on nested objects.

Field Group Key Columns Measured Fill Rate & Pipeline Behavior
Core Pricing & Specs priceNumeric, pricePerSqftAed, bedrooms, bathrooms, areaSqft, furnishing 50/50 on active residential runs; commercial desks under 180 sqft are isolated via isBusinessCentre: true
90-Day DLD Sold Benchmark dldMedianSoldPriceAed, dldMedianSoldPricePerSqftAed, vsDldSoldMedianPercent, dldSampleTransactionsCount, dldBenchmarkBasis Joined from Dubai Pulse open ready-sale records (90d window, trimmed below 150 or above 8,000 AED/sqft); requires dldSampleTransactionsCount >= 3 at building level before falling back to community
Velocity & Price Cuts daysOnMarket, priceChanged, previousPrice, priceDiffPercent daysOnMarket computed on first run directly from posted_at; tracks portal-recorded asking cuts
Visa & Portal Trust goldenVisaEligible, isVerified, verifiedAt, isTruBroker 50/50 on verifiedAt; goldenVisaEligible flags units at 2,000,000 AED+
Off-Plan Schedules developerName, handoverDate, paymentPlanDownPercent, paymentPlanPostHandoverPercent 4/50 to 7/50 in mixed Marina searches (populated strictly when primary off-plan listings publish payment milestones)
Broker Contacts agent1Name, agent1Phone, agent1MobileDigits, agent1Whatsapp, agencyName 50/50 on agent1Phone; agent1Whatsapp is scrubbed to null when an agency repeats one switchboard number across 80%+ of listings
DLD / Madhmoun Permit permitNumber 0/50 — Bayut's search index does not expose permit numbers, so the column stays null by design

Python Quickstart: Screen Ready Units Below 90-Day DLD Medians

pip install apify-client pandas
Enter fullscreen mode Exit fullscreen mode
import os
import pandas as pd
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])

run = client.actor("kotaro001/bayut-com-scraper").call(
    run_input={
        "listingType": "buy",
        "communities": ["Dubai Marina", "Business Bay", "JVC"],
        "completionStatus": "ready",
        "belowCommunityMedianPercent": 10,
        "goldenVisaOnly": True,
        "onlyVerified": True,
        "maxItems": 100,
    }
)

df = pd.DataFrame(client.dataset(run["defaultDatasetId"]).iterate_items())

cols = [
    "community",
    "towerOrBuilding",
    "bedrooms",
    "priceNumeric",
    "pricePerSqftAed",
    "dldMedianSoldPricePerSqftAed",
    "vsDldSoldMedianPercent",
    "dldBenchmarkBasis",
    "daysOnMarket",
    "agent1Phone",
]

# Negative vsDldSoldMedianPercent = asking PSF sits below 90-day DLD sold median
under_dld = df[df["vsDldSoldMedianPercent"] < -5.0].sort_values("vsDldSoldMedianPercent")
print(under_dld[cols].head(10).to_string(index=False))
Enter fullscreen mode Exit fullscreen mode

In vsDldSoldMedianPercent, ((pricePerSqftAed / dldMedianSoldPricePerSqftAed) - 1) * 100 measures the exact spread against registered DLD sales rather than portal-wide averages—for instance, -14.5 means the unit asks 14.5% per sqft below the 90-day DLD median for its building or community cohort.

Wiring Into Claude Code or Cursor via Apify MCP

Because all 91 columns are flat scalars and clean arrays (up to 12 photo URLs in images, total count in imagesCount), an LLM agent can filter 100 rows without context bloat. Drop the Streamable HTTP endpoint into .mcp.json:

{
  "mcpServers": {
    "bayut-dubai": {
      "type": "http",
      "url": "https://mcp.apify.com/?tools=kotaro001/bayut-com-scraper"
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

Authenticate via /mcp OAuth in Claude Code (or Authorization: Bearer <APIFY_TOKEN>), then query concrete parameters directly:

Run kotaro001/bayut-com-scraper for ready 2-bed apartments in Dubai Marina and Downtown Dubai (maxItems 60, onlyVerified true). Return units where vsDldSoldMedianPercent is below -10 and dldBenchmarkBasis is "building", sorted by daysOnMarket descending.
Enter fullscreen mode Exit fullscreen mode

Cost Math & Live Endpoints

Billing runs strictly on Pay-Per-Event with compute and retries baked in ($0 residential proxy fees, $0 detail-page fees, and free-tier row caps unlocked):

  • Free & Starter ($19/mo plan): $0.005 per actor start + $0.75 / 1,000 listings ($0.00075 per row). Apify's default $5.00 monthly free credit covers roughly 6,660 listings/month at zero out-of-pocket cost.
  • Scale ($199/mo+) & Business: $0.004 start + $0.55 / 1,000 listings.

Links to test or inspect the schema directly:

Top comments (0)