DEV Community: Happy Endpoint

Real Estate API MCP: One Dubai Property Search Costs an AI Agent 17,000 Tokens

Happy Endpoint — Sat, 07 Nov 2026 08:50:38 +0000

One page of 1-bedroom rentals in Dubai Marina from the Bayut API is 68,709 characters of JSON. At the usual rule of thumb of four characters per token, that is about 17,000 tokens of an AI agent's context window for 25 listings, and an agent that compares three areas on two portals has spent about 105,600 tokens before it writes a word of its answer.

I wanted to know where those tokens go, because "give the agent an MCP server" is now the default answer to "how do I connect an LLM to property data". So I connected three UAE real estate APIs the way an MCP setup does, called them for a simple question, and measured everything the model would have to read.

What I measured

The question was the one people actually ask an agent: what does a 1-bedroom flat rent for in Dubai Marina, Jumeirah Village Circle and Business Bay?

On 3 November 2026 I made 18 calls through three APIs: the Bayut API, the Property Finder API and the UAE Real Estate API, which serves both portals behind one key. For each area I looked the location up on both portals, ran one page of 1-bedroom rental search on each, and pulled Property Finder's lease transactions. I also fetched one full listing record and ran the UAE Real Estate search for both portals in Dubai Marina.

Every endpoint of these APIs is reachable over MCP through RapidAPI's gateway, one tool per endpoint (setup here). A tool's result is the API's response, so I measured the responses themselves: characters of compact JSON, divided by four for a token estimate. Real counts depend on the model's tokenizer (Anthropic's token counting docs show how to get exact ones), but the comparisons below hold either way, because every number gets the same treatment.

Cost one: the tool list, before the first question

An MCP client sends the model the name, description and input schema of every tool it can call, on every turn. I built those definitions from the three public OpenAPI files the way a gateway does, one tool per endpoint, with each query parameter as a property of the tool's input schema.

API	Tools	Characters	About tokens
Bayut	17	17,828	4,457
Property Finder	18	23,886	5,972
UAE Real Estate	11	10,725	2,681
All three	46	52,439	13,110

Property Finder costs the most: it has more endpoints, and its tools average about 1,327 characters against Bayut's 1,049, with parameter descriptions making up 48.3% of the text. Those descriptions list allowed values, defaults and examples for every filter. That detail is exactly what helps a model call the tool correctly, so I would not cut it. It is the reason to load only the APIs a task needs.

The other approach is a discovery connector. Free Live Data MCP, an open-source connector we maintain, puts 48 APIs behind four tools (search_apis, search_endpoints, describe_endpoint, call_endpoint). Its four definitions are 2,146 characters, about 536 tokens. The model pays for a parameter list only when it asks for one. The trade is an extra round trip or two per task.

Cost two: the responses

For Dubai Marina, one page of Bayut rental search cost about 17,177 tokens and one page of Property Finder search about 16,324. The full Property Finder listing record was about 3,216, the lease transactions about 1,794, and Property Finder's location lookup only 470. Bayut's lookup cost about 2,023, because it returned 25 candidate locations with their full paths and names, where Property Finder returned 20 short ones.

Across the three areas the search pages were remarkably stable: Bayut between 17,098 and 19,527 tokens, Property Finder between 16,324 and 18,126. The size is a property of the format, not of the area.

Where the characters go

A Bayut search row ran to between about 2,700 and 3,100 characters, depending on the area. The biggest field is location at 28.3% of the row. It is the whole location tree, four to six levels from UAE down to the building, and every level carries its name in several languages: name, then name_l1 in Arabic, name_l2 in Chinese and name_l3 in Russian. category takes another 16.8% for the same reason: two levels, each in several languages. The agent needs one word from each.

On Property Finder, images is 37.8% of a row: image URLs with long version strings. location_tree is another 9.0% and address 7.2%.

The part I did not expect was the contact data. On Bayut, the listing agent (ownerAgent), the phoneNumber and contactName fields together are 14.1% of every row. On Property Finder, agent_details is 14.7%. An agent that forwards raw rows is paying to read people's names and phone numbers it has no use for, and putting them into a model's context and logs. Trimming fixes both problems at once.

The fix: trim before the model reads it

The agent needs about nine fields to answer a rent question. Here is the Bayut version, runnable as printed with a RapidAPI key:

import json
import os

import requests

HOST = "uae-real-estate3.p.rapidapi.com"  # Bayut API on RapidAPI
BASE = os.environ.get("API_BASE", "https://" + HOST)


def headers():
    if "API_BASE" in os.environ:
        return json.loads(os.environ.get("API_HEADERS", "{}"))
    return {"x-rapidapi-host": HOST, "x-rapidapi-key": os.environ["RAPIDAPI_KEY"]}


def cost(x):
    """Characters of compact JSON, and a rough token count at 4 characters a token."""
    n = len(json.dumps(x, ensure_ascii=False, separators=(",", ":")))
    return n, round(n / 4)


def trim(row):
    """The nine fields an agent needs for a rent question, flat, with the unit in the key."""
    return {
        "id": row["externalID"],
        "title": (row.get("title") or {}).get("en"),
        "rent_aed": row.get("price"),
        "per": row.get("rentFrequency"),
        "beds": row.get("rooms"),
        "baths": row.get("baths"),
        "area_sqm": round(row["area"], 1) if row.get("area") else None,
        "furnished": row.get("furnishingStatus"),
        "building": (row.get("location") or [{}])[-1].get("name"),
    }


# One page of 1-bedroom rentals in Dubai Marina (Bayut location id 5003)
body = requests.get(
    BASE + "/search-property",
    params={"purpose": "for-rent", "location_ids": "5003", "rooms": "1", "page": 1},
    headers=headers(),
    timeout=60,
).json()
rows = body["data"]["properties"]
small = [trim(r) for r in rows]

print("rows:", len(rows))
print("full response: %d characters, about %d tokens" % cost(body))
print("trimmed rows:  %d characters, about %d tokens" % cost(small))
print(json.dumps(small[0], indent=2))

On my run it printed 25 rows, 68,709 characters (about 17,177 tokens) for the full response and 4,907 characters (about 1,227 tokens) after trimming. The first row came out as a furnished 1-bedroom in Marina Diamond 4 at AED 70,000 a year, 62.6 square metres: everything the agent needs to answer, nothing it does not.

Across the three areas the median Bayut page went from about 17,177 tokens to 1,227, a cut of 92.9%. Property Finder went from about 17,391 to 2,460, a cut of 85.5%. Its trimmed rows keep the listing URL, which an agent needs to show a link, and that URL is long. The six search pages together went from about 105,600 tokens to about 11,250.

Where the trim runs matters less than that it runs. In your own agent it is a few lines around the tool call. With a gateway you do not control, put a thin MCP server of your own in front of it that calls the API and returns the trimmed rows. A generic gateway has no way to know which fields matter to your task.

Pitfalls I hit in the data

Pretty-printed JSON costs half as much again. The same 18 responses were 52.4% longer indented with two spaces than in compact form. Indentation is free for a human reading a log and expensive for a model reading a tool result. Serialize with separators=(",", ":").

Bayut's area filter and area field use different units. The area_min and area_max filters take square feet. The area field on the 1-bedroom rows came back as values like 62.6, with fourteen decimals, which only makes sense as square metres. Property Finder's size says its unit ({"value": 592, "unit": "sqft"}). Put the unit in the key name (area_sqm, area_sqft) so the model cannot compare the two portals in different units without noticing.

The two portals disagree on types. Bayut's rooms is a number, Property Finder's bedrooms a string ("1"). Bayut's createdAt is a Unix timestamp in seconds, Property Finder's listed_date an ISO string. A model will usually cope, but normalizing once in the trim is cheaper than hoping.

Some Property Finder search rows have no rent at all. 14 of the 113 rows across my three rental searches carried no price.value. They are new-project cards mixed into the results, and 12 of them carry a price.from that is a sale price, such as {"from": 1000000, "down_payment_percentage": 20}. Drop them before the model sees the page, or it may read a million-dirham sale price as a rent.

Location ids belong to one portal. Dubai Marina is 5003 on Bayut and 50 on Property Finder. An agent that reuses an id across portals does not get an error; it gets the wrong area. Make the agent look the place up on each portal, or keep a small table of ids for the areas you care about.

Bayut search rows have no description. description came back as {"en": null} on all 75 search rows I pulled, so an agent that wants the listing text has to fetch the full record, which costs more tokens again. For a rent question it does not need it.

What I would do next

For a single-portal agent, one API's tools plus a trim on the search results is enough: about 4,500 to 6,000 tokens of tools and about 1,200 to 2,600 per trimmed search page. For an agent that has to choose among many sources, a discovery connector keeps the fixed cost near 500 tokens.

The APIs I used are the Bayut API (free plan on RapidAPI), the Property Finder API and the UAE Real Estate API, which is also a Postman collection if you want to try the calls before writing code. Each portal has its own site with agent setup notes: bayutapi.dev for Bayut, with a function-calling guide, and uaepropertyapi.com for Property Finder, with MCP setup for Claude, Cursor and ChatGPT. For what the Bayut data itself covers, see the Bayut API overview on LinkedIn.

Building an agent on UAE property data and want the trimmed version as a service, or a dataset instead of calls? Get in touch.

Scraping H&M Data with Python: Do Women Pay More? The Data Says Men Do

Happy Endpoint — Wed, 04 Nov 2026 09:46:27 +0000

On H&M's US site, the median men's T-shirt style lists at $24.99. The median women's T-shirt style lists at $19.99. For hoodies the gap is $44.99 against $29.99.

That is the opposite of what I expected. The "pink tax" (women paying more for comparable products) is well documented: a 2015 New York City study of 794 products found women's versions cost about 7% more on average, and 8% more for adult clothing. So I scraped H&M's catalogue with Python to see whether a fast-fashion retailer follows the pattern. For jeans, T-shirts and hoodies, it does not. Men pay more for all three, and the reason turns out to be the shape of the range more than the price tags.

The data

I used the H&M API, which returns H&M's search results as JSON. All of it came from a single endpoint, /product-search, which I called with a search word (jeans, t-shirt, hoodie), languageCode=en_us for the US site, pageSize=60, and a filters string that pins two of H&M's own facets: customerGroups (Woman or Man) and productTypes. I paged through every result for all six combinations on 24 October 2026: 37 pages, 2,114 listings.

A listing is one colour of one garment. Its id is a 10-digit article number, and the first 7 digits identify the style, so the 2,114 listings are 818 styles. Each listing has prices (a whitePrice, which is the regular price, and a redPrice when it is on sale), mainCatCode (H&M's internal category, such as men_jeans_baggy), and sizes, where every size carries a stock value.

The code

To run it, put a key from RapidAPI (there is a free plan) into RAPIDAPI_KEY and install requests and pandas. Each page is saved to hm_cache/ the first time, so the 37 calls are spent once and re-runs are free.

import json
import os
import re
from pathlib import Path

import pandas as pd
import requests

HOST = "h-m-hennes-mauritz1.p.rapidapi.com"
CACHE = Path("hm_cache")
TYPES = {"jeans": "Jeans", "t-shirt": "T-shirt", "hoodie": "Shirt"}  # H&M files hoodies under "Shirt"


def search(keyword, group, page):
    """One page of 60 results, cached on disk so a re-run costs no calls."""
    file = CACHE / f"{keyword}_{group}_p{page:02d}.json"
    if file.exists():
        return json.loads(file.read_text())
    r = requests.get(
        f"https://{HOST}/product-search",
        params={"query": keyword, "languageCode": "en_us", "pageSize": 60, "page": page,
                "filters": f"customerGroups:{group}||productTypes:{TYPES[keyword]}"},
        headers={"x-rapidapi-host": HOST, "x-rapidapi-key": os.environ["RAPIDAPI_KEY"]},
        timeout=90,
    )
    r.raise_for_status()
    CACHE.mkdir(exist_ok=True)
    file.write_text(r.text)
    return r.json()


rows, sizes = [], []
for keyword in TYPES:
    for group in ["Woman", "Man"]:
        page, last = 1, 1
        while page <= last:
            data = search(keyword, group, page)
            last = data["pagination"]["totalPages"]
            for p in data["searchHits"]["productList"]:
                price = {x["priceType"]: x["price"] for x in p["prices"]}
                rows.append({"type": keyword, "group": group, "id": p["id"],
                             "style": p["id"][:7],  # article id = style + colour
                             "name": p["productName"], "category": p["mainCatCode"],
                             "regular": price["whitePrice"], "sale": price.get("redPrice")})
                for s in p["sizes"]:
                    sizes.append({"id": p["id"], "label": s["label"], "out": s["stock"] == 0,
                                  "on_sale": "redPrice" in price})
            page += 1

df = pd.DataFrame(rows).drop_duplicates(["type", "group", "id"])
sz = pd.DataFrame(sizes).drop_duplicates(["id", "label"])

# Like for like: drop polos, multipacks, sportswear and maternity (and stray non-tees)
skip = (df["category"].str.contains("polo|multipack|maternity|^sportswear")
        | df["name"].str.contains(r"polo|\d-pack", case=False)
        | ((df["type"] == "t-shirt") & ~df["category"].str.contains("tops|tshirt")))
core = df[~skip]
styles = core.groupby(["type", "group", "style"])["regular"].median()
medians = styles.groupby(["type", "group"]).median().unstack()
medians["men_pay_more_%"] = ((medians["Man"] / medians["Woman"] - 1) * 100).round(1)
print(f"{len(df)} articles, {df['style'].nunique()} styles, {len(core)} kept for the comparison")
print(medians.to_string())

print(f"\nSize slots with stock 0: {sz['out'].mean():.1%} overall, "
      f"{sz[sz['on_sale']]['out'].mean():.1%} on sale items")
tops = sz[sz["id"].isin(df.loc[df["type"] != "jeans", "id"])]
print(tops.groupby("label")["out"].mean().reindex(["XS", "S", "M", "L", "XL", "XXL"]).round(3).to_string())

Two choices matter. First, the comparison is per style, not per listing: a women's basic tee that comes in 20 colours would otherwise count 20 times and drag the women's median down. Every colour of a style had the same price in this data, so the style price is unambiguous. Second, I left out what is not comparable across the two groups. The men's "T-shirt" results include 183 polo shirts, 91 sportswear tops and 34 multipacks; the women's results include maternity jeans and tops. That leaves 1,685 listings in 657 styles.

The first lines it prints:

2114 articles, 818 styles, 1685 kept for the comparison
group      Man  Woman  men_pay_more_%
type
hoodie   44.99  29.99            50.0
jeans    49.99  44.99            11.1
t-shirt  24.99  19.99            25.0

Men pay more for all three

The gap holds however I cut it. Per listing instead of per style, it gets wider: men's T-shirts $22.99 against $14.99, jeans $49.99 against $39.99. The most common price point tells the same story for jeans ($49.99 for men, on 77 listings; $39.99 for women, on 200) and hoodies ($49.99 against $24.99). The one exception is the T-shirt: $19.99 is the single most common price in both ranges.

Garment	Group	Listings	Styles	Median style	Cheapest	Styles under $20	On sale
T-shirts	Women	363	143	$19.99	$4.99	72.7%	5.8%
T-shirts	Men	348	117	$24.99	$7.99	25.6%	8.6%
Hoodies	Women	31	13	$29.99	$19.99	7.7%	0%
Hoodies	Men	114	50	$44.99	$24.99	0%	19.3%
Jeans	Women	662	286	$44.99	$19.99	0.7%	6.6%
Jeans	Men	167	48	$49.99	$34.99	0%	8.4%

The hoodie number needs a warning. The women's search returned only 33 hoodies, 13 styles once sportswear is out, so a few styles move that median. T-shirts and jeans have hundreds of styles on each side and are the numbers I would trust.

Why: women get a cheap tier, men get multipacks

This is not H&M charging men more for the same garment. The two ranges contain different garments, and I found no product sold in both that I could price both ways. What differs is the price ladder each shopper is offered.

The women's T-shirt range has a dedicated basics category, ladies_basics_tops_shortsleeve, with 46 styles at a median of $12.99 and the cheapest tee in the data, a "Fitted T-shirt" at $4.99. Nearly three in four women's T-shirt styles cost under $20; for men it is one in four. The men's results have no basics category at all. Men's cheap tees are sold in packs instead: 34 of the men's T-shirt listings are multipacks (not counting packs of polos), such as a "5-pack Regular Fit T-shirts" at $39.99, which is $8 a shirt. Buy the pack and the men's price per shirt drops to women's basics levels. Buy one shirt and it does not.

Jeans show the other half of the story: choice. Women get 286 styles of jeans, men get 48, six times fewer. The women's range reaches down to $19.99; the cheapest men's pair is $34.99. A wider range fills more price points, and most of the extra women's styles sit at the lower ones.

A third of all sizes are already gone

Every size in the response has a stock value. I only ever saw two: 2 and 0. Of 16,673 size slots across all 2,114 listings, 34.3% were 0. I read 0 as "not available in that size", which fits two checks: no listing in the data had every size at 0 (a fully sold-out item drops out of search), and the share of zeros rises sharply on sale items.

Small sizes are missing most often. In tops, S is the most often missing size (37.5% of slots), then XS (35.8%) and M (34.6%), while XXL is the easiest to find (22.0%). The edges of the range are thinner still: women's XXS was out on 70.1% of its 67 slots and men's 3XL on 51.7% of 174. Women's jeans follow the same slope, from 54.7% missing in size 0 to 23.3% in size 18 (the few size 24 and 26 slots are thin again).

Stock levels like these say nothing about demand on their own: a size can be missing because it sold well or because fewer were made. But if you are scraping H&M to track availability, sizing is where the signal is.

On sale, three in four sizes are gone

Only 189 of the 2,114 listings were on sale (redPrice present), but their sizes are mostly gone: 75.5% of size slots at 0, against 30.0% at full price, and 172 of the 189 had at least half their sizes missing. The discounts are deep, with a median of 40% to 52% off depending on the garment, but they are clearance of broken size runs rather than offers on the full range.

Men's listings were on sale more often in every garment: 8.6% of the men's T-shirts in the comparison against 5.8% of the women's, 19.3% of men's hoodies against none of the women's, and 8.4% of men's jeans against 6.6%. So the men's premium shrinks a little for anyone who shops the sale and wears whatever size is left.

What tripped me up in the data

There is no "Hoodie" product type. Filtering productTypes:Hoodie returned zero results for both groups. The facets in that empty response showed where the hoodies were: H&M files them under Shirt (125 men's, 33 women's). Read the facets block before you trust a filter value that seems obvious.

isMultipack is always false. All 47 listings named "3-pack", "5-pack" and so on had isMultipack: false, and so did every other listing. Catch packs by name or by mainCatCode (men_tshirtstanks_multipacks), and divide by the pack size before you compare anything.

stockState is always "Available". Every one of the 2,114 listings said so, including those with most sizes at 0. Availability lives in sizes[].stock, not in availability.stockState.

A listing is a colour, not a product. One style had 26 colour listings. If you count listings, your "most common price" is really the price of whichever style comes in the most colours. Group by the first 7 digits of id first.

The price facet reports sale prices. The priceRange facet for women's jeans said the range starts at $8.49, but no women's jeans had a regular price below $19.99. In all six searches the facet minimum matched the lowest current price, which is the redPrice wherever anything was on sale. Use whitePrice for list prices and redPrice for what a shopper pays today.

A "T-shirt" search is not only T-shirts. The men's productTypes:T-shirt results were 28% polo shirts by count, and the women's results included a few jumpers and maternity bottoms. mainCatCode is the more reliable label: it is H&M's own category path and it is present on every listing.

Sizes are not comparable across groups. Men's jeans use waist and length pairs (32/32), 42 distinct labels in all and a median of 18 sizes per listing; women's jeans mostly use US numbers (0 to 26), with a median of 11 sizes. Compare availability within a sizing system, not across them.

One listing appeared twice. The men's T-shirt search reported 657 hits and returned 656 distinct ids across its pages. Deduplicate on id after paging.

What I would do next

Three things. Run the same six searches weekly, which would show whether the men's sale share is a constant or a moment in the season. Repeat it in other markets through languageCode (en_gb, de_de), to see whether the women's basics tier exists everywhere. And call /supplier-details for a sample of styles from each side, to see whether the cheap women's basics and the men's multipacks come from different factories.

The data came from the H&M API. I wrote about the same kind of catalogue data for beauty retail in a LinkedIn article on Sephora product and price data, and for a custom fashion dataset, get in touch.

Realtor.com API in Python: Are New Homes Really Pricier per Square Foot?

Happy Endpoint — Tue, 03 Nov 2026 08:12:29 +0000

New single-family homes in Austin list at a median of $218 per square foot. Resale homes in the same Realtor.com search ask $304. On those two numbers alone, a new build is 28% cheaper than a used one, which is not what most buyers walking through a model home would expect.

I wanted to know whether that holds up, so I pulled new construction and resale listings for nine US metros with Python and compared them three ways: by city, by size band and inside the same ZIP code. The short version: the city numbers lie, and the reason they lie is the most useful thing in the data.

The data

Everything here comes from the Realtor.com Data API, which returns Realtor.com search results as JSON. On 24 October 2026 I ran three searches for each of nine cities: Austin, Houston, Phoenix, Charlotte, Atlanta, Nashville, Denver, Jacksonville and Raleigh.

New construction: /properties-for-sale with property_type=single_family, new_construction=true and hide_homes_not_yet_built=true, so only homes that exist or are being built, with a real address.
Resale: the same search with hide_new_construction=true instead.
Floor plans: one page of /new-construction, which turned out to be a different thing entirely (more on that below).

Both listing searches were sorted newest and I took two pages of 50, so each city contributes its 100 most recently listed homes of each kind. That is 2,195 rows from 45 search requests. After removing duplicates and listings without a price or square footage, 855 new homes and 847 resale homes were left. This is a snapshot of fresh inventory, not the whole market: the same searches report 6,735 new and 34,302 resale single-family listings in total across the nine cities.

The code

This is the core of the analysis. It runs as printed with requests and pandas on Python 3.9 or later, once RAPIDAPI_KEY is set in your environment. Every response is cached in cache/, so you pay for each page once and can rerun the analysis for free.

import json
import os

import pandas as pd
import requests

HOST = "realtor-com-data-api.p.rapidapi.com"
HEADERS = {"X-RapidAPI-Key": os.environ["RAPIDAPI_KEY"], "X-RapidAPI-Host": HOST}
METROS = ["Austin, TX", "Houston, TX", "Phoenix, AZ", "Charlotte, NC", "Atlanta, GA",
          "Nashville, TN", "Denver, CO", "Jacksonville, FL", "Raleigh, NC"]
BASE = {"property_type": "single_family", "sort": "newest", "limit": 50}
GROUPS = {
    "new": dict(BASE, new_construction="true", hide_homes_not_yet_built="true"),
    "resale": dict(BASE, hide_new_construction="true"),
}
os.makedirs("cache", exist_ok=True)


def search(name, params):
    """One page of /properties-for-sale, cached on disk so reruns are free."""
    path = os.path.join("cache", name + ".json")
    if not os.path.exists(path):
        r = requests.get("https://%s/properties-for-sale" % HOST,
                         headers=HEADERS, params=params, timeout=60)
        r.raise_for_status()
        with open(path, "w") as f:
            json.dump(r.json(), f)
    with open(path) as f:
        return json.load(f)["data"]


rows = []
for metro in METROS:
    slug = metro.lower().replace(", ", "-").replace(" ", "-")
    for group, params in GROUPS.items():
        for page in (1, 2):
            p = dict(params, location=metro)
            if page > 1:
                p["page"] = page
            for home in search("%s_%s_p%d" % (group, slug, page), p)["results"]:
                rows.append({
                    "metro": metro,
                    "group": group,
                    "property_id": home["property_id"],
                    "zip": home["location"]["address"]["postal_code"],
                    "price": home["list_price"],
                    "sqft": home["description"]["sqft"],
                })

df = pd.DataFrame(rows).drop_duplicates(["metro", "group", "property_id"])
df = df[(df.price > 0) & (df.sqft >= 400)]
df["ppsf"] = df.price / df.sqft

table = df.pivot_table(index="metro", columns="group", values="ppsf", aggfunc="median")
table["premium_%"] = (table.new / table.resale - 1) * 100
print(table.round(1).sort_values("premium_%"))

z = df.groupby(["metro", "zip", "group"]).ppsf.agg(["median", "count"]).unstack("group")
z = z[(z["count"]["new"] >= 3) & (z["count"]["resale"] >= 3)]
zip_premium = (z["median"]["new"] / z["median"]["resale"] - 1) * 100
print("%d ZIP codes, new dearer in %d, median premium %.1f%%"
      % (len(zip_premium), (zip_premium > 0).sum(), zip_premium.median()))

Two lines in there carry most of the weight. drop_duplicates on property_id matters far more in one city than you would guess, and sqft is the living area; the response also has lot_sqft, and dividing by that gives you a number that means nothing. The longer script I used for the charts and the data checks adds the floor plans, the size bands and the counts quoted in the pitfalls section.

City by city, new homes look cheap

Pooled across all nine cities, the median new home asks $239 per square foot and the median resale home $258, so new construction comes out 7.1% cheaper. City by city, new homes are cheaper per square foot in five of the nine.

Metro	Resale $/sqft	New $/sqft	Gap	Gap at 1,500 to 2,500 sqft	New share
Austin, TX	$304	$218	-28.1%	-24.8%	19.8%
Raleigh, NC	$254	$195	-23.0%	-14.0%	24.7%
Nashville, TN	$300	$273	-8.8%	+1.8%	18.7%
Phoenix, AZ	$286	$273	-4.3%	-11.0%	11.8%
Charlotte, NC	$228	$222	-2.3%	-4.0%	17.7%
Jacksonville, FL	$185	$192	+3.4%	-6.0%	20.5%
Houston, TX	$173	$194	+11.9%	+17.7%	18.6%
Atlanta, GA	$226	$265	+17.2%	+3.7%	9.8%
Denver, CO	$376	$461	+22.5%	+24.4%	5.0%

The spread is wide. Austin and Raleigh new homes ask roughly a quarter less per square foot than resale. Denver and Atlanta go the other way by about a fifth. A national "new construction premium" averaged over these cities would hide all of it.

Is it just size?

The obvious objection: price per square foot falls as homes get bigger, and new homes are bigger. They are. The pooled median new home has 2,258 sq ft against 1,950 for resale, and in Atlanta the gap is 3,251 against 1,925.

So I reran the comparison using only homes between 1,500 and 2,500 sq ft in both groups. New homes were still cheaper per square foot in five of nine cities. Austin barely moves (from -28.1% to -24.8%). Nashville flips to slightly more expensive, Jacksonville flips to slightly cheaper. Size explains some of the gap in some cities. It does not explain Austin or Raleigh.

It is location

What the city-level comparison really measures is where builders can find land. The location search covers the whole city, so a new home on the edge of town is compared with a resale home in an older, closer-in neighborhood. To take that out, I compared new and resale homes only inside the same ZIP code, keeping ZIPs with at least three of each.

That leaves 54 ZIP codes. In 31 of them, new homes ask more per square foot than resale homes. The median premium is 6.1%, and 15 of the 54 ZIPs show a premium above 20%.

The concentration is striking once you look at it. In Raleigh, 61 of the 99 new homes in my sample sit in a single ZIP code, 27616. In Jacksonville, 36 of 99 are in 32218, and in Austin 34 of 100 are in 78747. Those are the places where tract builders are working, and they are not the places the median resale listing comes from. Inside 32218, new homes ask 21.0% more per square foot than resale. In Charlotte's 28278, where 20 of the new homes are, the premium is 30.4%.

So the honest answer to the title is: yes, a new home usually costs more per square foot than a comparable used home nearby. It just tends to be built somewhere cheaper, and a citywide median mixes the two effects.

Where builders build

Every search also returns a total, the number of matching listings, which costs nothing extra and shows how different these markets are. Dividing the new-construction total by the new plus resale totals, new construction is 24.7% of single-family homes for sale in Raleigh, 20.5% in Jacksonville and 19.8% in Austin, but only 9.8% in Atlanta and 5.0% in Denver.

The two cities where new construction is rarest, Denver and Atlanta, have the two largest premiums, and two of the three where it is most common, Raleigh and Austin, have the two largest discounts. Denver looks like infill: its resale homes have a median year built of 1955, and it produced 11 of my 54 matched ZIP codes, more than any other city, so new and old homes share neighborhoods. Its new homes have a median of 3,212 sq ft and a median list price of $1,412,000, against 2,080 sq ft and $697,500 for resale. Large new houses in established neighborhoods are expensive per foot, and the city median shows it.

Pitfalls I hit in the data

This is the part I would want to read before running the script myself.

/new-construction is mostly floor plans, not houses. All 395 rows from that endpoint had status set to ready_to_build and the is_plan flag set. None of them had a year_built or a lot_sqft, and 25 had no price at all. Because I called it without a property_type, 101 of the 395 were not single-family (townhomes, condos and multi-family plans). A plan is a design a builder offers, so treat its price as an asking price for a design, not for a home on a lot. For comparisons with resale, the new_construction=true filter on the for-sale search, with hide_homes_not_yet_built=true, gives you homes with addresses. Where both existed, single-family plan prices per square foot were within 5% of the built new homes in six of the eight cities that had plans.

Builder listings look different from MLS listings. 60.9% of the new homes had no mls_status, and every one of those had a community advertiser: the builder's community listed them, not an agent through the MLS. These are thinner records: year_built was empty for 46.6% of new homes and set to a year after 2026 for 8.3% (homes still being built). lot_sqft was missing for 67.7% of new homes against 1.1% of resale homes, and only 33.6% of new homes carried a value estimate against 86.6% of resale. Any model that leans on lot size or estimates will quietly drop most new construction.

Atlanta lists the same house twice. I removed 78 duplicate property_id values, and 64 of them were in Atlanta (28 new, 36 resale). Looking up one of them with /property-details showed why: its property_history has two "Listed" events on 23 October 2026 at the same $299,900, one from GeorgiaMLS and one from FMLS. Atlanta listings can be entered in both of those MLSs, and a home entered in both comes back from search as two listings with two listing_id values. Without deduplicating on property_id, Atlanta's medians are built on a sample where a third of the rows are counted twice. Most of the remaining duplicates appeared on both page 1 and page 2, which is what happens when new listings arrive between two requests on a newest-first sort.

The search city is not the address city. 8.7% of all rows had an address city different from the city I searched, such as Manor or Pflugerville in the Austin search. Realtor.com's location search follows its own area boundary, so if you need municipal limits, filter on location.address.city yourself, or use postal_code.

Newest-first brings in coming-soon and pending homes. 76 resale and 18 new homes in the sample were flagged is_coming_soon, and 29 were pending or contingent. I kept them, because they still carry an asking price, but checked the result without them: new construction is still cheaper per square foot in five of nine cities, and Denver's premium moves from 22.5% to 19.6%.

Search results have no price per square foot. /property-details returns a price_per_sqft field (184 for the Atlanta home above, which is $299,900 divided by 1,632 sq ft, rounded), but search results do not, so you compute it yourself. Check that sqft is present first: 10 new and 10 resale homes had none.

What I would do next

Two things would sharpen this. First, run the same comparison on /properties-sold to see whether the gap survives at closing, since asking prices on builder listings and on MLS resale listings may not move the same way. Second, swap the ZIP code for a radius search with lat, lon and radius_miles around each new-home community, which compares a new home with the resale homes a buyer would actually tour instead of with a postal boundary.

Every parameter and field in this post, plus the rest of its 37 endpoints, is listed in the Happy Endpoint reference for this API. Keys come from its RapidAPI subscription page; you can try each call in a browser there before writing any code. If you need this pull for every metro on a schedule, get in touch.

Scraping IKEA Data with Python: the Same Products Cost 37% More in France Than in the Netherlands

Happy Endpoint — Mon, 02 Nov 2026 06:31:35 +0000

The white LACK side table, article 30449908, costs €9.99 in Germany, France, Spain, Austria and Belgium. In Portugal the same article number costs €12.99. Same product, same currency, a 30% gap, and no exchange rate to blame.

I wanted to know whether that was a quirk of one cheap table or a pattern, so I built a small IKEA price index in Python: the same article numbers, priced in nine euro-zone countries on the same day. It is a pattern. Not one of the 72 articles I could match had the same price in all nine countries, and on average the French site charges 37% more than the Dutch one for identical items.

The data

I used the IKEA Pro API, which returns IKEA's catalogue as JSON for a given countryCode. Everything here comes from one endpoint, /product-search-by-keyword: five IKEA families (BILLY, KALLAX, POÄNG, MALM, LACK) searched in nine euro-zone countries (Germany, France, Italy, Spain, the Netherlands, Austria, Belgium, Ireland, Portugal), first page of results only, all collected on 24 October 2026. That is 45 requests.

Each result carries an id (IKEA's article number), name, typeName, details.designText (the colour or finish), price.currentPrice, and a variants list where every colour variant has its own id and price. Counting the variants, those 45 searches returned 1,220 distinct article numbers. Only 72 of them showed up in all nine countries, and those 72 are the basket.

Sticking to the euro zone was deliberate. Once you compare euros with kronor or dollars, half of any gap you find is the exchange rate on the day you picked. Within the euro zone, a price difference is a price difference.

The code

You need a RapidAPI key (the API's RapidAPI page has a free tier) in the RAPIDAPI_KEY environment variable, plus requests and pandas. The script caches every response on disk, so it spends 45 calls once and nothing after that.

import json
import os
from pathlib import Path

import pandas as pd
import requests

HOST = "ikea-api-pro.p.rapidapi.com"
CACHE = Path("ikea_cache")
COUNTRIES = {"de": "en", "fr": "fr", "it": "it", "es": "en", "nl": "en",  # code: language
             "at": "de", "be": "en", "ie": "en", "pt": "en"}  # language changes text, not prices
KEYWORDS = ["billy", "kallax", "poang", "malm", "lack"]


def search(keyword, country):
    """Page 1 of a keyword search, cached on disk so a re-run costs no calls."""
    file = CACHE / f"{country}-{keyword}.json"
    if file.exists():
        return json.loads(file.read_text())
    r = requests.get(
        f"https://{HOST}/product-search-by-keyword",
        params={"keyword": keyword, "countryCode": country,
                "languageCode": COUNTRIES[country]},
        headers={"x-rapidapi-host": HOST,
                 "x-rapidapi-key": os.environ["RAPIDAPI_KEY"]},
        timeout=60,
    )
    r.raise_for_status()
    CACHE.mkdir(exist_ok=True)
    file.write_text(r.text)
    return r.json()


rows = []
for keyword in KEYWORDS:
    for country in COUNTRIES:
        for p in search(keyword, country)["data"]["products"]:
            # The listed product plus its colour variants, each with its own article number
            items = [(p["id"], p["details"]["designText"], p["price"])]
            items += [(v["id"], v["designText"], v["price"]) for v in p.get("variants") or []]
            for article, colour, price in items:
                rows.append({"country": country, "id": article, "name": p["name"],
                             "type": p["typeName"], "colour": colour,
                             "price": price["currentPrice"]})

df = pd.DataFrame(rows).drop_duplicates(["country", "id"])
wide = df.pivot_table(index="id", columns="country", values="price", aggfunc="first")
basket = wide.dropna()  # article numbers priced in all nine countries
rel = basket.div(basket.median(axis=1), axis=0)  # 1.0 = that article's median price
index = (rel.mean() * 100).round(1).sort_values()

print(f"{len(wide)} article numbers seen, {len(basket)} priced in all 9 countries")
print(index.to_string())
print("Cheapest (or tied) for this many articles:")
print(basket.eq(basket.min(axis=1), axis=0).sum().sort_values(ascending=False).to_string())

Two decisions in there matter more than the rest. The join key is the article number, never the name, for reasons the pitfalls section makes painfully clear. And the index treats every article equally: each price is divided by that article's median across the nine countries, then averaged per country, so a €179 shelving unit does not drown out a €4.99 wall shelf. An index of 100 means "typical euro-zone price".

The first lines it prints:

1220 article numbers seen, 72 priced in all 9 countries
country
nl     84.2
de     91.6
pt     97.2

France is the dearest IKEA in the euro zone, the Netherlands the cheapest

The Netherlands comes out at 84.2 and France at 115.2. Germany is the second cheapest at 91.6, and the other six countries sit within about five points of 100. Buying one of each of the 72 articles would cost €2,779.89 on the Dutch site and €3,835.46 on the French one, 38% more.

The ranking is not driven by a few outliers. The Netherlands was the cheapest country, or tied for cheapest, on 51 of the 72 articles and the dearest on none. France was the dearest, or tied, on 34. Ireland was never the cheapest country for a single article.

Country	Index	Index without VAT	One of each (72)	Cheapest or tied	Dearest or tied
Netherlands	84.2	84.5	€2,779.89	51	0
Germany	91.6	93.4	€2,945.44	29	5
Portugal	97.2	95.8	€3,255.74	12	8
Italy	102.3	101.7	€3,341.20	2	6
Spain	103.6	103.9	€3,325.39	8	13
Belgium	104.0	104.3	€3,443.39	5	6
Ireland	104.2	102.8	€3,334.00	0	10
Austria	105.3	106.5	€3,542.43	3	14
France	115.2	116.6	€3,835.46	1	34

The family matters too. France's premium is mostly KALLAX: on the 34 KALLAX articles in the basket its index is 125.1, against 105.3 on LACK. LACK is where Spain looks expensive (111.9) and Germany cheap (84.7). The Netherlands is the cheapest country on BILLY and KALLAX; on LACK, Germany edges it, 84.7 to 85.6.

Almost nothing has one euro-zone price

For each article I compared the dearest country with the cheapest. The median gap is 50.7%. Only 2 of the 72 articles stay within 25%, 43 differ by more than 50%, and 13 by 100% or more. The widest is a KALLAX insert with door: €7.50 in Portugal, €20 in France.

The LACK side table from the opening is a good example of how small these numbers can look while still being large in percentage terms: €8.99 in the Netherlands, €9.95 in Italy, €9.99 in five countries, €12 in Ireland and €12.99 in Portugal.

VAT does not explain it

The obvious objection is tax. EU shelf prices for consumers include VAT (the price indication directive defines the selling price as the final price including VAT), and the standard rates in these nine countries run from 19% in Germany to 23% in Ireland and Portugal (standard rates on the EU's Your Europe site).

So I divided every price by one plus its country's standard rate and rebuilt the index. Nothing important moved. France goes from 115.2 to 116.6 and the Netherlands from 84.2 to 84.5. The largest shift is Germany's, from 91.6 to 93.4, because it has the lowest VAT of the nine. A spread of 19% to 23% in tax cannot produce a 37% gap in prices, and France's 20% rate is in the middle of the range anyway.

What tripped me up in the data

The most famous bookcase is not in the basket. You would expect the white BILLY to be the anchor of any IKEA price index. It never made the 72, because Europe sells it under two sets of article numbers. Germany, the Netherlands, Austria, Belgium and Ireland list 00263850 (from €42.49 in the Netherlands to €65 in Ireland) and its siblings. France, Italy, Spain and Portugal list 50522040 (€44.99 to €49.99) and three siblings, the same four numbers the US site lists. This is not page-one noise: searching the exact article number 00263850 returns one result in Germany and zero in France, and 50522040 returns zero in Germany. The search response has no dimensions field, so I cannot tell from the data whether the two white bookcases are identical apart from the number; you can compare the German and French product pages yourself. Either way, joining on the number quietly drops BILLY white, and joining on the name compares two different articles.

Names are not keys. In Germany, page one of the KALLAX search has eight different article numbers that are all "KALLAX, Shelving unit, white", priced from €16.99 to €129. They are different sizes. Across the nine countries there were 101 such groups where name, type and colour were identical but the article numbers were not. Anything that matches on text will happily compare a small shelf with a large one.

Combinations have their own numbers, and they are local. IDs starting with s are combinations: a POÄNG armchair is a frame plus a cushion sold as one number. Of the 1,220 numbers I saw, 724 were combinations, and only 5 survived into the basket. The same POÄNG armchair and footstool in walnut effect with a Kelinge beige cushion is s29617733 in Germany, the Netherlands and Portugal (€149, €168 and €170) and s59617736 in Ireland (€199). POÄNG contributed nothing to the index for that reason.

Search is not a name filter. Page one of the "malm" search in Germany had 22 results: 5 named MALM and 10 named STORKLINTA. In Belgium only 2 of 22 were MALM. The total result count for the same keyword ranged from 75 in France to 615 in Germany. MALM contributed nothing either. If you want a specific article, search its number.

currentPrice is all you get, so promotions look like prices. The search response has a current price and no regular price. Seven of the nine Dutch BILLY prices in the basket end in .49 (€25.49, €42.49), and €42.49 is exactly 85% of €49.99, which looks like a 15% markdown. Portugal lists KALLAX door inserts at €7.50 where five other countries charge €15. I cannot prove either is a promotion from these fields, so I re-ran the index without BILLY and without the KALLAX inserts, 54 articles in all. The ends held: the Netherlands at 85.2, Germany 88.4, France 113.7. Austria moved up to 110.0.

formattedPrice is localised. Spain returns "44,99", Germany "59.99", the US "$79.00". Parse currentPrice, which is always a number. Badges are localised as well: the same idea arrives as "Top seller", "Best seller", "Bestseller", "Top ventes" and "Il più venduto", so filtering on one string misses most of them.

One page is a sample, not a catalogue. 615 of the 1,220 article numbers appeared in just one country. Most of that is ranking: I read page one of each search, and each country orders its results differently. Absence from page one does not mean IKEA does not sell it there, which is why I checked the white BILLY with exact-number searches before writing that paragraph.

What I would do next

Three things. Paginate with the page parameter, so the basket covers whole families instead of whatever ranked on page one. Run the same basket weekly, which is the only honest way to tell a 15% promotion from a list price. And add Sweden, Switzerland, the UK and the US with ECB reference rates for a stated date, accepting that the gaps then include currency.

The data came from the IKEA Pro API. If you want a full catalogue as a file instead, there is an IKEA US product dataset, and for anything custom, get in touch.

Scrape Tesco Data with Python: How Much Do Clubcard Prices Really Save?

Happy Endpoint — Thu, 29 Oct 2026 09:07:37 +0000

Every Tesco shelf edge has the yellow Clubcard Price tags, and the question I wanted answered is simple: how much do they actually take off, and how many products have one? I collected 14 aisles of Tesco UK on 24 October 2026 with Python. The median single-item Clubcard Price is 20.9% below the shelf price. But if you only scrape the first page of each aisle, as most tutorials do, you will think offers are about twice as common as they are.

The data

Tesco has no public product API, so I used the Tesco Data API, which returns Tesco's category pages as JSON: shelf price, unit price, isForSale and a promotions list with a description, an offerType, a startDate and an endDate for each offer. Keys come from RapidAPI.

I picked 14 aisles across food, drink, household and toiletries, from Fresh Fruit to Laundry. Together Tesco lists 8,231 products in them. Category pages hold 24 products each, and a big aisle like Wine runs to 72 pages, so I did not pull everything. For each aisle I took page 1, the middle page and the last page. That turned out to matter more than anything else in this analysis.

The run cost 54 API calls, including the category tree and one exploratory search. It returned 940 product rows. After removing duplicates, products that are not for sale and one row with no price, 920 products are left.

The code

This is the whole script. It looks up aisle IDs from the category tree (/get-category-ids-for-uk), pulls three pages per aisle from /category-page, caches every response to disk, and turns each promotion's text into a price per item.

# Scrape Tesco UK aisles with the Tesco Data API and measure Clubcard Prices.
# pip install requests pandas; export RAPIDAPI_KEY=...; python clubcard_core.py
import json
import math
import os
import re
from pathlib import Path

import requests

HOST = "tesco-data-api.p.rapidapi.com"
CACHE = Path(__file__).resolve().parent.parent / "data" / "raw"
HEADERS = {"X-RapidAPI-Key": os.environ.get("RAPIDAPI_KEY", ""), "X-RapidAPI-Host": HOST}


def get(path, params, cache_name):
    """Call the API once per page and keep the JSON, so reruns are free."""
    cache = CACHE / cache_name
    if cache.exists():
        return json.loads(cache.read_text())
    resp = requests.get("https://" + HOST + path, headers=HEADERS, params=params, timeout=60)
    if resp.status_code != 200:
        print("skipped", cache_name, resp.status_code)
        return None
    CACHE.mkdir(parents=True, exist_ok=True)
    cache.write_text(json.dumps(resp.json(), ensure_ascii=False))
    return resp.json()


def aisle_ids():
    tree = get("/get-category-ids-for-uk", {}, "explore-categories-uk.json")["data"]["taxonomy"]
    return {(d["name"], a["name"]): a["id"]
            for sd in tree for d in sd.get("children", []) for a in d.get("children", []) if "id" in a}


def load_aisle(cat_id, name):
    """Page 1, the middle page and the last page of one aisle."""
    get_page = lambda n: get("/category-page", {"categoryId": cat_id, "pageNo": n}, "uk-%s-p%d.json" % (name, n))
    first = get_page(1)
    if first is None:
        return []
    last = max(1, math.ceil(first["data"]["pageInformation"]["totalResults"] / 24))
    pages = [first]
    for n in sorted({math.ceil(last / 2), last} - {1}):
        page = get_page(n)
        if page is None and n == last and n - 1 > 1:
            page = get_page(n - 1)  # the computed last page sometimes fails
        if page is not None:
            pages.append(page)
    return [dict(p, pageNo=pg["data"]["pageNo"], position=i, aislePages=last)
            for pg in pages for i, p in enumerate(pg["data"]["products"])]


MONEY = r"(?:£(\d+(?:\.\d+)?)|(\d+)p)"


def pounds(m, i=1):
    return float(m.group(i)) if m.group(i) else int(m.group(i + 1)) / 100


def parse_offer(description, shelf):
    """Turn a promotion description into (kind, min_qty, member price per item)."""
    text = description.split(" - ")[0]
    if "Meal Deal" in text:
        return "meal deal", None, None
    m = re.match(r"Any (\d+) for " + MONEY, text) or re.match(r"(\d+) for " + MONEY, text)
    if m:  # Any 2 for £4, or 2 for £5 or 4 for £8 (first tier)
        qty = int(m.group(1))
        return "multibuy", qty, pounds(m, 2) / qty
    m = re.match(r"Any (\d+) for (\d+) ", text)
    if m:  # Any 3 for 2: cheapest free, so equal prices give this per item
        qty = int(m.group(1))
        return "multibuy", qty, shelf * int(m.group(2)) / qty
    m = re.match(r"Buy (\d+) or more Save (\d+)%", text)
    if m:
        return "multibuy", int(m.group(1)), shelf * (1 - int(m.group(2)) / 100)
    m = re.match(MONEY + r" .*Clubcard Price", text)
    if m:  # £3.00 Clubcard Price, 75p Clubcard Price, £3.50 Half Price Clubcard Price
        return "single", 1, pounds(m)
    return "other", None, None


AISLES = [
    ("Fresh Food", "Fresh Fruit"), ("Fresh Food", "Fresh Meat & Poultry"),
    ("Fresh Food", "Cheese"), ("Fresh Food", "Ready Meals"), ("Bakery", "Bread"),
    ("Treats & Snacks", "Chocolate, Sweets, Mints & Chewing Gum"),
    ("Treats & Snacks", "Crisps, Snacks, Nuts & Popcorn"), ("Food Cupboard", "Cereals"),
    ("Drinks", "Fizzy & Soft Drinks"), ("Drinks", "Beer & Cider"), ("Drinks", "Wine"),
    ("Drinks", "Coffee"), ("Household", "Laundry"),
    ("Toothpaste, Mouthwash & Toothbrush", "Toothpaste"),
]


def collect():
    ids, rows, seen = aisle_ids(), [], set()
    for dept, aisle in AISLES:
        name = "".join(c if c.isalnum() else "-" for c in aisle.lower()).strip("-")[:40]
        for p in load_aisle(ids[(dept, aisle)], name):
            price = (p.get("price") or {}).get("price")
            if p["id"] in seen or not p["isForSale"] or not price:
                continue  # duplicates, off-sale items and missing prices
            seen.add(p["id"])
            offers = [(o,) + parse_offer(o["description"], price) for o in p["promotions"]]
            singles = [unit for o, kind, qty, unit in offers if kind == "single"]
            multis = [unit for o, kind, qty, unit in offers if kind == "multibuy"]
            rows.append({
                "aisle": aisle, "id": p["id"], "title": p["title"], "page": p["pageNo"],
                "position": p["position"], "aisle_pages": p["aislePages"], "price": price,
                "unit": p["price"].get("unitOfMeasure"), "n_offers": len(offers),
                "kinds": sorted({kind for o, kind, qty, unit in offers}),
                "single_off": 1 - min(singles) / price if singles else None,
                "multi_off": 1 - min(multis) / price if multis else None,
                "offers": [{"description": o["description"], "kind": kind, "qty": qty,
                            "start": o["startDate"], "end": o["endDate"]}
                           for o, kind, qty, unit in offers],
            })
    return rows


def aisle_share(g, col):
    """Page 1 is one page of the aisle, not a third of it: weight it that way."""
    pages = g.aisle_pages.iloc[0]
    first, rest = g[g.page == 1][col].mean(), g[g.page > 1][col].mean()
    return (first + rest * (pages - 1)) / pages


if __name__ == "__main__":
    import pandas as pd

    df = pd.DataFrame(collect())
    df["any_offer"] = df.n_offers > 0
    df["clubcard_price"] = df.single_off.notna()
    summary = pd.DataFrame({
        "products": df.groupby("aisle").size(),
        "any_offer": df.groupby("aisle").apply(aisle_share, "any_offer"),
        "clubcard_price": df.groupby("aisle").apply(aisle_share, "clubcard_price"),
        "median_saving": df.groupby("aisle").single_off.median(),
    })
    print(summary.sort_values("clubcard_price", ascending=False).round(2).to_string())

The output, as shares (0.44 is 44%):

                                        products  any_offer  clubcard_price  median_saving
aisle
Chocolate, Sweets, Mints & Chewing Gum        50       0.61            0.44           0.21
Fizzy & Soft Drinks                           74       0.62            0.43           0.24
Toothpaste                                    67       0.36            0.36           0.48
Crisps, Snacks, Nuts & Popcorn                74       0.42            0.25           0.20
Bread                                         66       0.21            0.20           0.13
Beer & Cider                                  62       0.30            0.18           0.17
Cereals                                       73       0.15            0.13           0.19
Ready Meals                                   62       0.66            0.12           0.25
Fresh Fruit                                   73       0.19            0.10           0.23
Fresh Meat & Poultry                          64       0.24            0.08           0.17
Coffee                                        58       0.22            0.05           0.36
Laundry                                       60       0.07            0.04           0.16
Cheese                                        65       0.35            0.03           0.21
Wine                                          72       0.01            0.01           0.16

The part that does the real work is parse_offer. Tesco does not give you a member price as a number. It gives you a sentence like £2.95 Better than Half Price Clubcard Price or Any 2 for £4.50 Clubcard Price - Selected Sliced And Grated Cheese, and you have to turn that into a price per item yourself. I split the offers into three kinds: a single Clubcard Price (one item, lower price), a multibuy (you must buy 2, 3, 4 or 6) and a meal deal (a bundle price across several products).

Offers crowd onto page 1

This was the finding I did not expect. In the order the category pages come back, promoted products sit near the top of most aisles. In my sample, 55.8% of page-1 products had some promotion, against 37.0% on the middle page and 18.7% on the last page. The first three slots of page 1 were even more loaded: 54.8% of those 42 products had a single-item Clubcard Price.

The effect is strong but not universal. Page 1 had the higher share in 10 of the 14 aisles: coffee 74.1% against 19.4% deeper in the aisle, fresh fruit 59.3% against 15.2%, and wine 92.3% against 0.0%. In chocolate, fizzy drinks, ready meals and fresh meat, offers are spread through the whole aisle and page 1 had slightly fewer than the deeper pages. So there is no single correction factor; it has to be done aisle by aisle.

That means the raw sample says 39.8% of products have an offer and 22.8% have a Clubcard Price, and both numbers are too high, because page 1 is 39.1% of my sample but only one of the 6 to 72 pages in each aisle. Weighting page 1 as one page and letting the middle and last pages stand for the rest, the estimate across the 14 aisles is 26.5% with any offer and 14.1% with a single-item Clubcard Price. Roughly 1 product in 7.

That is an estimate from two deep pages per aisle, not a census. Treat the per-aisle figures as "a lot", "some" or "very few" rather than as exact numbers.

Where the Clubcard Prices are

Chocolate and sweets (43.6%) and fizzy drinks (43.1%) lead, with toothpaste close behind at 35.9%. Cheese (3.1%), laundry (3.7%) and wine (0.6%) sit at the bottom. Wine is the clearest case of the page-1 effect: a third of the products I sampled had an offer, all of them on page 1, and the middle and last pages of a 72-page aisle had none at all (0 of 46 products).

Ready meals look odd at first: 66.3% of products carry a promotion but only 12.3% a Clubcard Price. Most of the rest are meal deals, which I come back to below.

How much a Clubcard Price takes off

Across 210 single-item Clubcard Prices, the median saving is 20.9%. The middle half of offers sit between 16.7% and 30.2%, and 47.6% of them save less than a fifth. That lines up with the CMA's 2024 review of loyalty pricing, which found average member savings of 17% to 25% across five supermarkets.

Deep cuts are rare. Only 18 offers (8.6%) were half price or better. The biggest was 60%, Oral-B kids' toothbrush refills at £30.00 with a £12.00 Clubcard Price, followed by Costa instant coffee at £6.80 with a £2.95 Clubcard Price (56.6% off).

Depth varies by aisle more than frequency does. Toothpaste aisle offers save a median 48.5% and coffee 36.3%. Bread saves 13.3%. Coffee shows that frequency and depth are separate questions: only about 5% of the coffee aisle carries a Clubcard Price, but where there is one, it is big. If you build a price index, track both, or a handful of deep cuts at the top of the aisle will look like a cheap category.

The three offer kinds behave differently:

Offer kind	Offers	Median saving	Items you must buy	Median length
Single Clubcard Price	210	20.9%	1	42 days
Multibuy Clubcard Price	135	25.0%	2 to 6 (median 3)	110 days
Meal deal	52	not per product	a bundle	210 days
Not a Clubcard offer	1	not computed	1	10 days

Multibuys save a little more per item, 25.0% at the median and 33.3% at most, but you have to buy at least two, and 22 of the 135 need six (the wine "Buy 6 or more Save 25%" deal). They also run far longer than single Clubcard Prices: a median 110 days against 42.

Pitfalls in the Tesco data

offerType tells you almost nothing. 397 of the 398 offers I collected were CLUBCARD_PRICING. The only other value was one STANDARD_OFFER ("Save £4.10" on a laundry basket). Meal deals, 3 for 2s and straight member prices all share the same type, so you must parse description. Count anything with CLUBCARD_PRICING as a discount and Ready Meals looks like Tesco's most discounted aisle, when most of its offers are bundles like "WASABI DINE Meal Deal for £10 Clubcard Price - Buy 1 Main, 1 Sauce & 1 Side" on £6.00 items.

Compute the saving; do not read it from the label. The labels are accurate but rounded down. All 11 "Half Price" offers were exactly 50%. "Better than Half Price" ranged from 52.9% to 60.0%, "Save 25%" from 25.0% to 27.0%, and "Save 1/3" from 33.3% to 34.8%. Most offers have no label at all, only a price, so you need the arithmetic anyway.

Multibuy savings depend on the basket. "Any 2 for £4.50" on Tesco Grated Mozzarella 250g at £2.65 works out at £2.25 each, 15.1% off. But "any" means you can mix products from the offer range, and if the other item is cheaper your saving is smaller. "Any 3 for 2" gives the cheapest item free, so it only equals 33.3% when the prices match. And tiered offers like "2 for £5 or 4 for £8" need a choice; I used the first tier.

Page 1 returns more than 24 products. resultsPerPage says 24, but page 1 returned 24, 26 or 27 rows depending on the aisle. Two of the three duplicate IDs in my 940 rows were page-1 top slots that showed up again deeper in the same aisle. A third product, a Fit Foods ready meal, sat in both Fresh Meat & Poultry and Ready Meals. Deduplicate on id before counting anything.

Unit prices come in five units. Of 940 rows, 554 were priced per kg, 267 per litre and 111 per item, plus 4 per kg DR.WT (drained weight, for things like burrata in brine) and 3 per 100g. The toothpaste aisle alone mixes 43 products priced per litre with 24 priced per item, and the per-item group includes tubes of toothpaste, not only brushes and refills. Compare unitPrice only within the same unitOfMeasure.

Offers are a snapshot with a clock on them. Single Clubcard Prices run in short cycles: 37 of them ended on 4 November and 27 on 27 October. 67 of the 398 offers ended within a week of my collection date. A one-day scrape tells you about this week's cycle, not about Tesco in general. Some member prices are not short at all: 30 single Clubcard Prices ran for more than 90 days.

Off-sale products still appear. 17 rows had isForSale set to false, five of them in Bread. They still carry a price and sometimes promotions, so filter them before averaging.

Failed pages are gaps, not empty pages. A few of my requests for deep pages came back with a server error instead of products. The script prints and skips a failed page, and steps back one page when the last page fails; in a scheduled job, retry failures with a short backoff. If you treat a failed page as "no products", you will quietly bias every share towards page 1.

Some labels are not in the data at all. None of the 940 rows mention Tesco's Aldi Price Match, so if you need that badge, this endpoint will not give it to you.

What I would do next

One day of data shows the cycle, not the trend. The next step is to run the same aisles daily for a month, which would show how often products move in and out of Clubcard Prices and whether the shelf price moves when they do. The same script works on Tesco Ireland by swapping in /ireland-category-page and the Irish category tree.

The API and its endpoints are on the Tesco Data API page. If you want the whole catalog in one file instead, there is a Tesco UK products dataset, and the product side of the API is covered in our LinkedIn guide to Tesco grocery data. For a custom basket or daily tracking, get in touch.

I Used the Idealista API in Python to Rank Madrid Districts by Rental Yield

Happy Endpoint — Tue, 27 Oct 2026 05:07:59 +0000

A flat in Villaverde pays back its asking price in 13.9 years of asking rent. In Barrio de Salamanca, the same calculation gives 32.3 years.

I wanted to know how wide that gap really is across Madrid, and how much of it survives once you clean the data properly. So I pulled for-sale and for-rent listings for all 21 Madrid districts from Idealista with Python, computed a gross rental yield per district, and then went looking for everything that could distort it.

The data

Idealista's own developer API is granted by application, so for this I used the Happy Endpoint Idealista API, which returns the public search results as JSON. Two endpoints did all the work:

/sublocations with Madrid's location ID (0-EU-ES-28-07-001-079) lists the 21 districts with their location IDs.
/property-search with search_type=for_sale or for_rent, property_type=homes, sort_order=newest and result_count=50 returns the 50 most recent ads per district.

I took one page of 50 for every district and operation, and a second page for seven searches where the first page left a thin sample after cleaning. That came to 50 API calls and 2,280 listings (1,200 for sale, 1,080 for rent), collected on 24 October 2026.

The yield is the simplest version that people actually quote:

gross yield = 12 × median rent per m² per month ÷ median sale price per m²

Both medians are taken over flats in the same district, so it is a district-level estimate built from asking prices, not the yield of any particular flat.

The script

This is the whole thing. It caches every response in data/raw/, so you only pay for the calls once and can rerun the analysis as often as you like. You need a RapidAPI key in RAPIDAPI_KEY, plus requests and pandas.

"""Gross rental yield by Madrid district from Idealista asking prices.

Usage:  RAPIDAPI_KEY=... python madrid_yield.py
Needs:  requests, pandas. Every response is cached in data/raw/, so a re-run is free.
"""
import json
import os
import re
from pathlib import Path

import pandas as pd
import requests

HOST = "idealista17.p.rapidapi.com"
MADRID = "0-EU-ES-28-07-001-079"  # Madrid municipality
RAW = Path(__file__).resolve().parent.parent / "data" / "raw"
RAW.mkdir(parents=True, exist_ok=True)

SECOND_PAGE = {  # district code -> searches that get a second page of 50
    "04": ["for_sale"], "05": ["for_sale"], "09": ["for_sale"],
    "11": ["for_rent"], "12": ["for_rent"], "13": ["for_rent"], "18": ["for_rent"],
}
NOT_VACANT = re.compile(  # sale ads that are not a vacant flat: occupied, bare ownership, auction
    r"bare ownership|bare property|nuda propiedad|usufruct|occupied property|occupied by"
    r"|illegally occupied|okupa|squat|without possession|auction|subasta|judicial",
    re.I,
)


def get(endpoint, params, name):
    """GET an endpoint once, then read it from data/raw/<name>.json."""
    path = RAW / (name + ".json")
    if path.exists():
        return json.loads(path.read_text())
    resp = requests.get(
        "https://" + HOST + endpoint,
        headers={"X-RapidAPI-Key": os.environ["RAPIDAPI_KEY"], "X-RapidAPI-Host": HOST},
        params=params,
        timeout=60,
    )
    resp.raise_for_status()
    path.write_text(json.dumps(resp.json(), ensure_ascii=False))
    return resp.json()


def districts():
    body = get("/sublocations", {"country": "es", "location_id": MADRID,
                                 "property_type": "homes", "search_type": "for_sale"},
               "sublocations-madrid-sale")
    return [(loc["locationId"], loc["name"].replace(", Madrid", ""))
            for loc in body["data"]["locations"]]


def search(location_id, search_type, page=1):
    code = location_id.split("-")[-1]
    body = get("/property-search", {
        "country": "es", "location_ids": location_id, "search_type": search_type,
        "property_type": "homes", "sort_order": "newest", "page": page,
        "result_count": 50, "language": "en",
    }, "search-%s-%s-p%d" % (search_type, code, page))
    return body["data"]


def to_rows(listings, district):
    rows = []
    for ad in listings:
        labels = [lab["name"] for lab in ad.get("labels") or []]
        rows.append({
            "id": ad["propertyCode"], "district": district, "operation": ad["operation"],
            "typology": (ad.get("detailedType") or {}).get("typology"),
            "price": ad["price"], "size": ad.get("size"), "rooms": ad.get("rooms"),
            "neighborhood": ad.get("neighborhood"),
            "seasonal": "seasonalRental" in labels,
            "not_vacant": bool(NOT_VACANT.search(ad.get("description") or "")),
        })
    return rows


def load():
    """Every listing collected, one row each."""
    rows = []
    for loc_id, name in districts():
        for search_type in ("for_sale", "for_rent"):
            pages = 2 if search_type in SECOND_PAGE.get(loc_id[-2:], []) else 1
            for page in range(1, pages + 1):
                rows += to_rows(search(loc_id, search_type, page)["listings"], name)
    return pd.DataFrame(rows).drop_duplicates("id")


def clean(df):
    """Flats only, plausible sizes, no repeated ads, no seasonal lets, no occupied sales."""
    df = df[(df.typology == "flat") & df["size"].between(30, 150)].copy()
    df = df.drop_duplicates(["operation", "district", "neighborhood", "rooms", "size", "price"])
    df = df[~((df.operation == "rent") & df.seasonal)]
    df = df[~((df.operation == "sale") & df.not_vacant)]
    df["eur_m2"] = df.price / df["size"]
    return df


def yields(df):
    med = df.pivot_table(index="district", columns="operation", values="eur_m2",
                         aggfunc="median")
    n = df.pivot_table(index="district", columns="operation", values="id", aggfunc="count")
    out = pd.DataFrame({
        "sale_eur_m2": med["sale"].round(0),
        "rent_eur_m2_month": med["rent"].round(2),
        "gross_yield_pct": (12 * med["rent"] / med["sale"] * 100).round(2),
        "n_sale": n["sale"], "n_rent": n["rent"],
    })
    return out.sort_values("gross_yield_pct", ascending=False)


if __name__ == "__main__":
    table = yields(clean(load()))
    print(table.to_string())

Two details are worth knowing before you write your own version. The ads come back under data.listings, and short-term lets are flagged in a labels list on each ad ({"name": "seasonalRental", "text": "Short-term rental"}), not in a field of their own. Both matter below.

Results: the south and east out-earn the centre

After cleaning, 1,649 listings were left (946 for sale, 703 for rent). Across the whole city the median asking price is €5,255 per m² and the median asking rent is €20.77 per m² a month, which gives a city-wide gross yield of 4.74%. Nine of the 21 districts come in above 5%.

The top seven are Villaverde (7.19%), Puente de Vallecas (6.49%), Carabanchel (6.28%), Usera (6.13%), San Blas (5.80%), Villa de Vallecas (5.78%) and Latina (5.72%). The bottom four are the expensive core: Chamartín (3.70%), Chamberí (3.63%), Retiro (3.36%) and Barrio de Salamanca (3.10%).

The reason is in the two medians, not in the rent. A square metre in Salamanca asks €10,797, 3.31 times Villaverde's €3,260. Rent per square metre is only 1.43 times higher (€27.86 against €19.52). Outside Centro, Salamanca and Chamberí, the median asking rent sits between €17.08 and €23.93 per m² in every district. Rents in Madrid are compressed; sale prices are not. So the yield ranking is mostly a ranking of purchase prices, read upside down.

How precise is it? I bootstrapped each district 2,000 times (resampling its sale and rent ads separately). The 90% interval for Villaverde runs from 6.04% to 7.98%, for Salamanca from 2.77% to 3.38%. The top and bottom of the table are clearly different. The neighbours inside the table are not: Puente de Vallecas (6.07% to 7.15%) overlaps Villaverde, and Carabanchel and Usera overlap each other almost entirely. Treat the ranking as tiers, not places.

The catch: the high-yield districts have almost nothing to rent

Every search response also carries a total: the number of live ads in that district, not just the 50 I fetched. Comparing the for-sale and for-rent totals turned out to be the most interesting number in the whole exercise.

The seven top-yield districts had 615 homes for rent against 5,381 for sale: 11.4 rental ads per 100 sale ads. The other 14 districts had 52.7. Thin supply does not create a high yield by itself, though: Vicálvaro and Moratalaz are just as thin and sit in the lower half of the ranking, because they combine the two lowest asking rents in the city (€17.08 and €17.21 per m²) with sale prices well above the southern districts. Villaverde had 44 rental ads in total against 751 for sale. Centro had 1,912 rentals against 2,371 sales.

Two consequences. First, in four districts (Barajas, Moratalaz, Vicálvaro and Villaverde) there were 50 rental ads or fewer, so my "sample" of rentals was the entire live rental stock. Villaverde's 7.19% rests on 30 long-term rental ads after cleaning, and Moratalaz's on 14. Second, a thin rental market cuts both ways for an investor. Few competing ads can mean steady demand, but it also means the asking rent is set by a handful of landlords, and one new building of rentals can move the median.

What each cleaning step changed

Here is the city-wide yield after each step, applied cumulatively. It is the most useful table in this post if you are building your own version.

Step	Listings	Sale €/m²	Rent €/m²/month	Gross yield
All homes as returned	2,280	5,500	22.19	4.84%
Same, means instead of medians	2,280	6,243	24.61	4.73%
Flats only (no chalets)	2,217	5,511	22.23	4.84%
30 to 150 m² only	1,942	5,238	21.93	5.02%
Repeated ads removed	1,889	5,234	21.76	4.99%
Short-term lets removed	1,675	5,234	20.77	4.76%
Occupied and bare-ownership sales removed	1,649	5,255	20.77	4.74%

The headline barely moves, from 4.84% to 4.74%. That is reassuring, but it hides changes that matter a lot more at district level. These are the traps, in the order I hit them.

Pitfalls in Idealista data for yield work

Short-term lets are a separate market. Of the 917 rental flats left after removing repeated ads, 214 (23.3%) carried the seasonalRental label. In Centro it was 21 of 48 ads, or 43.8%. Spanish law treats these as seasonal contracts outside the ordinary rules for a primary home (the Ley de Arrendamientos Urbanos separates the two), so their rent is not what a long-term tenant pays. Removing them took the city figure from 4.99% to 4.76%.

What surprised me is where it mattered. In Centro, where the share is highest, the yield hardly changed (5.17% with them, 5.16% without), because short-term flats there asked €29.26 per m² against €29.17 for long-term ones. In Puente de Vallecas, a third of the rental ads were short-term, at €26.67 per m² against €19.17, and they pushed the yield from 6.49% to 7.24%. You cannot guess the direction from the share alone; you have to filter.

The same flat appears more than once. I grouped flats with the same operation, district, neighbourhood, rooms, size and price. That found 38 groups covering 91 ads. In 25 of the groups the ads came from different advertisers, which is what you expect in Spain, where a seller often lists with several agencies at once. In the other 13 it was one advertiser, either with several identical units or with the same flat posted twice. Either way, one flat or one building should not count three times in a median, so I kept one ad per group (53 removed).

Some "cheap" flats come with a tenant or a squatter. A text search of the sale descriptions (requested with language=en, so they come back translated) found 26 flats sold as occupied, without possession, as bare ownership or through an auction process. Their median price per m² was 15.8% below their district's median for ordinary flats. They are real listings, but they are not comparable with a flat you can let tomorrow, and in a small district a few of them pull the sale median down and the yield up.

Rentals are smaller than sales. The median rental flat is 71 m², the median flat for sale 81 m². Small flats ask far more rent per m² (€28.06 for 30 to 50 m², against €18.06 for 121 to 150 m²), while the sale price per m² is similar at both ends (€6,310 and €6,039):

Part of this is location (big flats cluster in Salamanca, Chamberí and Retiro), but the effect is large enough that the size band you pick changes the answer. Restricting both sides to 40 to 90 m² gives a city-wide yield of 5.18% instead of 4.74%. Whatever band you choose, use the same one for rent and sale, and say which it was.

priceByArea is rounded for rentals. Every ad has a priceByArea field, and on rentals it is a whole number of euros: 100% of the rental ads I collected had an integer value. That is up to 4.7% away from price divided by size (median gap 0.95%). It barely shifts most districts, but Latina moves from 5.72% to 5.84% if you use the field instead of computing it. Compute price / size yourself.

Use medians, and expect odd ads. Means push the city-wide asking price from €5,500 to €6,243 per m², because a few luxury flats sit far above everything else. There are odd ads at the other end too: one flat filed under Sol, in Centro, asked €176,000 for 70 m² (about €2,500 per m²), and its description turned out to be about a flat near Barcelona. A median shrugs that off; a mean does not.

These are asking prices, and the yield is gross. Asking prices are not closing prices, and nothing in a listing tells you the final figure. Sorting by newest also means few ads had been cut yet: only 4.8% of the sale ads and 3.6% of the rental ads carried a price drop. A gross yield ignores community fees, IBI property tax, insurance, empty months and income tax, which are what decide whether a 7% gross yield is still attractive net.

Newest first is a choice. sort_order=newest gives a fresh snapshot rather than the stale ads that never sold. In Centro, all 50 rental ads on the first page had been published within about 29 hours. The default sort is a weighted relevance ranking, and I did not want a ranking deciding which ads enter a price median.

What I would do next

Run the same script weekly and keep the snapshots, so each district gets a time series instead of one reading (I wrote up the snapshot-and-compare pattern for buyers in turning Idealista searches into shortlists). Go down to neighbourhood level in Centro and Carabanchel, where the samples are big enough. And for the thin districts, fetch every rental page rather than the newest 50, since the whole stock fits in one or two calls anyway.

Endpoints and parameters for everything above are documented on happyendpoint.com, and you can get a key and test the calls in the browser on RapidAPI. The same code works for Barcelona, Lisbon or Milan once you swap the location ID and country. If you need a larger pull, such as every listing in a city each week, get in touch.

I Collected Data on 19,994 UAE Real Estate Agents. Here Is What It Looks Like.

Happy Endpoint — Tue, 05 May 2026 09:10:25 +0000

The UAE real estate market gets covered endlessly in terms of property prices and transaction volumes. What gets less attention is the agent layer sitting underneath all of it. Who are these people? How experienced are they? What languages do they speak? Which agencies dominate by headcount?

I put together a structured dataset of every active agent listed on PropertyFinder UAE as of May 2026. This is what I found, and what the data is actually useful for.

What the dataset contains

19,994 agents, each with 51 fields covering:

Contact details: email, phone, WhatsApp, LinkedIn
Professional background: years of experience, position title, bio, nationality
Credentials: BRN license number, verified status, superagent status
Performance: platform ranking, average rating, full rating distribution, median listing quality score
Active listings: broken down by residential sale, residential rent, commercial sale, commercial rent
Transaction history: total deal count, total volume in AED, average sale and rent prices
Response behavior: average WhatsApp response time in minutes
Languages spoken
Top active locations (as location IDs, joinable against a separate locations table)
Agency: name, address, city, logo URL
Direct profile URL for each agent

A few things that stand out

The market is genuinely multilingual.

This is not a market where English is assumed. A meaningful portion of agents list languages like Arabic, Russian, Hindi, Urdu, Persian, French, and Mandarin. If you are building a lead routing system or a CRM that matches clients to agents, language matching is a real feature, not a nice-to-have.

Experience range is wide.

The experience_since_year field shows agents who have been active since the early 2000s sitting alongside people who started last year. If you are doing any kind of agent quality scoring, tenure is a useful baseline signal alongside rating and transaction volume.

Transaction volume tells a different story than listing count.

An agent can have 50 active listings and a claimed deal volume of zero. Another agent might have 8 listings and 200 million AED in claimed transactions. The dataset captures both, which means you can segment agents by activity type rather than just volume of listings on the page.

WhatsApp response time is tracked.

This surprised me. PropertyFinder tracks and exposes average WhatsApp response times per agent. For a market where WhatsApp is the primary sales communication channel, this is a real operational signal. The variance across agents is large.

Agency concentration.

The agency_id and agency_name fields let you count agents per agency. A handful of agencies account for a disproportionate share of total agents. If you are doing competitive research on agency market share, this is a straightforward query.

// Detect dark theme var iframe = document.getElementById('tweet-2051586550019047520-71'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=2051586550019047520&theme=dark" }

What the data is actually useful for

PropTech and CRM tools

If you are building anything in the UAE real estate space, you need a structured agent directory. This dataset gives you a ready-to-use starting point for agent profiles, lead routing, search, and filtering. Building it yourself means collecting, cleaning, and deduplicating thousands of records before you write a single line of product code.

B2B outreach and marketing

19,994 agents with email, phone, and WhatsApp. Segmentable by language, experience, specialty (residential vs commercial), agency, and location coverage. If you are selling software, financial products, or services to UAE real estate agents, this is a usable prospecting list.

Market research

Which nationalities are most represented in the agent population? How does average rating vary by agency? What is the distribution of experience across the market? These are answerable questions once the data is in a spreadsheet or a database.

Investment and partnership decisions

If you are a developer, an investment fund, or an agency looking to partner with top-performing agents in specific areas, the combination of transaction volume, claimed deal data, and location IDs gives you a shortlist starting point faster than reading individual profiles.

A note on the location IDs

The top_location_ids field contains pipe-separated PropertyFinder location IDs representing each agent's primary areas. These IDs reference a separate locations table (which I also have as a dataset) containing 15,248 UAE locations with names, types, coordinates, and hierarchy. Without that table, the IDs are opaque. With it, you can answer questions like "show me all verified agents whose top locations include Dubai Marina or Downtown Dubai."

Format and availability

The dataset is available as a CSV file. 19,994 rows, 51 columns, UTF-8 encoded, no JSON blobs. Every column is a readable value.

If you want a sample or have a specific use case in mind, get in touch.

Final thought

Real estate data in the UAE tends to get packaged at the transaction and property level. Agent-level structured data is less common, which is part of why it is useful. The people executing these transactions are not well represented in public datasets. This is one attempt to change that.

Wants to get the dataset, check here: https://payhip.com/b/bld1P

How to Build a Beauty Recommendation Engine with Real-Time Data

Happy Endpoint — Fri, 10 Apr 2026 14:23:29 +0000

Recommendation engines are the future of e-commerce. Netflix recommends movies. Amazon recommends products. Spotify recommends songs.

Beauty apps need recommendations too.

Here's how to build one using real-time Sephora data.

The Problem

Beauty shopping is overwhelming. 8,500+ products. 500+ brands. How do customers find what they need?

They need recommendations.

The Solution: Ingredient-Based Matching

Instead of recommending based on popularity, recommend based on ingredients.

If a customer likes products with hyaluronic acid, recommend other products with hyaluronic acid.

If a customer likes retinol products, recommend other retinol products.

How It Works

User Profile – Track which products the user likes
Extract Ingredients – Get ingredients from liked products
Find Similar Products – Find products with similar ingredients
Rank by Rating – Sort by customer rating
Recommend – Show top 5 recommendations

Code Example

import requests
from collections import Counter

class BeautyRecommender:
    def __init__(self, api_key):
        self.api_key = api_key
        self.base_url = 'https://real-time-sephora-api.p.rapidapi.com'

    def get_product_details(self, product_id):
        """Get product details including ingredients"""
        response = requests.get(
            f'{self.base_url}/product-details',
            params={'productId': product_id},
            headers={'x-rapidapi-key': self.api_key}
        )
        return response.json()

    def extract_ingredients(self, product):
        """Extract ingredients from product"""
        ingredients = product.get('ingredients', [])
        return [ing.lower() for ing in ingredients]

    def find_similar_products(self, liked_products, num_recommendations=5):
        """Find products similar to liked products"""
        # Extract all ingredients from liked products
        all_ingredients = []
        for product_id in liked_products:
            product = self.get_product_details(product_id)
            ingredients = self.extract_ingredients(product)
            all_ingredients.extend(ingredients)

        # Count ingredient frequency
        ingredient_counts = Counter(all_ingredients)
        top_ingredients = [ing for ing, count in ingredient_counts.most_common(5)]

        # Search for products with top ingredients
        recommendations = []
        for ingredient in top_ingredients:
            response = requests.get(
                f'{self.base_url}/search-by-keyword',
                params={'keyword': ingredient, 'pageSize': 20},
                headers={'x-rapidapi-key': self.api_key}
            )
            products = response.json()['products']
            recommendations.extend(products)

        # Remove duplicates and sort by rating
        unique_products = {p['productId']: p for p in recommendations}
        sorted_products = sorted(
            unique_products.values(),
            key=lambda x: x['rating'],
            reverse=True
        )

        return sorted_products[:num_recommendations]

# Usage
recommender = BeautyRecommender('YOUR_API_KEY')
liked_products = ['P427421', 'P458747', 'P495632']
recommendations = recommender.find_similar_products(liked_products)

for product in recommendations:
    print(f"{product['productName']}: {product['rating']} stars")

Advanced Features

1. Collaborative Filtering

Recommend products that similar users liked:

def collaborative_filtering(self, user_id, num_recommendations=5):
    """Recommend based on similar users"""
    # Find users with similar taste
    similar_users = self.find_similar_users(user_id)

    # Get products they liked
    recommendations = []
    for user in similar_users:
        products = self.get_user_liked_products(user)
        recommendations.extend(products)

    # Rank by popularity
    return self.rank_by_popularity(recommendations)[:num_recommendations]

2. Price-Based Recommendations

Recommend products at similar price points:

def price_based_recommendations(self, product_id, num_recommendations=5):
    """Recommend products at similar price"""
    product = self.get_product_details(product_id)
    price = product['price']

    # Search for products in price range
    response = requests.get(
        f'{self.base_url}/search-by-keyword',
        params={
            'keyword': product['category'],
            'minPrice': price * 0.8,
            'maxPrice': price * 1.2,
            'sortBy': 'TOP_RATED'
        },
        headers={'x-rapidapi-key': self.api_key}
    )

    return response.json()['products'][:num_recommendations]

3. Trending Products

Recommend products that are trending:

def trending_recommendations(self, num_recommendations=5):
    """Recommend trending products"""
    response = requests.get(
        f'{self.base_url}/search-by-keyword',
        params={
            'keyword': 'trending',
            'sortBy': 'NEWEST',
            'pageSize': num_recommendations
        },
        headers={'x-rapidapi-key': self.api_key}
    )

    return response.json()['products']

Real-World Results

I built a beauty recommendation engine using this approach. Here's what happened:

Click-Through Rate: 35% (vs. 5% for random recommendations)
Conversion Rate: 12% (vs. 2% for random recommendations)
Average Order Value: $65 (vs. $45 for random recommendations)
Customer Satisfaction: 4.5/5 stars

The Bottom Line

Recommendation engines drive revenue. They improve user experience. They increase engagement.

And with real-time Sephora data, you can build one in days, not months.

Ready to build? Get started with the Sephora API → https://rapidapi.com/happyendpoint/api/real-time-sephora-api/

How to Build a Dubai Real Estate Search App with Next.js and the Bayut API

Happy Endpoint — Wed, 08 Apr 2026 06:06:46 +0000

// Detect dark theme var iframe = document.getElementById('tweet-2037228542451474483-7'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=2037228542451474483&theme=dark" }

In this tutorial, we'll build a full Dubai property search app using Next.js 14 and the Bayut API.

By the end, you'll have a working app with:

Location autocomplete
Property search with filters
Property detail pages
Off-plan project listings

Let's build it.

Prerequisites

Node.js 18+
A RapidAPI account
Basic knowledge of Next.js and React

Step 1: Project Setup

npx create-next-app@latest dubai-property-app --typescript --tailwind --app
cd dubai-property-app
npm install axios

Create a .env.local file:

RAPIDAPI_KEY=your_rapidapi_key_here

Get your key by subscribing to the Bayut API:

https://rapidapi.com/happyendpoint/api/bayut14/

Step 2: API Client

Create lib/bayut.ts:

import axios from 'axios';

const client = axios.create({
  baseURL: 'https://bayut14.p.rapidapi.com',
  headers: {
    'x-rapidapi-host': 'bayut14.p.rapidapi.com',
    'x-rapidapi-key': process.env.RAPIDAPI_KEY!
  }
});

export async function searchLocations(query: string) {
  const { data } = await client.get('/autocomplete', {
    params: { query, langs: 'en' }
  });
  return data.data.locations;
}

export async function searchProperties(params: {
  purpose: 'for-sale' | 'for-rent';
  locationId?: string;
  propertyType?: string;
  rooms?: string;
  priceMin?: number;
  priceMax?: number;
  page?: number;
}) {
  const { data } = await client.get('/search-property', {
    params: {
      purpose: params.purpose,
      location_ids: params.locationId,
      property_type: params.propertyType,
      rooms: params.rooms,
      price_min: params.priceMin,
      price_max: params.priceMax,
      sort_order: 'popular',
      page: params.page || 1,
      langs: 'en'
    }
  });
  return data.data;
}

export async function getPropertyDetails(externalId: string) {
  const { data } = await client.get('/property-details', {
    params: { external_id: externalId, langs: 'en' }
  });
  return data.data;
}

export async function searchOffPlan(locationId?: string) {
  const { data } = await client.get('/search-new-projects', {
    params: {
      location_ids: locationId,
      property_type: 'residential',
      sort_order: 'latest',
      page: 1
    }
  });
  return data.data.properties;
}

Step 3: Search Page

Create app/page.tsx:

'use client';

import { useState } from 'react';
import { useRouter } from 'next/navigation';

export default function HomePage() {
  const [query, setQuery] = useState('');
  const [locations, setLocations] = useState([]);
  const [purpose, setPurpose] = useState<'for-sale' | 'for-rent'>('for-sale');
  const router = useRouter();

  async function handleLocationSearch(value: string) {
    setQuery(value);
    if (value.length < 2) return;

    const res = await fetch(`/api/locations?query=${value}`);
    const data = await res.json();
    setLocations(data);
  }

  function handleSearch(locationId: string) {
    router.push(`/properties?locationId=${locationId}&purpose=${purpose}`);
  }

  return (
    <main className="min-h-screen bg-gradient-to-b from-blue-900 to-blue-700 flex items-center justify-center">
      <div className="bg-white rounded-2xl p-8 w-full max-w-2xl shadow-2xl">
        <h1 className="text-3xl font-bold text-gray-900 mb-2">
          Find Property in Dubai
        </h1>
        <p className="text-gray-500 mb-6">
          Powered by Bayut API — real-time UAE property data
        </p>

        <div className="flex gap-2 mb-4">
          <button
            onClick={() => setPurpose('for-sale')}
            className={`px-4 py-2 rounded-lg font-medium ${
              purpose === 'for-sale'
                ? 'bg-blue-600 text-white'
                : 'bg-gray-100 text-gray-700'
            }`}
          >
            Buy
          </button>
          <button
            onClick={() => setPurpose('for-rent')}
            className={`px-4 py-2 rounded-lg font-medium ${
              purpose === 'for-rent'
                ? 'bg-blue-600 text-white'
                : 'bg-gray-100 text-gray-700'
            }`}
          >
            Rent
          </button>
        </div>

        <input
          type="text"
          value={query}
          onChange={(e) => handleLocationSearch(e.target.value)}
          placeholder="Search area, e.g. Dubai Marina..."
          className="w-full border border-gray-200 rounded-xl px-4 py-3 text-lg focus:outline-none focus:ring-2 focus:ring-blue-500"
        />

        {locations.length > 0 && (
          <ul className="mt-2 border border-gray-200 rounded-xl overflow-hidden">
            {locations.map((loc: any) => (
              <li
                key={loc.externalID}
                onClick={() => handleSearch(loc.externalID)}
                className="px-4 py-3 hover:bg-blue-50 cursor-pointer flex justify-between"
              >
                <span>{loc.name?.en}</span>
                <span className="text-gray-400 text-sm">
                  {loc.adCount?.toLocaleString()} listings
                </span>
              </li>
            ))}
          </ul>
        )}
      </div>
    </main>
  );
}

Step 4: API Routes

Create app/api/locations/route.ts:

import { NextRequest, NextResponse } from 'next/server';
import { searchLocations } from '@/lib/bayut';

export async function GET(req: NextRequest) {
  const query = req.nextUrl.searchParams.get('query') || '';
  const locations = await searchLocations(query);
  return NextResponse.json(locations);
}

Create app/api/properties/route.ts:

import { NextRequest, NextResponse } from 'next/server';
import { searchProperties } from '@/lib/bayut';

export async function GET(req: NextRequest) {
  const params = Object.fromEntries(req.nextUrl.searchParams);
  const results = await searchProperties({
    purpose: (params.purpose as 'for-sale' | 'for-rent') || 'for-sale',
    locationId: params.locationId,
    propertyType: params.propertyType,
    rooms: params.rooms,
    priceMin: params.priceMin ? Number(params.priceMin) : undefined,
    priceMax: params.priceMax ? Number(params.priceMax) : undefined,
    page: params.page ? Number(params.page) : 1
  });
  return NextResponse.json(results);
}

Step 5: Property Listing Page

Create app/properties/page.tsx:

import { searchProperties } from '@/lib/bayut';

export default async function PropertiesPage({
  searchParams
}: {
  searchParams: { locationId?: string; purpose?: string; page?: string };
}) {
  const data = await searchProperties({
    purpose: (searchParams.purpose as 'for-sale' | 'for-rent') || 'for-sale',
    locationId: searchParams.locationId,
    page: searchParams.page ? Number(searchParams.page) : 1
  });

  return (
    <div className="max-w-6xl mx-auto px-4 py-8">
      <h2 className="text-2xl font-bold mb-2">
        {data.total.toLocaleString()} Properties Found
      </h2>

      <div className="grid grid-cols-1 md:grid-cols-2 lg:grid-cols-3 gap-6 mt-6">
        {data.properties.map((property: any) => (
          <a
            key={property.externalID}
            href={`/property/${property.externalID}`}
            className="bg-white rounded-xl shadow hover:shadow-lg transition overflow-hidden"
          >
            {property.coverPhoto && (
              <img
                src={property.coverPhoto.url}
                alt={property.title?.en}
                className="w-full h-48 object-cover"
              />
            )}
            <div className="p-4">
              <p className="font-semibold text-gray-900 truncate">
                {property.title?.en}
              </p>
              <p className="text-blue-600 font-bold text-lg mt-1">
                AED {property.price?.toLocaleString()}
              </p>
              <div className="flex gap-4 text-gray-500 text-sm mt-2">
                <span>{property.rooms} bed</span>
                <span>{property.baths} bath</span>
                <span>{property.area?.toFixed(0)} sqm</span>
              </div>
            </div>
          </a>
        ))}
      </div>
    </div>
  );
}

Step 6: Deploy

npm run build
npx vercel --prod

Set RAPIDAPI_KEY in your Vercel environment variables.

What You've Built

A production-ready Dubai property search app with:

Real-time data from Bayut via API
Location autocomplete
Filterable property listings
Clean, responsive UI

Total time: ~4-6 hours.

Next Steps

Add /transactions for price trend charts
Add /agent-search for agent profiles
Add /search-new-projects for off-plan section
Add saved searches with localStorage or a database

Resources

🔗 Bayut API: https://rapidapi.com/happyendpoint/api/bayut14/
📖 Docs: https://bayutapi.dev/
🌐 Happy Endpoint: https://happyendpoint.com

Happy Endpoint - developer-friendly APIs for real-world data.

I Stopped Scraping Real Estate Websites. Here's What I Use Instead.

Happy Endpoint — Mon, 06 Apr 2026 06:10:25 +0000

Let me tell you about a mistake I made early in my developer career.

I was building a property comparison tool for the UAE market. I needed data — lots of it. Listings, prices, agent contacts, location data.

So I did what most developers do: I wrote a scraper.

It worked great. For about two weeks.

Then the website updated its HTML structure. My scraper broke. I fixed it. It broke again. I added proxy rotation. Got rate-limited anyway. Added delays. The data became stale. Added more infrastructure. Costs went up.

Six months in, I was spending more time maintaining the scraper than building the actual product.

There had to be a better way.

// Detect dark theme var iframe = document.getElementById('tweet-2037431884817604926-727'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=2037431884817604926&theme=dark" }

The Problem With Scraping Real Estate Data

Real estate portals are some of the most aggressively protected websites on the internet. They have:

Dynamic JavaScript rendering - data isn't in the HTML, it's loaded via JS
Anti-bot fingerprinting - they detect headless browsers
IP rate limiting - too many requests = ban
Captcha challenges - especially on search pages
Frequent structure changes - your selectors break constantly

Even if you get it working, you're looking at:

Proxy costs ($50-200/month minimum for reliable proxies)
Headless browser infrastructure (Puppeteer/Playwright servers)
Maintenance time every time the site updates
Legal grey areas around scraping terms of service

For a side project or early-stage startup, this is a massive overhead.

What I Use Now: The PropertyFinder UAE Data API

I switched to the PropertyFinder UAE Data API by Happy Endpoint.

It's a REST API on RapidAPI that gives you direct access to UAE property data - no scraping, no proxies, no maintenance.

👉 https://rapidapi.com/happyendpoint/api/propertyfinder-uae-data

Here's what changed:

Before (scraping):

Time to first data: 2-3 days setup
Monthly infrastructure cost: $100-200
Maintenance: 4-8 hours/month
Reliability: ~70% uptime
Data freshness: depends on scrape schedule

After (API):

Time to first data: 10 minutes
Monthly cost: API subscription (fraction of scraping costs)
Maintenance: zero
Reliability: 99.9% uptime
Data freshness: updated daily, Redis-cached for speed

A Real Example: Searching Dubai Marina Apartments

Here's how simple it is. In JavaScript:

const axios = require('axios');

async function searchDubaiMarinaApartments() {
  // Step 1: Get location ID
  const locationRes = await axios.get(
    'https://propertyfinder-uae-data.p.rapidapi.com/autocomplete-location',
    {
      params: { query: 'Dubai Marina' },
      headers: {
        'x-rapidapi-key': process.env.RAPIDAPI_KEY,
        'x-rapidapi-host': 'propertyfinder-uae-data.p.rapidapi.com'
      }
    }
  );

  const locationId = locationRes.data.data[0].id; // e.g., "50"

  // Step 2: Search for 1-2 bedroom apartments for rent
  const propertiesRes = await axios.get(
    'https://propertyfinder-uae-data.p.rapidapi.com/search-rent',
    {
      params: {
        location_id: locationId,
        property_type: 'apartment',
        bedrooms: '1,2',
        price_max: '120000',
        rent_frequency: 'yearly',
        sort: 'price_asc',
        page: '1'
      },
      headers: {
        'x-rapidapi-key': process.env.RAPIDAPI_KEY,
        'x-rapidapi-host': 'propertyfinder-uae-data.p.rapidapi.com'
      }
    }
  );

  return propertiesRes.data;
}

That's it. Clean, readable, maintainable code. No Puppeteer. No proxies. No headaches.

The Data You Get Back

Each property listing includes:

{
  "property_id": "12345678",
  "property_type": "apartment",
  "price": {
    "value": 85000,
    "currency": "AED",
    "period": "yearly"
  },
  "bedrooms": "1",
  "bathrooms": "1",
  "size": {
    "value": 750,
    "unit": "sqft"
  },
  "address": {
    "full_name": "Dubai Marina, Dubai"
  },
  "images": ["https://...", "https://..."],
  "agent_details": {
    "name": "Agent Name",
    "phone": "+971..."
  }
}

Structured, consistent, ready to use.

Beyond Basic Search

The API also gives you:

Market Intelligence

/price-trend-of-location - historical price charts by community
/property-insight - community-level analytics
/get-transactions - historical sales and rental records

Agent & Broker Data

/search-agents and /search-brokers - find real estate professionals
Direct contact info included (rare in the industry)

Commercial Properties

Office spaces, retail, warehouses, co-working spaces

New Developments

Off-plan projects and new launches

Who Should Use This

PropTech founders building UAE property platforms
Developers who need real estate data for a project
Data scientists analyzing UAE market trends
Investment firms tracking property valuations
Real estate agencies building internal tools

The Bottom Line

If you're building anything that needs UAE real estate data, stop scraping and start using an API.

Your future self will thank you.

👉 Try it: https://rapidapi.com/happyendpoint/api/propertyfinder-uae-data

🌐 Happy Endpoint: https://happyendpoint.com

Analyzed Dubai’s Property Market Using an API. Here’s What I Found.

Happy Endpoint — Sat, 04 Apr 2026 10:10:10 +0000

I spent a week pulling data from Dubai's property market.

Here's what the numbers actually say.

The Dataset

Using the Bayut API, I pulled:

10,000+ active property listings
12 months of transaction history
Data across 50+ Dubai neighborhoods
Both sale and rental markets

This is the kind of analysis that real estate consultancies charge $10,000 for. I did it with an API and a Python script.

Finding 1: Dubai Marina vs Downtown - The Price Gap Is Closing

Dubai Marina has historically been cheaper than Downtown Dubai. But the gap is narrowing.

Current average sale prices (AED per sqft):

Downtown Dubai: ~2,800 AED/sqft
Dubai Marina: ~2,400 AED/sqft
Palm Jumeirah: ~3,500 AED/sqft
Business Bay: ~1,900 AED/sqft
JVC: ~1,100 AED/sqft

Insight: JVC and Business Bay offer the best value for investors looking at rental yield.

Finding 2: Off-Plan Is Dominating

Of all new listings in the past 6 months, off-plan properties account for a significant share of the market.

Why? Payment plans. Developers are offering:

10% down payment
1% per month during construction
40-50% on handover

This makes off-plan accessible to a much wider buyer pool.

The /search-new-projects endpoint returns all active off-plan projects with completion percentage and developer info.

Finding 3: Studio Apartments Have the Best Rental Yield

When I calculated rental yield by bedroom count:

Studio: ~7-8% gross yield
1 bedroom: ~6-7% gross yield
2 bedroom: ~5-6% gross yield
3 bedroom: ~4-5% gross yield
Villa: ~3-4% gross yield

Insight: Studios are the best investment for pure rental yield. Villas are lifestyle purchases.

Finding 4: Furnished Properties Command a 20-30% Premium

Furnished apartments rent for significantly more than unfurnished ones in the same building.

The Bayut API's is_furnished filter makes it easy to compare:

import requests

def get_avg_price(location_id, furnished):
    response = requests.get(
        'https://bayut14.p.rapidapi.com/search-property',
        params={
            'purpose': 'for-rent',
            'location_ids': location_id,
            'property_type': 'apartments',
            'rooms': '1',
            'is_furnished': 'furnished' if furnished else 'unfurnished',
            'page': 1
        },
        headers={
            'x-rapidapi-host': 'bayut14.p.rapidapi.com',
            'x-rapidapi-key': 'YOUR_KEY'
        }
    )
    properties = response.data['data']['properties']
    prices = [p['price'] for p in properties if p['price']]
    return sum(prices) / len(prices) if prices else 0

furnished_avg = get_avg_price('5003', True)    # Dubai Marina
unfurnished_avg = get_avg_price('5003', False)

premium = ((furnished_avg - unfurnished_avg) / unfurnished_avg) * 100
print(f"Furnished premium: {premium:.1f}%")

Finding 5: Agent Concentration Is High

The top 10% of agents account for roughly 60% of all listings.

This creates an opportunity: most agents are underserved by technology. A simple CRM or listing management tool could capture a huge market.

The /agent-search and /agent-properties endpoints give you everything you need to build agent-focused tools.

Finding 6: Transaction Volume Peaks in Q1 and Q4

Looking at 24 months of transaction data via the /transactions endpoint:

Q1 (Jan-Mar): High volume — new year buying activity
Q2 (Apr-Jun): Moderate
Q3 (Jul-Sep): Lower - summer slowdown
Q4 (Oct-Dec): High - year-end deals

Insight: If you're building a price alert tool, set notifications to trigger in Q3 when sellers are more motivated.

How I Did This Analysis

import requests
import pandas as pd

API_KEY = 'YOUR_RAPIDAPI_KEY'
HEADERS = {
    'x-rapidapi-host': 'bayut14.p.rapidapi.com',
    'x-rapidapi-key': API_KEY
}

def get_transactions(location_id, time_period='12m'):
    all_transactions = []
    page = 1

    while True:
        response = requests.get(
            'https://bayut14.p.rapidapi.com/transactions',
            params={
                'purpose': 'for-sale',
                'location_ids': location_id,
                'time_period': time_period,
                'category_ids': 'residential',
                'page': page
            },
            headers=HEADERS
        )
        data = response.json()['data']
        transactions = data.get('transactions', [])

        if not transactions:
            break

        all_transactions.extend(transactions)
        if page >= data.get('totalPages', 1):
            break
        page += 1

    return pd.DataFrame(all_transactions)

# Analyze Dubai Marina (location ID: 5003)
df = get_transactions('5003')
print(df['price'].describe())
print(df.groupby('category')['price'].mean())

The Opportunity

Dubai's real estate market is one of the most data-rich in the world - but most of that data is locked behind portals.

The Bayut API unlocks it.

Whether you're building:

An investment analysis tool
A price comparison platform
A market research dashboard
A rental yield calculator

The data is there. You just need to use it.

Get the API

🔗 Bayut API on RapidAPI: https://rapidapi.com/happyendpoint/api/bayut14/

📖 Documentation: https://bayutapi.dev/

📩 Dataset inquiries: happyendpointhq@gmail.com

There is also bulk Bayut datasets (100K+ records) available on their website.

// Detect dark theme var iframe = document.getElementById('tweet-2038842577357881542-695'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=2038842577357881542&theme=dark" }

Happy Endpoint — real-world data APIs for developers.

🌐 https://happyendpoint.com | 🐦 https://x.com/happyendpointhq

5 PropTech Products You Can Build This Weekend Using the PropertyFinder UAE API

Happy Endpoint — Fri, 03 Apr 2026 19:43:40 +0000

The UAE real estate market is booming. Dubai alone recorded record transaction volumes in 2024, and the market shows no signs of slowing down.

For developers and indie hackers, this is a golden opportunity. The market is large, the data is rich, and there's genuine demand for PropTech tools.

The best part? You don't need months to build something useful. With the PropertyFinder UAE Data API, you can ship a working product in a weekend.

Here are 5 ideas to get you started.

1. 🏘️ UAE Property Price Tracker

What it does: Monitors price trends for specific communities and sends alerts when prices drop or rise significantly.

How to build it:

Use /price-trend-of-location to fetch historical price data for communities
Store snapshots in a database (Supabase or PlanetScale work great)
Compare current vs. previous data to detect significant changes
Send email/SMS alerts via Resend or Twilio

Endpoints used:

/autocomplete-location - let users search for communities
/price-trend-of-location - get price history
/property-insight - community-level metrics

Monetization: Freemium SaaS. Free for 3 communities, paid for unlimited tracking.

Time to build: 1-2 days

2. 🤝 UAE Real Estate Agent Finder

What it does: A directory of UAE real estate agents and brokers, searchable by location, specialty, and language.

How to build it:

Use /search-agents-detailed and /search-brokers-detailed to populate the directory
Add a search interface with location and filter options
Show each agent's active listings via /agent-properties
Let agents claim their profile (lead generation for you)

Endpoints used:

/search-agents and /search-agents-detailed
/search-brokers and /search-brokers-detailed
/agent-properties and /broker-properties

Monetization: Charge agents for featured placement or lead generation.

Time to build: 1-2 days

3. 📊 UAE Investment ROI Calculator

What it does: Helps investors calculate potential rental yield and ROI for UAE properties.

How to build it:

Let users search for properties using /search-buy
Show the purchase price alongside average rental prices for the same area from /search-rent
Calculate gross rental yield: (annual rent / purchase price) × 100
Add historical price appreciation data from /price-trend-of-location
Show comparable transactions from /get-transactions

Endpoints used:

/search-buy - purchase prices
/search-rent - rental prices for yield calculation
/price-trend-of-location - appreciation data
/get-transactions - comparable sales

Monetization: Freemium with premium reports. Affiliate commissions from mortgage brokers.

Time to build: 2-3 days

4. 🏢 UAE Commercial Property Finder

What it does: A specialized search tool for businesses looking for office space, retail units, or warehouses in the UAE.

Why this niche? Most property portals mix residential and commercial listings. A dedicated commercial tool serves a specific, high-value audience (businesses, not individuals).

How to build it:

Use /search-commercial-rent and /search-commercial-buy as the core search
Filter by property type: office-space, retail, warehouse, showroom, co-working-space
Add area filters (sqft) since commercial tenants care about space more than bedrooms
Show agent contact info prominently - commercial deals need direct contact

Endpoints used:

/search-commercial-rent
/search-commercial-buy
/property-details
/search-agents-detailed

Monetization: Lead generation for commercial real estate agents. B2B SaaS for relocation companies.

Time to build: 1-2 days

5. 🏗️ UAE Off-Plan Property Monitor

What it does: Tracks new off-plan project launches in the UAE and notifies subscribers when new developments are listed.

Why this works: Off-plan properties in Dubai often sell out within hours of launch. Investors and buyers need to be first.

How to build it:

Poll /search-new-projects regularly (daily or hourly)
Store seen project IDs in a database
When new projects appear, send notifications to subscribers
Let users filter by location, developer, price range, and property type
Show developer info from /real-estate-developers

Endpoints used:

/search-new-projects
/real-estate-developers
/autocomplete-location

Monetization: Subscription newsletter. Affiliate commissions from developers.

Time to build: 1 day

Getting the API

All five products above use the PropertyFinder UAE Data API by Happy Endpoint:

👉 https://rapidapi.com/happyendpoint/api/propertyfinder-uae-data

The API includes:

500K+ property listings
Residential, commercial, and new project data
Agent and broker directory with contact info
Market intelligence and price trends
Historical transaction records
UAE-wide coverage (Dubai, Abu Dhabi, Sharjah, and more)

The Stack I'd Use

For any of these weekend projects:

Frontend: Next.js + Tailwind CSS
Backend: Next.js API routes or a simple Express server
Database: Supabase (free tier is generous)
Deployment: Vercel (free for hobby projects)
API: PropertyFinder UAE Data API via RapidAPI

Total infrastructure cost to start: $0 (free tiers cover everything for an MVP).

Start Building

The UAE PropTech market is underserved. There's real demand for better tools, and the data is now accessible via API.

Pick one of these ideas, spend a weekend on it, and ship something.

🌐 Happy Endpoint: https://happyendpoint.com

📧 happyendpointhq@gmail.com

🔗 All APIs: https://rapidapi.com/user/happyendpoint