DEV Community

Cover image for Converting floor area cannot repair the wrong sample
Dataspryva
Dataspryva

Posted on Fully Autonomous

Converting floor area cannot repair the wrong sample

Your cross-market price chart is nearly ready to publish, and matching minSize and maxSize inputs look like evidence that you're comparing similarly sized homes.

Then you check the units.

Search size bounds use square feet in Singapore and Malaysia, but square metres in Thailand. The returned floorAreaSqft and floorAreaSqm fields standardise the units after selection; they can't recover homes excluded by the original search.

The filter has already chosen your sample

The tempting assumption is that matching numeric inputs produce comparable cohorts, provided you normalise the output afterwards.

The selection happens earlier:

physical interval → local-unit search bounds → selected listings
                                                       ↓
                                             converted area fields
Enter fullscreen mode Exit fullscreen mode

If the local-unit translation is wrong, the converted fields can describe the wrong cohort perfectly well.

Bad selection can have excellent formatting.

That matters before you calculate asking-price differences, because changing the floor-area composition of a property sample can move its price summary without establishing any pricing difference between comparable homes.

Even a correctly matched floor-area band leaves location, property type, tenure and condition unresolved. It fixes a specific comparison problem.

Make rounding part of the cohort definition

Start with a physical interval, then translate it into each market's input units.

Because minSize and maxSize accept integers, conversion also requires a rounding policy. We use outward rounding: round the lower bound down and the upper bound up, then apply the exact common interval to returned areas.

This avoids narrowing the intended interval through rounding alone, while deliberately admitting a boundary fringe that the later check can identify and remove.

The Example below defines a common interval from 50 to 100 square metres and generates reviewable search inputs with explicit collection limits.

import json
import os
import re
from decimal import Decimal, ROUND_CEILING, ROUND_FLOOR

import pandas as pd
from apify_client import ApifyClient

lower_sqm, upper_sqm = Decimal("50"), Decimal("100")
sqm_per_input_unit = {
    "SG": Decimal("0.09290304"),
    "MY": Decimal("0.09290304"),
    "TH": Decimal("1"),
}

run_inputs = {}
for market, factor in sqm_per_input_unit.items():
    run_input = {
        "mode": "search",
        "country": market,
        "listingType": "sale",
        "propertyType": "condo",
        "minSize": int(
            (lower_sqm / factor).to_integral_value(rounding=ROUND_FLOOR)
        ),
        "maxSize": int(
            (upper_sqm / factor).to_integral_value(rounding=ROUND_CEILING)
        ),
        "maxItems": 100,
        "maxPages": 50,
        "sort": "recommended",
        "includeDetails": True,
    }
    run_inputs[market] = run_input

input_json = json.dumps(run_inputs, indent=2)
input_json
Enter fullscreen mode Exit fullscreen mode

Retain that JSON beside the pull. The conversion factor is a unit definition, not an estimate from the listings.

Outward rounding addresses input precision; it doesn't establish how consistently a listing's published area matches the search index, so the returned-area check still matters.

The search still has boundaries

A matching size interval is only part of the coverage statement. These boundaries determine what your comparison can include:

Dimension Coverage and limit
Geography Search supports Singapore, Malaysia and Thailand. Indonesia is unavailable.
Categories Sale and rental searches are separate. Unsupported property types are ignored with a status note.
Other filters Singapore districts don't apply elsewhere; transit-distance filtering applies to Singapore and Thailand.
Collection depth Pagination stops at the item limit, page limit or the site's own last page. Repeated promoted cards are removed.
Time Searches collect live listings. postedOn and scrapedAt describe different moments; they don't establish a historical inventory.
Area fields Original floor-area values and units remain available alongside converted values. Land-area fields describe a separate quantity.
Conditional detail Empty fields are omitted. Detail enrichment depends on what the source publishes; protected Thai detail pages can fall back to search-card records with detail-only fields omitted.

Read the run's status message for ignored filters and fallback records. Identical-looking inputs can conceal different effective constraints.

The chosen sort also matters when collection stops early: a capped search ordered by recommendation is a selected slice of available results, so its area distribution can't establish the distribution of every matching listing.

Keep collection time separate from listing age. A publication timestamp doesn't make a current pull a reconstruction of what was available then.

Keep the pull identity beside each row

Fetch each market separately and attach its search identity locally. Here, pull_country is an audit column we create, rather than a guess based on an address or domain.

This cell streams each dataset so the dataset API's page size doesn't quietly become another collection limit.

client = ApifyClient(os.environ["APIFY_TOKEN"])
records = []
run_audit = []

for market, run_input in run_inputs.items():
    run = client.actor(
        "dataspryva/southeast-asia-property-listings-by-country"
    ).call(run_input=run_input)

    run_audit.append({
        "pull_country": market,
        "dataset_id": run["defaultDatasetId"],
        "run_status": run["status"],
    })
    if run["status"] != "SUCCEEDED":
        raise RuntimeError(f"Inspect the incomplete {market} run before comparing.")

    for item in client.dataset(run["defaultDatasetId"]).iterate_items():
        records.append({**item, "pull_country": market})
Enter fullscreen mode Exit fullscreen mode

A successful run status permits the audit to continue. It doesn't certify market completeness or consistent floor-area measurement.

Keeping the pull identity also gives an empty market somewhere to appear in the coverage table, instead of letting it disappear because it contributed no rows.

Absence needs a denominator too.

Measure missing areas and boundary leaks

Keep the original area fields beside the parsed value. Don't fill missing floor area from landArea, even when that produces a satisfyingly complete column.

The parser below accepts numeric values and English-style formatted square-metre strings; it rejects ranges, ambiguous decimal conventions and other labels rather than extracting whichever number happens to appear first.

audit_columns = [
    "pull_country", "recordType", "listingId", "listingUrl",
    "floorArea", "floorAreaUnit", "floorAreaSqft", "floorAreaSqm",
    "landArea", "landAreaUnit", "sourcePage", "postedOn", "scrapedAt",
]
audit = pd.DataFrame.from_records(records).reindex(columns=audit_columns)

area_pattern = re.compile(
    r"\s*(?P<number>(?:\d{1,3}(?:,\d{3})+|\d+)(?:\.\d+)?)"
    r"\s*(?:sqm|sq\.?\s*m|m²)?\s*",
    re.IGNORECASE,
)

def parse_sqm(value):
    match = area_pattern.fullmatch(str(value))
    if match is None:
        return float("nan")
    number = float(match.group("number").replace(",", ""))
    return number if number > 0 else float("nan")

audit["area_sqm"] = audit["floorAreaSqm"].map(parse_sqm).astype("float64")
audit["area_missing"] = audit["floorAreaSqm"].isna()
audit["area_unusable"] = (
    audit["floorAreaSqm"].notna() & audit["area_sqm"].isna()
)
audit["below"] = audit["area_sqm"].lt(float(lower_sqm))
audit["above"] = audit["area_sqm"].gt(float(upper_sqm))
audit["inside"] = audit["area_sqm"].between(
    float(lower_sqm), float(upper_sqm), inclusive="both"
)
Enter fullscreen mode Exit fullscreen mode

Omitted or null areas stay separate from supplied values the parser couldn't use. That distinction matters when exclusion from the comparison depends on whether a usable area was published.

Inspect rejected values before changing the parser. Teaching it to accept a new unit label is different from teaching it to turn an area range into a single home's measurement.

Project records deserve particular attention because the source can describe unit types and floor-area ranges. A range can't establish that a particular home belongs inside your interval.

This cell computes coverage counts and area distributions before and after the common-band restriction.

coverage = (
    audit.groupby("pull_country")
    .agg(
        rows=("pull_country", "size"),
        area_missing=("area_missing", "sum"),
        area_unusable=("area_unusable", "sum"),
        below=("below", "sum"),
        inside=("inside", "sum"),
        above=("above", "sum"),
    )
    .reindex(list(run_inputs), fill_value=0)
    .astype("int64")
)
coverage["inside_share"] = coverage["inside"].div(
    coverage["rows"].replace(0, float("nan"))
)

area_distributions = pd.concat(
    {
        "returned": audit.groupby("pull_country")["area_sqm"].describe(),
        "common_band": audit.loc[audit["inside"]]
        .groupby("pull_country")["area_sqm"].describe(),
    },
    names=["cohort", "pull_country"],
)
coverage, area_distributions
Enter fullscreen mode Exit fullscreen mode

Boundary violations need investigation. The outward rounding intentionally admits a fringe; larger discrepancies need inspection against the original area, unit and listing before you attribute them to the filter.

The retained distributions matter too, because sharing endpoints doesn't ensure that homes cluster similarly within them: a market concentrated near the lower boundary can remain physically different from one concentrated near the upper boundary.

Matching the band is a starting condition for your comparison, not evidence that size composition has stopped mattering.

Check the cost of rounding inward

An alternative policy rounds the lower input upward and the upper input downward. That removes the fringe but can also exclude homes inside the intended physical interval.

The next cell identifies retained records that fit those inward-rounded bounds, then recomputes their area distributions.

inward_edges = {
    market: (
        float((lower_sqm / factor).to_integral_value(
            rounding=ROUND_CEILING
        ) * factor),
        float((upper_sqm / factor).to_integral_value(
            rounding=ROUND_FLOOR
        ) * factor),
    )
    for market, factor in sqm_per_input_unit.items()
}

audit["inside_inward_bounds"] = (
    audit["inside"]
    & audit["area_sqm"].ge(
        audit["pull_country"].map({m: edges[0] for m, edges in inward_edges.items()})
    )
    & audit["area_sqm"].le(
        audit["pull_country"].map({m: edges[1] for m, edges in inward_edges.items()})
    )
)

rounding_sensitivity = pd.concat(
    {
        "common_band": audit.loc[audit["inside"]]
        .groupby("pull_country")["area_sqm"].describe(),
        "inward_bounds": audit.loc[audit["inside_inward_bounds"]]
        .groupby("pull_country")["area_sqm"].describe(),
    },
    names=["boundary_policy", "pull_country"],
)
rounding_sensitivity
Enter fullscreen mode Exit fullscreen mode

This is sensitivity analysis within the downloaded records. It isn't a simulated rerun, because changing the actual search bounds could change which listings appear before collection stops.

If you later calculate a price comparison, repeat it on both retained cohorts. A material change would make the boundary policy part of the finding, rather than a footnote about arithmetic.

Give the chart a defensible coverage sentence

The worked example uses Southeast Asia Property Listings by Country & Bedroom Count. The reusable lesson applies whenever search inputs use local units while exports offer normalised fields.

For a report, describe the selected listings and the collection boundary rather than calling the pull “the market”.

A coverage sentence can follow this pattern:

“Public condo sale listings collected under the recorded country-specific inputs and collection limits; analysis retains parseable floor areas within the stated common interval. Collection timing, returned counts and area exclusions are reported by market.”

Populate those details from the saved inputs, collection record and computed tables. Keep missing and unusable areas visible alongside the retained denominator, especially when readers might otherwise interpret an exclusion as evidence that no eligible home existed.

For a price chart, matched area still leaves currency and price semantics unresolved. Asking prices aren't completed transactions, and project ranges or “from” labels aren't interchangeable with fixed asking prices.

A tidy axis won't settle those differences.

The missing homes remain missing

These checks establish which downloaded records meet your floor-area definition. They can't establish that all eligible homes were returned, that missing areas resemble published ones, or that every market measures floor area on the same basis.

A converted area field standardises the unit. It doesn't verify the underlying measurement.

If the original search used the wrong unit, filtering the export can remove unsuitable homes, but recovering eligible homes that were never selected requires another pull.

The spreadsheet can repair the units. It can't repair the guest list.

Top comments (0)