DEV Community: uzie

An IDX/BEI financial panel: 4,408 company-years, 22 columns, and the caveats that matter

uzie — Sat, 07 Nov 2026 13:17:27 +0000

Indonesian listed-company data is public. Analysis-ready it is not.

The Indonesia Stock Exchange (IDX/BEI) publishes financial statistics for every listed
issuer, and anyone can open the pages one by one. The friction starts when you need a
panel: the same company across five years, with consistent ratios, in one table you can
feed to R or Stata. That is a data-engineering problem disguised as a download.

I built that panel: 4,408 company-year observations, 975 tickers, 11 IDX-IC sectors,
22 columns, 2021–2025. This post documents what broke, what I normalized, and what the
numbers actually mean — including the parts that make the dataset weaker than a sales page
would suggest.

Five things that make IDX data awkward

1. Each yearly bucket is a snapshot, not a fiscal year. When you pull a year, you get
whatever statement IDX had at collection time — predominantly the Q3 (30 September) report.
The column fs_date is in the dataset for exactly this reason. If your research requires
audited full-year figures, this panel is not a substitute.

2. Ratio scales were not stable. For 2021–2022 the source stores ROA, ROE, and NPM as
decimals; from 2023 onward they are percentages. Mixing them silently produces averages
that are wrong by two orders of magnitude. I normalized 2021–2022 ×100 and kept the flag in
the codebook. Those two years remain preliminary: they have not been reconciled line by
line against audited reports.

3. The panel is unbalanced. Coverage grows as more companies list:

Year	Observations
2021	769
2022	828
2023	906
2024	947
2025	958

That growth is real, not a bug. It also means naive year-over-year comparisons mix
composition changes with performance changes. Fixed-effects models handle it; a simple
mean by year does not.

4. Missing is not zero. An empty cell means IDX did not publish that figure. Filling it
with 0 destroys every ratio built on it. The panel keeps nulls, and the codebook says so.

5. Valuation multiples move with the snapshot. per and pbv are trailing values that
depend on the share price on collection day. They are useful as context, not as a clean
time series.

What is in the panel

Each row is one company-year:

Identity: kode_saham, nama_emiten, sektor, subsektor, tahun, fs_date, audit (U/A), sharia
Profitability and leverage: roa_pct, roe_pct, der, npm_pct
Per-share and valuation: eps_idr, book_value, pbv, per
Size: assets_miliar_idr, liabilities_miliar_idr, equity_miliar_idr, sales_miliar_idr, profit_attr_owner_miliar_idr, firm_size_lnassets

The 11 sectors follow IDX-IC: Financials, Energy, Basic Materials, Industrials, Consumer
Non-Cyclicals, Consumer Cyclicals, Healthcare, Properties & Real Estate, Technology,
Infrastructures, and Transportation & Logistic.

Design choices I would defend

Keep the source date and the audit flag. A panel without fs_date and audit invites
the reader to assume audited December figures. Most rows are neither.

Normalize loudly. The ×100 correction is documented in the codebook and repeated in the
listing. A silent fix is worse than a visible caveat.

Do not coerce missing values. null propagates honestly; 0 invents a fact.

Separate portal-computed multiples from your own. pbv and per come from the source
and will disagree at the edges with price / book value computed later. Label which is
which.

What it is good for — and what it is not

Good for: panel regressions with firm and year effects, sector comparisons, teaching
econometrics with real data, and quick descriptive work on Indonesian issuers.

Not good for: audited full-year analysis, investment advice, event studies that need
announcement dates, or market-wide estimates from a 25-row sample. A sample validates a
schema; it cannot estimate a market.

Inspect it first

A free 25-row sample with all 22 columns is here — no signup:

Sample CSV: https://fauzimatakupan.github.io/indonesian-financial-data/downloads/bei-panel-sample-2026-11-07.csv
Repository and column notes: https://github.com/FauziMatakupan/indonesian-financial-data

The sample uses 25 well-known tickers and the latest year, with values copied verbatim.
If the schema fits your work, the full panel adds all 4,408 rows, a quarterly version
(16,138 observations across 20 periods), per-sector files, a codebook, and runnable
R/Stata/SPSS syntax.

Disclosure: I assembled this dataset and sell the full version. The sample and the
documentation above are free, and the limitations are the same in both.

Dubai property listings: a schema for price-per-sqm analysis

uzie — Sat, 07 Nov 2026 08:57:01 +0000

Dubai property data looks easy until you try to compare two listings. One portal shows
"2BR, Dubai Marina, AED 2,100,000"; the next shows "2 Beds, 1,180 sqft, Marina, 2.1M".
Same building, different fields, no shared key. Multiply that by a few thousand listings
and any price-per-sqm analysis becomes a manual spreadsheet exercise.

This post documents the schema I settled on for Dubai listings, the parts that actually
matter for analysis, and a free 40-row sample you can inspect.

Why listing data is harder than it looks

The listing id is not the property id. The same unit can be re-listed by two brokers with different ids. Dedupe on the portal id, then cluster on coordinates + size + bedrooms if you need property-level counts.
Price is not comparable across transaction types. A sale price and a yearly rent both look like integers. Without transactionType and pricePeriod, averages are meaningless.
Size is published in both units, inconsistently. Some records carry sqft, some sqm, some both rounded differently. Keep both and store the conversion you used.
Community metrics are snapshots. "Average community price" is computed by the portal at collection time. It is useful context, not a time series.
Broker contact data is personal data. If you publish a sample, strip names, phones, emails, and full descriptions. The listing facts are enough for analysis.

The schema

Each record is one listing:

Field	Why it matters
`listingId`	Stable dedupe key across runs
`url`	Canonical source for verification
`price`, `currency`, `pricePeriod`	Comparable only when read together
`transactionType`, `propertyType`	Sale vs rent; apartment vs villa
`bedrooms`, `bathrooms`	Studio = `bedrooms: 0`, not null
`sizeSqft`, `sizeSqm`	Both units, no silent conversion
`pricePerArea`	Portal value when present; `null` otherwise
`city`, `community`, `subCommunity`	Location hierarchy for grouping
`completionStatus`	Ready vs off-plan changes the price logic
`isVerified`, `isExclusive`, `isNewConstruction`	Source flags, not guarantees
`latitude`, `longitude`	Approximate; useful for cluster checks
`listedDate`, `scrapedAt`	Snapshot integrity
`communityAvgPrice`, `communityPriceTrend12mPct`, `communityActiveListings`	Context at collection time
`rera`	Permit string when the portal publishes it

Two design choices I would defend:

Never coerce a missing price to zero. A zero in a price column silently destroys every average. Use null and filter explicitly.
Keep portal-computed fields separate from your own. pricePerArea from the portal and price / sizeSqft from your pipeline will disagree at the edges; label which is which.

The sample

A real 40-row snapshot collected on 6 November 2026 from public PropertyFinder search
results (sale and rent). Broker names, emails, phones, and descriptions are removed.

The sample is small on purpose. Forty rows validate a schema; they cannot estimate the
Dubai market.

Limitations and ethics

Public listings only. No login, no private data, no broker contact details.
Asking prices, not transaction prices.
Listings change daily; a snapshot is not a feed.
Respect the portal's terms and applicable data-protection rules. Do not republish personal data.

What I packaged

I turned the collection into an Apify Actor so pagination, normalization, and dedupe are
reusable:
https://apify.com/matakupanlabs/dubai-property-listings-scraper

Disclosure: this is my Actor and it is pay-per-listing. The sample and schema are free.
If you are building something similar, I am happy to compare notes on deduplication or
price-per-sqm normalization.

Scraping OLX Indonesia past Akamai: architecture notes and a 25-row sample dataset

uzie — Tue, 03 Nov 2026 11:59:45 +0000

OLX Indonesia (olx.co.id) is one of the largest classified marketplaces in Southeast Asia, but it is also one of the harder Indonesian sources to collect reliably. Search and listing pages sit behind an Akamai interstitial, and the data you want is not in the initial HTML — it is fetched by the page's own JavaScript after the challenge is cleared.

This article documents the architecture that worked for me, the normalized schema I settled on, and a small real sample dataset you can inspect. It is written for data engineers who need Indonesian marketplace data and for anyone evaluating whether a scraper is worth building.

Why plain HTTP and headless browsers fail

The first attempts were the obvious ones:

plain fetch / requests — returned the interstitial or an empty shell;
server-side rendered parsing — the listing JSON was not in the document;
headless Chrome without a real browsing context — the challenge sometimes passed, but the listing request was never captured correctly.

The pattern is familiar: the challenge is not the final boss. The final boss is the JSON request the page makes after the challenge, and that request only exists if the page is allowed to run like a normal browser.

What worked: run a real browser, intercept the response

The working approach was:

Launch a real browser context with a normal desktop fingerprint.
Navigate to the public search URL.
Let the page solve the interstitial naturally — no captcha-solving service, no login.
Intercept the JSON response the page's own frontend fetches.
Normalize the response into a flat record.

A simplified sketch:

page.on('response', async (response) => {
  const url = response.url();
  if (!url.includes('/api/') && !url.includes('listing')) return;
  const type = response.headers()['content-type'] || '';
  if (!type.includes('application/json')) return;

  const payload = await response.json();
  // map payload -> normalized listing
});

The important part is not a secret endpoint. It is the order of operations: real browser → challenge → page JavaScript → response interception. A proxy alone does not fix this; a different user agent does not fix this; parsing harder does not fix this.

The schema

I normalized each listing into a flat record. The exact fields depend on the marketplace, but this shape has been stable:

Field	Description
`adId`	Listing ID (stable key)
`name`	Listing title
`adDescription`	Seller-written description
`price`	Asking price in IDR (integer)
`priceDisplay`	Price as displayed
`condition`	`Bekas` / `Baru` when provided
`itemType`, `categoryId`	Marketplace taxonomy
`country`, `region`, `city`, `district`	Location hierarchy
`sellerType`, `sellerKyc`	Seller account type / platform flag
`favorites`	Saves at collection time
`createdAt`	Listing creation timestamp
`isNew`, `status`	Listing flags
`url`	Canonical public listing URL
`query`	Search query used

Two design choices worth calling out:

Keep the marketplace ID as the stable key. Titles change, prices change, and listings get republished. adId is what lets you deduplicate across runs.
Separate the display price from the numeric price. priceDisplay is useful for QA; price is what your pipeline should aggregate.

The sample dataset

I published a real 25-row snapshot (not synthetic) collected on 3 November 2026 from a public search for mechanical keyboard:

The sample omits seller names and exact coordinates. Descriptions are seller-written and may contain typos. Prices are asking prices, not transaction prices. It is a schema sample, not a representative market sample.

Limitations and ethics

Public data only. No login, no private messages, no seller contact details.
Respect the platform. Read the terms of service and applicable data-protection rules (UU PDP in Indonesia). Do not republish personal data.
Expect drift. Classified sites change their frontend often. A schema that survives a redesign is worth more than a clever selector.
Small samples lie. Twenty-five rows can validate a schema; they cannot estimate a market.

What I packaged

I turned this architecture into an Apify Actor so the browser lifecycle, challenge handling, pagination, and normalization are reusable:

https://apify.com/matakupanlabs/olx-indonesia-listings-scraper

Disclosure: this is my Actor and it is pay-per-listing. The sample dataset and schema notes are free. If you are building something similar, I am happy to compare notes on fields, deduplication, or handling large runs.

What would you add?

For marketplace price monitoring, the interesting question is time: is createdAt enough, or do you need first-seen / last-seen fields to build a real price time series? I would like to hear how other people model republished listings and seller churn.

Repository

The schemas and sample datasets also live in a public GitHub repository:

https://github.com/FauziMatakupan/indonesian-marketplace-data

It includes JSON Schema files, ethics notes, and a landing page:
https://fauzimatakupan.github.io/indonesian-marketplace-data/