DEV Community

Kuroshio Data
Kuroshio Data

Posted on

Parsing Japan's EDINET: UTF-16 CSVs, tabs inside fields, and semantics hidden in the element ID

Japan has a full securities disclosure system, EDINET, run by the Financial Services Agency. It is the structural equivalent of EDGAR: every listed company files there, the filings are public, and since v2 there is a documented REST API with free API keys. On paper it should be as easy to build on as EDGAR.

In practice, English-language coverage of Japanese corporate events runs days to weeks behind, and for small caps it often never arrives at all. Having spent a while pulling this API apart, I don't think the reason is access. It's that the payload fights you in three specific ways, and only one of them is "it's in Japanese."

Getting a document at all

Two endpoints matter. documents.json?date=YYYY-MM-DD&type=2 lists everything disclosed on one day with metadata. Then documents/{docID}?type=5 returns that filing — as a ZIP archive. Inside the archive, alongside the XBRL, there's a CSV that is far easier to work with than the raw XBRL tree.

That CSV has nine columns: element ID, item name, context ID, relative fiscal year, consolidated/individual, period or instant, unit ID, unit, and value. So far, reasonable.

Then you decode it and get mojibake, because the file is UTF-16LE, not UTF-8. That one is quick to spot and quick to fix. The next one is not.

The bug I shipped

The file is tab-separated, so the obvious parser is split('\n') then split('\t'). I wrote that, tested it against several filings, saw sensible output, and shipped it.

It is wrong, and it fails silently.

Values in the last column are quoted, and inside those quotes EDINET happily includes raw tab characters and raw newlines. This is not a corner case I constructed: it shows up in the body text of extraordinary reports, where the filer has pasted a table describing a business transfer, and it can show up in the "purpose of holding" field of large-shareholding reports.

The failure mode is the nasty kind. A stray tab inside a quoted value shifts every subsequent column by one, so value reads out of the unit column and comes back empty. A stray newline splits one logical record into two malformed ones. You don't get an exception. You get a filing that parses cleanly and quietly reports nothing where the interesting text should have been. If you're eyeballing a sample of output, everything looks fine — because the records that break are precisely the wordy, unusual ones you're least likely to have in your sample.

The fix is the boring one: scan character by character, track whether you're inside quotes, handle "" as an escaped quote, and only treat a tab or newline as a delimiter when you're outside. About thirty lines. The lesson I'd actually pass on is not "write a real parser" — everyone knows that — it's that a delimiter-based parser on unfamiliar data should be treated as unvalidated until you've deliberately hunted for the longest, ugliest free-text field in the corpus and confirmed it survives.

The part where EDINET is genuinely good

Now the payoff, and it's a real one.

Japanese listed companies file an extraordinary report (臨時報告書) when something material happens: an M&A decision, a change of major shareholders or parent company, a change of representative directors, shareholders-meeting resolutions. It's the rough analogue of an 8-K.

The obvious way to classify these would be to run the Japanese body text through a model and hope. You don't have to. In EDINET's XBRL taxonomy, the element ID that carries the report body already identifies the event type. The body of a share exchange decision arrives under DecisionOnShareExchangeTextBlock. A change of major shareholders arrives under ChangesInMajorShareholderTextBlock. Voting results arrive under ResolutionOfShareholdersMeetingTextBlock.

These map one-to-one onto the items of Article 19(2) of the Cabinet Office Ordinance — the legal list of things that trigger the filing in the first place. The regulation's structure is carried through into the taxonomy, so classification is a dictionary lookup and it is exact. No NLP, no confidence score, no drift.

For a sense of what actually flows through: across 94 corporate extraordinary reports I measured over six days in late July 2026, the most common events were 23 shareholders-meeting resolutions, 19 stock option issuances, 17 parent-or-subsidiary changes, 18 significant-financial-event disclosures (impairments, debt waivers, special losses), and 7 major shareholder changes, with M&A decisions — share exchanges, business transfers, subsidiary acquisitions, splits — making up a long tail of one to three each.

Those numbers sum to more than 94 on purpose. One report can carry several event elements at once, so a single filing shows up under multiple categories. If you're aggregating, count filings and events separately or you'll double-count your way into a phantom M&A wave.

Volume runs roughly 15–50 per business day, spiking past 150 in late June when AGM voting results land all at once.

The English-name asymmetry

Here's a structural quirk that took me a while to understand, and that I think is the real reason naive translation pipelines produce garbage here.

EDINET knows the official English name of every entity registered as a filer. So for extraordinary reports, where the company itself is the filer, you get its real registered English name — not a machine translation, the name the company itself uses.

But large-shareholding reports (Japan's 5% rule, the 13D/13G analogue) are filed by the holder, not the issuer. So you get the holder's English name, and the company being accumulated appears only as Japanese text plus a securities code. There is no reliable English name for it in the filing.

The temptation is to machine-translate the Japanese company name. Don't. Japanese corporate names are full of traps — the same characters have multiple valid readings, and the company's own chosen romanization frequently isn't any of them. The securities code is stable, unambiguous, and the thing you should be joining on anyway. Emitting the Japanese name plus the code is the honest output; inventing an English name is how you end up confidently wrong about which company just got a new 8% holder.

What the 5% filings actually tell you

One detail worth knowing if you ever look at these. Under Japan's rules, a holder crossing 5% must state the purpose of holding, in prose. Buried in that prose is a specific legal term, 重要提案行為 — "acts of important proposal." It isn't decorative. Declaring it changes which filing regime the holder is under, because it signals intent to push management on things like board composition or capital policy.

Which means the single most useful activist signal in the corpus is a legal phrase, not a sentiment. You can match on it exactly. A fund you've never heard of that declares it is worth more attention than a famous name that files a routine passive-investment purpose.

Versus EDGAR

For contrast, the U.S. side is easier in every mechanical way. SEC EDGAR needs no API key and no registration, the data is public domain, daily indexes are plain text, and Form 4 ownership documents are clean XML. The tradeoffs are volume — a busy day is 500 to 2,000 Form 4s — and that the interesting bit is a flag rather than prose: whether a trade ran under a pre-scheduled Rule 10b5-1 plan, which separates routine sales from discretionary ones.

Japan gives you fewer filings with more semantics baked into the schema. The U.S. gives you more filings with cleaner mechanics. Both are free and official, and neither is what most people building "financial data" products are actually scraping.


Everything above is doable with the official API and a free key; nothing here needs a vendor. I do also run hosted versions that emit this as English JSON — Japan 5% / activist filings, Japan corporate events, SEC Form 4 — but the parsing notes are the part I'd have wanted to read six weeks ago.

— kuroshio-data

Top comments (0)