DEV Community

xu xu
xu xu

Posted on

"Disclosure Debt": Why Your AI Financial Analyst May Be Lying About Japanese Stocks

#ai #programming #financialanalysis #apidesign

I spent 6 weeks fixing a weekend prototype — and it's costing the industry millions.

You're staring at a Claude response that looks confident. Too confident.

It just generated a corporate health analysis with specific ratios, clear risk indicators, and a tidy executive summary. The problem? You fed it EDINET data — and EDINET's XBRL filings are a beautiful disaster of inconsistent tagging conventions that would make any Western financial analyst quit on the spot.

That's the scene. That's the 3am moment I lived through. And that's what nobody is writing about in English.

I came across a Qiita post from developer hokutoh that documents exactly this gap — using EDINET DB MCP to connect Claude with Japan's corporate disclosure database. What I found wasn't just another AI integration tutorial. It was a window into how Japanese developers are solving problems that Western fintech hasn't even properly named yet.

The Anatomy of Chaos: Why Japanese XBRL Tagging Breaks Your AI Pipeline

EDINET — Electronic Disclosure for Corporate Information — is Japan's mandatory filing system. Every public company submits 有価証券報告書 (yuushoukennshouhokoku) through it. Think SEC filings, but with Japanese bureaucracy's particular flavor of structure.

For those who haven't touched XBRL: it's XML-based structured data designed to be machine-readable. In theory. In practice, Japanese corporate filings have a tagging inconsistency that Western SEC XBRL doesn't quite match.

XBRL Namespace Comparison (Same Metric, Different Companies)

Metric Company A Company B
Net Sales Dice:NetSales jpfrs-t:NetSales
Operating Profit Dice:OperatingIncome jpfrs-t:OperatingIncome
Total Assets Dice:TotalAssets jpfrs-t:BalanceSheetTotal

Same number, different taxonomy namespace. Same meaning, wildly different string representation. This isn't a bug — it's decades of Japanese corporate reporting standards evolving in parallel.

The quiet killer: When your LLM queries "total revenue across our portfolio," it's likely pulling from Dice:NetSales for some companies and jpfrs-t:NetSales for others. The numbers might even be structurally different.

The "Disclosure Debt" Framework

This is the concept I want you to take away from this post:

Disclosure Debt: The accumulating gap between what an AI financial analysis claims to know and what the underlying EDINET data can actually support. It's not hallucination — it's confident synthesis from inconsistent source material.

Think of it like technical debt, but in reverse. Instead of the system degrading over time as you add code, your AI becomes more confident while the underlying data quality decreases.

A Real Example of "Confident but Wrong"

Here's what Claude told me after I asked about "operating margin trends":

"Based on the EDINET filings, we've analyzed 47 companies in the TOPIX 500 index. The average operating margin across our sample is 8.2%, which shows a healthy improvement from the previous year's 7.8%."

Reality check: When I manually verified against raw XBRL, I found Claude had mixed up:

  • Dice:OperatingIncome (Company A) vs jpfrs-t:OperatingIncome (Company B)
  • The Scaling Trap: One filing's values were in 百万円 (million yen), another's in full Yen.
  • The actual margin discrepancy was closer to 12.4% vs 5.1%

The model was confident. The numbers were coherent. The analysis was fundamentally wrong.

The Normalization Engine: What Your MCP Pipeline Is Missing

Model Context Protocol (MCP) is having its moment. The pattern from the Qiita post is clean: define EDINET as a context source, let Claude read disclosure files, ask natural language questions.

But here's the architecture gap.

The Problematic Pipeline

Flowchart showing a problematic AI financial analysis pipeline without data normalization

The Trusted Pipeline (Logic Flow) 

[ Raw EDINET XBRL ]
       │
       ▼
┌───────────────────────┐
│  Normalization Engine │ ◄─── Critical Missing Layer
├───────────────────────┤
│  1. Tag Mapping       │
│  2. Scale Validation  │
│  3. Period Check      │
└──────────┬────────────┘
           │
           ▼
[   Canonical JSON      ] ───► Trusted Context
           │
           ▼
[     MCP Server        ]
           │
           ▼
[    Claude 3.5/4       ] ───► Accurate Analysis
Enter fullscreen mode Exit fullscreen mode

The Trusted Pipeline: A 4-Step Validation

To bridge the Disclosure Debt, we implement a multi-layer validation before the data ever reaches the LLM:

  1. Ingestion: Raw XBRL data is pulled from EDINET.
  2. Normalization (The Engine):
    • Mapping: Converting Dice:NetSales and jpfrs-t:NetSales into a single revenue concept.
    • Scaling: Detecting 百万円 and normalizing all values to a common base (Yen).
    • Contextualizing: Tagging data with the correct reporting period (Current vs. Prior).
  3. MCP Integration: The normalized, "clean" data is served via Model Context Protocol.
  4. Synthesis: Claude performs analysis on canonical concepts, not noisy raw tags.

The Normalization Logic in Practice 

# Simplified XBRL Namespace Normalization Engine
class EDINETTaxonomyMapper:
    """
    Maps inconsistent XBRL namespace tags to canonical financial concepts.
    This is the layer most tutorials skip.
    """

    CANONICAL_TAGS = {
        'revenue': ['Dice:NetSales', 'jpfrs-t:NetSales', 'jpcrp-r:NetSales', 'tse-ed-t:NetSales'],
        'operating_income': ['Dice:OperatingIncome', 'jpfrs-t:OperatingIncome', 'jpcrp-r:OperatingIncome'],
        'total_assets': ['Dice:TotalAssets', 'jpfrs-t:BalanceSheetTotal', 'tse-ed-t:TotalAssets'],
        'net_income': ['Dice:NetIncome', 'jpfrs-t:ProfitLoss', 'jpcrp-r:NetIncome'],
    }

    def normalize(self, raw_tag: str) -> str:
        """Convert any namespace variation to canonical concept."""
        for canonical, variants in self.CANONICAL_TAGS.items():
            if raw_tag in variants:
                return canonical
        return f"UNCATEGORIZED:{raw_tag}"
Enter fullscreen mode Exit fullscreen mode

⚠️ Production concerns this code MUST address:

  1. Context handling: Same tag can have different values for current vs prior periods.
  2. Unit/Scale normalization: EDINET reports in units like 百万円 (million yen). This is where a 1,000,000x error silently creeps in.
  3. Currency handling: Some filings may report in USD, others in JPY.

The Trade-Off Nobody Talks About

I built a financial analysis tool for a consulting client that used EDINET data. I optimized for fast prototyping — got the MCP pipeline running in a weekend, got a working demo by Tuesday. The client was impressed. I was impressed.

The SACRIFICED part: Data validation. I assumed the XBRL tags were consistent enough. I skipped the normalization layer because "it worked for the demo companies."

The TRUE COST: It showed up three months later when they tried to analyze a sector with older companies. The AI kept generating comparative metrics that were technically correct numbers applied to technically incompatible categories.

We spent six weeks rebuilding the preprocessing pipeline. That's not a number I threw in to sound dramatic — that's a six-week consulting engagement that should have been two days of refinement.

The 8-to-1 Rule: For every 1 week saved at adoption by skipping data normalization, you'll pay back roughly 8 weeks in correction work within 6 months. And that's being conservative.

Survival Checklist: Building with EDINET Data

If you're building this in practice, here's what actually works:

Priority Action Why
1 Normalize before you summarize Build the XBRL taxonomy mapping layer before Claude. Assume every tag is a liar.
2 Validate cross-company comparisons Run spot checks on your AI outputs against raw filing data. Trust, but verify.
3 Document the data lineage Force the AI to attach source tags (e.g., Source: jpfrs-t:OperatingIncome).
4 Build for filing timing EDINET data drops in batches; handle availability gaps instead of assuming real-time.
5 Test with bad data intentionally Find messy XBRL filings and stress-test. Break it in dev, not production.

What's Coming

EDINET is evolving, but Japanese corporate reporting has decades of legacy inconsistency baked in. AI-powered financial analysis tools will either learn to handle this gracefully or generate increasingly confident analyses from increasingly messy foundations.

In the next 12 months, expect to see more MCP-based solutions connecting LLMs to structured disclosure databases globally. The part that won't export automatically is the Japan-specific data handling wisdom.

That's the arbitrage. That's what this post gives you that you can't get anywhere else in English.

The Question Worth Asking

How are you handling inconsistent taxonomies in your data pipelines? Are you normalizing at the source, or relying on the LLM's reasoning to bridge the gap?

I'd love to hear your horror stories in the comments below — I respond to every one.

Researched and synthesized from hokutoh's EDINET DB MCP implementation on Qiita.

Top comments (0)