D-MO (Data Micro-Optimizer)

In day-to-day software development and data analysis, preparing and cleaning financial files is often one of the most repetitive and error-prone tasks. Dealing with rigid layouts, misaligned rows, and column names that change without warning creates constant operational friction.

To solve this problem at its root—while maintaining a strict focus on information security—I developed D-MO (Data Micro-Optimizer), a powerful ETL (Extract, Transform, Load) web-based processing tool that runs entirely on the client side.

The Origin: Privacy by Design

When handling banking reports or sensitive financial information, uploading them to external conversion platforms represents a critical security risk. D-MO was built on the principle of absolute privacy: all processing takes place locally in the browser's memory on the client side. Structured data is never sent to an external server, eliminating network latency and providing a zero-server-overhead environment.

Pipeline Architecture (Data Flow)

The system processes information sequentially through a decoupled pipeline, making it possible to transform complex files into production-ready datasets with a single click.

   [ Local File ] (.csv / .xlsx / .xlsb)
         │
         ▼
   ┌──────────────┐
   │   DropZone   │ ◄── File Extension and Size Validation
   └──────┬───────┘
         │ (Buffer / Plain Text)
         ▼
   ┌──────────────┐
   │ File Parser  │ ◄── Delimiter Detection and Dynamic Header Resolution
   └──────┬───────┘
         │ (Normalized JSON)
         ▼
   ┌──────────────┐
   │  ETL Engine  │ ◄── Business Rules, Alias Mapping, and CUSTOM Filters
   └──────┬───────┘
         │ (Clean Dataset)
         ▼
   ┌──────────────┐
   │ Export File  │ ◄── Generation of Clean, Ready-to-Use Reports
   └──────────────┘

Technical Core and System Layers

The application is built with Next.js 14 (App Router) and TypeScript, with its internal logic divided into three main components:

1. Interface and Coordination (page.tsx)

It acts as the orchestrator of the data lifecycle. It captures uploaded files through a drag-and-drop interface (DropZone), invokes the parsing utilities, feeds the transformation engine with the current state, and dynamically updates the UI metrics.

2. Parsing and Serialization Layer (file-parser.ts)

• Uses the xlsx (SheetJS) library to analyze workbooks, normalize raw data, and dynamically identify the correct header row in complex Excel files.
• Implements papaparse for fast in-memory parsing and automatic delimiter detection in CSV files.

3. Variation-Tolerant Rules Engine (etl-engine.ts)

This is the brain of D-MO. It uses a robust matching system based on string normalization functions. This allows the engine to identify dynamic headers through a flexible alias catalog, processing layouts with subtle variations in column names without breaking the transformation workflow.

// Enfoque conceptual del mapeo dinámico y tolerante a alias
export function resolveColumnHeader(header: string, aliasMap: Record<string, string[]>): string | null {
    const normalizedTarget = header.toLowerCase().trim().replace(/[\s_-]/g, '');

    for (const [key, aliases] of Object.entries(aliasMap)) {
        if (aliases.some(alias => alias.toLowerCase().trim().replace(/[\s_-]/g, '') === normalizedTarget)) {
            return key;
        }
    }
    return null;
}

Dashboard and Real-Time Indicators

To provide visual support and full traceability throughout the technical process, the UI includes two key sections:

• Log Console (LogConsole): Provides real-time visual auditing of every validation and transformation performed by the engine, categorized by status through a clean design (info, warn, error, success).
• Metrics Panel (StatsBar): Provides immediate analytical visibility into processed vs. exported rows, the number of resulting columns, and the exact execution time of the pipeline.

Approach Comparison

Factor	Traditional Processing (Server)	D-MO Approach (Client-Side)
Privacy	Potential risk when transferring data to third parties	Absolute privacy; data never leaves the browser
Infrastructure Costs	Requires dedicated servers and scalable computing resources	Zero infrastructure consumption; leverages the client's hardware
Speed	Subject to network upload and download latency	Instant processing directly in memory

---

Conclusion

D-MO demonstrates that it is not always necessary to delegate heavy operational data workflows to complex backend architectures. Moving the logic of a direct ETL pipeline to the client side using TypeScript not only reduces infrastructure costs, but also addresses the most critical factor in corporate financial environments: regulatory compliance and data protection.

Tip: When building client-side data tools, prioritize libraries that support chunked reading or streaming to avoid freezing the user interface's main thread when processing large datasets.

Try it: D-MO