DEV Community: Oluwatimilehin Adesoji

Finance-101 Part 1: Money as Data

Oluwatimilehin Adesoji — Thu, 06 Jun 2024 23:00:00 +0000

Have you ever seen a balance off by one cent? In finance, even the smallest error is a red flag. A rounding mistake that looks harmless in code can accumulate into losses, reconciliation headaches, or opportunities for abuse. The first principle of reliable money systems is treating money as data with exact rules.

What does it mean to treat money as data?

Money is not a generic number. It has precision rules set by the currency:

Each currency has a major unit (USD → dollars, NGN → naira, JPY → yen).
Each currency also defines a minor unit (USD → cents, NGN → kobo, JPY → none).
Amounts must be exact, not approximate.
Totals must remain consistent across systems, reports, and statements.

Ignore these, and you get drifting balances, failed reconciliation, and loss of trust.

Why floats fail

A float (floating-point number) is an approximate representation of real numbers in binary. Many decimal fractions, like 0.1, cannot be represented exactly, so arithmetic introduces tiny errors.

0.1 + 0.2
// 0.30000000000000004

That might be acceptable in physics. In finance, that “extra fraction of a cent” becomes a liability.

Instead:

Use integers for amounts stored in minor units.
Use fixed-point decimals (for example, DECIMAL(p,s) in SQL) only when more fractional precision is required, such as FX rates or interest calculations.

Major vs Minor Units

Minor unit (recommended for storage and computation)

Store the smallest indivisible unit as an integer. Example: 100.00 USD → 10000 cents.
Benefits: exact arithmetic, simple aggregation, consistent across services.

Major unit (allowed with care)

Store dollars or naira as 100.00. Use fixed-point decimals and enforce scale.
Risks: silent rounding in some languages and ORMs, accidental float usage, and harder migrations when adding currencies with different minor units.

💡 Best practice: store and compute in minor units (integers), and use major units for input and display at the edges (APIs, UI, reports).

Applying this in your system

When handling money in software, consistency matters more than anything else. Here’s a practical flow you can adopt:

Accept amounts in major units and convert to minor units internally, or accept minor units directly.
Validate the submitted amount against the currency’s allowed scale (minor unit).
- JPY minor unit is 0 → reject 100.15 JPY.
- USD minor unit is 2 → reject 100.123 USD.
- KWD minor unit is 3 → accept 10.125 KWD. > ❌ Never silently round. Always reject invalid input with a clear error.

Example error response:

   {
     "error": "INVALID_AMOUNT_SCALE",
     "message": "JPY does not allow fractional amounts. Submit an integer value."
   }

Convert to minor units on the server and include the currency code.
Use minor units for all calculations in your code.
Store minor units in the database.
Render back to major units in APIs, UI, and reports for readability.

TypeScript example (server-side):

type CurrencyMeta = { code: string; minorUnit: number };
const CURRENCIES: Record<string, CurrencyMeta> = {
  JPY: { code: "JPY", minorUnit: 0 },
  USD: { code: "USD", minorUnit: 2 },
  KWD: { code: "KWD", minorUnit: 3 },
};

function validateScale(amountMajor: string, currency: string) {
  const meta = CURRENCIES[currency];
  if (!meta) throw new Error("UNSUPPORTED_CURRENCY");
  if (!/^-?\d+(\.\d+)?$/.test(amountMajor)) throw new Error("INVALID_AMOUNT_FORMAT");
  const [, , frac = ""] = amountMajor.match(/^(-?\d+)(?:\.(\d+))?$/) || [];
  if (frac.length > meta.minorUnit) throw new Error("INVALID_AMOUNT_SCALE");
}

function toMinorUnits(amountMajor: string, minorUnit: number): bigint {
  const [intPart, fracPart = ""] = amountMajor.split(".");
  const frac = (fracPart + "0".repeat(minorUnit)).slice(0, minorUnit);
  return BigInt(`${intPart}${frac}`);
}

// Usage
validateScale("100.00", "USD");
const minor = toMinorUnits("100.00", 2); // 10000 cents

Migration considerations

If your system currently stores amounts in major units:

Audit where arithmetic is performed.
Add a new minor_amount column for minor units.
Backfill: minor_amount = amount * 10^scale(currency).
Switch reads and writes to minor_amount.
Keep accepting major units at the API but convert internally.
Run reconciliation before and after migration to confirm balances.

Takeaway

Applying this pattern ensures financial accuracy across your system:

Accept amounts in major units and convert to minor units internally, or accept minor units directly.
Convert to minor units internally.
Compute and store only in minor units.
Expose major units back at the edges.

Do this well, and you build a system that never loses a cent. This is also a solid foundation for accounts, double-entry, ledgers, and reporting.

Next in the series

Accounts and Balances: Understanding the different types of balances, where and how to use them.

Finance-101: Building Reliable Money Systems

Oluwatimilehin Adesoji — Thu, 30 May 2024 23:38:00 +0000

Have you ever lost a penny in your code? If you're a software engineer, the first time your code handles real money can be terrifying. One tiny bug could cause your software to charge a customer twice or misplace funds, with consequences far worse than a typical software glitch.

In financial systems, mistakes aren’t theoretical: a famous trading bug once cost a company $440 million in 45 minutes. Clearly, reliable money systems aren’t just a buzzword; they’re a necessity.

When I started in payments, I couldn’t find clear resources on the fundamentals. I’m writing this series to fill that gap for engineers, covering the essentials from representing money in code to ledgers and financial reporting.

What Are Reliable Money Systems?

Reliable money systems are software systems that handle financial transactions with a high degree of correctness, consistency, and trust. They power day-to-day solutions from payment processing in apps, e-commerce, checkouts, full accounting, trading, and banking platforms. Because they deal with actual money, there’s almost no margin for error.

At their core, reliable systems should ensure:

Accuracy: Every transaction reflects reality exactly, without rounding errors compounding over time.
Auditability: Every movement of money is traceable, with immutable logs that make it easy to verify where funds went.
Timeliness: Financial data is available when needed, whether as monthly reports or real-time balances.

In short: a reliable money system never loses a penny, never lies about the numbers, and never leaves you guessing where the money went.

Why This Matters

If your product touches payments, subscriptions, or balances, you need these concepts. Here’s why:

Money at Stake: Bugs directly cost money and trust. A rounding error of one cent across 10 million transactions means $100,000 unaccounted for.
User Trust: Customers forgive UI glitches, not missing money. One bad charge can ruin confidence permanently.
Bad Actors: Weak financial systems invite fraud. Exploiters actively look for inconsistencies in how money is tracked or moved.
Regulation: Finance is one of the most heavily regulated industries. Accurate, auditable systems are essential to meet regulatory obligations, reporting, and compliance requirements.
Complexity: Payments flow across services, processors, and databases. Without strong principles, money can be duplicated, lost, or stuck.

Understanding these foundations helps you design systems methodically instead of firefighting when things break.

What’s In It for You

This series is a practical guide to the building blocks of financial systems. Each article introduces a core concept, explains why it matters, and shows how it translates into software design. The goal is to help you avoid the mistakes I made early on and give you a solid foundation for building payment products with confidence.

Next Steps: Explore the Series

Here’s what’s coming next:

Money as Data: How money is represented in finance and how to represent it in your software.
Accounts and Balances: Understanding the different types of balances, where and how to use them.
Double-Entry Accounting: Understanding single and double entry accounting, and why debits and credits keep money systems consistent and auditable.
Journal Entries, Ledgers & Transactions: Understanding journals, ledgers, and transactions and how to utilize them correctly.
Financial Reporting Building Blocks: How raw ledger data rolls up into balance sheets and income statements.

Paginating with Cursor-based Pagination

Oluwatimilehin Adesoji — Sat, 20 May 2023 16:30:00 +0000

In the field of software engineering, managing enormous volumes of data is a major challenge. To address this, developers have often used offset-based pagination. Consider it like flicking through the pages of a book, where you choose where to start and how many pages to show. Isn't it straightforward? No, not at all.

The Traditional Approach: Offset-based Pagination

When you use offset-based pagination, you're asking the system, "Show me data starting from this point." For example, to obtain records 11 through 20, you might say, "Start at 10 and give me the next 10." It works. However, there are a few issues.

Some Issues with Offset-based Pagination
Some issues associated with offset-based pagination are mentioned below, shedding light on its performance challenges and inconsistent results due to dynamic data.

Performance Challenges: Performance degrades when the offset value rises because the database engine must process an increasing number of records. The end-user experience may be negatively impacted by retrieving pages farther into the dataset.
Inconsistent Results: Inconsistencies may arise because of the dynamic nature of data between queries. The intended flow of results may be broken by adding or removing entries between two paginated queries.
Resource Intensive: When using offset-based pagination, more data must frequently be retrieved than is required, which puts a burden on server and network resources. This inefficiency is especially noticeable when handling large datasets.

Cursor-based Pagination to the Rescue 🚀

Cursor-based pagination addresses the limitations of its offset counterpart by utilizing a more intelligent approach. Instead of relying on numerical offsets, it employs a "cursor" that points to a specific item in the dataset.
This cursor can be a unique identifier, a timestamp, or any other sortable attribute.

How it Works
Here’s an in-depth look at how cursor pagination operates and its nuanced advantages in database management.

Stateless Navigation: Cursors eliminate the need to maintain and manage offset values, making navigation stateless. Each request includes a cursor that points to a specific location in the dataset, assuring consistent and accurate results.
Efficient Queries: Cursors make queries more efficient. Database engines can swiftly discover and get the relevant information since they refer directly to a precise spot, eliminating the need for expensive actions such as skipping big portions of data.
Consistent Results: Cursors provide reliable outcomes, even with changing datasets. Developers may have a reliable and predictable experience as the pagination process remains unaffected by changes in the dataset because the cursor is linked to a single item.

Let's Dive into the Implementation

Let's explore a basic implementation of cursor-based pagination in MySQL using a fictional "products" table.

-- Initial query fetching the first page
SELECT * FROM products
ORDER BY id
LIMIT 10;

-- Subsequent query using cursor for the next page
SELECT * FROM products
WHERE id > :last_seen_cursor
ORDER BY id
LIMIT 10;

In this example, the id which is a signed integer serves as the cursor, ensuring that subsequent queries fetch records created after the last seen cursor.

Conclusion

The transition from offset-based to cursor-based pagination represents a big step towards efficiency and scalability in the changing world of backend programming. Understanding the limitations of standard pagination methods enables developers to use better alternatives, ensuring that apps can manage enormous datasets gracefully and quickly. Cursor-based pagination exemplifies the ongoing growth of backend engineering practices, optimizing data retrieval for the needs of current applications.

Processing millions of records to CSV using Node.js Streams

Oluwatimilehin Adesoji — Sat, 19 Nov 2022 10:38:00 +0000

Efficient data management is essential for backend engineering, especially when working with huge amounts of data. A typical task is to use Node.js to stream millions of entries from a MySQL database. This article will discuss the challenges and review how to implement a streaming solution for optimal outcomes.

The Role of Streams in Boosting Performance

Streams are important because they facilitate real-time responsiveness, minimize memory usage, and handle and process data in chunks effectively. The smooth piping and chaining of streams allow us to quickly construct modular code while guaranteeing a balanced data flow with backpressure management.

Navigating the Challenge of Streaming Millions of Records

Addressing the streaming problem of managing millions of data is essential. It might take a lot of time and resources to get many entries from a database. Conventional methods, such as retrieving every record at once, might cause memory problems and increase delay. We choose to broadcast the data in segments/ chunks to get around these challenges.

Now to the Implementation

We would use mysql2 that supports streaming to establish a connection to your MySQL database, and the @json2csv for the CSV report generation.

Install the Dependencies
First, we install 2 required dependencies using the command below:

npm install --save mysql2 @json2csv/node

Create a Database Pool
Next, we create a database connection pool using the mysql2 package installed earlier. This would allow us to manage connections to the database.

import mysql from 'mysql2/promise';

function createDatabasePool(): mysql.Pool {
  const { pool } = mysql.createPool({
    host: '127.0.0.1',
    user: 'USER',
    password: 'PASSWORD',
    database: 'DATABASE',
    port: 3306
  });

  return pool;
}

Stream to CSV
We create a function that accepts a readable stream as the source, and transforms and formats the data.

import stream, { Readable } from 'stream';
import { Transform } from '@json2csv/node';


function streamToCsv(data: {
  source: Readable;
  headers: string[];
  onError?: (data: { [key: string]: any }) => any;
  onFinish?: (data: { [key: string]: any }) => any;
}): { passThrough: stream.PassThrough } {
  const transformStream = new Transform(
    {
      fields: data.headers
    },
    {},
    {
      objectMode: true
    }
  );
  const passThrough = new stream.PassThrough();

  const pipe = stream.pipeline(data.source, transformStream, ps, (err) => {
    if (err) {
      console.log('Error => ', err);
      if (data.onError) {
        data.onError(err);
      }
    }
  });

  if (data.onError) {
    pipe.on('error', data.onError);
  }

  if (data.onFinish) {
    pipe.on('finish', data.onFinish);
  }

  return { passThrough };
};

function streamData (sqlQuery: string): Readable {
  const pool = createDatabasePool();
  const result = pool.query(sqlQuery).stream();
  return result;
}

Putting it all together
With everything set, we can now proceed to stream data to CSV. We start by first creating a connection to the database and then create a query stream from the SQL query. Afterward, we stream to CSV and propagate the data written to the passthrough into the Express response.

const sqlQuery = `SELECT id 'ID', name 'Name', email 'Email', created_at 'Date' FROM your_table`;

const readableStream = streamData(sqlQuery);

const { passThrough } = streamToCsv({
  source: readableStream,
  headers: ['ID', 'Name', 'Email', 'Date'],
  onError: (error) => console.error(error),
  onFinish: () => console.log('process complete')
});

passThrough.pipe(res); // res being Express response

Some Benefits of Streaming are

Streaming in Node.js has multiple benefits that improve application performance, scalability, and real-time responsiveness:

Memory Efficiency: Streams handle data in segments rather than loading full datasets into memory. This reduces the memory footprint, making Node.js well-suited for processing huge files.
Pipelining and Chaining: Streams provide a modular and reusable approach to code design since they are simple to pipe and link together. This makes it easier to create intricate workflows for data processing.
Improved Performance: Streams improve speed by starting data processing before receiving the whole data set, allowing for segmented data processing. It accelerates response times, which is particularly advantageous in high-concurrency scenarios.
Scalability: Your application can handle big datasets using streaming without reducing speed since it is more scalable. Streaming offers a more sustainable solution as data quantities rise, making it ideal for applications with changing data requirements.
Parallel Processing: Streaming makes it possible to process data in parallel, which improves system resource utilization. This is particularly helpful in situations when the streaming data has to undergo several procedures at once.
Real-time Processing: Streams provide the instantaneous processing of data upon its availability. Applications like monitoring systems or analytics dashboards applications that need to react instantly to changing data depend on this.

Conclusion

In this article, we have discussed how using streams can improve the performance of our application especially when working with large data. By breaking data into smaller chunks for processing, we can conserve memory and improve the performance of our application.

Thank you for reading!