Vonmo Trade Experiment. Part 1: Exchanges and modern technologies

#erlang #rust #functional #programming

This series of articles describes an attempt to create a reactive system performed by one person, with a minimum budget and in the shortest time possible.

The experiment aims at:

A deeper understanding of the subject and improvement of technical expertise;
Identifying strengths and weaknesses of functional languages and open source projects in trading systems development.

This article presents the motivational part and task decomposition.

I was planning to start this experiment in spring 2019, but then – life happened – and I had to reschedule. That is why here are my apologies to all those few followers of mine who have been waiting for the articles on this topic.

What brought me here?

When I realized that I’d got tired of day-to-day project routine, I wanted to create something unusual. I was thinking of various topics and directions which looked challenging enough for me. This way, the main requirements to the project-to-be appeared:

Field of studies that I find interesting;
Distributed and high-availability nature of the app;
High information capacity and big volume of stored data, typical for data-intensive apps;
Maximum use of open source projects.

As I’ve already had some experience in developing object storages, I came up with an idea. Why don’t I write a new implementation of s3-compatible storage using erasure coding and fast hash functions? But then I wanted something even more exciting. From a developer’s point of view, financial system are interesting due to their near real-time work, huge number of updates and volume of stored data. So, I set my mind on creating my own implementation of an exchange system.

About exchanges

A stock exchange is also known as bourse (from Latin “bursa” – “purse”). In a broad sense, this is a market where sellers and buyers conduct transactions. A traded asset defines the exchange type:

commodities exchange, including labour exchange
stock exchange / market
currency exchange
futures exchange

Historically, exchanges were literally a place which, at the appointed time, brought together sellers, buyers and brokers, or middlemen, who assisted with transactions.

The main functions of any exchange are the following:

Organizing the exchange trade:
- Defining trade rules under current legislation
- Setting product standards
- Staffing and material provision
Information activity. Providing information about stock instrument prices, markets and companies;
Drawing up standard contracts;
Settlement of disputes (arbitration);
Providing certain guarantees of order execution.

Modern exchanges are well-organized trading platforms which work by certain rules and create supply and demand. Exchange economy is crystal clear: the more transactions there are on the exchange, the more fees and commissions it gets for its service.

Now when we have a general idea, let's limit tasks to do for MVP realization.

MVP

Limited resources lead to limited functionality of the prototype.However, vital and fundamental things should be fully implemented.

Task decomposition and architecture choice

Technically speaking, an exchange is a public service system where its participants generate a flow of requests – or orders. The system, in its turn, operates and executes them by the rules defined in advance. Order service must be performed with minimum waste of time. The exchange must be fault-tolerant and highly available.

Let’s divide the project into a system part and a public part. The system part enables us to manage platform components and exchange trading process. The public part includes client interfaces: Web API, Trading API, notifications subsystem. Both parts will be based on distributed apps platform which is responsible for system scalability and reliability. As we are limited in budget, we’ll opt for tested open-source solutions for data storage organization. Tarantool is used for the system hot data, PostgreSQL for the cold data.

Let’s draw the system structure:

It might seem that both the architecture and the project itself are no different from any usual system. Where is the challenge, you may ask. To answer, let’s take a look at the process of creating orders and filling orders for a currency exchange.

Creating an order

Checking authorisation
Making sure that a user can now afford to place an order
Creating an order
Informing the user on success of the operation or failure

Filling an order

Finding a match
Checking fund sufficiency for both parties of the deal
Making the deal: money transfer between accounts + exchange fee transfer to the exchange commission account
Saving the history of order execution
Updating the information data: raw data and aggregated data for graphs
Notifying all parties concerned about the deal.

The trouble starts when we move from a monolith, working on a single machine, to a distributed fault-tolerant system deployed on a cluster. I think, the minimum performance level should make about 5-7k RPS completed transactions per market. This puts extra limitations on architecture and work with data.

Following the KISS principle, each app must perform only the functions which are necessary for processing the entity implemented in it: market, account, authorization etc.

VonmoTrade is based on horizontal scalability principle. Each application could have many instances to provide fault-tolerance and required performance level.

Technology stack

To avoid generating entropy, let’s limit the languages and approaches used.
For server, this set will include:

Erlang. To my mind, a perfect language to deal with infrastructure things.
Rust. Excellent for system-related things and optimization issues.
PostgreSQL. Serves as the main base for long-term data storage.
Tarantool. Serves as hot data storage (for MVP only)
Clickhouse. For logs analysis and deep analytics.
Linux. The system must support modern distributions.

The client software is implemented with the help of Vue framework and Vuex.

As the project presupposes a big number of components, we will write a lot of property-based tests and integrative tests to ensure an adequate level of quality of the final solution.