DEV Community: Daniel

Handling Account Balances in Financial Applications

Daniel — Thu, 27 Oct 2022 18:50:15 +0000

In software engineering, correctness is defined as an application behaving as intended for its expected use cases. Correctness is always important but never more so than when dealing with accounting data. I have been lucky enough to build software that has processed millions of pounds in payments, and these are some of the patterns & principles applied used to calculate and track account balances.

When building financial software, one of the tasks you undertake will be modelling how to calculate and store balances. An initial approach might include an attribute on an Account model, updated when a transaction occurs. Similar to the below:

The above is not a good solution because it breaks data normalisation. The balance of an account should equal the sum of transactions. Storing the calculated value (as the source of truth) will cause issues if the calculation becomes out of sync, and this will happen in an environment with lots of concurrent requests.

Race conditions occur when a system attempts to perform two or more operations simultaneously. In our case, we have a Read-Modify-Write race condition. This happens when two processes read a value and write back a new value. If you imagine multiple processes executing this, you should be able to see we have a race condition.

When we run the above code concurrently, you will see the potential for the processes to read the account balance simultaneously; therefore, once the execution finishes, the account_balance will be incorrect.

Executor A reads account balance = £100
Executor B reads account balance = £100
Executor B adds £50 funds. account_balance = £150
Executor A adds £50 funds. account_balance = £150
Executor A saves the account model with attribute account_balance = £150
Executor B saves the account model with attribute account_balance = £150

As you can see, with concurrent requests, the order of execution cannot be guaranteed. Executor B read the account’s current balance as 100 before Executor A had added the funds. The result is the account_balance is short by £50. This issue only gets worse as the system scales.

The first step in solving this problem will be to adjust how we check the current balance of an account. We will need to maintain a transaction log. Every time a credit or debit happens, we will write a record to this log with the details. We sum all transactions from the log for any given account to calculate the current balance.

This process does have a scaling issue because each time a transaction is added to the log, it increases the compute resources required to calculate the balance. You can alleviate this by implementing a statement system. Where you store the calculated balance at intervals (monthly works). You take the latest stored balance and sum the transactions since, to calculate the current balance.

The above code eradicates the possibility of concurrent requests causing the system to under-calculate the balance of an account at any one time. If we queried during the execution flow, we would have returned the correct balance, and after both requests finish, it will return £200.

However, it does not solve all of our problems. A classic problem in accounting is preventing overspend. Overspend happens when a request comes into an application to debit an account for funds. The application must first check if funds are available to process the request. Suppose concurrent requests to debit an account enter the system. In that case, we face a familiar situation where both checks may validate and write a debit transaction to the log, causing an overspend.

The overspend problem is applicable as long as there are balance constraints which should cause a debit transaction to be rejected, even if the system should also allow for negative balances (overdrafts).

Account ID has a balance of £100.
Executor A requests to debit the account for £70.
Executor B requests to debit the account for £50.
Executor B checks the balance of the account. The sum of transactions equals £100 credit. It can proceed.
Executor A checks the balance of the account. The sum of transactions equals £100 credit. It can proceed.
Executor B writes a debit transaction to the log.
Executor A writes a debit transaction to the log.

After the above processes have exited, the account balance may be £-20, allowing an unauthorised overspend on the account despite the explicit check to prevent this. Each goroutine calls svc.CalculateBalance(), but they may complete this check simultaneously (although before writing the transaction log).

Solution

There are several layers to solving the above (including constraints at the database level); however, I will discuss one we can implement within the application.

Distributed locks

A distributed lock will ensure only one goroutine at a time can access a resource (like a mutex but in a distributed environment). We are thoughtful about where we place the lock to guarantee only a single transaction is processed (per account) at any time.

One way to get and hold a distributed lock is via Redis — the example below.

In the code above, we first define a key. We can use this key whenever we wish to limit an aspect of our account execution flow to a simultaneous execution. Once the key is defined, we attempt to acquire a lock on it. If another goroutine already has the lock, we will wait for the specified duration (whilst continually retrying to acquire it).

Limiting the scope of our lock key by injecting the account ID ensures we maintain good throughput across the system. If, for example, we had used a generic lock key like ‘account-transaction’, we would only be able to process a single withdrawal at any time for ALL accounts.

Once we have acquired the lock, we can safely continue with the withdrawal request because no other goroutine can access funds in the account.

We have also deferred the lock’s release to guarantee it is released, even in the event of an application error in our function. There is also a timeout set on the lock, ensuring that even if our node was killed (and could not execute the defer function), we will still release the lock after a set amount of time.

Conclusion

Distributed locks are only a single aspect of the controls you need when designing a system that calculates balances. Still, by implementing the above, you will be developing safer, more scalable software.

As all of the example code was executed in a single process, I could have used a Mutex; however this will not work in distributed applications.

Validation and the Single Responsibility Principle in Object Orientated Programming

Daniel — Tue, 10 May 2022 16:08:43 +0000

When writing software, you will spend a large portion of time creating code that takes an object, validates it and then performs some actions. One of the core tenets of SOLID code is the Single Responsibility Principle, and this means there should be a separation of the code that validates the object and the code that executes the action.

An example scenario might be: User A submits payment of £100 to User B.

On the surface, this seems very simple and can be implemented using the following logic. The examples in this post will be PHP but apply to any object-orientated language

We use a PaymentManager to transfer the requested funds between two users from the above code. In this scenario, the PaymentRequest object holds the information about the origin/destination and funds sent.

Whilst this meets the criteria of moving the funds, there are checks that the system needs to conduct before executing such a request in the real world. Some of the checks could include:

Is the origin account active?
Is the destination account active?
Does the origin account have available funds to cover the payment amount?
Is the PaymentRequest amount valid? e.g. it is not logical to pay an amount of zero (or less)

The criteria listed are crucial to the application, and as a result, it may be tempting to conduct these checks similar to the below:

This code works, but it violates SRP as the function is now validating the request and calling the payment manager. As the business requirements get more complex, this class will continue to grow and become difficult to maintain.

For example - functionality has been added to the platform, enabling users to block one another. As a result, a user should not be able to send funds to a user who has blocked them. This would require an additional, conditional statement to be added to the function.

Validation Classes

A much better way of handling this scenario is to abstract the validation logic away from the execution. This can be achieved by implementing the below:

The validation logic has been extracted to its own class. This separates the code nicely and ensures we will not be forced to update this class every time rules are changed or added. It also has the benefit of making our unit tests much easier to write. To test the MakePaymentJob we can mock the validate() method as true or false and test whether our paymentManager->makePayment() is called.

The validation class itself functions as below:

In addition to extracting the rules to a validation class, I moved each rule to a private method. The descriptive naming of the method provides better context and improves readability whilst creating an excellent base for writing unit tests. I wrote a post about how we might test code like this previously - you can check it out here.

Improvements

The validation pattern above is vastly improved over the original code; however, it could still do with a few adjustments. For example, returning false from a failed validation attempt provides no context for the calling client why the request failed. If the paymentRequest failed because the user did not have enough funds, this is something we would want to communicate to the user.

We can achieve this by making minor adjustments to the PaymentValidator class.

The above is a crude demonstration - when running validations across a codebase it would be better to create validation methods for common assertions. In the example above it is a simple greater than conditional. There could easily be an agnostic method in a validation namespace to handle standard rules. Classes wishing to implement rules can then inject the required rules.

Summary

There are plenty of ways to handle the actual validation of the objects, but the important part is to ensure this logic is separate. It will make extending the rules much easier in the future and increase the testability of your code.

Solutions for storing state in a database

Daniel — Tue, 19 Apr 2022 16:08:55 +0000

When designing an application, you will often need to model 'state'. State, in this sense, is related directly to a single entity. An example may be:

I have an order, and the possible status' of the order can be:

Pending
Paid
Dispatched
Delivered
Cancelled
Refunded

The order may only be associated with a single state at any time. It would not make logical sense for an order to both be delivered and returned.

Transitioning between these states will drive the behaviour of your service. For example, when an order transitions to dispatched, it is an ideal time to notify the customer of the delivery tracking details.

When modelling this type of solution, engineers are often tempted by the simplicity of having a single column on the original entities table to describe the current state. This can be achieved in a few different ways.

First Solution (and the worst)

As we just discussed, the easiest solution is to add a column to the Orders table. This column can store the current state of order.

It can be achieved by storing the state as a string:

CREATE TABLE `orders` (
 `id` INT unsigned NOT NULL AUTO_INCREMENT,
 `customer_id` INT unsigned NOT NULL,
 `order_value` INT NOT NULL,
 `state` VARCHAR NOT NULL,
 `created_at` DATE NOT NULL,
 `updated_at` DATE NOT NULL,
 PRIMARY KEY (`id`)
);

This isn't good for a lot of reasons. The most obvious one is that the application can submit any string to the state column, even one that has not been designed for in the application. This can break the application and cause problems for the service.

We could solve that issue by making the state column an enum and only allowing expected values.

ALTER TABLE orders ADD state enum('pending','paid',...);

This is still a bad solution.

We have to run an ALTER on the table every time we wish to add new states.
We cannot store additional metadata about the state as we are limited to a single column with a specific string stored.
It can be challenging to disable values. What if the store is no longer offering refunds?
ENUM datatype is not standard SQL and is not supported by many ORM.
It is complicated if we wish to output the list of possible values for state in our application.

Slightly Better Solution

It is clear from the previous approaches that having a single column denoting the state is not ideal. The next logical solution we could design is creating an order states table and referencing it from the orders table via a foreign key.

CREATE TABLE `order_states` (
 `id` INT NOT NULL AUTO_INCREMENT,
 `key_name` VARCHAR NOT NULL,
 `created_at` DATE NOT NULL,
 `updated_at` DATE NOT NULL,
 PRIMARY KEY (`id`)
);
INSERT INTO order_states (key_name)
  VALUES('pending', 'paid',...);
ALTER TABLE orders 
ADD CONSTRAINT FK_OrderState 
FOREIGN KEY(state_id)
REFERENCES id(order_states);

This solves some of the previous issues. There is no need to run ALTER on the table when adding new states, and any additional metadata that needs to be stored against the state can be added as a new column in the order_states table. If a state needs to be disabled/removed, it is easy to select orders by state_id, and we also get the protection of valid states via foreign key constraints.

Best Solution

The solution we just implemented has solved most of our problems but we have issues with the visibility of an entities state over time. Because the order is assigned a state via a single FK reference it means we only have information about the current state of the order. If the order is currently in a state of Cancelled there is no easy way for me to tell when it was Paid.

We can solve this by adding another pivot table.

CREATE TABLE `order_state_history` (
 `id` INT NOT NULL AUTO_INCREMENT,
 `order_id` INT unsigned NOT NULL,
 `order_state_id` INT NOT NULL,
 PRIMARY KEY (`id`)
);

In addition to creating this table we will remove the state_id column from the Orders table.

This pivot table works as follows: Any time you wish to change the state of an order you INSERT a new record into this table. You never need to UPDATE any rows here. You will also likely never need to DELETE any rows (unless you are purging records).

The current state of an order is the most recent record in this table (associated with the order_id)

You can get the order with its latest state by using the query below:

SELECT orders.*, order_state_history.*
FROM orders O
JOIN order_state_history OSH1 ON (O.id = OSH1.order_id)
LEFT OUTER JOIN order_state_history OSH2 ON (O.id = OSH2.order_id AND (OSH1.created_at < OSH2.created_at OR (OSH1.created_at = OSH2.created_at AND OSH1.id < OSH2.id)))
WHERE OSH2.id IS NULL;

Drawbacks
It would not be objective if I did not at least point out some of the drawbacks of the above solution.

Additional pivot table required linking Order to State
Selecting data from Orders requires a more complex query (including JOINs)
You will end up having to store additional rows for each order. Worst case is Orders * count(possible_state_values)

Summary

The last solution discussed is more robust because it is INSERT only. We do not need to worry about race conditions.

If we receive a stream of updates, we simply write each record in the order we receive it. It also ensures we have a complete historical record of an order and the states it held.

This makes it trivial to look at the data set and ask questions like:

How many orders did I dispatch between 1/12/2021 and the 31/12/2022
What was the total value of orders refunded during August

These questions would have been impossible to answer with the other solutions because we did not retain the history of state movements - only the current one.

The additional code to be written is minimal, and any performance hit from using JOINs is trivial when using proper indexing.

Any concerns about the additional data stored can easily be rectified by running a simple query to remove unneeded state history items. In pseudocode terms -

Delete all order_state_history items IF created_at > 1 year ago AND
order_state_history_id < MAX order_state_history_id for any given record

Running a query like this would remove historical items (more than one-year-old) but retain the latest state for each Order (as this will remain important)

In conclusion, it is a much better and more complete way of managing the state in your application and should be utilised where plausible.

How to build high-performance engineering teams

Daniel — Thu, 20 Jan 2022 14:26:08 +0000

When building a tech business, one of the keys to success will be a great engineering team. After all, they are responsible for making the product. Teams that ideate, collaborate and iterate create excellent products.

It is crucial to create an environment that allows these teams to thrive. Unfortunately, there is no out of the box framework that you can apply, but there are some basic principles that you can follow to help along the way.

Autonomy and Curiosity

The best engineering teams are autonomous. They should not be viewed as a resource to ‘be used’, but integrated with the business, so solutions are built together.

To achieve this, you will need curious people in your engineering team. They will have used the product and be able to pick out pain points. One of the best ways to get engineers into this way of thinking is to have time set aside during onboarding to use the product and view user feedback.

By empowering engineering teams to take ownership of the product, you get a much better ROI.

Delivering value to users

The team should focus on the value delivered to the user, not by building technology and figuring out how to apply it later. The best way to do this is to work from the user backwards. Some companies even use feature voting to let users give technical teams direct feedback.

Once the value proposition has been established focus should be on making the initial implementation as simple as possible.

Try to tie feature work to metrics by utilising SMART targets, this has additional benefits beyond the obvious

It will “gamify” the work
Engineers will be able to use the ‘measurable’ part to see their impact on the business
You will begin collecting useful metrics
You can share these metrics with other areas of the business

Developer experience

You now have a great team and a backlog full of tickets that will deliver actual value to customers, it is time to execute.

When iteratively building products, the team will be producing a lot of code in small batches. The development pipeline itself should be as efficient and automated. A great CI/CD pipeline is essential to this and will help deliver higher quality code at an increased pace.

Some other things you can look at in your teams to improve the developer experience:

Agree on code style guidelines (and enforce these in your CI/CD suite)
Project scaffolding
Good Documentation
Defined processes for pull requests/code reviews

The development team should maintain a refined backlog of stories to improve the developer experience. They will also be responsible for balancing this work alongside product delivery goals.

Finally, the team should be tracking key metrics like time to build, time to first commit and time to first release. These metrics will allow the team to measure the changes that genuinely improve the experience.

Open collaboration

A good engineering team will have a group of people that are happy giving open and honest feedback and this should happen as often as possible between all members of the team regardless of seniority.

The teams time will be spent on solving problems (or figuring out which problems to solve) so it is imperative that everybody feels comfortable making suggestions. Ideas should be defended with facts and no one should be interrupted.

Unfortunately, the team may not always reach a consensus, in this situation, it is important that the decision-maker (usually a tech lead) picks an option. The team then works towards this goal, there should be no vetoing, foot-dragging or hesitation.

Conclusion

In summary, every team is a unique blend of personalities and there is no one size fits all approach. Some of the above might work for you and some of it won’t. The important thing is to measure the impact of any changes and iterate.

Tips for writing highly readable code

Daniel — Mon, 29 Jul 2019 15:27:01 +0000

What is readable code

You are writing for humans. Not the machine.

The above should always be at the forefront of a developers mind when designing a system. Assuming your are writing syntactically correct code that functions the machine could not care less about how it is structured, but the developers who work on it later (and that could still be you) certainly will!

"Good code" is highly subjective because whats constitutes as such varies from developer to developer, project to project and year to year. That being said there are still common strategies and guidelines we can adhere to make our lives easier in the future.

I'll go through some more specific examples below but in general:

Follow the project style guidelines (if it doesn't have any, suggest some)
Write testable code
Think SOLID
Use common design patterns (where appropriate)
Be consistent with file/class naming.

Some examples

Be descriptive

This example is extremely basic and I expect most people write like this by default.

First off - be descriptive with your class, method names and variables. Don't have vars called $i or $model.

Becomes

Abstract (complex) conditionals

Complex conditionals are hard to read and "break the flow" when reading through code.

Consider the following example:

Becomes

I am actually a fan of abstracting conditionals wherever possible. Even if they are relatively simple. They help make your code more testable and by naming your method appropriately you convey the purpose of the conditional in a really nice succinct way.

No magic numbers

Magic numbers are hard coded values that implicitly define logic. Not only does it require us to investigate the code-base to see what the value represents it could also change over time. If a magic numbers value changes we will be required to change it in multiple locations which can lead to inconsistencies.

Becomes

Review

It's easy to over complicate your code but as long as we remember we are writing for humans we can make a much nicer code-base for everyone.

How to write testable code

Daniel — Mon, 15 Jul 2019 10:27:53 +0000

This is something I put together for the engineering team at Howsy. Hopefully it will be of some assistance for others. The examples are in PHP but can be abstracted to any OOP language

What is Unit Testing?

It’s simply a test that runs against an individual ‘unit’ or component of software. We are testing the smallest possible implementation of a class/method. A unit test does not rely on any dependencies (other classes/libraries or even the database)

Why is it important?

I mean we could just stick with integration tests right? They are easy to write, however ...

The goal of any Software Project is to produce a maintainable, extensible codebase which meets business requirements. Writing effective unit tests help us to ensure we are writing SOLID code.

How can we write more testable code?

I have taken a very simple class and it is typical of some of the code I have seen over the years.

class ToActive
{
    public function transition(int $accountId): void
    {
        $account = Account::find($accountId);
        if ($this->canTransition($accountId)) {
            $account->setState('active')->save();
        }
    }

    private function canTransition(Account $account): bool
    {
        $validator = new ToActiveValidator();

        if ($validator->validate($account)) {
            return true;
        }
        return false;
    }
}

On the surface the class appears to work. It finds an account from the DB, calls a validation method in another class and returns true/false. Based on the result the Account may or may not be updated.

Although this class works it is impossible to unit test due to two different dependencies being instantiated rather than injected. The two offending pieces of code are

$account = Account::find($accountId);

$validator = new ToActiveValidator();

As they are in the body of our methods they will be called during unit testing. This will cause us to rely on the responses from other classes in order to run a test. This makes any resulting tests we write for this class not true unit tests.

We can refactor this class very easily to make it more easily testable and therefore more extensible and loosely coupled.

Refactoring

class ToActive
{
   private $account;
   private $validator;

   public function __construct(AccountInterface $account, StateTransitionValidatorInterface $validator)
   {
       $this->account   = $account;
       $this->validator = $validator;
   }

   public function transition(): void
   {
       if ($this->validator->validate($this->account)) {
           $this->account->setState('active')->save();
       }
   }
}

As you can see from the above we are now injecting dependencies.

Our class does not care about the concrete implementation of the Account or the Validator - only that they adopt the required interfaces. This again makes our code more loosely coupled. In the future maybe we may completely change the StateTransitionValidator implementation or even add additional validators. Our code is now less tightly coupled and therefore more testable.

Now let’s get to testing

Looking at the class we have one public method. We should not be trying to test private methods as they are implementation details. If you find your class has too many private methods or feel that they need testing it is a sure fire sign that your class could do with splitting out into smaller components.

There are two paths within the public method - the validation passes and we save the entity or the validation does not pass and we don't save it.

I will write the test assuming the validation passes first. We are going to be mocking our dependencies (something not possible with the original class architecture).


public function testToActiveWithPassingValidationShouldTransition()
{
   /** @var Mockery\MockInterface|Account $account */
   $account = Mockery::Mock(Account::class);
   $account->shouldReceive('setState')
       ->with('active')
       ->andReturn($account)
       ->once();
   $account->shouldReceive('save')
       ->andReturn($account)
       ->once();

   /** @var Mockery\MockInterface|ToNotActiveValidator $validator */
   $validator = Mockery::mock(ToActiveValidator::class);
   $validator->shouldReceive('validate')
       ->with($account)
       ->andReturn(true)
       ->once();

   $service = new ToActive($account, $validator);
   $service->transition();
}


public function testToActiveWithFailingValidationShouldNotTransition()
{
   /** @var Mockery\MockInterface|Account $account */
   $account = Mockery::Mock(Account::class);
   $account->shouldReceive('setState')
       ->with('active')
       ->andReturn($account)
       ->never();
   $account->shouldReceive('save')
       ->andReturn($account)
       ->never();

   /** @var Mockery\MockInterface|ToNotActiveValidator $validator */
   $validator = Mockery::mock(ToActiveValidator::class);
   $validator->shouldReceive('validate')
       ->with($account)
       ->andReturn(false)
       ->once();

   $service = new ToActive($account, $validator);
   $service->transition();
}

Breaking it down

First of all we use Mockery to mock our account class.

We tell our mock that we will call the method setState with an argument of active. We also assert that this function will be called exactly once (this is our actual test assertion)
We also assert that the save() method will be called exactly one time.

We also mock our concrete validator class. We tell our mock that the validate method shall be called once and that it will always return true.

The tests will pass and you can see we have done so without actually relying on our dependencies. These were mocked and injected into our class with all of their methods returning exactly what we defined in our test.

Testing the reverse

We also want to test the second path (what if the validation returns false, business rules dictate we should therefor not transition the object)

Take a look at the second test and you can see our test is pretty similar only now we are forcing the validator to return false. As such our assertions have changed and we now expect setState and Save to never be called on our account object.

What we achieved

The original code was hard to test and tightly coupled. The refactor has allowed us to make the code more testable but a great side affect of this is that the code is also now more flexible.

Let me know if you have any questions in the comments!