DEV Community: Akarsh Agarwal

A whole new world behind the scenes! - Part 3

Akarsh Agarwal — Sun, 09 Oct 2022 15:08:37 +0000

The primary goal was to process the transaction when we started building our payment engine in the previous blog. The preliminary checks about whether a transaction should happen took a back seat. Moreover, the system’s resiliency was never questionable in our last design. But is that how our current systems work? I believe everyone knows the answer to that question.

First, we’ll understand the numerous preliminary checks a payment processor performs before the addition and subtraction of balances. Then, post that, we’ll analyze different failure scenarios and improve our design to account for them. Hence, this blog is a bit of both theory and coding.

So, let’s get started!

N number of steps

Ideally, a transaction is the transfer of funds between two entities. Those two entities could be individuals, businesses, systems, or a mix-and-match. However, it’s the 21st Century, and our transactions involve much more than the addition and subtraction of balances.

Today, it is a series of processes that enables the transfer. For starters, KYC (Know Your Customer) is one such process. Validation of who you are is a crucial part of the transaction. Hence, we’ll look into some critical processes that perform checks before a transaction can proceed. However, due to my limited knowledge, the below is not an exhaustive list, and the sequence below is irrelevant. The sequence of checks is highly dependent on the payment processor and its rationale.

Identity Check

Today, KYE or Know Your Entity, to generalize, is as vital as transferring funds. KYE of individuals (KYC: Know Your Customer), businesses (KYB: Know Your Businesses), and even our interacting systems/partners defines whether a transaction should proceed or not.

Knowing who’s the sender and the receiver to avoid any illicit transactions is fundamental to today’s payment systems. It’s primarily to prevent fraud of any kind. What does that mean? If they represent themselves as Bob and Alice, the engine should ensure the fact, and the transfer should happen only between the two. Erroneous funds debiting or crediting can lead to losses and user experience.

I’m not oblivious to the disruption brought by Crypto in the last few years, where Anonymity has been one of the most extensive offerings by the network. However, this series focuses on the non-crypto Payment Engines that fuel our day-to-day transactions. I’ll take up Crypto a few blogs down the line.

Authorization Check

Let’s not confuse Authorization with Authentication. Authorization checks if the sender and the receiver are allowed to transact. For example, if I’ve blocked international usage on my account, it will not allow me to make international transactions. It is because I’m not authorized to perform it. Another example is if the receiver’s account is blocked due to suspicious activity. In that scenario, too, the transaction is not authorized.

Risk Check

When transacting, the payment processors need to ensure that the risk of the transaction is rightly managed. A famous example is OTP or One Time Password. When paying online, they usually redirect you to the OTP verification page to ensure additional authentication and avoid unwanted results.

Recently, the minimum threshold transaction amount for OTP has become configurable. Earlier, as you might recall, we received OTP for every transaction. Nowadays, we receive OTP for high-value transactions.

Compliance Check

Does your transaction comply with the rules and regulations of the region? It’s an essential check because the payment processors are answerable to their rules and regulations. Compliance checks are dependent on the transacting entities. It may differ for individuals and businesses.

For example, MAS (Monetary Authority of Singapore) allows only a S$30,000 yearly transaction limit for e-wallets.

Balance Check

Finally, last but not least, balance check. It is one of the most important checks to perform and needs no introduction or explanation. Balance check asks: Do you have the amount to transact?

Don’t worry! We’ll retry it for you!

We created our first payment engine in the last blog. The engine was capable of processing transactions in an all-good-nothing-fails scenario. However, failures are a part of our systems every day! So, how do we enhance our previous design to accommodate for errors? Let’s check that out.

We’re not working out anymore!

Our last design had the depositing and deducting steps in the same function. While we might know which step of the transaction failed, it’s pretty impossible to re-run the same function to process the second part of the transaction. It would deduct the receiver more than once. Hence, we’ll update our design to support retries and better failure management for erroneous scenarios.

Let’s separate the two states to become independent functions. Here’s how it would look:

func deductSender(tx Transaction) {
    tx.Sender.Balance -= amount
    tx.State = "deducted"
    save(tx)

    depositReceiver(tx)
}

func depositReceiver(tx Transaction) {
    tx.Receiver.Balance += amount
    tx.State = "deposited"
    save(tx)
}

One could run the depositReceiver function to complete the transaction if the transaction errored out after the deducted State. Does this look complete? Are we done?

Not really! The deducted state is tightly coupled with the deposited state as the deductSender calls the depositReceiver. Consider a scenario where we want to hold the money in escrow before depositing the receiver. The above logic doesn’t work, as we’ll need to update the deductSender function to add a step. Is this the best we can do?

Power up your SM!

Well, we’re using State Machines! Our states should define the execution and not the code. What do I mean by that? Our function should trigger events to process the transaction. For example, we can define that once a transaction reaches deducted state and an event next is called; it automatically executes the function depositReceiver.

Overall, it reduces the dependency on the code and moves it to the states. Now your functions become stateless, and you can add or remove states before or after the current function without affecting it.

I’ve added some sudo code for the State Machines to keep the details out. Here’s how it would look:

var sm *sm.StateMachine

func init() {
    sm := sm.NewSM()

    sm.Trigger("deducted", "next", depositReceiver) 
}

func executeTx() {
    tx := Transaction{
        ...
    }

    sm.Execute("deductSender", "next", tx)
}

func deductSender(tx Transaction) {
    tx.Sender.Balance -= amount
    tx.State = "deducted"
    save(tx)
}

func depositReceiver(tx Transaction) {
    tx.Receiver.Balance += amount
    tx.State = "deposited"
    save(tx)
}

The above is a short sudo code. So let’s dive into what I’m trying to achieve here. Please ignore the nuances, as it’s not a correct Go code, so it’s not usable.

The init function initializes the SM. Then, it creates a new SM object and defines a state transition rule. The rule says that any transaction in the deducted state triggering the next event should call the depositReceiver function.

Then, the sm.Execute function executes the SM by calling the deductSender function with the argument tx. Moreover, it defines the event name to trigger once the function completes. Hence, it would call the depositReceiver automatically post that.

One of the long-term benefits of the above approach is the maintainability and the possibility of branching when new flows are added. What do I mean by that? When we add an escrow in the middle, we don’t need to touch the deductSender function. We can update the SM’s Trigger function to call despositEscrow instead of the depositReceiver function. The functional logic remains the same. However, managing and adding functionality becomes easier without modifying the previous code.

Summary

That’s about it, folks!

In the first part, we covered a few behind the scene checks and understood the different steps (not necessarily in the same order as above) that define our transactions. In the second part, we improved our SM to define our execution flow and provide partial error handling. Finally, we improved the design and removed the dependency on code for executing the next step.

The following blog will target formalizing sudo code with appropriate libraries, steps, and function executions. We’d target to develop a CLI tool for executing a transaction and see through the state transition. I’ll add the code + the Github repo link for easier access.

See you next weekend!

A simple Payments Engine that just adds/subtracts balances! — Part 2

Akarsh Agarwal — Tue, 04 Oct 2022 13:54:56 +0000

What if I told you that writing your First Payments Engine was as simple as doing that math problem we learned in 4th grade?

Yes, we’ll learn to write our first Payments Engine in this blog. From scratch! We’ll target a simple engine and then increase the details as we understand more about the Payments domain over the following blogs.

To reference our learnings from the previous blog, we’ll focus on the Direct Charge mechanism for now.

Just FYI, I’ll be using Go to write some code below. It’s very similar to C and has been my first language since graduation, so I prefer that.

So, why keep waiting?

Our 4th grade Engine

Let’s start with understanding and developing what we’ve learned before.

What do we need for a transaction? A sender, a receiver, and two accounts, one to deduct from and another to deposit into. A transaction is adding the amount into the receiver’s account and subtracting the amount from the sender’s account.

Before we even start writing any code, let’s make 2 significant assumptions about the complexity of the problem at hand:

Bob has the required balance to pay Alice for the transaction, no matter how considerable an amount.
Our system runs on a single machine and doesn’t go down. It’s highly unlikely or impossible in today’s world, but as we progress further, we’ll work on this.

With this in mind, we can write a simple function to transact as follows:

func processTx(receiver, sender AccountDetails, amount int) {
    receiver.Balance = receiver.Balance + amount
    sender.Balance = sender.Balance - amount

    save(receiver)
    save(sender)
}

Well, that’s it? Yes, that’s our very first payment processing engine. It represents our knowledge from the 4th grade, where we would add/subtract balances to complete a transaction.

Now, let’s process an example for the same:

Bob needs to pay Alice $10 for Oranges

Let's say:
- Bob has $20 
- Alice has $0

processTx function:
-> Alice.Balance = Alice.Balance + $10
-> Bob.Balance = Bob.Balance - $10

After the transaction is complete:
-> Bob has $10
-> Alice has $10

Well, that’s it! Bob has paid Alice!

If I’m candid, this engine wasn’t that exciting to build or develop. We made significant assumptions that usually don’t hold in any payment engine.

Let’s tackle those assumptions one by one now.

Is Bob that rich to be able to pay every time?

Before developing our engine, one of the assumptions we made was that Bob had all the money in the world to pay Alice, no matter the transaction amount. However, we know that’ll seldom be true.

So, let’s take a scenario, where Bob doesn’t have the amount to pay. In this scenario, Bob’s account balance would go negative in the above function. The above engine doesn’t perform the necessary checks and hence, it’s a loss to our company. So, let’s add a balance check.

func processTx(receiver, sender AccountDetails, amount int) {
    // check Bob's balance
    if sender.Balance - amount < 0 {
        return
    }

    ...
}

Okay, now it looks a little better. We added a balance check for our sender to be 100% sure that they can pay. Uhm, interesting!

What if I told you that we’re not out of the woods yet? There’s a problem with the above check.

The problem is an underflow. If the Balance attribute is always defined as an unsigned integer, deducting an amount larger than the Balance would underflow the minimum 0.

Hence, the check above doesn’t help!

Similarly, in our original function processTx, deducting an amount greater than Balance would ideally increase the Balance of Bob. Woah! All in all, our engine was paying Bob money to pay Alice. How cool is that?

Jokes aside, how do we fix that? Well, we could do the following too:

func processTx(receiver, sender AccountDetails, amount int) {
    // check Bob's balance
    if sender.Balance < amount {
        return
    }

    ...
}

A classic 5th-grade problem of solving inequalities by moving the variables around. I’m sure every one of us learned this too.

So, we’re down to 1 last assumption.

Don’t worry! Our engine never blows off!

Really? Well, I guess then everyone would use our engine. Right?

Sadly, that’s one of the biggest lies anyone can assume for their software. Hence, the second assumption. Our systems are intelligent and fast but prone to failure too.

Let’s consider a scenario of failure:

func processTx(receiver, sender AccountDetails, amount int) {
    ...

    save(receiver)
    // Boom! Our server blows off!
    save(sender)
}

What happens if our server blows off after saving the receiver's balance increase? Technically, we paid Alice and didn't deduct any money from Bob. However, if this was the case, I'm sure Bob would advertise our engine to the world!

Okay, so the easiest solution is: Save the sender's balance first and then the receiver's. So:

func processTx(receiver, sender AccountDetails, amount int) {
    ...

    save(sender)
    // Boom! Our server blows off!
    save(receiver)
}

The problem now is that our engine would have deducted the sender's account. But our receiver got no money? So now, Bob will be discrediting our engine, and no one will use it further.

So, there’s no solution for it? Because at least one should come before the other. Right?

Welcome, SMs!

What if we could divide the transaction into two separate steps? One that deducts the sender and the other that deposits the receiver.

Can we do that? How would that work? Do we call the second function from inside the first one? Wouldn’t that cause a problem?

Welcome, State Machines! (SMs)

To put it without any technical jargon, a State Machine defines the steps of your process and tracks its status. In our case, the transaction. Hence, a state machine would follow which phase the transaction is in, in the depositing or deducting step.

One thing to note here is that we’ve introduced the transaction concept. Earlier, we only had a receiver and a sender, adding/subtracting balances, as we’ve always learned.

Here, it looks like a Transaction needs to exist independently and not depend on receiver/sender details and their contexts. In the previous version, the receiver and sender contained the transaction details internally. (I skipped the details stored inside the AccountDetails object that tracks the transactions for brevity.)

However, we now create a transaction as a separate entity in our database. Therefore, our functions could process a transaction based on the states defined.
How does a transaction look now?

type Transaction struct {
    Receiver AccountDetails
    Sender   AccountDetails
    Amount   int
    State    string
}

Let’s define our states:

"deposited"
"deducted"

So, any new transaction that comes in would first have no state. Post that, once the depositing to the receiver is complete, it moves to the deposited state. And after that, once the amount has been deducted from the sender, it moves to the deducted state. The deducted state marks our transaction as complete.

So, let’s see how our function would change now:

func processTx(tx Transaction) {
    // balance check

    tx.Receiver.Balance += amount
    tx.State = "deposited"
    save(tx)

    tx.Sender.Balance -= amount
    tx.State = "deducted"
    save(tx)
}

Now we know if a transaction was in the deposited state when our server blew off, the receiver has been paid. Hence, we only need to re-process the deduction step. But, again, I'm only mentioning the details necessary at the moment. A lot goes on behind that save function, and we'll see those details too.

Yay! We have Engine! Eureka!

Are we done? Looks good, right?

Well, we can process the transaction and track its progress. Right? As a payments engine, it would work okay or in scenarios where nothing fails. So, what did we miss?

Let's track all of it down, and we'll tackle it in the following article, as this one's getting a little longer. So, some of the questions to ask with the above approach are:

We deduct the sender AFTER we've deposited the receiver. It means we're paying the receiver out of our pocket before we even take the money from the sender.
1. Alright! Let's swap the steps. Is that okay? Are we done? Would that work?
What happens when the server dies in the middle of the SM? Do we restart or process the transaction from where it was last processed? What happens if a transaction dies before the first state is deposited? Should we ask the sender to retry?
Can this mechanism support multiple payment methods like VISA, MasterCard, and more?

And a few more.

Summary

This article introduces you to State Machines, a crucial part of our transactions worldwide. SMs are a Computer Science concept used extensively in multi-step processes where each step could exist individually.

In the following article, we'll try to tackle a few of the above questions and also understand what goes on behind the scenes before the processTx function executes. Also, we'll look into the race condition of balances and how it affects our sender's account, allowing them to deduct once but pay twice.

Do we really just add/subtract balances? - Part 1

Akarsh Agarwal — Tue, 04 Oct 2022 13:51:15 +0000

My first payments engine, or everyone out here, was calculating the total price of carrots to be paid by Bob to Alice, in 4th grade, by doing addition and subtraction of balances. And Voila! Bob paid Alice. End of Transaction.

Almost 2 decades later, I’m still helping Bob transfer money to Alice for the carrots received!

From barter systems and gold coins to e-wallets and crypto, we’ve come a long way in transacting from items as simple as food orders to as complicated as acquiring a company. But, one constant thing is the thirst for instant gratification of a transaction to provide you with your essentials or dreams. From a simple math problem, countless companies have built the Payments system so instantaneous and mammoth.

My last 1 year at Grab in the Payments Team has made me aware of the sophistications the day-to-day payment solutions bring to our table. For example, users want their payments provider to be flawless, whether it's the happy path or an errored-out flow. Well, I'm so glad to break it to you; it's easier said than done.

Hence, I'm cashing out on my drive to share what I've learned to provide insight into how convoluted the payment services are. Therefore, I intend to deliver some high-level details, complemented with some technical-but-not-so-technical information, over a series of articles to share what I understand so far.

Finally, although I understand Crypto quite a bit, I’m going to keep that out for now and maybe tackle it after a few articles. So, if that’s your expectation, I’d recommend not reading it further.

So, let’s get started!

Modes of Payment

First and foremost, we need to have a mode to pay to create a transaction even before it begins. It is part of the equation that holds money/amount/balance. Earlier, it was the vaults of Gold or even briefcases of cash. But, we all know about these, so let’s not dive into them.

Over the past decade, we’ve seen an exponential increase in online transactions. Today, if I want a flight ticket, I buy it using my GrabPay Wallet or my card. I travel around Singapore without cash, knowing everyone accepts online payment methods. Hailing from a cash-heavy country like India, I was surprised to see the enablement of online transfers even at that corner-of-the-road stall.

The progress has been Humongous! Unbelievable!

So, what have been these different payment methods that have recently grown in acceptance and usage?

Wallets

Technically, something similar has been around since the advent of Net-Banking, if you look closely. For example, using your balance to pay for online transactions and seeing that number on the screen go down or depositing money in the account to see that number go up.

Wallets are similar, with a bit of flexibility and complexities to provide you with a similar experience, sometimes even better. In addition, wallets have capitalised on the concept of rewards that can be won and used when using the providers' wallets to make payments. Rewards and offers have been around for a while. However, I find that wallets nowadays provide more of them to attract customers.

One of the cool things about them: They have more integrations and features to allow for a one-stop shop.

Virtual Private Address

VPA is a relatively new concept and one of the fastest-growing payment methods. It's like an email where you can transfer the amount and it’s received into your bank account. It removes the hassle of adding account numbers and SWIFT codes to your bank account to be able to transfer funds.

It's my fav payment method. It's swift, even though it doesn’t use SWIFT codes, and hassle-free.

Buy Now Pay Later

One of the latest offerings by almost all payment service providers is “Buy Now, Pay Later.” You get the goods and services today while promising to pay after a specific time, usually at 0% interest and in instalments. So, it’s not like your Credit Cards. Instead, it’s an extended version of the same.

The goods and services are provided to you on credit while ensuring you can pay, like attaching a default debit card or payment account to the profile. Each company offering BNPL has its due diligence process to enable this feature. For example, is your credit score good enough?

Here, the seller usually pays interest to the credit provider behind the scenes. Sellers are attracted to it as it enables more buyers to transact.

Types of Transactions

Now, here's where it starts to get interesting. I believe most of us have seen all the below payment methods, even if it might look like we're not. However, considering we know how complicated and prone to error distributed systems and databases are, we'll not dive into these details here. So instead, the idea is to introduce all of these below-mentioned payment types.

So, let's dive briefly into it.

Direct Charge

The simplest form of transaction, one that we learned in 4th grade.

Deduct from an account and add into another. All at the same time. Easy-peasy!

It is one of the most prominent transaction types in the history of payments. For example, transferring amount to your friend. Another partial example would be, paying at the grocery stores. Well, why partial? We’ll see it in a bit.

Postpaid

When I hear about postpaid, I think about those landline and telecom postpaid connections that send a monthly bill to the registered address. You'll be thrilled to know; it's precisely that. However, postpaid is now applied to your everyday necessities and purchases too.

It would be an offering of BNPL if I were to categorise it. However, it can also be considered like credit cards, where the bill comes due at the end of each billing cycle.

Instalments

I believe we've had EMIs for quite some time now. And yet, instalments are taking a turn. How so? In almost every scenario, an interest is attached to your EMI. It would be added to your monthly credit card bill. However, BNPL offers 0% instalments most of the time. Is that what every buyer has been craving? Maybe or maybe not. The trends suggest that the answer to that question is a huge Yes.

Again, it resembles almost all the properties of credit cards. However, as I mentioned earlier, BNPL, to me, feels like an extended version of Credit Cards.

Auth and Capture

Oh! It sounds cool! Auth and Capture!

What if I were to say that you've been using this payment mechanism if you've ordered food online from one of the online food delivery providers? Online food deliveries or ride-hailing apps, for that matter, usually use this mechanism.

So, what are Auth and Capture?

A transaction for a food order executes in 2 separate steps, Auth and Capture. In the two-step process, an amount gets blocked as per your order amount when you place an order. You've not been charged yet. It's blocked, so you cannot use that amount. When the food is delivered is the time when you get charged on your account for that order.

It works the same way for ride-hailing apps or services. You're not charged until the ride completes. After that, however, you cannot use the amount as it's blocked to be charged later by your payment provider.
It is to assert that you'll be able to make the payment once the service completes.

Hence, as I mentioned, we've all used this payment type before. It works behind the scenes and is oblivious to the buyer or rider.

Summary

We’ll stop here for now! It’s a journey to share what I’ve learned so far at Grab and how I’ve come to understand the intricacies of our day-to-day payment systems. Focusing more on online methods of payments and types of transactions, I introduced a few already known terms like postpaid and instalments. Moreover, we also picked up some of the new terminologies, like VPA and BNPL, as the payments ecosystem advances globally.

In the next blog, I’ll dive into the transaction entities involved. Additionally, we’ll learn about other processes apart from the addition and subtraction of balances crucial to a transaction. Finally, I’ll introduce you to a straightforward payment engine and how it works. Towards the end, we’ll target to understand how a single payment provider integrates and works with multiple providers and banks.

I look forward to seeing you at the next one! Please do leave constructive feedbacks or questions in the comments section below!