DEV Community: Raffi Sarkissian

Time zones are a nightmare for engineers

Raffi Sarkissian — Tue, 14 Mar 2023 14:04:25 +0000

Time zones suck. There’s a litany of posts advocating for the “end of Time zones”. There are pros and cons but the net is they won’t go anywhere.

The most fascinating thing about this is that there’s a lot of literature on “how to work across time zones” but so little on “how to build a product that has to handle timezones”.

We’ve come across this tweet from Peer at Cal.com, and the meme by Daniel that is just on point.

So here is how we approached time zones at Lago (we build open-source metering and usage-based billing) and why you might need to think about time zones for your own product too, if you’re addressing a global customer base.

Billing and Time Zones: why it matters

Yes. Your timezone may not match your customer's. This could lead to discrepancies in usage ingestion, subscriptions, and invoicing, which may not occur on the date you anticipated.
If your customer doesn’t understand what they’re billed for, it erodes trust and will incur “billing disputes”.

Expiry dates

The most telling example is how time zones impact expiry dates.
Expiry dates can be set for contracts, coupons, or wallets that hold prepaid credits to cover future usage.
If you’re a company based in Paris, France, and have a customer in San Francisco, USA, with a coupon expiring on 2023-02-01T00:00UTC (February 1st, 2023 at midnight). Your customer might expect to be able to use their prepaid credits until the end January 31st, 2023, their time: Pacific Standard Time.

With this day finishing nine hours earlier in the French time zone, that might trigger a “bad surprise” for the user. That kind of discrepancies can also impact subscriptions (when they start or end) and gaps in your customers’ financial records.

Double counting or missing consumption when the time zone changes

Then comes more tedious edge cases that can have considerable impact on billing and the end user’s experience.

Managing the boundaries of a billing period when a customer or an organization changes their timezone is one of them. For example, let’s say a customer had an active subscription, whose timezone is UTC, and for which the billing frequency is “monthly”. In February, we must take into account all the events that can impact their billing (e.g., additional consumption) that occurred between February 1, 2023, at 00:00:00 UTC and February 28, 2023, at 23:59:59 UTC.

If on February 15, this customer changes their time zone to Tokyo time (UTC+9) within our product, the "new billing period" that will be calculated will run from January 31, 2023, at 14:00:00 UTC to February 28, 2023, at 13:59:59 UTC.

The risk is that events received between January 31, 2023, at 14:00:00 UTC and February 1, 2023, at 00:00:00 UTC will be charged twice, once for January UTC and again for February UTC+9.

Therefore, we need to "realign" the start of the period to February 1, 2023, at 00:00:00 UTC.

If, instead of changing to Tokyo time, the customer switches to Los Angeles time (UTC-8), the "new billing period" will run from February 1, 2023, at 08:00:00 UTC to March 1, 2023, at 07:59:59 UTC.

In that case, we need to "realign" the start of the period to January 31, 2023, at 13:59:59 UTC.

Our learnings along the way

Vincent in our team was the lead on this feature, here are his own words:

1. What solutions have you considered for "translating" UTC into the end customer's timezone?

In general, it is considered best practice to store dates in a unified and comparable format from one record to another. That's why we decided to store dates in UTC (in the ISO 8601 format) because it provides a unique reference and is not sensitive to changes in time or modifications to the organization's or customer's timezone. Dates are only converted to the user's format at the last moment for display. We had considered storing dates in the database along with timezone information, but ultimately it made the recording process much more complicated, raised questions about updates, and introduced significant complexity in date comparisons because the organization's and customer's time zones may not be the same.

2. How did you learn or find best practices about this topic?

To be honest, I’ve scrolled the web to learn as many things as I could, I think Stackoverflow was the best source of info, for instance:

3. Did you iterate on the approach? What were your key learnings?

This is not the first time I have had to deal with this challenge, so I relied heavily on my personal experience and the path was "fairly clear" for me. However, we iterated on the "edge cases" which are numerous with this kind of problem, and making things understandable for the end-user is a constant challenge.

If I had to summarize our approach:

To manage time zones, we should not handle data in the "date" format (without the time information) because a date only makes sense in a particular time zone. This impacts things like usage limits for coupons, or the start and end of a subscription.
For certain objects, such as invoices, we need a blocked "accounting date" because it is displayed in a document. We have decided to always define it in the timezone applicable to the customer.
Automatic tasks should be processed on an hourly basis, never on a daily basis, because some elements might have “switched” day depending on the time zone, and that impacts billing. This means that instead of wondering "who should I bill today?", the real question is "what are the subscriptions for which we are currently on YYYY-MM-DD?" And safeguards need to be added to avoid billing the same subscription multiple times.
Postgres (our database) is very helpful for managing time zones, particularly thanks to the AT TIME ZONE operator.
There are many different time zones, depending on the offset from the GMT reference, daylight saving time, and other fun subtleties (for example, some zones in Asia/Oceania have offsets set on quarter-hours). We use a simplified list of 134 zones (source).
And another fun fact (We had a lot of fun!): Time Zones often have a "friendly name" in the format Continent/City|Island except for UTC and... GMT+12, which only covers two uninhabited American islands :D

The result

By default, Lago is ingesting usage and invoicing customers based on the system-wide UTC. However, you can decide to change the value at your organization level to make sure that every single customer is billed based on your timezone.

LAGO_URL="<https://api.getlago.com>"
API_KEY="__YOUR_API_KEY__"

curl --location --request PUT "$LAGO_URL/api/v1/organizations" \\
  --header "Authorization: Bearer $API_KEY" \\
  --header 'Content-Type: application/json' \\
  --data-raw '{
    "organization": {
      "name": "Name1",
      "webhook_url": "<https://test-example.example>",
      "country": "CZ",
      "address_line1": "address1",
      "address_line2": null,
      "state": "state1",
      "zipcode": "10000",
      "email": "org@email.com",
      "city": "city125",
      "legal_name": null,
      "legal_number": null,
      "timezone": "Europe/Paris", //Changes billing record to your organization timezone
      "billing_configuration": {
        "invoice_footer": "footer custom",
        "invoice_grace_period": 3,
        "vat_rate": 15.0
      }
    }
  }'

In addition to this, you can decide to overwrite this value per customer. The entire billing engine will then follow the exact time zone sets at a customer level. Usage ingestion, expiry dates and invoicing will be triggered based on the timezone of your customer.

LAGO_URL="<https://api.getlago.com>"
API_KEY="__YOUR_API_KEY__"

curl --location --request POST "$LAGO_URL/api/v1/customers" \\
  --header "Authorization: Bearer $API_KEY" \\
  --header 'Content-Type: application/json' \\
  --data-raw '{
    "customer": {
      "external_id": "5eb02857-a71e-4ea2-bcf9-57d3a41bc6ba",
      "address_line1": "5230 Penfield Ave",
      "address_line2": null,
      "city": "Woodland Hills",
      "timezone": "America/Los_Angeles", //Overwrite the value per customer
      "url": "<http://hooli.com>",
      "zipcode": "91364",
      "billing_configuration": {
        "payment_provider": "stripe",
        "provider_customer_id": "cus_12345",
      }
    }
  }'

This translates the entire billing logic to follow your customer’s timezone.

If this seems very simple within Lago, it’s on purpose: we worked hard to abstract the time zone logic, so that you don’t even have to think about it!

Et voilà !

Using Ruby’s BigDecimal to build a billing engine

Raffi Sarkissian — Mon, 05 Dec 2022 15:01:17 +0000

We’ve recently gone deep down the rabbit hole of 'BigDecimal'. Although we accumulated decades of experience in Ruby, this was more challenging than we initially thought, because of the few things we discovered along the way (that led to quite a bit of rework!).

Why we chose BigDecimal

Context: We build Lago, the open-source metering and usage-based billing.

Our need:

We handle objects with up to 5 decimals, without transforming them. Why? We need to keep each object as precise as possible, and delay the rounding to 2 decimal places to the last step: when our users choose to display a charge or an invoice in a specific currency, for instance.

Why didn’t we just store values in decimal? We need to support multiple decimals for complex pricing : e.g. $0,0015 per hour. We want to give the maximum flexibility to our users to fit their pricing needs.

The potential solutions:

BigInt: the ‘by default’ option with Ruby doesn’t support decimals, so we ruled it out.

Float objects: we hit accuracy limits very quickly as float in Ruby (like in many other languages) can lead to rounding issues, it’s designed for performance and not for precise calculations.

BigDecimal: we knew it ranked lower on the performance side, but it seemed to be the only relevant option. Moreover, PostgreSQL provides a Decimal type that is very well integrated with Rails Active Record, which, combined with BigDecimal provides an ‘out-of-the-box’ validation for scale and precision.

Example of a migration: With BigDecimal, we can specify precision and scale in the decimal column (PostgreSQL).
t.decimal :rate_amount, null: false, default: 0, precision: 5

Precision: total number of digits (including before and after the decimal point).

Scale: number of digits after the decimal point.

The problems we ran into

1. Rounding errors and execution time

Float rounding leading to potential errors

Execution time for a Float calculation

Execution Time for the same calculation, using BigDecimal

2. Front-End & Back-End coordination

Handling numbers can be tricky on the front-end side. We either have different types (string, float, integers), format (cents, decimal) or purpose (money, percent). For all of them we also need to change the UI display, depending on currency and user’s locale.

How we solved them

Back-End and Front-End agreed on some rules when dealing with numbers. If a value sent or received contains a decimal (no matter the precision), we’ll manage it as type ‘String’. On the other hand, all other values that won’t contain any decimal will be of type ‘Number’.

The Front-End component libraries can accept both types as inputs, so we don’t have to think about type management when building our interfaces. Internally, for manipulation and formatting, those components will systematically transform values into type ‘String’.

For display purposes, we use Javascript APIs that accept options to define the number of decimals or the display format (currency, percent).

Final words

We shared these notes and code snippets in the simplest form possible, because it’s what we were looking for when we researched the issue. From the questions we spotted in the Ruby community, it looks like other people were confused as well, so we hope this post proves useful to you as well.

6 steps to build a usage-based billing system with Lago (YC S21)

Raffi Sarkissian — Thu, 08 Sep 2022 13:13:41 +0000

Lago is an open-source billing API that helps engineers implement their billing systems faster. We do think the future of pricing is hybrid, which includes subscription fees and usage-based charges. Over the past few years, engineers have struggled to implement exotic billing systems but this is over!

Lago also provides a cloud-hosted application for those who don’t want to run it on their own environment.

We are going to build an entire hybrid billing system, including plans and usage-based features. Let’s try to replicate Segment’s pricing for instance!

Prerequisites

You can find the main Lago repository on GitHub.

Before we get started, you need to:

Install Docker on your machine;
Make sure Docker Compose is installed and available (it should be the case if you’ve chosen to install Docker via Docker Desktop);
Make sure Git is installed on your machine; and
A cup of tea (or coffee)! ☕️

This demo is made through basic CURL API calls, but Lago also provides API clients in order to make the implementation process easier.

1. Setting up your Lago billing engine

Start running Lago on your own environment. You will have access to Lago API and Lago UI (frontend admin panel to perform no-code tasks).

# Get the code
git clone https://github.com/getlago/lago.git

# Go to Lago folder
cd lago

# Set up environment configuration
echo "LAGO_RSA_PRIVATE_KEY=\"`openssl genrsa 2048 | base64`\"" >> .env
source .env

# Start (make sure docker is started)
docker-compose up

You can now open your browser and go to http://localhost/ to connect to the application. Lago's API is exposed at http://localhost:3000/.

If needed, you can override your environment variables (learn more).

2. Defining your billable metrics

The usage-based feature billed by Segment is the number of monthly tracked users (MTUs), which is related to the total number of unique users tracked monthly.

Lago’s billable metrics allow you to track and automatically aggregate usage to determine the number of units to be charged at the end of a billing period.

Here we’re going to define MTUs as a metered billable metric:

LAGO_URL="https://api.getlago.com"
API_KEY="__YOUR_API_KEY__"

curl --location --request POST "$LAGO_URL/api/v1/billable_metrics" \
  --header "Authorization: Bearer $API_KEY" \
  --header 'Content-Type: application/json' \
  --data-raw '{
    "billable_metric": {
      "name": "Monthly Tracked Users",
      "code": "mtu",
      "description": "Number of unique users tracked by Segment",
      "aggregation_type": "unique_count_agg",
      "field_name": "user_id"
    }
  }'

For this billable metric, the aggregation type is unique_count_agg, which is used to count the number of unique user_id values recorded during the billing period.

3. Creating your first plan

Now that we have our billable metric, we can create our first plan. The plan defines the billing period and all fees (i.e. subscription and usage-based charges).

Let’s take the example of Segment’s Team plan. The base subscription costs $120 per month month, including 10,000 free MTUs. If you track more users, you will be charged for the overage, according to Segment’s graduated pricing model.

LAGO_URL="https://api.getlago.com"
API_KEY="__YOUR_API_KEY__"

curl --location --request POST "$LAGO_URL/api/v1/plans" \
--header "Authorization: Bearer $API_KEY" \
--header 'Content-Type: application/json' \
--data-raw '{
  "plan": {
    "name": "Team",
    "code": "team",
    "interval": "monthly",
    "description": "Team plan, ideal for startups",
    "amount_cents": 12000,
    "amount_currency": "USD",
    "trial_period": 0,
    "pay_in_advance": true,
    "bill_charges_monthly": true,
    "charges": [
      {
        "billable_metric_id": "__BILLABLE_METRIC_ID__",
        "charge_model": "graduated",
        "properties": [
          {
            "to_value": 10000,
            "from_value": 0,
            "flat_amount": "0",
            "per_unit_amount": "0.00"
          },
          {
            "to_value": 25000,
            "from_value": 10001,
            "flat_amount": "0",
            "per_unit_amount": "0.012"
          },
          {
            "to_value": 100000,
            "from_value": 25001,
            "flat_amount": "0",
            "per_unit_amount": "0.011"
          },
          {
            "to_value": null,
            "from_value": 100001,
            "flat_amount": "0",
            "per_unit_amount": "0.010"
          }
        ]
      }
    ]
  }
}'

Let’s take a closer look at the request above.

Basic plan model

The request creates a basic plan at $120/month that is paid in advance (beginning of period), with no free trial. If can customise it using the following fields:

• interval: can also be set to yearly or weekly;
• amount_currency: all currencies are available;
• trial_period: set a number of trial days; and
• pay_in_advance: if set to false, the subscription fee will be paid in arrears (end of period).

Usage-based charges

In this example, we selected a graduated charge model for our usage-based feature (i.e. MTUs). Here’s what we can see in the user interface.

You can create as many charges as you want and there are many charge models available (e.g. volume pricing, standard pricing, percentage pricing, package pricing).

4. Creating your first customer

Congratulations, your pricing is ready! You can now create a customer, assign them a plan and start monitoring usage.

LAGO_URL="https://api.getlago.com"
API_KEY="__YOUR_API_KEY__"

curl --location --request POST "$LAGO_URL/api/v1/customers" \
  --header "Authorization: Bearer $API_KEY" \
  --header 'Content-Type: application/json' \
  --data-raw '{
    "customer": {
      "external_id": "5eb02857-a71e-4ea2-bcf9-57d3a41bc6ba",
      "address_line1": "5230 Penfield Ave",
      "address_line2": null,
      "city": "Woodland Hills",
      "country": "US",
      "email": "dinesh@piedpiper.test",
      "legal_name": "Coleman-Blair",
      "legal_number": "49-008-2965",
      "logo_url": "http://hooli.com/logo.png",
      "name": "Gavin Belson",
      "phone": "1-171-883-3711 x245",
      "state": "CA",
      "url": "http://hooli.com",
      "vat_rate": 12.5,
      "zipcode": "91364",
      "billing_configuration": {
        "payment_provider": "stripe",
        "provider_customer_id": "cus_12345",
        "sync": true
      }
    }
  }'

This request allows you to create a customer and also to define the default payment provider (i.e. the payment service provider that will be used to collect payments for this customer). We also set a tax rate of 12.5% for this customer.

5. Assigning a plan to a customer

To start billing a customer, you need to assign them a plan. This action will create a subscription.

LAGO_URL="https://api.getlago.com"
API_KEY="__YOUR_API_KEY__"

curl --location --request POST "$LAGO_URL/api/v1/subscriptions" \
  --header "Authorization: Bearer $API_KEY" \
  --header 'Content-Type: application/json' \
  --data-raw '{
    "subscription": {
            "external_id": "my_subscription_1",
      "external_customer_id": "5eb02857-a71e-4ea2-bcf9-57d3a41bc6ba",
      "plan_code": "team",
      "billing_time": "anniversary"
    }
  }'

When assigning a subscription, you can:

• Define your own external_id;
• Choose between anniversary or calendar for the billing time - a subscription based on calendar dates is billed on the first day of each week/month/year, while a subscription based on the anniversary date is billed on the day it was created (e.g. every 4 of the month); and
• Enter a subscription name that will be displayed on the invoice and will allow you to differentiate subscriptions that share the same plan (e.g. Project 1 on the Free plan and Project 2 also on the Free plan).

As the base amount of the Team plan has to be paid in advance, Lago automatically generates a first invoice. The usage-based charge related to MTUs will be billed at the end of the period.

This invoice can be customized with information about your organization, your logo and a custom footer. For compliance purposes, invoices have an incremental sequential_id and incremental invoice_number as well.

6. Ingesting usage-based events

Lago has an event-based architecture. When a subscription is associated with a customer, you can start pushing events related to your billable metrics. By using a unique transaction_id, we make sure that an event is not ingested twice. It’s what we call an idempotency key.

Below are three events ingested by Lago:

Event #1

LAGO_URL="https://api.getlago.com"
API_KEY="__YOUR_API_KEY__"

curl --location --request POST "$LAGO_URL/api/v1/events" \
--header "Authorization: Bearer $API_KEY" \
--header 'Content-Type: application/json' \
--data-raw '{
      "event": {
          "transaction_id": "transaction_1",
          "external_customer_id": "5eb02857-a71e-4ea2-bcf9-57d3a41bc6ba",
          "code": "mtu",
          "timestamp": 1662462118,
          "properties": {
            "user_id": "1234_5678_9012_3456"
          }
      }
  }'

Event #2

LAGO_URL="https://api.getlago.com"
API_KEY="__YOUR_API_KEY__"

curl --location --request POST "$LAGO_URL/api/v1/events" \
--header "Authorization: Bearer $API_KEY" \
--header 'Content-Type: application/json' \
--data-raw '{
      "event": {
          "transaction_id": "transaction_2",
          "external_customer_id": "5eb02857-a71e-4ea2-bcf9-57d3a41bc6ba",
          "code": "mtu",
          "timestamp": 1662548611,
          "properties": {
            "user_id": "1234_5678_9012_3456"
          }
      }
  }'

Event #3

LAGO_URL="https://api.getlago.com"
API_KEY="__YOUR_API_KEY__"

curl --location --request POST "$LAGO_URL/api/v1/events" \
--header "Authorization: Bearer $API_KEY" \
--header 'Content-Type: application/json' \
--data-raw '{
      "event": {
          "transaction_id": "transaction_3",
          "external_customer_id": "5eb02857-a71e-4ea2-bcf9-57d3a41bc6ba",
          "code": "mtu",
          "timestamp": 1662721411,
          "properties": {
            "user_id": "0000-1111-2222-3333"
          }
      }
  }'

We have three different events, however, two of them are sending the same user_id. As we use a unique count aggregation type for the billable metric MTUs, the number of units to be charged is 2.

Lago then automatically calculates the fees depending on the charge model of the plan. With Segment’s pricing, the first 10,000 MTUs are free of charge. Therefore, these two units cost $0.

This is how Lago can help you implement a usage-based billing system in just a few hours (not months). It’s easy to build but also easy to maintain.

In addition to this, Lago manages upgrades/downgrades, taxes, discounts, prepaid credits and prorated fees. Ready to give it a try?

The 5 reasons why we chose open source for our Billing API

Raffi Sarkissian — Mon, 05 Sep 2022 13:33:11 +0000

When we started Lago, we knew billing was still an unsolved problem, but we never considered launching another traditional billing SaaS.

We had experienced the pain of billing so deeply and for so long (our team built a billing system from scratch for a fintech unicorn) that we thought a traditional approach would lead to traditional results.

We want to make pricing and billing more programmable, open, and fair for all software companies.

1. Open architecture

Traditional billing or payment SaaS have become the source of financial truth for companies. However, it’s unclear how they interconnect with other internal business tools, and sometimes they don’t, by design: for instance, Stripe Billing can only be used with Stripe Payments.

Billing is at the heart of every B2B company’s survival but as there are myriad internal tool stacks (including accounting, payments, CRM, CPQ), the setup needs to be different for each company. Therefore, we decided to open-source our code and make the interconnections as easy as possible.

2. Developers first

None of the tools were made for engineers, so billing is still an engineering nightmare. It’s the most high-value, underrated and unsystematized business process we’ve seen.

It’s hard to know exactly why, but we believe the root cause is that existing billing solutions have catered to Finance people first. Just as payment solutions did before Stripe Payments came in with its API. Billing should address the needs of the Finance teams (obviously), but the implementation and maintenance of the system will always depend on engineers.

Therefore, we chose to go open-source, provide full transparency and be as close as possible to developers: they can submit issues on Lago's GitHub repo, talk to us on Slack, give us feedback on what we’ve built and comment on our product roadmap.

3. Fair pricing

We’re infuriated by the ‘rent seeker approach’ chosen by existing billing solutions. When you’re a billing middleman, a 1% fee on your customers’ hard-earned revenue doesn’t sound right.

That’s one of the reasons why we chose to build our home-made billing solution at our previous company, but it wasn’t perfect either: a lot of resources were pulled out of our core product to be allocated to this project.

Lago is open-source, forever free. We’ll also offer a cloud version for teams that don’t want to deal with the infrastructure themselves, which will allow us to monetize our solution and keep working towards a more programmable, open, and fair billing system for all software companies.

4. Composability

Each company’s pricing is different. The time of a one-size-fits-all subscription model is over and much ink has already been spilled by VCs on ‘usage-based pricing’. It’s a big shift but we’re convinced 80% of companies will switch to hybrid pricing: a mix of subscription and consumption-based model, with a million nuances in between.

A closed-source SaaS can’t embrace this diversity and, through Lago, we want to help build as many pricing models as possible.

Any company should be able to customize and deploy a billing system similar to that of Snowflake, Algolia, or Segment, 100x faster. And early-stage companies should be able to do so without hiring an army of engineers.

5. Participative approach

The most complicated part of billing is the long list of edge cases (we covered this topic in our previous article).

These are too small to be prioritized in any closed-source SaaS product roadmap, but they keep coming back on Stack Overflow, Hacker News, Reddit, and all the places where builders seek advice.

Beyond our product, we’re also aiming at creating this hub for shared knowledge, so that no more engineers have to struggle with billing edge cases.

If you share our vision, deploy our self-hosed version for free, or ask for an access to our cloud hosted version: https://www.getlago.com/get-started

😵‍💫 Why billing systems are a nightmare for engineers

Raffi Sarkissian — Thu, 01 Sep 2022 13:22:32 +0000

This article was initially published on Lago's blog, an open-source billing API, and was ranked #1 on HackerNews for 2 consecutive days.

"On my first day, I was told: 'Payment will come later, shouldn't be hard right?' I was worried. We were not selling and delivering goods, but SSDs and CPU cores, petabytes and milliseconds, space and time. Instantly, via API. Fungible, at the smallest unit. On all continents. That was the vision. After a week, I felt like I was the only one really concerned about the long road ahead. In ambitious enterprise projects, complexity compounds quickly: multi-tenancy, multi-users, multi-roles, multi-currency, multi-tax codes, multi-everything. These systems were no fun, some were ancient, and often 'spaghetti-like'. What should have been a one-year R&D project ended up taking seven years of my professional life, in which I grew the billing team from 0 to 12 people. So yes, if you ask me, billing is hard. Harder than you think. It's time to solve that once and for all."
Kevin Deldycke, former VP Engineering at Scaleway

This is a typical conversation we have with engineers on a daily basis.

After my last post about my 'pricing hack', some of you asked me why billing was that complex. My co-founder Raffi took up the challenge of explaining why it's still an unsolved problem for engineers.

We also gathered insights from other friends who went through the same painful journey, including Algolia, Segment, Pleo... don't miss them! Passing the mic to Raffi.

When you're thinking about automating billing, this means your company is getting traction. That's good news!

You might then wonder: should we build it in-house? It does not look complex, and the logic seems specific to your business. Also, you might want to preserve your precious margins and therefore avoid existing billing solutions like Stripe Billing or Chargebee, that take a cut of your revenue. Honestly, who likes this 'rent-seeker' approach?

Our team at Lago still has some painful memories of Qonto's internal billing system, that we had to build and maintain. Why was it so painful? In this article, I will provide a high-level view of the technical challenges we faced while implementing hybrid pricing (based on subscription and usage) and what we learned during this journey.

TL;DR: Billing is just 100x harder than you think

"Let's bill yearly as well, should be pretty straightforward" claims the Revenue team. Great! Everyone is excited to start working on it. Everyone, except the Tech team. When you start building your internal billing system, it's hard to think of all the complexity that will pop up down the road, unless you've experienced it before.

It's common to start a business with a simple pricing. You define one or two price plans and limit this pricing to a defined number of features. However, when the company is growing, your pricing gets more and more complex, just like your entire codebase.

At Qonto, our first users could only onboard on a €9 plan. We quickly decided to add plans, and 'pay-as-you-go' features (e.g. ATM withdrawals, foreign currency payments, capital deposit, etc.) to grow revenue.

Also, as Qonto is a neobank, we wanted to charge our customers' wallets directly, through a ledger connected to our internal billing system. The team started from a duo of full-time engineers building a billing system (which is already a considerable investment) to a dedicated cross-functional Pricing team.

This is not specific to Qonto of course. Pleo, another fintech unicorn from Denmark faced similar issues:

"I've learned to appreciate that billing systems are hard to build, hard to design, and hard to get working for you if you deviate from 'the standard' even by a tiny bit."
Arnon Shimoni, Product Billing Infrastructure Lead at Pleo

This is not even specific to fintechs. The Algolia team ended up creating a whole pricing department, now led by Djay, a pricing and monetization veteran from Twilio, VMWare, Service Now. They switched to a 'pay-as-you-go' pricing model based on the number of monthly API searches.

"It looks easy on paper — however, it's a challenge to bring automation and transparency to a customer, so they can easily understand. There is a lot of behind-the-scenes work that goes into this, and it takes a lot of engineering and investment to do it the right way."
Bernardette Nixon, CEO at Venture Beat

#1 - Dates

When implementing a billing system, dealing with dates is often the number 1 complexity. Somehow, all your subscriptions and charges are based on a number of days. Whether you make your customers pay weekly, monthly or yearly, you need to define the billing period.

Here is a non-exhaustive list of difficulties for engineers:

How to deal with leap years?
Do your subscriptions start at the beginning of the month or at the creation date of the customer?
How many days/months of trial do you offer?
Who decided February only holds 28 days?
Wait, point 1 is also important for February...
How to calculate usage-based charges (price per second, hour, day)?
Do I resume the consumption or do I stack it month over month? Year over year?
Do I apply a pro-rata based on the number of days consumed by my customer?

Although every decision is reversible, billing cycle questions are often the most important source of customer support tickets, and iterating on them is quite complex. For instance, Qonto switched from 'anniversary' dates to calendar dates for its billing periods (see here). This change was not trivial.

#2 - Upgrades & downgrades

You need to allow your customers to upgrade or downgrade their subscriptions. Moving from plan A to plan B seems pretty easy, but it's not. Let's zoom on potential edge cases.

What should we do when the user:

Upgrades or downgrades in the middle of a period;
Has paid for the original plan in advance;
Needs to pay for the original plan in arrears;
Downgrades from a yearly plan to a monthly plan;
Upgrades from a monthly plan to a yearly plan;
Upgrades or downgrades from a plan paid in advance to a plan paid in arrears (and vice-versa); or
Has a discount and upgrades/downgrades?

We did not have a free trial period at the time at Qonto, but Arnon from Pleo describes this situation here.

#3 - Usage-based computations

Subscription-based billing is the first step when implementing a billing system. Each customer needs to be affiliated to a plan in order to start charging the right amount at the right time.

But for a growing number of companies, like Qonto, other charges come alongside this subscription. These charges are based on what customers really consume. This is what we call 'usage-based billing'. Most companies end up having a hybrid pricing model: a monthly subscription fee and 'add-ons' or 'pay-as-you-go' charges on top of it.

These consumption-based charges are tough to track at scale, because they often come with calculation rules performed on a high volume of events.

Here are some examples:

Segment's pricing is based on the number of monthly tracked users. This means that they need to count the distinct number of users each month and reset this value to zero for the next billing period. In order to retrieve the number of unique visitors, they need to apply a 'distinct' function to deduplicate them.

{
  "transaction_id": "2345",
  "timestamp": "2022-03-16",
  "event_type": "tracked_users",
  "properties": {
            "user_id": "123456789"
    }
}

SELECT
    DATE_TRUNC('month', timestamp) AS month,
    COUNT(DISTINCT user_id) AS unique_users
FROM events
WHERE event_type = 'tracked_used'
GROUP BY month

Algolia tracks the number of API searches per month. This means they need to sum the number of monthly searches for each customer and reset this value to zero for the next billing period.

{
  "transaction_id": "2345",
  "timestamp": "2022-03-16T00:00:00Z",
  "event_type": "searches",
  "properties": {
            "api_search": "1"
    }
}

SELECT
    DATE_TRUNC('month', timestamp) AS month,
    SUM(api_search) AS monthly_searches
FROM events
WHERE event_type = 'searches'
GROUP BY month

It becomes even more complex when you start calculating charges based on a timeframe. For instance, Snowflake charges for the compute usage of a data warehouse per second. This means that they sum the number of gigabytes or terabytes consumed, multiplied by the number of seconds of compute time.

Consider the example of a utility company that charges $10 per kilowatt of electricity per hour. The illustration below shows what needs to be modeled and automated by the billing system.

• Hour 1: *10 KW used for 0.5 hour = 5 KW (10 x 0.5)
*• Hour 2: 20 KW used for 1 hour = 20 KW (20 x 1)
• Hour 3: *0 KW used for 1 hour = 0 KW (0 x 1)
*• Hour 4: 30 KW used for 0.5 hour = 15 KW (30 x 0.5)
• TOTAL = 40 KW used x $10 ⇒ $40

#4 - Idempotency done right

Billing errors sometimes occur. Charging a user twice for the same product is obviously bad for the customer experience, but failing to charge when needed hurts revenue. That's partly why Finance and BI teams spend so much time on revenue recognition.

For 'pay-as-you-go' companies, the billing system will process a high volume of events. When an event needs to be replayed, it needs to happen without billing the user again. Engineers call it idempotency, which means the ability to apply the same operation multiple times without changing the result beyond the first try.

It's a simple design principle, however, maintaining it at all times is hard.

#5 - The case for a Cash Collection Officer

Cash collection is the process of collecting the money that customers owe you. The pet peeve of cash collection is dunning: when payments fail, the merchant needs to persist and make repeated payment requests to their customers, while trying not to damage the relationship.

At Qonto, we called these 'waiting funds'. A client's status was 'waiting funds' when they successfully went through the sign-up, KYC and KYB process but their account balance was still 0.

For a neobank, the impact is twofold: you can't charge your service fees (e.g. monthly subscription) and your customer does not generate interchange revenues (e.g. when you make a payment of $100 with your card, the card issuer earns $0.5-$1 of interchange revenue through the merchant's fees).

Therefore, your two main revenue streams are 'null' but you did pay to acquire, onboard, KYC the user and then to produce and send them a card. We often half-joked about the need to hire a 'chief waiting funds officer': the financial impact of this is just as high as the problem is underestimated.

All companies face dunning challenges. For engineers, on top of the billing architecture, this means that they need to design and build:

A **retry logic **to ask for a new payment intent;
An invoice reconciliation system (e.g. if several months of charges are recovered);
An app logic to disable access to the service; and
An emailing workflow to urge a user to proceed to the payment.

Some SaaS are even on a mission to fight dunnings and have built full-fledged companies around cash collection, such as Upflow, which is used by successful B2B scale-ups, including Front and Lattice.

"Sending quality and personalized reminders took us a lot of time and, as Lattice was growing fast, it was essential for us to scale our cash collection processes. We use Upflow to personalize how we ask our customers for money, repeatedly, while keeping a good relationship. We now collect 99% of our invoices, effortlessly."
Jason Lopez, Controller at Lattice

#6 - The labyrinth of taxes and VAT

Taxes are challenging and depend on multiple dimensions. Taxes depend on what you are selling, your home country and your customer's home country. In the simplest situations, your tax decision tree should look like this:

Now, imagine that you sell different types of goods/services to different types of customers in 100+ countries. If you think the logic on paper looks complex, the engineering challenge is tenfold. Engineers need to think of an entire tax logic within the application. This pyramidal logic is based on customers and products, including:

1. Taxes at company level: your company will have a general tax rate that applies to all customers by default;
2. Taxes at customer level: the default tax rate can be overridden for each customer, depending on the dimensions mentioned above (see illustration); and
3. Taxes at feature level: this is mostly the case for the banking industry. At Qonto for instance, banking fees are not subject to taxes but a 20% VAT rate applies to non-banking features.

With billing, the devil's in the detail, which is why I always cringe when I see engineering teams build a home-made system, just because they think "it's not that complex".

If you've already tackled the topics listed above and think it's a good investment of your engineering time, go ahead and build it in-house, but make sure to budget for the maintenance work that is always needed. Another option is to rely on existing billing platforms, built by specialized teams.

To solve this problem at scale, we've adopted a radical approach: we're building an open-source billing API for product-led SaaS. Our API and architecture are open, so that you can embed, fork and customize them as much as your pricing and internal processes require.

If you're interested, you can reach out to the Lago team or take a look at our open-source repo: https://github.com/getlago/lago