DEV Community: Monite

API Versioning at Monite

Stanislav Zmiev — Thu, 29 Aug 2024 11:49:59 +0000

We all love to have shiny new tools but hate the chore of constantly updating them. This applies to anything: operating systems, apps, APIs, linux packages. It is painful when our code stops working because of an update and it is double the pain when the update was not even initiated by us.

In web API development, you are constantly at risk of breaking your users' code with every new update. If your product is an API, then these updates will be terrifying every time. Monite's main products are our API and our white-label SDK. We are an API-first company so we take great care of keeping our API stable and easy to use. Hence the problem of breaking changes is near the top of our priority list.

A common solution is to issue deprecation warnings to your clients and to release breaking changes rarely. Suddenly, your releases can now take months and some features have to stay hidden or even unmerged until each next release. This slows down your development and forces your users to update their integration every few months.

If you make releases faster, your users are going to have to update their integration too often. If you lengthen the time between releases, you will move slower as a company. The more inconvenient you make it for users -- the more convenient it is going to be for you, and vice versa. This is certainly not an optimal scenario. We wanted to move at our own pace without breaking anything for existing clients which would be impossible with a regular deprecation approach. This is why we picked an alternative solution: API versioning.

It is quite a simple idea: release any breaking changes at any time but hide them under a new API version. It grants you the best of both worlds: The users will not have their integrations routinely broken and you will be able to move at any speed you like. The users will migrate whenever they want -- free of any pressure.

Considering the simplicity of the idea, it feels perfect for any company. That is what you would expect to read in a typical engineering blog. Sadly, it is not so simple.

Beware of the price

API versioning is hard, very hard. Its illusory simplicity quickly fades away once you begin implementing it. Sadly, the internet never really warns you as there are surprisingly few resources on the topic. The absolute majority of them argue about where to put the API version but only a few scarce articles attempt to answer: "How do we implement it?". The most common ones are:

putting different versions of the same web application into separate deployments
copying single routes that have changed between versions
copying the entire versioned application for each version

Separate deployments can get really expensive and hard to support, copying single routes does not scale very well to large changes, and copying the entire application creates so much extra code that you will start drowning in it after just a few versions.

Even if you try to pick the cheapest one, the burden of versioning will catch up soon. At first, it will feel simple: add another schema here, another branch in business logic there, and duplicate a few routes at the end. But given enough versions your business logic will quickly become unmanageable, many of your developers will mistake application versions and API versions, and will start versioning the data within your database, and your application will become impossible to maintain.

You might hope that you will never have more than two or three API versions at the same time; that you will be able to delete old versions every few months. It is true if you only support a small number of internal consumers. But clients outside of your organization will not enjoy the experience of being forced to upgrade every few months.

API versioning can quickly become one of the most expensive parts of your infrastructure so it is critical to do diligent research beforehand. If you only support internal consumers, then you might have a simpler time with something like GraphQL but it can quickly get just as expensive as versioning.

If you are a startup, it would be wise to postpone API versioning until the later stages of your development when you have resources to do it right. Until then, deprecations and additive change strategy might suffice. Your API will not always look great but at least you will save a great deal of money by avoiding explicit versioning.

How do we implement API versioning?

After a few trials and many errors we were at the crossroads: our prior versioning approaches that we mentioned above were too expensive to maintain. As a result of our struggles, I devised the following list of requirements that would be required of a perfect versioning framework:

"Maintaining a large number of versions is easy" to make sure that versioning does not slow down our feature development
"Deleting old versions is easy" to make sure that we can clean up our code base without efforts
"Creating new versions is not too easy" to make sure that our developers are still incentivized to try to solve problems without versions.
"Maintaining a changelog between versions is easy" to make sure that both we and our clients can always be sure about the real differences between versions

Sadly, there were little to no alternatives to our existing approaches. This is when a crazy idea came to my mind: what if we try and build something sophisticated, something perfect for the job -- something like Stripe's API versioning?

As a result of countless experiments, we now have Cadwyn: an open-source API versioning framework that not only implements Stripe's approach but significantly builds on top of it. We will be talking about its Fastapi and Pydantic implementation but the core principles are language and framework agnostic.

How Cadwyn works

Version Changes

The problem of all other versioning approaches is that we are duplicating too much. Why would we duplicate the entire route, controller, or even application when only a tiny part of our contract got broken?

With Cadwyn, whenever API maintainers need to create a new version, they apply the breaking changes to their latest schemas, models, and business logic. Then they create a version change -- a class that encapsulates all differences between the new version and a prior version.

For example, let's say that previously our clients could create a user with an address but now we would like to allow them to specify multiple addresses instead of a single one. The version change would look like this:



class ChangeUserAddressToAList(VersionChange):
    description = (
        "Renamed `User.address` to `User.addresses` and "
        "changed its type to an array of strings"
    )
    instructions_to_migrate_to_previous_version = (
        schema(User).field("addresses").didnt_exist,
        schema(User).field("address").existed_as(type=str),
    )

    @convert_request_to_next_version_for(UserCreateRequest)
    def change_address_to_multiple_items(request):
        request.body["addresses"] = [request.body.pop("address")]

    @convert_response_to_previous_version_for(UserResource)
    def change_addresses_to_single_item(response):
        response.body["address"] = response.body.pop("addresses")[0]

instructions_to_migrate_to_previous_version are used by Cadwyn to generate code for older API versions of schemas and the two converter functions are the trick that allows us to maintain as many versions as we would like. The process looks like the following:

Cadwyn converts all user requests from older API versions to the latest API version by using change_address_to_multiple_items converter, and pipes them to our business logic
Business logic, its API responses, and database models are always tailored to the latest API version (of course, it must still support old features even if they were dropped in new versions)
After business logic produces the response, Cadwyn converts it to the older API version that the client requester is currently on by using change_addresses_to_single_item converter.

After our API maintainers have created the version change, they need to add it to our VersionBundle to tell Cadwyn that this VersionChange will be included in some version:



VersionBundle(
    Version(
        date(2023, 4, 27),
        ChangeUserAddressToAList
    ),
    Version(
        date(2023, 4, 12),
        CollapseUserAvatarInfoIntoAnID,
        MakeUserSurnameRequired,
    ),
    Version(date(2023, 3, 15)),
)

That's it: we have added a breaking change but our business logic only handles a single version -- latest. Even after we add dozens of API versions, our business logic will still be free from versioning logic, constant renaming, if's and data converters.

Version Chaining

Version changes are dependent on the public interface of the API and we almost never add breaking changes into existing API versions. This means that once we have released the version – it will not be broken.

Because version changes describe breaking changes within versions and there are no breaking changes within old versions, we can be sure that our version changes are completely immutable – they will never have a reason to change. Immutable entities are much easier to maintain than if they were a part of business logic because it is always evolving. Version changes are also applied one after another -- forming a chain of transformers between versions that can migrate any request to any newer version and any response to any older version.

Side effects

API contracts are much more complex than just schemas and fields. They consist of all endpoints, status codes, errors, error messages, and even business logic behaviors. Cadwyn uses the same DSL we described above to handle endpoints and status codes but errors and business logic behaviors are a different story: they are impossible to describe using a DSL, they need to be embedded into business logic.

This makes such version changes much more expensive to maintain than all others because they affect business logic. We call this property a "side effect" and we try to avoid them at all costs because of their maintenance burden. All version changes that want to modify business logic will need to be marked as having side effects. It will serve as a way to know which version changes are "dangerous":



class RequireCompanyAttachedForPayment(VersionChangeWithSideEffects):
    description = (
        "User must now have a company_id in their account "
        "if they want to make new payments"
    )

It will also allow API maintainers to check that the client request uses an API version that includes this side effect:



if RequireCompanyToBeAttachedForPayment.is_applied:

    validate_company_id_is_attached(user)

No silver bullets

Cadwyn has many benefits: It greatly reduces the burden on our developers and can be integrated into our infrastructure to automatically generate the changelog and improve our API docs.

However, the burden of versioning still exists and even a sophisticated framework is not a silver bullet. We do our best to only use API versioning when absolutely necessary. We also try to make our API correct on the first try by having a special "API Council". All significant API changes are reviewed there by our best developers, testers, and tech writers before any implementation gets moving.

Special thanks to Brandur Leach for his API versioning article at Stripe and for the help he extended to me when I implemented Cadwyn: it would not be possible without his help.

Measuring an engineering impact. Pyramid of needs for product engineers.

Vadim — Thu, 22 Aug 2024 14:10:53 +0000

Last time, we discussed how the role of a software engineer has evolved into the role of a product engineer and what the fundamental differences between them are. Today, I'd like to delve into the work routine and understand the key principles of operating as a product engineer.

Since the emphasis in this new role is on rapid iterations to gather feedback and validate your solution, it means we need specific tools to assess the code you release. When I talk about validation, I'm not referring to technical metrics, but rather how your solution impacts the overall business. Let's call it end-to-end (e2e) validation, encompassing the following points:

Impact on fundamental technical metrics such as database load, server resource consumption, and browser memory usage.
Impact on integration technical metrics, including the number of non-200 responses to users, increased response times for specific API endpoints, and a "laggy" interface.
Influence on local product metrics, such as the number of users who started using a feature after an improvement (adoption level) or the number of users who consistently use it.
Impact on business indicators - measured in the money generated by users utilizing the features you've worked on.

If we take another look at the points mentioned above, we'll find a common thread that unites them all - to measure impact, we need to "stay informed" about what's happening with the system in various dimensions. In the world of product development, there's a specific term for this - observability.

Observability in the context of software development refers to the system or application's ability to be easily observed and analyzed. This involves collecting data about the application's operation, its state, and performance to ensure transparency and the ability to quickly detect and address issues.

The list of previous points can be represented as an onion, where each ring is within the responsibility zone of different teams.

Level One: Engineering Metrics. Owners - Software Engineers.
Level Two: Integration Metrics. Owners - Site Reliability Engineers (SRE).
Level Three: Feature Metrics. Owners - Product Managers.
Level Four: Business Metrics. Owners - Analytics Team.

Each level has its key metrics, and each team is responsible for its level in the onion, providing a comprehensive assessment of the impact on the product and business.

For most modern companies, evaluating based on 1-2 of these points is quite commonplace. The more advanced ones try to assess changes against three. However, only A-players thoroughly assess all four parameters. Why is that? Because carrying out a detailed evaluation demands substantial effort, advanced skills in multiple areas, and smooth cooperation across various team departments. For very young startups, implementing such processes would be prohibitively costly, while the advantages would be minimal.

In the early stages of product development, the basic shortcomings are usually clear, and all the low-hanging fruits are visible. However, as your solution matures and it's time to enhance or discover new growth areas, such a system becomes a primary tool in your arsenal if you don't want to move blindly forward.

Returning to engineering…

All of this, of course, depends on how teams are organized within the company, but let's get back closer to our daily work routine. When working on a task, you're still writing code, there are no changes there. However, with such an approach, you need to think a bit further than just how to write code; you should also be interested in how your code works in production and, most importantly, how your code impacts the product.

I like to envision this as a pyramid, something like Maslow's hierarchy but with an engineering twist. Each level becomes a concern only after we've managed to fill the previous one, and for success at each level, we'll need specific tools that best address the needs.

So, we've written the code, initiated the deployment process, and now it's time to observe

Is my code working? 🛠️

Difficulty: Easy 🟢 
Owner: Developer 👨‍💻
Tools: Bugtracker 🐛
Evaluation time: up to 1 hour ⏲️

The bottommost level. Before evaluating further, you must ensure that your code fundamentally works. The primary metric is a binary answer: yes if your code somehow accomplishes its task, or no if it fails 100% of the time, even for the simplest user story without corner cases.

Typically, simple bug trackers like Sentry or Bugsnag are used as tools. Evaluation time is usually less than an hour. Depending on the release process, validation can occur at various stages—some use QA department testing, others rely on team tests, and some may test independently directly on sandbox/production environments. These days there's nothing new about this, and hopefully, we all do it.

Is my code working well? ✅

Difficulty: Medium 🟡
Owner: Developer, but you'll need initial setup of integrations with 
tools typically falling on DevOps/SRE shoulders ⚙️
Tools: Log management systems, times-series databases 📉
Evaluation time: 10 minutes to 6 hours. ⌚

After ensuring basic functionality, the next step is to assess how reliably our code is working and give it a quantitative evaluation. The primary metric here is the error/success rate.
Tools for this can include logging systems like the ELK stack or Grafana, or more time-series-oriented tools such as Prometheus, Datadog, or AWS CloudWatch if you operate under high load. To assess this parameter, you need to ensure that your code logs information or sends metrics to the relevant system. Configuring dashboards to have a visual representation is also crucial.

What's the ideal metric values to move to the next level? It all depends on the situation! For example, a billing system for processing orders on your site should strive for a success rate approaching 100%, while tasks related to unvalidated image classification or the use of an unreliable 3rd-party API may have a lower reliability level.

Sometimes evaluation can be challenging, especially if you need to introduce a new feature to users and teach them how to use it. In such cases, you can manually test the main user scenario with different inputs and settings 5-10 times.

Additionally, at this stage, automated alerts become your good friends. They send notifications if this metric starts deteriorating over time, allowing you to proactively respond to changes before your users become dissatisfied with the product

How does my code impact the system as a whole? 🌐

Difficulty: Medium 🟡
Owner: DevOps or developer 🧑‍💻
Tools: Log management systems, times-series databases 📊 
Evaluation time: 10 minutes to 6 hours ⏳

There are numerous technical metrics reflecting the health of your service. Depending on the product, their criticality may vary, but your changes can easily lead to spikes in any of them. For example, using a database query in a loop without attention can significantly increase the number of database queries, loading an entire file into memory can spike server memory consumption, or synchronously calling a 3rd-party API that responds slowly can increase the average HTTP response time for your users.

Most modern hosting or cloud providers cover these metrics out of the box. Many services, such as databases or queuing systems, also default to supporting data transmission to monitoring systems like Prometheus or OpenTelemetry. You need to put in some effort to start working with these, but the good news is that you only need to do it once for each tool.

Evaluation time: In the best-case scenario, you can identify the issue immediately. More often, problems arise with an influx of traffic, so detailed logging becomes your best friend in investigation to quickly understand what exactly is leading to deteriorating metrics. Ideally, this can be caught during performance testing, but maintaining load testing for every change, especially in the development of new products, can be a challenging task.

For example, in Monite, we use a set of multiple tools that support developers in collaboration with the DevOps team. For monitoring hardware metrics, such as database load, we use metrics exported to Prometheus. For a more detailed analysis of microservice performance, we have deep integration with Sentry. And for exploring end-to-end performance, we implement distributed tracing.

How does my code impact the success of a business feature? 📈

Difficulty: Hard 🟠
Owner: Product Manager, Analyst 📈
Tools: Google Analytics, Amplitude, Data Warehouse 🔍
Evaluation time: A few days for designing the A/B test, and from
a few weeks to several months for data accumulation and drawing 
conclusions 📅

Once we've ensured that technical metrics are alright, we can move to the next level - assessing the impact of your changes at a local product level. Analytical tools such as Google Analytics or Amplitude can be helpful here.

Let's consider a simple example - the checkout form on your service requires users to input their country in a strict format. Data show that the conversion to the next step is at 40%, and the simplest hypothesis is that users don't understand the expected input format, leading them to close the page after the first unsuccessful attempt. You decide to weaken the validation, and the conversion increases to 50%. However, you feel it's not enough, so you add autocomplete to the form, further increasing the conversion to 80%. This is the product impact of your changes.

It might not sound too difficult, but when trying to implement this in practice, many questions arise. What if I have a multi-platform application? How do I measure the impact of each change on a feature when we release them simultaneously? How can we understand that it's your changes affecting metrics and not external factors? Most of these questions can be answered with one solution - A/B testing. Conducting clean A/B tests requires analytical skills and, more importantly, a significant volume of traffic to draw statistically significant conclusions. The latter is a major hurdle for small companies.

Another reason I categorized this as “hard” is because such evaluation requires designing, bootstrapping, and implementing metrics for each feature of your product. As compensation, proudly use the "data-driven" badge on your resume.

How does my code impact the success of the product? 🚀

Difficulty: Insane 🔴
Owner: Product Manager, Analyst, Finance 🧑‍💼
Tools: Data warehouses, reporting tools, BI tools 💾
Evaluation time: from 1 month 🗓️

If in the previous point you are examining how users behave when interacting with your feature, transitioning here shifts your focus from the feature to the user. Who is the user? What problem does your product solve for them? Where did they come from? What keeps them from leaving for competitors? You literally need to know everything about them. This requires significant investments in a data collection, analysis culture and data infrastructure.

From an analytical standpoint, the complexity lies in establishing the correlation between key business metrics and local product metrics. You need to learn how changes, like increasing the form conversion rate by X%, will impact key business metrics (such as ARR, LTV, Churn rate, and any other metrics vital to your business). From a product standpoint, you need to understand not only your current users but also those contemplating using your product.

Honestly, this goes beyond the direct responsibilities of an engineer, but I believe that if you consider yourself a product engineer, you should at least be curious about how effective the solutions you release are and how they influence the successes and failures of the business as a whole.

How does your company measure the impact of engineering decisions on the product?

The Evolution of the Software Engineer Role into Product Engineer

Vadim — Wed, 28 Feb 2024 13:44:51 +0000

Over the past 10-20 years, software development approaches have undergone significant changes. What used to resemble the Wild West has now evolved into a vast ecosystem built from startups, accelerators, and venture funds. This transformation has inevitably influenced how companies structure their internal processes. Due to intense competition, the primary question for young companies has shifted from "how to build" to "what exactly to build." This has prioritized speed over high quality, emphasizing the velocity of hypothesis validation, market responsiveness, and adaptation to changes.

We observe how lean and scrum methodologies have displaced the waterfall model, focusing on addressing bottlenecks that are crucial in the new realities. However, are there other ways to optimize the process?

Typical team structure

Most modern development teams can be broadly categorized into two types:

Conservative Teams: These teams typically consist of a few engineers led by an engineering team lead. The focus is on a more traditional hierarchy, where the team lead guides the technical aspects and decision-making.

Cross-Functional Teams: These teams are more contemporary and are composed of a cross-functional mix. They include a product manager and a designer who are responsible for customer communication. The team is still led by an engineering team lead, and there are multiple developers working on tasks.

The choice between these models often depends on the nature of the project, the organizational culture, and the specific goals of the development effort.

However, both of these approaches share a common drawback - all feedback from a large number of stakeholders goes through the Product Manager. The feedback is somewhat processed by the Product Manager, then passed on to the team or team lead. Specifications are written, requirements are decomposed, and the task is handed over to the developer... except for the details and business context. The developer works with a task that's already prepared and doesn't directly interact with the client. If they have questions, they ask the Product Owner or team lead, and either the questions are passed on or the team lead attempts to solve them based on their own knowledge. All of this resembles an attempt to play a game of Chinese whispers.

At the end of the sprint, developers submit their task, roll it out to production, and it turns out they did something completely different. The entire process has to be initiated again. What's wrong with this approach? Each link in the team becomes a bottleneck for task progress. Due to limited capacity of team members, answering questions and passing context can take days or even weeks, significantly increasing the feedback loop for your new product features.

Is there a way to optimize this process? Certainly - by eliminating unnecessary links in the existing chain. That's how the role of a software engineer evolved into a product engineer.

Who is this product engineer of yours?

Product engineers possess a powerful combination of skills. They excel in communication, building strong relationships internally with colleagues and externally with customers. These strategic thinkers go beyond just completing tasks; they see their work as a key force driving towards a successful IPO. The most vital aspect is that product engineers deliver. They conceive, construct, and meticulously validate their work, ensuring that new features quickly reach customers.

To illustrate how the feedback process changes with the new approach, you can refer to the following diagram

So let's consider the features of a product engineer's work and how they differ from a software engineer:

Measure of success:

Software Engineer: Success criteria often revolve around the quality of the engineering system they develop.
Product Engineer: Focuses on solving user problems. Success is not always about the most productive code or code covering the most corner cases. It's about writing code that best addresses user problems, closes the most popular use cases, and provides the maximum possible UX for end users.

Scope of work:

Software Engineer: Works based on ready specifications and tasks (for example in JIRA).
Product Engineer: Concentrates on a user problem, usually described in the form of a user story.

Depth specialization:

Software Engineer: Often has a clear specialization in frontend or backend development. For software engineers focused on technical excellence, having a specialization and deep technical expertise is vital.
For a product engineer, it's crucial to balance speed and quality. They are often specialists without a strongly defined specialization in frontend or backend. They can contribute to any part of the product or are full-stack developers, following a T-shape or even W-shape approach.

Customer interaction:

Software Engineer: Main sources of requirements and feedback are product owners or their immediate supervisors.
Product Engineer: Emphasizes communication with users, involving both quantitative feedback through analytical systems and qualitative feedback through participation in customer development interviews to better understand user needs.

Development and delivery speed:

Software Engineer: Can afford to solve technical problems that do not always directly impact end users.
Product Engineer: Typically contributes to product development and drives shorter iterations, focused on obtaining feedback quickly from end users for their solutions.

What are the requirements for the role and how to become successful?

So, you've decided that you like this and want to try yourself in such a role. Should you just go and tell your boss about it? You can certainly try, but it doesn't always work. Transitioning to such a format usually requires changes in the overall company structure and internal processes, and not everyone is ready for such rearrangements. For larger companies with a strict and deep vertical structure, such a transition may prove more challenging than for horizontal companies or startups that are more aligned with them.

If we look at the job market, a similar correlation is evident. For example, in a LinkedIn job search, where larger companies post vacancies, a search for software engineer yields 143,146 job listings, while product engineer results in 1,773, which is only 1.2% of the total number of job listings. However, if we examine platforms where engineers are hired by younger companies, such as www.workatastartup.com, the distribution becomes much more interesting — 665 companies for product engineer compared to 686 for software engineer indicating a high demand for such specialists in young and growing companies.

Since this role involves continuous learning across adjacent tracks, the company culture should be geared towards growth and open communication among employees with different skill sets. If you are already an established technical professional, you will need to enhance your understanding in areas like product management, UX design, and analytics to identify issues in your product and propose user-effective solutions.

The next point may be a bit controversial, but I believe that to transition into the realm of product engineers, you should already be a software engineer with at least a middle+ or senior level. To address user problems, a developer must already have a well-developed technological worldview and experience to understand the pros and cons of specific solutions and make necessary trade-offs. The primary challenge in such a position should be product validation of the chosen solution, rather than writing the corresponding code.

By the way, when we talk about UX, we don't always mean the interface. It is more about how the user interacts with your product. For example, being an API-first company at Monite, 99.9% of user interactions with our product occur through the API. Therefore, we address this at the API level by adhering to industry standards, providing detailed documentation, and creating walk-through examples formatted in Postman that cover most user scenarios.

Does this bring value to the company?

Embracing a no-compromise approach, product engineering focuses on efficiently reducing communication noise. Forming small, agile teams leads to fewer meetings, rapid decision-making, and a streamlined feedback loop. By eliminating unnecessary layers of communication, product quality experiences a significant boost. When engineers have a comprehensive understanding and the authority to make decisions, the process becomes faster and more effective, preventing delays or errors.

When I discuss this topic with employers unfamiliar with such an approach, there's often a certain level of skepticism for two main reasons:

Developers have gained a reputation as technical geeks interested only in manipulating bits, optimizing algorithms, and scaling containers.
The approach where an employee is expected to be involved in more than one direction is often perceived as a lack of focus and a stress-inducing factor related to work.

Finally, I have a source to reference to refute these claims. In the DevOps report by Google released in 2023, there is a separate chapter dedicated to how a focus on users affects various team parameters. Among the findings, I would like to emphasize:

Teams with a strong user focus experience a 40% higher organizational performance.
Prioritizing user needs leads to a 20% higher job satisfaction.
User-focused approaches significantly improve other metrics.

Wrapping-up

Fundamental changes in the industry always impact the requirements for companies, which, in turn, influence approaches to development and the internal organization of teams. So, if, in addition to development, you are interested in:

Growing along a non-technical track
Gaining a better understanding of the domain you work in and the existing problems within it
More closely aligning your efforts with the successes or failures of the company

Then perhaps now is the perfect time to be at the forefront of these changes and consider embracing a new role for yourself.

If you prefer not to participate in this and feel more comfortable in the technical track, it doesn't mean you'll be left behind! Since all your teams are now focused on users and often do not pay full attention to engineering, there is a need for individuals who will address this aspect of your workflow. Therefore, roles such as architect or platform team (which I will revisit in future topics), providing a strong technological foundation for product engineers and preventing them from diverging too much, become even more crucial than before!

In my blog, I still focus on approaches to work, team building in the early stages of company development, the role of a product engineer, and establishing a data-driven culture within a company. If these topics interest you, feel free to subscribe for me and other Monite developers.

API-first Invoicing – A Guide to Simplifying Financial Document Generation

Andrey Korchak — Wed, 29 Nov 2023 11:28:39 +0000

Issuing invoices and other financial docs is something that rarely bothers tech teams. It’s something we all have to do, but there is no much value for customers in having access to financial documents.

Building yet another invoicing feature in your application looks like a simple and extremely boring thing to work on. Most engineers put together a bunch of HTML-based templates, and a PDF rendering library and start spitting out a bunch of boring financial documents.

Sometimes invoicing solutions come together as a part of payment services. Companies like Stripe supply their customers with pretty basic invoice generators integrated right into the payment gateway. That is a much simpler way to go, but it has disadvantages in terms of compliance in disadvantages.

These two solutions are equally boring and wrong at the same time.

First of all, invoices generated by your software system have to be compliant. Different countries and states have completely different understandings of what 'compliant' means. Some countries require valid business tax IDs to be present on every invoice. Some countries don't.
Some countries require to put quite sophisticated VAT rates on invoices. Some countries don't. Some countries have a specific look at financial documents, with strict requirements for the position of elements, IDs, and codes on documents. Some countries don't.

So most likely, if you just insert an HTML template into PDF render, you end up with an incompliant invoice in hand. What does 'incompliant' usually mean? it means 'loss of money'. Incorrect invoices may be not accepted by your/your customer's tax authorities, which leads to extra time and work spent on clarifying and re-doing financial paperwork. Missing compliant VAT information may lead to wrong VAT calculations which leads to extra taxes that must be paid to the government.

The second problem is that often invoice-related business scenarios are more complex than we used to think. There is a whole universe of invoice-related documents that often are involved in the process of receiving money or making payments from your account. These are purchase orders, credit notes, and full or partial cancellation notes. And all of these docs often have to be exported into 3rd party accounting software.

Both things, compliance, and different business cases, are boring, costly, and annoying to deal with. Especially if finances are not your main product and engineering expertise. That often happens in marketplaces, CRMs, and small-medium business management tools. In these software products, the main goal of the product team is to deliver features and customers and not to deal with annoying financial documents. But if you are not making invoices and other documents right, that will create frustration for your customers who have to deal with incompliant documents with missing VAT IDs (or God knows what else tax authorities wish to see on these PDF files).

Getting started

In order to avoid the situation of re-writing the same financial features using different programming languages, we designed an API that provides all necessary functionality and designed to provide solid foundations for your apps.

In short, Monite is like AWS for fintech, but instead of low-level services like servers and databases, Monite supplies users with high-level building blocks such as invoicing, payments, accounting, and many others. All building blocks can be connected to each other in all possible ways and they also can talk to each other. This leads us to API-level infrastructure which is able to provide a solid foundation for all your fintech ideas.

The Monite API platform uses a few different layers to secure access to any stored data within the platform:

Partner. A company that implements Monite in its app or platform. The development teams of Partners connect to the Monite API with admin-level access tokens. The admin tokens in this mandatory layer enable Partners to create and configure entities and access all resources of all entities they develop software for.
Entity. A customer of a Partner – an entity – is either an organization or an individual. Each Partner develops for one or more entities. With the ID of an entity, it is possible to obtain root access to all resources related to this specific entity only.
Entity user. The employees who work for an entity. This optional entity user access layer is for Partners who want to use Monite security for rapid development rather than build their own custom access control logic. Using the Monite API, Partners create customizable entity-level roles and permissions. Monite automatically monitors access policies for each API call.

For example, Maria is an accountant at Beispiel GmbH and Beispiel GmbH uses the online banking of Big Money Bank. Big Money Bank is a partner, Beispiel GmbH is an entity and Maria is an entity user.

To start building your invoicing solution, the first thing you as an API partner need to do is to register your partner account on the Monite Partner Portal.

Then you have to create your project and get credentials for it.

The API ID and secret need to be exchanged for an access token which is then used to authenticate the API calls.

curl -X POST 'https://api.sandbox.monite.com/v1/auth/token' \
     -H 'Content-Type: application/json' \
     -d '{
        "grant_type": "client_credentials",
        "client_id": "28c10852-7e78-43cf-abfb-efeed1834963",
        "client_secret": "615b3cfa-646b-41d9-b768-521f09315ac5"
     }'

Mapping your users to Monite

Usually, Monite is used when a tech team wants to add a fintech feature to an existing software product. It means users, roles, and permissions are already implemented inside of that product. But in order to make invoicing work, we somehow need to connect that existing access management function of the parental product with Monite API.

And in order to do that, we need to map existing users of our customers into Monite API and Monite will take care of all aspects of access control of invoicing feature.

First, we create an entity representing your customer. Let’s assume your software provides business services to Bob Jones who lives in Berlin. Via following API call we connect Bob Jones with Monite.

curl -X POST 'https://api.sandbox.monite.com/v1/entities' \
  -H 'Authorization: Bearer YOUR_PARTNER_TOKEN' \
  -d '{
    "type" : "individual",
    "email": "bob@example.com",
    "address" : {
        "country" : "DE",
        "city" : "Berlin",
        "state": "BE",
        "postal_code" : "10115",
        "line1" : "Flughafenstrasse 52"
    },
    "individual" : {
        "first_name" : "Bob",
        "last_name" : "Jones",
        "tax_id": "1234567890"
    }
  }'

Often multiple people operate business and they all have different permissions when we are talking about access to financial information. In order to separate different groups of people, you can create different roles with different permissions. For example, sales folks can only issue invoices, while CEO can do pretty much everything.

curl -X POST 'https://api.sandbox.monite.com/v1/roles' \
     -H 'X-Monite-Version: 2023-04-12' \
     -H 'X-Monite-Entity-Id: ENTITY_ID' \
     -H 'Authorization: Bearer ACCESS_TOKEN' \
     -H 'Content-Type: application/json' \
     -d '{
      "name": "View payables",
      "permissions": {
        "objects": [
          {
            "object_type": "comment",
            "actions": [
              {
                "action_name": "read",
                "permission": "allowed"
              }
            ]
          },
          {
            "object_type": "payable",
            "actions": [
              {
                "action_name": "read",
                "permission": "allowed"
              }
            ]
          }
        ] 
      }
    }'

And then we put all things together and create a record for a person who works for Bob Jones:

curl -X POST 'https://api.sandbox.monite.com/v1/entity_users' \
     -H 'X-Monite-Version: 2023-04-12' \
     -H 'X-Monite-Entity-Id: ENTITY_ID' \
     -H 'Authorization: Bearer YOUR_PARTNER_TOKEN' \
     -H 'Content-Type: application/json' \
     -d '{
       "login": "Gardner.Waelchi",
       "first_name": "Gardner",
       "last_name": "Waelchi",
       "role_id": "946141f3-ca01-44dc-b1a6-1024aa71f978",
       "email": "g.waelchi@example.com",
       "phone": "+15551234567"
     }'

To make API calls on behalf of an entity user, you need to use an access token of that user. To get this token, call POST /auth/token with the following request body:

curl -X POST 'https://api.sandbox.monite.com/v1/auth/token' \
     -H 'X-Monite-Version: 2023-04-12' \
     -H 'Content-Type: application/json' \
     -d '{
       "grant_type": "entity_user",
       "client_id": "2e0c68d6-00b7-447d-b26c-415bbcbfc026",
       "client_secret": "cf0de0bd-a59e-473f-a3dd-db5924bd8622",
       "entity_user_id": "0c76febf-aabb-451a-aabb-ea3b47689dc1"
     }'

Invoices and account receivables

All different invoice-related documents (such as cancellation notes and purchase orders) are part of a bigger concept which is called 'account receivables'. Monite API covers lots of different business cases related to AR and everything is available via API calls.

In order to issue your first compliant invoice via API, you have to do some preparatory work.

First, you have to dig into VAT rates. Monite manages VATs for you, but anyway you have to get connected to VAT rates API.
Then you have to map your user’s counterparts to our system. In order to become exportable, every financial document has to refer to a specific counterpart which also will be exported into accounting solutions along with financial data.
An invoice always contains information about line items such as goods and services, Every line item has to connect the valid title of service/good, price, VAT rate, and measurement units. Monite stores this information inside of API in order to make it reusable.
Every compliant invoice also has to contain payment terms information. Payment terms define the amount of time one has to pay an invoice (for example, 30 days from the invoice issue date). The terms can optionally include discounts for early payments to motivate the customers to pay sooner than the due date. Without payment terms, the invoice may be considered as incompliant!

And after all their preparatory steps you can go forward and start issuing invoices.

VAT classes

Monite maintains a database of VAT rates. API users can query the available rates via the API.

Currently, Monite API provides VAT rates for the following countries:

🇦🇴 Angola
🇧🇪 Belgium
🇧🇼 Botswana
🇪🇪 Estonia
🇸🇿 Eswatini
🇫🇷 France
🇩🇪 Germany
🇮🇪 Ireland
🇱🇸 Lesotho
🇱🇷 Liberia
🇲🇿 Mozambique
🇳🇦 Namibia
🇳🇱 Netherlands
🇿🇦 South Africa
🇪🇸 Spain
🇦🇪 United Arab Emirates (UAE)
🇬🇧 United Kingdom (UK)
🇿🇼 Zimbabwe

We are constantly adding new countries, and please let us know if you don’t see VAT rates for some countries you want to support right now. Maintaining a valid database of actual VAT rates is a significant effort and that’s why we are doing this for you.

You can get all available VAT rates using API call:

curl -X GET 'https://api.sandbox.monite.com/v1/vat_rates?counterpart_id=3a9c5...48df' \
     -H 'X-Monite-Version: 2023-04-12' \
     -H 'X-Monite-Entity-Id: ENTITY_ID' \
     -H 'Authorization: Bearer ACCESS_TOKEN'

Create and manage products

In order to make invoices right, your users need to specify products/services and measurement units for them.

curl -X POST 'https://api.sandbox.monite.com/v1/measure_units' \
     -H 'X-Monite-Version: 2023-04-12' \
     -H 'X-Monite-Entity-Id: ENTITY_ID' \
     -H 'Authorization: Bearer ACCESS_TOKEN' \
     -H 'Content-Type: application/json' \
     -d '{
       "name": "kg",
       "description": "Kilogram"
     }'

curl -X POST 'https://api.sandbox.monite.com/v1/products' \
     -H 'X-Monite-Version: 2023-04-12' \
     -H 'X-Monite-Entity-Id: ENTITY_ID' \
     -H 'Authorization: Bearer YOUR_PARTNER_TOKEN' \
     -H 'Content-Type: application/json' \
     -d '{
       "name": "Ice cream",
       "type": "product",
       "description": "A delicious vanilla ice cream",
       "price": {
         "currency": "EUR",
         "value": 1500
       },
       "measure_unit_id": "12188fc1-493d-48a7-aea8-382240dd7ce7",
       "smallest_amount": 1
     }'

Counterparts

Counterparts represent the suppliers and clients of an entity. They can be organizations or individuals. A counterpart must be created before you can create invoices, quotes, and other documents. That’s because in order to become exportable into accounting software, every financial document needs to be linked with a specific counterpart in the system.

curl -X POST 'https://api.sandbox.monite.com/v1/counterparts' \
     -H 'X-Monite-Version: 2023-04-12' \
     -H 'X-Monite-Entity-Id: ENTITY_ID' \
     -H 'Authorization: Bearer ACCESS_TOKEN' \
     -H 'Content-Type: application/json' \
     -d '{
      "type": "organization",
      "organization": {
        "legal_name": "Acme Inc.",
        "is_vendor": false,
        "is_customer": true,
        "phone": "+4930774619876",
        "email": "acme@example.com",
        "registered_address": {
          "country": "DE",
          "city": "Berlin",
          "postal_code": "10119",
          "state": "BE",
          "line1": "Flughafenstrasse 52",
          "line2": "Additional address"
        }
      }
    }'

Payment terms

Payment terms define the amount of time one has to pay an invoice (for example, 30 days from the invoice issue date). The terms can optionally include discounts for early payments to motivate the customers to pay sooner than the due date.

curl -X POST 'https://api.sandbox.monite.com/v1/payment_terms' \
     -H 'X-Monite-Version: 2023-04-12' \
     -H 'X-Monite-Entity-Id: ENTITY_ID' \
     -H 'Authorization: Bearer ACCESS_TOKEN' \
     -H 'Content-Type: application/json' \
     -d '{
       "name": "Net 30",
       "term_final": {
         "number_of_days": 30
       }
     }'

Issuing an actual invoice

And after all this preparatory work is done, you can finally issue an invoice on behalf of your user:

curl -X POST 'https://api.sandbox.monite.com/v1/receivables' \
     -H 'X-Monite-Version: 2023-04-12' \
     -H 'X-Monite-Entity-Id: ENTITY_ID' \
     -H 'Authorization: Bearer ACCESS_TOKEN' \
     -H 'Content-Type: application/json' \
     -d '{
       "type": "invoice",
       "currency": "EUR",
       "counterpart_id": "0414f84c-b039-4203-b09b-e42b49245435",
       "line_items": [
         {
           "quantity": 1,
           "product_id": "8755c86a-d630-4920-b6fd-fd2917d87dfb",
           "vat_rate_id": "479155c3-0995-4689-a3cf-7482ea5132a9"
         }
       ],
       "payment_terms_id": "e2cbbb5f-15e6-4b22-a942-8f51b7a81118",
       "vat_exempt": false,
       "entity_vat_id_id": "cb6c1c38-fdae-48f8-8e51-2d50d116b882",
       "entity_bank_account_id": "3f548c1b-8f18-4021-bdd1-5624cca65c3e"
     }'

What next?

Now you have a PDF document with all relevant financial data. But it’s just the beginning of the story. Invoices need to be delivered to recipients, then payments have to be accepted and matched against financial records. In some situations, people have to deal with recurring invoices and partially paid invoices, and all these cases need to be covered in invoicing products, otherwise, your invoicing solution will be half (or even quarter) backed. We covered these topics in our tech documentation.

The entire flow of API calls may seem a bit tricky for you. But there are ways to hide this complexity from your users. For example, VAT rates may be settled only once via a separate API call done in the background. Services and measurements can be also settled up only once if you are working, for example, on Upwork-like freelance platform, where people often produce invoices based on hours they spent working on projects.

How API works under the hood

All invoicing business flows are pretty much the same in different countries. Yes, there are differences in VAT rates, rules, and numbers, but we are talking, pretty much, about the same set of documents with the same set of properties.

We identified regional aspects of invoicing and made them configurable via configuration options coming from our internal microservices and JSON configuration files. Compliance officers and keeping these configs up to date so end users always receive complaint invoices.

Engineering Challenges in B2B and B2C Startups

Vadim — Wed, 22 Nov 2023 09:07:53 +0000

The internet is filled with articles comparing work cultures across different company types, especially in marketing or product management. But what about software developers? Do these differences impact them too? Absolutely! Understanding the nuances between B2B (Business-to-Business) and B2C (Business-to-Consumer) can be a game-changer. Each sector has its own unique set of characteristics, demands, and obstacles that significantly shape the work of software engineers.

Before diving in, let's set the stage. I'll use "B2C" to describe products that engage directly with end-users. This includes not only consumer-focused applications but also B2B products with a low average ARPU, where the primary revenue stream comes from the self-serve customers segment. These are platforms like Miro and Zapier, catering to a broad range of users who can onboard independently.

On the other hand, our B2B segment refers to enterprise-focused products. Such companies typically work with a smaller number of clients but with more substantial contracts, often in the tens of thousands, or hundreds of thousands of dollars.

Throughout my career, I primarily worked in B2C companies, occasionally advising teams involved in B2B. However, after joining Monite, I experienced the stark difference firsthand.

Alright, let’s go step by step through the main differences, analyze where they come from, and explore the opportunities they open up for your professional growth.

In-depth specialization vs. broad accessibility

In the B2B corner, you’re the tech wizard – called upon to solve specific challenges for major players. Corporate clients rely on you to provide robust solutions, sparing them the complexities of handling those issues themselves.

Sounds exciting, right? But here’s the thing: you always need to be at the top of your game. Your business must meet the high standards set by your clients. If you’re dealing with financial data and payments, get ready to navigate intricate regulations across the countries they operate in. Managing credit card data? Compliance with PCI-DSS is non-negotiable. And when working with medical data, you’ll become well-acquainted with HIPAA. And that’s just the beginning…

In the B2C space, your customers turn to your product for convenience or entertainment. They’re not looking for complex problem-solving; they want a user-friendly solution. While the B2C entry barrier is low, making it a popular choice, it's much like a bustling local cafe – lively but packed. While the vibe seems welcoming, brace yourself for fierce competition. Many are jostling for the spotlight in this vibrant arena.

Stability and fault tolerance

When a big client invests in your product with the expectation of saving their development team’s time and offloading certain tasks, any weaknesses on your part within their IT ecosystem can lead to a swift termination of your partnership. This applies not only to technical glitches, errors, and system failures but also to products that don't effectively meet their designated objectives.

Each failure also impacts your client's reputation with their end-users, as it directly compromises the quality of service they provide. This, in turn, can negatively influence your standing within the industry.

It’s also crucial to consider acquisition costs. Acquiring clients in the B2B sector is a resource-intensive and time-consuming task, demanding a personalized approach. So, when you lose a client, it's not just a hit to your wallet; it's hours of hard work down the drain and a chunk of your revenue gone.

On the flip side, the B2C world is a bit more forgiving of minor glitches or even partial downtime. Here, clients usually use your product for straightforward tasks. Even if there's a small issue or a short downtime, it's not the end of the world. Why? Because the average sale is probably between $100–200, and you're catering to a much larger crowd. This time partial failure or temporary degradation of one of its functions has a less severe impact on your entire user base.

But here's the thing: to really grow in the B2C space, you need a massive user base. And while there's a steady flow of users coming in and out, this model can handle a bit of client turnover.

All of this means one thing: B2B products can't afford mistakes. That's why there's so much emphasis on making them fault-tolerant and thoroughly tested, way more than B2C products. As a result, variations emerge in the development processes. When working on a B2B product, you encounter extended release cycles, stricter Quality Assurance procedures, and daily encounters with terms like release management and zero bug policy.

How working with customer feedback differs

In the B2B world, you're often working with a client base that's in the tens or hundreds. This smaller number means you can get to know each client personally. Your products in this space are typically designed to weave seamlessly into a client's existing infrastructure, which means the integration process can get pretty detailed. It's not uncommon to find yourself giving technical advice to engineers from your top-tier clients, and sometimes, you might even grab a drink together.

Within a B2B-focused company, you'll probably have several support tiers, each handling different levels of complexity. There might also be a specialized integration team. Their job? To offer deep technical support, help with integrating products, and sometimes even tweak the product to fit a particular client's needs.

Switch over to the B2C side, and things look a bit different. With potentially thousands or millions of users, it's tough to have a personal connection with each one. Most of the feedback you'll get will be in the form of data: numbers, charts, and graphs in your analytics dashboard. Sure, every now and then, you might chat with a super-user or an influencer. But mostly, conversations about your user base shift from talking about individual users to discussing broader groups or segments.

Because of this, B2C companies have a big appetite for product analytics. They need robust data engineering to gather, process, and make sense of heaps of data. This isn't just a numbers game; it's about making informed decisions based on a deep understanding of customer profiles and being able to measure how effective those decisions are.

Why rapid adaptation is important

In the enterprise B2B space, it's common to see long-term contracts. The cost of integrating these solutions? Sky-high, especially when you compare it to B2C, where businesses often lean on subscription models or even offer their services for free. Given this landscape, it's crucial to keep a close watch on what your competitors are up to and stay in tune with market shifts. This is especially true if you're in a young industry where competition is fierce, and you don't want to miss out on attracting new clients.

When it comes to product development in such an environment, the emphasis is on quick, short cycles aimed at fast delivery. You're probably not spending a ton of time drafting detailed tech documentation or plotting out product development for more than a few months in advance.

To keep up this speed and swiftly test out different business ideas, many turn to agile approaches like Scrum. It's all about iterative feature development, and when exploring new product avenues, the MVP (Minimum Viable Product) strategy is often the go-to.

Step growth vs. gradual growth

In the B2B space, the journey from customer interest to full integration can be a long one. This extended timeline often gives you a buffer to gear up for any unique demands a new client might bring. You can rally more developer manpower and fine-tune your infrastructure to cater to each new client's needs.

On the flip side, when you're in the B2C game, a big focus is on how quickly a user can jump in and start using your product. With such a low barrier to entry, you might see spikes in user activity out of the blue. Maybe you rolled out a catchy ad that's gone viral, got a shoutout as the "product of the day" on Product Hunt, or perhaps it's just that time of the year when everyone's looking for what you offer. Suddenly, your user count could skyrocket. This unpredictability means you've got to build your system to be super flexible. It should be ready to scale up in a snap and adapt on the fly to whatever comes its way.

Security

Beyond just meeting regulatory standards, it's crucial to sync up with your customers on security protocols. Today, a data breach can spell disaster for businesses big and small. However, big enterprise clients tend to put cybersecurity under the microscope before they even think about signing a contract or integrating your solution. They might ask you to secure certifications like ISO 27001, set up robust internal security guidelines, limit who can access production data, or even put your system through rigorous penetration tests.

On the other hand, small to medium-sized businesses (SMBs) aren't usually as stringent about these security deep dives. Addressing basic concerns, like preventing SQL injections, often ticks the box for them. And if you're a budding company, you might find that tasks like updating software to patch vulnerabilities get pushed down the priority list, making way for more immediate business-driven goals.

Brand visibility and recognition

Even though you're diving deep into complex challenges and working under tough conditions, your hard work might fly under the radar for those outside your field.

Your product often powers the big players from the shadows, without the general public getting a firsthand look. Contrast this with developing B2C tools or solutions for widespread SMB use. In those scenarios, explaining your latest project to friends and family is a breeze.

Plus, there's the added perk of getting personal recognition for that cool new feature you helped bring to life.

Overall thoughts (or a conclusion of sorts)

Launching a successful B2C company is no walk in the park. It demands a harmonious effort across every facet of the product.
Here's a breakdown:

User acquisition: Draw in the "right" users without breaking the bank.
Strategic planning: Choose the best paths forward for growth.
Backlog prioritization: Ensure you're always showing impressive month-over-month growth.
User support: Keep existing users happy and engaged.
Engineering: Never lose sight of the technical side of your product.

Given the fierce competition in the B2C space, this all comes with a hefty dose of uncertainty. This unpredictability can make venture capitalists wary of investing. If your leadership team isn't packed with industry bigwigs, securing substantial early-stage investment can be an uphill battle. With funds spread thin, it's tough to give every aspect of product development the attention it deserves.

At this juncture, zeroing in on a product-market fit is paramount. B2C startups are more prone to shifts in direction compared to their B2B counterparts, which usually have a clearer picture of their target audience and the problems they're addressing.

Budget constraints and the potential for sudden shifts in direction mean it's often unwise to pour too much into engineering. You'll likely be working with a tight budget, focusing only on the essentials to roll out the next product version swiftly for another round of testing. This environment is ripe for tales of quick adaptation, problem-solving on the fly, and product building.

However, B2B startups present a different picture:

They often have a clearer idea of client pain points.
Initial customer acquisition can be smoother, thanks to founders' connections or direct sales efforts.
The uncertainty is generally less, but the depth of knowledge required about the domain and product is greater.

So, if you're aiming to be a specialist in a niche area and prefer a more stable work environment (though priorities might still shift), B2B startups could be your calling.

Whether you're venturing into B2C or B2B, both paths come with their unique challenges and rewards. It's all about finding the right fit for your skills, passion, and the kind of challenges you're eager to tackle.

In my blog, I mostly focus on approaches to work, team building in the early stages of company development, the role of a product engineer, and establishing a data-driven culture within a company. If these topics interest you, feel free to subscribe for me and other Monite developers.