DEV Community: Vladimir Bobrov

Where did our monolith go?

Vladimir Bobrov — Thu, 01 Apr 2021 10:24:07 +0000

There is a lot to be said on the pros and cons of a monolith vs microservice software design (just Google a few). To briefly sum things up; while monolith applications are often still the best choice, organisations are increasingly leaning toward microservices architecture. Many companies recognise the importance of increasing overall engineering efficiency through team/unit autonomy and access to a wider labour market. And most are facing similar challenges in their journey to microservices.

Starting a transformation without a plan imposes significant risks. Many challenges must be tackled beforehand which play a foundational role in the success of adopting a monolith system. These include data input, output, processing, error handling, and the UI, which are usually written once and reused across the whole monolith.

Now, imagine multiplying these entities by the amount of stacks or a number of services we envision to expand into. Missing a common layer and principles across services will harm engineering efficiency by a lot. After all, if engineers are to produce tangible value. we must address common problems before we can put a foot in new water on a scale of 50+ people.

Get ready

At Catawiki, we’ve built a foundation with an app skeleton repo, automation and guidelines to enable teams to consistently deliver new services. Besides that, we've built libraries with the most essential tools helping us to streamline integration. This simple 2-step plan is for many adopters often just the tip of the iceberg. And the hidden part is bigger, further away we are from Cloud Native toolchain.

Organisational-wide standards are vital for the success of the microservices journey. It is best to start early with an internal SDK, OpenAPI specification and infrastructure as code. Nowadays, these problems are solved with a myriad of services, tools and well-established standards. Some monolith code will likely come in handy or become a blueprint for a common library that each app may depend on.

Focus on Data

Most applications are dealing with data and underlying storage. While an application layer is usually scaled by simply multiplying compute workload, a data layer eventually becomes a bottleneck.

Inefficient data structures and data-querying algorithms are the biggest problems we'll have to deal with when scaling an application. Frederick Brooks, the author of The Mythical Man-Month, says:

The programmer's primary weapon in the never-ending battle against slow systems is to change the intramodular structure. Our first response should be to reorganize the modules' data structures.

Microservices can bring in significant costs if the data design is not addressed properly. Replacing in-process code execution with cross-host API calls via extra virtual and physical processes costs user time and is more expensive for businesses to operate. In fact, the efficiency of any service depends on how data is stored and exchanged among services.

As the data architecture becomes so important, we'll have to be smart about how we denormalise and exchange it. Categorise data and its usage, establish data exchange rules and apply the right tools for specific use cases. It is always difficult to change data structures at a later stage, so best to ensure storage scales x10-x100 times and return the right value for the years ahead.

Realising the effort

When everything is moving fast in a growing business, it’s not easy to change overnight. To create a system developed from scratch by five people over five years, a quick estimate suggests the time needed amounts to at least 50.000 hours of engineering work. Of course, rebuilding a monolith into microservices might be less expensive than the original investment, as we may borrow from the old system some code and established business logic.

However, for an established business, a rapid rebuild from scratch is usually not an option as it requires us to stop feature development in the old system to avoid ending up a never-ending cycle of catching up with updates. This means that we have to build a new system and features working alongside the old one, exchanging and duplicating data and features, in order to provide clients with a gradual transition away from the original system..

It's true that an effort like this can cost any company a huge amount of time and resources. Nevertheless, organisational and innovation benefits of modern Service Oriented Architecture (and Microservices specifically) will easily outweigh any benefits of a Monolith system in the long run. To make a transition successful and get buy-in from the company at large, it is important to plan things ,with a clear rationale as to why you’re doing this and an understanding of the goals.

Be organised

An organisation with hundreds of people is a complex structure. After years of slow progress in our journey to Microservices, we've realised a lot of things. Perhaps the most important one is that this challenge is purely in the hands of engineers and as such must be driven by engineers.

Photo by airfocus on Unsplash

To better organise ourselves, we've built a company-wide Engineering Roadmap reflecting all steps of migration, accounting for people we have and/or will be hired this year. And with a clear vision of what this is going to bring us, we've committed to making a huge leap this year toward a better, more efficient and scalable foundation of our unique platform for passionate enthusiasts all around the world.

A few months later down the road, we're extremely satisfied with the results and the progress we've made so far. Although this year has just started, we are already planning for exciting engineering challenges in 2022 and beyond!

Cover photo by Boba Jaglicic on Unsplash.

Changing the rails

Vladimir Bobrov — Mon, 30 Nov 2020 17:23:32 +0000

When an online business takes off, there is always an imminent desire to be at the forefront of technology. An increasing belief that the business is on the right track fuels a fear of falling behind competitors — a feeling that can help push a company to success as much as it can hold it back. In the very beginning of our journey, Catawiki made a few technological choices it owes a large part of its success to.

Back in 2008, it was hard to find a more prominent web-development framework than Ruby on Rails. Even though Ruby as a programming language was never the most popular one, the shift of paradigm in web-development introduced by the Rails core team predetermined the success of the framework. As for many emerging Web 2.0 businesses, it was the obvious choice that helped us to focus on things that matter — continuous innovation.

Challenges faced

Six years after Catawiki was founded, the moment to make a bigger technological leap to support expansive growth arrived, facilitated by the company’s global recognition¹. By the end of 2014, we at Catawiki faced the inevitable question: “how can we best support our business growth with our current technology while keeping the promises we’ve made to our customers?”.

What is a simple question at first glance became more complicated when we look at what challenges we had already been facing and what that meant for scaling our platform 10 100 times. Some of the challenges included:

Regular outages and “Friday’s on fire” caused by the increase of visitors and bidders
On-premises infrastructure with a high degree of manual system management
Lack of development and staging automation constraining engineering team growth
Monolith service combining practically two different businesses — auctioneering and collectors platforms

Tackling the issues

Even though these challenges looked frightening, we had a great team of engineers who recognised the issues and acted on opportunities to improve every day. The most significant challenge that popped up on the radar was not just to double or triple the engineering team to cope with increased amounts of issues and support business objectives, but to rethink the way we build things for the future.

One of the first things we tried to do was to move away from the monolith service by building new features (or rebuilding old ones) in a standalone application. This approach, which at first seemed promising, proved to be unsuccessful in the end, yet it brought us lots of learnings.

Monolith systems can be successful and in the early stages of business development this is the only way forward. Many businesses wouldn’t reach their success if the only thing they cared about was pure engineering excellence. In the end what makes great products successful is actually trying, failing a few times before finally succeeding in what you believe could work for many people. This in turn creates a large amount of technical debt a successful company should solve sooner or later when the business takes off.

By the beginning of 2015, we were a few months away from doubling our engineering team to about 25 engineers and we realised we needed to make our own shift in how we were going to shape our technological future.

Stay tuned to find out more about Catawiki's journey and what led us to success!

https://www2.deloitte.com/gr/en/pages/technology-media-and-telecommunications/articles/fastest-growing-tech-companies-emea.html ↩

So how did we begin?

Vladimir Bobrov — Thu, 05 Nov 2020 11:34:17 +0000

Perhaps it would be good to take a tour far back to 2008 and give our audience a picture of where, we as a business, came from.

Originally Catawiki was built by two Dutch gentlemen (René and Marco) with the idea to help collectors around the world organise and exchange their items. As it usually happens, one of the founders (René) thought the idea could actually work as he was himself a passionate Tintin comic strips collector. And the other founder (Marco) knew how to build it.

Catawiki went live in 2008 and was designed as a catalog system with detailed information about items as if it were a wiki. Hence where the Catawiki name comes from!

Thanks to WaybackMachine we can even show you how it looked like back then

via: web.archive.org/web/20080914004650/http://www.catawiki.nl

Just a few years after, in the mid-November of 2011, Catawiki hosted its very first auction. Since then the Catawiki business as we know it today started taking off.

You can even find how the advertisement campaign of this exciting moment looked like (on the right side of this screenshot)

via: web.archive.org/web/20111215035153/http://www.catawiki.nl

And if you are curious about what was the very first lot that was auctioned and successfully sold on Catawiki, here you go

“Storm 1 - De diepe wereld, Hardcover, 1e Druk 1978” was our very first lot sold in Comics Auction (Dutch). Starting with the initial bid of just €5 the price rose x17 times and it went for an impressive €85!

Storm is a soft science fiction/fantasy comic book series, just one of many others you can find on Catawiki platform among many-many other objects. Stamps, coins, paintings, jewelry, cars are just some of the dozens of object categories. Whether you are a passionate enthusiast or a collector seeking for a missing piece, you have a pretty high chance to find what you’re looking for!

Wow, what a “selling” detour we’ve just made! 😊

Let's think for a moment about the tools we employed back then to bring Catawiki online:

Rails - the most modern and powerful web development framework the world has ever seen by that time!
HTML - Internet as we know it wouldn’t exist without.
CSS - without it Catawiki website would look like from ‘90s.
jQuery - breaking through DHTML technology, still used by 73% of the 10 million most popular websites by mid-’19 (according to Wikipedia).
Solr - enterprise-search platform using Lucene search engine.
MySQL - “the World’s Most Popular Open Source Database” which was hard to argue about.
Linux - to run all of the above on the Internet.

Throughout the years we’ve changed quite a few things: half of the stack was replaced with more modern tools and another half has changed so much that you can hardly find a similarity. And of course, we have added a few dozens of newer tools and technologies! 😅

What stayed the same was continuous change. That thing which is not always easy and many times compared with “changing the tires of a car which is running at full speed, staying on top of it”.

And this is exactly what we will be talking about in our next blog posts!

Hello World!

Vladimir Bobrov — Thu, 15 Oct 2020 07:49:35 +0000

Perhaps any Engineer remembers the excitement of seeing this very first message when their first program or webpage comes to life! And so here we are at Catawiki, excited about the launch of our Engineering Blog

The idea behind having a Catawiki Engineering blog didn’t come overnight and was rather hanging in the air for a few years. We’ve put a good amount of diligence to embark on this journey, being completely sure we will provide useful content for our visitors and engage with our audience in a meaningful way.

At Catawiki, we’re building our amazing product with the help of incredible engineering concepts and tools like Cloud, Kubernetes, Microservices and BigData. It is not a secret that those can be used and built in many different ways, and despite our struggle to find the best solution for a specific purpose, we cannot claim that our way is the only true way to build things right.

Nevertheless, we believe that sharing the Catawiki Engineering stories will help many of our peers. Especially those who may be looking for examples of the same or similar problems they are faced with as part of an emerging tech organisation.

We are aiming to share our stories of successes and failures. Although you may find more of the former ones and less of the later ones, we will make sure to share the essential learnings we went through on the way to success. We believe we can achieve greater success if we help our readers avoid falling into traps, as those might be easy to overlook in the landscape of the current technology environment.

We also think it is important to share our stories in a structured form. We will post stories on different topics: Frontend, Backend, Mobile, Systems, Data, Security, Reliability, etc. Any post will be tagged with a corresponding topic name. And that’s not all!

Many companies during their growth from a small startup to a unicorn follow very similar phases. Especially during the migration to Microservices! Thus we would like to present our readers with some content which clearly pertains to different phases of our development. To make the posts stand out, our authors will tag them by corresponding phase names.

Phase	Years	What is this phase about?
Catawiki 0.0	2011-2014	Ancient times, before the expansive growth of Catawiki.
Catawiki 1.0	2015-2017	Our journey to Service Oriented Architecture (Microservices) begins, we can even call it the Great Expansion. Phase when we’ve focused on building facades around our Monolith system.
Catawiki 2.0	2018-2020	Phase when our Microservices Universe cooled down and we started departing from Monolith. We also call it Independence from Monolith.
Catawiki 3.0	2021+	Time when we’ll finally say “Farewell Monolith...” and open the door to the new world!

We hope you will find our posts as useful and interesting as we’ve found it challenging to wade through the building of modern Engineering at Catawiki!