DEV Community: MIKAMAI

A Serverless Architecture to handle image generation from bulk data

valeriadibattista — Mon, 11 Jul 2022 07:30:44 +0000

A real case study on how AWS serverless ecosystem can give your product a lot of value with minimum effort

Publications about serverless adoption often miss a practical approach that can make it affordable in real-life use case scenarios. In Neosperience Cloud Services (formerly Mikamai) we challenge this pure theoretical adoption of serverless technologies, testing our skills in large scale projects that can leverage the full potential of serverless. One of the most interesting use cases is our BeatIdentity client.

Their platform contains thousands of instrumental music tracks, offered on an ecommerce platform in different takes. A user can select them one by one, or as part of a playlist.
The platform has to create a cover for each track and make clear when it belongs to a single or a playlist. Sometimes, due to marketing reasons, the client may need a cover image with the BeatIdentity logo as a watermark.
Since doing this task manually is extremely time consuming and stressful for content creators, we supported the company in building the BeatId Generator.

Thanks to the work of the Neosperience design team, along with the customer, we created a tool, using Processing, to procedurally generate the cover image from track details such as title, artist, instruments, whether it is part of a playlist or needs the logo.

A sample of the resulting images is as follows, either with a standalone song (the pink images) or a playlist (the purple images):

This core logic needs to be packaged within a service capable of storing the images in a way they could be easily retrieved in the future by third party applications (e.g. the ecommerce platform) or by BeatIdentity employees. Infrequent and unpredictable access patterns as well as maintenance considerations suggest this could be built as a cloud native application, leveraging all the benefits coming from serverless.

Project Overview

The application should support the following use cases:

A general user can get an already generated cover in high resolution or in a custom defined size;
A third party application can upload a single track, letting the system generate the high resolution versions of the cover;
A BeatIdentity Admin can import a single track or massively import multiple tracks uploading a CSV file. At the end of the CSV import process the user can download a zip archive containing all the generated files.

The AWS Cloud Development Kit (CDK) lets define the AWS cloud infrastructure in a general-purpose programming language. Among the several available ones, we chose Typescript to take advantage of the benefits this programming language serves. Instead of writing the resulting Cloud Formation Stacks using the native JSON or YAML format, Typescript makes infrastructure design, deployment and the overall coding experience more enjoyable, even allowing good practices such as code reviews, unit tests, and source control to make the infrastructure more robust.
The overall architecture, implementing the beforehand use cases leverages the following AWS services:

Lambda: it’s the core of our serverless application, since it allows us to run the code without caring about provisioning or managing servers (and only pay per use!);
Amazon API Gateway: it sits in front of our lambdas exposing them as REST APIs and taking care of authentication;
Amazon Cognito: this takes care of authenticating our users;
Amazon S3: as a storage service, it helps us to manage data in every format we will need, storing them as objects in Buckets;
AWS SQS: it’s a queue service and we use it as a decoupling mechanism, to avoid losing messages;
Amazon DynamoDB: the fully managed NoSQL service offered by AWS. We use it to persist generation info and let users and third parties know the status of each of their imports;
Amazon CloudFront: simply put, it’s a CDN fully integrated with the AWS ecosystem. We use it for caching and for generating resized covers on the fly;
AWS Code Pipeline and AWS Code Build: to take advantage of the CI/CD approach in order to automatically build and deploy our code.

We need to store on Amazon S3 different sets of data, thus we defined four buckets starting from the access patterns:

CSV Bucket: used for uploading the CSVs (via presigned URLs);
Track Bucket: used for storing the track metadata. This bucket has no public access and is used only by the lambdas to write and read them;
Image Bucket: this bucket is served by Cloudfront and contains the generated cover and the zip archives;
Frontend Bucket: for storing the frontend web application, written in React. The bucket assets are then served by Cloudfront.

Using multiple buckets (instead of just one bucket for everything) makes our life easier for handling permissions and reduces the chance of human error.

The Cover Generation Lambda

The Lambda which translates the Processing algorithm must reproduce all the expected behaviors, such as receiving track details, generating some variables and colors, drawing shapes and applying stickers, if needed.
All the Processing APIs used by the algorithm were available as part of the Canvas API. So we decided to convert the Processing code in Typescript and use the node-canvas package to reproduce the same behavior. This package needs some native libraries which can be easily provisioned on the Lambda function via an existing AWS Lambda Layer.
Once we were able to replicate the algorithm result, making it work as a Lambda function was the easiest part: we just needed to slightly change our function definition to adhere to the lambda event specification.
The function would need to access several static assets which would make the deploy artifact bigger. We could have used S3 for them, but this would result in a lot of unnecessary API requests, so we moved these assets in a Lambda Layer too.
Having the cover generation lambda complete, the rest of the product was already appearing as an easy task. We divided the rest of this serverless architecture into three main parts, described in the following paragraphs.

Single Track Creation

The simplest flow is the Single Track Image Creation. The flow is thought to be used by a third party user and the BeatIdentity admin.

There are two APIs that need to be called by third party services. They are implemented as Lambda Functions and exposed via API Gateway and authenticated against an API Key.
Calling the “Create Single Track” API will insert the corresponding record in DynamoDB, save the JSON details in the JSON Bucket and start the Step Function workflow, which is async. This way the HTTP request is not blocked by the cover generation, and the user can know if the generation is complete by calling the Status API, which fetches the status from the corresponding record on DynamoDB. The state machine provided by the AWS Step function allows us to perform all the tasks needed to call the “Cover Generation” Lambda, keep the corresponding record on DynamoDB updated, and handle errors.

CSV Upload

The aim of the second flow is to allow BeatIdentity admins to massively import hundreds of tracks at the same time uploading a CSV in which each row represents a track.

At the very start of the flow, the frontend application calls the “Presigned URL” API to obtain a presigned URL which allows uploading the CSV on the S3. This technique guarantees a secure way of:

Keeping the CSV bucket private;
Offloading the file from our service to S3, reducing the overhead of receiving the file on the backend and then uploading it from the backend onto S3.

More info on this topic can be found here.
When the upload on S3 is complete, a message is propagated to a SQS Queue, and consumed by a Lambda Function (Start State Machine) which is responsible for starting the Step Function responsible of processing the CSV (a Step Function cannot be started directly from an SQS queue). If any error occurs, a message will be requeued and, if the error persists, after some time it will be archived in a DLQ queue for further analysis.
The Step Function of this flow is slightly different with respect to the previous one, since it needs to create images from each CSV row as fast as possible. Thus, the iterator operator was implemented to perform the actions in parallel and to process rows in groups of 5 elements for each iteration. At the end of the generation a zip archive is created containing all the generated Covers and stored in the Image Bucket to be later downloaded.

Cover Request

Until this point each generated image can be fetched in its high resolution size. But someone may also need scaled down versions. To allow this we implemented a common pattern through Lambda@Edge.

When a cover image is requested to Cloudfront (its path starts with “/covers”), the request is sent to S3 to retrieve an object from there. The S3 response is then handled by a Lambda@Edge which is a special Lambda acting as a middleware.
If the S3 response is a “Not Found Object” and the requested path is referring to a cover image, the Lambda@Edge will fetch the high resolution image from S3, size it down to the desired dimension, store it on S3 and return the image content. This way on the next request for the same object, S3 would return the resized version and the Lambda would just ignore the message.
This pattern is explained in detail here.

What we got at the end of the day

We could have delivered this product in a plain, classic way: a simple server configured with Java and Processing and a couple of PHP web pages. The user would have used the web pages to let PHP handle the Processing sketch and generated the images, maybe storing them on the EC2 EBS itself. This would have reduced (slightly) the development time, but at what cost?

Scaling is not obvious and needs some additional work;
Security is not by-default. An EC2 server needs additional work on this point too;
The instance needs periodic updates;
The infrastructure has fixed costs, even if we don’t use the platform.

Instead we decided to aim for a fully serverless architecture:

The cover generation lambda and the resize lambda@edge function complete their work in about 2 seconds, which means that generating covers for 1 MILLION tracks will cost 34$ which is slightly less than paying a t3a.large EC2 instance;
We don’t have to take care of infrastructure security at the same level of classic infrastructures, we just need to ensure the correct permissions are set and that our code is not faulty;
We have built-in decoupling between the different components, which means changing a peace of our design is pretty easy;
Thanks to CDK we have a single repository, put in CI/CD, which contains both our infrastructure and application logic, in a homogeneous language, making it easy to understand how data is flowing.

Overall, the final result exceeded the client expectations, and the effort required to both translate the original generation algorithm to a different language and to develop the product with a serverless approach required the same time we would have needed to provision and properly configure a classic infrastructure.

Looking at the repository, it’s easy to see which components do what in our architecture, because the stack code appears as an imperative function of code composing pieces together and adding behaviors. For example, we create a csvBucket, then we create a “CSVGenerationFSM” (which is a construct for our step function) then we do “fsm.bindToS3Bucket(csvBucket)” to imply that our step function will start when an event is triggered on the csv bucket.

So, our advice is, don’t be afraid of the serverless world, and don’t play safe! Just start playing with it and look at some other architectural examples (like the one we talked about in this post) to get inspiration for improving your design more and more.

Some advice for the beginners:

The AWS world aims for security over all. If you have a bucket and a lambda, you have to explicitly give permission to the lambda to write or read on that bucket, and you can also scope this permission to specific objects or prefixes. This applies to any AWS service and to any action you make on them. So, even if it may seem complicated when you start, once you’ll get used to it, you’ll discover your products have never been so robust and secure!
It’s not you that are not good at googling for documentation! This is indeed one the few things on which AWS could improve. And they are doing it, because if you look at the CDK documentation, it’s awesome!

Co-Authored with: Antonio Riccio

Why we went from Slack to Discord

Matteo Joliveau — Thu, 09 Apr 2020 15:56:01 +0000

The beginning of 2020 was one of the most difficult moments of the last decade. The rapid spread of SARS-COVID-2 and the consequent quarantine imposed by the various countries of the world has led many companies to discover and approach remote work for the first time, in order to protect their employees while still being able to work without attending the office. However, choosing the right tools to support remote collaboration is almost as difficult a challenge as adopting "smart" working methodologies that allow you to keep productivity high. The panorama of available choices is vast, from open-source tools such as Jitsi and Nextcloud to commercial services such as Slack and Google GSuite, each with its own strengths and peculiarities.

At Mikamai we have always been used to working in a "smart" way, preferring asynchronous and written collaboration wherever possible to make our business more streamlined and agile. This allows us to expand our operational area to the entire world, both in terms of customers and collaborators. In fact, many colleagues do not work from our main office in Milan, Italy, but from more comfortable places for them, such as other Italian or European cities, even other continents. We have long been GSuite users for shared documents, GitHub for hosting our projects, Trello for managing activities and Slack for internal communication.

However, we recently started to feel the limitations of Slack’s free plan. Mikamai has grown, the number of people and messages written every day has grown significantly with it, and we have found ourselves to frequently exceed the limit of 10,000 messages, thus losing potentially important conversations and messages. The lack of group video calls forces us to use Google Meet, an excellent video conferencing solution but which lacks the immediacy of starting a call directly from the text chat. However, paid plans, the price of which increases for each employee present in the workspace, are not very appealing to us. So we started scouting for a tool that would better meet our needs.

In the past few months we have tried several online communication tools, in particular Mattermost, an open source alternative to Slack, JetBrains Space, a collaboration and management service that includes a chat very similar to Slack, and Discord, a communication platform designed for gamers and online communities, but with very interesting features also for companies that work in a distributed way.

The latter has caught our attention, and here are the reasons that led us to adopt it as our corporate communication tool.

Voice chat is only one click away

Despite working mainly in an asynchronous way through email, Trello cards and comments on the code repositories, direct conversation is a very important tool especially for creative activities that benefit from a rapid and continuous exchange of ideas between the participants. The ability of immediately entering a group voice chat, with excellent audio quality and with the support of text chat and screen sharing is perhaps Discord's main strength, and what really convinced us in attempting the transition.

Brainstorming and planning sessions can be started quickly without the overhead that changing tools entails when using services outside the chat, as was our case with Google Meet. Everything is much more natural and immediate, promoting communication and the exchange of ideas rather than discouraging them.

It is also very convenient for organizing virtual coffee breaks, allowing you to maintain human contact with colleagues that you risk losing by working remotely.

Granular control over roles and permissions

Discord was created to host online communities, mainly in the gaming world but extending over time to all those groups of users who need a safe digital space in which to meet and chat. For this reason, the platform provides a wide range of permissions, which can be grouped into roles to be assigned to users, allowing you to granularly control which actions are granted to which users, and in which channels.
In our particular case we do not manage a public space but a private server, however it is not used solely for business purposes. We have dedicated some channels to external activities, like video games or role-playing games, which involve the presence of users who are not part of the company staff, such as family members and playmates, former colleagues and partners of other companies. Thanks to Discord's powerful permission system, these users have access only to the channels for which they were authorized, while the company's internal chats are hidden and inaccessible.

This also allows us to invite our customers to join the server to facilitate the exchange of information with them. For each project, a text channel and a corresponding voice room are created, along with a role that allows access to them. When the customer enters the server and is assigned the role, he can then access these channels and actively collaborate with us, without being able to see reserved areas and spaces dedicated to other active projects.

We have also created a dedicated announcement channel, in which only Mikamai's administrative staff can write, to support the internal mailing list in quickly sharing official news and communications with colleagues.

External integrations

Slack boasts a multitude of integrations with third-party systems, from ticketing and project management platforms to infrastructure alarms and monitoring systems. Discord does not have the same spread in the business and development world, but provides the possibility to configure the integrations in a Slack compatible mode, basically allowing to integrate any system that can interact with Slack.
Discord also has a robust and diverse chatbot ecosystem, small applications that act like automatic users and increase native chat features. For example, a very successful bot in Mikamai’s server is Groovy, a bot that allows you to play music to all users connected to a voice channel as a kind of virtual jukebox. Thanks to it, we can share our music with colleagues, allowing everyone to add songs to the playlist being played.

Cost

Obviously one aspect that we have not underestimated during our evaluation is the monetary cost compared to the competition. All of Discord’s core features are completely free, providing paid additions mainly aimed at the gaming world, especially streamers and youtubers, such as the ability to add custom emojis, customize the profile and improve audio and video quality. This, added to the points previously discussed, was one of the key points for Mikamai in making the decision to migrate to Discord.

Testimonials

Given that I’m not a very techy person, which often limits me in discovering the full potential of this type of tool, I must say that I find Discord very versatile. I really appreciate the possibility of enabling audio channels that allow an immediate switch between text communication and voice communication (especially group chats). Having an unlimited number of messages always available free of charge is, of course, another nice advantage.

Debora, Office Manager

Discord adds syntax highlighting for code snippets, unlimited message history, integrated calls, screenshare with up to 50 people (during the COVID-19 period), a really advanced role and permission system, and is very easy to integrate with external tools. I don't know what more you could ask for.

Nicola Racco, Tech Lead

Nicola RaccoFollow

I have been using Discord for more than a year mainly for gaming but also to talk or see my friends. I immediately found it intuitive and with a youthful graphic feeling.
Its strength is absolutely being free, I still remember when I had to communicate with other players simultaneously, we used TeamSpeak and it required us to spend a lot of money every month for a dedicated server, with an increasing cost based on the amount of users who could connect at once, a real nightmare.
With Slack, I suffered (and still suffer) because of the limited chat history, 10,000 messages or a month of history and then that’s it. How many lost links, how many chats that have fallen by the wayside ("I wrote it to you on Slack", "When ???") ...
Discord is, in my opinion, the tool par excellence of remote communication both for gaming, chatting and companies.

Daniela, Frontend Developer

DanielaFollow

Conclusions

Discord has proved to be an excellent corporate communication platform despite being mainly geared towards the world of gaming and digital communities. The speed with which it is possible to start talking with colleagues and organize virtual meetings has allowed us to facilitate the transition even for those who were not previously used to working completely remotely, and has given us back a part of the warmth that sharing the day with our coworkers transmit, reducing the alienation due to the emergency situation in which we find ourselves. We will have the opportunity to test it thoroughly in the coming months, but we are sure that it will prove itself to be a valid and central tool for our daily work.

DEV Milan has started, and it was awesome!

Matteo Joliveau — Tue, 24 Sep 2019 13:12:34 +0000

On Monday 23rd September we kicked off the first event of the DEV Community Milanese chapter, and it was a blast! 🇮🇹

We as Mikamai have fostered and helped to gather many tech communities in Milan over the course of the years, and it with great pleasure that we organized this one ourselves to thank all these awesome people and their daily effort in bringing Italian developers together.

We hosted two talks (actually one big talk divided into two sections) exploring the magical world of GraphQL APIs first on the backend side, and then on the frontend.

The Mighty Backend

I started the evening with the backend talk, introducing new people to what GraphQL is and isn't and quickly explaining the inner working of the language, then moved on to more practical advices and strategies for implementing clean and effective GraphQL APIs.

We touched topics like types and scalars, datasources, batch loading and evolving schemas, and tried to give actionable suggestions on how to cleanly and efficiently tackle all these issues.

Participants were very active and asked clever questions that made the whole experience much more inclusive and enjoyable!

When in Italy, just pizza 🍕

After the first talk, it's time for a break! We offered pizzas, chips and beverage to all attendees and had a fun time networking, discussing and generally have a good time. It was really fun and I'm very proud of how the evening turned out.

The Glorious Frontend

Then it was time for the second half of the event. My colleague Mattia Panzeri, one of our finest frontend engineers, talked about the challenges of modern frontend development, from multiple devices and screen resolutions to finicky mobile networks and voice assistants, and how GraphQL help clients tailor API requests exactly to their needs. He provided examples of similar and related technologies like Netflix Falcor and why they failed to achieve what GraphQL does.

All of this explained with his witty and funny style that made the whole speech very lighthearted and enjoyable.

We close the evening with a live demonstration using GitLab's GraphQL Explorer to showcase how queries work in the wild, and of course, this being a live demo it went horribly wrong with authentication problems and the window manager of my laptop ultimately deciding that it didn't like our projector so much, but all in all we had good fun.

We gave out stickers to everyone and announced a followup of this event, that we absolutely want to make again and possibly on a regular basis. We are thinking one per month or every two months, depending on speakers' availability.

I really want to thank the DEV Team for launching IRL and supporting this kind of initiatives. There is still work to be done on the logistic side of things, but in the end, the event was a success and everyone enjoyed it.

Thank you for reading and I hope to see you next time!

You can find the slides here.

// Detect dark theme var iframe = document.getElementById('tweet-1176193547768389633-453'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1176193547768389633&theme=dark" }

Effective Service Objects in Ruby

Matteo Joliveau — Mon, 04 Jun 2018 19:47:48 +0000

As a Java developer that recently transitioned to a Ruby on Rails company, I felt kinda lost when I discovered that the use of models directly inside of controllers was a common practice.

I have always followed the good practices of Domain Driven Design and encapsulated my business logic inside special classes called service objects, so in Java (with Spring) a controller would look like this:

@Controller
@RequestMapping("/api/users")
public class UserController {

    private final UserService userService;

    public UserController(UserService userService) {
        this.userService = userService;
    }

    @GetMapping
    public ResponseEntity<Iterable<User>> getAllUsers() {
        return ResponseEntity.ok(userService.getUsers());
    }
}

Verbosity aside, this is nice and clean with good separation of concerns. The actual business logic that retrieves the user list is delegated to the UserService implementation and can be swapped out at any time.

However, in Rails we would write this controller as such:

class Api::UserController < ApplicationController
    def index
        @users = User.all
        render json: @users
    end
end

Yes, this is indeed shorter and even cleaner than the Java example, but it has a major flaw. User is an ActiveRecord model, and by doing this we are tightly coupling our controller to our persistence layer, breaking one of the key aspects of DDD. Moreover, if we wanted to add authorization checks to our requests, maybe only returning a subgroup of users based on the current user's role, we would have to refactor our controller and putting it in charge of something that is not part of the presentation logic. By using a service object, we can add more logic to it while being transparent to the rest of the world.

Let's build a service object

In Java this is simple. It's a singleton class that is injected into other classes by our IoC container (Spring DI in our example).
In Ruby, and Rails especially, this is not quite the same, since we can't really inject anything in our controller constructor. What we can do, however, is taking inspiration by another programming language: Elixir.
In Elixir, a functional language, there are no classes nor objects, only functions and structs. Functions are grouped into modules and have no side effects, a great feature to ensure immutability and stability in our code.
Since Ruby too has modules, we can use them to implement our service object as stateless collections of methods.

Our UserService can look something like this:

module UserService
    class << self
        def all_users
            User.all
        end
    end
end

And then will be used like this:

class Api::UserController < ApplicationController
    def index
        @users = UserService.all_users
        render json: @users
    end
end

This doesn't sound like a smart move, does it? We just moved the User.all call in another class. And that's true, but now, as our application grow we can add more logic to it without breaking other code or refactoring, as long as we keep our API stable.
One small change I'll make before proceding. Since we may want to inject some data into our service on every call, we'll define our methods with a first parameter named ctx, which will contain the current execution context. Stuff like the current user and such will be contained there.

module UserService
    class << self
        def all_users _ctx # we'll ignore it for now
            User.all
        end
    end
end

class Api::UserController < ApplicationController
    def index
        @users = UserService.all_users { current_user: current_user }
        render json: @users
    end
end

Applying business logic

Now let's build a more complex case, and let's use a user story to describe it first. Let's imagine we're building a ToDo app (Wow, how revolutionary!).
The story would be:

As a normal user I want to be able to see all my todos for the next month.

The RESTful HTTP call will be something like:
GET /api/todos?from=${today}&to=${today + 1 month}

Our controller will be:

class Api::TodoController < ApplicationController
    def index
        @ctx = { current_user: current_user }
        @todos = TodoService.all_todos_by_interval @ctx, permitted_params
        render json: @todos
    end

    private

    def permitted_params
        params.require(:todo).permit(:from, :to)
    end
end

And our service:

module TodoService
    class << self
        def all_todos_by_interval ctx, params
            Todos.where(user: ctx[:current_user]).by_interval params
        end
    end
end

As you can see we are still delegating the heavy database lifting to the model (throught the scope by_interval) but the service is actually in control of filtering only for the current user. Our controller stays skinny, our model is used only for persistence access, and our business logic doesn't leak in every corner of our source code. Yay!

Service Composition

Another very useful OOP pattern we can use to enhance our business layer is the composite pattern. With it, we can segregate common logic into dedicated, opaque services and call them from other services. For example we might want to send a notification to the user when a todo is updated (for instance because it expired). We can put the notification logic into another service and call it from the previous one.

module TodoService
    class << self
        def update_todo ctx, params
            updated_todo = Todos.find ctx[:todo_id]
            updated_todo.update! params # raise exception if unable to update
            notify_expiration ctx[:current_user], updated_todo if todo.expired?
        end

        private

        def notify_expiration user, todo # put in a private method for convenience
            NotificationService.notify_of_expiration { current_user: user }, todo
        end
    end
end

Commands for repetitive tasks

As the Gang of Four gave us a huge amount of great OOP patterns, I'm going to borrow one last concepts from them and greatly increase our code segregation. You see, our services could act as coordinators instead of executors, delegating the actual work to other classes and only caring about calling the right ones. Those smaller, "worker-style" classes can be implemented as commands. This has the biggest advantage of enhancing composition by using smaller execution units (single commands instead of complex services) and separating concerns even more. Now services act as action coordinators, orchestrating how logic is executed, while the actual execution is run inside simple, testable and reusable components.

Side Note: I'm going to use the gem simple_command to implement the command pattern, but you are free to use anything you want

Let's refactor the update logic to use the command pattern:

class UpdateTodo
    prepend SimpleCommand

    def initialize todo_id, params
        @todo_id = todo_id
        @params = params
    end

    def call
        todo = Todos.find @todo_id

        # gather errors instead of throwing exception
        errors.add_multiple_errors todo.errors unless todo.update @params
        todo
    end
end

module TodoService
    class << self
        def update_todo ctx, params
            cmd = UpdateTodo.call ctx[:todo_id], params

            if cmd.success?
                todo = cmd.result
                notify_expiration ctx[:current_user], todo if todo.expired?
            end

            # let's return the command result so that the controller can
            # access the errors if any
            cmd
        end

        private

        def notify_expiration user, todo # put in a private method for convenience
            NotificationService.notify_of_expiration { current_user: user }, todo if todo.expired?
        end
    end
end

Beautiful. Now every class has one job (Controllers receive requests and return responses, Commands execute small tasks and Services wire everything together), our business logic is easily testable without needing any supporting infrastructure (just mock everything. Mocks are nice.) and we have smaller and more reusable methods. We just have a slightly bigger codebase, but it's still nothing compared to a Java project and it's worth the effort on the long run.
Also, our services are no longer coupled to any Rails (or other frameworks) specific class. If for instance we wanted to change the persistence library, or migrate one business domain to an external microservice, we just have to refactor the related commands without having to touch our services.

Are you using service objects in your Ruby projects? How did you implement the pattern and what challenges did you solved that my approach does not?

Technical Debt: Definition and Practical Approach

Nicola Racco — Thu, 15 Jun 2017 14:14:38 +0000

I know this post doesn’t invent anything new and you may have read about technical debt other hundred of times. This post is just another take on the Technical Debt matter, with the description of the practical approach we follow in Mikamai.

The process of software development is traditionally composed by three phases: development, testing, release. These phases can then be further detailed or splitted: the development phase may comprehend frequent deploys on an alpha environment which will contain incomplete features but will allow to receive feedback easily; testing might then happen on different environments, sequentially (i.e. the first one is an isolated environment with an empty data set, the second one is an isolated environment with a realistic data set, the third one is accessible from the internet and can be used for stress testing, and so on)

The one thing happening at the end of the traditional development life-cycle is the release, where the term “release” is much more like a “furniture shipment” than a digital good delivery. In this traditional lifecycle the product is indeed considered stable, without any issues, and we can rapidly forget it and move on to the next product. There can be a period of controlled operativity where we’re admitting that the app may be affected by some bugs, that will be fixed. But anything weird happening after this period is considered expected.

Problem is: a software feature is not a desk, or a seat.

After the release, even after a reasonable amount of time, we could discover that the developers didn’t fully understand the specs, with the needing now of rewriting parts of code in a hurry, without considering too much themes like code readability.

It may also happen that the delivered feature is good, it’s delivering the business value we had requested, be performant and without bugs. And even in this case, even with this excellent development, a change in the company strategy may require changes to the product some months or years from now.

Even in the most lucky cases, just the passing of time increases day after day the chance that the code need to be changed again, to accommodate a technological upgrade for example.

Whatever way it will happen, suddenly we’re handling this horrific code, written by developers that left the team years ago, written with an ancient technology, and even a small tiny little change, which normally would require just a day, is now requiring weeks.

For example we may found ourselves to have written an e-commerce in a month two years ago, but to need now two months to change a single feature, arguing on whose fault is it and if it’s still reasonable to keep maintaining a so pricey tool.

We’re exaggerating, but not that much.

What is technical debt?

The concept of technical debt is a metaphor created by Ward Cunningham to describe the complexity in software projects. His idea is that releasing a first-time code is like going into debt, which can be paid back with a refactoring. Every minute spent on this not-quite-right code increases the interest on the debt.

Cunningham didn’t just find a good metaphor to describe complexity, but this same metaphor also works quite well to describe the worst case scenario, if development doesn’t follow the correct methodology, the interest on the debt may go up so much to paralyze the project.

Obviously there are developers and Developers, consultants and Consultants. Experience helps a lot in writing better code, in understanding the specifications and write the best implementation on the first try. But this is not enough: sometimes even the best code is anyway difficult to understand and change, thus increasing the technical debt.

Usually we tend to consider technical debt like something that can affect only the “old code”. But it’s not so obvious that a code written in the ‘70s will be more complicated of another one written in the ‘90s.

Another definition of Technical Debt, more commonly accepted, is the one provided by Michael Feathers in Working Effectively with Legacy Code: it’s code without test coverage. It’s a good definition, which can helps us to avoid the mistake of associating the term technical debt with the term old code.

The reason why writing tests is such a good precaution against technical debt is that tests provide an implicit documentation on the application functioning.

So instead than talking of just “untested code”, we’re talking of:

– Code not easily readable;

– Code that doesn’t contain explanations of the logic;

– Not having explanations of the ideas and the decision process that lead to writing the code.

Which offers a completely different point of view on this matter. Because at this point technical debt is not only a technical problem, but it’s also a communication problem.

How is technical debt generated?

Let’s use the e-commerce example described at the beginning.

After the first 100 lines of code this technical debt could be paid back showing the code to a colleague. This simple operation would mean sharing the knowledge, having a feedback on the logic correctness, and a confirmation that the code is readable by others.

After a month, when the e-commerce was released, we had the minimum required to have a functioning platform. In this moment the technical debt could be paid back showing the buying process to our colleagues, checking again the code so we can be sure there are no dark spots too complex to read, and writing some automatic tests that can grant the purchase process is functioning. This technical debt is much higher than before, but still acceptable.

Two months after the release we’re asked of adding an asiatic country: everything need to change, from taxes to customer fields. Given that any change can break the purchase process developers have started testing their changes on the staging environment after each push. This is slowing down the development, and the technical debt is now so high that we’re starting to feel the first problems. To pay back this debt we should now completely change our strategy, starting to dedicate some time to write tests and clean the more problematic points of the code.

Six months have passed. On each new release we’re spotting new bugs, and for this reason it has been hired a QA specialist, whose duty is to visiting the staging environment before each production release to help spot and fix bugs. In this moment the interest on the technical debt is so high that we’re effectively paying a new employee, which would not be needed otherwise.

After an year: Company grew up, and now we need to add the brand concept to the e-commerce. This requires a complete refactoring to the app. The team is doing their best, but development is going slowly. I like to think that at this point the technical debt has gained a physical form: it’s a snake swallowing its own tail. Technical debt is not so high that any development is slower, that the team is tired to work on the project but at the same time is so busy to not have time for refactoring.

We’re not so close from the worst case scenario I mentioned before: rewriting the app costs too much (and rewriting from scratch is never a good choice), changing the app is instead too risky.

Summarizing, every time we’re talking of:

– Code difficult to change;

– Code difficult to read;

– Unstable app;

– Slow development cycle;

– Team hard to expand;

– Application hard to upgrade;

– Application hard to maintain;

– Low performances.

We’re talking of technical debt.

How can we reduce the technical debt?

Technical debt is a communication problem, and this inevitably brings to mind the Conway Law: organizations which design systems … are constrained to produce designs which are copies of the communication structures of these organizations.

Even before solving the technical debt into detail, changing the code, we need to solve the communication problem. We need to determine which communication protocols we’re going to use to document the project itself, the functional specifications, the ideas behind the code, and so on. Only after having decided which protocols to use we can delve into the code.

What are this communication protocols, which can help so much into reducing the technical deb?

– Behavioural tests: automatic tests which test features at a high level (i.e. Using the browser in a web product). More than their value as tests, they’re also usually easy to understand and can reveal a lot about the intentions of the developers;

– The code itself:

– Comments and code formatting. Even a simple blank line between to code blocks can communicate a lot (for example two blocks that deal with two distinct phases of the process);

– Naming of variables, objects, methods, functions. Naming a variable “value” doesn’t say that much on what it may contain, and naming a method “calc” doesn’t say that much on what it’s calculating;

– Errors: raising errors with a detailed description of what is happening may save your day in a year from now, when that error will pop up from nowhere during a process that is considered ultra-stable;

– Again on the errors: there is nothing worse than not raising errors. The worst case is when you’re not receiving any error, still you don’t get any result (e.g. You see the “purchase successful” page but no order has been generated). Errors are useful, always.

– Version control system: A well written commit on Git may be the most precious tool to solve an incomprehensible bug, or when you’re reading an obscure line of code.

– Typical “project documentation”: setup procedures, deploy procedures, how to create a staging/production environment, how to execute a system task, and so on;

– Company chatroom, slack, forums, mailing lists. These tools have a persistent history and we can use them to keep track of discussions on the decision process;

– And there are a lot more than these, some of them more obvious than others.

Using the right communication protocols can help us to build something new, something that can allow knowledge sharing between developers (present and future ones), something that acts as a communication bridge between the owner of the platform and the developers of the platform.

Once we’ve set the communication protocols, we can start analyzing the project as it’s an ancient tomb, documenting each finding and trying to give an explanation to each object.

But the real cleaning should be done gradually, while we’re also delivering new features maybe. But we should avoid changing everything at once. Rewriting a product is almost never a good idea , because rewriting causes the loss of “implicit” specifications, of those hidden features that were contributing to give the product the experience we knew.

There are some other side effects when doing a refactoring on large scale:

– It requires a substantial budget;

– It requires to stop developing new features while cleaning the code;

– It increases the chance to generate bugs because we could clean features whose effects are not fully understood, or because the testing process cannot test all their edge cases.

Let’s imagine that our house is messy and some guests are coming (buying a new house and sell the old one is not an option). If we try to clean all the rooms together we’re risking to go out of time (and guests would see our house in a messy state). We could try to do just a rough cleaning of all the rooms, but our house would be still messy, only less messier than before. We could instead ignore the rooms when guests will not see (our restroom and the kitchen), and concentrate on the ones they’ll visit for sure (the living room and the bathroom).

We could proceed with the same approach when refactoring. We create new features using the new communication protocols, and every time our feature deals or need to change an old feature, we also clean that feature upgrading it to the new standard.

A practical approach

Experience has taught us that we need to follow a method when a client contact us for refactoring, upgrading or maintaining legacy applications.

In these cases, the first step is a Code Inspection , to evaluate the situation. We do a code analysis (with the help of some tools), so we can get an idea of what points of the code are more harder to read or to change, of what security vulnerabilities are present, of the status of the documentation. As a result we deliver to the customer a documentation which describes what we have found and what steps the customer could take to improve the situation.

At this point it’s up to the customer to decide if this code inspection should result in starting a process of improvement. If the process starts, it’s time to define a team :

– If there’s already a senior technical team our developers usually joins the existing team;

– If there’s a technical team but it’s composed mainly by junior developers, we usually try to leave the new developments to the existing team (followed by a mentor). At the same time another team composed by specialists works on the maintenance;

– If there’s no technical team, we can provide one.

Then there’s the definition of a release life-cycle :

– In the worst cases we cannot do this without also creating an operations team;

– If there are no counter issues we usually try to put in place a continuous release process;

– In special cases (for example when each release requires to produce specific documentation for users of the platform) we advise to work with milestones and scheduled releases;

And finally we can define the ceremonies and the communication protocols : this step changes a lot from customer to customer, because some of them have enough time to participate to each communication protocol, some others may only join us for a review/planning meeting each week, some others don’t like video conferencing tools, and so on. The following is a typical setup:

– 15 minutes standup every morning at 10:00 (except Wed);

– 30 minutes review meeting every Wednesday at 15:00;

– 30 minutes planning meeting every Wednesday at 15:30;

– Production release every Thursday at 02:00;

– Retrospective and progress report every first friday of the month at 17:00;

– Definitions of Done. Example:

– For a feature to be released in staging is needed:

– Automatic unit tests;

– Code Review from a colleague;

– Being documented in the Changelog;

– To have at least one integration test.

– For a feature to be released in production is needed:

– To be approved at the Review meeting by the Product Owner;

– Be translated in each supported language.

– For a bugfix to be released is needed:

– To have at least one test that is testing the absence of the bug.

Now the team can start working on the project. The objective is to create stability, to reach that point of equilibrium between growth and stability in which developments at par while keeping the technical debt low.

Usually during time we also find ourselves reducing the team size because:

– The amount of automatic tests reduce the regressions caused by any new feature;

– The code becomes more readable and more manageable;

– Junior developers learn how to write code with a low (or absent) level of technical debt.

The wealth we’re creating at the end of this process allow us to deliver new features easily. At the same time this creates a new class of developers, more virtuous and careful about certain themes. In the end we can say that this process is becoming part of the company culture, changing the result of the Conway Law.