DEV Community: Jérémy Basso

StopCovid app: overview of the backend server code

Jérémy Basso — Wed, 10 Jun 2020 10:32:51 +0000

StopCovid is a mobile application launched by the french government in order to perform contact tracing of COVID-19 infections and inform potential new infected.
The codebase of the application is released publicly here.

I was quite interested in seeing how they have done things because the app has a lot of expectations, and the press about it is quite harsh. One of the questions was the lack of time to implement the app because of the situation.

I decided to perform a quick overview of the backend part, source code can be found here, and share it with you.

I am not an backend development expert (yet), so expect not to agree with me on some of points ! Also, this article is not an audit of the code, but reflects my personal opinion on what I checked.

Tech choices

Language : Java 1.8

 <java.version>1.8</java.version>

The language used is Java in 1.8 version, which is a logical choice for an application of this size. We could expect PHP, Javascript on Node.js as well.
The use of Java 8 can be discussed, since Oracle stopped updating in 2016, and Java 13 has a good adoption so far. Java 8 still works perfectly fine, but my choice would go to a newer version.
Kotlin (still on JVM) was an option as well, but still has less adoption than Java.

Dependencies management: Maven over Gradle

The dependencies management is handled by Maven. When working with Java projects, the two main possibilities for this subject are Maven and Gradle, and both are totally fine.

While Maven was the first to go out, I personally prefer Gradle because of its API thats seems easier to use. Yet, Maven is a valid choice.

RPC framework : gRPC and REST

<grpc.version>1.29.0</grpc.version>  
<protobuf.version>3.11.0</protobuf.version>

I totally expected a classic REST or even SOAP API in this backend, and was very surprised to notice the presence of gRPC.
This framework allows to exchange requests between servers and clients, based on HTTP/2 protocol and Protocol Buffers. One of its strength is the ability to develop backend services, generate gRPC "interface" contract, and then generate connectors to this interface for any platform or language.

For the record, StopCovid frontend's is native Android and native iOS, so gRPC makes total sense. Indeed, you will only have to develop interface once on the backend-side, and the frontend will only need to generate interfaces from this. This is when gRPC shines : avoid duplicate development and mistakes when dealing with multiple performances.

Otherwise, gRPC is as well is centered on high-performance, so another good point here.

When diving into code, I found out REST was used. REST is the most used and most standard protocol out there, so nothing to note here.

To sum up, they used gRPC for all the crypto operations around the app, and REST for "classic" API calls.

Database: PostgreSQL and MongoDB

<postgresql.version>42.2.12</postgresql.version>

I expected that kind of database, which is an industry standard. There are a lot of well known databases out there, my guess would have been PostgreSQL or MariaDB.

<dependency>  
 <groupId>org.mongodb</groupId>  
 <artifactId>bson</artifactId>  
</dependency>

There seems to be some MongoDB as well, and it is a bit surprising to find multiple databases in the same project, but why not.

Diving into code, MongoDB seems to be the most used database in this project.

Spring Boot

<parent>  
 <groupId>org.springframework.boot</groupId>  
 <artifactId>spring-boot-starter-parent</artifactId>  
 <version>2.2.6.RELEASE</version>  
 <relativePath /></parent>

They choose the Spring Boot framework that is very popular, and it is always a good choice. The version is one of the latest, which is good.

Code

Architecture

Modules

<modules>  
 <module>robert-server-ws-rest</module>  
 <module>robert-server-batch</module>  
 <module>robert-server-database</module>  
 <module>robert-server-crypto</module>  
 <module>robert-server-common</module>  
 <module>robert-crypto-grpc-server</module>  
 <module>robert-crypto-grpc-server-messaging</module>  
 <module>robert-crypto-grpc-server-storage</module>  
</modules>

The application is divided in 8 modules , which is understandable. Each module has its own dependencies, which is a bit harder to maintain, but a tradeoff for more separation and readability.

As a result, each module has minimal logic implemented.

Module structure

Every module share the same core package structure, even though some packages are not needed sometimes.
The most complete module I based my reflexion on is robert-server-ws-rest. Here are the packages and how I understand them :

service and service.impl: business logic (interfaces and implementation
utils: utils classes
exception: exceptions and handlers
vo and vo.mapper: entities for web services and mapping from and to business entities
dto: entities for data transfer (as you can guess from the name)
controllerand controller.impl: controller for API (REST or gRPC)
database: database functions
models: business entities

This structure is pretty simple and does the job. In that kind of structure you do not split into few layers (like Data/Domain/API), but you just split every different type of file by its role, which leads to more packages.

Code quality

Testing

A lot of tests follow the pattern Given/When/Then, which provides a good readability. Yet, the tests seems a bit simple and test only a unit of code.

I did not find workflow test of the applications (like end-to-end tests), which I personally like since it helps to understand code, and check that the full workflow is working, not only the different parts separately.

It would be interesting to know the test coverage of the backend, my guess is above 90%, since every important function seems to have one test tied.

Comments and javadoc

My general opinion is that sometimes this code lacks of comments.
There are good comments when something weird is implemented, but there are missing comments about standard stuff, like fields description. Some fields have name like "ebid" or "ecc", and without any comment it is impossible to know what those fields are about.

The project contains 11 TODO's, which is surprising for a released product. Most of them are tied with commented code.

The javadoc around the code is very inequal, it depends on the modules. Some modules contains javadoc on all functions (robert-server-crypto for instance), when other zero (robert-server-database for instance).

When browsing around the code, I was surprised to notice huge amount of commented code.

This if of course bad practice, and I was very surprised to find that in production version of the code. This kind of mistakes is easy to check and remove, I guess time was the missing resource here.

Magic Numbers

The code contains a lot of magic numbers. Magic numbers are variables that contain numbers, initialized in the code instead of stored in a constant elsewhere, as an example :

@NotNull  
@Size(min = 6, max = 36)  
@ToString.Exclude  
private String token;

Those numbers 6 and 36 should be stored elsewhere and accessed trough constants.
Imagine you want to change the size of the token max from 36 to 40. If you change the number below, you could break your code elsewhere because the size 36 would be hard coded somewhere else. By storing it to a constant, you could modify it in one place and the change would be effective everywhere else.

General quality

From my point of view, the overall quality of code is good. Most of the logic is related to crypto functions, and it is difficult to find any issue with those functions without testing it.

I think what is missing is a tool for quality of code that would help detect bad practices from the start and fix most of the mistakes quoted before (Sonarqube as an example), and probably some time to work with it.

Conclusion

First of all, it was very interesting to gain access to the code base of the application, and I hope this will become a standard for government projects. It was a great experience for me to check how things are done for a backend of this importance, and I learned a lot.
I find out it is really difficult to properly evaluate a code like this without spending lot of time on it. While it is easy to find basic mistakes, it is very hard to assess how logic is implemented without a complete understanding of the business rules.

To conclude about StopCovid backend, I think most of the choices technology-wise are good. The overall quality of the code is good, but you can notice the lack of time they had to implement it with some bad practices.

My point here is not to cast disgrace on the backend development team, I think they did their best with what they had and they can be proud of the result. The issue is more about releasing an app when time is running out and development seems not totally finished.

It seems dangerous to release the StopCovid at this state, because when someone with bad intentions want to break the app, they will manage to, and they did during the bug bounty.

Also, it is interesting to note that the app got released on June 2nd, few days only after opening the code source and the related bug bounty.

At the moment I write (June 10th), the StopCovid app has more than one million downloads, which is huge, even though the interesting metric would be the daily users.

Since this app had a lot of discussions in France, here is some interesting content about other sides of StopCovid :

Thanks for reading !

Node.js : Upload binary image "on-the-fly" with Fastify and Cloudinary

Jérémy Basso — Thu, 07 May 2020 16:17:30 +0000

In this article I will describe "on-the-fly" upload of a binary image, using Fastify and Cloudinary in an Node.js environnement.

Context

Fastify is a high performance web framework built for Node.js. In my opinion it is the best web framework nowadays (as I like to call it "Express 2.0") for backend purposes.

Cloudinary is a Content Delivery Network, which allow us to perform file upload and storage in a very efficient way. I like it a lot because of all the features around pictures manipulations, and because it provide very good package for free users.

The whole reason when using a CDN is to avoid storing files in your backend storage (wether it is database or file system), because it can lead to performance issues or storage overflow. This is the reason why I implemented a quick way to Upload image from a distant device to a CDN without storing any file.

My use case is written in Typescript, but the equivalent can be done with standard Javascript.

Requirements

First, you will need to install some packages

npm install fastify fastify-multer cloudinary

You will also need to create an account on Cloudinary, you can do it for free.

The core logic

Getting the data from outside

We need to initialize the fastify router, and bind it fastify-multer plugin :

import fastify from 'fastify'
import multer from 'fastify-multer'

const server = fastify()
server.register(multer.contentParser)

server.listen(8080, (err, address) => {
  if(err) {
    console.error(err)
    process.exit(1)
  }
  console.log(`Server listening at ${address}`)
})

Then, we add a new POST route, /api/profilePicture with a specific preHandler from multer package :

import fastify from 'fastify'
import multer from 'fastify-multer'
const storage = multer.memoryStorage()  
const upload = multer({ storage: storage })

... // See above

server.post('/api/profilePicture', {preHandler: upload.single('file')},  
 this.uploadProfilePicture.bind(this))

async uploadProfilePicture(req, res): Promise<string> {  
   const binaryData = req.file.buffer
   // TODO : do stuff
   return ''
}

We just set the storage mode for multer to memory storage, so it does not store the file in the file system, but as a Buffer object.

To send a picture, we consume a content of type multipart/form-data, that provides key-value couples.
With the preHandler option, we specify which key provides the data to read from the form data, in our case then name of the key is file.

So, what does multer does with the uploaded data ?
This is when the magic happens, multerstores the file data into req.file. We can access the content of this file with the buffer field.

There is nothing more to do for this part, we already have our file ready to upload on binaryDate variable !

Uploading the data to Cloudinary

First of all, you will need to setup your Cloudinary client using 3 credentials you can find on your account's dashboard : cloud name, api key, api secret.

cloudinary.config({  
  cloud_name : process.env.CDN_CLOUD_NAME,  
  api_key: process.env.CDN_API_KEY,  
  api_secret: process.env.CDN_API_SECRET  
})

Then, you can use your client to upload the data to Cloudinary, and retrieve the result. To do so, I like to "promisify" the function in order to ease the potential logic around.

In my example, I perform an eager transformation on my picture, allowing to have a version of the picture cropped and centered in the face of the person (for a profile picture as an example) built by Cloudinary.
Cloudinary provides tons of options to deal with pictures and it is likely that you find happiness among them.

uploadPicture(content: Buffer): Promise<object> {  
    return new Promise((resolve, reject) => {  
        cloudinary.uploader.upload_stream(  
            {  
                folder: 'profile_pictures',  
                eager : [{ width : 400, height : 400, crop : 'crop', gravity : 'face'}]  
            }, (error, result) => {  
                if (error) {  
                    throw Exception('Upload failed')
                  } else {  
                    resolve(result)  
                 }  
            }  
        ).end(content)  
    })  
}

Wrapping everything up

The final route code looks like this :

import fastify from 'fastify' import multer from 'fastify-multer' const storage = multer.memoryStorage() const upload = multer({ storage: storage })
const storage = multer.memoryStorage()  
const upload = multer({ storage: storage })

const server = fastify()
server.register(multer.contentParser)

server.listen(8080, (err, address) => {
  if(err) {
    console.error(err)
    process.exit(1)
  }
  console.log(`Server listening at ${address}`)
})

server.post('/api/profilePicture', {preHandler: upload.single('file')},  
 this.uploadProfilePicture.bind(this))

async uploadProfilePicture(req, res): Promise<string> {  
   const binaryData = req.file.buffer
   const result = await uploadPicture(binaryData) // Watch below for details
   return result.url
}

In this example I only return the URL of the uploaded picture, but in a more real life use, I would store it in my database, in order to send to frontend the URL of the picture when needed.

I really liked this way of working with file uploads, and I highly recommend you to use that kind of strategy for your projects.

Thanks for reading !

Kotlin Spring Boot use-case: dependency injection of a list of services

Jérémy Basso — Tue, 28 Apr 2020 16:40:26 +0000

Context

With Spring boot framework, it is possible to inject a list of beans as a dependency. I always found the use of this feature useful and elegant, so I am sharing my take on this subject.
My example will be using Kotlin, but we can assume that it works in Java just as well.

Our example will solve the problem of authentication with multiple providers : standard auth, Facebook, Google or anything else. To keep it simple, we will focus only on the login feature.

Setting up the providers

When injecting dependencies, we will create an interface that will define the beans to inject. Here, we will call this interface ILoginProvider :

interface ILoginProvider {  

    fun login(request: AuthRequest): Account 

    fun supports(request: AuthRequest): Boolean  

}

The login function is where the login logic happens, the supports function is in charge of returning if the AuthRequest object is supported by each provider or not.
We pass an object AuthRequest as an argument for login and supportsfunction:

data class AuthRequest(
    val type: AuthType = AuthType.OTHER // enum : [EMAIL,FACEBOOK,OTHER]

    val email: String = ""

    val credentials: String? = null // only filled with email/password

    val socialToken: String? = null // only filled with FB
)

This interface will be implemented by a new class for each source. For our example we will create EmailLoginProvider and FacebookLoginProvider (example below), that will both implement ILoginProvider.

@Service  
class FacebookLoginProvider() : ILoginProvider {

override fun login(request: AuthRequest): Account {
    // TODO : implement login function
    }

override fun supports(request: AuthRequest): Boolean {
    return request.type == AuthType.FACEBOOK
    }
}

Setting up the service

Now it is time to inject those providers, and it is definitely the cool part. When using Spring Boot, you can use the magical @Autowiredannotation to perform injection. In our case we want to inject all beans implementing ILoginProvider in our LoginService :

@Service 
class LoginService() {

@Autowired
lateinit var loginProviders: List<ILoginProvider>

}

The last thing to do is to make good use of this list of providers, by correctly wiring each AuthRequest to the accurate provider: You can do the following:

@Service 
class LoginService() {

@Autowired
lateinit var loginProviders: List<ILoginProvider>

fun login(request: AuthRequest): Account {
    val provider = loginProviders.firstOrNull{it.supports(request)} 
        ?: throw Exception("Provider not found")
    return provider.login(request)
    }
}

Conclusion

By injecting lists of providers, we are able to maintain genericity between different sources of providers. In our example, it is very easy to disable a source (by making supportsreturn false), add a new one (by adding new AuthType and creating a new provider).
It is very useful when the context is a constantly changing product

You can scale this approach to any number of providers, and add more logic for provider selection through supports function..

Some common features benefits a lot from this approach, such as authentication, payment. This approach can be adapted to any language or framework, but my point was to highlight how elegant it was using Spring boot framework.

Thanks for reading !

Why you should build a starter project to enjoy your side-projects

Jérémy Basso — Thu, 16 Apr 2020 16:34:26 +0000

Background story

When you are working as a developer for a company, and you want to keep working on side-projects at the same time, you will probably struggle finding a well-known and desirable resource : time. Directly linked to time, another important resource is energy. It is necessary, in my opinion, to save free time for non work-related tasks, in order to relax and maintain enough energy. Finally, the third resource is attention, when a big task is ahead of you, it gets harder to maintain focus.

I mentioned time, energy and attention for a reason : those are often quoted as the components of the TEA framework. The idea behind TEA is that if you lack any of the 3 resources, you will never be able to achieve optimal productivity. In the case of side-projects, I think of "productivity" not as the amount of code you can deliver, but as the good balance that will provide pride, fulfillment and blooming from working on your side-projects.

At the beginning of my carrer, I tried really hard to combine the work projects, and the side-projects. As a result, I lost motivation for my side-projects because :

I had less time to invest for the side projects.
It was hard for me to start something from scratch because the "motivating" part needed some boring setup first
I did not want to spend all my free time working on my side-projects, I just wanted to jump directly to the fun part

Few weeks ago, an unexpected situation occurred : the COVID19 lockdown. Having to spend all my free time at home, I decided to give my side-projects a try again, and tried to find the tricks that could work for me.
I wanted to share with you the trick that worked best, it is not a miracle solution, yet it worked very well for me : what I call starter projects.

What is a starter project ?

As a developer, whether you are an expert of a specific language, or more of a Swiss Army knife that likes to try a lot of languages, you probably have 2-3 "go-to" languages and frameworks, that you like and will always choose for a side-project. It is not necessary that the chosen language is something you work with your company, side projects can be a good opportunity to try something different.

As an exemple, in my company I work mostly as backend engineer with Kotlin, but on my side projects I often go for Node.js + Typescript for backend, and Flutter for mobile frontend. It allows me to keep learning outside of my job, and most importantly have a blast developing in languages that seduces me (not that I dont like Kotlin, but I am getting used to it).

So here we are, you picked a language, a framework you like, and you are ready to start !

Where to start ?

Functional scope

First of all, try to find out what are the features your starter should have. Pick few of your side-projects ideas or existing applications, and try to define what features are always needed among those. This should help you finding the scope of your starter project. You should think of this as a potential starting point for every of your projects. Your incoming projects should be a fork of this one.

Example for backend starter : authentication email/password, database connection, account management, admin routes

Architecture

Once you decided your functional scope, you should spend some time on your architecture. That's the point of having a starter project : you need a good architecture, that you like, and that you feel confortable with. That's an opportunity for you to try something new, this starter is designed by you and for you, so feel free to try new things, as long as you can handle it.
My tip for architecture is to use a whiteboard to visualize what you want to do, it will help you iterate over your architecture and spot potential issues.

Example for backend starter: hexagonal architecture

Implementation

It is getting real now ! You can now start working on what you planned beforehand.
Take the time to decide if you want to integrate some libraries into you starter. The idea is to limit as much as possible the number of libraries on your project, only the minimum necessary.

Some tips I am sharing to you (take it or leave it):

• Comment and document correctly your code. All of your projects using the starter will contain your starter.

• Setup the test environment. Tests are good, make tests

• Have a full implementation of a dummy resource (not a complex one). It allows you to immediately have a complete process to copy paste in your projects. For backend it would be from database to web services, and frontend from the API call to the display screen (even ugly).

• Use a keyword for your project name : it will be helpful for search and replace afterwards

• Setup minimal CI/CD. It could be pipeline to Heroku for backend, Bitrise for mobile dev, anything you like.

• For object-oriented projects, think SOLID. It is more of a general advice, but still a good one.

What now ?

Let's start !

Congratulations, you built your starter ! It took some time but think of all the time you will save on your incoming projects.

To start a new project from the starter, I usually do the following :

git clone myrepo/mystarterproject
cd mystarterproject
rm -R .git
git init

Then you search for your starter project name and replace occurrences with your new project name. Another approach would be to fork the starter project, which I personally dont like because it means your starter project is public, I rather keep it private.

Going further

If you want to go further, here are some things you can do to keep improving your starter.

First of all, you are free to refactor, or change things on your starter. This is a good way of trying new things and then iterate on what you learned.

Secondly, you can add features that you have done in one of your project to your starter, in order to save time for the next project that will need it.
To do so, you can branch from the master branch and include the code that you want to save for further projects. This way, the next time you will start a project, before deleting the git directory, you can merge into your local branch the features you want to include into your new project.
This approach can also be used to separate different technologic choices, such as database choice.

Example for backend starter: my master branch uses MongoDB database, but I have a branch that uses MySQL instead. I have as well a branch where I included full code for Facebook authentication.

Why starter projects are cool ?

Building starter projects helped a lot with my involvement in side-projects, and here is why.

First of all, while I used to dislike building the basic features for my side-projects, building the starter became a side-project on itself, and it was easier to maintain attention and energy knowing that what I was building will save me tremendous time later.

Accurate picture of me and my T.E.A.

Then, it allows to summarize all your knowledge around a technology, and to have something practical that shows where you locate yourself. You can see yourself improving when improving the starter, it is setting interesting milestones

I can finally start projects without spending hours of configurations and boring features, and directly jump to the fun part. The idea that drives me is : the more projects I create (and continue !) from my starter, the more the time spent on building the starter is worth.

Finally, I even started created starters from my starters, for specific types of applications. Yeah I know, this is pretty meta.
The idea is that certain types of applications always have the same core features, and the only difference between them is a standout factor that will decide if the product is successful or not. So imagine you have a starter for that kind of product : you can just try out without big investment different standout factors, until you find the one that makes a successful product.
I am pretty convinced this approach has a potential, maybe it will come within an incoming article.

Thanks for reading !

It was my first article there so any feedback is appreciated !