DEV Community: Qovery

The importance of SemVer for your applications

Romaric P. — Sun, 23 Jan 2022 09:43:09 +0000

For some developers, SemVer can look just cosmetic, nice to have, or simply useless. But SemVer format is mandatory to make reliable software.

I'll explain how over one year, we encountered 2 issues related to SemVer. The first one was critical and led to a production outage, while the other was a lot of trouble for several companies to upgrade a managed service.

What is SemVeR?

First of all, let's remind us what SemVer is: SemVer (for Semantic Versioning) is the process of assigning either "uniqueversion' names or "uniqueVersion" numbers to unique states of computer software. Within a given version number category (e.g., major, minor), these numbers are generally assigned in increasing order and correspond to new developments in the software. [Wikipedia]

1st issue: Patch version

To get a bit of context, we are using EKS (Kubernetes on AWS) for Qovery production, and we wanted to remove the SSH access (remote) from our Kubernetes nodes.

Config change

On Terraform, it was just a few lines to remove:

resource "aws_eks_node_group" "eks-cluster-workers" {
   cluster_name    = aws_eks_cluster.eks_cluster.name
 ...
    // lines removed
   remote_access {
     ec2_ssh_key = "qovery"
     source_security_group_ids = [aws_security_group.eks_cluster_workers.id]
   }
 ...
}

The applying workflow for those changes was:

Force the deployment of new EKS nodes (EC2 instances behind it)
Move pods (a set of containers) from old nodes to fresh new nodes
Delete old nodes

Before running the changes, we checked that every pod on the cluster was running smoothly; there was no issue. So we decided to apply the change as everything looked OK and the operation "as usual".

Outage

The rollout occurred, and all pods moved from the old nodes to the new ones. Pods were able to start without problems...instead of one! Our Qovery Engine (written in Rust), is dedicated to infrastructure deployments.

The pod was crashing 2s after starting during the initialization phase and was in CrashloopBackOff status:

2022-01-16T16:52:09Z DEBUG app::utils: message: Requesting deployment task at engine.local.infrastructure                                                    
thread 'tokio-warp-http' panicked at 'called `Result::unwrap()` on an `Err` value: Other("Failed to parse patch version")', /usr/local/cargo/registry/src/git
hub.com-1ecc6299db9ec823/procfs-0.9.1/src/lib.rs:303:34

We've got an unwrap() here (non caught error) on a Result. So this issue was voluntary, not handled, or definitively not expected.

Investigation

Looking deeper into our Cargo dependencies (Cargo.toml), we did not use this "procfs" library directly. Meaning it's a library dependency, so we dug into the Cargo.lock to find the dependency and bingo! Prometheus library was using it:

[[package]]
name = "prometheus"
version = "0.12.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5986aa8d62380092d2f50f8b1cdba9cb9b6731ffd4b25b51fd126b6c3e05b99c"
dependencies = [
 "cfg-if 1.0.0",
 "fnv",
 "lazy_static",
 "libc",
 "memchr",
 "parking_lot 0.11.2",
 "procfs 0.9.1",
 "protobuf",
 "thiserror",
]

Alright, we see the procfs version is 0.9.1 as described in the error message. So looking at the code in src/lib.rs line 303 we got this:

 /// The version of the currently running kernel.
    ///
    /// This is a lazily constructed static.  You can also get this information via
    /// [KernelVersion::new()].
    static ref KERNEL: KernelVersion = {
        KernelVersion::current().unwrap()
    };

We can see inside current() that it reads a file (the kernel version):

    /// Returns the kernel version of the currently running kernel.
    ///
    /// This is taken from `/proc/sys/kernel/osrelease`;
    pub fn current() -> ProcResult<Self> {
        read_value("/proc/sys/kernel/osrelease")
    }

And the function in charge of this is splitting on dot and ASCII digits:

    pub fn from_str(s: &str) -> Result<Self, &'static str> {
        let pos = s.find(|c: char| c != '.' && !c.is_ascii_digit());
        let kernel = if let Some(pos) = pos {
            let (s, _) = s.split_at(pos);
            s
        } else {
            s
        };
        let mut kernel_split = kernel.split('.');
...
        let patch = patch.parse().map_err(|_| "Failed to parse patch version")?;

        Ok(Version { major, minor, patch })
}

Here comes the fun fact! Let's take a look at the kernel version on AWS EC2 instances deployed by EKS:

4.14.256-197.484.amzn2.x86_64

And we expected to have this struct:

pub struct Version {
    pub major: u8,
    pub minor: u8,
    pub patch: u8,
}

So here is the problem in the kernel number (4.14.256-197.484.amzn2.x86_64).

256: we expected to have a patch in u8, whereas the max of a u8 is 255, so we needed at least a u16. (https://github.com/eminence/procfs/pull/140).

It's not common to have such a high number, but not rare. We can see it on Chrome, vanilla Kernel, and many other big projects. After the patch version corresponds to a build change, it's also common to find the "-xxx" version.

Unfortunately, this library wasn't expecting to have numbers higher than 255, and SemVer discussion about this topic doesn't look to be simple: https://github.com/semver/semver/issues/304.

Solutions and fix

At this moment, we had 3 solutions in mind:

Temporarily remove Prometheus lib
Fix by ourselves those issues report them upstream. It can take some time, depending on the number of changes and impact behind it (primarily for lib retro-compatibility)
Look into commits, and try to move to newer versions to see if those issues were fixed

Temporarily remove Prometheus lib

We decided not to remove Prometheus lib, because at Qovery, it's not only used for observability purposes but also to manage the Kubernetes Horizontal Pod Autoscaler (HPA). It helps us to absorb a load of infrastructure changes automatically and scale accordingly to the number of Engines for deployment requests:

Fix by ourselves those issues

We wanted to go fast. Making a fork, patching libs, and later on, making a PR to the official project repository sounds a good solution, but it would take time. So before investing this time, we needed to be sure that newer commits of profs and Prometheus were not embedding fixes.

Move to newer versions

Finally, "procfs" in the latest released version had the bug fixed! Prometheus lib was not up to date with this latest release of procfs version, but the main branch was up to date.

So thanks to Rust and Cargo, it's effortless to point to git commit. We updated the Cargo.toml file like this:

[dependencies]
prometheus = { git = "https://github.com/tikv/rust-prometheus", rev = "ac86a264223c8d918a43e739ca3c48bb4aaedb90", features = ["process"] }

And finally, we were able to quickly release and deploy a newer Engine version with those fixes.

2nd issue: AWS Elasticache and Terraform

This one was a pain for a lot of companies. AWS decided to change the version format of their Elasticache service. Before version 6, SemVer was present:

5.0.0
4.0.10
3.2.10

When version 6 was released, they decided to use 6.x version format! (https://web.archive.org/web/20210819001915/https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/SelectEngine.html)

As you can imagine comparing 2 versions with 2 different types (int vs. string), would fail on a lot of software like Terraform. You can easily find all related issues due to this change on the "Terraform AWS provider" GitHub issue page: https://github.com/hashicorp/terraform-provider-aws/issues?q=elasticache+6.x

At this time, it was a headache for people using Terraform (like Qovery) who wanted to upgrade from a previous version and "Terraform AWS provider" developers. AWS understood their mistake as they recently removed 6.x and switched back to SemVer format (https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/SelectEngine.html). And, so...here we go again on the "Terraform AWS provider" on GitHub: https://github.com/hashicorp/terraform-provider-aws/issues/22385

Conclusion

Big corporations like AWS or small libraries maintained by a few developers can easily have a considerable impact when SemVer rules are not strictly respected with the correct type. Versioning your applications is not "cosmetic" at all, as you can see.

Following some rules and good practices is essential if we make reliable software in our tech industry. SemVer is definitively one of them!

Deploy JHipster with PostgreSQL in less than a minute

Romaric P. — Tue, 19 May 2020 17:56:36 +0000

JHipster is one of the most popular Java frameworks. Build on top of Spring Boot, it is very convenient to bootstrap a web project. And now it's possible to deploy it in production in less than a minute.

ps: I can't copy/paste the generated markdown due to custom components used

A month after switching from Slack to Discord in a remote-first tech startup

Patryk Jeziorowski — Thu, 14 May 2020 18:02:36 +0000

Nowadays, Slack is often the default choice for an online communication tool in the tech industry. Startups, enterprises, open-source projects - they all use Slack to communicate within their organization but sometimes also with their users and community. Our startup was no different. In this article, I explain what made us look for Slack alternative and what are our impressions after a month of switching to Discord, so you can learn from those before you make your decision.

The background

Qovery is a remote-first startup founded at the end of the last year. I was hired as the first full-time employee. We are all distributed and work remotely. All our communication happens online. Meetings, plannings, brainstorming sessions, talks with founders, technical discussions, and all the rest - it all happens online without exception.

Our product is focused on solving the developers' problems. We help them in deploying their applications on the cloud. We work with all types of programmers - passionate hobbyists, innovative startup guys, serious enterprise architects but also programming newbies. This is why we work with many different kinds of applications and need to be there to answer various community questions.

This all makes it crucial for us to have a tool that allows communicating flawlessly with our community but also internally within our startup, as all our communication happens online.

What made us switch

Slack is not a bad tool. In fact, there's a reason why it's so popular. But, like all tools, it's not perfect.

What we didn't like was primarily the message history limit. On Slack, you have to pay per user. If you want to preserve the chat history for a growing number of community users, this can get pretty expensive quickly. In this regard, it doesn't really support our needs. Don't get me wrong - we are willing to pay for the tools we use, but Slack pricing model is not well suited for a community communication tool.

We talk (voice) a lot. We think it's important to communicate effectively using our tools, especially when that's the only channel of our verbal communication. Slack was not doing its job well in this regard. Sound, connection issues, and noticeable delay of sound despite decent internet connection made the experience at least not perfect. It's not only Slack issue - we tried many other tools, too, and they all often face similar problems.

Solution?

Discord? It's a tool for online gamers, isn't it? Maybe, but... "On paper" it looked like it could fix all the problems we had with Slack. No history limit, focus on great voice communication... We decided to ignore the "it's a tool for gamers only' stereotype and took a bold move to give it a try in our organization. What's the result?

What was expected to be better is in fact better

In practice, all promises Discord gave us, in real day-to-day usage, turned out to be true:

Pricing model - we no longer have to pay per user to keep our chat history. You could use it, in fact, for free. We pay less than 80$ per month now for even better audio quality and HD video calls. The price won't change with our growing number of community users.
Voice / Video chats. This is where Discord shines. The audio quality is excellent, screen sharing works flawlessly, the latency between you speaking and your listeners hearing it is much lower than on Slack and other solutions. Also, what I really like is you do not have to call somebody directly or prepare a group calls - you can just enter a voice channel (Teamspeak and Ventrilo nostalgic memories...).

What was expected to be worse, is in fact worse

Integrations - we were aware that as a mainly gamers' tool, Discord lacks some of the integrations we used on Slack. Slack has a huge advantage in this space. However, thanks to webhooks combined with Zapier, we were able to configure all integrations we had before also Discord.
Threaded messages - we were aware of the fact it's missing in Discord, and in fact, it's painful. I do miss it. I wish it was there.

What surprised positively

What surprised me from the first second I launched Discord app was the smoothness and responsiveness of the app. It runs truly, noticeably, SMOOTHER and FASTER than Slack. Using Slack in comparison to Discord feels like being in a slow mode for me now. I completely didn't expect this (in fact I was not aware that Slack was so clunky). It's simply a pleasure to use Discord.

Another thing that surprised me is that Discord seems to grow really fast. In just a month of our usage, the added noise suppression, which eliminates random noises from your background and an option to turn on the webcam during talks on group channels.

What surprised negatively

What surprised me negatively the most is a very inconvenient upload of very long texts (e.g. application logs). If your copy-paste a very long text, it won't be displayed in the chat in any way - it appears as a text file to download instead.

Another thing - file upload size limits are a bit restrictive. There are no limits for the workspace like on Slack, but for a single file upload, the limits are pretty harsh (8MB / 50MB/ 100MB depending on your plan). This means you'll have to share your files elsewhere.

My final thoughts after one month of switching to Discord

I'm selfish. I wish Discord would switch focus from gamers to professionals. It would be then the perfect tool for any IT organization. But for now, these are just my dreams - the reality is that Discord is much better for voice conversations. The app works much smoother and feels more responsive & modern. Chat is worse due to the lack of threaded messages.

In my opinion, you should use Discord if:

you value quality voice conversations
you communicate a lot with your users/community
you don't mind using gamers focused tool (the UI is less suited for enterprise usage than Slack)

I think Discord may gradually turn into a more business-oriented tool. They add features that are useful for IT teams. There's yet a bit to be done, but I hope it will naturally go in this direction. There are a lot of open-source communities using Discord to build their communities at the moment (including Rust, Django, Hasura, Typescript, and many others) and I believe Discord's popularity will grow in our industry.

And, it the end, they use Rust so no wonder why Discord works so smooth! :)

P.S. If you want to try Discord or ask me or our team any question, feel free to join our Discord server.

Productive and simple way to deploy Dockerized applications

Patryk Jeziorowski — Wed, 06 May 2020 18:51:22 +0000

In recent years, Docker has been becoming more and more popular tool used to deploy web applications. According to Datadog in 2018, the adoption of Docker in large organizations reached about 47 percent and almost 20 percent in small organizations. This report it two years old - no doubt Docker is even more common now.

In my opinion, knowing Docker basics is an essential tool in the toolbox of every software engineer, especially in the web development ecosystem. In this article, I'll demonstrate the easiest way to Dockerize and deploy a simple application. Before we dive deep into practical steps, let's first answer two essential questions - "What is Docker" and "Why should I use it" in the first place.

Docker in a nutshell

Docker is a tool that makes it easy to build and deploy your applications, typically to the cloud environment. It allows you to package your application in a container that contains your app with all of the things it needs, such as libraries and other dependencies. Then, this package can be run on any machine with a Docker engine installed, no matter the underlying configuration or system distribution.

Why should I use Docker?

It works on my machine sentence has become a meme in the software world. You can even get a sticker on your laptop:

Making applications run consistently in various environments is one of the issues addressed very well by Docker.

Docker makes sure that your containerized applications run in the same way on your machine, on your friend's machine, and on the AWS server (and anywhere else where Docker engine is installed). It is truly a superpower. As a developer, you no longer need to worry about the underlying system. After you Dockerize your app, you can be sure that it behaves in the same manner in your development, testing, and production environments, as well as on your local machine. It makes building and testing applications way more comfortable than it was before.

Another reason why you should be interested in Docker is the popularization of cloud, microservices, and Kubernetes. Docker is the first-class citizen in the cloud-native world, so if you want to take the full advantage of scalable, cloud-native application architectures, Docker is the way to go.

How to deploy Docker containers

Let's move on to the practical application and usage of Docker. We'll now build a very simple web application that responds to HTTP requests, dockerize it and deploy to Qovery - a scalable Container as a Service platform.

Create a simple application

For the sake of simplicity, we'll create a simple Node.js application that returns a Hello, World text in response to HTTP requests. I choose Node.js here because it's simple and popular technology, but you can use Docker with basically any language and framework.

Let's create an empty folder for our new application and initialize an empty Git repository:

mkdir deploying-docker
cd deploying-docker
git init

Now, create app.js file with the source code of our server:

const http = require('http');

const hostname = '0.0.0.0';
const port = 3000;

const server = http.createServer((req, res) => {
    res.statusCode = 200;
    res.setHeader('Content-Type', 'text/plain');
    res.end('Hello World');
});

server.listen(port, hostname, () => {
    console.log(`Server running at http://${hostname}:${port}/`);
});

It is a very simple server that returns "Hello World" text on its root endpoint. After it's done, we want to make this app run in a Docker container. To do so, we need to create a Dockerfile.

What is Dockerfile?

Besides containers, Docker uses the concept of Images. Image is a template used to create and run containers. Dockerfile describes the steps required to build the image. Later on, this image is used as a template to run containers with your application.

You can think about images and containers as a good analogy to classes and objects (instances of a given class) in the Object-Oriented Programming world.

Create a Dockerfile that will allow us to run our Node.js app in a container. Create a file named Dockerfile with the following content:

FROM node:13-alpine

RUN mkdir -p /usr/src/app

WORKDIR /usr/src/app

COPY . .

EXPOSE 3000

CMD node app.js

Let's discuss all lines of the Dockerfile:

FROM node:13-alpine specifies the base of our Docker image. It's a base used to get started with building an image.
RUN mkdir -p /usr/src/app creates a new empty folder in /usr/src/app
WORKDIR /usr/src/app defines the working directory of our container
COPY . . adds the contents of our application to the container
EXPOSE 3000 informs Docker that the container listens on the specified network port at runtime
and, finally: CMD node app.js is the command that starts our application.

Now we got all basic things we need to run our application in a Docker container! Let's try it out:

Build Docker image of the app using docker build testing/docker .
Run a container with our application by executing docker run -p 3000:3000 testing/docker

the -p 3000:3000 flag makes container port 3000 accessible on your localhost:3000.

Great! The container is up. Run docker ps to see the list of running containers and confirm that it is indeed running.

Now open a browser at http://localhost:3000 to see that the application in a container responded with Hello, World message.

Did it work? Great. Our app works well in the Docker container. It's adorable, but we want to share our app with the world - running applications only on our own machine won't make us millionaires!

Container as a Service

To deploy our Dockerized application, we'll use Qovery. It's a Container as a Service platform that allows us to deploy Dockerized apps without any efforts. Qovery is free up to three applications (and databases!) in the community version.

Install Qovery CLI

To Sign Up and install the CLI, you can follow the steps described in this link.

After you have access to Qovery, it's time to deploy the application.

Deploy the docker container

Run qovery init
Choose application name, e.g., node-app
Choose project name, e.g., testing-docker
Commit and push your changes to Github: git add . ; git commit -m "Initial commit" ; git push -u origin master" (create an empty repository beforefor your application on Github before if it's not done yet)

Voila! That's all. Your Dockerized application is being deployed as a Docker container. To deploy a Docker container on Qovery, all you need is a Dockerfile that describes containers with your application + running qovery init command to initialize Qovery. From now on, Qovery will build and deploy your Dockerized application after you make any changes in your repository to scalable Kubernetes clusters as a Docker container.

To check that your application is in fact deploying, run qovery status:

BRANCH NAME       | STATUS  | ENDPOINTS                            | APPLICATIONS    | DATABASES
master            | running | https://some.url.qovery.io           | node-app  | 

APPLICATION NAME  | STATUS  | DATABASES
node-app          | running | 

DATABASE NAME     | STATUS  | TYPE       | VERSION | ENDPOINT | PORT     | USERNAME | PASSWORD | APPLICATIONS

Summary

In this guide, you learned the essential basics of Docker. You also learned why you should be interested in using it, and how to deploy your application to the cloud as a Docker container. This is all you need to know to improve your development experience and deploy your application to the cloud with ease! If you have any questions or feedback please do let me know in the comments or join Qovery Discord server and feel free to speak your mind.

Heroku vs Qovery

Romaric P. — Wed, 15 Apr 2020 08:46:30 +0000

The first question we always get from developers before trying out our product is - “What’s the difference between your platform and Heroku?”. We think it’s very important to answer your questions clearly and this article is meant to be our answer.‍

At Qovery, applications, databases, storage, and brokers are first-class citizens

Heroku provides three managed services to all customers:

Heroku Postgres
Heroku Redis
Apache Kafka on Heroku

All other integrations are made as third-party, paid add-ons. They’re not treated on pair with your applications.

At Qovery, we have taken a different approach. Applications, storage, databases, and message brokers are treated as first-class citizens. We treat them all in a uniform way. They have the same pricing and limits of storage, CPU or memory no matter if it's your backend application, MongoDB, PostgreSQL or RabbitMQ. Their configuration is also simplified - connecting your backend to the database is very declarative. **You declare what you need, and we make it available to your application¨¨. This speeds up the development and simplifies the management of services in production a lot.

The list of officially supported services by Heroku is very limited. Our goal at Qovery is to make any service offered by cloud providers effortlessly accessible in your application.

Here’s an example of how easy it is to add PostgreSQL to your application - just add those few lines of configuration:

...
databases:
- type: postgresql
  version: "11.5"
  name: my-postgresql-db
...

That’s all - you'll get a managed PostgreSQL to use in your app.

Qovery is built for teams

One of the superpowers of Qovery is the concept of isolated environments. Have you ever had a situation when you could not test your changes in a deployed environment because it would block your teammate from testing his own feature? This is a common problem of sharing testing and staging environments. Qovery is a remedy for this issue. With Qovery, you have the ability to have an isolated environment for each of your developers. This will greatly speed up the development cycle and the productivity of your team.

Creating a new remote branch results in a clean, remote environment, where you can freely test your code. These are not empty words - you get a fully functional copy of your environment. This includes all data, database replication and even traffic mirroring between environments. This way, you’ll be independent, save your precious time, avoid frustration and introduce fewer bugs to your project. Having a chance to test new code on a production-like environment before going to production is invaluable.
‍
All of this happens automatically, without you having to configure anything at all. Every application, database or broker has replicated automagically. This is the superpower brought to you by Qovery, that other platforms do not provide.

Docker, Dynos & language support

Another important difference is the runtime of applications and language support. On Heroku, the number of supported languages and frameworks is limited. You can use languages officially supported by Heroku in containers called Dynos. If you need to use something else, you can provide a Docker image. However, the support for Docker containers on Heroku is somewhat troublesome. Using Docker on Heroku requires a lot more effort and configuration than on Qovery. Besides this, Docker containers on Heroku are affected by the same limitations as Dynos, as in fact they're run in Dynos.

At Qovery, we have taken a different, opinionated approach of running your applications. Docker integration with Qovery is effortless - all you have to do is to create a Dockerfile in your repository. Qovery takes care of the rest - building your Docker image, deploying it and integrating with your Git workflow.

This way, we support all languages and frameworks you can imagine with minimal effort for you as a developer. All your applications are built and deployed in the same way, no matter what technology or language you use.

Different approach to open source and side projects

At Qovery, we highly value passionate developers and open-source. We provide a generous free tier, in which you can deploy up to three applications (e.g. frontend application, backend application, and a database) and set it up with your custom domain.
‍
Unlike Heroku, we never turn your applications off, even on a free tier! If you’re an author of an open-source project, please contact us - we are glad to give you up to 100% discounts for running your open source projects.

Join the community now

6 benefits of Kotlin for building server-side applications

Romaric P. — Thu, 09 Apr 2020 18:52:15 +0000

Kotlin is a programming language over the JVM (like Java). It is well known for being the official programming language for Android. But Kotlin is a reliable and powerful programming language that can also be used for server-side applications.

I have been using Kotlin for 4 years to develop server-side applications. Here are the 6 reasons why it is a great choice for building your next backend:

Java Ecosystem

Java is one of the vastest ecosystems in the programming world. Kotlin is 100% interoperable with Java, which allows it to benefit from all its libraries/frameworks without having to recreate new ones (which was difficult with Scala).

Interoperability is what made its great success on Android, the interface with existing APIs is painless. It is the same observation on the backend side. Would you like to use a framework like Spring Boot in Kotlin? Not a problem, it's even officially supported.

Then, Kotlin starts with the advantages of thousands of existing Java libraries, frameworks and potential dev adopters.

Concise

There's no need to argue the point, it makes sense. You just have to see the declaration of a class in Kotlin vs Java

Kotlin example

data class User(val firstName: String? = null, val lastName: String? = null)

Java example

class User {

    private String firstName;
    private String lastName;

    public User() {
    }

    public User(String firstName, String lastName) {
        this.firstName = firstName;
        this.lastName = lastName;
    }

    public String getFirstName() {
        return firstName;
    }

    public void setFirstName(String firstName) {
        this.firstName = firstName;
    }

    public String getLastName() {
        return lastName;
    }

    public void setLastName(String lastName) {
        this.lastName = lastName;
    }

    @Override
    public boolean equals(Object o) {
        if (this == o) return true;
        if (o == null || getClass() != o.getClass()) return false;
        User user = (User) o;
        return Objects.equals(firstName, user.firstName) &&
                Objects.equals(lastName, user.lastName);
    }

    @Override
    public int hashCode() {
        return Objects.hash(firstName, lastName);
    }

    @Override
    public String toString() {
        return "User{" +
                "firstName='" + firstName + '\'' +
                ", lastName='" + lastName + '\'' +
                '}';
    }

}

In these two examples, the bytecode will be the same. Except that in Kotlin the compiler does much more than in Java. This improves code maintainability and readability, meaning engineers can write, read, and change code more effectively and efficiently. Features such as type inference, smart casts, data classes, and properties help achieve conciseness.

Note: Java 14 includes the concept of Record - Which is more or less the concept of data class in Kotlin. Let's keep an eye out to see how it evolves over the next few years.

Null Safety

Kotlin brings the "optional" concept used in functional languages with the "?" operator. It prevents crashes when a value can be potentially null and therefore takes into account the case where this would be the case.

When designing a RESTful API we can receive null data, if the case has not been handled, then it is a possible crash. These errors are greatly reduced with Kotlin. Goodbye NullPointerException (the billion dollar mistake)

See the null safety page

Data Transformation

The functional programming concepts are present natively in Kotlin. This makes it possible to chain actions intuitively. Do you need to transform a list of X objects into Y objects? It' s as simple as...

listOf(X(1), x(2), X(3)).map { x -> x.toY() }

Sometimes I can chain up to 10 operations without having to create a single intermediate variable. When you get a taste for it, it's tough to live without it.

Reliability and performance

Kotlin is Open Source and was initially developed by Jetbrains (the company behind the IntelliJ IDE). Its success is due to its performance and reliability close to what you would have in Java. The final code is also ByteCode and contrary to Groovy which had a very bad reputation for performance, here no magic, no "dynamic invoke" in all directions.

Growing community

The community around Kotlin keeps growing month over month. Companies such as Kodein Koders and Jetbrains are doing their best to animate it and make Kotlin one of the top 5 programming languages used by developers in the coming years.

How to build and deploy realtime GraphQL APIs in a few minutes

Patryk Jeziorowski — Thu, 09 Apr 2020 09:18:50 +0000

Before we dive deep into details, I'll quickly describe the tools I used to build and deploy a realtime GraphQL API and tell you why you should fall in love with GraphQL and all the tools I used.

First, why to use GraphQL?

GraphQL is a query language for APIs and a runtime for fulfilling those queries with existing data. GraphQL provides a schema that describes the API and allows clients (e.g. your frontend or mobile application) to easily fetch data they want and nothing more.

Here is what you get from using GraphQL instead of standard RESTful APIs:

GraphQL queries get exactly what you need, nothing more and nothing less
Instead of making multiple requests to fetch required data, you make just one request to one endpoint
GraphQL schema is typed, what makes the contract between frontend and backend clear and understandable

If you are a frontend engineer, you will not like to consume other APIs than GraphQL after trying it out. It makes your life so much more pleasurable and easy.

You don't need to know GraphQL to follow this article. All you need to know is that GraphQL allows you to define contract between frontend and backend and do operations on the data you are interested in.

Productivity boosting tools

Hasura is an open source engine that connects to your databases & microservices and auto-generates a production-ready GraphQL backend. By using Hasura in conjunction with Qovery (platform that combines the power of Kubernetes, the reliability of AWS and the simplicity of Heroku to allow developers deploy their apps with pleasure), you get a blazing fast, auto-scallable and extensible solution to quickly build your applications.

Why Hasura?

Consuming GraphQL APIs is a pleasure. We'd like to have more GraphQL APIs. But those APIs do not come out of nowhere. Somebody has to implement them first - the data won't just flow out of the database through the schema to your frontend automagically, right? Well... with Hasura it will!

Hasura allows you to bootstrap a realtime GraphQL API in seconds by simply modeling your data. Hasura will do the hard work of translating your needs into queries to the database and translating them to GraphQL schema. Thanks to this, all you need to do is to define the data you want to store in the database - Hasura will do the rest.

This is unbelievable how much time it saves. If you don't believe, try implementing a GraphQL server yourself - with all the featuers and options that Hasura offers.

If you have doubts about flexibility - you don't have to worry. If you need to perform a very custom business logic, you can implement this part in any language you want and connect it to Hasura engine. This way you will not only save a lot of time, but also have flexibility to write your custom code if needed.

Why Qovery?

Managing infrastructure is hard and takes time. Developers want to focus on building their apps instead of wasting time on managing servers or databases. Qovery is tool that does it all for you - all you have to do is to write your application code. It's *powered by Docker and Kubernetes * underneath, so you get all the benefits of using those modern tools without the complexity and costs of learning and managing them.

Qovery is also a great fit for Hasura - its free plan allows you to deploy Hasura and database for free, without any limits, performance degradations or putting your app to sleep like it's done on other platforms.

If you have any questions regarding this post or other things, feel free to reach me on Discord.

Hasura deployment on Qovery

Deploying Hasura on Qovery is really easy. All you have to do is to bootstrap a project using Qovery CLI in a Git repository & export environment variables required by Hasura.

1/ Bootstrap a project with Qovery CLI (the script will ask you for project and application name, which you can choose as you like)

qovery template hasura

2/ Point Hasura to your Postgres database and enable Hasura Console using environment variables

qovery application env add HASURA_GRAPHQL_DATABASE_URL '$QOVERY_DATABASE_MY_POSTGRESQL_DATABASE_CONNECTION_URI'
qovery application env add HASURA_GRAPHQL_ENABLE_CONSOLE true

3/ Commit and push your changes

git add .
git commit -m "Deploy Hasura on Qovery"
git push -u origin master

Well done! After pushing changes, Postgres and Hasura deployment will start. You can use ‍qovery status --watch to track the progress. Once the deployment is done, you’ll see your Hasura application URL in the status:

Creating realtime GraphQL APIs

Navigate to your Hasura application URL and choose Data tab in the console:

In this section we'll configure our data model. Now, click on the Create Table button.

You’ll see the table creator. We are going to create a simple "Todo" items table. We'll name it "todos" and the table will contain three columns:

1. id - unique identifier of given "Todo" item
2. title
3. description - optional description of "Todo" item

Fill the form as in the screenshots below to prepare the table.

At the end, we should specify our id column as a Primary Key.‍

The table is ready to be created. Click Add Table button at the bottom of the page.

Voila! The table has been created in Postgres and Hasura has exposed GraphQL APIs to interact with our data.

Testing GraphQL APIs

To test the GraphQL API, navigate to the GraphiQL tab and run the following query:

mutation query {
  todos {
    id
    title
    description
  }
}

As you can see, Hasura returned an empty array of "Todo" items. Let’s add a "Todo" item by executing the following query:

mutation {
  insert_todos(objects:[
    {
      title: "My first TODO"
      description: "It's very important TODO item"
    }
  ]) {
    affected_rows
  }
}

After you run the above query, in the response you'll see information about one affected row. Congrats! You have created a first "Todo" item. Let's now move further to a more interesting topic.

GraphQL realtime APIs

It's time to use a realtime GraphQL APIs - GraphQL Subscriptions. Subscription allows you to fetch data and get updates about any changes that occur in data you are interested in.

In the GraphiQL, run the following query:

subscription {
  todos {
    id
    title
    description
  }
}

In the response in the right-hand of console you'll see a "Todo" item you have created previously. That's great. Let's now test if the subscription really works - open one more Hasura console in a separate browser tab and navigate to GraphiQL.

Execute the following query a few times:

mutation {
  insert_todos(objects:[
    {
      title: "Another TODO to test subscriptions"
      description: "Subscriptions!"
    }
  ]) {
    affected_rows
  }
}

At the same time, keep an eye at the subscription. Each and every newly created "Todo" item automagically appears in the subscription response!

Conclusions

By following this article you quickly deployed a realtime GraphQL backend using Qovery, Hasura and Postgres database.

Using this stack saves you tons of time. Deploying it on Qovery is extremely easy. We take care of your application and your database. With Qovery and Hasura all you have to do to expose quality, realtime GraphQL backend is just a few clicks. After minutes your application is ready - define your data schema and expose GraphQL API to the world!

How to choose the right database?

Romaric P. — Tue, 07 Apr 2020 23:44:13 +0000

Hello the community,

I'm hosting an online event called "As a developer, how to choose the right database?" - From my experience as an SRE and backend developer, I've noticed that choosing a database is not necessarily easy for everyone. With my colleague Pierre (CTO @ Qovery, ex SRE Criteo & Red Hat), we propose a small webinar to give you an overview of what exists in SQL and NoSQL databases. We will cover MongoDB, PostgreSQL, Cassandra, Elasticsearch, Redis and others... If you are interested in participating you can register here

Free hosting for Open Source projects

Romaric P. — Tue, 31 Mar 2020 10:48:13 +0000

Qovery combines the power of Kubernetes, the reliability of AWS and the simplicity of Heroku to deploy modern application seamlessly!

I am pleased to announce that Qovery supports individual developers / open source and non profit projects to host up to 3 applications (database included) for free. Our business model is based on Enterprise hosting, which gives us the possibility to offer generous free plans.

Start using Qovery now and join the community!

Why we moved from Slack to Discord?

Romaric P. — Wed, 18 Mar 2020 16:08:45 +0000

Why we moved from Slack to Discord?

Today, at Qovery we have moved from Slack to Discord. We are a software company solving developer problems - application deployment. Solving developer problems require to have a close contact with the developers community. For team communication, we used a dedicated Slack workspace, for community communication - another one. Discord has been a real lifesaver for us. Let's start to tell you why..

Slack limitations

Slack was created to meet the needs of companies. It's a wonderful communication tool that has successfully found its place in companies at all stages. From R&D to marketing to finance, Slack is omnipresent. However, like all tools, it is not perfect. In teamwork it is perfectible, and in community management it is totally non-existent.

Here are the 3 limitations that pushed us to look for something else (from the most important to the least-important):

The 10,000 message limit - you have to pay per user to avoid losing your message history, it's truly restrictive! You have to pay per user on your Slack to make it disappear. Do the math, for a community of 1000+ developers it's pretty expensive.
We are a "remote first" company - we need to be able to talk to each other quickly and to be able to share our screen very often, with slack it's a real pain in...
There is no member hierarchy - It's impossible to know who's in the staff, who's an active member, who's a beta tester. It's hard to know who is who and who does what. How do you give visibility to new members? It's simply not possible.

After testing several solutions such as Tandem, Zoom, Google Meet, Slack, each solution was able to satisfy one or two points, but never all of them... Until we found Discord.

Discord

Built for online community

Discord is a communication tool that has been created for online gamers. Online gamers have high expectations. Written and voice communication is ubiquitous and must be flawless for them. It is in this context that Discord was made. The funny part is, it is also perfectly suited for team communication in a company. And even more so when this company targets developers, which is essentially a world of enthusiasts. The community can meet and talk on Discord in a very natural way. The experience is intuitive.

Advantages over Slack

Here is the advantages of using Discord:

No more 10,000 message limit - you don't even have to pay, Discord won't limit you. All the features are accessible without paying 1 cent. It's quite incredible considering the quality of the product.
Video and Voice conversations - Discord is great at supporting video and audio communication. There are no limits on this side either. Want to organize a quick meeting? Just go to a voice channel and it's done.
Hierarchy - In a company, as in a community, not everyone has the same status. Discord has perfectly understood this and materializes it through the role system. Each member can have zero to many roles (R&D Engineer, Marketing, Finance, Founder, Contributor...). Each channel will then be accessible or not according to its roles. This is very practical to make only some channels accessible to the community and the rest to the staff.

Discord sets the standard very high! The tool was designed with the idea of improving teamwork - obviously on the gaming side. But this is perfectly transposable to the needs of companies. In fact, today online games are taken very seriously and are considered a real business activity. Pricing is another very important point, let's talk about it...

Interesting pricing model

At Qovery, we have no problem paying for the tools we use. The problem is that we can't afford a tool that charges us a per-user license fee! This is where Discord stands out from its competitors by offering pricing to the global "server" rather than to the user. By the way, if you don't want to do HD voice communication no need to pay. It's a very virtuous and collaborative pricing model. Because your users can help you pay the bill at their convenience. In our case we pay $74.85/month for +150 developers. And even if tomorrow we're at 100,000 developers it won't change anything on the bill! Incredible isn't it?

Missing features from Slack :'(

Telling you that being in Discord is heaven on earth would be a lie. It lacks some important features to make Discord the uncontested leader :)

Integrations: Discord was created for gamers, which means you won't find all the integrations you have for Slack. That's really the big negative! However, you can manage your integrations yourself thanks to webhooks. Thanks to this we were able to (almost) upload all the integrations we had on Slack (Stripe, Intercom, Mailchimp, Typeform and Gitlab) - few of them are still in progress.
Thread messages: Discord does not allow threaded chat, even if the community requests access to this feature, there are no plans at this time. On the other hand, there is the possibility to "quote" a message.
Gaming oriented: As I said above, Discord was made for gamers. Which means that sometimes certain usage options are not appropriate. Nothing too serious, but it can be confusing if you're planning to get people from other departments than R&D in there.

What's next?

At Qovery we are really pleased with this change, especially in these bad times due to the Covid-19. Discord is for us very appropriate for our default remote office needs. On the community side we lack sufficient hindsight to tell you this, but the first feedback are positive. We will keep you informed of how the product evolves. We secretly hope that Discord will take into account that their product can be an alternative to Slack for business.

Join us now on Discord 🚀

Create a URL shortener API with Kotlin, the micro-framework Ktor and PostgreSQL

Romaric P. — Sun, 15 Mar 2020 11:29:24 +0000

The source code for this post can be found on this github repo

Introduction

Ktor is a brand new micro-framework created by the Jetbrains team and running over the JVM. Jetbrains are the authors of Kotlin - which is now the official programming language for Android, and one of the most popular programming language on the JVM. Kotlin is gaining popularity on server-side and multi-platform application development.

Ktor is a framework for building asynchronous servers and clients in connected systems using the powerful Kotlin programming language.

In this article you will learn:

How to design a simple URL shortener.
How to use the Ktor micro-framework with Kotlin
How to deploy a Ktor application

I personally have +4 years of experience using Spring and I wanted to give a try to Ktor which seems promising. Creating a URL shortener is a good way to start.

What is a URL shortener?

A URL shortener is a simple tool that takes a long URL and turns it into a very short one

It is commonly used for 3 reasons:

Tracking clicks
Make URL much more concise.
Hide original URL

One famous freemium provider is bit.ly (see here)

In this article we will make a basic bit.ly like URL shortener. Let’s go

Ktor principles

Before starting I want to introduce the 3 main principles of Ktor.

Kotlin

Kotlin is the language used to develop on Ktor. It is an object-oriented and functional language. It is very stable and runs on the JVM. Kotlin is 100% interoperable with Java and allows you to benefit from its ecosystem (libraries, build system, etc.).

Functional programming

Ktor leverages the power of Kotlin and has a very functional approach. When writing code, everything seems obvious. It's very similar to what you can see on NodeJS. For me, coming from the Spring world, I find it very efficient to read and use.

Asynchronous

Kotlin provides asynchronous code execution, thanks to coroutines. Ktor exploits this feature to its full potential, and even if you have the impression that you are writing code in a blocking manner, this is not the case. Ktor makes your life easier.

HTTP Server

Here is a complete and simple example of how to expose an HTTP server (http://localhost:8080) with Ktor.

fun main(args: Array<String>): Unit = io.ktor.server.netty.EngineMain.main(args)

@kotlin.jvm.JvmOverloads
fun Application.module(testing: Boolean = false) {
    routing {
        get("/") {
            call.respondText("Hello World", contentType = ContentType.Text.Plain)
        }
    }
}

URL Encoder

The URL encoder will translate an incoming address into a smaller URL. The idea is to provide an ID that will identify the final URL. Using a hash function is perfect for this operation. However, the operation is non-reversible, meaning you can’t retrieve the final URL by the generated identifier.

Function to transform a long URL into a shorter URL

// String extension
fun String.encodeToID(): String {
    // hash String with MD5
    val hashBytes = MessageDigest.getInstance("MD5").digest(this.toByteArray(Charsets.UTF_8))
    // transform to human readable MD5 String
    val hashString = String.format("%032x", BigInteger(1, hashBytes))
    // truncate MD5 String
    val truncatedHashString = hashString.take(6)
    // return id
    return truncatedHashString
}

We expose the function through the REST API

// Request object
data class Request(val url: String) {
    fun toResponse(): Response = Response(url, url.encodeToID())
}

// Response object
data class Response(val originalURL: String, private val id: String) {
    val shortURL: String = "http://localhost:8080/$id"
}

@kotlin.jvm.JvmOverloads
fun Application.module(testing: Boolean = false) {
    install(ContentNegotiation) {
        jackson {
            enable(SerializationFeature.INDENT_OUTPUT)
            propertyNamingStrategy = PropertyNamingStrategy.SNAKE_CASE
        }
    }

    // Hash Table Response object by ID
    val responseByID = mutableMapOf<String, Response>()

    routing {
        post("/api/v1/encode") {
            // Deserialize JSON body to Request object
            val request = call.receive<Request>()

            // find the Response object if it already exists
            val retrievedResponse = responseByID[request.url.encodeToID()]
            if (retrievedResponse != null) {
                // cache hit
                log.debug("cache hit $retrievedResponse")
                return@post call.respond(retrievedResponse)
            }

            // cache miss
            val response = request.toResponse()
            responseByID[request.url.encodeToID()] = response
            log.debug("cache miss $response")

            // Serialize Response object to JSON body
            call.respond(response)
        }
    }
}

Handle identifier collision

Using a hash function makes no guarantee that it is not already being used. If it is in use, then you need to change it to another one. Note: even if the probability to have a collision is very low, you should handle this case.

// String extension (function signature has changed)
fun String.encodeToID(truncateLength: Int = 6): String {
    // hash String with MD5
    val hashBytes = MessageDigest.getInstance("MD5").digest(this.toByteArray(Charsets.UTF_8))
    // transform to human readable MD5 String
    val hashString = String.format("%032x", BigInteger(1, hashBytes))
    // truncate MD5 String
    val truncatedHashString = hashString.take(truncateLength)
    // return id
    return truncatedHashString
}

//...

@kotlin.jvm.JvmOverloads
fun Application.module(testing: Boolean = false) {
        // ...
    // Hash Table Response object by id
    val responseByID = mutableMapOf<String, Response>()

    fun getIdentifier(url: String, truncateLength: Int = 6): String {
        val id = url.encodeToID()

        val retrievedResponse = responseByID[id]
        if (retrievedResponse?.originalURL != url) {
            // collision spotted !
            return getIdentifier(url, truncateLength + 1)
        }

        return id
    }

    routing {
        post("/api/v1/encode") {
            // Deserialize JSON body to Request object
            val request = call.receive<Request>()

            // find the Response object if it already exists
            val id = getID(request.url)
            val retrievedResponse = responseByID[id]
            if (retrievedResponse != null) {
                // cache hit
                log.debug("cache hit $retrievedResponse")
                return@post call.respond(retrievedResponse)
            }

            // cache miss
            val response = request.toResponse()
            responseByID[id] = response
            log.debug("cache miss $response")

            // Serialize Response object to JSON body
            call.respond(response)
        }
    }
}

URL Decoder

Decoding the URL is the process of returning the original URL from the short URL. This is the reverse operation made by the URL Encoder

val shortURL = getShortURL(request.url)
val retrievedResponse = responseByID[shortURL]
retrievedResponse?.originalURL // return original URL or null

Redirect

When a user clicks on a short URL, the user is redirected to the final URL. HTTP protocol allows to do this naturally by returning a 302 status code and a redirection URL.

With Ktor the redirection is as simple as calling a method with the final URL as a parameter.

call.respondRedirect("https://www.qovery.com")

What we expect is that when the user visits http://localhost:8080/fbc951 he is redirected to https://www.qovery.com. If the URL is incorrect then redirect to https://www.google.com

@kotlin.jvm.JvmOverloads
fun Application.module(testing: Boolean = false) {
        // ...
    routing {
        get("/{id}") {
            val id = call.parameters["id"]
            val retrievedResponse = id?.let { responseByID[it] }

            if (id.isNullOrBlank() || retrievedResponse == null) {
                return@get call.respondRedirect("https://www.google.com")
            }

            log.debug("redirect to: $retrievedResponse")
            call.respondRedirect(retrievedResponse.originalURL)
        }
        // ...
    }
}

Stats: clicks over time

Something that is really useful on products like bit.ly is the stats provided (click over time, referrers, country of visitors). Here is how to store click over time and make them available through the API

// added
data class Stat(val clicksOverTime: MutableList<Date> = mutableListOf())

// Response object (modified with Stat)
data class Response(val originalURL: String, private val id: String, val stat: Stat = Stat()) {
    val shortURL: String = "http://localhost:8080/$id"
}

@kotlin.jvm.JvmOverloads
fun Application.module(testing: Boolean = false) {
        install(ContentNegotiation) {
        jackson {
            // ...
            // add this line to return Date object as ISO8601 format
            disable(SerializationFeature.WRITE_DATES_AS_TIMESTAMPS)
        }
    }
        // ...
    routing {
            // ...
        get("/api/v1/url/{id}/stat") {
            val id = call.parameters["id"]
            val retrievedResponse = id?.let { responseByID[it] }

            if (id.isNullOrBlank() || retrievedResponse == null) {
                return@get call.respond(HttpStatusCode.NoContent)
            }

            call.respond(retrievedResponse.stat)
        }
        // ...
    }
}

Try the API

Run the application

$ ./gradlew run
//...
2020-03-12 09:28:08.150 [main] INFO  Application - No ktor.deployment.watch patterns specified, automatic reload is not active
2020-03-12 09:28:08.606 [main] INFO  Application - Responding at http://0.0.0.0:8080

Then execute the commands

# generate a short URL
$ curl -X POST -d '{"url": "https://www.qovery.com"}' -H "Content-type: application/json" "http://localhost:8080/api/v1/encode"
{
  "original_url": "https://www.qovery.com",
  "stat": {
    "clicks_over_time": []
  },
  "short_url": "http://localhost:8080/fbc951"
}

# generate 4 fake clicks
$ curl -X GET 'http://localhost:8080/fbc951'
$ curl -X GET 'http://localhost:8080/fbc951'
$ curl -X GET 'http://localhost:8080/fbc951'
$ curl -X GET 'http://localhost:8080/fbc951'

# show stat
$ curl -X GET 'http://localhost:8080/api/v1/url/fbc951/stat'
{
  "clicks_over_time": [
    "2020-03-11T21:10:52.354+0000",
    "2020-03-11T21:10:54.093+0000",
    "2020-03-11T21:12:34.987+0000",
    "2020-03-11T21:12:37.223+0000"
  ]
}

Connect to a PostgreSQL database with Exposed

By storing the data in memory, we lose all the data every time the application restart. Which is problematic for running in production. To make the data persistent we will store it in a PostgreSQL database. We will have to add 1 new dependency - Exposed. Exposed (with Hikari Connection Pool) is a lightweight SQL library on top of JDBC driver for Kotlin. With exposed it is possible to access databases in two flavours: typesafe SQL wrapping DSL and lightweight Data Access Objects (DAO).

Add the dependencies to your build.gradle (or POM.xml)

repositories {
  jcenter()
}

dependencies {
  // Connection Pool and PostgreSQL driver
  implementation("com.zaxxer:HikariCP:3.4.2")
  implementation("org.postgresql:postgresql:42.2.11")

  // Exposed
  implementation("org.jetbrains.exposed:exposed-core:0.22.1")
  implementation("org.jetbrains.exposed:exposed-dao:0.22.1")
  implementation("org.jetbrains.exposed:exposed-jdbc:0.22.1")
  implementation("org.jetbrains.exposed:exposed-java-time:0.22.1")
}

We need to have 2 distincts tables, one containing all the final URLs with their correspond identifier

object ResponseTable : Table("response") {
    val id = varchar("id", 32)
    val originalURL = varchar("original_url", 2048)
    override val primaryKey: PrimaryKey = PrimaryKey(id)
}

And a second one with all the clicking points

object ClickOverTimeTable : Table("click_over_time") {
    val id = integer("id").autoIncrement()
    val clickDate = datetime("click_date")
    val response = reference("response_id", onDelete = ReferenceOption.CASCADE, refColumn = ResponseTable.id)
    override val primaryKey: PrimaryKey = PrimaryKey(id)
}

We need to create the tables as defined above programmatically

fun initDatabase() {
    val config = HikariConfig().apply {
        jdbcUrl = "jdbc:postgresql://127.0.0.1:5432/exposed"
        username = "exposed"
        password = "exposed"
        driverClassName = "org.postgresql.Driver"
    }

    Database.connect(HikariDataSource(config))

    transaction {
        // create tables if they do not exist
        SchemaUtils.createMissingTablesAndColumns(RequestTable, ClickOverTimeTable)
    }
}

@kotlin.jvm.JvmOverloads
fun Application.module(testing: Boolean = false) {
    initDatabase()
    // ...
}

We have to replace the Hash Table used to store the data by the PostgreSQL database (see the final code here)

Deploy in the Cloud with Qovery

Qovery is going to help us to deploy the final application in the Cloud without the need to configure the CI/CD, network, security, load balancing, database and all the DevOps tasks

Qovery is a Container as a Service platform for developer - developers can deploy their application in the Cloud in just a few seconds

Pre-requisites:

Have an account on Qovery (register here)
Your code need to be hosted on Github
You need to accept the Qovery Github app
Package your Ktor application to build and run it on Docker

To deploy on Qovery 2 files are mandatories

.qovery.yml - a very simple way to declare the dependencies that you need (e.g: PostgreSQL) and where you want to run it (here on AWS and eu-west-3 / Paris)

application:
  name: api
  project: url-shortener
  cloud_region: aws/eu-west-3
  publicly_accessible: true
databases:
  - type: postgresql
    version: "11.5"
    name: my-pql-db
routers:
  - name: main
    routes:
      - application_name: api
        paths:
          - /*

Dockerfile - to build and run your application on Qovery

FROM openjdk:8-jre-alpine

ENV APPLICATION_USER ktor
RUN adduser -D -g '' $APPLICATION_USER

RUN mkdir /app
RUN chown -R $APPLICATION_USER /app

USER $APPLICATION_USER

COPY ./build/libs/ktor-url-shortener.jar /app/ktor-url-shortener.jar
WORKDIR /app

CMD ["java", "-server", "-XX:+UnlockExperimentalVMOptions", "-XX:+UseCGroupMemoryLimitForHeap", "-XX:InitialRAMFraction=2", "-XX:MinRAMFraction=2", "-XX:MaxRAMFraction=2", "-XX:+UseG1GC", "-XX:MaxGCPauseMillis=100", "-XX:+UseStringDeduplication", "-jar", "ktor-url-shortener.jar"]

Connect to PostgreSQL

Qovery add dynamically all required environment variables to connect to the database at the runtime of the container.

To list all of them

$ qovery application env list
  SCOPE    | KEY                                                          | VALUE
  BUILT_IN | QOVERY_JSON_B64                                              | <base64>
  BUILT_IN | QOVERY_BRANCH_NAME                                           | with_postgresql
  BUILT_IN | QOVERY_IS_PRODUCTION                                         | false
  BUILT_IN | QOVERY_DATABASE_MY_PQL_DB_NAME                               | my-pql-db
  BUILT_IN | QOVERY_DATABASE_MY_PQL_DB_TYPE                               | POSTGRESQL
  BUILT_IN | QOVERY_DATABASE_MY_PQL_DB_VERSION                            | 11.5
  BUILT_IN | QOVERY_DATABASE_MY_PQL_DB_CONNECTION_URI                     | <hidden>
  BUILT_IN | QOVERY_DATABASE_MY_PQL_DB_CONNECTION_URI_WITHOUT_CREDENTIALS | <hidden>
  BUILT_IN | QOVERY_DATABASE_MY_PQL_DB_HOST                               | <hidden>
  BUILT_IN | QOVERY_DATABASE_MY_PQL_DB_FQDN                               | <hidden>
  BUILT_IN | QOVERY_DATABASE_MY_PQL_DB_PORT                               | <hidden>
  BUILT_IN | QOVERY_DATABASE_MY_PQL_DB_USERNAME                           | <hidden>
  BUILT_IN | QOVERY_DATABASE_MY_PQL_DB_PASSWORD                           | <hidden>
  BUILT_IN | QOVERY_DATABASE_MY_PQL_DB_DATABASE                           | postgres
  BUILT_IN | QOVERY_APPLICATION_API_HOSTNAME                              | <hidden>
  BUILT_IN | QOVERY_APPLICATION_API_HOST                                  | <hidden>

To use them

fun initDatabase() {
    val config = HikariConfig().apply {
        jdbcUrl = "jdbc:${System.getenv("QOVERY_DATABASE_MY_PQL_DB_CONNECTION_URI_WITHOUT_CREDENTIALS")}"
        username = System.getenv("QOVERY_DATABASE_MY_PQL_DB_USERNAME")
        password = System.getenv("QOVERY_DATABASE_MY_PQL_DB_PASSWORD")
        driverClassName = "org.postgresql.Driver"
    }

    Database.connect(HikariDataSource(config))

    transaction {
        // create tables if they do not exist
        SchemaUtils.createMissingTablesAndColumns(RequestTable, ClickOverTimeTable)
    }
}

Deploy

Deploying your app with Qovery is as simple as commit and push your code.

$ git add .qovery.yml Dockerfile
$ git commit -m "add .qovery.yml and Dockerfile files"
$ git push -u origin master

To get the public URL

$ qovery status
  BRANCH NAME     | STATUS  | ENDPOINTS                                   | APPLICATIONS | DATABASES | BROKERS | STORAGE
  with_postgresql | running | https://qavcgggy6g6dlkbj-main-gtw.qovery.io | 1            | 1         | 0       | 0

  APPLICATION NAME | STATUS  | ENDPOINT                                                | DATABASES | BROKERS | STORAGE
  api              | running | https://ete6bq97amj9n82c-qavcgggy6g6dlkbj-app.qovery.io | 1         | 0       | 0

  DATABASE NAME | STATUS  | TYPE       | VERSION | ENDPOINT | PORT     | USERNAME | PASSWORD | APPLICATIONS
  my-pql-db     | running | POSTGRESQL | 11.5    | <hidden> | <hidden> | <hidden> | <hidden> | api

Conclusion

We have seen that creating an URL shortener API with Ktor and Kotlin is extremely simple. The connection of the application with the PostgreSQL database is done in a very easy way with the Exposed library. In just a few lines of code the service is fully functional, and can be deployed in production very quickly with the help of Qovery. In the next part we will see how to create a web interface connecting to this API to convert our URLs without using the curl command.

Part 2: bind a web interface to the API - [link coming soon]

3 ways of cloning an application and a database per git branch

Romaric P. — Thu, 27 Feb 2020 22:52:21 +0000

Back in the early days of software development, having multiple developers working on the same application was a tough challenge. That’s why VCS (Version Control System) like Git was created and methodology like Feature Branching was introduced.

1 feature = 1 branch

The basic idea of working per git branch (also known as Feature Branching) is that when you start to work on a feature, you take a branch of your repository (e.g: git) to work on that feature.

The advantage of feature branching is that each developer can work on their own feature and be isolated from changes going on elsewhere. This concept has been initiated for stateless application. However, most applications rely on databases (also known as “stateful application”).

The question is: How to use the concept of Feature Branch with a stateful application?

Let’s take a concrete example:

We have a NodeJS application that connects to a PostgreSQL database, for which we have 3 distinct branches: master, staging and feature_1.

No matter which branch we are working on, we are always connected to the same database. If our application writes, modifies or deletes data in the database, while we are on the "feature_1" branch, all branches will also be impacted by these changes. Meaning, we violate the main Feature Branch principle - isolation. There is nothing more stressful than having to keep in mind that we can lose data and break everything at any time.

so -

How to be compliant to the feature branch principle (isolation) with a database?

One possible solution is to have one copy of the database per branch. Each database can be modified without the risk of modifying the others.

How to have one database per branch?

3 main choices are possible (manual, partially automated and fully automated) with all their advantages and disadvantages.

Manual

Manual means that you need to install all your required services (DNS, databases, VPC...) manually. This approach is fast enough to bootstrap, but it is hard to maintain over time.

Advantages

Fast to bootstrap

Disadvantages

You need to configure system and network services (DNS, network, security...)
You need to manually create the databases according to the number of branches
You need to manually synchronize the data between the databases
You need to setup observability, monitoring and alerting
Hard to maintain over time
Error prone

Partially automated

Partially automated means that you will spend time to set up a complete system that is provisioning your required services for your project. This type of architecture required times and effort from experienced DevOps. It's a good choice for large corporations that can support the cost, but most of the time a really bad for smaller one.

Advantages

Automatic system and network configuration + database provisioning with tools like Terraform (Infrastructure as Code)
Automatic data synchronization between databases (with a custom script)
Perfectly fit your need

Disadvantages

You need to manually create the databases according to the number of branches
Required months to fully set up
Expensive maintenance over time (experienced DevOps engineer required)
You need to setup observability, monitoring and alerting

Fully automated (with Qovery)

Fully automated means that all the required resources by the developer will be deployed no matter his needs. With Qovery, all resources are automatically provided and the developer don't even have to change their habits to deploy their application. Feature Branching is supported out of the box.

Qovery gives to any developer the power to clone an application and a database without having to change their habits

Let's take the example of our 3 branches with a NodeJS application and a PostgreSQL database

Here is the commands necessary to have a fully isolated application and database on each branch

$ pwd
~/my-nodejs-project 

# Github, Bitbucket, Gitlab seamless authentication
$ qovery auth
Opening your browser, waiting for your authentication...
Authentication successful!

# Wizard to generate .qovery.yml
$ qovery init

$ git add .qovery.yml
$ git commit -m "add .qovery.yml file"

# Deploy master environment
$ git push -u origin master

# Show master environment information
$ qovery status

# Create branch staging
$ git checkout -b staging
# Deploy staging environment!
$ git push -u origin staging

# Show staging environment information
$ qovery status

# Create branch feature_1
$ git checkout -b feature_1
# Deploy feature_1 environment!
$ git push -u origin feature_1

# Show feature_1 environment information
$ qovery status

Advantages

Accessible to any developer
No set up time
Programming language agnostic
Integrated to git (no other dependencies required)
Compliant to Feature Branching concept

Disadvantages

Know how to create a Dockerfile
Deployment only available on AWS, GCP and Azure
Only integrated to Github, Gitlab and Bitbucket

Conclusion

In this article, we have seen that the purpose of the Feature Branch is to be able to develop a feature without being impacted by changes that can be made to other branches.

However, the Feature Branch concept is difficult to apply when our application needs to access a database. Because each application (coming from several branches) access to the same database. This is contrary to the Feature Branch isolation principle and brings serious data safety problems.

Qovery allows application and database to be seamlessly duplicated from one branch to another. And thus to respect the isolation principles of the Feature Branch.

Try Qovery now!

Useful links: Feature Branching and Continuous Integration from Martin Fowler - Stateless vs Stateful from StackOverflow