DEV Community: The Latino CTO

NestJS API Versioning: Supercharging Our CICD

The Latino CTO — Fri, 23 Feb 2024 10:07:03 +0000

Joining a startup from its inception was thrilling. From architecting frameworks to coding frontends and backends, each day brought new challenges. However, as clients came on board, so did a flood of bugs with every release. Hot-fixes and rollbacks became routine despite rigorous testing.

Reflecting on our practices, a pattern emerged: constant modifications to existing code. This approach was unsustainable for a startup with limited time and resources.

How do we minimise constant modifications?

Instead of endlessly modifying tested code, we focused on adding new features without disrupting the existing framework. This shift in mindset led to streamlined development and sustainable growth.

By decoupling new functionalities from existing code, we minimised disruptions and maximised efficiency. This approach empowered our team to adapt swiftly to client needs.

To tackle our challenges, I proposed a gradual approach: implementing API versioning. This simple change paved the way for safer feature delivery using tools like Feature Flags and A/B testing.

NestJS Versioning

NOTE: You can skip reading and go directly to the code and inspect it here:

https://github.com/RaulCornejo/tutorials/tree/main/nestjs-version-example

Assuming you took a look at the github repository, you will be able to see the app.controller.ts which contains this:

@Controller('/api')
export class AppController {
  constructor(private readonly appService: AppService) {}

  @Get('/greetings')
  getHello(): string {
    return this.appService.getHello();
  }
}

You can start the project by running

yarn start:dev
or
npm start:dev

You'll see something like this:

[10:36:08 PM] Starting compilation in watch mode...

[10:36:14 PM] Found 0 errors. Watching for file changes.

[Nest] 24054  - 02/23/2024, 10:36:15 PM     LOG [NestFactory] Starting Nest application...
[Nest] 24054  - 02/23/2024, 10:36:15 PM     LOG [InstanceLoader] AppModule dependencies initialized +16ms
[Nest] 24054  - 02/23/2024, 10:36:15 PM     LOG [RoutesResolver] AppController {/api} (version: Neutral): +26ms
[Nest] 24054  - 02/23/2024, 10:36:15 PM     LOG [RouterExplorer] Mapped {/api/greetings, GET} (version: 1,Neutral) route +3ms
[Nest] 24054  - 02/23/2024, 10:36:15 PM     LOG [RouterExplorer] Mapped {/api/greetings, GET} (version: 2) route +0ms
[Nest] 24054  - 02/23/2024, 10:36:15 PM     LOG [NestApplication] Nest application successfully started +4ms

There's an example route called getHello

You can send a get request to it using curl/postman whatever you like:

raul@local:~|⇒  curl http://localhost:3000/api/greetings
Hello World!%

It responds with a Hello World! string.

Great!

Imagine this scenario:

The "greetings" endpoint has been functioning flawlessly for ages. Suddenly, a request arises: the need to return greetings in both English and Spanish.

One of the recurring challenges we faced was the constant need to tweak endpoints and controllers to accommodate new functionalities. This perpetual cycle not only increased the likelihood of introducing bugs but also posed significant challenges despite our rigorous testing and monitoring efforts.

No matter how thorough our manual and automated testing processes were, there always seemed to be an edge case waiting to disrupt our workflow and cause headaches.

Most people, including myself, usually think about versioning APIs by adding a version number to the URI such as

ie: https://example.com/v1/route and https://example.com/v2/route

However, that solution didn't quite fit our needs. It entailed additional configuration steps on our API gateway, load balancers, and beyond. Moreover, such versioning mechanisms seemed better suited for major API changes, rather than the incremental updates we required. What we truly sought was a seamless method to introduce new versions of an endpoint, conduct thorough testing, and gracefully retire the old version once validated.

Interestingly enough, every time I read the Nestjs docs, there's something new that surprises me. This was one of the cases. They have an opinionated way of handling versions and they have multiple options on how to implement versioing:

URI Versionining: The version will be passed within the URI of the request (default)

Header Versioning: A custom request header will specify the version

Media Type Versioning: The Accept header of the request will specify the version

Custom Versioning: Any aspect of the request may be used to specify the version(s). A custom function is provided to extract said version(s).

As we mentioned before, URI versioning wouldn't work for us. All of the other 3 types are quite similar, we decided to use Header Versioning.

import { NestFactory } from '@nestjs/core';
import { AppModule } from './app.module';
import { VERSION_NEUTRAL, VersioningType } from '@nestjs/common';


async function bootstrap() {
  const app = await NestFactory.create(AppModule);

  // enable nestjs versioning
  app.enableVersioning({
    type: VersioningType.HEADER,
    header: 'route-version',
    defaultVersion: [VERSION_NEUTRAL],
  });

  await app.listen(3000);
}
bootstrap();

The code above creates a nest app, and then it activates versioning using enableVersioning.

VersioningType.HEADER and VERSION_NEUTRAL are part of @nestjs/common

header: 'version',

Can have any value you like. call it route-version, v, api-version, etc. We'll talk about this later.

defaultVersion: [VERSION_NEUTRAL],

We configured the defaultVersion to VERSION_NEUTRAL. Essentially, this means that in the absence of a version header in the request, and when multiple versions are available, NestJS will automatically serve the version designated as VERSION_NEUTRAL.

Creating a New Version of getHello()

Our AppController has this endpoint

  @Get('/greetings')
  getHello(): string {
    return this.appService.getHello();
  }

Some might suggest a straightforward solution: just add a parameter to getHello() and adjust appService.getHello() accordingly.

Yes, theoretically, we could do that. However, I was hesitant to tamper with an endpoint that had undergone thorough testing by both our team and our clients. (Not to mention, our real-world scenarios were far more intricate, making any changes significantly more complex.)

What can Nestjs Versioning do for us?

    @Version(['1', VERSION_NEUTRAL])
    @Get('/greetings')
    getHello(): string {
        return this.appService.getHello();
    }

    @Version(['2'])
    @Get('/greetings')
    getHelloV2() {
        // return this.appService.getHelloV2();
    }

As you can see, I added

@version(['1', VERSION_NEUTRAL])

The @arul47 annotation in NestJS informs the framework that the route below it corresponds to version 1. However, the true magic lies in *VERSION_NEUTRAL. When multiple versions are available for the same route and no version is specified in the request, NestJS will serve the version marked with VERSION_NEUTRAL. If the request includes a route-version=1 header, then this specific version will also be served.

Conversely, if the request specifies version 2, then the route annotated with @Version('2') will be served. This flexible approach allows for seamless version management and ensures that the appropriate version is delivered based on the request parameters.

Let's give it a try and call old version 1

raul@local:~|⇒  curl http://localhost:3000/api/greetings -H 'route-version: 1'
Hello World!%

Now version 2

raul@local:~|⇒  curl http://localhost:3000/api/greetings -H 'route-version: 2'
i'm version 2%

However, the critical functionality lies in the fact that our frontend comprises numerous instances where api/greetings is called. We're not yet ready to make modifications on the frontend, or perhaps we prefer to implement changes gradually. Therefore, it's imperative that the old implementation of api/greetings continues to be served by the old version until we're prepared for the transition.

curl http://localhost:3000/api/greetings
Hello World!%

Exactly. Since we haven't included a version header in the request, and our defaultVersion is set to 1 (our NEUTRAL_VERSION), the old version 1 is automatically served.

This functionality is incredibly useful. It means we can confidently modify our service, introducing new or partially new methods to accommodate greetings in additional languages, without disrupting the existing functionality.

Modify /greetings version 2 so it accepts a language parameter

    @Version(['2'])
    @Get('/greetings')
    getHelloV2(@Query('language') language): string {
        return this.appService.getHelloV2(language);
    }

Now, implement the new functionality on appService.getHelloV2

  getHelloV2(language: 'en' | 'es'): string {
    // if language is not provided, default to English (using a case)
    switch (language) {
      case 'es':
        return '¡Hola Mundo!';
      default:
        return 'Hello World!';
    }
  }

Go ahead now and test our new version!

raul@local:~|⇒  curl http://localhost:3000/api/greetings\?language\=es -H 'route-version: 2'
¡Hola Mundo!%

Great! we can now hit route-version: 2 and we pass the ?language=es query parameter to get a greeting in Spanish!

If we also call version 2 with language=en or without a language parameter we should get Hello World

raul@local:~|⇒  curl http://localhost:3000/api/greetings\?language\=en -H 'route-version: 2'
Hello World!%
raul@local:~|⇒  curl http://localhost:3000/api/greetings -H 'route-version: 2'
Hello World!%

And finally, what if we don't specify a version? or if we specify version 1?

raul@local:~|⇒  curl http://localhost:3000/api/greetings
Hello World!%
raul@local:~|⇒  curl http://localhost:3000/api/greetings -H 'route-version: 1'
Hello World!%

Even better, if someone accidentally calls version 1 (with or without headers) and passes a language parameter:

raul@local:~|⇒  curl http://localhost:3000/api/greetings\?language\=en -H 'route-version: 1'
Hello World!%
raul@local:~|⇒  curl http://localhost:3000/api/greetings\?language\=en
Hello World!%

Since both cases get served by version 1, the greeting is still always Hello World!

Summary

Indeed, simplicity often conceals the brilliance of a solution, and this case is no exception. Embracing different versions of an endpoint, coupled with feature flags, has been a game-changer for us.

Transitioning from GitLab-flow to GitHub-flow marked a significant milestone in our development journey. We've overcome challenges associated with prolonged deployment cycles and bottlenecks in our pre-production environment. Previously, we encountered delays due to multiple features awaiting testing and debugging. With versioning and feature flags in place, we've streamlined our workflow, enabling more efficient testing and deployment processes.

Our ability to adapt and learn from our mistakes has propelled us forward. Today, we're operating at a much faster pace, and our releases are more valuable than ever before.

This topic is a testament to the power of continuous improvement, and I look forward to revisiting it in the future.

Do Not Commit Secrets!

The Latino CTO — Fri, 16 Feb 2024 09:04:27 +0000

Picture this: every team I've had the pleasure (or sometimes, the pain) of working with has had one thing in common – someone, somewhere, committing secrets to Git.

In my recent adventures as a Technical Lead, CTO, and all-around code wizard, I've witnessed firsthand the struggles of distinguishing between public and private keys. And you know what's even trickier? Accidentally spilling those private keys and secrets onto Git for all to see. It's like leaving your front door wide open in the middle of the night and wondering why your house got burgled.

I've seen .env files with more secrets than my ex-wife and Axios requests spilling the beans on sensitive credentials.

Now, you might be thinking, "What's the harm? Can't we just commit another version without the credentials?" Ah, if only it were that simple.

Well, let's conduct a little experiment, shall we? Create a local Git repository (assuming Git is already in your system)



raul@mbp:~|⇒  mkdir fake-git-repo
raul@mbp:~|⇒  cd fake-git-repo
raul@mbp:~/fake-git-repo|⇒  git init
Initialized empty Git repository in /Users/raul/fake-git-repo/.git/
raul@mbp:~/fake-git-repo|master ⇒  touch .env

Add some super secret values to the .env file ```

raul@mbp:~/fake-git-repo|master ⇒ vi .env


Add something like this:

AWS_ACCESS_KEY_ID = 'AKJEGCKAQNSGFN2N68''
AWS_SECRET_ACCESS_KEY = 'mC4gRDpo/o+M1uOfTWgdfgqMqYe38eGmt'
~
~
~
:wq


Save the changes by typing `wq` (write and quit)

- Now commit those changes! although, you shouldn't!

raul@mbp:~/fake-git-repo|master⚡ ⇒ git add .
raul@mbp:~/fake-git-repo|master⚡ ⇒ git commit -m "let's commit something we shouldn't commit"
[master (root-commit) 2be5583] let's commit something we shouldn't commit
1 file changed, 2 insertions(+)
create mode 100644 .env


- Now, for the sake of the story, let's assume you follow a Code Review procedure and someone caught you committing these secrets. They told you it's not safe. You immediately said, "OK! I'll remove them," so you go to `vi`, remove those values from the file, and commit again without the secrets.

AWS_ACCESS_KEY_ID =
AWS_SECRET_ACCESS_KEY =
~
~
~
:wq


- Let's use `git` to double-check our changes before commit

diff --git a/.env b/.env
index 90912b1..3c569d6 100644
--- a/.env
+++ b/.env
@@ -1,2 +1,2 @@
-AWS_ACCESS_KEY_ID = 'AKJEGCKAQNSGFN2N68''
-AWS_SECRET_ACCESS_KEY = 'mC4gRDpo/o+M1uOfTWgdfgqMqYe38eGmt'
+AWS_ACCESS_KEY_ID =
+AWS_SECRET_ACCESS_KEY =


If you pay attention, the lines with a `-` are the ones deleted, and the ones with `+` are the added ones. So we're all good! We removed the values of our super secrets! Yay!

Now, commit the safer .env file.

raul@mbp:~/fake-git-repo|master⚡ ⇒ git commit -a -m 'removes secrets'
[master 61a83f4] removes secrets
1 file changed, 2 insertions(+), 2 deletions(-)


If you open the file now, secrets are no longer there

raul@mbp:~/fake-git-repo|master ⇒ cat .env
AWS_ACCESS_KEY_ID =
AWS_SECRET_ACCESS_KEY =


### You're Wrong!

You might believe you're all safe now. But the whole purpose of Git is to keep track of changes and allow us to go back in the history of our files if necessary. Let's investigate the history of our `fake-repository` and `.env` file.

- Use `git log -p -2`.
  - `-p` means patch, and when used along with log, it will show what changed (removed, added, modified).
  - `-2` tells the log to show the latest 2 commits.

commit 61a83f419b174f0750d7dab56c9e0469b0c27784 (HEAD -> master)
Date: Fri Feb 16 20:57:26 2024 +1300

removes secrets

commit 2be5583e6c94c7b70e0f4232fea5ff1fb13603a8
Date: Fri Feb 16 20:46:59 2024 +1300

let's commit something we shouldn't commit


If you take a deep look, `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` appear twice. But one with a `-` and another with a `+`, meaning that the line with `-` was removed and the one with `+` was added.

So, we could see what was the old value (with our secrets) which we thought we safely removed from the public!

Now, you might ask...why do I care so much about this?


### Real-Life Example from GitHub

If the previous example doesn't seem scary, let's head over to `github.com` and see if secrets are being leaked or not, and why we should be worried.

Here's a direct link for you to search for commits on GitHub public repositories:

[https://github.com/search?q=remove+secrets&type=commits](https://github.com/search?q=remove+secrets&type=commits) 

You'll see a lot of repositories with commits named `remove secrets`


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/bxbbspc357f0mlygsrxz.png)

If you click on some of them, you might be able to see files like this:


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/aymqcdm0j28gbko3fuu9.png)

That's actually `google cloud computing secrets! I blurred the values, and I hope this person already rotated this certificate or they could be in serious problems having people accessing their Google Cloud services and probably incurring charges.

Look at this other example where someone tried to remove AWS secrets, MySQL DB connection strings, etc.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/l50wy58jro6rg3yfav7e.png)

## Summary

I hope this helps people understand why there's a Tech Lead, Senior Dev, etc., asking you not to commit secrets.

I've come to realize we usually tell others what not to do without giving a reason of why it shouldn't be done.

Hopefully, this blog will help me too, to teach some of the developers I work with. And if in the meantime, someone else can learn something, well, you're welcome!