DEV Community: Iury Souza

Kotlin Multiplatform under the surface

Iury Souza — Wed, 18 Aug 2021 01:29:54 +0000

The mobile development community has always had many alternatives for creating cross-platform apps. Since the days of webview-based solutions we've come a long way, but we still have many alternatives. The most recent one is the Jetbrains led, Kotlin Multiplatform Mobile project.

If you're curious about how does it stand against the competition and what are its pros and cons, this brief overview may help you get started.

Cross-platform vs Multiplatform

As you may have noticed, Jetbrains calls its solution: Kotlin Multiplatform Mobile. This might get overlooked by a lot of people, me included, up until I decided to write this down. But then it hit me: what's the difference between Cross-platform and Multiplatform?

After some research, I got the answer.
From the user's point of view, this is plain simple. In short: there's no difference.
Multiplatform and Cross-platform implementations achieve the same thing. It simply means that the application will run on - you guessed it. Multiple platforms.

Since we're focusing on mobile dev, Multiplatform means Android and iOS. But this can translate to web, desktop... whatever. Just... multiple platforms.
It doesn't say anything about how the app was created. If you have a code-base for an app written in Kotlin targeting android and another written in swift targeting iOS, you have a multiplatform app.

Now, cross-platform is different. Cross-platform means that the same code you wrote will run on, for example, android and iOS. This is usually done via some kind of runtime that needs to be shipped to the hosting platform that enables that code to be interpreted for that particular environment it is running on.

So what about Kotlin Multiplatform?

The main idea behind Kotlin Multiplatform is code-sharing. If you enter the home page, you'll see them mentioning how you can write the code once and have it running on both platforms. So -you may be asking, isn't that cross-platform... like Flutter or React? What gives?

The main advantage of Kotlin Multiplatform is that it delivers native code for multiple target platforms/architectures by taking the same code as input.
It is able to do this because of the Kotlin/Native Compiler, so to understand how Kotlin Multiplatform works we first need to understand what is Kotlin/Native.

Kotlin/Native

Quoting from Kotlin's landing page:

Kotlin/Native is a technology for compiling Kotlin code to native binaries, which can run without a virtual machine. It is an LLVM based backend for the Kotlin compiler and a native implementation of the Kotlin standard library.

There's a lot to unpack here. Let's take the first sentence in:

So Kotlin/Native will give us native binary executables that can run on an OS using no virtual machine.

A binary is the machine code, ready to be run on the specified target architecture. So it's the machine code for, for example, the iOS targeting a specific CPU architecture, like arm64 (which is the official iPhone architecture name)

Then it goes on to mention an LLVM based backend for the Kotlin compiler. This one is dense.

LLVM

The LLVM Compiler Infrastructure Project is probably the biggest open-source compiler project that exists to build native binaries. Languages such as C, C++, Haskell, Rust & Swift compile into native binaries through LLVM.

Ok, fine... but what about the backend? More specifically, what is a backend for a compiler?

The Kotlin Compiler, frontends, and backends

The Kotlin compiler first compiles your Kotlin source code into an Intermediate Representation, or IR. This is what they call the Frontend.

Then, the LLVM compiler takes this IR as input and generates machine code for several supported CPU architecture (like the aforementioned arm64). This is the compiler backend.

Batteries included

(...) and a native implementation of the Kotlin standard library.

This last part refers to the stdlib. Kotlin multiplatform also includes a sound standard library. This allows us to get up and running with it. So when writing common code, we'll be using things like kotlin.String instead of java.lang.String. The Compiler Backend will automatically map the IR a Kotlin.String into the correct type.

If you think about it, since the native side doesn't have access to, for example, List, this means you would have to roll your own when targeting that platform. Not cool.
So the Kotlin team built this to enable us to hit the ground running when targeting native. They also did that for things like IO, networking, and more - all of which are included.

Kotlin.String   ----jvm-----> java.lang.String
                -----js-----> string
                ---native---> KString

Long story short

The Kotlin Compiler compiles code into an Intermediate Representation with the Compiler Frontend
Then, it takes this IR and sends it to the LLVM based Compiler Backend.
Kotlin has a stdlib that covers most basic use-cases, like IO and networking which is compiled to the appropriate target representation.

Not Cross-platform, but truly Multiplatform

Write once, run anywhere (with the jve).
-java

Kotlin, probably:

Kotlin Multiplatform takes advantage of its great tooling paired with the amazing LLVM project to create true native code that you only need to write once.

It is not Cross-platform because it doesn't need a runtime to be able to run anywhere. It is just plain native code.

Cool, right?
I would say that Kotlin is able to take that direction because of its tooling/architecture and the stage that LLVM project has achieved. Let's see what the future holds for this tech.
There are certainly many challenges that we haven't discussed here, but anyways, the foundation of the tech looks pretty interesting to me.

There's this talk from KotlinConf18 by Kevin Galligan where he goes on to make the case for this multiplatform idea and why he thinks this is the future.
I also recommend this great post!

A Continuous Integration approach for Kotlin projects

Iury Souza — Sun, 24 Jan 2021 23:48:33 +0000

Continuous integration

Continuous integration is one of the words that you often hear nowadays. I remember not really getting it when I first read about it. I think that most people don't get it right away. It's one of those things that you don't see the benefit before you need it, like an insurance or a sun lotion.

So, let's try to tackle this one once again. What is Continuous Integration?
Integrate what? And why should we continually do it?

As you can see, there's clearly something missing in this term: the subject. Maybe it should be called Continuous CODE Integration, but that may not sound so appealing.
Now you might be wondering what this integration is. Here it refers to code merging. Let's jump to a full definition:

Continuous Integration (CI) is a phase in the software development cycle where code from different team members or different features are integrated together. This usually involves merging code (integration), building the application and carrying out basic tests all within an ephemeral environment.

Successful Continuous integration means new code changes to an app are regularly built, tested, and merged to a shared repository. It aims to improve the quality of software, and to reduce the time taken to deliver it, by replacing the traditional practice of applying quality control after completing all development.

It’s also a solution to the problem of having too many branches of an app in development at once that might conflict with each other.
The main benefits here is making sure that the code quality doesn't decrease while keep this process small and automated.

You can check my DevOps series articles, where I go deeper about some concepts behind CI and automation.

A primer

Iury Souza ・ Dec 17 '20

#devops #automation

The First Way

Iury Souza ・ Dec 17 '20

#devops #automation #ci #cd

Pipelines

The pipeline is another loaded term. We lend its usage from the lean manufacturing process. The basic intuition is that each software change should go through an automated code quality verification assembly line instead of using a manual, artisan like approach to review the changes.

To achieve that we need a clear process to guarantee software quality after every change.
CI pipelines can vary a lot depending on tools, type of project and many other factors, but a common flow has these steps.

Pushing to the code repository
Running Static code analysis
Unit testing
Manual code revision
Merge code on the main branch

An android library example

CI/CD tools can help a team automate their development, deployment, and testing. Some tools specifically handle the integration (CI) side, some manage development and deployment (CD), while others specialize in continuous testing or related functions.

One of the best known open source tools for CI/CD is the automation server Jenkins. Jenkins is designed to handle anything from a simple CI server to a complete CD hub.

You can even use GitHub with Jenkins, via the ghprb-plugin to set up a pipeline workflow to be triggered after a Pull Request is opened.

Let’s take a look at how this flow works on an android library project.

Our Pipeline

This is an overview of our final pipeline. Now let's see what each stage does.

Opening a Pull Request

This is the starting point. Developers should open a pull request where they explain the context of ther code changes. Every project have their own pull request process and that should be clear to everybody contributing on to the code-base.
This pull request should trigger a CI build. While that happens, other developers can contribute to the PR making questions and reviewing the code.

Merge

Your integration server will then checkout the code on your branch and merge with the main branch. If there are issues here, your build will fail and that is reported directly on the PR.

Static analysis

In this pipeline this stage does two things using the same tool:

Code Review Automation

To improve the code review process you can automate some parts of it. To do that you can use Danger. This tool "provides another logical step in your build, through this Danger can help lint your rote tasks in daily code review." This way you can codify your team norms. E.G.: PRs with no description or with no tests are instantly reported back directly on the PR, speeding up the review process.

Actual static analysis

Static analysis is done on the code base of the application without the need to run the software. The goal here is to ensure the code doesn’t have bugs and conforms to standard formatting and styling. You might want to check this article I've written about it with my approach for Kotlin projects.

Ideally, developers should have fast feedback for issues during this step, so they can start fixing them ASAP.
To do that, I've forked the danger-checkstyle-format plugin and added some changes to enable reporting issues from both ktlint and detekt directly on the PR, making developer iteration faster.

If you want to try it out you can get it here:

iurysza / danger-checkstyle_format

Report checkstyle violations to danger.systems

danger-checkstyle_format

Danger plugin that parses a checkstyle format file and turns it into comments on your Pull Request.

Installation

$ gem "danger-checkstyle_format", github: "iurysza/danger-checkstyle_format"

Checkstyle format

Many static analysis tools can create this type of file as an output This is a typical checkstyle format file:

<?xml version="1.0" encoding="utf-8"?>
<checkstyle version="4.3">
    <file
        name="/Users/john-doe/project/src/main/kotlin/com/john/doe/storage/ExternalStorageService.kt">
        <error
            column="18"
            line="42"
            message="Caught exception is too generic. Prefer catching specific exceptions to the case that is currently handled."
            severity="error" 
            source="detekt.TooGenericExceptionCaught"
        />
    </file>
</checkstyle>

This plugin will parse that file and create comments directly on your pull requests using Danger's api. It also offers some basic configurations like:

Run a

…

View on GitHub

Build

In this step we use a gradle task to build our code.

Test

Here we run all unit tests, also with gradle.

Publish reports

This step happens after the pipeline has finished. It runs jacoco jenkins plugin, a java code coverage tool to create coverage reports, then we store the test results in Jenkins. This provides a way for us to track code coverage and investigate flaky tests later on.

Bonus

You can also add some basic commands to your Jenkins pipeline, like the rebuild or rerun-tests commands. To help you work around flakiness on your project's build or testing stages.

Conclusion

The benefits you get from having a CI pipeline in place pile up like Loan Shark interests. I strongly advise that you and your team consider creating one. If you already have one, I'd love to hear some other stages or ideas you've added to it. Please share in the comments =)

Publishing an obfuscated kotlin library

Iury Souza — Sun, 27 Dec 2020 03:19:56 +0000

Being an android developer, I've heard about code obfuscation pretty early in my career. Since it's preconfigured in android's gradle plugin (it's actually proguard under the hood), but you don't really mess with it unless you want to.
On the other hand, when you're developing a java/kotlin library and you want to use code obfuscation, things get a little different. In this article I wanted to give a brief overview of my strategy for publishing obfuscated kotlin libraries.

Code obfuscation

If you're not sure what that means, code obfuscation is the modification of the source code to make it unintelligible, meaning it is impossible for a third-party to understand.
It's makes for a great AppSec initiative and often takes care of the bare minimum security you can have when publishing a source code out in they wild. By making the code hard to reverse engineer, you ensure that your library's intellectual property is guarded against security threats, unauthorized access, and discovery of application vulnerabilities.

Proguard

ProGuard is the most popular optimizer for Java bytecode and it also provides obfuscation for the names of classes, fields and methods.

Proguard is basically a command line tool that takes input jars (aars, wars, ears, zips, apks, directories) and then performs shriking, optimization, obfuscation and preverification to output a new jar.

Here we're just interested in the obfuscation step which will rename the classes, fields, and methods using short meaningless names making it really hard to read the code.

An obfuscation strategy

Choosing our tools

There are probably many ways to do this, but since I have a mobile development background, we'll be doing this using proguard's gradle plugin, which will make things are little more simple, specially during the publication part. Also, I'll use gradle's groovy dsl, just because I like its convenience better than kotlin's now (for gradle scripts). I may need another article to justify that, but, bear with me. This is how we integrate proguard's gradle plugin:


 groovy
buildscript {
    repositories { jcenter() }
    dependencies {
        classpath "com.guardsquare:proguard-gradle:7.0.1"
    }
}

We now need to use it to obfuscate our library's jar. To do that we're going to use gradle's syntax to create a task that does just that.


 groovy
task("obfuscateArtifact", type: ProGuardTask, dependsOn: jar)

Let's dissec the task signature for a second. Here we're declaring that this is a task of type ProguardTask and that it depends on jar task, meaning it has to run after it. This jar task comes from either the kotlin or java plugin, that you're already using and it is responsible for packaging your application artifacts into a jar file. The reason our obfuscateArtifact task needs to run after this one can be seen in the image above. Proguard takes a jar file as the input, so we need this task to create the jar and then we'll apply our task on top of it.

Now let's configure the input/output of our task:


 groovy
def artifactName = "$libraryArtifactId-${libraryVersion}.jar"
def obfuscatedFolder = "$buildDir/obfuscated"

injars "$buildDir/libs/$artifactName"
outjars "$obfuscatedFolder/$artifactName"

The jar task will create, by default, the outputfile file in buildDir/libs folder. By default, its name follows the pattern: artifactId-version.jar. The artifactId is defined in the publishing task that we'll see later on. It's useful to have those values as variables declared in a gradle.properties file.

Next properties are for handling deobfuscation. These are meant to be used when you want to understand your stacktraces.


 groovy
printseeds "$obfuscatedFolder/seeds.txt"
printmapping "$obfuscatedFolder/mapping.txt"

Handling dependencies

After that, we have to make proguard understand our project's dependencies. To do that we need to provide it with java's standard library and ours runtime, and compile-time dependencies. A quick explanation of the last two:

The compile-time classpath contains the classes that you’ve added in order to compile your code. This is the equivalent of the classpath passed to javac (or any other java compiler).
The runtime: classpath contains the classes that are used when your application is running. That’s the classpath passed to the java executable. This is all the code your library is using when it's running.

In proguard we can use the libraryjars property to provide these dependencies:


 groovy
libraryjars "${System.getProperty('java.home')}/lib/rt.jar"
libraryjars configurations.runtime
libraryjars sourceSets.main.compileClasspath

The first line has a path pointing to a rt.jar. This stands for runtime JAR and it has all the classes from the Core Java API. This is the path you use when you're targeting java 8 and below. The other two lines handles runtime and compile-time dependencies that we get by using gradle's dsl.

Wrapping up

The last piece of this puzzle is proguard's configuration file. I prefer doing that in a separated file. If you're using intelij you can get nice syntax highlight for this file, which in my book is always a nice to have.


 groovy
configuration files("$projectDir/gradle/configuration.pro")

The configuration you're going to use is up to you, even if you don't add anything in the config file, proguard will handle code obfuscation by default. We'll come back to this file again.

Ok, now if you want to obfuscate your code you just need to:

run jar task
run obfuscateArtifact task

Problem solved! Or maybe not.

Automating the publication pipeline

Assuming you get the artifactId right, what we did until now may actually work for youu. The obfuscate task will trigger a build and then obfuscate the jar. But that's not ideal. We don't want to manually run this specific task every time. To me, the obfuscation step should be just a part of our library's publication pipeline now. We shouldn't have to remember about it anymore. In order to fix this we need to take a look at this so called, publication pipeline.

If you already have one, it may be just a simple publishing task, something like this:


 groovy
group = "com.myLibrary.group."
version = "$myLibraryVersion"

publishing {
    publications {
        create(MavenPublication) {
            artifactId = "$myLibraryArtifactId"
            from components["java"]
            name = "My Library's name"
            description = "My Library's description"
        }
    }
}

The key here is the from components["java"] declaration. Quoting from gradle's docs:

Components are defined by plugins and provide a simple way to define a publication for publishing. They comprise one or more artifacts as well as the appropriate metadata. For example, the java component consists of the production JAR — produced by the jar task — and its dependency information.

So, this is how our publication task gets the project's artifacts and dependencies. Therefore, if we want to publish an obfuscated library we need to remove that line and provide our publication with the output produced by obfuscateArtifact task.

So, how can we do that?
As we saw in the documentation the components["java"] is automatically defined by the java/kotlin plugin and includes all classes from the default sourceSets. If we want to define our own artifacts that we want to be published, we can use the artifact property.
This is how you do it, inside the create block, add:


 groovy
artifact("$buildDir/obfuscated/${libraryArtifactId}-${version}.jar") {
    builtBy obfuscateArtifact
}

Now, let's read this backwards:

builtBy tells which task is going to build the artifact
the argument to the artifact function is the path where this task is going to output it. This path was defined earlier by the outputJar property, in our obfuscateArtifact task.

Almost there! Now, we need to provide the dependency information. This comes automatically when using the components["java", but here we need to manually provide it. Still in the create block, add:


 groovy 
pom.withXml {
    def dependencies = asNode().appendNode("dependencies")
    configurations.implementation.allDependencies.each { dep ->
        def depNode = dependencies.appendNode("dependency")
        depNode.appendNode("groupId", dep.group)
        depNode.appendNode("artifactId", dep.name)
        depNode.appendNode("version", dep.version)
    }
}

And that's about it!
Now, we can forget about obfuscation. We just run the publish task and get our obfuscated code published seamlessly.

Naming conflicts

If you're using this library in another codebase that will also obfuscate its code you might get some naming conflicts due to proguard using the same name for classes/methods in that projects own obfuscation process. To fix this you can provide proguard with a custom dictionary.


 txt
//class-dictionary.txt
Lorem
ipsum
dolor
sit
amet
consectetur
adipiscing
...

You'll then need tell proguard about it in the configuration.pro file, by adding the line.


 pro
-classobfuscationdictionary class-dictionary.txt

Now, proguard will use the words in that file to name our classes during the obfuscation step, avoiding naming conflicts.

The last thing I like to do is to create a different artifactId for the unobfuscated version of the library and to publish both versions. You can see the complete script bellow:

This diagram shows our end product.

References

Proguard home page
Code obfuscation guide

Continuous Kotlin Static Analysis

Iury Souza — Sun, 20 Dec 2020 20:45:00 +0000

Static analysis is one of those terms that people toss around that usually scares junior devs, (it certainly scared me) but it's a simple and powerful tool to have under your belt.
In this article I wanted to give a brief overview about its definition, best practices and common pitfalls. In the end I'll provide a solution that's been working for me.

What is it anyway?

Static analysis is a way to find errors and other issues in a source code without actually executing it (hence, static). The process provides an understanding of the code structure and can help ensure that the code adheres to industry standards and capture issues early.

In sum, it's a glorified spellchecker.

This is not a new idea, the first one of these tools is lint which dates back from the late 70's and is the most common kind of static analysis tool. Most languages have a linter nowadays.
The automation provided by these tools is fundamental to improve code quality, as it greatly increase code reviewing speed while also being less prone to human error.

Let machines do what they're best at!

What can it do?

Static analysis can be broken into formal, cosmetic, design properties, error checking and predictive categories.

Formal checks if the code is correct;
Cosmetic checks if the code syncs up with style standards;
Design properties check the level of complexities;
Error checking looks for code violations;
Predictive guesses how code will behave when run.

But who needs this anyway?

In short, everyone does. Quoting John Carmack, lead developer of Wolfenstein, Doom, Quake and their sequels:

The first step is fully admitting that the code you write is riddled with errors. That is a bitter pill to swallow for a lot of people, but without it, most suggestions for change will be viewed with irritation or outright hostility. You have to want criticism of your code.

Automation is necessary. It is common to take a sort of smug satisfaction in reports of colossal failures of automatic systems, but for every failure of automation, the failures of humans are legion. Exhortations to "write better code" plans for more code reviews, pair programming, and so on just don't cut it, especially in an environment with dozens of programmers under a lot of time pressure. The value in catching even the small subset of errors that are tractable to static analysis every single time is huge.

Your first Quality Gate

Static code analysis is a common DevOps practice and makes for a great first quality gate, since it provides early feedback for developers making it easier to fix issues.
So, if you've reached this far, I'm hoping that I was able to convince you, but how can we get static analysis added to your project?

Kotlin static analysis landscape

So, in the Kotlin community two tools usually come up when talking about static analysis: ktlint and detekt.

ktlint prides itself for taking a no-configuration approach to linting. The idea is that it will always reflect kotlin's official code style so the idea here is to be a cosmetic static analysis tool.
Detekt on the other hand provides many other features like complexity reports, code smell analysis while also being extensively configurable.

Basic integration pitfalls

Both tools provide many ways to integrate in your project. You can add a plain gradle task, a gradle plugin or install the command line interface. The recommended route is via a gradle plugin.
After integrating these tools in your project you can use tasks like detektCheck and ktlintCheck to run over all your project and report the issues. And that's it, you're done!

Yes, but also no.

The problem with the basic integration is that in real life things tend to be a little different.
If you're adding static analysis to a greenfield project, it's usually fine to run those tasks over the whole project, but if that is not your case, you'll probably want to avoid auto-formatting your whole project or getting 5k+ warnings every time you run one of those tasks. Warnings just turn into noise.
You could use a baseline, which is a configuration that helps incremental adoption on a project, but you're still going to stumble in some issues related to running these over your whole codebase.

A better solution

If you stop and think about it to be able to get the benefits I've mentioned in the beginning of this post, ideally:

You want these tasks to run fast, so you can run them often.
You want to incrementally adopt this in your codebase
And last, but certainly not least: you don't want to keep forgetting to run these tasks.

Configuring both tools is really similar, so I'll highlight some points that happen in both scripts. You can get detekt's here.

So, the magic here is to get the files that were changed in our commit

There's git command that does just that: $ git diff --name-only origin/main --relative In line 38 we call it by using groovy's api. This will output all the change files names.
In line 44 we process the output so that we can pass the filename list to ktlint/detekt.
From lines 13 to 20 we decide whether we need to run static analysis or just ignore all files.

And that's about it! The rest of the code (from both scripts) just handles the report output location.

Good! Almost there now!

We just need a way to make this a part of our workflow. This way we won't forget to run these tasks. And yeah, we can use a git-hook like the pre-commit hook to do just that.

If you're not familiar with git-hooks, you can check them here, but if you want to keep reading, just think about them as a piece of code that is going to be called when you do something in git.

The pre-commit hook will run when you try to commit something. It runs before git actually creates the commit, so the idea is to only allow a new commit when our static analysis passes successfully, effectively creating our first quality gate. Woot woot!

That's it! This makes sure that every piece of code you commit has to follow the minimum quality standards that you and your team agreed upon.
This is not yet a final solution as people can bypass those checks by running git commit --no-verify, but it's a solid first step towards a better codebase.

Besides, if you think about it, these scripts can be easily integrated to a CI environment. Wouldn't it be nice if it could send those warnings as inline comments in your pull request? Stay tuned for more =)

References

Carmac's blog
Static analysis definition

The First Way

Iury Souza — Thu, 17 Dec 2020 11:50:12 +0000

The idea behind the First Way is that by identifying, measuring, and improving the flow of work through our value stream, we can optimize and refine the ability to deliver value to our customers.

The first step towards that is to split every piece of work in the value stream and automate it. This is often called a pipeline.

The foundation of our pipelines

You're probably already using a build system to automate some parts of our value stream, but we can do better. Quoting from the Devops Handbook:

In order to create fast and reliable flow from Dev to Ops, we must ensure that we always use production-like environments at every stage of the value stream. Furthermore, these environments must be created in an automated manner, ideally on demand from scripts and configuration information stored in version control, and entirely self-serviced, without any manual work required from Operations. Our goal is to ensure that we can re-create the entire production environment based on what’s in version control.

The authors go on to say that we should enable on demand creation of dev, test, and production environments so developers can run production-like environments anywhere, even on their own computers, created on demand and self-serviced.

This way, developers can run and test their code in a production-like environment as part of their daily work,
providing early and constant feedback on code quality.

We've all been there before:

The "It works on my machine!" issue comes from the fact that in any complex system there's just too many variables to manage. If you google The Matrix of Hell you'll might see something like this:

Different kind of applications will have different rows and columns in this table, but you get the idea: - Multiple environments with multiple third party dependencies, create a combinatorial explosion that no sane human being can manage.

Most companies will have a test and production environments, maybe you even have a dev environment where you can safely run your app with the hottest new features. But as the project evolves, non-production environments tend to degrade. It's hard for them to keep up with prod because of lack of automation or the discipline of constantly updating it. After a while the dev environment will have so many small things that work differently that it becomes hard to trust it. Some months down the line and people will stop believing anything that happens in the dev environment and only newcomers, still unaware of these issues, will be using it at all.

How to fix that?

The issue is that if these environments were manually created, you can't expect people to maintain them equally.
By automating their creation we enable anyone to quickly spin up a new environment on demand, enabling every developer to have their own local environment when they need one.
The automation scripts will make sure that the correct configuration is loaded on each environment, avoiding hundreds or thousands of hours of unneeded headaches over a misconfiguration due to human error.

This way we're effectively giving the development team access to production-like environments by treating your infrastructure and configuration as code, making immutable infrastructure, ensuring where we operate, develop, and test is comparable and a reliable indicator of system operation for customers — such that new value contributed by developers can easily and predictably be deployed to customers.

It's hard, but it pays in the long run!

This is not a simple issue to solve. And you should be mindful about when and where to start the process.

In my experience, small startups or software studios tend to overlook this, because it's too much to worry about this stuff at their product stage.

If your project doesn't have to scale, fine. You can get away from following these ideas in smaller projects, but the engineering team has to stay vigilant for when they should stop and implement a solution.

If you get big enough the cost of not having on-demand, dev/prod environment parity will also scale, even faster than your product. And hiring more people will only make the cost increase.
Hundreds or thousands of development hours go down the drain because of this issue. Bugs that are impossible to reproduce make debugging even harder, fear of changing "buggy" parts of the application causes code quality to suffer and everybody suffers.

Every team and organization is different and deciding when to start with this approach has to made in case by case scenario, but I'd argue that it's certainly cost-effective to gradually implement this idea in the early stages of a project.

Create our single repository of truth for the entire system

To ensure that we can restore production service repeatedly and predictably (and, ideally, quickly) even when catastrophic events occur, we must check in all the assets needed to recreate the system state.

That includes:

All application code and dependencies (e.g., libraries, static content, etc.)
Any script used to create database schemas, application reference data, etc.
All the environment creation tools and artifacts
Any file used to create containers (e.g., Docker or composition files)
All supporting automated tests and any manual test scripts

Ideally this should happen using immutable infrastructure where no manual changes occur - make it easier to destroy
and rebuild than to configure.

Encoding knowledge and sharing it through a repository is one of the powerful mechanism for propagating knowledge.
Particularly when it includes every library (internal or external), and those each have an owner responsible for ensuring it successfully passes all tests.

Wrapping up

We've seen how dev/prod parity begets automation. If you're not yet embracing these concepts, well, you quite frankly should, or at least be aware of the issues that are coming your way.
But fret not! We've all been there and those who came out alive have some very interesting ideas about how to solve it. We just have to listen to them.

Coming up next: Test automation! Stay tuned.

A primer

Iury Souza — Thu, 17 Dec 2020 00:16:22 +0000

Recently, I've been trying to improve my understanding of good CI and CD practices and I often saw this word: DevOps.
I mean, I thought I sort of understood what DevOps meant: - The team who keeps our infrastructure up and running and that manages some crazy scripts for on our pipelines, right? Well... no.

So I decided to take some time to understand this. Picked a book on the subject, read some articles here and there...
And in this series I'll try to talk about some of what I learned and about my own experiences.

The book I picked up was The DevOps Handbook, subtitled: How to Create World-Class Agility, Reliability, & Security in Technology Organizations. This is an incredible source of knowledge, written by some very smart folks from the DevOps community. After that, one thing led to another, and I stumbled upon The Twelve-Factor App. This one is a methodology developed by people working for the Heroku platform. They provide key insights for anyone that develops software delivered as a service (SaaS), but a lot of what they talk about is useful for any kind of software. You should really check them out if you have the chance. But enough with that. Let's dive in.

So, what is DevOps?

Quoting Wikipedia:

DevOps is a set of practices that combines software development and IT operations.
It aims to shorten the systems development life cycle and provide continuous delivery with high software quality.
DevOps is complementary with Agile software development; several DevOps aspects came from Agile methodology.

So, this is basically agile at a macro-level. When you take a birds-eye view of all the agile methodologies,
clean-code best practices, conventions, collaboration tools, etc, what you and your organization really
want is to deliver value to the costumer. Ok, makes sense, right?

It does! And the people from the DevOps Handbook even came up with this idea of a value-stream. Now, what is a value stream is in the first place?

Quoting from the book:

In the DevOps, we typically define a value stream as the process required to convert a business hypothesis into a technology-enabled service that delivers value to the customer.

The Three Ways

Right off the bat, the authors split DevOps practices into three ways:

The First Way enables fast left-to-right flow of work from Development to Operations to the customer. In order to maximize flow, we need to make work visible, reduce our batch sizes and intervals of work, build in quality by preventing defects from being passed to downstream work centers, and constantly optimize for the global goals.
The Second Way enables the fast and constant flow of feedback from right to left at all stages of our value stream. It requires that we amplify feedback to prevent problems from happening again, or enable faster detection and recovery.
The Third Way enables the creation of a generative, high-trust culture that supports a dynamic, disciplined, and scientific approach to experimentation and risk-taking, facilitating the creation of organizational learning, both from our successes and failures. Furthermore, by continually shortening and amplifying our feedback loops, we create ever-safer systems of work and are better able to take risks and perform experiments that help us learn faster than our competition and win in the marketplace.

The book is divided into three parts, one per way. Each part documents the relevant technical practices with supporting case studies and experiences sourced from industry behemoths like Netflix and Amazon.

But this is not a book review, and for the purpose of this series I wanted to share some insights on topics I've found particularly compelling. Let's dive in!

Setting up MockServer

Iury Souza — Sat, 07 Nov 2020 18:41:01 +0000

Main goals:

Simple and declarative testing environment setup.
Runs everywhere

Strategy

Set up a replicable local server.

Docker - Containerized applications (actually write once, runs anywhere)

Provide an expectations json where all the requests are mapped out.

MockServer - Creates a local server with complex request/response expectations through a json resource or a REST API.

Making it happen

Install Docker.
Install Docker Compose.
Make sure your docker daemon is up and running.
Create a folder for your MockServer setup.



  $ mkdir /path/to/mockserver_container/ && cd /path/to/mockserver_container

Create the docker-compose.yml .
- This is where you'll setup your container for MockServer.



  $ touch ./docker-compose.yml && vim ./docker-compose.yml

Paste this code snippet:



version: "2.4"
services:
  mockServer:
      image: mockserver/mockserver:latest
      ports:
        - 1080:1080
      environment:
        MOCKSERVER_WATCH_INITIALIZATION_JSON: "true"
        MOCKSERVER_PROPERTY_FILE: /config/mockserver.properties
        MOCKSERVER_INITIALIZATION_JSON_PATH: /config/initializerJson.json
      volumes:
        - type: bind
          source: .
          target: /config

This is saying:

Create a service named mockserver using the docker image from mockserver/mockserver:latest.
Assign the host port 1080 to the container port 1080.
Set the enviroment variables to the provided paths.
Create a volume binding from the current directory (this file's dir) to /config inside the container.
- Setup the Expectation Initializer JSON.
Create the initializerJson



  $ touch ./initializerJson.json && vim ./initializerJson.json

Paste this code snippet:



  [
    {
      "httpRequest" : {
        "method" : "GET",
        "path" : "/api/v1/weather",
        "queryStringParameters" : {
          "code" : ["10969"]
      }
    },
      "httpResponse" : {
        "body" : "{\"wheather\": 20.5}",
        "statusCode": 200
      }
    }
  ]

Full documentation reference can be found here

Running your container

Now that we've got the plumbing out of the way, it's time to run it.

Inside the folder you created the .yml type:



  $ docker-compose up

Your terminal should print something like this:



  Creating mockserver_mockServer_1 ... done
  Attaching to mockserver_mockServer_1
  mockServer_1  | 
  mockServer_1  | java  -Dfile.encoding=UTF-8 -jar /opt/mockserver/mockserver-netty-jar-with-dependencies.jar  -server
  mockServer_1  | 
  mockServer_1  | 2020-04-11 15:52:30  org.mockserver.log.MockServerEventLog  INFO  creating expectation:
  mockServer_1  | 
  mockServer_1  |   {
  mockServer_1  |     "id" : "3f3962d2-9c03-4dce-95cd-2522962ceccb",
  mockServer_1  |     "priority" : 0,
  mockServer_1  |     "httpRequest" : {
  mockServer_1  |       "method" : "GET",
  mockServer_1  |       "path" : "/api/v1/weather",
  mockServer_1  |       "queryStringParameters" : {
  mockServer_1  |         "code" : [ "10969" ]
  mockServer_1  |       }
  mockServer_1  |     },
  mockServer_1  |     "times" : {
  mockServer_1  |       "unlimited" : true
  mockServer_1  |     },
  mockServer_1  |     "timeToLive" : {
  mockServer_1  |       "unlimited" : true
  mockServer_1  |     },
  mockServer_1  |     "httpResponse" : {
  mockServer_1  |       "statusCode" : 200,
  mockServer_1  |       "body" : "{\"wheather\": 20.5}"
  mockServer_1  |     }
  mockServer_1  |   }
  mockServer_1  |  
  mockServer_1  | 2020-04-11 15:52:30  org.mockserver.cli.Main  INFO  logger level is INFO, change using:
  mockServer_1  |  - 'ConfigurationProperties.logLevel(String level)' in Java code,
  mockServer_1  |  - '-logLevel' command line argument,
  mockServer_1  |  - 'mockserver.logLevel' JVM system property or,
  mockServer_1  |  - 'mockserver.logLevel' property value in 'mockserver.properties' 
  mockServer_1  | 2020-04-11 15:52:30  org.mockserver.log.MockServerEventLog  INFO  started on port: 1080

Sending requests

Open a new terminal window and enter:

To see info about your available containers enter again:



$ docker ps -a

You can now see some info about your running container.



$ curl localhost:1080/api/v1/weather\?code\=10969

You should see the response:



{"wheather": 20.5}

And thats it!

You've got yourself a local server where you can mock simple and complex requests and responses.

Useful links:

REST API Swagger docs

Concise and type-safe scripting with kotlin

Iury Souza — Sat, 07 Nov 2020 18:32:44 +0000

A kotlin script is a way to compile and run Kotlin code easily.
Scripts are a powerful tool when you want to automate any kind of work. The most used languages for scripting tend to be python
or bash/shell but, what if you could get the same task done with Kotlin?
But not just that. What if you had access to any java or kotlin library in your scripts?

With Kscript you can easily do that! Besides it also offers a lot of features.

Scripts caching: Running the same script will be way faster the second time.
Maven dependencies
IntelliJ support
Bootstrap header
Check the kscript repo for the full feature set.

Trying it out

Install it with sdk man:

$ sdk install kscript

Now let's create a .kts file and try importing some libraries to create an embedded server:

#!/usr/bin/env kscript

@file:MavenRepository("bintray-ktor","https://dl.bintray.com/kotlin/ktor")
@file:DependsOnMaven("io.ktor:ktor-server-netty:1.2.6")

import io.ktor.application.*
import io.ktor.http.*
import io.ktor.response.*
import io.ktor.routing.*
import io.ktor.server.engine.*
import io.ktor.server.netty.*

println("starting server...")

val server = embeddedServer(Netty, port = 8080) {
  routing {
    get("/") {
      call.respondText("Hello World!", ContentType.Text.Plain)
    }
  }
}
server.start(wait = true)

Now, lets run it!

$ kscript kscript-server.kts

Open your browser on localhost:8080/

Cool, right?

But if you’re like me you probably do most of your work on the IDE. So, let’s try that now.

If you have Intelij Idea on your machine you can type

$ kscript --idea kscript-server.kts

and it will generate a Gradle Kotlin project with all dependencies declared with the @file convention and add them to a Gradle file.

#Protip: If syntax highlighting is not working properly, just right click on the build.gradle file select Import Gradle project