DEV Community: Antoine Veuiller

A Generic Approach to Troubleshooting

Antoine Veuiller — Wed, 21 Sep 2022 01:48:09 +0000

When working with a production system, you may encounter errors at any point, originating either from an application or its underlying host. While it may become natural for a seasoned engineer to pinpoint its origin, it can be overwhelming at times if you are unsure of where to look.

This article aims to provide a generic framework to tackle production issues. The framework will serve as a basis for any engineer aiming to learn about troubleshooting basics. However, as with any generic framework, it may need adjustments for specific use cases.

Environment

In this article, we consider that you are working on a project that already implements the best practices in terms of monitoring so that you have access to sensible metrics, defined for instance through the USE Method. Those metrics can be used to provide performance dashboards and declare alerts triggered in case of faulty or suspicious behavior. For a more visual reference, the following diagram shows a basic monitoring architecture using Prometheus, Grafana, and Alert Manager.

Upon the reception of an alert for an issue impacting a production service, the actions to take can be broken down into three steps: Analysis, Correction, and Post-Mortem. As some of the traces found or produced during the incident can be transient, it is recommended to keep track of any event and action that was taken on the production during the analysis and correction phases. This will ease the post-mortem phase.

Analysis

Explore The Issue

The first step when receiving an alert will be to find any known information about it. Usually, a Runbook will explain the reason why this alert fired and the steps to take to solve it.

Even if the solution seems simple, you may want to find the scope of the problem. Your response might differ if the issue is impacting one or hundreds of hosts. Likewise, an issue crippling production will require a low response time to issue corrective action, while a less critical issue, like a decrease in performance, may give you more time for analysis.

While your production servers should behave in the same way, in some cases you will need to investigate the error before being able to determine the actual number of impacted hosts.

Check The Basic Metrics

If you have multiple hosts impacted, take one of them as an analysis host before you have a clear view of which metrics are important in the current case. If you have the chance to have metrics exported to a dashboard, for instance through node-exporter, you will be able to see the issue quite easily from the different indicators. Otherwise, you will need to jump on the host and use the usual tools to get more insights into the host's health. In any case, you will be looking for issues with the CPU, RAM, Load, Disk, and Network.

This first step will indicate if the issue is purely applicative, or if your server is over-used to some extent. This will also help you reduce the scope of where to look further down the investigation.

Regardless of the result of the previous check, you will need to look at the application-specific metrics, both for performance through CPU and RAM usage, as well as applicative metrics such as response time, workload queues, and more. You also need to check the applicative logs looking for anything that stands out of the ordinary.

From there, if you have enough information to at least recognize the issue and perform a temporary corrective action, you should go for it and fix the production as soon as possible before digging Further.

Dive Deeper into The Application Metrics

If you didn't find the root cause of the error yet, you need to dig into the application behavior and its interaction with the system. This step is mostly dependent on the application, but there are some common checks that could show irregularities.

On the system side, you can take a look at:

The network communications and connectivity (e.g. tcpdump, netstat, telnet, mtr).
The Kernel and related logs (e.g. journalctl, dmesg, …)

On the applicative side, you can take a look at:

The configuration through files and environment variables (e.g. /proc).
The opened file handles and connections (e.g. lsof).
The system calls that are performed by the application (e.g. strace).
The application performances in a specific code path (e.g. gdb, pprof, …).

Correction

Protect The Production

It is important to protect production from the issue as soon as possible. This can take many forms but usually, you will either implement a quick fix, that will enable the instance to still run until a long-term solution is implemented, or isolate the faulty application from the production pools. The latter can be done for instance by redirecting the load-balancer flow to other instances.

With orchestration solutions and stateless services, it could be appealing to simply restart the application and use a brand-new instance. This may work in some cases, but be sure to backup all data required for further investigation before doing so, otherwise, you may end up in the same situation later.

Implement a Long-Term Solution

Once the production is secured, you can catch your breath and start digging further into the data you collected to find the actual root cause of the alert that was triggered. Even if the situation is stable for now, you may want to improve the overall code quality so the future you don't have to investigate this issue again.

This section is entirely specific to the encountered issue, the actual fix could range from host configuration or code update to a complete architectural refactor in some cases. I would advise performing the long-term fix as soon as possible, but you may be compelled to write down the Post-Mortem beforehand if the fix is big or impacts multiple components.

Post-Mortem

Write The Actual Post-Mortem

The main point of a Post-Mortem is to hold all relevant events and actions that took place during the incident. This helps to understand the issue and improve the response process for future incidents. It also helps fellow engineers to map potential side effects they experienced to the main incident.

On top of the precise timeline, you need to describe the actual root cause from which the incident originates and the action you and your team are proposing to avoid this issue from recurring. In case of a corrective action impacting multiple components, you will need to have this post-mortem reviewed by the right stakeholders before taking action. The post-mortem should hold enough information for a peer to have a strong opinion about the proposal.

Improve Runbooks And Alerts

Depending on the corrective action you took, you may need to create new alerts to cover different edge cases and modify the runbook sections with up-to-date data.

Even if the changes you performed are trivial, use this step as a feedback loop on the data you were missing during the investigation to help the future you during the next investigation.

Conclusion

This article aims to provide a baseline of investigation in case of a generic production incident. The procedure will not show you the error if you don't ask the right questions but defines a framework to help you find the right questions to ask, and where to write the answers.

As a picture is worth a thousand words, here is a summary of the article through a simple flow chart.

About design patterns: Dependency Injection

Antoine Veuiller — Wed, 13 Oct 2021 10:23:02 +0000

What is dependency injection?

Dependency injection (DI) is a very simple concept that aims to decouple components of your software and ease their integration and testing. It does so by asking for their sub-components instead of creating them.

During this article, we will also mention inversion of control (IoC), which is commonly used along with dependency injection. This pattern aims to avoid asking for implementations but rather interfaces while injecting dependencies.

This article will use a simple example in Java to present dependency injection but aims towards a technology-agnostic explanation of the concept and its advantages. Moreover, even if it is an object-oriented design pattern, you can still adapt the behaviour in many programming languages.

Let’s clarify all this using an example!

We will present a weather service that shows an intelligible representation of the weather. In the current implementation, we rely solely on a thermometer.

Let’s start without dependency injection.

As you can see on the diagram, the WeatherService is relying on a Thermometer, which can be configured with a TemperatureUnit.
Not using dependency injection will result in a code creating a new instance of Thermometer in the service, and a Thermometer configuring the TemperatureUnit to use:

public class Thermometer {
  private final TemperatureUnit unit;

  public Thermometer() {
    this.unit = TemperatureUnit.CELSIUS;
  }
}

public class WeatherService implements WeatherContract {

  private final Thermometer thermometer;

  // This constructor is not using dependency injection
  public WeatherService() {
    this.thermometer = new Thermometer();
  }
}

Now let’s imagine that we want to use a Thermometer configured to use Fahrenheit degrees instead of Celsius. For this, we add a parameter to switch between both units.

public Thermometer(boolean useCelsius) {
  if (useCelsius) {
    this.unit = TemperatureUnit.CELSIUS;
  } else {
    this.unit = TemperatureUnit.FAHRENHEIT;
  }
}

One can also argue that the user of our program won’t always have access to an actual thermometer on their device, thus you may want to be able to fall back to another implementation in this case.
For instance, an API sending the current temperature in your area. Integrating multiple implementations inside the service could be done as shown below.

public WeatherService(boolean useRealDevice, 
                      boolean useCelsius,
                      String apiKey) {
  if (useRealDevice) {
    this.thermometer = new Thermometer(useCelsius);
  } else {
    this.thermometer = new ThermometerWebService(useCelsius, apiKey);
  }
}

As a result, initializing the service can be done as follows:

public static void main(String[] args) {
  // Not using dependency injection
  WeatherContract weather = new WeatherService(true, true, null);
}

Even if it is easy to use, our current version of the WeatherService is not evolutive. If we take a closer look at its constructor, we can see multiple design flaws that will haunt us in the long run:

The constructor is choosing its Thermometer. Adding a new type of Thermometer would require some parameter tricks to guess the implementation to use.
The constructor is managing the Thermometer constructor parameters. Adding the ThermometerWebService forced us to add a new apiKey parameter to it, even if unrelated to the WeatherService.

As a result, any change to any Thermometer implementation may require changes on the WeatherService constructors. This behaviour is unwanted and breaks the Separation of Concerns principle.

Will dependency injection improve my project?

Dependency injection, associated with inversion of control, is a good way to cover this use case. It allows you to choose which kind of thermometer you want in your program depending on the situation. The following diagram gives a quick overview of our new architecture:

The inversion of control is represented in this diagram by the fact that our WeatherService implementation is linked to ThermometerContract rather than any of its implementations. That’s nothing more than this.

As for dependency injection, WeatherService will now take a ThermometerContract in its constructor, requiring the block using the service to build an instance filling this contract:

public class WeatherService implements WeatherContract {
  // We now use the Interface   
  private final ThermometerContract thermometer;

  // New constructor using dependency injection    
  public WeatherService(ThermometerContract thermometer) {
    this.thermometer = thermometer;
  }
}

As a result, the initialization of a WeatherService for both constructors will look like the following:

public static void main(String[] args) {
  // Using dependency injection
  TemperatureUnit celsius = TemperatureUnit.CELSIUS;
  ThermometerContract thermometer = new Thermometer(celsius);
  WeatherContract weather = new WeatherService(thermometer);
}

Now, our ThermometerContract can be fully configured by an external part of the software. More important so, the WeatherService doesn’t need to know any of the available implementations of ThermometerContract, thus decoupling your software packages.

This could seem like nothing important, but this simple switch of responsibility is critical leverage for multiple aspects of software design. It enables you to control the instance creation from your software entry point by chaining dependencies. You won’t have to take care of the instantiation until it is necessary. This behaviour could be compared to raised exceptions, that are ignored until taken care of in a significant context.

That’s all there is to dependency injection?

It is important to know that even if you can find libraries that help you manage your dependency injection, it is not always necessary to use them.

Those libraries tend to cover a lot of cases thus be offputting to developers not comfortable with the pattern in the first place. In reality, they simply ease the instantiation of complex dependency trees and are not required at all.

The following section is an example of injecting our service using Guice, a dependency injection framework for Java made by Google. The concept is to reference bindings of every component you can inject in your program, so that the library can generate a class of any type, automatically.

Let’s consider that we have two implementations with the following constructors:

public class WeatherService implements WeatherContract {
  private final ThermometerContract thermometer;

  @Inject
  public WeatherService(ThermometerContract thermometer) {
    this.thermometer = thermometer;
  }
}

public class Thermometer implements ThermometerContract {
  private final TemperatureUnit unit;

  @Inject
  public Thermometer(@Named(WeatherModule.TEMPERATURE_UNIT) 
                     TemperatureUnit unit) {
    this.unit = unit;
  }
}

The injection module should be configured to bind all needed interfaces to a given implementation.
It should also be able to inject any object without a specific interface, such as the enumerate TemperatureUnit.
The injection will then be bound to a specific name, “temp_unit” in this case.

public class WeatherModule extends AbstractModule {
  public static final String TEMPERATURE_UNIT = "temp_unit";

  @Override
  protected void configure() {
    // Named input configuration bindings
    bind(TemperatureUnit.class)
      .annotatedWith(Names.named(TEMPERATURE_UNIT))
      .toInstance(TemperatureUnit.CELSIUS);

    // Interface - Implementation bindings
    bind(ThermometerContract.class).to(Thermometer.class);
    bind(WeatherContract.class).to(WeatherService.class);
  }
}

Ultimately, the module can be used as follow, here instantiating a WeatherContract.

public static void main(String[] args) {
  // Creating the injection module configured above.
  Injector injector = Guice.createInjector(new WeatherModule());

  // We ask for the injection of a WeatherContract, 
  // which will create an instance of ThermometerContract
  // with the named TemperatureUnit under the hood.
  WeatherContract weather = injector.getInstance(WeatherContract.class);
}

Such modules usually provide a good power of customization to the injected elements, thus we can consider configuring the injection depending on the available implementations.

As a result, using a library is not required when integrating dependency injection. However, this could save a lot of time and cumbersome code in big projects.

Show me some tests!

As a side effect of decoupling your code, the dependency injection pattern is a real asset to improve unit testability of each component. This section contains an example of unit tests for our WeatherService.

As said above, making WeatherService asking for a ThermometerContract enables us to use any implementation we want. Hence, we can send a mock in the constructor, then control its behaviour from the outside.

public void testTemperatureStatus() {
  ThermometerContract thermometer = Mockito.mock(ThermometerContract.class);
  Mockito.doReturn(TemperatureUnit.CELSIUS).when(thermometer).getUnit();
  WeatherContract weather = new WeatherService(thermometer);

  Mockito.doReturn(-50f).when(thermometer).getTemperature();
  assertEquals(
    TemperatureStatus.COLD,
    weather.getTemperatureStatus()
  );

  Mockito.doReturn(10f).when(thermometer).getTemperature();
  assertEquals(
    TemperatureStatus.MODERATE,
    weather.getTemperatureStatus()
  );
}

As you can see, we can then control our thermometer without a struggle from outside our tested class.

Conclusion

Dependency injection is a way of thinking your code architecture and can be simple to implement by yourself. In bigger projects, integrating a dependency injection framework can save you a lot of time in the long run.

Dependency injection provides multiple non-negligible advantages such as:

Code decoupling: use the contracts and ignore implementation specificities.
Enhanced testability: Unit tests almost become a pleasure to write.
Configurability: you can more easily swap injected instances.

You can find the full code example in my design tutorials repository on GitHub

Is GitHub Copilot a Threat to Developers? (Spoiler: It’s Not

Antoine Veuiller — Wed, 28 Jul 2021 04:30:06 +0000

Computer science provides more and more abstraction layers with time, to the point that we don’t really know what’s running underneath.

GitHub recently released Copilot in technical preview. Copilot is an AI-based assistant that will automatically generate code from the context of your application, with the aim of easing software development.

A question that arises with simplification and automation in computer science is generally “will the developer still be needed?”. And every time it happened, the answer was “yes”. Abstraction in computer science reduces the cost of entering the domain, thus attracts a larger population. But it also usually creates opportunities with new technologies, practices, and more complex systems.

This short article aims to summarize the evolution of abstraction layers in computer science to help understand the impact that GitHub Copilot may have.

A Bit of History

The example that seems the oldest and most sensible would be the Turing Machine. It was created with the specific idea to break the cryptographic code Enigma, but the concepts behind it are now considered as a ground reference of algorithmic capabilities for programming languages.

Since then, we created processors, able to take a random set of instructions and process them generically. Those instructions were originally described using machine code, which is hard to manipulate for a human. This is why programming languages of higher abstraction appeared.

Generally speaking, we can describe the goal of a programming language as to write human-readable specifications that can be converted into machine code.

Evolution of Programming

Programming languages evolved from low (abstraction) level languages to higher levels, each iteration adding automation and helpers to the common operations. Those abstractions can take many forms, from garbage collection to libraries reading CSV files.

These evolutions enable developers to add more complexity to their software, as there is less complexity on the technical parts. They can in turn abstract away even more for others. Thanks to this, taking on programming is becoming easier with time. We even see some programming languages using almost natural languages (e.g. Gherkin) and requiring next to zero computer science knowledge to write.

This does not mean that people will stop using languages with lower abstraction levels. On the contrary, even if they are generally harder to master and more verbose, they can be way more flexible for some use cases, memory management for instance.

We can also note the appearance of Machine Learning, creating dynamic processes over data that would have been tedious to analyse, either by hand or through specific code. This enables writing potentially complex behaviours with a few lines of code in some cases. Even then, there is some automation of it to the point where you only have to provide data to get working results.

That being said, having a minimum of knowledge on the layers underneath remains very important to really comprehend the code execution. Otherwise, it would be difficult to adapt to issues that may, and will, arise.

Evolution of Infrastructure

We talked about code, but the same kind of evolution happened on the servers too. At first, companies were starting on personal computers until they could get a hold of some space in a datacenter, renting one or more baremetal servers. This was costly and required technical knowledge for both the installation and deployment processes.

The virtualization technology evolved to the point where you can now rent a virtual machine hosted on a physical host anywhere through cloud providers. This tremendously reduces the costs and enhances accessibility, but still requires technical knowledge to administrate correctly the machine on top of the deployments pipelines.

Since more recently, the technology of containerization is heavily used with powerful tools like Docker and Kubernetes. Those tools can help you deploy applications on a VM without necessarily mastering the system underneath*. In the same fashion as machine learning, there are projects aiming to further reduce the knowledge required to use such technologies.

On top of that, a new abstraction layer called serverless is emerging. This enables you to directly execute code on a remote container, without having to take care of the hosting problematics at all. While this is currently mostly used for stateless operations, there are no doubts about the growth of use cases and tools around this ecosystem.

*Note: Those tools may be better used with knowledge on the underlying layer, but you can redirect your learning to a more practical proficiency.

A Glimpse on The Future of Copilot

GitHub Copilot is a pretty interesting advancement in software engineering, even if not completely on point at the moment. According to the FAQ, the system gets the generation right around 50% of the time, and it is recommended to stay wary of the generated code.

If we look a bit differently on GitHub Copilot, we can consider this as a new programming language. The input of this language is the documentation and function name and those are transpiled into source code for another programming language.

The generation is today limited to methods but we can imagine that this will evolve to classes or even packages. We can also imagine the generation becoming almost flawless for well-known use cases.

Even if everything is one day working on a larger scale and flawlessly, the generated code will still require developers to handle the direction it takes, and connect the even bigger pictures altogether.

Conclusion

In regard to what we just saw, it makes no doubt that GitHub Copilot will further abstract development and ease the creation of software. This breakthrough has the power to move the development industry, as other abstraction layers did before.

Having said that, developers will probably never lack opportunities, as the scope of the projects will get bigger thanks to this new abstraction layer. The day to day work done as a developer may change, but developers will still be required for a long time.

As a matter of comparison, we already saw system administration positions evolve to SRE, still working on providing reliable hardware, but on a scale never seen before. In the same way, developers may evolve to something close, adapting to the new tools.

Thanks to Sarra Habchi for the review

Introduction to Flaky Tests by Example

Antoine Veuiller — Fri, 16 Jul 2021 12:45:13 +0000

Tests are an essential part of software development as they give a ground truth about the code sanity. As developers, we reasonably expect our unit tests to give the same results if the source code does not change. It can happen, however, that the result of a unit test changes over multiple executions of a test suite without any change in the code. Such a test is named a flaky test.

A flaky test is not dangerous per se but reduces the confidence a developer can give to his test suite, diminishing the benefits of the latter. It is thus recommended to eradicate such issue as soon as possible.

However, depending on the origin of the flakiness, one may find out only a few days, months or even years later that the tests are flaky. It may be hard to dive back into those and find the root causes, so usually, we tend to put those tests aside to make them less annoying or we rerun them until success.

As a real-world example of flaky tests and the logic behind their resolution. I will talk about two interesting cases I had the opportunity to fix during my career.

Storytime!

During my career, I stumbled onto a couple of flaky tests issues. There are two instances that, in my opinion, are quite symptomatic of test flakiness, with quite different contexts.

The examples are voluntarily adapted to a simpler context than the original ones to keep a short and focused article.

Story 1: 5 days a month isn’t a big deal

I entered a project where continuous integration was broken during the 5 first days of each month. I was told that this wasn’t a big deal since we don’t need to deploy this project at the beginning of a month. The priority of fixing those tests was so low that it has remained like this for years before we took the time to tackle this issue.

def add_data(input_datetime, data):
    # Store the data along with the input date
    # [...]

def retrieve_data(lower_date):
    # Retrieve all data from lower date to now 
    # [...]

def compute_stats():
    # Compute some statistics about stored data
    month = arrow.get().floor('month') 
    data = retrieve_data(month)
    return len(data)

def test_compute_stats():
    # Test method checking the behaviour of compute_stats
    now = arrow.get()

    add_data(now.shift(days=-5))
    add_data(now.shift(days=-1))
    stat = compute_stats()

    assert stat == 2, "We retrieve the two data input"

The faulty feature was computing statistics about the current month. As the developer creating the initial tests wanted to take all cases into consideration, he created a test that gave as input multiple dates relative to the current datetime.

Among those inputs, one was 5 days before the current date, and the test was always computing the statistics as if it was part of the same month. As a result, it led to the tests being faulty at the beginning of each month. We can then imagine that the flakiness was detected under one to three weeks after the feature development and from then on, ignored.

This test is time-dependent because compute_stats will call for the date of analysis itself. As a result, the computation will always be dependent on the current date. One way of fixing such issues would be to sandbox the execution of the tests in order to control the current date.

At first, we wanted to rely on dependency injection and make compute_stats ask for a month to compute the statistics. This would create an easy way of sandboxing the execution and also potentially open the door to new features. However, in this project, this wasn’t trivial to implement because there was a lot of code dependent on this feature.

Another way of doing so would be to inject the value directly to the method. Python has a very good library to sandbox the tests when using the built-in datetime objects: freezegun. Once again, and unfortunately for us, the project was using arrow so this was not a possibility.

Fortunately, and thanks to some previously well-thought environment on the project, we had a central method to provide the current date, which was initially intended to prevent the use of a wrong timezone.

By mixing this method to the awesome patch decorator of python mock library (which is part of the standard unittest library since 3.3), we solved the issue with a simple modification.

def add_data(input_datetime, data):
    # Store the data along with the input date
    # [...]

def retrieve_data(lower_date):
    # Retrieve all data from lower date to now 
    # [...]

def compute_stats():
    # Compute some statistics about stored data
    month = arrow.get().floor('month') 
    data = retrieve_data(month)
    return len(data)

def test_compute_stats():
    # Test method checking the behaviour of compute_stats
    now = arrow.get()

    add_data(now.shift(days=-5))
    add_data(now.shift(days=-1))
    stat = compute_stats()

    assert stat == 2, "We retrieve the two data input"

By sandboxing the execution to a given point in time, we ensured the reproducibility of the build at any given time.

Story 2: We use that configuration!

In another project, while creating a new feature we happened to break tests unrelated to our changes. This case could have been tedious to pinpoint, fortunately, due to the project organization, we were certain that the new feature did not affect the code covered by the now failing tests.

The code below is a synthetic representation of what happened, a global config object was interacting with both the existing and new features.

# Global state configuration
config = {}

def existing_feature():
    if "common_entry" not in config:
        raise ValueError("Not configured")

    # Process [...]
    return True

def our_new_feature():
    if "common_entry" not in config:
        raise ValueError("Not configured")

    # Process [...]
    return True

From the isolation of the two features, we knew that the new tests had to be the ones creating a faulty global state. There were globally two possibilities for the faulty state. Either the new test was injecting something new to the global state, or removing something essential to the existing test.

The test cases below were always run in the specific order ConfiguredFeatureTest, ExistingFeatureTestCase before integrating the new feature, then in the order ConfiguredFeatureTest, NewFeatureTestCase, ExistingFeatureTestCase.

class ConfiguredFeatureTest(unittest.TestCase):
    def setUp(self):
        config["entry"] = "anything"

    def test_configured(self):
        self.assertIsNotNone(config.get("entry"))


class ExistingFeatureTestCase(unittest.TestCase):
    def test_feature_one(self):
        self.assertTrue(existing_feature())


class NewFeatureTestCase(unittest.TestCase):
    def setUp(self):
        config["entry"] = "anything"

    def tearDown(self):
        config.clear()

    def test_new_feature(self):
        self.assertTrue(our_new_feature())

In order to understand the behaviour of the existing test, we ran it alone, both with and without the new changes. It appeared that the test was failing in both cases. This gave us the information that this test was using an existing global state, and that we might be cleaning this state. So we took a deeper interest in the tearDown method.

It happened that the global configuration was injected and cleared in our new test suite. This configuration was used but rarely cleared in other tests. As a result, the existing test was relying on the execution of the previous ones to succeed. Clearing the configuration removed the context required by the existing test, thus made it fail.

By “chance” the tests were always run in the right order for years. This situation could have been detected way earlier by using a random execution order for tests. It happens that python has simple modules to do so.

To fix the tests and avoid this situation to happen in the future, we decided to force the configuration clearing in the project’s test suite superclass. This meant to fix a bunch of other tests failing after this but also enforced a clean state for all new tests.

Bonus story: A good flakiness

On top of the previous stories, where flakiness is obviously a bad thing, I also stumbled into a case where I found flakiness somehow beneficial to the codebase.

In this particular case, the test intended to assert some data consistency for any instances of the same class. To do so, the test was generating numerous instances of the class with randomized inputs.

This test happened to fail during some executions as the inputs were creating a behaviour not accounted for by the feature. That enabled to extract a specific test case for the input and fix the behaviour in this case.

While I agree that edge cases should be analyzed during the development process, sometimes the input scope is too wide to consider all of them, let alone test all possibilities.

In those cases, randomizing the input of a method that should keep a consistent output is a good way to assert the codebase sanity in the long run.

Conclusion

This article showed some real-life examples of flaky tests. They were not the worst to track down, but they pinpoint the fact that flakiness can resurface at any time, even years after their introduction!

Once they appear, the flaky tests need to be fixed as soon as possible out of fear that the failing test suite will be considered as a normal state. The developers may then not rely on the tests suites anymore.

Flakiness is mostly due to non-deterministic behaviour in the code, in this article we had an example of:

Specific execution time. If the code is dependent on time, there may be failing tests at specific dates.
Randomness. Using random values in the main code or in the tests needs extra care or the behaviour may vary depending on those random values.
Modified global state. Using a global state in a project can create inconsistencies in tests if the state is not managed correctly.

Some behaviour can help to limit the amount of flakiness that hides in the tests, such as:

Control your test execution environment to keep reproducible execution.
Avoid global states to minimize the side effects of environment settings.
Randomize the test execution order to determine dependencies between tests.

Continue on the subject:

If you are curious about the context that led to the apparition of those flaky tests, my former manager Kevin Deldycke provides a more detailed view in a very interesting post: Billing Pipeline: A Critical Time Sensitive System.

Apache Kafka: Apprentice Cookbook

Antoine Veuiller — Thu, 01 Jul 2021 07:03:57 +0000

Apache Kafka is a distributed event streaming platform built over strong concepts. Let’s dive into the possibilities it offers.

Apache Kafka is a distributed event streaming platform built with an emphasis on reliability, performance, and customization. Kafka can send and receive messages in a publish-subscribe fashion. To achieve this, the ecosystem relies on few but strong basic concepts, which enable the community to build many features solving numerous use cases, for instance:

Processing messages as an Enterprise Service Bus.
Tracking Activity, metrics, and telemetries.
Processing Streams.
Supporting Event sourcing.
Storing logs.

This article will see the concepts backing up Kafka and the different tools available to handle data streams.

Architecture

The behaviour of Kafka is pretty simple: Producers push Messages into a particular Topic, and Consumers subscribe to this Topic to fetch and process the Messages. Let’s see how it is achieved by this technology.

Infrastructure side

Independently of the use, the following components will be deployed:

One or more Producers sending messages to the brokers.
One or more Kafka Brokers, the actual messaging server handling communication between producers and consumers.
One or more Consumers fetching and processing messages, in clusters named Consumer Groups.
One or more Zookeeper instances managing the brokers.
(Optionally) One or more Registry instances uniformizing message schema.

As a scalable distributed system, Kafka is heavily relying on the concept of clusters. As a result, on typical production deployment, there will likely be multiple instances of each component.

A Consumer Group is a cluster of the same consumer application. This concept is heavily used by Kafka to balance the load on the applicative side of things.

Note: The dependency on Zookeeper will be removed soon, Cf. KIP-500

Further Reading:
Design & Implementation Documentation
Kafka Basics and Core Concepts: Explained — Aritra Das

Applicative side

A Message in Kafka is a key-value pair. Those elements can be anything from an integer to a Protobuf message, provided the right serializer and deserializer.

The message is sent to a Topic, which will store it as a Log. The topic should be a collection of logs semantically related, but without a particular structure imposed. A topic can either keep every message as a new log entry or only keep the last value for each key (a.k.a. Compacted log).

To take advantage of the multiple brokers, topics are sharded into Partitions by default. Kafka will assign any received message to one partition depending on its key,
or using a partitioner algorithm otherwise, which results in a random assignment from the developer's point of view. Each partition has a Leader responsible for all I/O operations, and Followers replicating the data. A follower will take over the leader role in case of an issue with the current one.

The partition holds the received data in order, increasing an offset integer for each message. However, there is no order guarantee between two partitions. So for order-dependent data, one must ensure that they end up in the same partition by using the same key.

Each partition is assigned to a specific consumer from the consumer group. This consumer is the only one fetching messages from this partition. In case of shutdown of one customer, the brokers will reassign partitions among the customers.

Being an asynchronous system, it can be hard and impactful on the performances to have every message delivered exactly one time to the consumer. To mitigate this, Kafka provides different levels of guarantee on the number of times a message will be processed (i.e. at most once, at least once, exactly once).

Further Reading:
Log Compacted Topics in Apache Kafka — Seyed Morteza Mousavi
(Youtube) Apache Kafka 101: Replication — Confluent
Replication Design Doc
Processing Guarantees in Details — Andy Briant

Schema and Registry

Messages are serialized when quitting a producer and deserialized when handled by the consumer. To ensure compatibility, both must be using the same data definition. Ensuring this can be hard considering the application evolution. As a result, when dealing with a production system, it is recommended to use a schema to explicit a contract on the data structure.

To do this, Kafka provides a Registry server, storing and binding schema to topics. Historically only Avro was available, but the registry is now modular and can also handle JSON and Protobuf out of the box.

Once a producer sent a schema describing the data handled by its topic to the registry, other parties (i.e. brokers and consumers) will fetch this schema on the registry to validate and deserialize the data.

Further Reading:
Schema Registry Documentation
Kafka tutorial #4-Avro and the Schema Registry— Alexis Seigneurin
Serializer-Deserializer for Schema

Integrations

Kafka provides multiple ways of connecting to the brokers,
and each can be more useful than the others depending on the needs. As a result, even if a library is an abstraction layer above another, it is not necessarily better for every use case.

Kafka library

There are client libraries available in numerous languages which help develop a producer and consumer easily. We will use Java for the example below, but the concept remains identical for other languages.

The producer concept is to publish messages at any moment, so the code is pretty simple.

public class Main {
  public static void main(String[] args) throws Exception {
    // Configure your producer
    Properties producerProperties = new Properties();
    producerProperties.put("bootstrap.servers", "localhost:29092");
    producerProperties.put("acks", "all");
    producerProperties.put("retries", 0);
    producerProperties.put("linger.ms", 1);
    producerProperties.put("key.serializer", "org.apache.kafka.common.serialization.LongSerializer");
    producerProperties.put("value.serializer", "io.confluent.kafka.serializers.KafkaAvroSerializer");
    producerProperties.put("schema.registry.url", "http://localhost:8081");

    // Initialize a producer
    Producer<Long, AvroHelloMessage> producer = new KafkaProducer<>(producerProperties);

    // Use it whenever you need
    producer.send(new AvroHelloMessage(1L, "this is a message", 2.4f, 1));
  }
}

The code is a bit more complex on the consumer part since the consumption loop needs to be created manually. On the other hand, this gives more control over its behaviour. The consumer state is automatically handled by the Kafka library. As a result, restarting the worker will start at the most recent offset he encountered.

public class Main { 
  public static Properties configureConsumer() { 
    Properties consumerProperties = new Properties();

    consumerProperties.put("bootstrap.servers", "localhost:29092");
    consumerProperties.put("group.id", "HelloConsumer");
    consumerProperties.put("key.deserializer", "org.apache.kafka.common.serialization.LongDeserializer");
    consumerProperties.put("value.deserializer", "io.confluent.kafka.serializers.KafkaAvroDeserializer");
    consumerProperties.put("schema.registry.url", "http://localhost:8081");
    // Configure Avro deserializer to convert the received data to a SpecificRecord (i.e. AvroHelloMessage)
    // instead of a GenericRecord (i.e. schema + array of deserialized data).
    consumerProperties.put(KafkaAvroDeserializerConfig.SPECIFIC_AVRO_READER_CONFIG, true);

    return consumerProperties;
  }

  public static void main(String[] args) throws Exception {
    // Initialize a consumer
    final Consumer<Long, AvroHelloMessage> consumer = new KafkaConsumer<>(configureConsumer());

    // Chose the topics you will be polling from.
    // You can subscribe to all topics matching a Regex.
    consumer.subscribe(Pattern.compile("hello_topic_avro"));

    // Poll will return all messages from the current consumer offset
    final AtomicBoolean shouldStop = new AtomicBoolean(false);
    Thread consumerThread = new Thread(() -> {
      final Duration timeout = Duration.ofSeconds(5);

      while (!shouldStop) {
        for (ConsumerRecord<Long, AvroHelloMessage> record : consumer.poll(timeout)) {
          // Use your record
          AvroHelloMessage value = record.value();
        }
        // Be kind to the broker while polling
        Thread.sleep(5);
      }

      consumer.close(timeout);
    });

    // Start consuming && do other things
    consumerThread.start();
    // [...]

    // End consumption from customer
    shouldStop.set(true);
    consumerThread.join();
  }
}

Further Reading:
Available Libraries
Producer Configuration
Consumer Configuration

Kafka Streams

Kafka Streams is built on top of the consumer library. It continuously reads from a topic and processes the messages with code declared with a functional DSL.

During the processing, transitional data can be kept in structures called KStream and KTable, which are stored into topics. The former is equivalent to a standard topic, and the latter to a compacted topic. Using these data stores will enable automatic tracking of the worker state by Kafka, helping to get back on track in case of restart.

The following code sample is extracted from the tutorial provided by Apache. The code connects to a topic named streams-plaintext-input containing strings values, without necessarily providing keys. The few lines configuring the StreamsBuilder will:

Transform each message to lowercase.
Split the result using whitespaces as a delimiter.
Group previous tokens by value.
Count the number of tokens for each group and save the changes to a KTable named counts-store.
Stream the changes in this Ktable to send the values in a KStream named streams-wordcount-output.

public class Main {
  public static void main(String[] args) throws Exception {
    Properties props = new Properties();
    props.put(StreamsConfig.APPLICATION_ID_CONFIG, "streams-wordcount");
    props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:29092");
    props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
    props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.String().getClass());

    final StreamsBuilder builder = new StreamsBuilder();

    builder.<String, String>stream("streams-plaintext-input")
            .flatMapValues(value -> Arrays.asList(value.toLowerCase(Locale.getDefault()).split("\\W+")))
            .groupBy((key, value) -> value)
            .count(Materialized.<String, Long, KeyValueStore<Bytes, byte[]>>as("counts-store"))
            .toStream()
            .to("streams-wordcount-output", Produced.with(Serdes.String(), Serdes.Long()));

    final Topology topology = builder.build();
    final KafkaStreams streams = new KafkaStreams(topology, props);
    final CountDownLatch latch = new CountDownLatch(1);

    // attach shutdown handler to catch control-c
    Runtime.getRuntime().addShutdownHook(new Thread("streams-shutdown-hook") {
      @Override
      public void run() {
        streams.close();
        latch.countDown();
      }
    });

    // The consumer loop is handled by the library
    streams.start();
    latch.await();
  }
}

Further Reading:
Kafka Streams Concepts
Developer Guide
Kafka Stream Work Allocation — Andy Briant

Kafka Connect

Kafka Connect provides a way of transforming and synchronizing data between almost any technology with the use of Connectors. Confluent is hosting a Hub, on which users can share connectors for various technologies. This means that integrating a Kafka Connect pipeline is most of the time only a matter of configuration, without code required. A single connector can even handle both connection sides:

Populate a topic with data from any system: i.e. a Source.
Send data from a topic to any system: i.e. a Sink.

The source will read data from CSV files in the following schema then publish them into a topic. Concurrently, the sink will poll from the topic and insert the messages into a MongoDB database. Each connector can run in the same or a distinct worker, and workers can be grouped into a cluster for scalability.

The connector instance is created through a configuration specific to the library. The file below is a configuration of the MongoDB connector. It asks to fetch all messages from the topic mongo-source to insert them into the collection sink of the database named kafka-connect. The credentials are provided from an external file, which is a feature of Kafka Connect to protect secrets.

{
  "name": "mongo-sink",
  "config": {
    "topics": "mongo-source",
    "tasks.max": "1",
    "connector.class": "com.mongodb.kafka.connect.MongoSinkConnector",
    "connection.uri": "mongodb://${file:/auth.properties:username}:${file:/auth.properties:password}@mongo:27017",
    "database": "kafka_connect",
    "collection": "sink",
    "max.num.retries": "1",
    "retries.defer.timeout": "5000",
    "document.id.strategy": "com.mongodb.kafka.connect.sink.processor.id.strategy.BsonOidStrategy",
    "post.processor.chain": "com.mongodb.kafka.connect.sink.processor.DocumentIdAdder",
    "delete.on.null.values": "false",
    "writemodel.strategy": "com.mongodb.kafka.connect.sink.writemodel.strategy.ReplaceOneDefaultStrategy"
  }
}

Once the configuration complete, registering the connector is as easy as an HTTP call on the running Kafka Connect instance. Afterwards, the service will automatically watch the data without further work required.

$ curl -i -X POST -H "Accept:application/json" -H  "Content-Type:application/json" \
  http://localhost:8083/connectors -d @sink-conf.json

Further Reading:
Getting Started Documentation
Connector Instance API Reference
(Youtube) Tutorials Playlist — Confluent

KSQL Database

Ksql is somehow equivalent to Kafka Streams, except that every transformation is declared in an SQL-like language. The server is connected to the brokers and can create Streams or Tables from topics. Those two concepts behave in the same way as a KStream or KTable from Kafka Streams (i.e. respectively a topic and a compacted topic).

There are three types of query in the language definition:

Persistent Query (e.g. CREATE TABLE <name> WITH (...)): Creates a new stream or table that will be automatically updated.
Pull Query (e.g. SELECT * FROM <table|stream> WHERE ID = 1): Behaves similarly to a standard DBMS. Fetches data as an instant snapshot and closes the connection.
Push Query (e.g. _SELECT * FROM <table|stream> EMIT CHANGES_): Requests a persistent connection to the server, asynchronously pushing updated values.

The database can be used to browse the brokers' content. Topics can be discovered through the command list topics, and their content displayed using print <name>.

ksql> list topics;
 Kafka Topic      | Partitions | Partition Replicas
----------------------------------------------------
 hello_topic_json | 1          | 1
----------------------------------------------------

ksql> print 'hello_topic_json' from beginning;
Key format: KAFKA_BIGINT or KAFKA_DOUBLE or KAFKA_STRING
Value format: JSON or KAFKA_STRING
rowtime: 2021/05/25 08:44:20.922 Z, key: 1, value: {"user_id":1,"message":"this is a message","value":2.4,"version":1}
rowtime: 2021/05/25 08:44:20.967 Z, key: 1, value: {"user_id":1,"message":"this is another message","value":2.4,"version":2}
rowtime: 2021/05/25 08:44:20.970 Z, key: 2, value: {"user_id":2,"message":"this is another message","value":2.6,"version":1}

The syntax to create and query a stream, or a table is very close to SQL.

-- Let's create a table from the previous topic
ksql> CREATE TABLE messages (user_id BIGINT PRIMARY KEY, message VARCHAR) 
> WITH (KAFKA_TOPIC = 'hello_topic_json', VALUE_FORMAT='JSON');

-- We can see the list and details of each table
ksql> list tables;
 Table Name | Kafka Topic      | Key Format | Value Format | Windowed
----------------------------------------------------------------------
 MESSAGES   | hello_topic_json | KAFKA      | JSON         | false
----------------------------------------------------------------------

ksql> describe messages;
Name                 : MESSAGES
 Field   | Type
------------------------------------------
 USER_ID | BIGINT           (primary key)
 MESSAGE | VARCHAR(STRING)
------------------------------------------
For runtime statistics and query details run: DESCRIBE EXTENDED <Stream,Table>;

-- Appart from some additions to the language, the queries are almost declared in standard SQL. 
ksql> select * from messages EMIT CHANGES;
+--------+------------------------+
|USER_ID |MESSAGE                 |
+--------+------------------------+
|1       |this is another message |
|2       |this is another message |

Kafka recommends using a headless ksqlDB server for production, with a file declaring all streams and tables to create. This avoids any modification to the definitions.

Note: ksqlDB servers can be grouped in a cluster like any other consumer.

Further Reading:

Official Documentation

KSQL Query Types In Details

(Youtube) Tutorials Playlist — Confluent

Conclusion

This article gives a broad view of the Kafka ecosystem and possibilities, which are numerous. This article only scratches the surface of each subject. But worry not, as they are all well documented by Apache, Confluent, and fellow developers.
Here are a few supplementary resources to dig further into Kafka:

The complete experimental code is available on my GitHub repository.

Thanks to Sarra Habchi, and Dimitri Delabroye for the reviews

Spring Boot: Apprentice Cookbook

Antoine Veuiller — Mon, 07 Jun 2021 06:54:07 +0000

Spring Boot is a web framework built on top of the framework Spring. It is designed for easier use and quicker implementation. It does so by configuring the application and its environment as automatically as possible. As a newcomer, I can say that it makes the framework really easy to get into.

My learning led me to read most of the reference documentation, which is well written and gives you a lot of insights into the internal behavior of Spring Boot. This documentation gives a lot of details, so this article aims to take the counter approach and pinpoint the concepts you will need to implement an API using Spring Boot. I will complement each section with a set of links to related documentation, may you want to dig further.

As a side note, this document will be using version 2.4.2 of the framework, on a Java project using Gradle as the build system. However, the information remains applicable to any compatible language and build system.

This article will cover the following aspects of creating an API with Spring Boot:

Bootstrap the project
Create REST endpoints
Handle errors
Connect to a persistence layer
Paginate the results
Test the application
Package the Application

Bootstrap the project

This part may be the easiest, as Spring Boot is providing a package generator at https://start.spring.io/. We can select all required modules and retrieve an archived project with the build system, dependencies, and main application class.

Outside of this generator, to declare a RESTful API, our project should define the Spring Boot starter web dependency. The starter dependencies are a set of ready to use features packaged by Spring Boot.

plugins {
  id 'org.springframework.boot' version '2.4.2'
}

dependencies {
  implementation 'org.springframework.boot:spring-boot-starter-web'
}

The application’s main method should be contained in any class, on which we should apply the annotation @SpringBootApplication. This annotation is responsible for a lot of automatic configurations, namely the components injection and web server startup.

@SpringBootApplication
public class MyApplication {
    public static void main(String[] args) {
        SpringApplication.run(MyApplication.class, args);
    }
}

Starting the server is as simple as using the embedded command ./gradlew bootRun. The server will start, but we don’t have any endpoint to serve at the moment.

Documentation links:
@SpringBootApplication
List of starter dependencies

Create a REST endpoint

To create a controller, we simply have to annotate any class with @RestController. We can then configure any method inside this controller as an endpoint using @RequestMapping.

@RequestMapping help us configuring the endpoint by providing an URL, the HTTP verb, the expected data type, and more. It can be applied both on a class and a method, the configurations applied on the class will be inherited by the methods underneath and the path concatenated.

To control our endpoint status codes we will return aResponseEntity, holding both the response message and HttpStatus.

@RestController
@RequestMapping(value = "/hello",
        consumes = MediaType.ALL_VALUE,
        produces = MediaType.APPLICATION_JSON_VALUE)
public class HelloWorldController {

    @RequestMapping(value = "/world", method = RequestMethod.GET)
    public ResponseEntity<Map<String, String>> index() {
        HashMap<String, String> output = new HashMap<>();
        output.put("message", "Hello World!");
        return new ResponseEntity<>(output, HttpStatus.OK);
    }
}

The ResponseEntity will be automatically transformed to an HTTP response, using the HttpStatus as response code and transforming the message to a JSON object. On top of transforming Maps to JSON objects, Spring Boot configure Jackson to map all public attributes or getters of any class to a JSON object.

$ curl -i "localhost:8080/hello/world"
HTTP/1.1 200
{"Hello":"World"}

Documentation links:
@RestController and @RequestMapping
@RequestMapping API doc
Customize Json Serialization

Advanced endpoint configuration

Now that we have a controller, we may want to define dynamic HTTP endpoints. To do so, the main annotations to keep in mind are:

@RequestBody : Defines a body structure through a java Class.
@PathVariable: Defines a variable subpart of the endpoint URL.
@RequestParam : Defines a query parameter.

The controller below showcases the three annotations with two endpoints, each returning a custom “Hello World” depending on the query.

@RestController
@RequestMapping(value = "/hello",
        consumes = MediaType.ALL_VALUE,
        produces = MediaType.APPLICATION_JSON_VALUE)
public class HelloWorldController {

    // The behavior is not representative of a typical POST request
    // and only here as a matter of example.
    @RequestMapping(value = "", method = RequestMethod.POST)
    public ResponseEntity<Map<String, String>> greetFromBody(@RequestBody HelloBody helloBody) {
        HashMap<String, String> output = new HashMap<>();
        output.put("message", "Hello " + helloBody.getName());
        return new ResponseEntity<>(output, HttpStatus.OK);
    }

    @RequestMapping(value = "/{name}", method = RequestMethod.GET)
    public ResponseEntity<Map<String, String>> greet(@PathVariable String name,
                                                     @RequestParam(required = false,
                                                                   defaultValue = "0") int amount_exclamation) {
        HashMap<String, String> output = new HashMap<>();
        StringBuilder b = new StringBuilder("Hello ");
        b.append(name);
        for (int i = 0; i < amount_exclamation; i++) {
            b.append("!");
        }
        output.put("message", b.toString());
        return new ResponseEntity<>(output, HttpStatus.OK);
    }
}

class HelloBody {
    String name;

    public HelloBody() {
        // Used by Jackson
    }

    public String getName() {
        return this.name;
    }
}

The endpoints defined above can be used as follows:

curl -i "localhost:8080/hello/jack?amount_exclamation=4"
HTTP/1.1 200
{"message":"Hello jack!!!!"}

# -d automatically creates a POST request.
$ curl -i "localhost:8080/hello" -d '{"name": "Bob"}' -H "Content-Type: application/json"
HTTP/1.1 200
{"message":"Hello Bob"}

Documentation links:
@RequestBody
@PathVariable
@RequestParam

Handle errors

By default, Spring Boot will return the HTTP code 200 for any successful request, 404 if the endpoint is not registered, and 500 for any error. We already saw that using ResponseEntity enables us to override this behavior for successful requests, but we still need to handle error codes more finely.

To do so, we will define custom API exceptions that will be automatically transformed into HTTP codes. This transformation is done by a class extending ResponseEntityExceptionHandler and annotated with @ControllerAdvice. In this class, we can define methods to handle exceptions using the annotations @ExceptionHandler and @ResponseStatus.

@ControllerAdvice
public class MyApplicationControllerAdvice extends ResponseEntityExceptionHandler {

    @ExceptionHandler(ApiException.class)
    @ResponseStatus(HttpStatus.BAD_REQUEST)
    public void handleBadRequest() {
    }

    @ExceptionHandler(NotFoundException.class)
    @ResponseStatus(HttpStatus.NOT_FOUND)
    public void handleNotFound() {
    }
}

public class ApiException extends Exception {
}

public class NotFoundException extends ApiException {
}

After defining the ControllerAdvice in your project, any exception thrown by your controllers will be parsed and transformed to the bound ResponseStatus.

@RestController
@RequestMapping(value = "/exception")
public class ExceptionController {

    @RequestMapping(value = "/404", method = RequestMethod.GET)
    public ResponseEntity<Map<String, String>> notFound() throws NotFoundException {
        throw new NotFoundException();
    }    

    @RequestMapping(value = "/400", method = RequestMethod.GET)
    public ResponseEntity<Map<String, String>> badRequest() throws ApiException {
        throw new ApiException();
    }

    @RequestMapping(value = "/500", method = RequestMethod.GET)
    public ResponseEntity<Map<String, String>> ise() throws Exception {
        throw new Exception();
    }
}

$ curl -i "localhost:8080/exception/500"
HTTP/1.1 500

$ curl -i "localhost:8080/exception/404"
HTTP/1.1 404

$ curl -i "localhost:8080/exception/400"
HTTP/1.1 400

Our exception handling is very simple and does not return any payload, but it is possible to implement exception parsing in the methods of ResponseEntityExceptionHandler.

Documentation links:
ResponseEntityExceptionHandler
@ControllerAdvice
@ExceptionHandler
@ResponseStatus

Connect to a persistence layer

Configuration

To use a database, we will need the Java Persistence API (JPA) package and the implementation of any persistence layer. The former will install interface APIs, while the latter will provide the implementations and drivers.

To pinpoint the minimal changes required to switch between two distinct databases, we will show the integration with both PostgreSQL and H2 at the same time. First, let’s declare our dependencies:

dependencies {
  implementation 'org.springframework.boot:spring-boot-starter-data-jpa'

  // Dependencies to your used dbms
  implementation 'org.postgresql:postgresql:42.2.1'
  implementation 'com.h2database:h2:1.4.200'
}

The second step is to configure the accesses in application.properties. The property file is the first and the last time we will have to worry about our persistence configuration. In this file, the 3 lines commented out are the only part to change to switch from PostgreSQL to H2.

spring.datasource.username=user
spring.datasource.password=password
spring.datasource.generate-unique-name=true
# Automatically create & update the database schema from code.
spring.jpa.hibernate.ddl-auto=update

#spring.datasource.url=jdbc:h2:mem:database_name
spring.datasource.url=jdbc:postgresql://localhost:5432/database_name

#spring.datasource.driver-class-name=org.h2.Driver
spring.datasource.driver-class-name=org.postgresql.Driver

#spring.jpa.database-platform=org.hibernate.dialect.H2Dialect
spring.jpa.database-platform=org.hibernate.dialect.PostgreSQL10Dialect

Documentation links:
Database configuration
Available properties

Define a model

Defining a model is as simple as using annotations defined on JSR-317. These annotations are available through the package javax.persistence, which is available through the JPA dependency.

For instance, the code below creates a Delivery entity. Our entity identifier is the field id, which will be automatically initialized and increased on each new saved entity in the database thanks to the annotation @GeneratedValue.

Note: All attributes publicly available will be set into the JSON representation of the entity in the API responses.

@Entity
@Table(name = "delivery")
public class Delivery {
    @Id
    @GeneratedValue(strategy = GenerationType.AUTO)
    long id;

    @Column(nullable = false)
    @NotNull
    @Enumerated(EnumType.STRING)
    DeliveryState state;

    @Column(nullable = false)
    @NotNull
    String location;

    public Delivery() {
        // Used by Jackson2
    }

    public Delivery(@NotNull DeliveryState state, @NotNull String location) {
        this.state = state;
        this.location = location;
    }

    public long getId() {
        return id;
    }

    public DeliveryState getState() {
        return state;
    }

    public void setState(DeliveryState state) {
        this.state = state;
    }

    public String getLocation() {
        return location;
    }

    public void setLocation(String location) {
        this.location = location;
    }
}

enum DeliveryState {
    PENDING, DELIVERING, WAITING_AT_ARRIVAL, RETURNING, RETURNED, PICKED_UP;
}

To ensure consistency of our data class, we applied @NotNull validations from JSR-303, these validations can be enforced on endpoints as we will see during the next section. The constraints are contained in the package javax.validation.constraints, available through the dependency spring-boot-starter-validation.

dependencies {
  implementation 'org.springframework.boot:spring-boot-starter-validation'
}

Documentation links:
Entity Declaration
javax.persistence API documentation (@Entity, @Column, @Enumerate, …)
@GeneratedValue
javax.validation.constraints API documentation (@NotNull)

Expose the model

To interact with our models, we have to define a Repository, for instance, a CrudRepository. Doing so is as easy as extending the class with an empty class. Spring Boot will automatically implement functions to interact with the entity.

@Repository
public interface DeliveryRepository extends CrudRepository<Delivery, Long> {
}

We annotate this component @Repository to make it available to dependency injection. Then we can inject and use the repository in any class, for example directly in a controller. Using@Autowired will automatically retrieve the @Repository declared above.

Note: @Repository and @Service behave exactly as the main injection annotation@Component, it simply enables to mark a semantic difference.

@RestController
@RequestMapping(value = "/delivery",
        consumes = MediaType.APPLICATION_JSON_VALUE,
        produces = MediaType.APPLICATION_JSON_VALUE)
public class DeliveryController {

    private final DeliveryRepository deliveryRepository;

    @Autowired
    public DeliveryController(DeliveryRepository deliveryRepository) {
        this.deliveryRepository = deliveryRepository;
    }

    @RequestMapping(value = "", method = RequestMethod.POST)
    public ResponseEntity<Delivery> post(@Valid @RequestBody Delivery delivery) throws ApiException {
        try {
            delivery = deliveryRepository.save(delivery);
        } catch (Exception e) {
            throw new ApiException();
        }
        return new ResponseEntity<>(delivery, HttpStatus.OK);
    }

    @RequestMapping(value = "/{id}", method = RequestMethod.GET)
    public ResponseEntity<Delivery> get(@PathVariable long id) throws ApiException {
        Optional<Delivery> delivery = deliveryRepository.findById(id);
        if (delivery.isEmpty()) {
            throw new NotFoundException();
        }
        return new ResponseEntity<>(delivery.get(), HttpStatus.OK);
    }
}

We used the annotation@Valid to ensure that our constraints defined above are met on the sent Delivery body.

$ curl -i "localhost:8080/delivery" -H 'Content-Type: application/json' \
  -X POST -d '{"state": "PENDING"}'                  
HTTP/1.1 400 

$ curl -i "localhost:8080/delivery/1" -H 'Content-Type: application/json'
HTTP/1.1 404 

$ curl -i "localhost:8080/delivery" -H 'Content-Type: application/json' \
  -X POST -d '{"state": "PENDING", "location":"Budapest"}'
HTTP/1.1 200 
{"id":1,"state":"PENDING","location":"Budapest"}

$ curl -i "localhost:8080/delivery/1" -H 'Content-Type: application/json'                                                                                 130 ↵
HTTP/1.1 200
{"id":1,"state":"PENDING","location":"Budapest"}

Note: H2 is an in-memory database so the data will be wiped out at each server restart.

Documentation Links:
CrudRepository API Documentation
Spring Component Declaration
javax.validation API documentation (@Valid)

Paginate the results

This section illustrates how well Spring Boot integrates some classic features of a web API. To paginate the access to our previous entity Delivery, we simply have to change the repository’s extended class from CrudRepository to PagingAndSortingRepository.

@Repository
public interface DeliveryRepository extends PagingAndSortingRepository<Delivery, Long> {
}

This repository implementation provides a new method findAll(Pageable) returning a Page. The class Pageable configures the page and page size to return.

@RestController
@RequestMapping(value = "/delivery",
        consumes = MediaType.APPLICATION_JSON_VALUE,
        produces = MediaType.APPLICATION_JSON_VALUE)
public class DeliveryController {

    private final DeliveryRepository deliveryRepository;

    @Autowired
    public DeliveryController(DeliveryRepository deliveryRepository) {
        this.deliveryRepository = deliveryRepository;
    }

    @RequestMapping(value = "", method = RequestMethod.GET)
    public ResponseEntity<Page<Delivery>> index(@RequestParam(required = false, defaultValue = "0") int page) {
        Pageable pageable = PageRequest.of(page, 50);
        return new ResponseEntity<>(deliveryRepository.findAll(pageable), HttpStatus.OK);
    }
}

The endpoint will then serve the whole Page object’s data upon request.

$ curl "localhost:8080/delivery" -H 'Content-Type: application/json' | jq                                                                                   4 ↵
{
  "content": [
    {
      "id": 1,
      "state": "PENDING",
      "location": "Budapest"
    }
  ],
  "pageable": {
    "sort": {
      "sorted": false,
      "unsorted": true,
      "empty": true
    },
    "offset": 0,
    "pageNumber": 0,
    "pageSize": 50,
    "paged": true,
    "unpaged": false
  },
  "totalPages": 1,
  "totalElements": 1,
  "last": true,
  "first": true,
  "size": 50,
  "number": 0,
  "sort": {
    "sorted": false,
    "unsorted": true,
    "empty": true
  },
  "numberOfElements": 1,
  "empty": false
}

Documentation links:
PagingAndSortingRepository API Documentation
PageRequest API Documentation
Page API Documentation

Test the application

Spring Boot provides every tool to easily test controllers with a set of APIs and mocks. Mostly, MockMvc will enable us to send requests and assert response content without having to worry about technicalities.

As an example, we are testing the POST endpoint from the section above. One of these tests is successfully creating a Delivery entity, and the second one simulates an error coming from the database.

To avoid relying on a physical instance of a persistence layer, we injected our DeliveryRepository instance using @MockBean, which creates and injects a mock of our component.

@SpringBootTest
@AutoConfigureMockMvc
class DeliveryControllerTest {
    @Autowired
    private MockMvc mvc;

    @MockBean
    DeliveryRepository deliveryRepository;

    @Test
    void testPostDeliveryOk() throws Exception {
        ObjectMapper mapper = new ObjectMapper();
        Map<String, String> delivery = getValidDelivery();
        String body = mapper.writeValueAsString(delivery);
        MockHttpServletRequestBuilder accept =
                MockMvcRequestBuilders.post("/delivery")
                        .accept(MediaType.APPLICATION_JSON)
                        .contentType(MediaType.APPLICATION_JSON)
                        .content(body);
        mvc.perform(accept).andExpect(status().isOk());
    }

    @Test
    void testPostPersistIssue() throws Exception {
        ObjectMapper mapper = new ObjectMapper();
        Map<String, String> delivery = getValidDelivery();
        String body = mapper.writeValueAsString(delivery);
        Mockito.when(deliveryRepository.save(Mockito.any())).thenThrow(new RuntimeException());

        MockHttpServletRequestBuilder accept =
                MockMvcRequestBuilders.post("/delivery")
                        .accept(MediaType.APPLICATION_JSON)
                        .contentType(MediaType.APPLICATION_JSON)
                        .content(body);

        mvc.perform(accept).andExpect(status().is4xxClientError());
    }

    private Map<String, String> getValidDelivery() {
        Map<String, String> delivery = new HashMap<>();
        delivery.put("state", "PENDING");
        delivery.put("location", "Rome");
        return delivery;
    }
}

Documentation links:
@SpringBootTest
@AutoConfiguredMockMvc
@MockBean
MockMvc api Documentation

Package the application

Spring boot also eases the application packaging either as a standalone jar or a docker image.

To create a ready to run fat jar, execute ./gradlew bootJar.
To build a docker image, execute ./gradlew bootBuildImage.

Note that docker does not like uppercase characters in the image name, but we can easily customize the image name and version.

// Only use lowercase on docker image name
tasks.named("bootBuildImage") {
    imageName = "${rootProject.name.toLowerCase()}:${version}"
}

Documentation links:
Create an application fat jar
Configure Docker Image

Conclusion

Spring Boot can be used with a handful of annotations and will manage most of the configuration for you. However, most of the configuration can be overridden to provide your own behavior if necessary. This makes it a good framework to design proof of concepts while keeping room for optimization if the project grows specific needs.

If you want to know more about the framework, I can’t stress enough the quality of the Reference Documentation, which gives really good details.

If you want to play around with some code, you can find all those concepts on an example delivery API on GitHub.

Kubernetes: Apprentice Cookbook

Antoine Veuiller — Tue, 11 May 2021 08:05:08 +0000

You probably already heard of Kubernetes, a powerful orchestrator that will ease deployment and automatically manage your applications on a set of machines, called a Cluster.

With great power comes great complexity, even in the eyes of Google. Thus, learning Kubernetes is oftentimes considered as cumbersome and complex, namely because of the number of new concepts you have to learn. On the other hand, those very same concepts can be found in other orchestrators. As a result, mastering them will ease your onboarding on other orchestrators, such as Docker Swarm.

The aim of this article is to explain the most used concepts of Kubernetes relying on basic system administration concepts, then use some of these to deploy a simple web server and showcase the interactions between the different resources. Lastly, I will lay out the usual CLI interactions while working with Kubernetes.

This article mainly focuses on the developer side of a Kubernetes cluster, but I will leave some resources about cluster administration at the end.

Terminology and concepts

Architecture

The Kubernetes realm is the cluster, everything needed is contained within this cluster. Inside it, you will find two types of nodes: the Control Plane and the Worker Nodes.

The control plane is a centralized set of processes that manages the cluster resources, load balance, health, and more. A Kubernetes cluster usually has multiple controller nodes for availability and load balancing purposes.
As a developer, you will most likely interact through the API gateway for interactions.

The worker node is any kind of host running a local Kubernetes agent Kubelet and a communication process Kube-Proxy. The former handles the operations commanded by the control plane on the local container runtime (e.g. docker), while the latter redirects connectivity to the right pods.

Namespaces

After some time, a Kubernetes cluster may become huge and heavily used. In order to keep things well organized, Kubernetes created the concept of Namespace. A namespace is basically a virtual cluster inside the actual cluster.

Most of the resources will be contained inside a namespace, thus unaware of resources from other namespaces. Only a few kinds of resources are completely agnostic of namespaces, and they define computational power or storage sources (i.e. Nodes and PersistentVolumes). However, access to those can be limited by namespace using Quotas.

Namespace-aware resources will always be contained in a namespace as Kubernetes creates and uses a namespace named default if nothing is specified.

There is no silver bullet on the way to use namespaces, as it widely depends on your organization and needs. However, we can note some usual namespaces usages:

Divide the cluster by team or project, to avoid naming conflict and help repartition of resources.
Divide the cluster by environment (i.e. dev, staging, prod), to keep a consistent architecture.
Deploy with more granularity (e.g. blue/green deployment), to quickly fall back on an untouched working environment in case of issue.

Further reading:

Namespace Documentation

Manage The Cluster Namespaces

Glossary

Kubernetes did a great work of remaining agnostic of any technology in their design. This means two things: handle multiple technologies under the hood and there is a whole new terminology to learn.

Fortunately, these concepts are pretty straightforward and can most of the time be compared to a unit element of classic system infrastructure. The table below will summarize the binding of the most basic concepts. The comparison might not be a hundred per cent accurate but rather here to help understand the need behind each concept.

Abstraction Layer	Physical Layer	Uses Namespace	Description
Pod	Container	✅	A Pod is the minimal work unit of Kubernetes, it is generally equivalent to one applicative container but it can be composed of multiple ones.
Replicaset	Load Balancing	✅	A ReplicaSet keeps track of and maintain the amount of instances expected and running for a given pod.
Deployment	-	✅	A Deployment keeps track of and maintain the required configuration for a pod and replicaset.
StatefulSet	-	✅	A StatefulSet is a Deployment with insurance on the start order and volume binding, to keep state consistent in time.
Node	Host	❌	A Node can be a physical or virtual machine that is ready to host pods.
Service	Network	✅	A Service will define an entrypoint to a set of pods semantically tied together.
Ingress	Reverse Proxy	✅	An Ingress publishes Services outside the Cluster.
Cluster	Datacenter	❌	A Cluster is the set of available nodes, including the Kubernetes controllers.
Namespace	-	➖	A Namespace defines an isolated pseudo cluster in the current cluster.
StorageClass	Disk	❌	A StorageClass configures filesystems sources that can be used to dynamically create PersistentVolumes.
PersistentVolume	Disk Partition	❌	A PersistentVolume describe any kind of filesystem ready to be mounted on a pod.
PersistentVolumeClaim	-	✅	A PersistentVolumeClaim binds a PersistentVolume to a pod, which can then actively use it while running.
ConfigMap	Environment Variables	✅	A ConfigMap defines widely accessible properties.
Secret	Secured Env. Var.	✅	A Secret defines widely accessible properties with potential encryption and access limitations.

Further reading:

Official Kubernetes Glossary

Official Concepts Documentation

Definition files

The resources in Kubernetes are created in a declarative fashion, and while it is possible to configure your application deployment through the command line, a good practice is to keep track of the resource definitions in a versioned environment. Sometimes named GitOps, this practice is not only applicable for Kubernetes but widely applied for delivery systems, backed up by the DevOps movement.

To this effect, Kubernetes proposes a YAML representation of the resource declaration, and its structure can be summarized as follow:

Field	File type	Content
`apiVersion`	All files	Version to use while parsing the file.
`kind`	All files	Type of resource that the file is describing.
`metadata`	All files	Resource identification and labeling.
`data`	Data centric files (Secret, ConfigMap)	Content entry point for data mapping.
`spec`	Most files (Pod, Deployment, Ingress, ...)	Content entry point for resource configuration.

Watch out: some resources such as StorageClass do no use a single entry point as described above

Further reading:

Guide on apiVersion

Yaml Specifications

Metadata and labels

The metadata entry is critical while creating any resource as it will enable Kubernetes and yourself to easily identify and select the resource.

In this entry, you will define a name and a namespace (defaults to default), thanks to which the control plane will automatically be able to tell if the file is a new addition to the cluster or the revision of a previously loaded file.

On top of those elements, you can define a labels section.
It is composed of a set of key-value pairs to narrow down the context and content of your resource. Those labels can later be used in almost any CLI commands through Selectors. As those entries are not used in the core behavior of Kubernetes, you can use any name you want, even if Kubernetes defines some best practices recommendations.

Finally, you can also create an annotations section, which is almost identical to labels but not used by Kubernetes at all. Those can be used on the applicative side to trigger behaviors or simply add data to ease debugging.

# <metadata> narrows down selection and identify the resource
metadata:
  # The <name> entry is required and used to identify the resource
  name: my-resource
  namespace: my-namespace-or-default
  # <labels> is optional but often needed for resource selection
  labels:
    app: application-name
    category: back
  # <annotations> is optional and not needed for the configuration of Kubernetes
  annotations:
    version: 4.2

Further reading:

Naming and Identification

Labels and Selectors

Annotations

Data centric configuration files

Those files define key-value mappings that can be used later in other resources. Usually, those resources (i.e. Secrets and ConfigMap) are loaded before anything else, as it is more likely than not that your infrastructure files are dependent on them.

apiVersion: v1
# <kind> defines the resource described in this file
kind: ConfigMap
metadata:
  name: my-config
data:
  # <data> configures data to load
  configuration_key: "configuration_value"
  properties_entry: |
    # Any multiline content is accepted
    multiline_config=true

Infrastructure centric configuration files

Those files define the infrastructure to deploy on the cluster, potentially using content from the data files.

apiVersion: v1
# <kind> defines the resource described in this file
kind: Pod
metadata:
  name: my-web-server
spec:
  # <spec> is a domain specific description of the resource.
  # The specification entries will be very different from one kind to another

Resources definition

In this section, we will take a closer look at the configuration of the most used resources on a Kubernetes application. This is also the occasion to showcase the interactions between resources.

At the end of the section, we will have a running Nginx server and will be able to contact the server from outside the cluster. The following diagram summarizes the intended state:

ConfigMap

ConfigMap is used to hold properties that can be used later in your resources.

apiVersion: v1
kind: ConfigMap
metadata:
  name: simple-web-config
  namespace: default
data:
  configuration_key: "Configuration value"

The configuration defined above can then be selected from another resource definition with the following snippet:

valueFrom:
  configMapKeyRef:
    name: simple-web-config
    key: configuration_key

Note: ConfigMaps are only available in the namespace in which they are defined.

Further reading:

ConfigMap Documentation

Secret

All sensitive data should be put in Secret files (e.g. API keys, passphrases, …). By default, the data is simply held as base64 encoded values without encryption. However, Kubernetes proposes ways of mitigating leakage risks by integrating a Role-Based Access Control or encrypting secrets.

The Secret file defines a type key at its root, which can be used to add validation on the keys declared in the data entry. By default, the type is set to Opaque which does not validate the entries at all.

apiVersion: v1
kind: Secret
metadata:
  name: simple-web-secrets
# Opaque <type> can hold generic secrets, so no validation will be done.
type: Opaque
data:
  # Secrets should be encoded in base64
  secret_configuration_key: "c2VjcmV0IHZhbHVl"

The secret defined above can then be selected from another resource definition with the following snippet:

valueFrom:
  secretKeyRef:
    name: simple-web-secrets
    key: secret_configuration_key

Note: Secrets are only available in the namespace in which they are defined.

Further reading:

Secrets Documentation

Available Secret Types

Pod

A Pod definition file is pretty straightforward but can become pretty big due to the quantity of configuration available. The name and image fields are the only mandatory ones, but you might commonly use:

ports to define the ports to open on both the container and pod.
env to define the environment variables to load on the container.
args and entrypoint to customize the container startup sequence.

Pods are usually not created as standalone resources on Kubernetes, as the best practice indicates to use pod as part of higher level definition (e.g. Deployment). In those cases, the Pod file's content will simply be embedded in the other resource's file.

apiVersion: v1
kind: Pod
metadata:
  name: my-web-server
spec:
  # <containers> is a list of container definition to embed in the pod
  containers:
    - name: web
      image: nginx
      ports:
        - name: web
          containerPort: 80
          protocol: TCP
      env:
        - name: SOME_CONFIG
          # Create a line "value: <config_entry>" from the ConfigMap data
          valueFrom:
            configMapKeyRef:
              name: simple-web-config
              key: configuration_key
        - name: SOME_SECRET
          # Create a line "value: <config_entry>" from the Secret data
          valueFrom:
            secretKeyRef:
              name: simple-web-secrets
              key: secret_configuration_key

Note: Pods are only available in the namespace in which they are defined.

Further reading:

Pod Documentation

Advanced Pod Configuration

Fields available in Pod entry

Fields available in Pod entry

Deployment

The Deployment is generally used as the atomic working unit since it will automatically:

Create a pod definition based on the template entry.
Create a ReplicaSet on pods selected by the selector entry, with the value of replicas as a count of pods that should be running.

The following file requests 3 instances of an Nginx server running at all times. The file may look a bit heavy, but most of it is the Pod definition copied from above.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-web-server-deployment
  namespace: default
  labels:
    app: webserver
spec:
  # <selector> should retrieve the Pod defined below, and possibly more
  selector:
    matchLabels:
      app: webserver
      instance: nginx-ws-deployment
  # <replicas> asks for 3 pods running in parallel at all time
  replicas: 3
  # The content of <template> is a Pod definition file, without <apiVersion> nor <kind>
  template:
    metadata:
      name: my-web-server
      namespace: default
      labels:
        app: webserver
        instance: nginx-ws-deployment
    spec:
      containers:
        - name: web
          image: nginx
          ports:
            - name: web
              containerPort: 80
              protocol: TCP
          env:
            - name: SOME_CONFIG
              # Create a line "value: <config_entry>" from the ConfigMap data
              valueFrom:
                configMapKeyRef:
                  name: simple-web-config
                  key: configuration_key
            - name: SOME_SECRET
              # Create a line "value: <config_entry>" from the Secret data
              valueFrom:
                secretKeyRef:
                  name: simple-web-secrets
                  key: secret_configuration_key

Note: Deployments are only available in the namespace in which they are defined.

Further reading:

Deployment Documentation

Service

A pod might be deleted and recreated at any time. When it occurs the pod's IP address will change, which could result in a loss of connection if you are directly contacting it. To solve this issue, a Service provides a stable contact point to a set of Pods, while remaining agnostic of their state and configuration. Usually, Pods are chosen to be part of a Service through a selector entry, thus based on its labels. A Pod is selected if and only if all the labels in the selector are worn by the pod.

There are three types of services that are acting quite differently, among which you can select using the type entry.

The ClusterIP service is bound to an internal IP from the cluster, hence only internally reachable. This is the type of service created by default and is suitable for binding different applications inside the same cluster.

A NodePort service will bind a port (by default in range 30000 to 32767) on the nodes hosting the selected pods. This enables you to contact the service directly through the node IP. That also means that your service will be as accessible as the virtual or physical machines hosting those pods.

Note: Using NodePort can pose security risks, as it enables a direct connection from outside the cluster.

A LoadBalancer service will automatically create a load balancer instance from the cloud service provider on which the cluster is running. This load balancer is created outside the cluster but will automatically be bound to the nodes hosting the selected pods.

This is an easy way to expose your service but can end up being costly as each service will be managed by a single load balancer.

If you are setting up your own Ingress as we will do here, you may want to use a ClusterIp service, as other services are made for specific use cases.

apiVersion: v1
kind: Service
metadata:
  name: simple-web-service-clusterip
spec:
  # ClusterIP is the default service <type>
  type: ClusterIP
  # Select all pods declaring a <label> entry "app: webserver"
  selector:
    app: webserver
  ports:
    - name: http
      protocol: TCP
      # <port> is the port to bind on the service side
      port: 80
      # <targetPort> is the port to bind on the Pod side
      targetPort: 80

Note: Services are defined in a namespace but can be contacted from other namespaces.

Further reading:

Service Documentation

In Depth Service Comparison

Create an External Load Balancer

Ingress

Ingress enables you to publish internal services without necessarily using a load balancer from cloud service providers. You usually need only one ingress per namespace, where you can bind as many routing rules and backends as you want. A backend will typically be an internally routed ClusterIP service.

Please note that Kubernetes does not handle ingress resources by itself and relies on third-party implementations. As a result, you will have to choose and install an Ingress Controller before using any ingress resource. On the other hand, it makes the ingress resource customizable depending on the needs of your cluster.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: simple-web-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
    # Using <host> redirects all request matching the given DNS name to this rule
    - host: "*.minikube.internal"
      http:
        paths:
          - path: /welcome
            pathType: Prefix
            backend:
              service:
                name: simple-web-service-clusterip
                port:
                  number: 80
    # All other requests will be redirected through this rule
    - http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: simple-web-service-clusterip
                port:
                  number: 80

Note: Ingresses are defined in the namespace but may contact services from other namespaces and are publicly accessible outside the cluster.

Further reading:

Ingress Documentation

Available Ingress Controllers

Enable Ingress on Minikube

Nginx Ingress Annotations

CLI Usage

Create and manage resources

This section showcases the basic CLI commands to manipulate resources. As said before, while it is possible to manually manage resources, a better practice is to use files.

# <kind> is the type of resource to create (e.g. deployment, secret, namespace, quota, ...)
$ kubectl create <kind> <name>
$ kubectl edit   <kind> <name>
$ kubectl delete <kind> <name>

# All those commands can be used through a description file.
$ kubectl create -f <resource>.yaml
$ kubectl edit   -f <resource>.yaml
$ kubectl delete -f <resource>.yaml

To ease resources manipulations through files, you can reduce the interactions to the CLI to the two following commands:

# Create and update any resource
$ kubectl apply   -f <resource>.yaml
# Delete any resource
$ kubectl delete  -f <resource>.yaml

Further reading:

Managing Resources

Monitor and Debug

Fetch resources

You can see all resources running through the CLI using kubectl get <kind>. This command is pretty powerful and lets you filter the kind of resources to display or select the resources you want to see.

Note: if not specified, Kubernetes will work on the default namespace. You can specify -n <namespace> to work on a specific namespace or -A to show every namespace.

# Fetch everything
$ kubectl get all
NAME                                            READY   STATUS    RESTARTS   AGE
pod/my-web-server-deployment-58c4fd887f-5vm2b   1/1     Running   0          128m
pod/my-web-server-deployment-58c4fd887f-gq6lr   1/1     Running   0          128m
pod/my-web-server-deployment-58c4fd887f-gs6qb   1/1     Running   0          128m

NAME                                   TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
service/simple-web-service-clusterip   ClusterIP      10.96.96.241     <none>        80/TCP,443/TCP               60m
service/simple-web-service-lb          LoadBalancer   10.108.182.232   <pending>     80:31095/TCP,443:31940/TCP   60m
service/simple-web-service-np          NodePort       10.101.77.203    <none>        80:31899/TCP,443:31522/TCP   60m

NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/my-web-server-deployment   3/3     3            3           136m

NAME                                                  DESIRED   CURRENT   READY   AGE
replicaset.apps/my-web-server-deployment-58c4fd887f   3         3         3       128m

# We can ask for more details
$ kubectl get deployment -o wide
NAME                       READY   UP-TO-DATE   AVAILABLE   AGE    CONTAINERS   IMAGES  SELECTOR
my-web-server-deployment   3/3     3            3           121m   web          nginx   app=webserver

# Some resources are not visible using "all" but available
$ kubectl get configmap
NAME                DATA   AGE
kube-root-ca.crt    1      38d
simple-web-config   3      3h17m

Dig into a particular resource

This section will show you how to dig into resources. Most of the required day-to-day operations are doable through the three following commands.

The first command will give you the resource's complete configuration, using kubectl describe <kind>/<name>.

# Let's describe the ingress for the sake of example
$ kubectl describe ingress/simple-web-ingress
Name:             simple-web-ingress
Namespace:        default
Address:          192.168.64.2
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
  Host                 Path  Backends
  ----                 ----  --------
  *.minikube.internal
                       /welcome   simple-web-service-clusterip:80 (172.17.0.4:80,172.17.0.5:80,172.17.0.6:80 + 1 more...)
  *
                       /   simple-web-service-clusterip:80 (172.17.0.4:80,172.17.0.5:80,172.17.0.6:80 + 1 more...)
Annotations:           nginx.ingress.kubernetes.io/rewrite-target: /
Events:
  Type    Reason  Age                 From                      Message
  ----    ------  ----                ----                      -------
  Normal  UPDATE  7m6s (x6 over 23h)  nginx-ingress-controller  Ingress default/simple-web-ingress

Another important command is kubectl logs <kind>/<name>, as you might expect it shows you the resources' logs if applicable. As the logs are produced by Pods, running such a command on a resource above a Pod will dig through Kubernetes to display the logs of a randomly chosen Pod underneath it.

$ kubectl logs deployments/my-web-server-deployment
Found 3 pods, using pod/my-web-server-deployment-755b499f77-4n5vn
# [logs]

Finally, it is sometimes useful to connect on a pod, you can do so with the command kubectl exec -it <pod_name> -- /bin/bash. This will open an interactive shell on the pod, enabling you to interact with its content.

# As for logs, when called on any resource enclosing Pods,
# Kubernetes will randomly chose one to  execute the action
$ kubectl exec -it deployment/my-web-server-deployment -- /bin/bash
root@my-web-server-deployment-56c4554cf9-qwtm6:/# ls
# [...]

Conclusion

During this article, we saw the fundamentals behind deploying and publishing stateless services using Kubernetes. But you can do a lot more complex things with Kubernetes. If you want to learn more about it, I can recommend you to look at these resources:

Read the Kubernetes reference documentation.
Install a sandbox locally with Minikube, and play with it.
Watch the video Kubernetes Tutorial for Beginners - TechWorld with Nana.
Manually bootstrap a Kubernetes cluster: Kubernetes The Hard Way.

Incidentally, there are multiple subjects I could not deeply talk about in this article and that may be of interest.

On the developer side:

On the cluster administrator side:

Operators
Access control
Secret encryption
Quotas
Network Plugins (e.g. VPP and Weaveworks)

Furthermore, if you are interested in the ecosystem around Kubernetes, you may want to take a look at the following technologies:

Openshift is wrapping Kubernetes with production friendly features.
Helm is a charts manager for Kubernetes helping improve re-usability of configuration files.
ArgoCD is keeping your Kubernetes Cluster up to date with your configurations from Git.

Appendix

Resources' repository

The resources definitions used in this article are available in the following GitHub repository.

CLI equivalents - Docker and Kubernetes

Managing containers with Docker and pods with Kubernetes is very similar,
as you can see on the following table describing equivalent operations between both technologies.

Operation	Docker	Kubernetes
Running containers	`docker ps`	`kubectl get pods`
Configuration details	`docker inspect <name>`	`kubectl describe <name>`
Show logs	`docker logs <name>`	`kubectl logs <name>`
Enter container	`docker exec -it <name> /bin/bash`	`kubectl exec -it <name> -- /bin/bash`

Thanks to Sarra Habchi, Dimitri Delabroye, and Alexis Geffroy for the reviews