DEV Community: Philipp Gysel

Learn Aspect Oriented Programming by Example

Philipp Gysel — Mon, 01 Feb 2021 12:04:33 +0000

This post offers a great way for you to learn Aspect Oriented Programming by studying concrete examples. In particular, I will showcase SpringBoot AOP by implementing 4 Aspects.

Table of Contents:

What is an Aspect?
@Cacheable: a standard Spring Advice
Log REST calls (with a custom Aspect)
Performance monitoring (with AOP)
Retry mechanism (with AOP)

If you’re the person who wants to skip lengthy descriptions and just look at concrete code, I’ve got you covered:

pmgysel / aop-examples

Some examples of Aspect Oriented Programming (AOP) with SpringBoot

What is an Aspect?

So there are some great resources out there for an overview of Spring AOP, including this Baeldung article and the official Spring AOP documentation. But since we don’t wanna focus on boring theory and rather keep things practical, here’s a really short summary how AOP works:

We’ll need the following terms in this tutorial:

Advice: the method which implements some common task like logging or caching
Pointcut: a pattern expression which matches the places where your Advice should be invoked
Aspect: The Advice plus the Pointcut expression
Bonus - Join point: All places in your code that represent candidates for a Pointcut

`@Cacheable`: a standard Spring Advice

Let’s start simple and consider an already implemented Advice by Spring, namely the @Cacheable annotation. Say your web service must compute numbers of the Fibonacci series.

If you don’t know what the Fibonacci series is: it’s the series starting with 0 and 1 and each consecutive number is the sum of the previous two numbers.

We implement the Fibonacci computation in a @Service class:

@Service
public class FibonacciService {
  public Long nthFibonacciTerm(Long n) {
    if (n == 1 || n == 0) {
      return n;
    }
    return nthFibonacciTerm(n-1) + nthFibonacciTerm(n-2);
  }
}

Next, we use this service class in our REST controller:

@RestController
public class WebController {
  @Autowired private final FibonacciService fibonacciService;

  @GetMapping(path = "/api/fibonacci/{number}")
  public Long fibonacci(@PathVariable(value = "number") Long number) {
    return fibonacciService.nthFibonacciTerm(number);
  }
}

Our implementation is recursive and thus rather slow. So how do you make your web service faster? One way would be to use a faster algorithm, but let’s solve the problem with Spring’s @Cacheable feature. This annotation creates a cache in the background where all previous results get stored. All we must do, is add the @Cacheable annotation to our method:

@Cacheable("Fibonacci")
@GetMapping(path = "/api/fibonacci/{number}")
public Long fibonacci(@PathVariable(value = "number") Long number) { ... }

Now we’re ready to test our caching mechanism by firing a REST request to http://localhost:8080/api/fibonacci/40. I tried to compute the 40th Fibonacci on my own laptop and here are the results:

First REST call: 1902ms
Second REST call: 1ms

Pretty good result eyyy🤙😎

One last thing I’d like to mention: in order to activate Spring’s cacheable feature, you have to add @EnableCaching to a @Configuration class.

Log REST calls with a custom Aspect

That was pretty easy right? So let’s move on to a more advanced use case: now we create a custom Aspect!

Our goal is to create a log message every time some REST method gets called. Since we might wanna add this functionality to future REST methods too, we want to generalize this task in an Aspect:

@Before("@annotation(com.example.aop.LogMethodName)")
public void logMethodName(JoinPoint joinPoint) {
  String method = joinPoint.getSignature().getName();
  String params = Arrays.toString(joinPoint.getArgs());
  System.out.println("Method [" + method + "] gets called with parameters " + params);
}

The first line defines the Pointcut expression, and the subsequent method represents the Advice. Let’s break the two down one by one:

Pointcut:
The Pointcut expression defines the places where our Advice is inserted to. In our case, the Aspect is applied before every method with a @LogMehtodName annotation. Note that @LogMethodName is our custom annotation which we use as Pointcut marker.

Advice:
The advice method is the piece of logic that generalizes a task common to many different objects. In our case, the Advice finds the originating method’s name as well as its calling parameters and logs them to the console.

With our Aspect in place, there are three additional code lines required to get everything working:

First, add the marker @LogMethodName to our fibonacci() method
Second, we have to add @Aspect to the class containing our Aspect
Third, enable Spring’s Aspect scanning with @EnableAspectJAutoProxy in any @Configuration class

That’s it, we’ve implemented our own Advice!🙌 Let’s run a test! We fire a REST request to the web service to compute the 40th Fibonacci number and have a look at the console output:

Method [fibonacci] gets called with parameters [40]

It goes without saying that such log messages will be of great help if you ever must track down bugs in your application.

Performance monitoring with AOP

In the previous example, we used a Pointcut expression of type @Before - here, the Advice runs before the actual method. Let’s switch gears and implement an @Around Pointcut. Such an Advice runs partly before the target method and partly after it.

Our goal now is to monitor the execution time of any REST call. Let’s go ahead and implement the monitoring requirement in a generalized fashion, namely an Aspect:

@Around("@annotation(com.example.aop.MonitorTime)")
public Object monitorTime(ProceedingJoinPoint joinPoint) throws Throwable {
  long startTime = System.currentTimeMillis();
  Object proceed = joinPoint.proceed();
  long duration = System.currentTimeMillis() - startTime;
  System.out.println("Execution took [" + duration + "ms]");
  return proceed;
}

Pointcut:
Like before, we create a new custom annotation @MonitorTime for marking our Pointcuts.

Advice:
An @Around Aspect should have an argument of type ProceedingJoinPoint. This type has a proceed() method which triggers the execution of the actual target method. So in our Advice, we first query the current time in milliseconds. After the target method is executed, we measure the current time again, and from there we can compute time difference.

Let’s go ahead and mark our target method with the @MonitorTime annotation:

@MonitorTime
@LogMethodName
@Cacheable("Fibonacci")
@GetMapping(path = "/api/fibonacci/{number}")
public Long fibonacci(@PathVariable(value = "number") Long number) { ... }

By now, our REST method has quite some Pointcut markers attached to it😉 Anyways, let’s go ahead and test our performance monitoring feature. As before, we compute the 40th Fibonacci number:

Method [fibonacci] gets called with parameters [40]
Execution took [1902ms]

As you can see, this particular REST call took 1902ms. With this @Around Aspect in place, you’re definitely an advanced AOP programmer!💪

Retry mechanism with AOP

Distributed systems can experience concurrency issues. One such example would be when two web service instances are simultaneously trying to access the same record in a database. Oftentimes, such a lock problem can be resolved by retrying the operation. The only requirement here is that the operation is idempotent.

Let’s go ahead and create an Aspect which transparently retries an operation until it succeeds:

@Around("@annotation(com.example.aop.RetryOperation)")
public Object doIdempotentOperation(ProceedingJoinPoint joinPoint) throws Throwable {
  int numAttempts = 0;
  RuntimeException exception;
  do {
    try {
      return joinPoint.proceed();
    } catch(RuntimeException e) {
      numAttempts++;
      exception = e;
    }
  } while(numAttempts < 100);
  throw exception;
}

Pointcut:
Our Advice runs around any method with the custom annotation @RetryOperation.

Advice:
In the try statement, we run the target method. This method might throw a RuntimeException. If this happens, we increment the numAttempts counter and simply rerun the target method. As soon as the target method succeeds, we exit the Advice.

For demonstration purposes, let’s create a REST method for storing a String. This method will fail 50% of the time:

@RetryOperation
@LogMethodName
@PostMapping(path = "/api/storeData")
public void storeData(@RequestParam(value = "data") String data) {
  if (new Random().nextBoolean()) {
    throw new RuntimeException();
  } else {
    System.out.println("Pretend everything went fine");
  }
}

Thanks to our @RetryOperation annotation, the above method will be retried until it succeeds. Moreover, we use our @LogMethodName annotation so we can see every method invocation. Let’s go ahead and test our new REST endpoint; for this purpose we fire a REST request to localhost:8080/api/storeData?data=hello-world.

Method [storeData] gets called with parameters [hello-world]
Method [storeData] gets called with parameters [hello-world]
Method [storeData] gets called with parameters [hello-world]
Pretend everything went fine

In the above case, the operation failed 2 times and only succeeded on the third try.

Conclusion

Congrats, you’re a professional AOP programmer now🥳🚀 You can find a fully working web service with all Aspects on my Github repo:

pmgysel / aop-examples

Some examples of Aspect Oriented Programming (AOP) with SpringBoot

Thanks so much for reading, please leave a comment if you have any questions or feedback!

Aspect Oriented Programming with Java and SpringBoot

Philipp Gysel — Wed, 06 Jan 2021 13:26:26 +0000

This blog covers the basics of Aspect Oriented Programming (AOP). I’ll show a simple working example of AOP: a common task like logging of REST calls can be generically defined in an Advice, which can then be applied to various target places in our code (so called Pointcuts) via an Aspect.

For the impatient: you can find the end result on my Github repo🏎️

pmgysel / aop-demo

Demo for Aspect Oriented Programming

Programming Paradigms

Back in the 1996, when Java 1.0 was released, Java developers were all hyped up about object oriented programming, also called OOP. While OOP was one of the foundational drivers for Java, the programming language itself has gone a long way since then and now supports many programming paradigms.

Here’s a list of the primary programming paradigms supported in Java (in historical order):

Procedural programming
Object oriented programming (OOP)
Functional programming
Aspect oriented programming (AOP)

For this post, we focus on AOP and show how to create Aspects. You’ll learn both the basics of AOP as well as how to use it with SpringBoot.

Why AOP?

Most bigger companies have programming guidelines, and so does mine. One of our guidelines states that every REST endpoint execution must be logged (name of the Java method + the parameters).

Here’s how you could solve this:

@RestController
public class MyRestController {
  @GetMapping(path = "/api/hello/")
  public String hello() {
    System.out.println("Method [hello] gets called with 0 parameters");
    return "Hello world!";
  }
}

The code snippet above does the following:

@RestController: Make sure SpringBoot knows this class contains REST endpoints
@GetMapping: A method which replies to HTTP GET requests
System.out.println(...): adhere to the aforementioned coding guidelines
return value: the method simply returns a greeting message of type String

In a realistic app, you will have many such REST calls, in many different classes. Doing the exact same logging in all those methods is cumbersome. Moreover, if the coding guidelines slightly change, you will have to change the logging message in each method.

Here’s where AOP comes to the rescue: with the help of AOP, we can nicely add common functionality to many different places in our code, without interfering with existing code. In literature jargon, AOP is all about the separation of cross-cutting concerns.🤓 In more human understandable language, AOP enables the modularization of common tasks across different objects.😎

Maven Dependency

To start using AOP in SpringBoot with AspectJ annotations, we need to import the following dependencies in our pom.xml:

<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-aop</artifactId>
</dependency>
<dependency>
  <groupId>org.aspectj</groupId>
  <artifactId>aspectjweaver</artifactId>
</dependency>

Advice

Let’s create a function which performs our required logging in a generic way:

public void logMethodExecution(JoinPoint joinPoint) {
  String method = joinPoint.getSignature().getName();
  String params = Arrays.toString(joinPoint.getArgs());
  System.out.println("Method [" + method + "] gets called with parameters " + params);
}

This generic function is called an Advice. Note that it can log the name and parameters of any method; let’s break the Advice down step by step:

JoinPoint: this object contains all information about the Join point, that is, the "location" where our Aspect will be inserted. In our case, this would be the REST method for which we want to create a log message for.
joinPoint.getSignature() and joinPoint.getArgs() extracts the method signature as well as the calling arguments
System.out.println(...): do the necessary logging

Pointcut

So where do we want to insert the above Advise method? Well, we want each REST endpoint to be tracked. While there are many ways to mark our REST endpoints, we choose to use a custom annotation to define the Pointcut:

@Before("@annotation(LogMethod)")
public void logMethodExecution(JoinPoint joinPoint) {...}

As you can see, the Pointcut definition is just a one-liner:

@Before: we run the Advice before the REST call gets answered
@annotation: we mark Pointcuts via an annotation
LogMethod: this is the name of our custom annotation

Now we’re ready to mark our REST method with our custom annotation:

@LogMethod
@GetMapping(path = "/api/hello/")
public String hello() {
  return "Hello world!";
}

Note that we prepended the REST method with the annotation @LogMethod. Moreover, we removed the logging inside the method, this is now done by our Aspect.

Aspect

An Aspect is a Pointcut plus an Advice. So, let’s put the two together, and we get:

@Aspect
@Component
public class LoggingAspect {

  @Before("@annotation(LogMethod)")
  public void logMethodName(JoinPoint joinPoint) {
    String method = joinPoint.getSignature().getName();
    String params = Arrays.toString(joinPoint.getArgs());
    System.out.println("Method [" + method + "] gets called with parameters " + params);
  }
}

Here’s what we have:

@Aspect: SpringBoot expects all Aspects to be in classes annotated with @aspect
@Before(...): the Pointcut
logMethodName(...){...}: the Advice

So all we did here was just bring together the previously shown Pointcut expression plus the Advice, and wrap everything in a class. Bring out the champagne, we have our Aspect all finished and working 🥂

Enable AOP

To wrap up, we have to enable AspectJ for our Spring configuration:

@Configuration
@EnableAspectJAutoProxy
public class AspectConfig {

}

Remember that we want to be able to work with Beans when using Spring. At the moment, our @RestController class only contains the REST-call logic, but not our Advice. Spring can create Proxies for such Beans which contain this additional logic (the Advice), and this is enabled by @EnableAspectJAutoProxy.

Conclusion

That’s all! You now have a fully working example of AOP😀💪🍾

We implemented an Advice to be run anytime a method with annotation @LogMethod is executed. Thanks to our AOP approach, we can add this annotation to future REST methods which will then be advised with the same Aspect!

Checkout the fully working example on Github:

pmgysel / aop-demo

Demo for Aspect Oriented Programming

This is a simple web service which uses AOP. The REST methods are advised by multiple Aspects.

Dependencies

Java JDK 15
Maven:
- spring-boot-starter-aop
- aspectjweaver

Aspects for REST calls

Log method name and parameters
Log duration of method

Usage

Compile and run web service:

mvn clean install
mvn spring-boot:run

Sample REST calls:

GET http://localhost:8080/api/greeting/{name}
GET http://localhost:8080/api/order/{menu}

Sample aspect output

Method [greeting] gets called with parameters [John]
Exeution took [21ms]

View on GitHub

This Github repo also contains a second advice of type @Around: Each time a REST method is called, we log the execution time. This can come in handy to measure, monitor and compare the performance of different REST endpoints.

Thanks for reading, please leave a comment if you have any feedback or questions!😀

From Java 8 to Java 15 in Ten Minutes

Philipp Gysel — Mon, 07 Dec 2020 15:13:51 +0000

This blog will give you samples of awesome new feature added since Java 7. I’ll showcase at least one major improvement for each Java version, all the way to Java 15 which was released in fall 2020. Java now fully supports lambdas, functional programming, type inference via var, immutable collections with simple constructors, and multi-line strings. Moreover, there are exciting new experimental features like data classes (record), and sealed classes. Finally I’ll talk about the Java REPL which offers high value for quick experiments.

Table of Contents:

Functional programming (Java 8)
Streams (Java 8)
Optional (Java 8)
JShell (Java 9)
Factory Method for Immutable Collections (Java 9)
Type Inference with var (Java 10)
Single Source File Launch (Java 11)
Switch Expression (Java 12: experimental, full feature: Java 14)
Multi-line Strings (Java 13: experimental, full feature: Java 15)
Data Classes: record (Java 14: experimental)
instanceof without Cast (Java 14: experimental)
Sealed classes (Java 15: experimental)
Bonus: Updated Licensing Terms Starting with Java 8

Functional Programming (Java 8)

In Java 8, functional programming and lambdas were added as language features. The two core paradigms of functional programming are immutable values and elevating functions to first class citizens. Data goes through a pipeline of modification steps, where each step takes some input and maps it to a new output. Functional programming can be used with Streams and null-safe monads (Optional) in Java as shown below...

Streams (Java 8)

For your average computer program, you often have to work with a list of values and perform a given transformation on each value. Prior to Java 8, you had to use a for loop for this transformation, but from now, you can use Streams as follows:

Stream.of("hello", "great")
    .map(s -> s + " world")
    .forEach(System.out::println);
> hello world
> great world

The map function takes as input a lambda, which will be applied to all elements in the stream.

Streams can work on Lists, Sets, and Maps (via transformation). Thanks to Streams, you can get rid of pretty much all loops in your code!👌

Optional (Java 8)

Another common problem in Java were Null Pointer Exceptions. So, Java introduced Optional – a monad that wraps a reference which might or might not be null. Applying updates to this Optional can be done in a functional way:

Optional.of(new Random().nextInt(10))
    .filter(i -> i % 2 == 0)
    .map(i -> "number is even: " + i)
    .ifPresent(System.out::println);
> number is even: 6

In the snippet above, we create a random number, wrap it inside an Optional object, and then only print the number if it is even.

JShell (Java 9)

Finally, we have a REPL for Java, and its name is JShell!😊 In a nutshell, JShell allows to experiment with Java snippets without writing and compiling a full Java class. Instead, you can execute one command at a time, and you immediately see the result. Here’s a simple example:

$ <JDK>/bin/jshell
jshell> System.out.println("hello world")
hello world

Folks familiar with interpreted languages like JavaScript or Python have had the pleasure of a REPL for a long time, but so far, this feature was missing in Java. JShell allows to define variables, but also more complex entities like multi-line functions, classes, and perform loops. Moreover, JShell supports auto-completion, which comes in handy if you don’t know the exact methods offered by a given Java class.

Factory Methods for Immutable Collections (Java 9)

Simple initialization of Lists has been missing in Java for a long time, but those times are over now. 😅 Previously you had to do something like this:

jshell> List<Integer> list = Arrays.asList(1, 2, 3, 4)
list ==> [1, 2, 3, 4]

This is now simplified as follows:

jshell> List<Integer> list = List.of(1, 2, 3, 4)
b ==> [1, 2, 3, 4]

This fancy of(...) method exists for List, Set and Map. They all create an immutable object in just one simple line of code.

Type Inference with `var` (Java 10)

Java 10 introduced the new var keyword which allows to omit the type of a variable.

jshell> var x = new HashSet<String>()
x ==> []

jshell> x.add("apple")
$1 ==> true

In the snippet above, the type of x can be inferred to be HashSet by the compiler.

This feature helps to reduce boilerplate code and improve readability. There’s some limitations to it though: you can only use var inside of method bodies, and the compiler will infer the type at compile time, so everything is still statically typed.

Single Source File Launch (Java 11)

Previously, when you had written a simple Java program consisting of one file, you had to first compile the file with javac and then run it with java. In Java 11, you can do both steps with one command:

Main.java:

public class Main {
  public static void main(String[] args) {
    System.out.println("hello world");
  }
}

$ java ./Main.java
hello world

For simple starter programs or experiments consisting of just one Java class, this feature for launching single source files will make your life easier.

Switch Expression (Java 12)

Java 12 brought us Switch expressions. Here’s a quick showcase of how the expression differs from the old switch statement.

The old switch statement defines the flow of the program:

jshell> var i = 3
jshell> String s;
jshell> switch(i) {
   ...>     case 1: s = "one"; break;
   ...>     case 2: s = "two"; break;
   ...>     case 3: s = "three"; break;
   ...>     default: s = "unknown number";
   ...> }
jshell> s
s ==> "three"

In contrast, the new switch expression returns a value:

jshell> var i = 3;
jshell> var x = switch(i) {
   ...>     case 1 -> "one";
   ...>     case 2 -> "two";
   ...>     case 3 -> "three";
   ...>     default -> "unknown number";
   ...> };
x ==> "three"

To sum up, the old switch statement is for program flow, and the new switch expression resolves to a value.

Notice that this new switch statement is a sort of mapping function: there’s one input (in the above case i), and there’s one output (here x). This is actually a pattern matching feature which helps to make Java more compatible with the functional programming principles. A similar switch statement has been available in Scala for a while.

A couple of things to note:

Instead of double points, we use arrows ->
There’s no need for break
The default case can be omitted when all possible cases are considered
To enable this feature with Java 12, use --enable-preview --source 12

Multi-line Strings (Java 13)

Did you ever have to define a long multi-line String like JSON or XML? So far, you’d probably squash everything on one line and use newline characters \n, but this makes the String much harder to read. Here comes Java 13 with multi-line Strings!💪

Sample case:

public class Main
{ 
  public static void main(String [] args)
  {
    var s = """
        {
            "recipe": "watermelon smoothie",
            "duration": "10 mins",
            "items": ["watermelon", "lemon", "parsley"]
        }""";
    System.out.println(s);
  }
}

Now, we run the main Method via single-file-launch:

java --enable-preview --source 13 Main.java

{
    "recipe": "watermelon smoothie",
    "duration": "10 mins",
    "items": ["watermelon", "lemon", "parsley"]
}

The resulting String spans multiple lines, quotation marks "" are left intact, and even tabs \t are preserved!

Data Classes: `record` (Java 14)

Of all the new features in this article, this is probably the one I’m most excited about: finally, there are data classes in Java! These classes are declared with the record keyword and have automatic Getters, a constructor, and equals() method etc. In short, you can get rid of a huge chunk of boilerplate code!🙌🎉

jshell> record Employee (String name, int age, String department) {}
|  created record Employee

jshell> var x = new Employee("Anne", 25, "Legal");
x ==> Employee[name=Anne, age=25, department=Legal]

jshell> x.name()
$2 ==> "Anne"

Scala has a similar feature with case classes, and Kotlin with data classes. In Java, lots of developers used Lombok so far, which offered pretty much the features that now inspired records for Java 14. More details can be found in this Baeldung article.

`instanceof` without Cast (Java 14)

Prior versions of Java already contained the instanceof keyword:

Object obj = new String("hello");
if (obj instanceof String) {
  System.out.println("String length: " + ((String)obj).length());
}

The unfortunate part: First we check that s is of type String, then we cast it again to retrieve its length.

Now with Java 14, the compiler is smart enough to infer the type automatically after the instanceof check:

Object obj = new String("hello");
if (obj instanceof String mystr) {
  System.out.println("String length: " + mystr.length());
}

Sealed classes (Java 15)

With the sealed keyword, you can restrict which classes can extend a given class or interface. Here’s an example:

public sealed interface Fruit permits Apple, Pear {
    String getName();
}

public final class Apple implements Fruit {
    public String getName() { return "Apple"; }
}

public final class Pear implements Fruit {
    public String getName() { return "Pear"; }
}

So how does this help us? Well, now you know how many Fruits there are. This is actually an important step into the direction fully supported pattern matching, where you can sort of treat classes like enums. This sealed feature goes together nicely with the new switch expression explained previously.

Bonus: Updated Licensing Terms Starting with Java 8

One last topic for this article: licensing. Most of you heard that Oracle stopped updates for Java 8 (for the free commercial version). So here are your options:

Use a newer Oracle JDK version (Oracle offers free security updates for only 6 months after each release)
Use and old JDK version at your own risk
Use an old OpenJDK Java version, those still get security updates from the open source community or a third party vendor
Pay Oracle for premier support (e.g. Java 8: support until 2030)

Below you can see the tentative Oracle support durations per JDK:

Oracle’s new licensing model is influenced by the new release cycle: Oracle will bring out a new Java version every 6 months. This new release cycle helps Oracle to improve Java at a faster pace, get quicker feedback via experimental features, and catch up with more modern languages like Scala, Kotlin and Python.

If you’re interested in more licensing details, checkout this Medium article.

Wrap up

Java has gone a long way in the last 6 years, there's actually been 8 new Java releases since then!🚀 All these awesome new features help to make Java a competitive option compared to other JVM-based rivals (Scala and Kotlin).

If you're looking for even more in-depth details on new Java features since Java 8, I can recommend this DEV article by Andrew as well as this article by David Csakvari.

I had a blast writing this article🔥 Thanks for reading😊 What's your favorite feature in a recent java release? I'd love to get a feedback from you! Cheers!

From Legacy Projects to CI/CD Pipelines

Philipp Gysel — Tue, 17 Nov 2020 14:46:42 +0000

Recently I received the maintenance responsibility for two legacy projects in my company. One of the projects is 8 years old, the other is so old even our tester doesn’t know when the project started – definitely before he joined 12 years ago. Obviously, both projects use pretty old technologies like Ant-scripts and both have many features which aren’t even used any more.

So how do you deal with such an old project - as a motivated and proactive engineer?

In this blog, I’ll show you how you can make the best of such a situation, and how you can have a positive and significant impact. 🚀😎

CI / CD Theory

First off, here's the theory for CI/CD, which stands for continuous integration, delivery and deployment:

From what I hear, there's some misconceptions out there as to what CI/CD really means. In the graph above, you can see there are three parts to it:

compilation
bringing the software to a test system
and finally releasing it to the productive environment

Flee from the Problem?

Now if you’re ever stranded on a legacy project, one obvious solution is to polish up your LinkedIn and start sending out resumes😜

Remember though that every company has legacy projects. So rather than complaining, this article is about the positive impact you can have on a legacy project.

Before we get into the details, let me just mention that I’m lucky enough to only work part time on these legacy projects – the rest of my time, I spend on developing new software from scratch. There’s a brand-new web service I’m currently working on that uses the latest Java version, Spring Boot, Docker, an elaborate CI/CD pipeline and cloud-hosting, so this keeps me happy😊

Version Control

When I started on these 2 legacy projects, the first thing I did was migrating to a decent version control system. Both projects used SVN for versioning, but luckily, migrating to a new version control system is straight forward and worth the time. So, I used svn2git and created new Git repositories. After completion, I could use branches for quick experiments, plus git is much, much faster than SVN, and finally, you can perform commits without the need to immediately push your changes to the server.

Compile

Next, I analyzed the build system of my legacy projects. The question looks easy: how do I compile this code? Well my legacy projects consisted of many sub-components, some of which were easy to compile, some of which were hard to compile. There were easy cases of plain old Java-Maven projects where I could just run mvn clean package. One particular component though was much harder to compile: it was an old Grails app and getting all dependency versions right like Grails, Java and Maven was tricky.

In the end, you want to have a terminal command that will compile the project for you. Once you have that, you’re good, since you have a command, and this can be automated. So, in my case, I set up a Jenkins job for every component which performed the building process for me.

Note that in rare cases, there might be a need to re-engineer the compilation process. I've seen Java projects that use Ant and checked-in Jar dependencies. This is an outdated and cumbersome way to do things, and it would really make sense in this case to move to Maven or Gradle instead and use a registry for dependencies.

Code quality

Is your application secure and your code clean? An easy way to advance on these questions and possibly improve your code quality is by using a static code analysis tool. Inspecting your code is non-intrusive, all you do is add a new step to your continuous integration pipeline. There are many great tools out there. For my projects I chose SonarQube

Deploy to Test System

First off: yes, you need a test environment. If your legacy project doesn’t have one, make sure you get one!💪 You can figure out the dependencies from the productive environment. Installation of third party software like the runtime environment will be easy, however installing (mock versions of) the external systems might be trickier.

Anyways, the specific legacy project I worked on used Ant scripts to deploy the distributable Java archive on the test system. What the Ant script basically did was take the tarball, copy it to the test system machine, inflate the package, stop the old application, and start the new one. The command for starting the application was a Java command including all kinds of important parameters like the JVM version, Maven configuration file, the log file location and so on. As it turned out, it was pretty simple to automate this deployment: just use some CD tool like Jenkins and run the Ant script. There were still some bits to get right, like adding additional support Jars to the class path of the Ant command, but I figured that out too.

If you ever must work with a legacy app, your technology stack might be different from mine, but the basic concept of bringing your app to the test system and running it there remains the same.

End-to-End Tests

Anyways, now I had automated things far enough so I could just hit a button to turn my source code into a running application on the test system. Now the last step was to automatically run end-to-end tests. For my legacy projects, I was lucky to have a plethora of existing E2E tests. My specific project used Java FitNesse tests. Since Jenkins offers a FitNesse plugin to run such tests, my job was straight forward.

Now what do you do if you have no automated E2E tests? Well the most basic thing you can do is perform a health check. Every component should have an HTTP health endpoint which will tell whether the app was started properly and if all external systems can be connected to like the database or queues. It’s even better though to perform functional tests. If your app has a REST or SOAP interface, you can use it to perform more advanced queries like read/write/update operations. Anyways, the important part is to program out these tests, so you don’t have to manually fire requests via Postman or SOAP UI. Moreover, if you're starting from scratch with new E2E tests, there are many modern and powerful frameworks out there, like Cucumber and Gauge.

Remember to pay attention to external systems. Since you are performing end-to-end tests, you want to make sure all surrounding systems are present on your test system. Most often, engineers will create mocks for these systems with the most important functionality replicated, or you can ask the maintainers of those externals systems if they already have a mock.

The Final CI/CD Pipeline

Here’s a snapshot of the final result, the Jenkins pipeline for continuous integration and delivery:

Notice that I hadn't changed any line in the old source code of the legacy app, yet I had made the maintenance so much more fun and easy.🎉🥂

If you ever work on a legacy project yourself, I hope this article will give you some helpful information. Cleaning up legacy code will mostly be tough, plus the customer will probably not want to pay for it. Instead, what I suggest is to first focus on everything surrounding your code base. With a CI/CD pipeline in place, you will be much more motivated to start on a change request, since you can just hit one button which will verify your change all the way from compilation to packaging, deployment and E2E testing.

Happy coding💻😃🔥

Credits: Title image from Unsplashed
This post was inspired by an article from Felipe Lopes

Log Capturing App with MongoDB + Spring Boot + Swagger UI

Philipp Gysel — Mon, 14 Sep 2020 11:56:22 +0000

In this story, we will look at a log capturing app using the following technologies:

MongoDB
Java + Spring Boot
Swagger UI

This demo application showcases the ability of MongoDB to store data without schema. All log entries can have arbitrary fields and we can search logs via those arbitrary fields. To make the app easy to test, we’ll visualize its REST API via Swagger UI.

The Challenge of Schemaless Data

In this story we’ll build an app which captures log data from 3rd party apps. The main challenge here: we don’t know how the log data from these 3rd party apps looks like. Still, we want to store the data in a compact format, plus we want to support queries for this log data.

Let’s look at a concrete example. Here’s two sample logs we want to store, in JSON format:

We have one microservice PAYMENT which creates logs during a payment process. On the right hand, we have a log from another app called LOGIN. Some of the log fields are common like severity, but some are unique to an app like creditCardProvider which only exists in the PAYMENT app.

The SQL Approach

If we wanted to store this data in an SQL store, we’d have a hard time to figure out a neat solution. Either we’d create a huge table which has a column for every log field we ever see. But this would result in many null entries and we would waste storage. Another solution is to store all data in JSON format, so you’d only have two rows in your table: the primary key and the JSON document. However, with this solution, you wouldn’t be able to search by JSON fields in a fast way.

The NoSQL Approach

NoSQL Document Stores come to our rescue: they store data in JSON format without any fix schema. Basically, a document store contains JSON documents – and nothing else. Even the ID of a given document is contained in the JSON itself.

Now, we can do really fast querying by JSON field, so you can do something like this:

> db.logEvents.find({_appType:"PAYMENT"})

This would return the first log above. More interestingly, you can perform the following query:

> db.logEvents.find({creditCardProvider:"VISA"})

The document store is robust to the fact that not every log contains the field creditCardProver!

Implement the Log Capturing App

Enough talk - let’s get started implementing our app!

MongoCollection Configuration

We use MongoCollection from the official MongoDB Java Driver for DB access. Let’s go ahead and create a Bean with a properly configured MongoCollection:

@Configuration
public class MongoConfiguration {
  @Bean
  public MongoCollection<Document> getMongoCollection() {
    MongoClient mongoClient = MongoClients.create();
    MongoDatabase db = mongoClient.getDatabase("mydb");
    return db.getCollection("logEvents");
  }
}

MongoDB Queries

Let’s use the above declared Bean to search entries in the MongoDB by id:

public class MongoDbService {
  @Autowired
  MongoCollection<Document> collection;

  public Optional<Document> findById(String id) {
    Document doc = collection.find(eq("_id", new ObjectId(id))).first();
    return Optional.ofNullable(doc);
  }
}

Using the @Autowired annotation, we can pull in our Bean of type MongoCollection. This class offers many query capabilities like searching by id, by key-value pair, and even more complex queries like greater-than etc. On my Github repo, you can see other query samples.

One more note: all DB data is represented as Documents, which are described in the official documentation as follows:

The Document class can represent dynamically structured documents of any complexity ... Document implements Map<String, Object>

Java Date Model

Inside our logging app, we don’t want to use Documents, but use a custom POJO instead:

import lombok.Data;
@Data
public final class LogEvent {
  private Map<String, String> logEvent;
}

We represent each log entry as a map with String keys and values.

Side note: the code snippet above uses Lombok which offers a neat way to create setter, getters, and constructors.

Document to LogEvent Mapper

Our DB layer represents logs as Document, but our service layer uses LogEvent. Therefore, we need a mapper method:

public LogEvent toLog(Document document) {
  Map<String, String> map = new HashMap<>();
  document.forEach((key, val) -> map.put(key, val.toString()));
  return new LogEvent(map);
}

REST Controller

Now it’s time to create the controller layer. We’ll use REST Controllers, which can be nicely created using Spring Boot’s @RequestMapping annotation:

@RequestMapping(method = RequestMethod.GET, path = "/api/log/{id}")
public ResponseEntity<LogEvent> getLogById(
    @PathVariable(value = "id") String id) {
  return mongoDbService.findById(id)
      .map(documentToLogMapper::toLog)
      .map(ResponseEntity::ok)
      .orElse(ResponseEntity.notFound().build());
}

Everything we’ve coded so far comes together now!😎 So let’s go through the above code snippet step by step! First, our REST controller will serve HTTP GET requests to the URI "api/log/{id}", id is the identifier of the log event. Second, the REST call will return a LogEvent in JSON format. Third, we use functional programming in the method body. We start with the DB query, pass the result to our mapper, and finally check if everything went fine – and if not: return a 404 NOT_FOUND result.

Spring Boot Service

Only one thing is missing to make our app executable:

@SpringBootApplication
public class LogCaptureApp {
  public static void main(String[] args) {
    new SpringApplication(LogCaptureApp.class).run();
  }
}

That’s it, that’s all necessary code😊 Now your logging app can serve REST requests to read logs!

Swagger UI Documentation

One more thing – let’s add a simple GUI to our log capturing app. We use Swagger UI for this purpose. Swagger UI allows to visualize your API without any further hustle. All you need is configure Swagger:

@Configuration
@EnableSwagger2
public class SwaggerUIConfiguration {
  @Bean
  public Docket apiDocu() {
    return new Docket(DocumentationType.SWAGGER_2)
        .apiInfo(apiInfo())
        .select()
        .apis(RequestHandlerSelectors.basePackage("com.example.controller"))
        .paths(PathSelectors.any())
        .build();
  }
  private ApiInfo apiInfo() {
    return new ApiInfoBuilder()
        .title("Log capturing app with MongoDB + SpringBoot")
        .build();
  }
}

Run the App

The complete code can be found on Github:

pmgysel / mongodb-logging-app

Demo app for MongoDB: manage log events of varying schema type

My Github repo contains more functionality than what we saw in this story, like searching log events by date range and storing log events from JSON data.

So, if you have implemented the log app as you read through this story, go ahead and run it now. If not, no problem, you can use my fully working example from Github.

Also, make sure you have a MongoDB Server running locally on the standard port. Then, compile and run the app:

$ git clone https://github.com/pmgysel/mongodb-logging-app.git
$ cd mongodb-logging-app
$ mvn clean package
$ java -jar target/mongodb-logging-app-0.0.1-SNAPSHOT.jar

Go to the Swagger UI page in your favorite web browser: http://localhost:8080/swagger-ui.html

You can use the log capturing app as follows:

createLogRadom: Create random log entries to get some data into the DB
Alternatively, create arbitrary log entries using the createLog endpoint. Swagger UI will show sample request to make your life easy 😊
Now search for logs by key-value pair (getLogByOneField) or via date range (getLogByDateRange)
Sample search:

Conclusion

So, in this story, we built a fully working log capturing app in Java. Since our requirements were to support flexible log entries, we chose MongoDB as database solution.

This scenario is very well suited to NoSQL Document Stores. Note that every application calls for another database – if your data has a fix schema, you might be better off with a traditional SQL store.

Feel free to drop a comment with your ideas! Also, like ❤️ this story if it was helpful for you!

MongoDB CRUD Queries from Java and Shell

Philipp Gysel — Mon, 31 Aug 2020 13:44:56 +0000

Liquid syntax error: Tag '{% raw %}' was not properly terminated with regexp: /\%\}/

Introduction to MongoDB and Document Databases

Philipp Gysel — Thu, 27 Aug 2020 13:08:36 +0000

This tutorial teaches you the basics of NoSQL Document Databases. To keep things practical, we’ll also look at a concrete implementation, namely MongoDB.

First, we’ll start with the data model in document databases and compare it to the SQL data model. Second, we dive into the characteristics of MongoDB, like: queries, data replication, sharding, and consistency. We’ll wrap things up with some real world uses cases well suited for MongoDB.

Data Model

The fundamental difference between SQL and NoSQL is how the data gets stored. Relational databases are like Excel tables with rows and columns. This way of storing data means each row has the exact same fields. In contrast, NoSQL allows for more flexibility in what you store. It’s easier to persist arrays, fields of unknown length, or add new data fields. Let’s look at an example.

UML data model:

Above you see the data model for a simple online store: we store customers and their orders. A given customer can have multiple orders, and each order is associated with a delivery address and all items purchased.

SQL tables:

In SQL, a convenient way for storing this data is by having 4 tables (see graph above). We would connect the orders to the customer and the items through foreign keys. In order to fetch a given order, you’d perform a join over the Order, Address and Item table.

Let's switch from the relational storage model to the non-relational one. In a document store, we store all data in so called documents (either in JSON, XML or BSON format).

NoSQL document:

// in orders
{
 "_id": 99,
 "orderDate": "2020-08-22",
 "customer": {
  "customerId": 1,
  "firstName": "Philipp",
  "lastName": "Gysel"
 },
 "address": {
  "street": "Main ln",
  "city": "Bern"
 },
 "items": [
  {
   "itemId": 88,
   "price": 9.90,
   "description": "NoSQL Distilled"
  }
 ]
}

The above code snippet uses 1 collection to store all data (as opposed to 4 tables in SQL). Each document in the collection orders contains the delivery address and all items. Moreover, the customer data is also present in each order. Note that we leverage a powerful feature here: different orders will have varying lengths, since the items element is an array of arbitrary size.

How to Store and Manipulate Data

In SQL:

No Arrays: We can’t store arrays. Thus, we need one table row per order.
Normalization: Data is normalized. There’s no data duplication. In case the customer wants to change his/her name in the online store, we just need to update the customer name in one place.

In NoSQL document store:

Arrays are supported: We can store all items of a given order together.
Data duplication: Each order has the customer data. Now if a customer changes name, we have to perform this change in each order of this customer, unfortunately.
Aggregation: In document databases and NoSQL in general, we use aggregates, which contain a mix of data. We typically draw the aggregate boundaries according to how the application accesses the data. In our case, orders contains more than just the order itself, which makes sense since the application has to read all data anyways.
Not optimized for transactions: Newer NoSQL versions do offer support for transactions, but they are not primarily designed for distributed ACID behavior. For most use cases though, this is not a problem, especially when all logically connected data lives in the same aggregate.

MongoDB

In order to make things more tangible, let’s look at a concrete implementation! In the next few sections we cover the basics of MongoDB.

MongoDB Queries

MongoDB is a document store which persists a key-value map, where each value is a JSON document of varying schema. As a case in point, a collection can contain the following two documents:

{
 "_id": 1,
 "firstName": "Philipp",
 "lastName": "Gysel"
}
{
 "_id": 2,
 "firstName": "John",
 "lastName": "Smith",
 "age": 22
}

Using MongoDB Shell, we could query for a customer with first name “Philipp” as follows:

> db.customers.find({firstName:"Philipp"})

... which would return us the first customer in JSON format.

MongoDB offers a variety of query features for CRUD operations, projection, sorting etc. For more on queries, watch out for my next post in this series.

MongoDB Naming Convention

With MongoDB, there are no rows and columns, instead, we deal with documents and JSON elements:

Oracle	MongoDB
table	collection
row	document
rowid	_id

The primary key of a MongoDB document is always called _id and naturally you can do queries by primary key.

MongoDB - No Schema

MongoDB has no predefined schema. If you look again at the documents above, you can see that “John” has an age element, which is missing from “Philipp”. Moreover, you can even change the schema of an existing document:

> db.customers.updateOne({_id:1},{$set:{city:"Bern"}})

... this will add a city field to the existing document.

MongoDB Replica Sets

For read scalability, MongoDB supports replica sets. A replica set contains one master and multiple slaves; each node contains all data. MongoDB allows to add new nodes to a running database when more data traffic needs to be supported. By default, all requests from the application go to the master node.

In the image above, the application performs a write operation to the database, which is sent to the master and then passed on to all slave nodes.

Read scalability is achieved through the fact that each slave contains all data. All you need to do is specify that reads from slaves are ok:

Mongo mongo = new Mongo(“localhost:270127”);
Mongo.slaveOk();

Given 3 slaves, you now have 4 times higher read throughput!

MongoDB Sharding

Read scalability: check✔️. What about write scalability❓ Replica sets won’t help us here, since all writes need to go through the master. That’s where sharding comes in: With sharding, we split our data into partitions, and each partition is stored in a different shard. By using some simple rule like applying Modulo on the primary key of a document, each document is assigned to a shard.

Now the writes get distributed over different nodes. This can be especially helpful for write heavy applications like log capturing (e.g. NewRelic). Also, MongoDB makes sharding easy from an application perspective and performs all complicated work automatically in the background, like balancing shards or figuring out which document lives on which shard.

MongoDB Consistency

As soon as a MongoDB cluster uses replication, special care needs to be taken for consistency. Hypothetically, an application can persist a document, then query for it, but the DB will throw a "not found" error. The reason lies in the topology of the cluster: In a master-slave configuration, a write doesn’t reach each slave immediately, and it might take a sub-second before all slaves receive the update. While SQL offers immediate consistency, MongoDB guarantees only eventual consistency – one is guaranteed to have all updates on all nodes in the end.

For better control, MongoDB allows to specify a writeConcern for each insert and update:

db.companies.insert(
   {_id: 1, firstName: "Philipp", lastName: "Gysel"},
   {writeConcern: {w: "majority", wtimeout: 5000}}
)

Here, we mandate that an insert is propagated to the majority of nodes and the DB call is blocked until this happens or the timeout of 5 seconds is reached. Beware though: consistency comes at a price (latency), so think carefully about your specific use cases and how consistent your data needs to be.

MongoDB Transactions

Since MongoDB 4.0, multi-document transactions are supported. Thanks to this feature, documents living in different collections can be updated in an atomic fashion. Beware though, as the official MongoDB documentation states:

In most cases, multi-document transaction incurs a greater performance cost over single document writes, and the availability of multi-document transactions should not be a replacement for effective schema design.

As a rule of thumb, you should try to pack all interconnected data into one document, if possible, so that you don’t need to worry about transaction.

MongoDB: Suitable Business Cases

MongoDB isn’t a hammer to be used for every nail. Make sure you analyze the business and non-functional requirements of your application, and then search for a matching database solution. While MongoDB has many advantages (scalability, simple query language, plus it’s open source), there are also disadvantages like slow transactions compared to SQL.

So here are some use cases well suited for MongoDB:

Logging applications: Apps which track huge logs are well suited for MongoDB. Immediate consistency is not necessary, as a new log doesn’t have to be available right away. However, performance is really important, especially when many applications with verbose logs are involved. Big write volumes can be nicely handled through sharding.
Blogs / social media: For a Twitter-like app, each entry has a different format. Some contain text, some images, some videos, but everything is optional. This matches well to documents which can contain arbitrary elements. Also, if the app wants to support new types of blogs in the future – no problem, no database changes are required, you can simply add a new field to new documents. What's more, immediate consistency isn’t important, it’s fine to see a particular entry with a delay of 1 second.

Conclusion

I hope this helped you learn more about MongoDB. Needless to say, that the best way to learn a technology is by actually using it😉.

Add a ❤️ if you liked the MongoDB tutorial. Leave a comment if you have any questions / feedback.

For more on MongoDB, checkout my next post in this series which covers MongoDB queries from MongoDB Shell and Java!

NoSQL: Introduction and Concepts

Philipp Gysel — Mon, 24 Aug 2020 12:22:37 +0000

So you know SQL and want to get started with NoSQL? This starter tutorial will be perfect for you 😊.

What we'll cover:

Comparison: The strengths of SQL and NoSQL
4 NoSQL families: What NoSQL types exist?
Selling points for NoSQL: Dive deeper into NoSQL advantages

SQL vs. NoSQL

Most developers know SQL, but NoSQL has been in its shadow for a long time. In the past decade though, more and more business cases emerged that can't be handled easily by SQL solutions. Let’s go ahead and compare the two database concepts:

Advantages of NoSQL

Strength	Explanation
Scalability	NoSQL solutions are at home in the cloud; thus scalability and big data applications are nicely supported.
Flexibility	NoSQL can deal with changing schemas without any explicit update scripts.
Cost	Many NoSQL databases are open source, there are no expensive licensing terms. What’s more: NoSQL runs well on commodity machines which again saves costs.
Replication	Most NoSQL solutions have built-in support for sharding and replication sets. This allows to split huge amounts of data across many servers and at the same time replicating the data, thus creating higher availability.

Advantages of SQL

Strength	Explanation
Transactions	Querying and updating data across multiple tables in an atomic fashion is better supported in SQL.
Consistency	In SQL, updates are immediately visible to all clients. NoSQL uses eventual consistency – meaning the data will be uniform, but it might take half a second until all nodes hold an update.
Support and maturity	Most SQL databases have been around for a while and are thus very stable. Moreover, if you choose an enterprise solution like Oracle DB, you get professional support.
Data normalization	In SQL, we store all data items just once – no repetitions necessary. This allows for simple updates. On the other hand, updates in NoSQL can require writes to multiple aggregates.

NoSQL families

One of the confusing things about non-relational databases is the fact that many varying database concepts belong to this family. All those families share one property though: data is NOT arranged into tables with fixed column sizes. For example, in Key-Value and Document Stores, each row can have a different length and format.
Here are the 4 families:

Database	Example	Description
Key-Value Store	Dynamo, Redis	A key-value map where the value is of arbitrary data type like String, Array or JSON.
Document Store	MongoDB, Couchbase	Each data entry is a document (e.g. as JSON) of varying schema. It's possible to query by JSON elements.
Column-Family Store	Cassandra, HBase	Store data in column families. Great for data compression and aggregate queries (e.g. SUM)
Graph Store	Neo4J, Giraph	Do computations on graphs, where the majority of data comes from edges rather than nodes.

If you want to get to know NoSQL better, a great starting point is the learn and use a concrete store. For me, when I started to learn about non-relational databases, I simply started with MongoDB and tried out some queries / inserts etc. This helped me tremendously to grasp the concept of NoSQL.

Selling points for NoSQL

The web era

NoSQL emerged around 2000 when companies like Google and Amazon felt they needed an alternative to traditional SQL databases. The primary driver for NoSQL was the tremendous growth of data volume as well as the frequent changes in the data schema. Thus highly influential papers like BigTable and Dynamo were released which showed unprecedented ways for storing and managing big data.

Scalability

While most SQL databases run on one server (with lots of RAM), NoSQL was designed from the start to run on clusters. Thanks to this capability, a data pool can easily be increased if need be, by simply adding more nodes. When you think about a company like Google, Amazon, Netflix or Facebook, you can easily imagine that their data wouldn’t even fit on 100 hard drives. Anyways, there’s an increasing number of businesses where huge data volumes are common, like for example NewRelic which stores log data from your applications and allows you to easily query and display that data. If you’ve already seen a large business application running in verbose logging mode, you know how hard it can be to browse through the log, so a Logging solution like the one from NewRelic can be really helpful.

Flexibility

Another huge selling point for NoSQL are flexible schemas – in fact, you don’t need any schema, you can start storing data right away. Now obviously, the application dealing with your database has to understand how to make sense of the bits and bytes in your data store. So implicitly, there will be some schema, it's simply not in your database. Nevertheless, having the ability to quickly change the structure of new data records has tremendous advantages. For SQL, you’d first have to change your table schema – but what do you do with existing records? You might even have to create a data migration script and you obviously need an SQL script altering existing tables. In the non-relational world, all you do is change the logic in your application – and that’s it.

As a case in point, lets consider a retail store, which stores product info in a NoSQL database. Once a new piece of information gets introduced – like the eco-friendliness of a given product – then you can simply add this information to newly introduced products, but you can leave old products as is. All you have to do is make sure your application can handle both data formats – products with and without eco-friendly-scale.

Availability

With NoSQL, we use replication of data to ensure high availability. While replication comes in different forms (master-slave, peer-to-peer), the result is the same: One node can become unavailable, but querying data will still work, as other nodes take over the traffic in question. There are many ways how a data server can become unavailable – hard disk failure, temporary problems with the internet or a power outage – and while most of those happen only once a year or less frequently, there are a number of business cases where such downtimes are intolerable. Note though that SQL solutions can also offer high availability – with concepts like backup databases.

Low Cost

In the NoSQL world, there are many open source solutions, which one can simply install on his or her own cluster (MongoDB, Cassandra, CouchDB, Hypertable, Neo4J). So, when you choose a NoSQL open source solution, you are not at the mercy of a company’s licensing terms which might change unfavorably over time. What’s more, non-relational solutions are the ideal candidate for commodity servers. One can achieve really good throughput and availability without buying expensive high-end servers. For SQL, the game is different, here people often scale vertically, meaning an existing server is replaced by one with more RAM and faster CPU cores. This typically infers higher costs than scaling horizontally.

Conclusion

Bonus points if you're still reading this😊 So we've covered the most important general concepts of NoSQL.

Note that this post has only scratched the surface of NoSQL. If you are like me an prefer more concrete examples – well I’m glad you asked, I’ll be posting a tutorial on Document Stores and MongoDB shortly!

Also, if you're new to NoSQL and enjoy learning from books, I can highly recommend NoSQL Distilled by Sadalage and Fowler.

Finally, please remember to like ❤️ this article if you found it useful!

From a Docker Image to a Deployment on OpenShift

Philipp Gysel — Mon, 03 Aug 2020 13:35:15 +0000

In this tutorial, I’ll show you a working example for deploying a docker container on OpenShift V4. You can find all Kubernetes resources from this tutorial on my Github repo. If you only have some basic knowledge of Docker and Kubernetes and don’t know OpenShift yet – don’t worry, this tutorial will be perfect for you!

Quick History

From Native Applications to Docker Containers to Kubernetes

In the olden days, software companies deployed their applications on “bare metal”, meaning one physical computer hosted one application. Over time, things improved thanks to virtual machines, which allowed multiple isolated applications to run on the same physical server. In the last couple of years, a more light-weight virtualization solution became popular – docker containers. Once companies started to heavily use docker containers, their deployment landscape became pretty messy, and so orchestration tools like Kubernetes came to the rescue. Kubernetes allows to plan, perform, and manage the deployment of applications in the form of Docker containers and to make deployments reproduceable and easy to customize.

From “Bare-metal” to “PaaS”

Parallel to the development of new tools, the hosting landscape also changed significantly. While most companies were hosting their applications on machines in their basement a decade or two ago, more and more software development companies started to investigate letting third party companies host their applications. This greatly helped to separate the concerns of application development vs. application deployment and monitoring. Today there are many great cloud providers which take care of several important challenges like hardware scaling, geo redundancy, monitoring of running clusters and configuring security related components. Companies which offer a Platform as a Service include AWS, Google Cloud, IBM Cloud, Microsoft Azure, and many more.

Why OpenShift?

OpenShift is the PaaS solution we’ll be discussing in this blog. It is developed by Red Hat and builds on top of Docker and Kubernetes. OpenShift can be run on any cloud infrastructure, and many popular providers have an offering, including the aforementioned providers AWS, Google Cloud, IBM Cloud, and Microsoft Azure.

So, what’s the advantage of using OpenShift over a Kubernetes cluster? Well OpenShift offers several additional features while also supporting all Kubernetes resources:

Simple to use: Thanks to many preconfigured parameters and a nice GUI, OpenShift is fairly simple to learn.
Secure by default: OpenShift apps are secure by default. Security experts from Red Hat have put a lot of time and effort into making the platform resilient against attacks.
Additional tools: OpenShift offers additional tools compared to a simple Kubernetes cluster: Source-to-Image build, Kibana monitoring, OpenShift registry, etc.

Note though that some people might prefer pure Kubernetes cluster infrastructures, for the following reasons:

Avoid vendor look-in: OpenShift supports all Kubernetes objects, so by using only standard Kubernetes resources for the deployments, a future switch to another cloud provider is possible without extra effort. Unfortunately, there are some minor differences between OpenShift and Kubernetes, for example OpenShift’s Route resource (which doesn’t exist in Kubernetes).
Pricing: There are many providers out there for both OpenShift and Kubernetes, so comparing the costs is a tricky endeavor. Generally speaking, you get more from an OpenShift platform, which is why this solution tends to cost slightly more.

Overall, OpenShift is a great solution that is getting adapted by many companies all over the world, so my guess is that it is a solution that will stay for quite some time. Without any further ado, let’s jump into the working example!

DIY OpenShift Example

Alright, now it’s time to run your first app on OpenShift V4. All you need is an OpenShift account and the Kubernetes resources from the Github repo.

Create Free OpenShift Account

For experimental purposes, OpenShift offers a solution termed “OpenShift online” which can be used for free for 2 months.

Create Red Hat account: https://www.redhat.com/ ‑> Login ‑> Register now
Confirm email
Start OpenShift online trial: https://www.openshift.com/products/online/ ‑> Choose Starter plan and follow the instructions
Login to OpenShift online web console: https://manage.openshift.com/sign_in
Create your first project: in “Administrator” view, go to Home ‑> Projects ‑> click Create Project

Deploy App on OpenShift (via web console)

To deploy your first app, you’ll need two items:

A Docker image
The Kubernetes resources

As for the Docker image: I have packaged a simple Java app into a Docker image and published it on Docker Hub. The docker image is directly reference in the Kubernetes resources, so all you need to do is download the Kubernetes resources from my Gihub repo.

Let’s start with the deployment. Head over to the web console where you’ve already created an OpenShift project, then:

In “Developer” view, go to “+Add” and select “YAML”
Copy and paste all Kubernetes resources: deployment.yaml, service.yaml and route.yaml (you’ll find the files on Github)
That’s it! Your app is getting deployed on OpenShift. To check the status, go to “Administrator” view ‑> Workloads ‑> Pods

You should see a Pod called “spring-boot-app-*” starting up. As soon as it is up and running, let’s test it like this:

In “Administrator” view, go to Networking ‑> Routes
For “spring-boot-app-route”, select the “Location”: this is a public URL to access your running app.
Append “/api/ping” to the URL – and you should see an answer from your app.

That’s it folks 😊 You’ve got an application running on OpenShift!

Deploy App on OpenShift (via CLI)

In a real development project, you’ll want to automate as much work as possible. People typically use a CI/CD pipeline like Jenkins to compile new app versions, test them, build the docker image and deploy the app in the cloud. For this purpose, OpenShift offers the command line tool oc – which allows to show the status of your running apps, alter deployment configurations and start new deployments.

First, you probably want to remove all existing resources on OpenShift. The simplest way to do so is just to remove the active OpenShift project and create a new one.
Install the oc CLI by following the instructions under “?” ‑> “Command Line Tools”
Login to OpenShift via oc CLI by following the instructions “Copy Login Command” on the previous page.

$ oc login --token=… --server=…

Make sure you’re in the correct OpenShift project

$ oc projects

Fetch the Kubernetes resources from Github

$ git clone https://github.com/pmgysel/cloud-stack_openshift-kubernetes-ressources.git
$ cd cloud-stack_openshift-kubernetes-ressources

Upload Kubernetes resources

$ oc apply -f deployment.yaml
$ oc apply -f service.yaml
$ oc apply -f route.yaml

That’s it folks 😊 You just performed your first OpenShift deployment via command line!

Our Kubernetes Resources

You have created 3 Kubernetes resources on the OpenShift platform. Let’s quickly get an overview of how these 3 resources work together. This is a fairly simple primer on Kubernetes, so feel free to skip this part if you already have advanced knowledge in this technology.

deployment.yaml: The Kubernetes Deployment specifies how your containers should be deployed, including the number of instances, the port to be exposed and the amount of RAM per Pod.
service.yaml: A Kubernetes Service acts as entry point to running pods. You don’t need to worry about the exact number of running pods, how to distribute the load, how to detect ready pods and so on. The Service will do all the work for you.
route.yaml: All Services and Pods are only accessible from within the OpenShift cluster. To publish your service to the outside world, you can use an OpenShift Route. Just a small hiccup here: Routes are an OpenShift only feature and don’t exist in Kubernetes. In pure Kubernetes, there’s an analog resource termed “Ingress”.

Summary

Bonus points if you’re still reading this 😊 So in this post, we learned about the advantages of OpenShift and ran a simple app on the OpenShift platform. I hope this was informative for you!

Some more resources:

OpenShift has an official OpenShift tutorial
Kubernetes resources: Deployments, Services, and Routes
OpenShift’s oc CLI is an extension to Kubernete’s kubectl. The OpenShift website has a docu on oc

Remember to like ❤️ this blog if it was helpful to you and see you next time!

Learn Spring Boot by Annotations

Philipp Gysel — Mon, 11 May 2020 10:14:20 +0000

This blog will teach you the basics of Spring Boot by looking at the available annotations. A lot of the magic in Spring Apps happens via annotations such as @Bean, @RestController, and @Component.

Why We Care about Spring Annotations

Spring has some powerful concepts like dependency injection (DI) which is made possible thanks to Spring-managed components. Here, all Classes annotated with @Component are candidates for DI and can be used by other Classes without explicit instantiation. Spring offers a feature called Inversion of Control Container which will do all necessary injections in the background, all magic happens without any work from the developer😊 This is a great example for learning Spring by studying annotations: The Java code of a given Class may look completely normal, but when we consider this metadata, much richer functionality is possible (like DI). There are many of those annotations that significantly add functionality unique to Spring.

I dare to argue that a great way of learning Spring is by understanding the most common annotations. In the rest of this article, we will look at different annotations (from Spring as well as javax.persistence), describe their purpose, and show some code samples. I also created a Java Spring Boot app with all the described annotations included, so if you wanna look at a fully working example, checkout my Github repo.

Spring Annotations Overview

First let’s get an overview of what I consider the most important Spring annotations. Let me state that my selection is aimed at people interested in RESTful webservices. For other people, the optimal selection might vary slightly. Anyways, apart from Spring annotations, we’ll also cover metadata tags from the Javax Persistence API. The reason for this: we nearly always use a database in our backend, and this is a feature supported by Javax, not Spring, so I decided to include it here.

Here's a list of annotations we'll cover:

As you can see in the table above, there’s quite some annotations to cover in this blog, so I decided to put them into functional groups. Let’s get started with syntactic metadata used in the Main class:

Main Class Annotation

Covered in this section:

@SpringBootApplication

Thankfully, in a Spring app, our Main Method doesn’t have to do much: we don’t need to code up any logic to assemble our service components and boot it up. Instead, all we need is to annotate the Main class with @SpringBootApplication. This will mainly trigger two things: First, Spring Boot will apply the default configuration such as socket read timeout. Secondly, Spring will search for all Spring-managed components, e.g. Classes annotated with @RestController.

Here’s how our Main class would look like:

@SpringBootApplication
public class SpringBootAnnotationsApplication {
  public static void main(final String[] args) {
    SpringApplication.run(SpringBootAnnotationsApplication.class, args);
  }
}

A fully working example can be found in my Github repo in SpringBootAnnotationsApplication.java.

REST Endpoint Annotations

Covered in this section:

@RestController
@RequestMapping
@PathVariable
@RequestBody

Say our web service should be able to answer to several REST requests. To implement this, we just need a Class annotated with @RestController which contains methods annotated with @RequestMapping. Each such method represents one REST endpoint.

When a REST request arrives, Spring will automatically search for the corresponding method, deserialize the incoming request, put all incoming data into our method parameters, and we’re ready to perform our own business logic. In the end, our result (a Java object) will get serialized by Spring and returned over HTTP. All we need to worry about is having the right footprint for our method, which should include the following: URI, URI path parameters, URI query parameter, HTTP request body, and HTTP answer body.

@RequestMapping(method = RequestMethod.PATCH, path = "/api/account/{accountId}")
public ResponseEntity<AccountRJ> activateAccount(
    @PathVariable(value = "accountId") Integer accountId,
    @RequestBody Integer startBalance) {
  // implement method ...
}

Here’s an example for a corresponding REST request:

PATCH https://www.../api/account/5
Request Body: "10"

In our code snipped, the accountId would be 5 and startBalance would be 10. The REST answer should be an object of type AccountRJ. To be more specific, the method return value should be of type ResponseEntity<T>. This is a Spring class containing all HTTP reply data such as status code, body, header, etc.

A fully working example can be found in my Github repo in WeatherRestController.java.

Periodic Task Annotations

Covered in this section:

@Scheduled
@EnableScheduling

Some web services need to do regular work in the background, such as archiving old data, aggregating data to create statistics etc. Spring makes the implementation of periodic tasks as easy as this:

@Configuration
@EnableScheduling
public class TaskConfig {
  @Scheduled(cron = "* * * * * ?")
  public void doTask() {
    System.out.println("hello world");
  }
}

As you can see, all we need is a Class annotated with @EnableScheduling and a method annotated with @Scheduled(cron=...). This method will run in the background according to our cron expression, e.g. every 10 minutes, or every hour.

A fully working example can be found in my Github repo in WeatherStatsTask.java.

Bean Annotation

Covered in this section:

@Configuration
@bean

To get your app up and running quickly, Spring Boot has lots of sensible default configurations like the REST configuration (e.g. read timeout, connect timeout). We can overwrite configuration defaults by defining our own Beans.

@Configuration
public class RestConfig {
  @Bean public RestTemplate restTemplate(final RestTemplateBuilder builder) {
    return builder
        .setConnectTimeout(Duration.parse("PT5s"))
        .setReadTimeout(Duration.parse("PT10s"))
        .build();
  }
}

As displayed above, our Beans are put in a @Configuration-class and the method is annotated with @Bean. The return type is essential here: whenever Spring needs to work with a RestTemplate, it will configure it according to our specification.

A fully working example can be found in my Github repo in JpaConfig.java.

Quick Note on Dependency Injection

Let's quickly look at the concept of dependency injection, as this is essential to understand Spring-managed components.

Spring offers an Inversion of Control (IoC) Container: Instead of manually initializing all members of an object, Spring will inject all necessary dependencies. In the graphic below, you can see a Class requiring an object of another Class:

The Class Parent contains a Child object:

public class Parent {
  private final Child child;
  public Parent(Child child) { this.child = child; }
}

We can now annotate the dependency with @Component as follows:

@Component
public class Child { … }

Now, when Spring IoC must create an object of type Parent, it will automatically create an object of Child, then pass the object as argument to the Parent constructor, thus creating an object where all dependencies are initialized.

Note that the actual dependency chain in a complex application can have several hierarchy levels. Spring IoC will also work for complicated cases where one Class will require another one which again will require a third one and so on. Moreover, Spring IoC even works for Interface-Dependencies: when a Class requires an object of the type of an interface, Spring will search for a Class implementing that interface and inject it.

Annotations for Spring Managed Components

Covered in this section:

@Component
@Service
@Repository

Now that you understand the concept of dependency injection, let’s focus on the three different annotations that specify Spring-managed components:

@Component: This Class is registered in the Spring-ApplicationContext and becomes a candidate for dependency injection.
@Service: has the same meaning as @Component, but is used for the service layer of an app (with business logic).
@Repository: same as @Component, but for Classes performing database access. The standard database exceptions will automatically be caught by Spring and dealt with appropriately.

Note that all of these annotations are to be applied at class-level.

A fully working example can be found in my Github repo in:

@Components -> AnnotationsAppProperties.java
@Service -> WeatherService.java
@Repository -> WeatherRepository.java

Persistence Annotations

Covered in this section:

@entity
@ Id
@GeneratedValue
@EnableJpaRepositories
@EnableTransactionManagement

Similar to the previous sections, annotations come in handy for data persistence: all we need is the right annotations! For the following example, let’s just focus on an SQL use case, and let’s use Spring Data Jpa. To do the ORM mapping, we need an @entity class as follows:

@Entity
public class WeatherPO {
  @Id
  @GeneratedValue(strategy=GeneratorionType.IDENTITY)
  private Long id;
  private Integer temperature;
}

In the above code, the WeatherPO class holds all data elements which we want to represent from our relational database. The first member is also annotated with @Id which indicates that this is the primary key. Moreover, we define a strategy to generate new primary keys, again with an annotation!

Now we are ready to define a JpaRepository with which we can fetch WeatherPO objects from the database:

@Repository
public interface WeatherRepository extends JpaRepository<WeatherPO, Long> {}

When Spring sees the @Repository annotation, it will automatically create an implementing class to access the DB. We can use the Repository via dependency injection:

@Service
public class WeatherService {
  private final WeatherRepository repo;
  public WeatherService(WeatherRepository repo) { this.repo = repo; }

  public WeatherPO find(Long id) {
    return repo.getOne(id);
  }
}

For a complete list of all JpaRepository functions, like getOne() used in the code above, checkout out the official documentation here.

So far, we’ve covered all Java code necessary to access a database. To make everything work, we additionally need the right Maven dependency: Spring’s spring-boot-starter-data-jpa. There exists a myriad of SQL databases, if we choose to use the simple H2 database, we need one more Maven dependency: com.h2database.h2.

Optionally, we can configure our database (username, password, URL, etc). Note that the H2 database will work out of the box, but for others like an Oracle DB, you would need this additional configuration. Namely, you can force the search of repositories with @EnableJpaRepositories and you can create your own database related beans with the @EnableTransactionManagement annotation.

A fully working example can be found in my Github repo in:

@entity, @ Id, @GeneratedValue -> WeatherPO.java
@Repository -> WeatherRepository.java
@EnableJpaRepositories, @EnableTransactionManagement -> JpaConfig.java

Miscellaneous Annotations

Covered in this section:

@Autowired
@ConfigurationProperties

@ConfigurationProperties can be used to automatically fetch configuration values from a .properties or .yaml file and put them into our Java class with the same data structure. This allows us to set config values in a configuration file, which can be easily altered later.

@Autowired forces dependency injection. We need to start the dependency-injection-mechanism at the root of the dependency chain. When we ask Spring to initialize the top-level object, this will trigger the initialization and injection of all objects in the dependency chain. The @Autowired annotation can be used for this purpose and put before the top-level object.

A fully working example can be found in my Github repo in:

@ConfigurationProperties -> AnnotationsAppProperties.java
@Autowired -> WeatherIntegrationTest.java

Annotations for Testing

Covered in this section:

@SpringBootTest
@AutoConfigureMockMvc

Spring offers annotations just for testing purposes, a great feature to make your integration tests simpler! Since this post is already a bit longer than originally intended, I’ll stop here and refer you to the fully working example in WeatherIntegrationTest.java.

Summary

Congrats, you made it to the finish of this (rather lengthy) post😊 I hope this post has helped you to understand the most important annotations of Spring. If you want to know more details, I encourage you to have a look at my Github repo with a fully working Spring Boot service using all annotations from this post.

Also, if you have a Spring annotation which you consider essential and it’s not covered here, please leave a comment and I will try to extend this article or creating a new one which includes your annotation.

Finally, please remember to heart❤️ this article if you found it useful!

Simple working example of REST Web Service with Spring Boot

Philipp Gysel — Fri, 01 May 2020 09:19:01 +0000

In this we will look at a very basic web service which can respond to HTTP requests. This blog is perfect for people who want to get started using Spring Boot.

Why Spring Boot?

So why should you as Java developer care about Spring Boot? Well there are many good reasons! 😊 First of all Spring is open source, meaning it is continuously maintained and tested by community and it is free or charge. Second, according to Hotframeworks, it is the most widely used Java web framework of 2019. Third, there’s an excellent way to get your Spring application up and running quickly, which is where Spring Boot comes into play: Thanks to Spring Boot, you don’t need to worry about a lot of the boiler plate code and configuration. Spring Boot automatically sets a lot of config defaults for you, but you can always overwrite those if needed. For that purpose, Spring Boot is opinionated, meaning the people in the Spring team chose some configs for you, but those are well accepted by the community.

Btw, why should we care about web frameworks at all? Well there are many items which are used over and over in typical web services, such as answering to HTTP request, spanning new threads for each incoming request, security mechanisms like HTTPS and OAUTH2 and so forth. We do not want to reinvent the wheel every time we create a new web service, and for that purpose we can use web frameworks with all those common mechanisms provided. Additional features from web frameworks include data base access, scheduling of tasks, inversion of control etc. All these nice features are included in Spring Boot and thus you have more time for other stuff like drinking a good cappuccino☕

As a final introductory remark, let me mention that Spring is not only compatible with Java, but also with Kotlin, a language very popular for Android apps.

Prerequisites

We will now create a hello-world web service. All necessary code is given here, and the final solution is also available on my Github repo.

Requisites to follow all the steps:

Maven
Java JDK 8 or higher
Command line

For this blog, we will do all the work from the command line. Alternatively, you can use an IDE like IntelliJ. In fact, I will soon release a post on IntelliJ and cover introductory topics like code completion, searching for a given code snippet in your project, compilation, debugging etc.

Using the Spring initializr

We use Maven as build tool, and Spring Boot offers a great way to create your POM file: Head over to https://start.spring.io/ and enter all the details of our app like below:

You can use a newer version of Spring Boot and Java – of course - if you prefer to. Anyways, remember to add “Spring Web” as a starter dependency – we will use it for our REST endpoints. Once you have filled in all details, use the “GENERATE” button. This will download a ZIP file with the initial Java project structure, and most importantly, the initial pom.xml file.

Let us have a closer look at the generated POM file. At the top of the POM, you can see we inherit from spring-boot-starter-parent, which contains all necessities for a Spring-Boot app.

  <parent>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-parent</artifactId>
    <version>2.2.6.RELEASE</version>
    <relativePath/> <!-- lookup parent from repository -->
  </parent>

Further down in the POM, under dependencies, you can see we will use spring-boot-starter-web:

    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-web</artifactId>
    </dependency>

You can find a nice description of this dependency on mvnrepository.com:

Starter for building web, including RESTful, applications using Spring MVC. Uses Tomcat as the default embedded container.

Anyways, so far, we have only looked at one important file: the pom.xml file. Next let us focus on the Main Class, which you can find under src/main/java/com/example/springbootexample/SpringBootExampleApplication.java:

@SpringBootApplication
public class SpringBootExampleApplication {

  public static void main(final String[] args) {
    SpringApplication.run(SpringBootExampleApplication.class, args);
  }

}

What is interesting here is just the annotation at the top: @SpringBootApplication. Among several things, this annotation makes sure our Spring Boot app gets configured with the default Spring Boot properties (like timeouts for HTTP requests and many, many other things).

Hello-World REST endpoint

Since we want to create a REST endpoint later on, we need our Main class to search for Servlets, and therefore we need to add one more annotation to our Main class: @ServletComponentScan (again, if today is your lazy day and you don’t want to do any coding, you can look at the completed code in my Github repo).

Next, let us create a REST endpoint. For this purpose, we create a new Java class and call it PingRestController.java (you can use the same folder as for the Main class).

The content of PingRestController.java should look like so:

package com.example.springbootexample;

import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import org.springframework.web.bind.annotation.RestController;

@RestController
public class PingRestController {

  @RequestMapping(method = RequestMethod.GET, path = "/api/ping")
  public ResponseEntity<String> getPing() {
    return ResponseEntity.ok("pong");
  }

}

The annotation @RestController signifies that this class contains REST endpoints. Each REST endpoint is a method annotated with @RequestMapping. In this particular case, we have only one such method: getPing. This method is executed every time the corresponding REST call arrives at our server. Let us look more in detail at the @RequestMapping annotation: We specify a method and a path variable. These two variables specify that we want to capture HTTP GET request to the URI “/api/ping”. Also, note the return type of our getPing method: A ResponseEntity wraps the HTTP answer, and the HTTP body should be just a String. Thus, the response to the HTTP call will always looks as follows:

Headers: Status: 200, ContentType: text/plain;charset=UTF-8
Body: "pong"

With the modified Main class and the PingRestController class, we have all pieces ready to run our service. In the terminal, type:

mvn clean install
java -jar target/spring-boot-example-0.0.1-SNAPSHOT.jar

Now, in your favorite web browser, type:

localhost:8080/api/ping

You should see the “pong” response!

What happens in the background is that your browser fires a HTTP GET request to localhost, which is handled by your Spring Boot app and responded to with the String “pong”.

Integration Test

A great way to make sure our REST endpoint really works, is by writing an integration test. This test will run every time we build our application. Why do we use integration tests? First, because we developers want to automate everything and do not like testing manually. Second, because this adds stability to future development: As our web service will be extended, this test will still run with every build, and we can be sure this feature still works.

What is an integration test? Contrary to unit tests which only look at one class, an integration test is for our app as a whole, where all components get integrated together. We typically mock third party systems like data bases, so we can test independent of (sometimes unreliable) surrounding systems. In our case, we want to really boot up our web service, but if we had a data base, then we would just simulate it.

We implement our integration test in src/test/java/com/example/springbootexample/PingIntegrationTest.java:

package com.example.springbootexample;

import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.autoconfigure.web.servlet.AutoConfigureMockMvc;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.test.context.junit.jupiter.SpringExtension;
import org.springframework.test.web.servlet.MockMvc;

import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.get;

@ExtendWith(SpringExtension.class)
@SpringBootTest
@AutoConfigureMockMvc
public class PingIntegrationTest {

  @Autowired
  private MockMvc mvc;

  @Test
  public void testHelloWorldEndpoint() throws Exception {
    String response = mvc
        .perform(get("/api/ping"))
        .andReturn().getResponse().getContentAsString();
    assertEquals("Hello world", response);
  }
}

As you can see, testing a REST endpoint takes slightly more code. For now, let us just focus on the interesting bits, and I will leave it to you to understand every single line of code. So here are the important points:

@SpringBootTest: Start the web server locally and have it ready to answer to test REST calls
private MockMvc mvc: A MockMvc object allows us to fire test HTTP calls to our web server
@Test: Just as with Junit-Tests, each test is implemented in a method annotated with @@test.
mvc.perform(get(“api/ping”)): Here we trigger an HTTP GET request

You can run the test with the following command:

mvn -Dtest=PingIntegrationTest test

Aaaaand... the integration test fails🙈 What happened? Well no need to worry, there is just a tiny error in the above test case. I will leave it up to you to find the error cause and fix it!

Spring Configuration

Spring Boot automatically configures many properties of your service. For example, when we pinged our own service, we had to use port 8080. Now we did not define this anywhere… this is just a Spring Boot default. All those default properties can be found in the official docu here. To change this default behavior, all we need to do is create an application.properties file and overwrite the appropriate value. So, go ahead and modify src/main/resources/application.properties:

server:
  port: 8082

Now, when you recompile and restart the service, the REST endpoint will be available on port 8082.

Conclusion

We have covered all code necessary to create a simple REST service. Using Spring Boot, we just needed a total of 23 lines of Java code to create a working REST endpoint! Moreover, there was zero XML configuration needed. Pretty cool!

I hope this post has helped you to get started with Spring Boot, please remember to heart❤️ it and leave a comment below! 😊

Introduction to Convolutional Neural Networks

Philipp Gysel — Tue, 21 Apr 2020 13:55:49 +0000

In this introductory post on Convolutional Neural Networks, we will cover:

Some applications
The most important layer types

While talking about the different layers, we won’t focus on the math but rather on the benefit of each layer as well as the simplified forward path.

Introduction to Convolutional Neural Networks (CNN)

Machine Learning offers many different models for supervised learning, and CNNs are one such model type. CNNs are part of the artificial neural network family, with the important addition that they contain convolutional layers. CNNs have many applications, such as image recognition and natural language processing. Let’s also mention that CNNs are inspired by the visual cortex of animals: neurons in the network only respond to the stimuli of a restricted region from the previous layer.

Why should you care about CNNs? Well a lot of the most challenging AI problems are actually solved with CNNs!😊

Applications

Let’s look at three innovative applications that work thanks to CNNs!

Object detection:

MobileNets, developed by Google Research, can be used for many tasks, among them object detection. The advantage of MobileNets: this neural network family is very resource efficient, which can be crucial for applications running on smartphones or autonomous cars.

Face recognition:

Stevens Institute of Technology used CNNs for face detection, one possible application: focus smartphone camera on the people, not the background.

Board game Go:

DeepMind used a combination of deep neural network and tree search to beat the best human player at the game of Go.

The layers

Convolutional neural networks typically contain many layers – some models up to 100 layers and more! Each layer is supposed to extract features – that is, interesting characteristics – from the input. Since CNNs often consist of so many layers, they are part of the deep learning models.

Image credit: Google Research

Above, you can see the picture of a deep CNN called GoogleNet. As the name suggests, it was designed by Google. On the very left of the picture, the input (that is, an image in RGB format) is fed into the CNN. The first layers extract very simple features like lines, dots and edges in the image. The next layers extract more advanced features like objects and shapes. At the very right of the picture above, the network will extract high level features which enable to the network to distinguish between 1000 image categories.

You might ask: So which layers exist and when should I use them? Well I’m glad you asked: This blog will answer exactly this question! 😊 First let’s get an overlook of the most important layer types:

Layer type	Benefit	Has learnable parameters?
Convolution	Extract feature from small region of input	Yes
Fully connected	Extract feature from whole input	Yes
ReLU	Recognize non-linear properties	No
Pooling	Resilient against small distortion in input	No
Batch Normalization	Keep neuron values in range	Yes
Softmax	Get class probabilities	No
Loss	Calculate the prediction error	No

Now let’s go through all layer types:

Convolutional layer

Convolutional layers are typically where most of the computation happens in a deep CNN: this layer type extracts features using a K x K convolution filters. An example is given below, where the green input is convolved with the yellow filter and thus the red output is generated. In this example, the filter dimension K equals 3. The output shows high values in spatial locations where the input looks similar to the filter.

Image credit: Stanford Tutorial

As you can see, we extract the same feature at different spatial locations. This means the output only depends on a small region of the input.

In reality, things are slightly more complicated than in the image above: the input to a convolutional layer consists of multiple feature maps, and the output too. So, for N input feature maps and M output feature maps, we need filters in the total size of N * M * K * K.

Convolutional layers are typically the bottle neck in terms of arithmetic operations. In most deep learning frameworks, convolution is implemented via data rearranging and matrix multiplication. Also, deep learning training is very resource intense, but offers a lot of room for parallelism. For this purpose, people often use Graphics Processing Units for training, and matrix multiplications are heavily optimized on modern GPUs.

For a great way to learn more about convolution, you can check this course offered by Stanford University. They also cover more details on CNNs in general.

Fully Connected Layer

Alright, let’s move on to the next layer type, namely the fully connected layer. This layer extracts features over the whole input, meaning each output node depends on all input nodes. Mathematically, the output vector y is the result of the weighted input vector x:

y = W * x

The weight matrix W is of dimension |x|*|y|, which brings us to the drawback of this layer type: The size of the parameters tends to be very large, which is why most modern network architectures avoid fully connected layers.

Activation Function: ReLU

Convolutional neural networks can capture complex properties, which wouldn’t be recognized by a simple linear regression. This is largely thanks to activation functions inside a neural network: They introduce a non-linearity into the forward path of the network. Activation functions are typically appended to the convolutional layers and they are applied to each output node independently.

Traditionally, people used different activation functions like Sigmoid, Tanh and ReLU. The last one has proven to be the most effective for deep learning. The picture below shows all three activation functions:

The activation function ReLU (Rectified Linear Unit) keeps all positive values untouched, but negative values get set to zero:

y = max(0, x)

Pooling Layer

Pooling layers reduce the dimension by combining several neurons to one. There are two pooling types: average pooling and max pooling. Many networks use max pooling: here, the output node is equal to the maximum of the input region.

Image credit: Stanford Tutorial

Batch Normalization

For different reasons which we’ll discuss in a second, we want our activations to stay in a reasonable number range during training. However, the above presented ReLU activation function is unbounded. To make up for this, we typically use normalization layers at regular intervals within the network. There are two important normalization layer types: LRN (local response normalization) and BN (batch normalization). While older networks used LRN, newer ones often use BN. BN was originally proposed by Google Research and it helps in many ways: the training is easier to set up and goes faster, and the end accuracy is better. What BN basically does, is it rescales and re-centers the data.

So why is it important to keep our data in a reasonable range? There are two main reasons: First we need to represent our data using some number format (typically with 32-bit or 64-bit floating point), and we simply can’t represent huge numbers accurately. Second, we want high values in our feature maps to represent strong feature existence. But values down the layer chain would possibly become huge, thus we can’t easily estimate whether a given feature is actually present or not.

Once the network is trained and then used for inference, the BN layer can be simplified to a linear transformation:

y = γ * x + β

Softmax

The output of a classification network is a vector of probabilities. To achieve this, we use a Softmax layer, which normalizes the output nodes such that the sum of all nodes is 1. The formula of Softmax is as follows:

y = exp(x) / sum(exp(x))

Loss

The loss layer is used during training only. It compares the network output with the ground truth and computes the “error” of our prediction. The resulting loss is then backpropagated through the network to adjust all learnable parameters. Various loss functions exist, and the choice of loss function will often influence the final result significantly.

For multi-class prediction networks, we typically use Sparse Cross Entropy as loss function, which is equal to the negative log probability of the true class. If our network is sure of the correct class, the loss is zero, and if the network is totally wrong, the loss is infinitely big.

Conclusion

In this post, we first covered the benefits of neural networks: for tasks like object detection, they deliver state-of-the-art results. The second part of the blog was dedicated to the different layer types which are commonly used in CNNs. If you wish to dive deeper and gain more hands-on experience with CNNs, I can suggest this TensorFlow CNN Tutorial.

Thanks so much for reading this blog, I hope it has helped you to get started with CNNs.😊

DEV Community: Philipp Gysel

Learn Aspect Oriented Programming by Example

pmgysel / aop-examples

Some examples of Aspect Oriented Programming (AOP) with SpringBoot

What is an Aspect?

@Cacheable: a standard Spring Advice

Log REST calls with a custom Aspect

Performance monitoring with AOP

Retry mechanism with AOP

Conclusion

pmgysel / aop-examples

Some examples of Aspect Oriented Programming (AOP) with SpringBoot

Aspect Oriented Programming with Java and SpringBoot

pmgysel / aop-demo

Demo for Aspect Oriented Programming

Programming Paradigms

Why AOP?

Maven Dependency

Advice

Pointcut

Aspect

Enable AOP

Conclusion

pmgysel / aop-demo

Demo for Aspect Oriented Programming

Demo for Aspect Oriented Programming

Dependencies

Aspects for REST calls

Usage

Sample aspect output

Further readings

From Java 8 to Java 15 in Ten Minutes

Functional Programming (Java 8)

Streams (Java 8)

Optional (Java 8)

JShell (Java 9)

Factory Methods for Immutable Collections (Java 9)

Type Inference with var (Java 10)

Single Source File Launch (Java 11)

Switch Expression (Java 12)

Multi-line Strings (Java 13)

Data Classes: record (Java 14)

instanceof without Cast (Java 14)

Sealed classes (Java 15)

Bonus: Updated Licensing Terms Starting with Java 8

Wrap up

From Legacy Projects to CI/CD Pipelines

CI / CD Theory

Flee from the Problem?

Version Control

Compile

Code quality

Deploy to Test System

End-to-End Tests

The Final CI/CD Pipeline

Log Capturing App with MongoDB + Spring Boot + Swagger UI

The Challenge of Schemaless Data

The SQL Approach

The NoSQL Approach

Implement the Log Capturing App

MongoCollection Configuration

MongoDB Queries

Java Date Model

Document to LogEvent Mapper

REST Controller

Spring Boot Service

Swagger UI Documentation

Run the App

pmgysel / mongodb-logging-app

Demo app for MongoDB: manage log events of varying schema type

Conclusion

MongoDB CRUD Queries from Java and Shell

Introduction to MongoDB and Document Databases

Data Model

How to Store and Manipulate Data

MongoDB

MongoDB Queries

MongoDB Naming Convention

MongoDB - No Schema

MongoDB Replica Sets

`@Cacheable`: a standard Spring Advice

Type Inference with `var` (Java 10)

Data Classes: `record` (Java 14)

`instanceof` without Cast (Java 14)