DEV Community: Cosmas Gikunju

Consuming and Testing third party API's using Spring Webclient

Cosmas Gikunju — Mon, 19 Feb 2024 16:33:06 +0000

Introduction

More often than not developer's build and call API's. Therefore it's guaranteed that you are going to have to build or consume an API sometime in your day to day development. In this article I will show you how you can build a springboot service that calls a third-party api and most importantly we are going to write test to guarantee our consuming service works as expected.

Getting Started

Start a new Spring Boot Project on Spring Initializr here or just create from the command line using a http request to Spring Initializr as follows:



curl --location 'https://start.spring.io/starter.zip?type=maven-project&language=java&bootVersion=3.2.2&baseDir=ms-xcoffee&groupId=com.xcoffee&artifactId=ms-xcoffee&name=ms-xcoffee&description=Demo%20project%20for%20Spring%20Boot&packageName=com.xcoffee.ms-xcoffee&packaging=jar&javaVersion=21&dependencies=webflux%2Clombok%2Cvalidation' | tar -xzvf -

Add Spring Webflux, Validation and Lombok as the starting dependencies. (Above is your first first API interaction).

Building

We are going to use https://sampleapis.com/api-list/coffee as our third party API. They have provided us a very nice API endpoint that accepts a GET request and responds with a list of JSON representation of the coffee data.



GET /coffee/iced HTTP/1.1
Host: api.sampleapis.com

Curl Request:



curl --location 'https://api.sampleapis.com/coffee/iced'

Response:



[
    {
        "title": "Iced Coffee",
        "description": "A coffee with ice, typically served with a dash of milk, cream or sweetener—iced coffee is really as simple as that.",
        "ingredients": [
            "Coffee",
            "Ice",
            "Sugar*",
            "Cream*"
        ],
        "image": "https://upload.wikimedia.org/wikipedia/commons/d/d8/Blue_Bottle%2C_Kyoto_Style_Ice_Coffee_%285909775445%29.jpg",
        "id": 1
    }, ...
]

Next we will build a POJO(plain old java object) representation of the json response.



package com.xcoffee.msxcoffee.schema.pojos;

import com.fasterxml.jackson.annotation.JsonProperty;
import lombok.AllArgsConstructor;
import lombok.Getter;
import lombok.NoArgsConstructor;
import lombok.Setter;

import java.util.List;


@Getter
@Setter
@AllArgsConstructor
@NoArgsConstructor
public class CoffeeResponse {
    @JsonProperty("title")
    private String title;
    @JsonProperty("description")
    private String description;
    @JsonProperty("ingredients")
    private List<String> ingredients;
    @JsonProperty("image")
    private String image;
    @JsonProperty("id")
    private int id;
}

The above class maps the json data to a java object we can work with. We use Lombok to generate constructors, getters and setters for our code and the Jackson Project to handle serialization and deserialization of json to pojo . We know the response is an array of objects representing the coffee and so above data structure is fit for this.

Webclient Action

We will build an interface in which we will base our api consumer implementation. The interface supports Polymorphism and allows us to use different implementations for different purposes e.g. testing.



package com.xcoffee.msxcoffee.services;

import com.xcoffee.msxcoffee.schema.pojos.CoffeeResponse;
import reactor.core.publisher.Mono;

import java.util.List;

public interface CoffeeService {
    Mono<List<CoffeeResponse>> getHotCoffees();
    Mono<List<CoffeeResponse>> getIcedCoffees();
}

Next we dive into the actual implementation.
We will create a class called CoffeeServiceImpl.java that implements our interface methods.



package com.xcoffee.msxcoffee.services.impl;

import com.xcoffee.msxcoffee.schema.pojos.CoffeeResponse;
import com.xcoffee.msxcoffee.services.CoffeeService;
import reactor.core.publisher.Mono;

public class CoffeeServiceImpl implements CoffeeService {
    @Override
    public Mono<List<CoffeeResponse>> getHotCoffees() {
        return null;
    }

    @Override
    public Mono<List<CoffeeResponse>> getIcedCoffees() {
        return null;
    }
}

Then we build our api consumption logic.

But wait, we need to configure WebClient , here's the implementation.



package com.xcoffee.msxcoffee.config;

import io.netty.channel.ChannelOption;
import io.netty.handler.ssl.SslContext;
import io.netty.handler.ssl.SslContextBuilder;
import io.netty.handler.ssl.util.InsecureTrustManagerFactory;
import io.netty.handler.timeout.ReadTimeoutHandler;
import io.netty.handler.timeout.WriteTimeoutHandler;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.http.client.reactive.ClientHttpConnector;
import org.springframework.http.client.reactive.ReactorClientHttpConnector;
import org.springframework.web.reactive.function.client.ExchangeStrategies;
import org.springframework.web.reactive.function.client.WebClient;
import reactor.netty.http.client.HttpClient;

import javax.net.ssl.SSLException;
import java.time.Duration;
import java.util.concurrent.TimeUnit;

@Configuration
public class WebClientConfig {

    @Bean
    public WebClient getWebClient() throws SSLException {

        ExchangeStrategies exchangeStrategies = ExchangeStrategies.builder()
                .codecs(configurer -> configurer.defaultCodecs()
                        .maxInMemorySize(1048576)) // Set buffer size to 1 MB
                .build();

        // Disable ssl verification
        SslContext context = SslContextBuilder.forClient()
                .trustManager(InsecureTrustManagerFactory.INSTANCE)
                .build();

        HttpClient httpClient = HttpClient.create()
                .secure(t -> t.sslContext(context))
                // .proxyWithSystemProperties() // Use JVM level System proxy
                .responseTimeout(Duration.ofSeconds(30))
                .option(ChannelOption.CONNECT_TIMEOUT_MILLIS, 30 * 1000)
                .doOnConnected(conn -> conn
                        .addHandlerLast(new ReadTimeoutHandler(30,
                                TimeUnit.SECONDS))
                        .addHandlerLast(new WriteTimeoutHandler(30)));

        ClientHttpConnector connector = new ReactorClientHttpConnector(httpClient);

        return WebClient
                .builder()
                .exchangeStrategies(exchangeStrategies)
                .clientConnector(connector)
                .build();
    }
}

Then we can use WebClient in our Service Impl class:

Here's the code:



package com.xcoffee.msxcoffee.services.impl;

import com.xcoffee.msxcoffee.config.AppConfig;
import com.xcoffee.msxcoffee.schema.pojos.CoffeeResponse;
import com.xcoffee.msxcoffee.services.CoffeeService;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.http.HttpMethod;
import org.springframework.http.HttpStatusCode;
import org.springframework.http.MediaType;
import org.springframework.stereotype.Service;
import org.springframework.web.reactive.function.client.WebClient;
import reactor.core.publisher.Mono;

import java.util.List;

@Service
@RequiredArgsConstructor
@Slf4j
public class CoffeeServiceImpl implements CoffeeService {

    private final WebClient webClient;
    private final AppConfig cfg;

    @Override
    public Mono<List<CoffeeResponse>> getIcedCoffees() {
        return webClient.method(HttpMethod.GET)
                .uri(cfg.getCoffeeURL() + "/iced")
                .contentType(MediaType.APPLICATION_JSON)
                .retrieve()
                .onStatus(HttpStatusCode::isError, response -> {
                    log.error("An Error Occurred Getting Iced Coffees");
                    return Mono.just(new Exception("An Error Occurred Getting Iced Coffees"));
                })
                .bodyToFlux(CoffeeResponse.class)
                .collectList();
    }

    @Override
    public Mono<List<CoffeeResponse>> getHotCoffees() {
        return webClient.method(HttpMethod.GET)
                .uri(cfg.getCoffeeURL() + "/hot")
                .contentType(MediaType.APPLICATION_JSON)
                .retrieve()
                .onStatus(HttpStatusCode::isError, response -> {
                    log.error("An Error Occurred Getting Hot Coffees");
                    return Mono.just(new Exception("An Error Occurred Getting Hot Coffees"));
                })
                .bodyToFlux(CoffeeResponse.class)
                .collectList();
    }
}

The code above uses org.springframework.web.reactive.function.client.WebClient to perform GET request to the respective coffee endpoints and then map the responses to the List<CoffeeResponse object which is a List of Coffees.

@RequiredArgsConstructor creates a webclient and config constructor for the service class.

We also created a config class to externalize configurations such as API URLS :



package com.xcoffee.msxcoffee.config;


import lombok.Getter;
import lombok.Setter;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.context.annotation.Configuration;

@Getter
@Setter
@Configuration
@ConfigurationProperties(prefix = "app")
public class AppConfig {
    private String coffeeURL;
}

Unit Testing and Mocking

Suppose the HTTP calls to the coffee endpoints perform a data update or a paid transaction, then we cannot perform test by calling the real endpoint. So here comes Mocking .

We will use Square’s Mock Webserver to spin up a mock server which we can use to simulate real api's request to the get coffee endpoint.

Add the dependency as follows:



        <dependency>
            <groupId>com.squareup.okhttp3</groupId>
            <artifactId>mockwebserver</artifactId>
            <version>5.0.0-alpha.12</version>
        </dependency>
        <dependency>
            <groupId>com.squareup.okhttp3</groupId>
            <artifactId>okhttp</artifactId>
            <version>5.0.0-alpha.12</version>
        </dependency>

Checkout the latest version here on maven central https://central.sonatype.com/artifact/com.squareup.okhttp3/mockwebserver/5.0.0-alpha.12

Then we can write our test as follows:



package com.xcoffee.msxcoffee.services.impl;

import com.xcoffee.msxcoffee.config.AppConfig;
import com.xcoffee.msxcoffee.schema.pojos.CoffeeResponse;
import com.xcoffee.msxcoffee.services.CoffeeService;
import jakarta.validation.constraints.NotNull;
import okhttp3.mockwebserver.MockResponse;
import okhttp3.mockwebserver.MockWebServer;
import org.junit.jupiter.api.*;
import org.springframework.http.HttpHeaders;
import org.springframework.http.HttpStatus;
import org.springframework.http.MediaType;
import org.springframework.web.reactive.function.client.WebClient;
import reactor.core.publisher.Mono;
import reactor.test.StepVerifier;

import java.io.IOException;
import java.util.List;
import java.util.Objects;

class CoffeeServiceImplTest {


    private static MockWebServer mockWebServer;
    private static CoffeeService coffeeService;

    private static final String COFFEE_API_URL = "dummy";

    @BeforeAll
    static void setUp() {
        AppConfig cfg = new AppConfig();
        mockWebServer = new MockWebServer();
        WebClient mockedWebClient = WebClient.builder()
                .baseUrl(mockWebServer.url(COFFEE_API_URL).toString())
                .build();

        coffeeService = new CoffeeServiceImpl(mockedWebClient,cfg);

    }

    @AfterAll
    static void tearDown() throws IOException {
        mockWebServer.close();
    }

    @Test
    @DisplayName("Unit | Get Hot Coffees Success")
    void getHotCoffeeSuccess() {
        // Enqueue a successful response
        mockWebServer.enqueue(
                new MockResponse().setResponseCode(HttpStatus.OK.value())
                        .setHeader(HttpHeaders.CONTENT_TYPE, MediaType.APPLICATION_JSON_VALUE)
                        .setBody(getCoffeeSuccess()));

        // Test the success case
        Mono<List<CoffeeResponse>> responseMono = coffeeService.getHotCoffees();

        StepVerifier.create(responseMono)
                .assertNext(response -> {
                    Assertions.assertInstanceOf(List.class, response);
                })
                .verifyComplete();

    }

    @Test
    @DisplayName("Unit | Get Hot Coffees Failure")
    void getHotCoffeeFailure() {
        // Enqueue a failed response
        mockWebServer.enqueue(new MockResponse()
                .setResponseCode(HttpStatus.INTERNAL_SERVER_ERROR.value())
                .setBody("{}")
        );

        // Test the failure case
        Mono<List<CoffeeResponse>> hotCoffees = coffeeService.getHotCoffees();

        StepVerifier.create(hotCoffees)
                .expectErrorMatches(throwable -> throwable instanceof Exception
                        && Objects.equals(throwable.getMessage(),
                        "An Error Occurred Getting Hot Coffees"))
                .verify();

    }

    @NotNull
    private String getCoffeeSuccess() {
        return """
                [
                     {
                         "title": "Iced Coffee",
                         "description": "A coffee with ice, typically served with a dash of milk, cream or sweetener—iced coffee is really as simple as that.",
                         "ingredients": [
                             "Coffee",
                             "Ice",
                             "Sugar*",
                             "Cream*"
                         ],
                         "image": "https://upload.wikimedia.org/wikipedia/commons/d/d8/Blue_Bottle%2C_Kyoto_Style_Ice_Coffee_%285909775445%29.jpg",
                         "id": 1
                     }
                 ]
                """;
    }
}

We initialize the components to be mocked via setUp() method annotated with @BeforeAll, here we also configure the mock server and mock webclient and inject to our coffeeService , remember the interface we created? That helps us easily modify the implementation of coffee service and this is what achieves mocking.

We Enqueue success and failure responses respectively inside the test methods and finally use project reactor Step Verifier to test the events that happen upon subscription.

Finally run the test with IntelliJ, VS Code, Maven or tool of your choice they run as follows:

IntelliJ
Maven

mvn test

Conclusion

In this article we learnt how we can consume API's using non blocking WebClient and mock API calls using MockServer.

How to add SonarQube Code Coverage to Spring Boot

Cosmas Gikunju — Wed, 07 Feb 2024 12:34:21 +0000

1. Overview

SonarQube is a self-managed static code analysis tool for continuous codebase inspection provided by SonarSource.

It's a popular choice used by organizations to :

Finding and fix bugs and security vulnerabilities in code.
Analyze code with Static Application Security Testing (SAST).
Detect a broad range of security issues such as SQL injection vulnerabilities, cross-site scripting (XSS) code injection attacks, buffer overflows, authentication issues, cloud secrets detection and much more.
Perform branch analysis to spot and eliminate bugs.

In this article we will look at how to add Coverage to your Spring Boot and Java application.

2. Integrating Sonarqube to your spring boot project

Add JaCoCo plugin to your dependencies on the pom.xml file as follows:



<dependency>
    <groupId>org.jacoco</groupId>
    <artifactId>jacoco-maven-plugin</artifactId>
    <version>0.8.11</version>
</dependency>

Work with the version of choice , you can search at Maven Central https://central.sonatype.com/artifact/org.jacoco/jacoco-maven-plugin

Then add the following under build plugins:



<build>
   <plugins>
      <plugin>
         <groupId>org.jacoco</groupId>
         <artifactId>jacoco-maven-plugin</artifactId>
         <version>0.8.11</version>
         <executions>
            <execution>
               <id>prepare-agent</id>
               <goals>
                  <goal>prepare-agent</goal>
               </goals>
            </execution>
            <execution>
               <id>report</id>
               <goals>
                  <goal>report</goal>
               </goals>
            </execution>
         </executions>
      </plugin>
   </plugins>
</build>

There is a very good post at https://community.sonarsource.com/t/coverage-test-data-importing-jacoco-coverage-report-in-xml-format/12151 that explains importing JaCoCo coverage report in XML format.

And voila, that's all you need to do.

3. Testing

Download and run sonarqube via docker: docker run -d -p 9000:9000 sonarqube

Then access the dashboard at : http://localhost:9000

Back at your project directory run mvn clean install to build your code then mvn sonar:sonar to sync to sonarqube.
Back at your sonar dashboard you will see your coverage info as follows:

4. Caveat

To exclude packages or files from the coverage add them as following in the properties section of your pom.xml :



<properties>
   <java.version>21</java.version>
   <jacoco.version>0.8.11</jacoco.version>
   <sonar.exclusions>**/schemas/**,**/config/**</sonar.exclusions>
   <sonar.coverage.exclusions>**/schemas/**,**/config/**</sonar.coverage.exclusions>
</properties>

Run mvn clean install then mvn sonar:sonar and your coverage will update. If a devops pipeline is set, just push your changes and you will see them at your sonarqube dashboard.

You can also add the Sonarlint plugin/extension to your IDE or Code Editor to allow you catch most of the issues before you commit or build.

How to Set Up a Local Development Workflow with Docker for your Go Apps (with MongoDB and Mongo Express)

Cosmas Gikunju — Fri, 15 May 2020 14:59:42 +0000

Cover Photo by Etienne Girardet on Unsplash

Setting up a development workflow can be time consuming and frustrating especially when there is a need to install a lot of tools. But with the advent of containers this became easy, way easy. In this article I'll explain how I set up my dev workflow for Go Apps being my primary working tool.

So in setting up a new project I start out by listing out the tooling I need for maximum productivity along the way. In this guide, I'll build an extremely easy Go api with just two endpoints, one that receives a GET Request to the base endpoint / and returns a welcome message and another one that simply receives a book as json object from a POST Request to the api endpoint /books then saves this book in a MongoDB collection. So to work on this project I need the following tools:

Golang and the apps dependencies
Mongo DB as the database
Mongo Express (optional) for my database admin interface

To start things of, open your terminal or cmd window and navigate to your desired working directory. In my case I have a folder named projects in my home directory where I organize my projects.

Clone this repository https://github.com/ItsCosmas/docker-go-dev
```
git clone https://github.com/ItsCosmas/docker-go-dev
```
Navigate to the newly created directory
```
cd docker-go-dev
```
Once Inside the docker-go-dev directory, open with your favorite editor. In my case VS Code.
```
code .
```
Make Sure your directory tree looks like this.

Open the go-app folder and Look at the Dockerfile.

# Pull base image
FROM golang:1.18-alpine

# Install git
RUN apk update && apk add --no-cache git

# Where our files will be in the docker container 
WORKDIR /opt/go-app

# Copy the source from the current directory to the working Directory inside the container 
# Source also contains go.mod and go.sum which are dependency files
COPY . .

# Get Dependency
RUN go mod download

# Install Air for hot reload
RUN go install github.com/cosmtrek/air@latest

# The ENTRYPOINT defines the command that will be ran when the container starts up
# In this case air command for hot reload go apps on file changes
ENTRYPOINT air

I have well commented the Dockerfile but I will explain a few bits.

A Dockerfile is a text document that contains all the commands a
user could call on the command line to assemble an image.

We start by pulling the base go image from the docker hub. The alpine tag indicates the alpine version which is bit lightweight and should work well in development, I won't go to much optimization for the moment.
Then install git which is a dependency to get some packages.
Then we set the work directory inside the container in my case it is /opt/go-app that is where our files will live inside the container.
Then we copy the source from the current directory to the working Directory set above. Our Source Code also contains go.mod and go.sum which are dependency files
After that we get the app service dependencies using go mod download
We will need Live Reload for our app so we install air https://github.com/cosmtrek/air which is a live reload utility for Go apps.
Finally we pass an ENTRYPOINT which defines the command that will be ran when the container starts up, in our case its air the Live reload utility.

That's it for the go-app service now we move a directory up to the docker-compose.yml file.

This is the content of the compose file:

version: '3.7'
services:
    # The Go App
    go-app:
        build: ./go-app
        container_name: go_app
        depends_on:
            - 'mongo'
        environment:
            - PORT=8000
        ports:
            - '8000:8000'
        volumes:
            - './go-app:/opt/go-app:cached'
    # MongoDB
    mongo:
        image: 'mongo:4.4'
        container_name: mongo
        ports:
            - '27017:27017'
        volumes:
            - ./mongodata:/data/db
        restart: always
    # Mongo Express Web-based MongoDB admin interface
    mongo_express:
        image: 'mongo-express:0.54.0'
        container_name: mongo_express
        depends_on:
            - 'mongo'
        ports:
            - '8081:8081'
        restart: always

I'll explain the core parts of this file.

Compose is a tool for defining and running multi-container Docker applications. With Compose, you use a YAML file to configure your application’s services. Then, with a single command, you create and start all the services from your configuration.

To learn more about docker-compose.yml file head over to the official docs at https://docs.docker.com/compose/.

First we define our version tag for the compose file in our case 3.7 as at the time of writing.

go-app service

Next We will define the services needed for our application. We start with the go-app service which we just created above, remember that Dockerfile?
The go-app service uses an image build from the Dockerfile inside the go-app directory.
We give it a custom name in our case go_app.
Then We indicate that the service is dependent on another service called mongo which will be our database service.
Finally We pass in some environment variables for use inside the container. In our case PORT.
Last We bind the container and the host machine to the exposed port 8000.
Finally we define a volume which maps our source directory to the work directory inside the container, that way changes in our source code from our host machine will reflect inside the container.

Mongo Service

First we define the image that shall be used in our mongo service, in our case an official image with the bionic tag.
Then we give it a custom name mongo which is the default, could be anything.
Then We bind the container and the host machine to the exposed port 27017.
Then We define a volume which maps a local directory mongodata to the data directory inside the container, that way we shall have persistent storage even if our container fails.
Lastly we add a restart:always flag to always restart the container if it stops.

Mongo Express (Optional)

A Web-based MongoDB admin interface that will allows to manage our mongo database in the browser.
First We define the image , in our case the official mongo-express image from docker hub.
We give it a custom container name
We add a depends_on flag so that the container only runs if mongo service is running.
Then We bind the container and the host machine to the exposed port 8081.
Lastly we add a restart:always flag to always restart the container if it stops.

Action

Now that everything is set up, it's time to see our containers in action.To start our containers we run:

docker-compose up

If everything is correct docker will build the containers and you should see this output in your terminal indicating your containers were built and started successfully:

There's obviously more output but I'm focusing on this. Once that's done you can hit the base endpoint http://localhost:8000 to verify its working. You can use a tool like postman https://www.postman.com/ or just curl on your terminal.

Open a new Terminal and input:

curl http://localhost:8000; echo

If your app worked, you should see similar output:

{"message":"Welcome to your App on Docker"}

The GET endpoint actually works.
Now let us test the POST Endpoint. On your terminal enter this command:

curl -d '{
    "name":"My Best Book Ever",
    "pages": 12
}' -H "Content-Type: application/json" -X POST http://localhost:8000/books; echo

If it works you should see this Response:

{"code":200,"msg":"Book Published","data":{"name":"My Best Book Ever","pages":12}}

POST Endpoint on Postman:

If success, you should see this output:

To Check Live Reload, Just Edit a file e.g. The `app/app.go` :

    // API Home
    r.GET("/", func(c *gin.Context) {
        c.JSON(http.StatusOK, gin.H{
            "message": "Welcome to your App on Docker",
        })
    })

You can change the message to whatever you want and once you save your file you should see a similar output on your terminal:

To check whether the data was saved to Mongo We'll use Mongo-Express:

Open http://localhost:8081/ in your browser. You should see the database books-db and if not just refresh your browser:
Click on books-db and it should redirect to the Database Page as seen below::
Click on the Collection you wish, in this case book collection and the records from the collection should be displayed as below:
You can also view a Single Document if you wish by just clicking on it and it should redirect to a document page similar to this:

So In this article I have shown you how I set up my local dev environment and workflow for go apps, Mongo and Mongo Express. The process is almost similar for other stacks, the key is to have a standardized dev environment for maximum productivity. By the way shipping to production is very easy with docker you require just a few optimizations and that's all, no OS dependency issues ever.

I have updated the article and the github repository so that this tutorial can be helpful even now as at 2022.

And that's it for the article. PS: Trying to do some regular writing 🙂.