DEV Community: charles1303

Security Using Spring and JWT

charles1303 — Sun, 23 Jun 2019 13:14:24 +0000

Introduction

I've finally been able to get this article out. It took this long because of two major reasons; firstly I was enmeshed in a very interesting project over the past few weeks. It really felt good when we deployed it, did a test run and saw that all the stakeholders were happy about how the product met the expected criteria and the desired features. Secondly and on the sad side, my hard disk crashed and the recovery process was quite daunting. (I'm still taking stock of what I've lost so far especially my write ups and documentations). But thankfully most of my project codes were on their respective git repositories and the delta wasn't that much with what was on my laptop. Which is why I would like to use this opportunity to remind we developers again that we should always push to our code repositories as often as we can to a current working branch. Even if the task is not completely done or the code ain't compiling after working on it over time. It doesn't have to be everyday but at least make sure you commit and push within the week. That push could make you gush later in the future.

Now that we have that out of the way, let's focus on what I really want to talk about in this article. In my last two articles here and here, I demonstrated on how to build REST APIs and integrate them with external APIs using Design patterns and best practices in developing applications. In this article I will be demonstrating on how to implement Security using JWT to secure our REST APIs. I will also highlight other strategies in securing our application especially with respect to data. I know this might be familiar topic, but they are usually written in isolation from the overall context of developing application. So we will be building on what we have developed in the previous articles I wrote. I will still be using the same approach in terms of class, component and method referencing so you might want to access the repo here and also take a look at the previous articles to play catch up a bit.

Code Implementation

Imagine you are a consultant, and you are about to book an appointment with a new client of yours. You know that the drill is to first register with the client as a consultant where you provide all your details that will be kept in their records. This is akin to the Registration process that we do online when you want to use an application for the first time. After registration you then proceed to booking an appointment at an agreed upon date. Upon arrival at the premise, you identify yourself at the reception and if all checks out you are then given a visitor's tag or card which grants you access to certain parts or floors of the building. This process could be mapped to the login process of the application using your credentials when you registered. This is the Authentication process and if your credentials are valid, you are granted access to the application. At the same time you are also issued a JWT(token) (or a session is generated for you in a stateful web application or a cookie for the browser). This token represents the visitor's tag that was issued to you. And just like the tag, token will grant you controlled access to resources in the application domain based on the role(s) assigned to the token. This is used in the Authorization process to determine whether you can access a resource or not based on your role(s) (just as how the visitor's tag will only grant you access to certain floors in the building). Now just like the tag which can only be used for that day only, the token also has an expiration date and becomes invalid hence resulting in authorization failures and denial of access to resources. In some cases though, assuming you are not done with your consultation for that day, the tag can be reused for the next day based upon agreement. In the same vein, the token can also be refreshed upon request and a new expiration date is set for it after validating the old token. In the event, you are done with your consultation, you have to submit the tag before leaving the building. Likewise when you are done using the application, you need to logout so that the token is invalidated. And once you are out of the building or logged out of the application, you will have to repeat the identification process at the reception or the login process again to access the facility or the application in order to perform your activities. Now that we have a picture of how Authentication, Authorization and Token Generation and Management work together, Let's connect the dots using the Spring Security Framework and JWT:

First we create our JwtManager class to handle generating, validating and retrieval of information from our token. Then we create our CustomUserDetails class. This class will store details of the Spring authenticated user or prinicipal. By implementing the org.springframework.security.core.userdetails.UserDetails interface, it can access information such as username, password and the authorities granted to the principal via the user's assigned role(s). Next we create our CustomUserDetailsService class. This class defines the custom implementation of the loadUserByUsername method in the org.springframework.security.core.userdetails.UserDetailsService interface that it implements and based on the outcome returns the CustomUserDetails instance we defined earlier. In this case we are querying the database with the username supplied to see if it exists or not.

We then create our JwtAuthenticationFilter class. This is class filters every incoming request against a protected resource to inspect and validate the token that is sent using the validateToken method defined in our JwtManager class. If valid, it extracts the username (subject) and the roles from the token claims using the getUsernameFromJWT and getRolesFromJwt methods respectively that were defined in our JwtManager class and uses it to create a valid secuity context for the user so the user remains authenticated in the application. We could have equally used a second approach with just the username and the loadUserByUsername method of the CustomUserDetailsService class to achieve this with this code snippet

        String jwt = getJwtFromRequest(request);

            if (StringUtils.hasText(jwt) && jwtManager.validateToken(jwt)) {
                String username = jwtManager.getUsernameFromJWT(jwt);

        //Using an injected CustomUserDetailsService instance
        CustomUserDetails customUserDetails = customUserDetailsService.loadUserByUsername(username);
        UsernamePasswordAuthenticationToken authentication = new UsernamePasswordAuthenticationToken(customUserDetails.getPrincipal(), "", customUserDetails.getAuthorities());
                authentication.setDetails(new WebAuthenticationDetailsSource().buildDetails(request));
                SecurityContextHolder.getContext().setAuthentication(authentication);
            }else {
                SecurityContextHolder.getContext().setAuthentication(null);
            }

The disadvantage with the second approach is that there will be a performance database hit using the loadUserByUsername method for every request unlike the first approach where we gained application performance by retrieving the user's detail from the sent token. On the other hand, the second approach allows for real time modification of granted authorities when user roles are changed whereas the first approach will require the token to expire before it takes effect. Whichever approach you employ here is usually based on a design and requirement decision.

Next we will create our CustomRestAccessDenied and CustomRestAuthenticationEntryPoint classes. As you can see, they implement the org.springframework.security.web.access.AccessDeniedHandler and the org.springframework.security.web.AuthenticationEntryPoint interfaces respectively. The handle method in the CustomRestAccessDenied class is meant to be triggered for requests where users have been authenticated but do not have the permissible role to access the resources and returns a HTTP 403 status while the commence method in the CustomRestAuthenticationEntryPoint class is triggered for unauthenticated users and returns a HTTP 401 status. You will see how this is setup when we look at our Security Configuration class implementation. Also note how the response is tailored to return in a particular format. This is so that we adhere to a consistent response structure in our application. For those who saw my last article on Exception handling, you maybe wondering why didn't we use that approach in this scenario. The reason for this is that the Access Denied and Unauthorized exceptions are thrown at an outer level than the Spring's ControllerAdvise and Exception can handle hence we have to handle these exceptions at the level which they occur.

Now let's take a look at our Security configuration class which is the SecurityConfig class. The SecurityConfig class which extends org.springframework.security.config.annotation.web.configuration.WebSecurityConfigurerAdapter is what glues all the previous security components and classes we have defined so far. It determines the behaviour and roles they play in our application security implementation. The securedEnabled, jsr250Enabled and prePostEnabled annotations allows us to determine what roles should have access to a particular resource. This can be configured either on the methods you want to apply these restrictions to (for example in the controllers) or append it to the antMatchers as can be seen in the configure method of the SecurityConfig class. The CustomUserDetailsService class we defined earlier is injected and specified to be used as our authentication component in the overridden public void configure(AuthenticationManagerBuilder authenticationManagerBuilder) method. We also configure a PasswordEncorder bean to use the Spring's BCryptPasswordEncoder to validate the user's password also in this method. In the overridden protected void configure(HttpSecurity httpSecurity) method, we configured the Exception handlers for the unauthorized and access denied scenarios to use the injected CustomRestAuthenticationEntryPoint and CustomRestAccessDenied classes respectively. We disabled the csrf so that requests from other domains can access our endpoints and then we declare the session management to be stateless so as to conform to REST specifications. We then configure our injected JwtAuthenticationFilter class as the filter that will be called for every request to any of our secured resources. And finally we disable our security for endpoints within the /api/auth/ route as this will be called by unauthenticated users during registration and login.

Let's now create the components that will allow users to be able to register and login on our application. First we create our User class entity that will map to our users table in the database. We also create the Role class entity as well that will map to the roles table. We then insert into the roles table the following roles: ROLE_USER and ROLE_ADMIN. Then we create the UserRepository and UserService classes for managing our User entity and the RoleRepository and RoleService classes to manage our Role entity. For integrity, we create a RoleType enum that maps to the roles we just created in our roles table. We then create our AuthController class to handle registration and login. As you can see, the registration endpoint is pretty straightforward. We create a user with the assigned roles and store in the database. For the login endpoint, you supply a registered username and password and if authentication by the Spring Security framework is successful, a valid secuity context for the user is set using the Spring's Authencation object. The generateToken method of the JwtManager then takes in the authentication as a parameter and sets a Claims object's subject with the username and puts the authorities gotten from the Principal which were all gotten from the authentication object as roles into the Claims object. A token is then generated using the constructed Claims object and setting the expiry date, time it was issued and finally signed with our token secret key.

Test Run

All is now set so we can launch our spring application using mvn spring-boot:run and then we try to register by hitting the endpoint using our SignUpRequest dto class structure:

    POST    http://localhost:8080/api/auth/register
    {

    "name":"user1",
    "username":"username1",
    "email":"username1@email.com",
    "password":"password1"
    }

       Response:
    {
        "status": 0,
        "message": "User registered successfully",
        "result": null
    }

Repeat the above for another user to create and admin

    POST    http://localhost:8080/api/auth/register/admin
    {

    "name":"user2",
    "username":"username2",
    "email":"username2@email.com",
    "password":"password2"
    }

        Response:
    {
        "status": 0,
        "message": "Admin registered successfully",
        "result": null
    }

Login with both users using our LoginRequest dto class structure and get the respective tokens that will be returned for both users

    POST http://localhost:8080/api/auth/login

    {
    "username":"username1",
    "password":"password1"
    }

        Response:
    {
        "status": 0,
        "message": "Token generated successfully",
        "result": {
        "accessToken": "eyJhbGciOiJIUzUxMiJ9.eyJzdWIiOiJ1c2VybmFtZTEiLCJyb2xlcyI6WyJST0xFX0FETUlOIl0sImlhdCI6MTU2MDk4MDU1MywiZXhwIjoxNTYwOTgyMzUzfQ.boqHQh3gLPgNWBP0GiATBGg-25bwMfg33-zn5BFK9AiFuhYcWcmSSFp_isjlhL_xJ9WxMW6A7UWaVOXMdx94ng"
        }
    }

Calling any of our endpoints without a token or an invalid or expired token will give you a response triggered by our CustomRestAuthenticationEntryPoint handler with a HTTP 401 status code

    {
        "timestamp": "2019-06-19T22:51:37.189",
        "message": "Unauthorized user",
        "status": 401
    }

While calling an endpoint with an unauthorized role say

http://localhost:8080/card-scheme/stats?start=1&limit=10

with the user1 generated token will give a response triggered by our CustomRestAccessDenied handler with HTTP 403 status code because only Admin roles can access it.

    {
        "timestamp": "2019-06-19T22:51:57.213",
        "message": "Access Denied",
        "status": 403
    }

Unit Testing

Let's now look at how we will setup writing our tests. Because we are concerned about Authentication and Authorization, we will be restricting our test scopes to not go beyond the controller layer as this is where these restrictions are placed. To achieve this I created the SecurityTest class and annotated it with the @WebMvcTest annotation. Then I mocked every dependencies that are needed at the controller layer except for JwtManager which I spied. The reason being that I will actually be calling the real methods in the JwtManager class in our test for token generation and validation. Then I created AppTestConfig configuration class and set the base package scanning at the controller layer. However I had to create an EntityManagerFactory bean that will be using an in-memory H2 database for startup and bootstrapping. In my setUp method, I then added the JwtAuthenticationFilter instance as a filter for validating the token. And finally I enabled Spring security with .apply(springSecurity()). Running the tests will give us the expected outcomes as defined in our test class.

Before we conclude on this article, let's talk about other aspects of securing our application.

Data Encryption

Data Encryption has been around for as long as the ability for humans to communicate came into being. I will not go into historical accounts of these evidences but data encryption is almost just as important as the exchange of information between individuals or parties. Data Encryption is simply the act of making information useless or cryptic to unintended recipients except for the intended recipient(s).

In today's world, Data Encryption is based on two different forms of cryptography:

Symmetric key
Asymmetric key.

The symmetric key form which uses the AES algorithm, uses a single key both for encryption and decryption and is shared by parties involved. On the other hand, asymmetric key which uses the RSA algorithm involves two keys, public and private keys. The public key can be shared with anyone, but the private key must remain a secret. Both can be used to encrypt a message, and the opposite key from the one originally used to encrypt that message is then used to decode it. Also there is the concept of signing the encrypted data which is very useful for Non-repudiation. This invloves using the private key to sign the encrypted and the public key is then used to verify the signature.

Although there was really no use case to implement any of these encryption forms in our application, you can take a look at the RSAEncryption, AESEncryption and EncryptionAndSignTest classes I created to see the available utility methods and how they are used to perform various operations. Notice how in the RSAEncryption class how you can either generate keys dynamically by calling the generateKeyPair method or load from a keystore file by calling the getKeyPairFromKeyStore method. The keystore file can be generated using the Java keytool utility. One guiding rule in determining which form to use (RSA or AES) is the size of the data to encrypt. For large amounts of data, it is advised to use the AES form as it's faster and less computationally intensive while the RSA can be used for smaller amounts of data.

Data encryption can be done both at rest and in motion.

A common example of data at motion encryption is the use of the HTTPS protocol. What happens here is that when the browser makes a request to a SSL server, it gets the public key and generates a random based key. It then uses the public key to encrypt the generated key and sends it back to the server. The server then uses the private key to decrypt and retrieve the generated key. It is this generated key that both the server and the browser use for encryption and decryption when exchanging information. The advantage of this is that only the browser session and the server knows the generated key used for exchange of information.

For non browser applications, we can simulate the above scenario as well whereby key pairs are generated (public and private keys using the encryption classes I just mentioned above). The involved applications then exchange their public keys and is/are used to encrypt and verify signatures while the private keys are used to decrypt and sign the data.

So in our case here, we can equally achieve data encryption for data in motion by leveraging our application on the HTTPS protocol. Hence our solution will be more of an architectural implementation rather than a coding one.

In the case of data encryption at rest, this usually involves protecting data within our data storage components or systems such as our databases (be it RDBMS, NOSQL or even our caching systems) and even our configuration file values. In other words, using the encryption utility classes I created earlier, we can store our data in an encrypted format and only decrypt when it is about to be used within the application. This approach will make the data in our datastore cryptic and unreadable should it fall into the wrong hands. We can also choose to encrypt our data without expecting to decrypt it back so long as we know the algorithm we used in encrypting it in the first place. This is a lot safer than encryption and it is known as Hashing.

A classic example of hashing here is the storage of users' passwords in our database (Recently there was an outcry of Facebook storing passwords of Instagram users in plain text). We achieved it here using the hashing implementation which is unidirectional. In otherwords unlike the encryption technique discussed earlier, it can't be decrypted (or impossible in practicality) after it's been hashed. The principle here is that hashing a particular sequence of string with a particular algorithm will always produce the same encrypted string (except in the case of Salting which I will discuss shortly). Here we used the BCryptPasswordEncoder encode method to achieve this as can be seen in our SecurityConfig class where it is being used as our PasswordEncoder and used when creating our users during the registration process.

Merely hashing our passwords isn't just enough to secure them. The availability of Rainbow tables has proven this to be so. The online availability of these tools have made cracking passwords a hobby for hackers especially the very simple and common ones such as popular words, sequential alphabets and numbers, and even personal information such as names, dates etc. This has led to the concept known as Salting. Salting is a technique whereby random sequence of characters are added as part of the parameters required for password hashing. It makes using rainbow tables very difficult to use as even the simple password instances mentioned above will no longer have entries in these tables as a result of salting.

In older implementations, Spring provided the MessageDigestPasswordEncoder class which allowed us to provide our own salt string for hashing our password. An efficient usage of this was to provide a different salt string for every password string to be hashed (e.g time in milliseconds at point of registration). However this has been deprecated in favour of the BCryptPasswordEncoder which we are currently using in our application. The BCryptPasswordEncoder not only hashes our password but internally generates a random salt string for us in the process as can be seen when looked at its internal implementation. This implementation unburdens us with the overhead of managing the generation of salt strings as this crucial in the integrity of our passwords. As a result of this, a different hashed string produced everytime you hash the same password string. Now the question arises how do we determine compare and match the passwords? The BCryptPasswordEncoder implementation breaks down the hashed string into the salt string, cost parameter and the hashed value all of which can be obtained from the hashed string and since the salt determines the generated hash, it can be used to determine whether the supplied passwords match or not with what we have in store.

And finally, I'd like to talk about securing our application configuration file which is usually either named as application.properties file or application.yaml as in our case. Although in our case you will notice that sensitive information like the database credentials are currently being retrieved from environment variables, I've seen cases where these details are stored as plain text within the file. Even though the use of environment variables mitigates to an extent the risk of having them as plain text in the file, another approach will be the use of Jasypt Spring boot starter module. This module allows us to store values in its encrypted form and then decrypted for use at runtime.

To implement this in our application, we first need to add the jasypt-spring-boot-starter to our pom.xml

<dependency>
    <groupId>com.github.ulisesbocchio</groupId>
    <artifactId>jasypt-spring-boot-starter</artifactId>
    <version>2.1.1</version>
</dependency>

Then we run the command below

java -cp $M2_REPO/org/jasypt/jasypt/1.9.2/jasypt-1.9.2.jar  org.jasypt.intf.cli.JasyptPBEStringEncryptionCLI input="yourpassword" password=jasypt_password algorithm=PBEWithMD5AndDES

where

$M2_REPO is your maven repoistory directory
input argument is the password u need to encrypt say for example your database password
password argument is the Jasypt password or key for encryption you will be supplying when starting up your application
algorithm is the Password Based Encryption (PBE) algorithm of your choice provided by Jasypt (PBEWITHMD5ANDDES, PBEWITHMD5ANDTRIPLEDES, PBEWITHSHA1ANDDESEDE, PBEWITHSHA1ANDRC2_40)

The above command will produce an encrypted value of the input argument password value you provided. You can repeat the command for any other values you may wish to encrypt in the application.properties or yaml file.

You then set the properties or environment variable value(s) with these encrypted value in this format ENC(encrypted_value) for example in our yaml you can change the password value like this

.
.
.
spring:
  profiles: development
  application:
    name: binList
  datasource:
     password: ENC(encrypted_password_from_jasypt_operation)
     .
     .
     .

or you simply leave the yaml file as is and set the SPRING_DATASOURCE_PASSWORD environment variable with it

export SPRING_DATASOURCE_PASSWORD=ENC(encrypted_password_from_jasypt_operation)

Then you either start your application like this

mvn -Djasypt.encryptor.password=jasypt_password spring-boot:run

or you export the JASYPT_ENCRYPTOR_PASSWORD environment variable and simply just run mvn spring-boot:run

export JASYPT_ENCRYPTOR_PASSWORD=jasypt_password

To further tighten security on our application server, you can do the following steps sequentially

Export and set all the environment variables mentioned earlier using a batch script (.sh or .bat depending on your OS)
Start up the application as a background service like this mvn spring-boot:run & or as a Windows Service
unset all the environment variables using a batch script since these values are only requested for once on application start up.
Delete the batch script(s) so long as you have a copy elsewhere.

The reason for these steps is to prevent users querying your application server using the ps and history commands to view the passwords or environment variables from previously ran commands.

So there you have it. We have secured our application although as we all know security can never be total but it should slow down and require more effort for someone to maliciously use our application. There are other security strategies I have left out here especially at Data Access Object layer such as query input parameterization for SQL injection due to lack of SQL or JQL implementations in our application. You can access the updated source code for this article here.

In my next article if time permits me I will be demonstrating how to achieve Scalability and Availability with our application using Docker and some code refactorings. Also I hope to recover the Laravel and Node(NestJs) implementations of what we have done so far from my laptop crash and share it. That's all folks! Happy Coding.

Exception Handling and Logging

charles1303 — Mon, 01 Apr 2019 19:59:37 +0000

Introduction

In my last article here, We discussed the implementations and steps that were taken to integrate our application to an external API. We highlighted the design patterns and strategies used to carry out the task. They were divided into the following sections.

Design patterns
Performance
Testing

In this article, I will be building on what we did in the last article on how to implement the following requirements:

Exception handling
Logging

Exception Handling

Exception handling can be seen as a way of providing Resilience to our application. It allows the application to continue to behave normally even when things go wrong but at the same time report what went wrong. Imagine we get a 404 HttpException when trying to access the external API due to its unavailability at some point or we get a database connection exception which will mostly likely lead to a 500 HttpException. We wouldn't want our application to become unusable or behave unxepectedly due to this scenarios. It should still function properly by giving a predfined response adhering to the expected response format or structure of the application. And this is a good practice because a system being down should not mean that every other system or feature in our deployment infrastructure should be down or inaccessible. Rather our application should still be accessible and respond normally but report that something went wrong in a presentable manner and behaviour. We can think of exception handling as a sort of prescriptive analytics in the Big Data world.

In our Spring Implementation, we create a class and annotate it with ControllerAdvice as seen in the CardDetailControllerAdvise class. Then we define methods to handle the exception scenarios. Ensure that in your controllers that you throw the appropriate anticipated exceptions as seen in the CardDetailController class. I have also introduced two new classes RecordNotFoundException and GeneralException that are used in my business use cases. I also introduced the HttpStatusCodeException class in the CardDetailControllerAdvise class to cater for HTTP level exceptions (in this case our 404 and 500). Of course you can extend or decorate the getHttpExceptionDetails method in the advice class to add other HTTP Exceptions.

On a side note, when implementing Exception handling, a clean approach is to avoid using Exceptions in your decision making. For example rather than do this:

    public int execute () {

    try{
        // Code to execute
        return 1;
     }catch(Exception ex) {
       return -1;
    }

}

either do this

    public int execute () {

        int status = -1;

        try {
            //Code to execute
            status = 1;
        }catch(FileNotFoundException ex ) {
          //Log exception details
        }
        return status;
    }

or you simply throw the exception like this so it propagates to the caller of the method who then decides how to proceed as in our case

    public int execute () throws CustomException {

        int status = -1;

        try {
            //Code to execute
            status = 1;
        }catch(FileNotFoundException ex ) {
          throw new CustomException("Could not access file resource.", e);
        }
        return status;
    }

Here we are wrapping our exception in a CustomException class that is thrown when this exception scenario occurs. We simply override the constructor of the java.lang.Exception class that takes in a message (which is the message we would like to present to the client) and a Throwable object (in this case the Exception object that is thrown).

    public class CustomException extends Exception {  
        public CustomException(String message, Exception ex) {
        super(message, ex);
        }
    }

The above implementation can be seen in the getCardRequestLogsCountGroupedByCard method of the CardDetailService class. This gives a response format consistent with the application's predefined response format and also allows you to control the exception message content using a more centralized approach with the help of the CardDetailControllerAdvise class.

Now Guess what! Our CacheIntegrationTest class is now failing because it is now throwing an Exception based on the above implementation. In this case, because our CardDetailRequestLogRepository class is actually being mocked, its getCardRequestLogsCountGroupedByCardNumber method will return null resulting in throwing the GeneralException class. So we need to create an ExpectedException rule in the CacheIntegrationTest class and test for the GeneralException that is expected to be thrown.

It is good practice to wrap our API exceptions so that one can localize the source and cause of the exception at a high level of analysis and investigation.

Also due to the inherent increased I/O operation when writing to logs and memory usage which occurs as a result of throwing exceptions, we should either log the exception or throw it but not both as shown earlier. Doing this can lead to what is commonly referred to as the Hot Potato anti pattern design. This creates a scenario whereby a lot of CPU intensive work is done with relatively little valuable output leading to application spikes in CPU usage and memory consumption. Throwing and logging an exception will also result in a bunch of log messages which aren't really needed. Hence amount of text will reduce the visibility of the logs.

Apart from the above approach of handling exceptions, people do employ the strategy of caching data responses from an external API into a data store. These data responses are then accessed from the cached system when the external API is unavailable or something went wrong. This strategy is a design pattern known as the Circuit Breaker. The fault tolerance library Hystrix is a common implementation of this pattern as I will show you in a bit.

First of all we include the spring-cloud-starter-netflix-hystrix dependency in our project.

    <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-starter-netflix-hystrix</artifactId>
        <version>2.1.1.RELEASE</version>
    </dependency>

Then we create a method verifyCardDetailHystrix in the CardDetailService class and annotate it with the HystrixCommand and setting the necessary HystrixProperty values. Then we create a fallback method defaultCardDetailDto that will be called when the API is unreacheable after a period so as to return our cached or default data. Finally we enable the circuit breaker by annotating our Application class with EnableCircuitBreaker. And that's it. The defaultCardDetailDto method would be called whenever the API is unavailable. Apart from the ones I used, there are a number of other HystrixProperties you can tweak to suit your application specification.

Logging

We have talked about how to keep track of application activities, present default or cached data and notify users of the application state when something goes wrong. What if we also want to keep track of application states when all goes well without any exceptions (i.e the Happy Path)? The advantage of this that we will then have a holistic view of all the activities that have taken place in the application. It can also help us to replay all the activities of an application between any given points in time during the lifecycle of the application.

Even though we introduced the concept of logging in our pseudo codes earlier, we will talk in depth as to ways and strategies of implementing application logging for auditing.

Logging is a means of auditing or keeping track of every activity that has taken place in the application or system. This is crucial as this can tell us the activities that took place at any particular point in time during the lifecycle of the application. There are quite a number of approaches to implementing this but I will highlight on two common approaches.

In the first approach, we can sprinkle our codes with logging information using standard libraries such as Log4j or Sl4j implementations. Here we add these libraries as dependencies in our projects and then call the methods that prints out whatever we want printed. To avoid unnecessary I/O operations, we can call these methods at the beginning and at the end of a method body.

    public class CardDetailService {
        Logger logger = LoggerFactory.getLogger(CardDetailService.class);

        public int execute () throws CustomException {

           logger.debug("execute method started");

            int status = -1;

            try {
                //Code to execute
                status = 1;
            }catch(FileNotFoundException ex ) {
              throw new CustomException("Could not access file resource.", e);
            }
              logger.debug("execute method ended");
            return status;
        }

    }

(For those in the PHP Laravel world, there is a shipped in Log facade that has the same signature call as in above i.e Log::debug('An informational message.');)

This approach tells us the activities/operations that took place in the application at a given point time. The downside to this is that it doesn't tell us the states of objects being acted upon during that period. Note however that with the above implementation, if an exception is thrown, it will be logged as well.

We can also control the logging level to reduce the amount of logging information hence I/O operations. This can be done either using the application.yml and logback-spring.xml files in our project or at the container/application server level. Notice how I set the logging levels in the application.yml file based on java packages to ERROR level based on our earlier recommendation. We also set the output pattern of every log event that will be written to our log files such that it prints the log time and level, class where the logs is being triggered and the message to print. In the logback-spring.xml file, we set the RollingFileAppender properties to control the name and size of each log file that will be generated, the maximum number and size of total log files to be kept on the server.

The other approach of logging uses a very common design pattern known as Event Sourcing. Here we store all the events/activities and the corresponding changes made of objects or models in our application. Then, to retrieve an object's or model's state we then read the different events/activities related to it and apply them one by one. It ensures that all changes to application state are stored as a sequence of events. This is quite similar to JPA Hibernate Envers implementation (where we annotate the JPA entities with @Audited leading to mirror images of model states being created in the database) except that in Event Sourcing, these changes are tied to the actual end-user activity that led to these model state changes.

In the Laravel world, this is implemented by adding the either the EventSauce or prooph dependency. You create your models, then define the Events, create the Projector and handle side effects such as notifications using Reactors. For sake of scope of this article and the framework we are using, I will not go into details on this but the concept is the same and this pattern can be applied on other technology stacks as well.

In our case here, using the Spring framework, I will show a high level and basic implementation.

First we bring in the dependency like below into our pom.xml file :

        <dependency>
            <groupId>org.axonframework</groupId>
            <artifactId>axon-spring-boot-starter</artifactId>
            <version>4.1</version>
        </dependency>
        <dependency>
            <groupId>org.axonframework</groupId>
            <artifactId>axon-test</artifactId>
            <version>4.0.3</version>
            <scope>test</scope>
        </dependency>

Then download the axon server, extract to a folder of your choice. This is what we will be using as our EventStore.

We go ahead and create the Aggregate (CardDetailRequestAggregate), Event (CardDetailRequestCreatedEvent) and Command (CreateCardDetailRequestCommand) classes. Then we create the command and event handler methods in the CardDetailRequestAggregate for the CardDetailRequestCreatedEvent and CreateCardDetailRequestCommand classes. In very simple terms, the command class and the method annotated with CommandHandler are responsible for determining which action or event was triggered based on a user input via the CommandGateway interface. It's similar to the Command design pattern in implementation. While the Event class and the corresponding annotated EventSourcingHandler methods are responsible via the EventStore interface for the side effect that occurred as a result of the event or action that was triggered. The event components are also responsible for retrieval from and persistence of the Aggregate models also via the EventStore interface. The usage of the CommandGateway interface and the EventStore interface can be seen in the CardDetailCommandComponent class. The method logCardDetailRequest which was our existing function in the CardDetailService class earlier in our last article is overloaded to accept the CardDetailRequestCreatedEvent object and annotated with the EventHandler. The EventHandler annotation will trigger the logCardDetailRequest method when we send our command object via the CommandGateway as can be seen in the createCardDetalRequestLog method of the CardDetailCommandComponent class.

We can now start up our axon server using java -jar axonserver.jar command and the spring boot application. When we hit the endpoint, our function will be called as expected. If you go to the Axon Server dashboard, you will see an entry having the CardDetailRequestCreatedEvent event that was triggered with the value of the cardNumber and date you made a request with. The beauty of this approach is that we no longer burden the verifyCardDetailHystrix method in the CardDetailService class with the responsibility of logging as in the previous implementation whereby we called the logCardDetailRequest method. Hence we have in a way applied the Single Responsibility principle to that method which is one of the tenets of clean coding (S.O.L.I.D).

Because logging is more or less a non-functional requirement, one crucial factor to consider here is to ensure that the logging process does not impact on the application performance hence user experience. For this reason, it is advisable to make the logging process asynchronous as can be seen the reason why I am using the CompletableFuture in the service class for the return object which of course is the default implementation of the Axon framework. Another point of consideration if you have the resources, is to point your Event Sourcing implementation to a different datastore from the primary application database as its event store. In our case, this is being handled by the Axon Server we are running locally. The Axon implementation can also be configured to use other data stores such as RDBMS or NOSql databases as its event store. This way you can have a sort of backed up data to replay your application lifecycle and restore previously generated data should something go wrong with the primary application database.

So there you have it, our application now has resilience, tolerance as well as best practices for exception handling and logging which are requirements for a well architected system. And not to worry, I have updated our repo here to include all the implementations discussed in this article. In my next article, I will be highlighting on Security, a very crucial requirement for building applications and how to achieve Scalability and Availability with our application using Docker as the virtualization technology. Happy Coding!

A Use Case Implementation for External API Integration

charles1303 — Mon, 18 Feb 2019 17:14:18 +0000

Introduction

Phew finally! This will be my first post on this community and I hope to get better using them markdowns as I post more articles here. So here it goes...

I did a simple task of integrating with the Bin List API and returning an expected client response different from the response gotten from the Bin List API. I will like to point out some of the basics in designing and building a system for such a task.
The BinList API is a public API that is used to fetch payment card details. It can be accessed via here Bin List API. It exposes an endpoint where you provide the first eight (8) digits of the card number and it returns the card details. For example below is a request and corresponding response made to the API:

  GET https://lookup.binlist.net/45717260
    {
        "number": {
        "length": 16,
        "luhn": true
        },
        "scheme": "visa",
        "type": "debit",
        "brand": "Visa/Dankort",
        "prepaid": false,
        "country": {
        "numeric": "208",
        "alpha2": "DK",
        "name": "Denmark",
        "emoji": "🇩🇰",
        "currency": "DKK",
        "latitude": 56,
        "longitude": 10
        },
        "bank": {
        "name": "Djurslands Bank",
        "url": "www.djurslandsbank.dk"
        }
    }

I won't be doing much of code screen shots but you can access every class and method names and signatures highlighted boldly in my github repo.

I will place emphasis on the following areas for sake of time and space.

Design pattern implementations (DTO, Service Facade, Repository, Cacheless Cow)
Performance(Caching)
Testing(Unit and Integration)

Tools/Platform used

Spring Boot 2.1.2 RELEASE
Maven 3.5.3
Java 8
Mysql 5.7.25
Ubuntu 16.04

Design Pattern Implementation

Because I was doing an integration which required data transfer and operations, these design patterns stood out immediately as to the strategies to be employed

Data Transfer Object (DTO) pattern
Service Facade pattern
Repository pattern
Cacheless Cow anti pattern

The Data Transfer Object(DTO) pattern is used when you want to transfer data across services or domains in a well structured format. It also allows for data structure transformation between domains. It's expected that the DTOs should be serializable. So while the BinListResponse class encapsulated the data retrieved from the BinList API (whose url I have configured in the application.yml), the CardDetailDto class contained the transformed data format that will be understood by the client. The transformation is done in the convertToCardDetailDto(BinListResponse binListResponse) method of the CardDetailService class. An advantage of this approach is that either the BinList API response or the client's expectation can be changed internally without affecting either. Also this will prevent breakage of the BinList API interface should the data format decide to change. This approach also makes use of Encapsulation in Object Oriented Programming (OOP) paradigm. All DTOs employed in the application can be located in the com.projects.binlist.dto.responses package.

The Service Facade or simply Facade is a strategy allows you to encapsulate and hide the complexities of carrying out an operation thereby exposing a simple interface to the client or caller of the method. This interface with this characteristics is sometimes referred to as being coarse. For example the getCardRequestLogsCountGroupedByCard(Pageable pageable) in the CardDetailService class does a bunch of things before returning the data to the client, i.e check the cache store, call the repository and transform the data retrieved to suit the client's expectation. The advantage of this is that it exposes a neat and simple interface and encapsulates related and seemingly linked operations that depend on one another as a single unit of operation thereby maintaining the integrity of the system's operation and output to the client. The client also has an advantage in that these operations are clearly transparent and they do not need to know the implementation of retrieving the data. (I like to keep my controllers thin and simple).

The Repository pattern as we all know allows for easy interchange of the Data Access Layer implementation. It's usually made up of interfaces implemented by the Data Access Object (DAO) (DAO which is another design pattern) classes. Spring provides different Repository interfaces which can be extended and implemented depending on our data storage engine (Mysql, MongoDB etc). As you can see, my repository domain is just a bunch of generics interfaces with the entity it's acting on as the type parameter as seen in the CardDetailRequestLogRepository class. The Repository pattern allows for loose coupling and easy testing as we will see in a bit. Also I have the advantage of writing customized queries and I can easily change them depending on either the data storage engine or the ORM implementation being used.

I will be talking about the Cacheless Cow anti pattern in the next section.

Performance(Caching)

Remember when I said in the last section concerning the Service Facade about how it exposes a simple interface to call by encapsulating a bunch of operations as a single unit of operation to produce the expected data output? Well this comes at a cost in terms of time it will take to produce that output for every call. However we can eliminate this cost by storing the data output and retrieving it whenever we need that output without going through the steps of reprocessing it provided we have some sort of identifier or key that we can do a lookup for that maps to that particular data we want to retrieve. This is similar to Dynamic Programming when we implement Recursion Algorithm by storing a previously computed value with a key in a data structure (Array or Map) and retrieving it without having to make a recursive call for that value so as to reduce time complexities.

In our application, we used a cache implementation to store that value so we don't have to make expensive database calls and data transformations for every service call. Caching in this context simply means

Get me that data as is if you have it without reprocessing or recomputing it.

This is how we handle what we normally term the Cacheless Cow anti pattern I referred to earlier.

To implement this in our code, we configure a Spring Cache manager bean in one of our configuration classes, the CacheConfig class using the ConcurrentMapCacheManager implementation (we could have also easily used the EhCache implementation as well). Then we annotate the service method getCardRequestLogsCountGroupedByCard(Pageable pageable) in our CardDetailService class with @Cacheable providing the necessary values (the value represents the cache name in our configuration file while the key is the key naming strategy we are using). Here in the key naming strategy, we concatenate the method name with the pageNumber attribute value of the Pageable parameter we are sending to generate a key and store the returned data with that key. So the first request will go through the processing steps to retrieve the data and then store it in the cache using the generated key. This took about 276ms for a 20 record table. Subsequent requests will ignore the processing steps and just fetch the stored data from the cache taking about 5-10ms (Wow!). The reprocessing of the data will occur only when you send a different pageNumber attribute value of the Pageable parameter that was not previously sent. A warning here however is to find a way to clear the data from the cache if the data from the backend has changed to prevent stale data being returned. Hence when the logs are updated in the database in the logCardDetailRequest(String iinStart) method, the @CacheEvict annotation on it clears the cache so it will force the method call to getCardRequestLogsCountGroupedByCard(Pageable pageable) to actually fetch the data from the database as it will no longer be in the cache. I will show you in a bit how we can verify this in our test class.

Testing

Writing tests can never be over emphasized during application development. Writing tests against your functionalities guarantees integrity and raises red flags if any new code change breaks your application. Before I continue, I would like to ask a rhetorical question here:

Why do you write tests or why are you writing that test?

For those of us who thought

I want to verify that my functionalities or endpoints works properly.

, chances are that you may be inclined to write your tests around what we refer to as Integration (or Functional) tests. For others who thought

I want to verify that my method works properly.

,chances are that you will end up writing Unit tests. There is nothing wrong in having both tests in your test suite but you need to consider the following when you decide which one to implement or even implement both.

Integration tests are normally environment dependent such that one that was written and passed successfully on your local machine might fail on another machine due to different data states in the different database instances or unavailability of a resource dependency due to network failure. Such a scenario can be seen when you try to run the CardDetailControllerITTest or the CardDetailControllerTest classes. These test classes expect the following in any environment it's running to be true

That the Bin List API is accessible
That the database is available and accessible
That the datasets in the databases of any of the environment must be the same.

Despite these constraints, a clear advantage of integration test is that you get to actually see how your dependencies behave in a real deployed scenario in the production environment. For example, the CacheIntegrationTest test class asserts that the responses gotten in both service calls are the same (i.e the cached response created in the first call is returned in the second call). It also verifies that the repository layer was only accessed once after both calls indicating that the second call didn't go to the database but retrieved the object from cache.

On the other hand, Unit tests are independent of the data states of the environment and will yield the same results regardless of the environment it's being run in. The reason for this is that the dependencies, resources, outputs and data are mocked, hence are not required to be available or accessible. Only pure logic and component method behaviours are tested here. For example, in the CardDetailServiceTest class, the RestTemplate dependency which is used to access the Bin List API is mocked along with the response. (This is like using the Guzzle MockHandler in the PHP or nock in the Node worlds). Also you will observe that logCardDetailRequest(String iinStart) method behaviour of the CardDetailService class is equally mocked by instructing it to do nothing as this interacts with the database (Thanks to the @Spy annotation). In the end, all we are testing is our DTO pattern implementation (i.e convertToCardDetailDto(BinListResponse binListResponse) method), to ensure the mocked BinListResponse object state that will be gotten from the Bin List API is transformed to the appropriate mocked CardDetailDto object state and returned to the client or method caller.

Can you guess the downside of this approach of testing? Yeah that's right, it doesn't simulate the true behaviours of your dependencies when they are deployed in production environment. Choosing which tests(unit or integration) to implement in your code is a function of the coding review requirements, use case or business requirements and the strategy employed in the deployment pipeline.

I have left out other features such as Security, Exception handling and Logging in this task as they were not necessarily a fundamental requirement. I hope I can get to share my insights on this as well sometime soon. This approach outlined in executing this task is by no means cast in stone as I welcome comments and other better ways or approaches in implementing such a task. Please feel free to pull the project here and review. Afterall this is a community, Ain't That Right or should I say Ain't It? :) . Happy coding!