DEV Community: Davey

Configuring Cache Control for S3 Hosted Resources

Davey — Sat, 01 Jan 2022 21:57:43 +0000

When doing web development, one of the most important aspects is to get a site performing well with SEO (Search Engine Optimisation). One contributing factor to this (there are many), is effective resource caching. In this article, I'm going to show how to achieve this when resources are stored within a S3 bucket.

Within Chrome, we can generate a Lighthouse report which will identify any images that are not cached correctly.

In this image, we can see that the default "cache policy" applied to resources in a S3 bucket is to tell the browser to not cache the resource.

To cache an resource, we need to send a Cache-Control HTTP header with a suitable timeout to cache. The timeout is measured in seconds and is identified as the max-age of the resource.

So for example, to tell the browser to cache resources for 1 day, we would use a max age of 60s * 60m * 24h = 86400. To tell the browser to cache for 1 week, we would use a max-age of 60s * 60m * 24h * 7d = 604800.

So, how do we configure S3 to specify a cache time ?

Within the S3 Console, we can select either an individual resource, or an entire directory and specify this value. To do this, we select to edit the Metadata of the selected objects.

On the resulting screen, we select the Add Metadata section and add a System Defined piece of metadata with a key of Cache-Control and a value of max-age=<number of seconds>

In this image, you can see that I've set the max-age to 7 days (60s * 60m * 24h * 7d).

That's all there is to it! Save the changes to the metadata and S3 will start hosting your resources with an effective cache control.

Credits

Photo by Marc-Olivier Jodoin on Unsplash.

Changing Default Search Provider in Firefox on Linux Mint

Davey — Thu, 09 Dec 2021 22:53:42 +0000

On Linux Mint, the default version of Firefox is installed and configured to allow the following search engines to be queried directly from the address bar:

Yahoo!
Startpage
DuckDuckGo
Wikipedia

Mint defines these as the default available set of search engines based upon 3 criteria (funding to Linux Mint, privacy support and whether the search engine is non-commercial).

Other search engines such as Google, Bing or Twitter, etc. can easily be added into the default version of Firefox however.

To add a different search provider, browse to Search Engines

At the bottom of the page, click on the icon of the requested search engine, then click on the ... button in the URL bar and select the Add Search Engine option.

You then have the option to change the default search engine within Firefox preferences to your new choice.

An Application Structure for Python REST APIs

Davey — Tue, 05 Oct 2021 19:28:35 +0000

Introduction

In this article, I'm going to show the structure that I use for REST based APIs using Python. In Python, I use Flask to create APIs as I find this is easy to use, yet very powerful. This article isn't a tutorial on using Flask, rather I'm showcasing the project structure I currently use that allows me to develop and test Flask applications easily.

If you've never used Flask before, its a lightweight framework that allows Python developers to create both Web Applications and Web APIs. It includes all the tools necessary and allows you to write easily testable applications. Flask can easily be installed into an application using pip

pip install flask

Top level structure

The basic structure I like to use is shown below.

.
├── application
│   ├── api
│   │   ├── <api definition>
│   │   └── <api definition>
│   ├── factory.py
│   ├── <logic classes>
├── app.py
├── Makefile
├── requirements.txt
└── tests
    ├── api
    │   ├── <api test>
    │   └── <api test>
    ├── context.py
    └── test_<logic classes>.py

Within this structure, there are two top level folders, application and tests. These are used for storing the application code and the test code respectively. I like to keep the test code separate from the application code and use this structure rather than placing the tests in the same folder as the application code.

Within the application and tests folders, there are individual files for each api I want to develop. Each of these corresponds to a different url.

Main Python Classes

Within this directory structure, there are 3 Python files that are standard across each application:

app.py
application/factory.py
tests/context.py

Lets take a look at each of these in turn.

app.py

This is the main file to bootstrap the application and invokes the create_app() method in the application/factory.py class:

from application.factory import create_app

app = create_app()

if __name__ == '__main__':
    app.run()

application/factory.py

from flask import Flask
from application.api.moon import moon
from application.api.sun import sun


def create_app():
    app = Flask(__name__)
    app.register_blueprint(api1, url_prefix=<api1>.url_prefix)
    app.register_blueprint(api2, url_prefix=<api2>.url_prefix)

    return app

This file creates a Flask application and registers the Blueprints for the different API endpoints.

Flask describes a blueprint as

a set of operations which can be registered on an application, even multiple times.

Essentially in this example, this means that we can register a set of APIs within the application with each API being stored in a separate file. This allows us to easily create larger applications whilst maintaining smaller testable components.

tests/context.py

This file allows the tests to be able to import the application logic when the tests are stored in a different folder to the applicaton source code.

import application
import os
import sys


sys.path.insert(0, os.path.abspath(
    os.path.join(os.path.dirname(__file__), '..')))

An example API

Using the ephem library and the basic API structure above, lets see how we can write an API to return the phase of the moon on a given date.

Defining Endpoints

To obtain the phase of the moon, I'm going to create three endpoints:

/moon/
/moon/phase
/moon/phase/yyyy/mm/dd

To create the API, create a file called application/api/moon.py. To initialise the API with Flask, we need to register the Blueprint:

from flask import Blueprint, Response
import json
import ephem
from ..moonphase import MoonPhase
import datetime
from datetime import date

moon = Blueprint('moon', __name__)
moon.url_prefix = '/moon'

This creates a Blueprint called moon at the endpoint of /moon

To create the first of our endpoints (/moon/), we need to define a method with a route of / (as the API itself has the url_prefix of /moon).

@moon.route('/')
def index():
    return Response(json.dumps("Moon API"), mimetype=APPLICATION_JSON)

To create our second endpoint (/moon/phase), we're going to return the Moon phase for the current date. This can be achieved with the following code:

@moon.route('/phase')
def phase_today():
    m = ephem.Moon()
    today = date.today()
    m.compute(today.strftime('%Y/%m/%d'))
    return Response(json.dumps(MoonPhase(round(m.moon_phase * 100, 1), today.year, today.month, today.day).__dict__),
                    mimetype=APPLICATION_JSON)

In this code, the method is decorated with the route /phase making the url end with /moon/phase. This method uses the json.dumps method to return a JSON representation of an instance of a MoonPhase class as the response.

MoonPhase is a @dataclass defined within the application/moonphase.py file

from dataclasses import dataclass

@dataclass
class MoonPhase:
    '''Data object to be passed out from the api to the caller'''
    phase: float
    year: int
    month: int
    day: int

Finally, getting back to moon.py, the final API endpoint (/moon/phase/yyyy/mm/dd) can be defined as:

@moon.route('/phase/<year>/<month>/<day>')
def phase_specific_day(year, month, day):
    m = ephem.Moon()
    m.compute(f'{year}/{month}/{day}')
    return Response(json.dumps(MoonPhase(round(m.moon_phase * 100, 1), year, month, day).__dict__),
                    mimetype=APPLICATION_JSON)

This method is almost identical to the previous, except that the year, month and day are passed in as parameters into the REST API.

Testing the API

To test the API, we can use PyTest and write a test within the tests/api folder, for example:

from application.factory import create_app
import pytest
from ..context import application


@pytest.fixture
def client():
    return create_app().test_client()


def test_home_route(client):
    response = client.get('/moon/')
    assert response.get_json() == 'Moon API', 'Route should return "Moon API"'


def test_moon_phase_today(client):
    response = client.get('/moon/phase')
    assert response.status_code == 200, 'Invalid status code'
    assert float(response.get_json()['phase']) > 0.0, 'Invalid phase'
    assert int(response.get_json()['year']) > 0, 'Invalid year'
    assert int(response.get_json()['month']) > 0, 'Invalid month'
    assert int(response.get_json()['day']) > 0, 'Invalid day'

Conclusion

In this article, I've shown a standard template that can be used to develop and test Flask applications. I've then shown an example of adding an API using this template.

The code is stored in a modular fashion so the application can easily be enhanced with more APIs. Also, test application is structured so that tests can easily be written for each of the APIs and other classes within the application.

The complete source code associated with this article is available on GitHub

Credits

Photo by Raghav Bhasin on Unsplash

Connecting to an AWS MySQL RDS using a Bastion Host

Davey — Wed, 08 Sep 2021 22:02:22 +0000

Introduction

In my previous article, I showed how to connect to an Amazon RDS host by changing the security group and allowing direct access to port 3306. This works but does have security implications in that potentially the entire internet can start pinging your RDS server.

In this article, I’m going to show how to use a Bastion host which does not need us to expose the database access port (port 3306 in the case of MySQL) to the internet.

What is a Bastion Host ?

Wikipedia defines a Bastion Host as:

a special-purpose computer on a network specifically designed and configured to withstand attacks. The computer generally hosts a single application or process

What this means, in terms of connecting to a MySQL RDS instance, is that MySQL is not installed on the Bastion server, rather we use the Bastion server as a “jump” point to allow us to get to the real database server.

I'm not going to go into details on how to harden a bastion server, I'm concentrating on how to access MySQL through the server. Check out this blog post for more details on how to controll network access using a bastion server.

When using a Bastion server, the network topology looks something like:

In terms of AWS, to create a Bastion Server, we need to create a EC2 instance (in my case, I created a t2.micro instance). We need to ensure that the EC2 instance is in the same VPC as the RDS database otherwise they will not be able to connect to each other. This is a key point. If the bastion server and the RDS database are not in the same VPC, then this technique will not work.

For the security groups, the Bastion server should be configured to only allow ssh traffic from a known host(s) which is your local network. Note that it is not necessary to allow network traffic into the bastion host on the database port (3306). The only access required from the global internet is via port 22 (ssh port).

The security group for the RDS instance should be configured to allow traffic from within the VPC which enables access from the Bastion host.

SSH Tunneling

Once we've set up the networking and got an EC2 instance that can be connected via ssh from a local machine, we need to set up a ssh tunnel from our local machine through to the RDS database via the bastion host. This is achieved with the ssh command. If you're a Linux/Mac user, this will most likely already be installed. For Windows users, it is available from within Powershell.

An ssh tunnel is basically a way of routing network traffic from one place to another via a ssh host. In our case, we are routing traffic from our local machine to a RDS instance via a Bastion host. For more information about ssh tunnelling, check out this article.

To create a ssh tunnel, we execute the ssh command as follows:

ssh -N -L <local port>:<database host name>:<remote port> \
  <bastion-user>@<bastion-host> -i <bastion-host-key-file>

<local port> is the source port on the local machine that we can use for connecting to from local applications. It makes sense to set this the same as the remote port if possible, but this can be set to any free port. For connection to MySQL, I set this to 3306

<database host name> is the host name of the RDS instance. This can be found in the RDS console but will be along the lines of xxx.xxx.us-east-1.rds.amazonaws.com

<remote port> is the port number we want to connect to on the remote RDS instance. For MySQL, this is usually 3306

<bastion-user> is the username that can be used to connect to the bastion host. For a standard EC2 instance, this would usually be ec2-user

<bastion-host> is the host name or ip address of the bastion server.

<bastion-host-key-file> is the full path to the .pem key file created when the EC2 instance was created.

-N tells ssh not to execute a remote command, essentially telling it to create a tunnel only and no interactive terminal connection.

The full ssh command would therfore look something like:

ssh -N -L 3306:xxx.xxx.us-east-1.rds.amazonrds.com:3306 \
  ec2-user@xxx.xxx.xxx.xxx -i ~/bastion_key.pem

Connection to MySQL

After creating a tunnel, we can connect to MySQL as though the database were running locally. The host name will be 127.0.0.1 and the port will be 3306 (127.0.0.1 is implicitly implied by the ssh command above and 3306 is the local port we defined above).

Note that we have to use 127.0.0.1 instead of localhost when connecting via the ssh tunnel, or the MySQL client will believe we are attempting to connect to a local instance via shared memory or via a Unix socket and will not connect. For more details check out the MySQL Documentation.

~ ❯ mysqlsh  -u admin -h 127.0.0.1
Please provide the password for 'admin@127.0.0.1': **********
Save password for 'admin@127.0.0.1'? [Y]es/[N]o/Ne[v]er (default No): v
MySQL Shell 8.0.26

Copyright (c) 2016, 2021, Oracle and/or its affiliates.
Oracle is a registered trademark of Oracle Corporation and/or its affiliates.
Other names may be trademarks of their respective owners.

Type '\help' or '\?' for help; '\quit' to exit.
Creating a session to 'admin@127.0.0.1'
Fetching schema names for autocompletion... Press ^C to stop.
Your MySQL connection id is 22
Server version: 8.0.23 Source distribution
No default schema selected; type \use <schema> to set one.
 MySQL  127.0.0.1:3306 ssl  JS >

Conclusion

In this article, we've seen how we can use the ssh command to create a tunnel between our local machine and a RDS server via a bastion host. This allows us to connect to a RDS instance without having to expose the RDS instance directly to the internet.

Credits

Photo by Andrew Neel on Unsplash

Accessing a RDS Database Locally

Davey — Fri, 27 Aug 2021 21:55:59 +0000

Elastic Beanstalk (EB) is a Platform As A Service from AWS that allows you to easily deploy applications without having to worry about setting up the base infrastructure, such as HTTP servers, or load balancers. One of the benefits of EB is that is allows us to create a database on RDS when creating a new application, making the entire deployment of an application much easier.

RDS however does not provide a user interface to execute SQL against a created database (unless its been created as an Aurora Serverless Database), so how can we execute SQL commands against the database? If we were deploying our application to an EC2 instance, we could simply log onto the instance and connect directly to the database using a tool such as mysql. We can’t do this with EB, however, but we can connect to the database from our local PC.

When creating the database from EB, we get the option of specifying whether we want the database to be publicly accessible. This is the first step to being able to connect to an RDS database. So, let’s try and connect and see what happens.

Browsing to the RDS console, we can see the connection details for all our RDS databases, for example:

On this screen, we can see the endpoint, so let’s try connecting from the MySQL Workbench client. (Don’t try connecting to my database – I’ve deleted it for security purposes so the endpoint in this article doesn’t exist anymore).

If we enter the endpoint and username and password in MySQL Workbench, we’ll see an error indicating that a connection could not be made to the database.

So why do we get this error? Well, when we created the EB application, AWS created a VPC and added some security groups that explicitly deny access from outside of the VPC.

From the RDS Console, we can see the Security Group Rules and can see that we have both an Inbound and Outbound set of rules.

The outbound rule is set to allow all traffic to any address, however the inbound rule only allows inbound traffic from within the VPC.

We can change this by editing the Inbound Rules and adding a new rule, specifying the address we will allow inbound data to originate from.

This source address can be any custom IP address, however, the option My IP is available to explicitly only allow access from your current IP address.

Note, if you have a dynamic IP address, next time you connect to the internet your IP address may change and you won’t be allowed to access, but the user with your previous IP address will. Please be aware of this!

Once we’ve added a new rule, we can connect from MySQL workbench on our local PC and execute the required SQL against the database.

Conclusion

In this post, I’ve shown how you can connect to a RDS instance from a local application (MySQL Workbench in this example).

Please be careful when editing Security Groups in AWS, as you can lock yourself out of your system, or can easily allow access for potentially malicious users. For more information, please checkout the documentation at AWS.

Credits

Photo by benjamin lehman on Unsplash.

Creating a VM on Google Cloud Platform

Davey — Thu, 15 Jul 2021 20:18:32 +0000

Google Cloud Platform is a set of services that Google offer to customers providing different type of computing functionality, for example, Virtual Machines, Databases, Message Queues etc. These services are running on the same global infrastructure that Google uses for its products. In this article, I’ll show how to create and connect to a Virtual Machine running on the Google Cloud.

To complete the actions in this article, you’ll need a Google Cloud Platform account. These can be created at https://cloud.google.com. There is a charge for using the Google Cloud Platform, but when you sign up you get $300 off free credit. There are also several services that are always free as long as you are within the free usage tier. Information about GCP pricing can be found at https://cloud.google.com/pricing/

Creating a VM

The easiest way of creating a VM within GCP is from within the Web Console. Navigate to https://console.cloud.google.com/ to display the web console. Along the top of the page, you see the menu and the name of the project you have selected. When signing up, GCP creates an initial project for you. If you do not have one created, navigate to https://console.cloud.google.com/projectcreate to create a new project and then return to the console home page.

The screenshot above shows the GCP console together with the My First Project selected.

From within the console, press the menu button at the top left. A menu is displayed down the left hand side of the page showing all of the different options that can be managed from within the console.

For this article, we’re interested in creating a new Virtual Machine. VM’s are part of the GCP Compute Engine, so select the option Compute Engine from the menu.

On the Compute Engine page, a list of VM’s is displayed that you have previously created. If this is the first VM you are creating, you will see the welcome page and be presented with the option of creating a new VM. Simply press the Create button to continue.

In the following page, we can enter details of the VM we want to create, e.g. we can name it, select the region to create it in, select the amount of RAM and disk space it has as well as all the other options that go to define a VM. For this article, since we’re just creating a basic VM, we’ll go through the common options that are needed to create a VM.

Name

This is the name of both the VM when it is running, and the name that is displayed in the console where you manage your instances. The name must be in lowercase but can include numbers or hyphens.

Region / Zone

This defines the geographic location of where the VM is created. This defaults to us-east1 which is in South Carolina. For the purposes of this article, we will leave the default region as us-east1 and the default Zone as us-east1-b

Machine Type

This option allows us to configure the number of cores, memory and GPUs that the VM will possess. Note that GPUs are not presently available in all regions, so if you select a region other than us-east1, the options may be different from those shown here.

Selecting the CPU drop down, allows the number of CPUs used by the VM to be defined. Obviously, the more CPUs you select, the higher the cost of the VM per month. The estimated cost of the VM is always displayed on the right hand side of the page whilst customising the VM.

For this article, select the Machine Type as Micro. This defaults the VM to 1 shared CPU with 0.6Gb RAM and no GPU. This is the smallest of the VMs that can be created, and this VM type can be run in the free-tier. For more information about the Free Tier, and what you can run in it, see https://cloud.google.com/free/docs/gcp-free-tier?hl=en_GB&_ga=2.180966501.-319154563.1548685428#always-free

Boot Disk

Next, we get to define the Boot disk and operating system for the VM.

Within this section, we can select the OS, the size of the boot disk and a number of different image types. The default option is to use Debian Linux 9 (stretch) with a 10 GB boot disk. This is fine for demonstration purposes, but can be changed to whatever is best for your use case. A variety of different Linux and Windows operating systems can be selected from here if required.

Identity and api access

The Identity and api access section allows us to define what account, and therefore what security, the applications running on the VM will use. For this article, we aren’t running any applications on the VM, so we can leave the default settings.

Firewall

The next section allows us to define whether HTTP and HTTPS traffic are allowed to the instance. Since we’re not deploying a web server, we can leave these settings unchecked.

ssh access

Once our VM is running, we’ll want to connect to it via SSH so that we can see that everything is there as expected. To be able to do this, we need to add our SSH public key onto the Virtual Machine. Your public key is usually stored within the ~/.ssh/id_rsa.pub file. If you don’t have a public / private key, now would be a good time to create one :)

Take a copy of your public key and paste it into the keydata box on the Security tab. If you’re on a Mac, you can copy your public key into the clipboard with

cat ~/.ssh/id_rsa.pub | pbcopy

Starting the VM

That’s all that’s needed to get a basic VM up and running. Of course, there are many other options available which I’ll go through in another article. For the moment though, let’s start the VM by clicking the Create button.

Upon pressing this button, GCP will start to provision the VM. This will take up to a minute, however when it’s complete, you will see a list of VM’s that you’ve created along with tools to manage the VMs.

Connecting to the VM

In the list of VMs, you can see the name (webserver1 in this example) along with its internal and external ip addresses. To connect to the VM, we need to ssh using the username we defined previously and the public IP address.

From the terminal:

david$ ssh david@104.196.139.230
The authenticity of host '104.196.139.230 (104.196.139.230)' can't be established.
ECDSA key fingerprint is SHA256:cxuubyAwHqGG6SDyGdHHlVtPLnQsYJNpT57E2mok4Dg.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '104.196.139.230' (ECDSA) to the list of known hosts.
Linux webserver1 4.9.0-8-amd64 #1 SMP Debian 4.9.130-2 (2018-10-27) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
david@webserver1:~$

Destroying the VM

That’s all there is to it. We’ve created a Debian VM, and connected to it. We can now use the VM for whatever we need.

Finally, to delete the VM (and ensure there are no recurring costs), select the VM in the list of instances and press the Trash Can Icon. Remember, VMs are charged by usage, so if you’re not using it, delete it to stop recurring costs.

Credits

Photo by Jordan Harrison on Unsplash

Creating MicroProfile Applications From The Command Line

Davey — Tue, 13 Jul 2021 21:16:07 +0000

Introduction

The MicroProfile initiative provides the excellent web based Starter application to allow developers to easily scaffold MicroProfile based applications.

MicroProfile Starter REST Interface

Recently, a new version of the Starter has been published that exposes a REST interface allowing MicroProfile applications to be easily created from the command line. In this post, we’ll show how this can be used.

Supported Servers

To create a project with the default settings, a HTTP GET request must be made to the following url: https://start.microprofile.io/api/1/project?supportedServer=<SERVER>

This url takes as a minimum one parameter, <SERVER>, which specifies which application server (that supports MicroProfile) to use.

So, how do we get a valid <SERVER>. Well, not all application servers provide the same level of support for different versions on MicroProfile, so first, we must decide which version of MicroProfile we wish to target. Once we know that, we can select an appropriate application server and then create a project.

The first stage therefore, is to get a list of different versions of MicroProfile. We can then query the service to establish which application servers support the requested version.

Supported Versions

To get a list of MicroProfile versions, we must perform a HTTP GET request to https://start.microprofile.io/api/1/mpVersion

david ~ $ curl https://start.microprofile.io/api/1/mpVersion
["MP22","MP21","MP20","MP14","MP13","MP12"]

This tells us that there are currently 6 versions of MicroProfile that are supported by the REST service: from MicroProfile 1.2 through to MicroProfile 2.2

As different application servers support different versions of MicroProfile, we can query the service to establish which application servers are supported for a specified version of MicroProfile. For example, to get a list of application servers that support MicroProfile 2.0, we would execute the following command:

david ~ $ curl https://start.microprofile.io/api/1/mpVersion/MP20
{"supportedServers":["KUMULUZEE","LIBERTY","PAYARA_MICRO","TOMEE"],"specs":["CONFIG","FAULT_TOLERANCE","JWT_AUTH","METRICS","HEALTH_CHECKS","OPEN_API","OPEN_TRACING","REST_CLIENT"]}
This api call tells us that Kumuluzee, Open Liberty, Payara Micro and TomEE support version 2.0 of MicroProfile and they support the MicroProfile APIs:
Fault Tolerance
JWT Authentication
Metrics
Health Checks
Open API
Open Tracing
REST Client

We now have sufficient information to create a basic MicroProfile 2.0 application. In this example, we’ll use the TomEE application server.

Creating an Application

So, to create a basic MicroProfile application, execute the following command:

david ~ $ curl -O -J https://start.microprofile.io/api/1/project?supportedServer=TOMEE
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 12555  100 12555    0     0  44818      0 --:--:-- --:--:-- --:--:-- 44679
curl: Saved to filename 'demo.zip'

The -O parameter tells curl to download a file rather than output to the console. The -J parameter tells curl to correctly name the file after it is downloaded, in this case, demo.zip. If the -J parameter was not specified, the downloaded file would be called project?supportedServer=TOMEE

So, what does the created project look like? A bit like this :)

demo/src/test/java/.gitkeep
demo/src/main/webapp/WEB-INF/beans.xml
demo/src/main/java/com/example/demo/HelloController.java
demo/pom.xml
demo/src/main/java/com/example/demo/health/ServiceHealthCheck.java
demo/src/main/resources/META-INF/microprofile-config.properties
demo/src/main/java/com/example/demo/resilient/ResilienceController.java
demo/src/main/webapp/.gitkeep
demo/src/test/java/com/example/demo/JWTClient.java
demo/src/main/java/com/example/demo/config/ConfigTestController.java
demo/src/test/resources/privateKey.pem
demo/src/main/java/com/example/demo/DemoRestApplication.java
demo/readme.md
demo/src/main/resources/.gitkeep
demo/src/main/java/com/example/demo/secure/ProtectedController.java
demo/src/test/java/com/example/demo/MPJWTToken.java
demo/src/main/resources/publicKey.pem
demo/src/main/webapp/index.html
demo/src/main/java/com/example/demo/metric/MetricController.java

The first thing you notice here is that the package names are of the format com.example.demo. We can easily refactor this in an IDE, or we could specify the Maven Group and Artifact Id’s and have the project created exactly as we want it, by appending the groupId and artifactId arguments to the call.

For example, to create the project with a groupId of com.acme and an artifactId of customer-service, we would append the groupId and artifactId parameters as:

david ~ $ curl -O -J 'https://start.microprofile.io/api/1/project?supportedServer=TOMEE&groupId=com.acme&artifactId=customer-service'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 13420  100 13420    0     0  45698      0 --:--:-- --:--:-- --:--:-- 45802
curl: Saved to filename 'customer-service.zip'

Now, note that the download is called customer-service.zip and its contents are:

customer-service/src/main/java/com/acme/customer/service/metric/MetricController.java
customer-service/src/main/java/com/acme/customer/service/resilient/ResilienceController.java
customer-service/pom.xml
customer-service/src/main/java/com/acme/customer/service/CustomerserviceRestApplication.java
customer-service/src/main/webapp/index.html
customer-service/src/main/webapp/WEB-INF/beans.xml
customer-service/src/main/java/com/acme/customer/service/HelloController.java
customer-service/src/main/java/com/acme/customer/service/config/ConfigTestController.java
customer-service/src/main/webapp/.gitkeep
customer-service/src/main/resources/META-INF/microprofile-config.properties
customer-service/src/main/resources/.gitkeep
customer-service/src/test/resources/privateKey.pem
customer-service/src/test/java/com/acme/customer/service/JWTClient.java
customer-service/src/main/java/com/acme/customer/service/health/ServiceHealthCheck.java
customer-service/src/main/resources/publicKey.pem
customer-service/src/main/java/com/acme/customer/service/secure/ProtectedController.java
customer-service/src/test/java/.gitkeep
customer-service/readme.md
customer-service/src/test/java/com/acme/customer/service/MPJWTToken.java

With the starter web application, we can also request to add examples for the different MicroProfile specifications (such as Config or Fault Tolerance) into the generated application. We can also do this with the REST client by utilizing the selectedSpecs parameter. The values used for this parameter are those returned in the curl shown at the beginning of this post.

So for example, to add an example for Config and Fault Tolerance, we would use the following curl:

david ~ $ curl -O -J 'https://start.microprofile.io/api/1/project?supportedServer=TOMEE&groupId=com.acme&artifactId=customer-service&selectedSpecs=CONFIG&selectedSpecs=FAULT_TOLERANCE'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  6107  100  6107    0     0  17580      0 --:--:-- --:--:-- --:--:-- 17548
curl: Saved to filename 'customer-service.zip'

In this post, we’ve seen how we can easily utilize the MicroProfile Starter REST interface to easily create projects.

Happy MicroProfiling !

Credits

Photo by Athul Cyriac Ajay on Unsplash

Creating a MongoDB Capped Collection In Java

Davey — Tue, 13 Jul 2021 21:02:53 +0000

In MongoDB, it’s possible to preserve the insertion order of documents into a collection in a circular fashion. These types of collections are called Capped Collections in MongoDB. The MongoDB documentation describes Capped Collections:

“Capped collections are fixed-size collections that support high-throughput operations that insert, retrieve, and delete documents based on insertion order. Capped collections work in a way similar to circular buffers: once a collection fills its allocated space, it makes room for new documents by overwriting the oldest documents in the collection.”

OK, so we know what a capped collection is, but how to we create one?

From the MongoDB shell, we would create a capped collection using the db.createCollection command:

db.createCollection("logs", {capped: true,
                             size: 4096,
                             max:5})

This command tells MongoDB to create a collection called “logs” with a maximum size of 4096 bytes which can hold a maximum of 5 documents. When the 6th document is added, the first document is removed form the collection ensuring that there is only ever a maximum of 5 documents in the collection. The “size” parameter is mandatory, however the “max” parameter is optional.

In Java, there are two common ways to communicate with MongoDB; the MongoDB Java Driver and with Morphia (the lightweight type-safe mapping library for mapping Java objects to/from MongoDB).

First, let’s first look at using the Java Driver.

Java Driver

With the Java Driver, again, we use the db.createCollection command, this time passing a BasicDBObject as the parameter. This parameter has the fields “capped”, “size” and “max” which specify that the collection is capped, the maximum size of the collection in bytes and the maximum number of entries in the collection. The following code snippet shows how to connect to a local instance of MongoDB and create a capped collection.

MongoClient mongoClient = new MongoClient(new MongoClientURI("mongodb://localhost"));
DB db = mongoClient.getDB("test");

DBCollection collection;
if (!db.collectionExists("cappedLogsJavaDriver")) {
    BasicDBObject options = new BasicDBObject("capped", true);
    options.append("size", 4096);
    options.append("max", 5);
    collection = db.createCollection("cappedLogsJavaDriver", options);
} else {
    collection = db.getCollection("cappedLogsJavaDriver");
}

Once we’ve created a collection, we can insert documents into it to ensure that it is working as expected. The following code snippet shows how to insert 8 documents into the collection (remember, only the last 5 of these will be stored as it’s a capped collection).

for (int i = 0; i < 8; i++) {
    BasicDBObject logEntry = new BasicDBObject("logId", i);
    collection.insert(logEntry);
}

Using the MongoDB interactive shell, we can verify the documents that are now stored in the collection are as expected.

> db.cappedLogsJavaDriver.find()
{ "_id" : ObjectId("54a1ca44a82617da4f72e025"), "logId" : 3 }
{ "_id" : ObjectId("54a1ca44a82617da4f72e026"), "logId" : 4 }
{ "_id" : ObjectId("54a1ca44a82617da4f72e027"), "logId" : 5 }
{ "_id" : ObjectId("54a1ca44a82617da4f72e028"), "logId" : 6 }
{ "_id" : ObjectId("54a1ca44a82617da4f72e029"), "logId" : 7 }

Morphia

Now that we’ve seen how to create a collection via the Java Driver, let’s see how we can accomplish the same thing by using Morphia.

The essence of Morphia is to map Java classes to/from MongoDB. Classes that we wish to persist in MongoDB are annotated with the @Entity annotation, which are then stored within a collection usually named after the class being annotated. To create a capped collection, we must add additional values onto the @Entity annotation to specify the maximum number of entries in the collection and the size of the collection. Modelling the same type of object as used in the example for the Java Driver, we would create a LogEntry class as follows:

@Entity(value="cappedLogsMorphia", cap=@CappedAt(count=5, value=4096))
public class LogEntry {

    private int logId;

    @Id
    private ObjectId id;

    public LogEntry(int logId) {
        this.logId = logId;
    }

    public int getLogId() {
        return logId;
    }

    public void setLogId(int logId) {
        this.logId = logId;
    }
}

We can see that this class is annotated with @Entity specifying that the collection should be capped with a maximum of 5 documents and a size of 4096 bytes.

With Morphia, the capped collection is created at start-up time by invoking the .ensureCaps() method on the Morphia Datastore as shown below.

MongoClient mongoClient = new MongoClient(new MongoClientURI("mongodb://localhost"));
DB db = mongoClient.getDB("test");

Morphia morphia = new Morphia();
morphia.map(LogEntry.class);

Datastore datastore = morphia.createDatastore(mongoClient, "test");
datastore.ensureCaps();

Again, as before, we can insert 8 documents into the collection to verify that only the last 5 are stored.

for (int i = 0; i < 8; i++) {
    LogEntry logEntry = new LogEntry(i);
    datastore.save(logEntry);
}

> db.cappedLogsMorphia.find()
{ "_id" : ObjectId("54a1ce9da82629642c64f5d9"), "className" : "com.cloudblogaas.cappedcollection.LogEntry", "logId" : 3 }
{ "_id" : ObjectId("54a1ce9da82629642c64f5da"), "className" : "com.cloudblogaas.cappedcollection.LogEntry", "logId" : 4 }
{ "_id" : ObjectId("54a1ce9da82629642c64f5db"), "className" : "com.cloudblogaas.cappedcollection.LogEntry", "logId" : 5 }
{ "_id" : ObjectId("54a1ce9da82629642c64f5dc"), "className" : "com.cloudblogaas.cappedcollection.LogEntry", "logId" : 6 }
{ "_id" : ObjectId("54a1ce9da82629642c64f5dd"), "className" : "com.cloudblogaas.cappedcollection.LogEntry", "logId" : 7 }

Checking a collection status

Once we’ve created a capped collection in MongoDB, we can check its status by executing the .stats() method on the collection from within the Mongo DB interactive shell.

> db.cappedLogsJavaDriver.stats()
{
    "ns" : "test.cappedLogsJavaDriver",
    "count" : 5,
    "size" : 180,
    "avgObjSize" : 36,
    "storageSize" : 4096,
    "numExtents" : 1,
    "nindexes" : 1,
    "lastExtentSize" : 4096,
    "paddingFactor" : 1,
    "systemFlags" : 1,
    "userFlags" : 0,
    "totalIndexSize" : 8176,
    "indexSizes" : {
        "_id_" : 8176
    },
    "capped" : true,
    "max" : 5,
    "ok" : 1
}

Here we can see that the collection is indeed capped (“capped”=true) and that the maximum number of entries in the collection is 5 (“max”=5).

Credits

Photo by Jon Tyson on Unsplash

Custom JDK Versions Per Project With SDKMAN!

Davey — Tue, 13 Jul 2021 20:57:24 +0000

SDKMAN! allows us to easily install and use different JDKs. It can become frustrating to have to change JDK versions manually for different projects. Fortunately, SDKMAN! has a solution to this in the form of the sdk env command. This command allows us to define a different JDK per project which can be easily switched to, without having to remember the JDK version. Additionally, it can be configured to automatically change to the required JDK when changing directories.

For more information about installing and using SDKMAN!, check out my article - Managing JDK Versions With SDKMAN!

Updating SDKMAN!

Before we can start using the sdk env command, we must first ensure we are using the latest version of SDKMAN! The software can be updated by using the sdk selfupdate command

% sdk selfupdate
No update available at this time.

Specifying a JDK Version For A Project

SDKMAN! uses a hidden file called .sdkmanrc which contains a parameter specifying which JDK version to use for a project. To configure and use this file is a simple 3 step process.

1 Create the .sdkmanrc file. This can easily be created by navigating to the required directory and executing sdk env init

% sdk env init
.sdkmanrc created.

2 Configure the version. Once we've created the file, edit it and add a line java=x.x.x.x specifying the version of Java to use. For example to use Adopt Open JDK version 11, the file would look like:

% cat .sdkman
java=11.0.2.hs-adpt

3 Initialize the environment. After editing the .sdkmanrc fille, we can configure the environment by executing the sdk env command.

% sdk env

Using java version 11.0.2-open in this shell.
%
% java -version
openjdk version "11.0.2" 2019-01-15
OpenJDK Runtime Environment 18.9 (build 11.0.2+9)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.2+9, mixed mode

Specifying a JDK Automatically

Finally, we can get SDKMAN! to automatically change JDK to the version defined within .sdkmanrc by changing configuration. To do this, edit the ~/.sdkman/etc/config file and add/edit the property sdkman_auto_env=true. This will probably already exist and be set to false, so make sure you edit this setting if it is there rather than duplicating it.

Now, whenever you change into a directory containing a .sdkmanrc file in it, the JDK will automatically be changed to your desired version. Fantastic !

% cd project1 

Using java version 11.0.2-open in this shell.
%
% cd project2

Using java version 14.0.1-open in this shell.

Credits

Photo by Octavian Dan on Unsplash

Managing JDK Versions With SDKMAN!

Davey — Thu, 08 Jul 2021 19:32:23 +0000

SDKMAN! describes itself as:

a tool for managing parallel versions of multiple Software Development Kits on most Unix based systems. It provides a convenient Command Line Interface (CLI) and API for installing, switching, removing and listing Candidates.

What this means, is that it is a powerful tool that allows you to easily change between different versions of the JDK without having to modify JAVA_HOME or other environment variables.

Installing SDKMAN!

Installation is as simple as running a bash command. For Linux and Mac users, simply execute the following command and follow the on-screen instructions.

curl -s "https://get.sdkman.io" | bash

For Windows users, the recommended way of installation is via WSL. This is covered in the installation documentation.

Installing a JDK

After installing SDKMAN!, the next stage is to decide which JDK versions you wish to install. SDKMAN! allows you to install different version of the JDK by different vendors. To get a complete list of the different JDK's available, execute the sdk list java command:

% sdk list java
================================================================================
Available Java Versions
================================================================================
 Vendor        | Use | Version      | Dist    | Status     | Identifier
--------------------------------------------------------------------------------
 AdoptOpenJDK  |     | 14.0.1.j9    | adpt    |            | 14.0.1.j9-adpt      
               |     | 14.0.1.hs    | adpt    |            | 14.0.1.hs-adpt      
               |     | 13.0.2.j9    | adpt    |            | 13.0.2.j9-adpt      
               |     | 13.0.2.hs    | adpt    |            | 13.0.2.hs-adpt      
               |     | 12.0.2.j9    | adpt    |            | 12.0.2.j9-adpt      
               |     | 12.0.2.hs    | adpt    |            | 12.0.2.hs-adpt      
               |     | 11.0.7.j9    | adpt    |            | 11.0.7.j9-adpt      
               |     | 11.0.7.hs    | adpt    |            | 11.0.7.hs-adpt      
               |     | 8.0.252.j9   | adpt    |            | 8.0.252.j9-adpt     
               |     | 8.0.252.hs   | adpt    |            | 8.0.252.hs-adpt     
 Amazon        |     | 11.0.7       | amzn    |            | 11.0.7-amzn         
               |     | 8.0.252      | amzn    |            | 8.0.252-amzn        
               |     | 8.0.202      | amzn    |            | 8.0.202-amzn        
...

This command will list all the different vendors and all the different JDK's available from each vendor.

In the snippet above, you can see, for example, that Amazon provides (amongst others) Java version 11.0.7 and AdoptOpenJDK supports version 14.0.1 (again amongst others).

To install a specific JDK, we use the sdk install java command, noting the Identifier column. This is used to uniquely specify vendor and JDK version.

So, for example to install AdoptOpenJDK version 14, we would execute sdk install java 14.0.1.hs-adpt

% sdk install java 14.0.1.hs-adpt 

Downloading: java 14.0.1.hs-adpt

In progress...

#################### 100.0%
#################### 100.0%

Repackaging Java 14.0.1.hs-adpt...

Done repackaging...
Cleaning up residual files...

Installing: java 14.0.1.hs-adpt
Done installing!

Do you want java 14.0.1.hs-adpt to be set as default? (Y/n): y

Setting java 14.0.1.hs-adpt as default.

We can now check the Java version:

 % java -version
openjdk version "14.0.1" 2020-04-14
OpenJDK Runtime Environment AdoptOpenJDK (build 14.0.1+7)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 14.0.1+7, mixed mode, sharing)

In this example, we selected to install this version of Java as the default. What if we just want to specify a different version of the JDK to use rather than installing a version?

Changing JDK Version

To use a specific JDK version, you must first install it as shown in the previous section. Then, simply execute the sdk use java ... command to change the current shell to be a specific version.

% sdk use java 11.0.7.hs-adpt

Using java version 11.0.7.hs-adpt in this shell.
david@Davids-MacBook-Air ~ % java -version
openjdk version "11.0.7" 2020-04-14
OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.7+10)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 11.0.7+10, mixed mode)

If we want to permanently change the JDK version, we can use the sdk default java ... command

% sdk default java 11.0.7.hs-adpt

Default java version set to 11.0.7.hs-adpt

Now, when we open a new shell or terminal, the default JDK will have changed to the one we just specified. We can use the sdk current java command to display the currently selected JDK version.

Its not Just Java though!

Throughout this article, we've made reference to specifying different versions of the Java JDK. SDKMAN! is much more than a tool to change Java versions - it can do the same job for different SDK's and tools, for example, Apache Ant, Maven or Spring Boot.

To get a full list of managed SDKs, execute sdk list, or visit the prioject documentation.

Conclusion

In this article, we've seen how to install SDKMAN! and how to install and select different JDK versions. This is only one way of defining different JDKs, but its very powerful and very simple.

Credits

Photo by Tom Hermans on Unsplash

Accessing Multiple Databases From a Spring Boot Application

Davey — Thu, 08 Jul 2021 19:13:38 +0000

When developing any application, it’s quite common to have to access multiple databases. Out of the box, Spring Boot provides easy access to a single datasource, in the simplest case just by specifying the JDBC driver on the class path!

Accessing multiple databases however, is still straightforward with Spring Boot. This article shows how to connect to two different MySql datasources from a Spring Boot application.

To showcase how to connect to to different databases, consider a products database and a customer database, with the following simplistic schema and data.

Database One - Products Database

Schema

create table PRODUCT(id integer, name varchar(255));

Data

insert into PRODUCT(id, name) values (1, ‘XBox');

Database Two - Customer Database

Schema

create table CUSTOMER(id integer, name varchar(255));

Data

insert into CUSTOMER(id, name) values (1, 'Daphne Jefferson’);

To access the databases, we need to declare a JdbcTemplate for each database. In Spring, JdbcTemplates are created from a DataSource which has a set of connection properties (url, username, password etc.)

@Configuration
public class DataSourceConfig {

  Bean
  @Qualifier("customerDataSource")
  @Primary
  @ConfigurationProperties(prefix="customer.datasource")
  DataSource customerDataSource() {
    return DataSourceBuilder.create().build();
  }

  @Bean
  @Qualifier("productDataSource")
  @ConfigurationProperties(prefix="product.datasource")
  DataSource productDataSource() {
    return DataSourceBuilder.create().build();
  }

  @Bean
  @Qualifier("customerJdbcTemplate")
  JdbcTemplate customerJdbcTemplate(@Qualifier("customerDataSource")DataSource customerDataSource) {
    return new JdbcTemplate(customerDataSource);
  }

  @Bean
  @Qualifier("productJdbcTemplate")
  JdbcTemplate productJdbcTemplate(@Qualifier("productDataSource")DataSource productDataSource) {
    return new JdbcTemplate(productDataSource);
  }
}

In the above code we can see that a @Configuration bean has been declared that defines a customerDatasource and a customerJdbcTemplate. Each of these beans are annotated with the @Qualifier('customer...') to identify them as relating to the customer database.

Similarly, the above code defines a productDataSource and a productJdbcTemplate. Again these are annotated with @Qualifier('product...') to identify them as relating to the product database.

Finally, each DataSource Bean is annotated with the @ConfigurationProperties(prefix="...datasource") annotation. This tells Spring Boot what properties within the application.properties file should be used for connecting to each database. The application.properties file therefore looks like the following:

product.datasource.url = jdbc:mysql://localhost:3306/dbOne
product.datasource.username = user1
product.datasource.password = password
product.datasource.driverClassName = com.mysql.jdbc.Driver

customer.datasource.url = jdbc:mysql://localhost:3306/dbTwo
customer.datasource.username = user2
customer.datasource.password = password
customer.datasource.driverClassName = com.mysql.jdbc.Driver

Now that we've seen how to create a DataSource and JdbcTemplate, the JdbcTemplate can be injected into a @Repository for use, e.g.

@Repository
public class CustomerRepository {

  private static final String SELECT_SQL = "select NAME from CUSTOMER where ID=?"; 
  @Autowired
  @Qualifier("customerJdbcTemplate")
  JdbcTemplate customerJdbcTemplate;

  public String getCustomerName(int id) {
    String name = customerJdbcTemplate.queryForObject(SELECT_SQL, new Object[] {id}, String.class);

    return name;
  }
}

Again, note the use of the @Qualifier annotation to specify which JdbcTemplate is required for the different repositories.

The ProductRepository is similarly written to access the productJdbcTemplate

@Repository
public class ProductRepository {

  private static final String SELECT_SQL = "select NAME from PRODUCT where ID=?"; 
  @Autowired
  @Qualifier("productJdbcTemplate")
  JdbcTemplate productJdbcTemplate;

  public String getProductName(int id) {
    String name = productJdbcTemplate.queryForObject(SELECT_SQL, new Object[] {id}, String.class);

    return name;
  }
}

With a few simple steps, Spring Boot allows us to easily connect to multiple databases when using JdbcTemplates.

Credits

Photo by Jan Antonin Kolar on Unsplash

Debugging SpringBoot Application In IntelliJ Idea CE

Davey — Thu, 08 Jul 2021 18:58:17 +0000

When debugging a SpringBoot application in IntelliJ Idea Community Edition, additional steps need to be taken.

If you have defined your run configuration as spring-boot:run, you will find that the application runs, but does not stop at breakpoints as expected.

An easy way to resolve this is to set the spring-boot.run.fork property to false.

This can be set wither within the pom.xml file, or within the run configuration. To set the run configuration, set the "Command line:" within the Run Configuration to be:

spring-boot:run -Dspring-boot.run.fork=false

The default value of this property is true, which is usually the best setting, but not when debugging from IntelliJ Idea CE. If the application is forked, then the relevant JVM properties are not passed on to the forked process and debugging does not work as expected.

Changing the value of this property all the time can cause problems with Spring Boot DevTools, so its only recommended to set at runtime and not in the pom.xml file where it will be used every time the application is started.

Credits

Photo by Zan on Unsplash