DEV Community: Marco Villarreal

Creating a Spark Standalone Cluster with Docker and docker-compose(2021 update)

Marco Villarreal — Sun, 27 Jun 2021 01:53:25 +0000

Back in 2018 I wrote this article on how to create a spark cluster with docker and docker-compose, ever since then my humble repo got 270+ stars, a lot of forks and activity from the community, however I abandoned the project by some time(Was kinda busy with a new job on 2019 and some more stuff to take care of), I've merged some pull quest once in a while, but never put many attention on upgrading versions.

But today we are going to revisit this old fella with some updates and hopefully run some examples with scala and python(yeah 2018 version didn't support python, thanks to the community to bring pyspark to this baby).

Requirements

Docker (I am using version 20.10.7)
docker-compose (I am using version 1.21.2)
This repo ;)

Project Structure

The following project structure will be used

|
|--|apps # Apps directory for volume mounts(any app you want to deploy just paste it here)
|--|data # Data directory for volume mounts(any file you want to process just paste it here)
|--|Dockerfile #Dockerfile used to build spark image
|--|start-spark.sh # startup script used to run different spark workloads
|--|docker-compose.yml # the compose file

Creating The Image

In the 2018 version, we've used a base image and a separated image for each spark workload(one image for the master one for the worker and one for spark-submit). In this new approach we will use docker multi stage builds to create a unique image that can be launched as any workload we want.

Here's the dockerfile used to define our apache-spark image:


# builder step used to download and configure spark environment
FROM openjdk:11.0.11-jre-slim-buster as builder

# Add Dependencies for PySpark
RUN apt-get update && apt-get install -y curl vim wget software-properties-common ssh net-tools ca-certificates python3 python3-pip python3-numpy python3-matplotlib python3-scipy python3-pandas python3-simpy

RUN update-alternatives --install "/usr/bin/python" "python" "$(which python3)" 1

# Fix the value of PYTHONHASHSEED
# Note: this is needed when you use Python 3.3 or greater
ENV SPARK_VERSION=3.0.2 \
HADOOP_VERSION=3.2 \
SPARK_HOME=/opt/spark \
PYTHONHASHSEED=1

# Download and uncompress spark from the apache archive
RUN wget --no-verbose -O apache-spark.tgz "https://archive.apache.org/dist/spark/spark-${SPARK_VERSION}/spark-${SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}.tgz" \
&& mkdir -p /opt/spark \
&& tar -xf apache-spark.tgz -C /opt/spark --strip-components=1 \
&& rm apache-spark.tgz


# Apache spark environment
FROM builder as apache-spark

WORKDIR /opt/spark

ENV SPARK_MASTER_PORT=7077 \
SPARK_MASTER_WEBUI_PORT=8080 \
SPARK_LOG_DIR=/opt/spark/logs \
SPARK_MASTER_LOG=/opt/spark/logs/spark-master.out \
SPARK_WORKER_LOG=/opt/spark/logs/spark-worker.out \
SPARK_WORKER_WEBUI_PORT=8080 \
SPARK_WORKER_PORT=7000 \
SPARK_MASTER="spark://spark-master:7077" \
SPARK_WORKLOAD="master"

EXPOSE 8080 7077 6066

RUN mkdir -p $SPARK_LOG_DIR && \
touch $SPARK_MASTER_LOG && \
touch $SPARK_WORKER_LOG && \
ln -sf /dev/stdout $SPARK_MASTER_LOG && \
ln -sf /dev/stdout $SPARK_WORKER_LOG

COPY start-spark.sh /

CMD ["/bin/bash", "/start-spark.sh"]

Notice that in the dockerfile we reference a script called start-spark.sh, it's primary goal is to run spark-class script with the given role (master, or worker).

#start-spark.sh
#!/bin/bash
. "/opt/spark/bin/load-spark-env.sh"
# When the spark work_load is master run class org.apache.spark.deploy.master.Master
if [ "$SPARK_WORKLOAD" == "master" ];
then

export SPARK_MASTER_HOST=`hostname`

cd /opt/spark/bin && ./spark-class org.apache.spark.deploy.master.Master --ip $SPARK_MASTER_HOST --port $SPARK_MASTER_PORT --webui-port $SPARK_MASTER_WEBUI_PORT >> $SPARK_MASTER_LOG

elif [ "$SPARK_WORKLOAD" == "worker" ];
then
# When the spark work_load is worker run class org.apache.spark.deploy.master.Worker
cd /opt/spark/bin && ./spark-class org.apache.spark.deploy.worker.Worker --webui-port $SPARK_WORKER_WEBUI_PORT $SPARK_MASTER >> $SPARK_WORKER_LOG

elif [ "$SPARK_WORKLOAD" == "submit" ];
then
    echo "SPARK SUBMIT"
else
    echo "Undefined Workload Type $SPARK_WORKLOAD, must specify: master, worker, submit"
fi

To build the image just run:

docker build -t cluster-apache-spark:3.0.2 .

After some time the image will be successfully created, it will take some time depending on how fast the dependencies and the spark tarball are dowloaded (fortunatelly these steps get cached as a layer thanks to the multistage setup).

The compose File

Now that we have our apache-spark image is time to create a cluster in docker-compose

version: "3.3"
services:
  spark-master:
    image: cluster-apache-spark:3.0.2
    ports:
      - "9090:8080"
      - "7077:7077"
    volumes:
       - ./apps:/opt/spark-apps
       - ./data:/opt/spark-data
    environment:
      - SPARK_LOCAL_IP=spark-master
      - SPARK_WORKLOAD=master
  spark-worker-a:
    image: cluster-apache-spark:3.0.2
    ports:
      - "9091:8080"
      - "7000:7000"
    depends_on:
      - spark-master
    environment:
      - SPARK_MASTER=spark://spark-master:7077
      - SPARK_WORKER_CORES=1
      - SPARK_WORKER_MEMORY=1G
      - SPARK_DRIVER_MEMORY=1G
      - SPARK_EXECUTOR_MEMORY=1G
      - SPARK_WORKLOAD=worker
      - SPARK_LOCAL_IP=spark-worker-a
    volumes:
       - ./apps:/opt/spark-apps
       - ./data:/opt/spark-data
  spark-worker-b:
    image: cluster-apache-spark:3.0.2
    ports:
      - "9092:8080"
      - "7001:7000"
    depends_on:
      - spark-master
    environment:
      - SPARK_MASTER=spark://spark-master:7077
      - SPARK_WORKER_CORES=1
      - SPARK_WORKER_MEMORY=1G
      - SPARK_DRIVER_MEMORY=1G
      - SPARK_EXECUTOR_MEMORY=1G
      - SPARK_WORKLOAD=worker
      - SPARK_LOCAL_IP=spark-worker-b
    volumes:
        - ./apps:/opt/spark-apps
        - ./data:/opt/spark-data
  demo-database:
    image: postgres:11.7-alpine
    ports: 
      - "5432:5432"
    environment: 
      - POSTGRES_PASSWORD=casa1234

For both spark master and worker we configured the following environment variables:

Environment	Description
SPARK_MASTER	Spark master url
SPARK_WORKER_CORES	Number of cpu cores allocated for the worker
SPARK_WORKER_MEMORY	Amount of ram allocated for the worker
SPARK_DRIVER_MEMORY	Amount of ram allocated for the driver programs
SPARK_EXECUTOR_MEMORY	Amount of ram allocated for the executor programs
SPARK_WORKLOAD	The spark workload to run(can be any of master, worker, submit)

Compared to 2018 version the following changes were made:

Removed the custom network and ip addresses
Expose 2 workers instead of 3, and expose each worker port in the range of(9090...9091 and so on)
Pyspark support thanks to community contributions
Include a postgresql instance to run the demos(both demos store data in jdbc)

The final step to create your test cluster will be to run the compose file:

docker-compose up -d

To validate your cluster just access the spark UI on each worker & master URL

Spark Master: http://localhost:9090

Spark Worker 1: http://localhost:9091

Spark Worker 2: http://localhost:9092

Database Server

To check database server just use the psql command(or any database client of your choice):

psql -U postgres -h 0.0.0.0 -p 5432
#It will ask for your password defined in the compose file

The Demo Apps

The following apps can be found in apps directory, this apps are used as proof of concept of our cluster behavior.

NY Bus Stops Data [Pyspark]

This programs just loads archived data from MTA Bus Time and apply basic filters using spark sql, the result are persisted into a postgresql table.

The loaded table will contain the following structure:

latitude	longitude	time_received	vehicle_id	distance_along_trip	inferred_direction_id	inferred_phase	inferred_route_id	inferred_trip_id	next_scheduled_stop_distance	next_scheduled_stop_id	report_hour	report_date
40.668602	-73.986697	2014-08-01 04:00:01	469	4135.34710710144	1	IN_PROGRESS	MTA NYCT_B63	MTA NYCT_JG_C4-Weekday-141500_B63_123	2.63183804205619	MTA_305423	2014-08-01 04:00:00	2014-08-01

To submit the app connect to one of the workers or the master and execute:

/opt/spark/bin/spark-submit --master spark://spark-master:7077 \
--jars /opt/spark-apps/postgresql-42.2.22.jar \
--driver-memory 1G \
--executor-memory 1G \
/opt/spark-apps/main.py

MTA Bus Analytics[Scala]

This program takes the archived data from MTA Bus Time and make some aggregations on it, the calculated results are persisted on postgresql tables.

Each persisted table correspond to a particullar aggregation:

Table	Aggregation
day_summary	A summary of vehicles reporting, stops visited, average speed and distance traveled(all vehicles)
speed_excesses	Speed excesses calculated in a 5 minute window
average_speed	Average speed by vehicle
distance_traveled	Total Distance traveled by vehicle

To submit the app connect to one of the workers or the master and execute:

/opt/spark/bin/spark-submit --deploy-mode cluster \
--master spark://spark-master:7077 \
--total-executor-cores 1 \
--class mta.processing.MTAStatisticsApp \
--driver-memory 1G \
--executor-memory 1G \
--jars /opt/spark-apps/postgresql-42.2.22.jar \
--conf spark.driver.extraJavaOptions='-Dconfig-path=/opt/spark-apps/mta.conf' \
--conf spark.executor.extraJavaOptions='-Dconfig-path=/opt/spark-apps/mta.conf' \
/opt/spark-apps/mta-processing.jar

You will notice on the spark-ui a driver program and executor program running(In scala we can use deploy-mode cluster)

Conclusions

We've created a simpler version of a spark cluster in docker-compose, the main goal of this cluster is to provide you with a local environment to test the distributed nature of your spark apps without making any deploy to a production cluster.
The generated image isn't designed to have a small footprint(Image size is about 1gb).
This cluster is only necessary when you want to run a spark app in a distributed environment in your machine(Production use is discouraged, use databricks or kuberetes setup instead).

What's left to do?

Right now to run applications in deploy-mode cluster is necessary to specify arbitrary driver port through spark.driver.port configuration (I must fix some networking and port issues).
The spark submit entry in the start-spark.sh is unimplemented, the submit used in the demos can be triggered from any worker.

Deploy a Serverless Api on Cloud Run with Github Actions

Marco Villarreal — Mon, 21 Jun 2021 23:36:53 +0000

Google cloud Run is a serverless runtime designed for containerized applications, it allows to run high availability services with few configurations.

In this post we will deploy a Go Rest Api on Cloud Run using Github Actions as our CI/CD Tool.

Requirements

Google cloud cli
Terraform - I am using v1.0.0
Docker - I am using 20.10.7
Go - I am using go1.16.4
This Repo :)

Configuring Google Cloud Project

We need the following steps to setup a GCP project with cloud Run enabled and cloud SQL database instance prepared.

PROJECT_NAME="mvillarreal-demo-platform"
# Create GCP project(You must enable a billing account in your project)
gcloud projects create $PROJECT_NAME

# Set project as current running project
gcloud config set project $PROJECT_NAME

# View the current project
gcloud config get-value project

# Enable cloud Run api
gcloud services enable run.googleapis.com

# Enable resource manager api
gcloud services enable cloudresourcemanager.googleapis.com

# Enable vpc access api
gcloud services enable vpcaccess.googleapis.com

# Enable compute engine(for serverless vpc access)
gcloud services enable compute.googleapis.com

# Enable container Registry
gcloud services enable containerregistry.googleapis.com

# Enable Cloud SQL services
gcloud services enable sqladmin.googleapis.com

# Enable networking services
gcloud services enable servicenetworking.googleapis.com

# Service account for Github actions
gcloud iam service-accounts create mvillarrealb-gha-saccount \
--description "Main service account for github actions" \
--display-name "mvillarreal-gha-saccount"


# Assign editor role for service account(for terraform)
gcloud projects add-iam-policy-binding $PROJECT_NAME \
--member serviceAccount:mvillarrealb-gha-saccount@$PROJECT_NAME.iam.gserviceaccount.com \
--role roles/editor 

# Adding networking admin permission
gcloud projects add-iam-policy-binding $PROJECT_NAME \
--member serviceAccount:mvillarrealb-gha-saccount@$PROJECT_NAME.iam.gserviceaccount.com \
--role roles/servicenetworking.networksAdmin

# Export service account key for terraform(keep this in a safe place)
gcloud iam service-accounts keys create $(pwd)/terraform/service-account-key.json \
--iam-account mvillarrealb-gha-saccount@$PROJECT_NAME.iam.gserviceaccount.com

Setup Terraform

Our Cloud Run service uses a database and is deployed inside a private network, in order to create these additional resources we will use Terraform, to do so execute the following commands:

# Initialize terraform dependencies
cd terraform && terraform init

# Preview terraform plan
terraform plan

# Apply Terraform(it will take about 10 minutes, cloud sql instance take some time)
terraform apply

Post Terraform Tasks

After the installation some additional steps are required:

Create database poi_manager in your cloud SQL instance, you can do it directly on Google Cloud Console or via Cloud SQL proxy.
Generate a new password for the postgres User and copy it into some safe place(we will need it later)

Prepare Dockerfile

To create a containerized version of our api, we will use docker multi stage build and take advantage of Go static compilation and create a lightweight image.


#Build step
FROM golang:1.15 as builder
RUN mkdir -p /poi-api/api
WORKDIR /poi-api
ADD api ./api
COPY go.mod go.sum main.go ./
#static compilation options for go
RUN go build -ldflags "-linkmode external -extldflags -static" -o main .

#Run step
#Scratch image is an empty image to add our binary, so the image will be as small as possible
FROM scratch
#Environments for dataase connection
ENV DATABASE_HOST="127.0.0.1" \
DATABASE_PORT="5432" \
DATABASE_USERNAME="postgres" \
DATABASE_PASSWORD="password"
#Copy binary from builder
COPY --from=builder /poi-api/main ./main
CMD ["./main"]

Cloud Run Deploy with Github Actions

Finally to deploy the api in cloud run we will create a pipeline using Github actions, our pipeline structure is as follows:

name: poi-api
on:
  push:
    branches:
    - master
env:
  REGION: us-east1 # Cloud Run zone
  PROJECT_ID: mvillarreal-demo-platform # GCP project
  BASE_IMAGE: gcr.io/mvillarreal-demo-platform/poi-api #Container registry entry for the api
  DATABASE_INSTANCE: mvillarreal-pg-sql # Cloud sql instance name
  SERVICE_NAME: poi-api #Cloud run service name
  DATABASE_IP: 10.85.0.3 # My database private IP address
jobs:
  build: # Definition for Build Job
  deploy: # Definition for deploy Job
  test: # Definition for Test Job

Now we must define each job:

Build Job

In the build stage we will use our dockerfile to build and push to gcr the created image

    runs-on: ubuntu-latest
    steps:
      - name: Setup Project # Setup
        id: checkout
        uses: actions/checkout@master
      - name: Login to GCR # Login to GCP
        uses: docker/login-action@v1
        with:
          registry: gcr.io
          username: _json_key
          password: ${{ secrets.GCR_JSON_KEY }}
      - name: Build & Publish Image # Use the dockerfile to publish image
        uses: docker/build-push-action@v2
        id: build
        with:
          context: .
          push: true
          tags: ${{ env.BASE_IMAGE }}:${{ github.sha }}

Deploy Job

The deploy stage will use the deploy-cloudrun action to create a successful cloud run api

    runs-on: ubuntu-latest
    needs: [build]
    steps:
      - name: Deploy to Cloud Run
        id: deploy
        uses: google-github-actions/deploy-cloudrun@main
        with:
          region: ${{ env.REGION }}
          service: ${{ env.SERVICE_NAME }}
          image: ${{ env.BASE_IMAGE }}:${{ github.sha }}
          credentials: ${{ secrets.GCP_SA_KEY }}
          env_vars: "DATABASE_HOST=${{ env.DATABASE_IP }},DATABASE_USERNAME=${{ secrets.DATABASE_USERNAME }},DATABASE_PASSWORD=${{ secrets.DATABASE_PASSWORD }}"
          flags: "--allow-unauthenticated --vpc-connector vpc-conn --add-cloudsql-instances '${{ env.PROJECT_ID }}:${{env.REGION}}:${{env.DATABASE_INSTANCE}}'"

Let's take a closer look on some settings we've added in here:

env_vars

Environment variables set on cloud run application in the format env=value,env2=value2, the variables assigned were:

Variable	Description
DATABASE_HOST	Database Host, in this case the private ip specified in the env section
DATABASE_USERNAME	Database Username loaded in the secrets of our repo
DATABASE_PASSWORD	Database Password loaded in the secrets of our repo

flags

Cloud run specific settings used to configure the api:

Setting	Description
--allow-unauthenticated	Allow requests from unauthenticated users
--vpc-connector vpc-conn	Specifiy a serverless vpc connector vpc-conn was the connector we created on the terraform file
--add-cloudsql-instances 'env.PROJECT_ID:env.REGION:env.DATABASE_INSTANCE'	Links the cloud sql instance to the api to be able to use it

Test Job

Last but not least we will run a postman collection to test our deployed service

    runs-on: ubuntu-latest
    needs: [build, deploy]
    steps:
      - name: Generate Variable File
        run: |
          echo {\"BASE_URL\": \"{{ needs.deploy.outputs.url }}\"} > variables.json
      - name: Run e2e Test
        uses: matt-ball/newman-action@master
        id: test
        with:
          collection: poi-e2e.postman_collection.json
          environment: variables.json

Before pushing your changes, we must configure some secrets under the Settings > Secrets in our repository:

The secrets we've added are the following:

Secret	Description
DATABASE_USERNAME	Cloud SQL database username
DATABASE_PASSWORD	Cloud SQL database password(we adquired this in previous steps)
GCP_SA_KEY	Base 64 encoded service account key
GCR_JSON_KEY	Json format service account key

After a push into the master branch a job will be triggered deploying our service to cloud run:

Post Installation Steps

If you want to use the geocoding endpoint and reference the points of interest you can load the sql file in GCP:

# Create a bucket to load initial data
gsutil mb gs://h3-indexes

# Upload files
gsutil cp $(pwd)/data/*.sql gs://h3-indexes

# Load data(Remember to add ACL permissions to the h3-indexes directory)
gcloud sql instances import mvillarreal-pg-sql gs://h3-indexes/PE-Lima.sql \
--database poi_manager

What Have We done so far?

Setup a Google Cloud project
Provision basic services with Terraform
Dockerize a Go Rest Api using multi stage builds
Use Github Actions to enable a CI/CD pipeline

What's Left to do?

Add a custom Domain for your Cloud Run api
Disable public access for your api
Add some unit test on the codebase

Conclussions

Cloud Run can be a good option if you are starting a serverless based architecture, or can be an intermediary step if you are thinking on kubernetes on a long term(both solutions are container based).
Github actions is a robust CI/CD tool with strong community based actions
Using golang you can reduce the footprint of your end product using static compilation :)

References

E2E Sandbox with Test-containers

Marco Villarreal — Thu, 10 Jun 2021 21:57:29 +0000

End to End tests can be cumbersome the mayority of the times, we need to wire up a lot of services (cache servers, database engines, message brokers etc) resulting in a big ball of stuff to do in order to run a simple test.

These tests are the most expensive ones in the test pyramid, but are the ones closer to a real production scenario.

While unit tests ensures our business logic is ok, e2e tests ensures that our platform will perform accordingly with no surprises.

In this article we are going to put hands up with testcontainers to create an e2e sandbox and run some postman assertions through newman.

What is TestContainers?

Tescontainers is a java library to run containers in JUnit tests, it provides lightweight containers environment to create infraestructure service and test real case scenarios(Database connections, Cache servers etc.).

While the intended use of testcontainers is to run containers in the context of a Junit test, we will use it to create a sandbox environment outside junit environment.

Tools Needed

The following tools are needed to run this example:

Java (I am using openjdk 11.0.11 2021-04-20)
Docker (I am using Docker version 20.10.7, build f0df350)
Postman collection(Attached in the repo)
Newman (I am using 5.2.3) + newman-html-reporter
This repo :D

Our Service

Our service it is a continuation of my previous post on database migration. TLDR; we are working on a really simple book registry api.

However as you can see there are several infraestructure services to provide a fully featured service:

The services we need to provide in the sandbox are:

Google Pub/sub: To publish Events regarding the Book registry
Google Cloud Storage: To upload book covers
PostgreSQL: To store books and author data.

Creating The Spring Profile

To create the sadbox environment we will take advantage of the spring profiles and conditional beans, to achieve that, the following steps are required:

Configure testcontainers dependencies

We will add testcontainers gcp module and of course testcontaiers for postgresql

    implementation "org.testcontainers:gcloud:1.15.3"
    runtimeOnly 'org.testcontainers:postgresql'

Create Profile application-test.yaml

Apart from our application-default.yaml, we will add an additional yaml file for our test profile

+--resources
|   +--db
|   +--application-default.yaml
|   +--application-test.yaml #Sandbox configuration file

The content of application-test.yaml looks like this:

server:
  port: 8080
  profiles: test # Define the applied profile
logging:
  level:
    root: WARN
    org.springframework.web: WARN
    org.mvillabe.books: INFO
    org.hibernate: WARN
spring:
  liquibase:
    enabled: true
    change-log: "classpath:/db/changelog.yaml"
  datasource:
    url: "jdbc:tc:postgresql:11.7-alpine:///book_demo" #Testcontainers jdbc URI format
    driverClassName: "org.testcontainers.jdbc.ContainerDatabaseDriver" #Tescontainers jdbc driver to hook automatically
  cloud:
    gcp:
      project-id: "test-containers"
      credentials: # Dummy encoded key
        encoded-key: "${BASE64_ENCODED_DUMMY_SVC_ACCOUNT}"
      storage: 
        enabled: false # Disable default bean wiring for storage
      pubsub:
        emulator-host: "${EMULATOR_HOST:127.0.0.1:8085}"

Configuring Sandbox beans

To configure the sandbox beans we will use the following class:

@Profile("test") //This configuration bean will only be available on test profile
@Configuration
public class BeanConfiguration {
  /*Your bean definition*/
}

Each service configuration must be added to BeanConfiguration class:

Configuring Postgresql

Automatically wired with testcontainers jdbcDriver(org.testcontainers.jdbc.ContainerDatabaseDriver) - Sweet!

Configuring Cloud Storage

Disable default spring cloud wiring with property spring.cloud.gcp.storage.enabled = false <- we did this in our application-test.yaml
Define a StorageEmulator container:

    @Bean
    public StorageEmulator storageEmulator() {
        StorageEmulator storageEmulator = new StorageEmulator();
        storageEmulator.start();
        return storageEmulator;
    }

Manually register Storage Bean:

    /**
    As you can see Storage bean uses as dependency the previously created StorageEmulator bean, this is because
    we need to wire up the emulator first and provide an endpoint for cloud storage to work it
    */
    @Bean
    @Primary
    public Storage storage(StorageEmulator emulator, GcpProjectIdProvider gcpProjectIdProvider) {
        return StorageOptions.newBuilder()
                .setProjectId(gcpProjectIdProvider.getProjectId())
                .setHost(emulator.getEmulatorEndpoint())
                .build()
                .getService();
    }

Configuring Pub/Sub

Define a PubSubEmulatorContainer container:

    @Bean("pubSubEmulatorContainer")
    public PubSubEmulatorContainer pubSubEmulatorContainer() {
        PubSubEmulatorContainer container = new PubSubEmulatorContainer(
            DockerImageName.parse("gcr.io/google.com/cloudsdktool/cloud-sdk:316.0.0-emulators")
        );
        container.start();
        return container;
    }

Override bean GcpPubSubProperties (Spring Cloud AutoConfiguration Bean) to reload EMULATOR_HOST environment variable with the emulator Host

    /*
    * As we did with Storage, the Pubsub must be wired up with the emulator to provide the container host:port,
    * in this case we are overriding de auto congiguration bean initialized before PubSubTemplate
    */
    @Primary/*Override default bean*/
    @ConfigurationProperties("spring.cloud.gcp.pubsub")/*Load pubsub properties*/
    @Autowired
    @Bean
    public GcpPubSubProperties configurationProperties(PubSubEmulatorContainer pubSubEmulatorContainer) {
        System.setProperty("EMULATOR_HOST", pubSubEmulatorContainer.getEmulatorEndpoint());
        return new GcpPubSubProperties();
    }

With all these settings we are ready to run our application via command line or IntellijIdea(just consider adding an environment variable with: SPRING_PROFILES_ACTIVE=test )

Running the application

Run your application with the following command:

export SPRING_PROFILES_ACTIVE=test && ./gradlew bootRun

If everything started ok, then you will see some basic logs:

If you check your docker containers you will notice some activity:

Each individual container represents your app services, test containers will also create an additional container representing your app.

Testing with Postman/Newman

The whole purposse of all this configuration is to run our quality assurance team postman collection with newman, so let's do so:

newman run books-e2e.postman_collection.json -r html

#if you have a custom html reporter
newman run books-e2e.postman_collection.json -r html --reporter-html-template htmlreqres.hbs 

# To export test results in junit format(Good for CI/CD tools like Azure Devops)
newman run books-e2e.postman_collection.json -r junit

And that's it, we successfully runned a E2E test on a sandbox environment.

Caveats and Conclusions

The idea of a sandbox environment is to run stuff as isolated and reproducible as posible to provide better feedback on how things work on the intended infraestructure.
A sandbox is also nice for developers who does not want to have a lot of containers running all the time to test a microservice.
If you are going to use a sandbox environment for your microservices architecture, consider isolating a dependency with the bean configuration boilerplate(of course adapted to your stack :D).
Why not use JUNIT instead: As I said before, testcontainers was designed to run smoothly in junit environments, is it's natural behavior, however when we are working on a team with quality Assurance professionals, they are more familiar with tools like postman, newman, gatling etc. They are not interested on creating java code to assert basic e2e behavior.
Wiring up a successful sandbox can be difficult, and a little bit tricky, in my case I've found overriding some beans really challenging (you must know what you are touching and why, and of course ensure your production code doesn't get polluted by this beans/configurations).

As a bottom note I encourage you to play with testcontainers outside Junit with your own stack and discover what works for you.

Database Migrations for Micronaut/Spring With Liquibase

Marco Villarreal — Sun, 30 May 2021 22:44:55 +0000

Wheter you are starting on a new project or mantaining an existing codebase, mantaining database changes as controlled and reproducible as possible is a must, there are a varieity of tools for achieving this goal but today we going to focus on liquibase, an opensource(and paid) database migration/versioning tool.

Database Migration 101

A database migration(or schema migration) is a software engineering technique based on a structured and incremental version control of a database schema, can be compared to a git repository, where you can perform incremental commits adding new functionality to a codebase.

Given that in mind we are ensuring that every version(or commit) to our database can be traced and reproduced in different environments(Local development, UAT, Testing Sandbox).

Liquibase In a Nutshell

In liquibase database migrations are structured in a changelog a file to track every version of your database, usually a changelog contains relevant information about every version such as: version number, comments, author and of course the changes themselves.

Liquibase enables a variety of formats for your changelog(sql, xml, properties and yaml files) in this case and since we are using spring and micronaut, we are going to create yaml files as our changelog.

The following example is a changelog(yaml based) with sqlFiles:

databaseChangeLog:
  - changeSet:
      id: v_1_0_0
      author: "Marco Villarreal"
      comment: "A comment for your version"
      sqlFile:
        encoding: utf8
        relativeToChangelogFile: true
        stripComments: true
        path: "v_1_0_0/main-changelog.sql"

On a closer look we can identify the following properties:

comment: Description for your database version
author: Author of the current version
id: The id of the version, you are totally free to use whatever format you like
sqlFile: An object with the sql file configuration
- encoding: Encoding of the sql file
- relativeToChangelogFile: Determines if the sql file path is relative to the changelog path
- stripComments: Remove any comments from the sql file
- path: The path for the sql file itself

With this in mind we can use the following directory structure:

+--db
|   +--changelog.yaml
|   +--v_1_0_0
|   |   +--main-changelog.sql

And of course we need an initial database version.

--main-changelog.sql

CREATE TABLE author(
  author_id BIGSERIAL,
  author_name VARCHAR(200) NOT NULL,
  created_at TIMESTAMP WITHOUT TIME ZONE DEFAULT NOW(),
  updated_at TIMESTAMP WITHOUT TIME ZONE DEFAULT NOW(),
  PRIMARY KEY(author_id)
);

CREATE TABLE book(
  book_id BIGSERIAL,
  author_id BIGINT NOT NULL,
  book_isbn VARCHAR(200) NOT NULL,
  book_name TEXT NOT NULL,
  created_at TIMESTAMP WITHOUT TIME ZONE DEFAULT NOW(),
  updated_at TIMESTAMP WITHOUT TIME ZONE DEFAULT NOW(),
  PRIMARY KEY(book_id),
  FOREIGN KEY(author_id)
  REFERENCES author(author_id)
  ON UPDATE CASCADE
  ON DELETE RESTRICT,
  CONSTRAINT unq_book_isbn UNIQUE(book_isbn)
);

Note: We are not using a CREATE DATABASE statement, is a required step to have an empty database to execute a changelog.

Creating a new Database Versions

To create a new database version the following steps are required:

Add a new directory for your version

+--db
|   +--changelog.yaml
|   +--v_1_0_0
|   |   +--main-changelog.sql
|   +--v_1_0_1 # new Directory
|   |

Create a sql file for your version

+--db
|   +--changelog.yaml
|   +--v_1_0_0
|   |   +--main-changelog.sql
|   +--v_1_0_1
|   |   +--adding-comments.sql # new version file

-- adding-comments.sql
COMMENT ON COLUMN author.author_name IS 'Author name';
COMMENT ON COLUMN author.author_id IS 'Author numeric identifier';

COMMENT ON COLUMN book.book_id IS 'Book numeric identifier';
COMMENT ON COLUMN book.book_isbn IS 'Book International Standard Book Number';
COMMENT ON COLUMN book.book_name IS 'Book name';
COMMENT ON COLUMN book.author_id IS 'Author numeric identifier(author reference)';

Add a reference to your new version in the changelog

#changelog.yaml
databaseChangeLog:
  - changeSet:
      id: v_1_0_0
      author: "Marco Villarreal"
      comment: "Initial dummy library database schema"
      sqlFile:
        encoding: utf8
        relativeToChangelogFile: true
        stripComments: true
        path: "v_1_0_0/main-changelog.sql"
  - changeSet:
      id: v_1_0_1
      author: "Marco Villarreal"
      comment: | # We can use yaml multi-line syntax for more descriptive changelog comments
        * Adding comments for author's table
        * Adding comments for book's table
      sqlFile:
        encoding: utf8
        relativeToChangelogFile: true
        stripComments: true
        path: "v_1_0_1/adding-comments.sql"

Configuring automatic migrations with SpringBoot & Micronaut

Now that we have our database changelog organized and ready to go, we can implement it automatically in our java projects:

SpringBoot configuration

We need to add liquibase-core, spring data jpa(datasource connection) and postgresql driver to our project dependencies:

implementation 'org.liquibase:liquibase-core'
implementation 'org.springframework.boot:spring-boot-starter-data-jpa'
runtimeOnly 'org.postgresql:postgresql'

And enable liquibase beans in our project's configuration:

spring:
  liquibase:
    enabled: true
    change-log: "classpath:/db/changelog.yaml"

To download my setup check the spring initializr project

Micronaut configuration

We need to add micronaut-liquibase and micronaut-data jpa(datasource connection)

implementation("io.micronaut.liquibase:micronaut-liquibase")
implementation("io.micronaut.data:micronaut-data-hibernate-jpa")

liquibase:
  enabled: true
  datasources:
    default:
      change-log: 'classpath:db/changelog.yaml'

To download my setup check the micronaut launch project

Creating Database Migrations for a existing Database

If you are on a project with an existing database with no versioning at all, fear not, we can create a changelog with liquibase, in this case we need to have liquibase installed.

Once installed we have to execute the following steps:

Create the recomended folder structure

In this case we create the db folder, an empty changelog.yaml file and v_1_0_0 directory.

+--db
|   +--changelog.yaml
|   +--v_1_0_0
|   |

Get the changes from your existing database

liquibase \
--driver=org.postgresql.Driver \
--classpath=~/classpath/postgresql-42.2.20.jar \
--url="jdbc:postgresql://127.0.0.1/existing_database" \
--changeLogFile=db/v_1_0_0/main-changelog.postgresql.sql \
--username=postgres \
--password=casa1234 \
generateChangeLog

This command will create a sql based changelog, however since we are using yaml files to organize our changelog, we can use the generated sql file as an input for a yaml changelog.

Create a changelog

Now we create a changelog v_1_0_0 targeting the generated v_1_0_0/main-changelog.postgresql.sql file

#changelog.yaml
databaseChangeLog:
  - changeSet:
      id: v_1_0_0
      author: "Marco Villarreal"
      comment: "Inherited changelog from existing database"
      sqlFile:
        encoding: utf8
        relativeToChangelogFile: true
        stripComments: true
        path: "v_1_0_0/main-changelog.postgresql.sql"

Sync the changelog to your database

liquibase \
--driver=org.postgresql.Driver \
--classpath=~/classpath/postgresql-42.2.20.jar \
--url="jdbc:postgresql://127.0.0.1/existing_database" \
--changeLogFile=db/v_1_0_0/main-changelog.postgresql.sql \
--username=postgres \
--password=casa1234 \
changelogSync

With the executed command we successfully referenced a changelog for the existing database.

Conclusions and Caveats

Liquibase creates 2 additional tables; databasechangelog and databasechangeloglock:
- databasechangelog: Keep a track of each executed version
- databasechangeloglock: Used to lock executions and avoid conflicts at runtime.
When using liquibase, each executed version is inmutable, each version creates a hash of the file, if you modify it it will change the hash causing a corrupted changelog(we need to avoid this).
Having a versioned database is as good as having a versioned codebase, enables a clean tracking of your schema's evolution.
If you feel uncomfortable adding liquibase to your project's runtime, you can use gradle or maven plugin, or even the liquibase cli itself.
Enabling tools like liquibase can be considered a "vendor locking", this is because we are creating sql scripts with vendor specific syntax. If this is a problem for you, consider enabling jpa automatic changes(of course this has it's own caveats).

You can find the used projects in the github repository