DEV Community: sumanthshastry

Flyway Set-up and Integration with Spring Boot, a simple Hands-On

sumanthshastry — Sun, 20 Apr 2025 14:44:09 +0000

This article is about how to setup and integrate flyway migration tool to manage Database Schema in a Spring Boot application with simple setup. Here we use HSQL runtime database to minimize the setup and to have quick hands-on.

Prerequisites:

Below are the dependencies for this sample Flyway demo. Used https://start.spring.io/ to create the Spring Boot application.

JDK 17
Gradle-Groovy
HSQL Database
Spring boot Starter Web
Spring boot Starter Data JPA
Lombok
Flyway Migration

Below is the sample Project structure.

1. Create Database Baseline/V1:
Follow the below steps to integrate and test Flyway and create DB baseline.

Download/Generate the Spring boot project with prerequisites mentioned above and import it to any IDE. In this case I am using IntelliJ
build.gradle file should be similar to below.

plugins {
 id 'java'
 id 'org.springframework.boot' version '3.3.5'
 id 'io.spring.dependency-management' version '1.1.6'
}

group = 'com.shastry.flyway.demo'
version = '0.0.1-SNAPSHOT'

java {
 toolchain {
  languageVersion = JavaLanguageVersion.of(17)
 }
}

configurations {
 compileOnly {
  extendsFrom annotationProcessor
 }
}

repositories {
 mavenCentral()
}

dependencies {
 implementation 'org.springframework.boot:spring-boot-starter-data-jpa'
 implementation 'org.springframework.boot:spring-boot-starter-web'
 implementation 'org.flywaydb:flyway-core'
 implementation 'org.flywaydb:flyway-database-hsqldb'
 compileOnly 'org.projectlombok:lombok'
 runtimeOnly 'org.hsqldb:hsqldb'
 annotationProcessor 'org.projectlombok:lombok'
 testImplementation 'org.springframework.boot:spring-boot-starter-test'
 testRuntimeOnly 'org.junit.platform:junit-platform-launcher'
}

tasks.named('test') {
 useJUnitPlatform()
}

Configure application.properties or application.yml file with HSQL DB properties.

spring:
  application:
    name: FlywayDemo
  datasource:
    url: jdbc:hsqldb:mem:testdb;sql.syntax_pgs=true
  jpa:
    show-sql: true
    hibernate:
      naming.physical-strategy: org.hibernate.boot.model.naming.PhysicalNamingStrategyStandardImpl

Note:
As for this example we are using DBeaver sample database which has CamelCasing for columns, setting
spring.jpa.hibernate.naming.physical-strategy=org,hibernate.boot.model.naming.PhysicalNamingStrategyStandardImpl
will prevent convering to Snake Case while executing SQL queries from the application.

Create first flyway migration file V1__Initial_DB_tructure.sql
by following the flyway file naming convention. DBeaverprovides us a sample database if installed. For our use case I have taken couple of tables with Data. Using HSQL for easy testing.

-- Employee definition
CREATE TABLE Employee
(
    EmployeeId INTEGER  NOT NULL,
    LastName NVARCHAR(20)  NOT NULL,
    FirstName NVARCHAR(20)  NOT NULL,
    Title NVARCHAR(30),
    ReportsTo INTEGER,
    BirthDate DATETIME,
    HireDate DATETIME,
    Address NVARCHAR(70),
    City NVARCHAR(40),
    State NVARCHAR(40),
    Country NVARCHAR(40),
    PostalCode NVARCHAR(10),
    Phone NVARCHAR(24),
    Fax NVARCHAR(24),
    Email NVARCHAR(60),
    CONSTRAINT PK_Employee PRIMARY KEY  (EmployeeId),
    FOREIGN KEY (ReportsTo) REFERENCES Employee (EmployeeId)
    ON DELETE NO ACTION ON UPDATE NO ACTION
    );

CREATE UNIQUE INDEX IPK_Employee ON Employee(EmployeeId);
CREATE INDEX IFK_EmployeeReportsTo ON Employee (ReportsTo);

INSERT INTO Employee (EmployeeId,LastName,FirstName,Title,ReportsTo,BirthDate,HireDate,Address,City,State,Country,PostalCode,Phone,Fax,Email) VALUES
(1,'Adams','Andrew','General Manager',NULL,'1962-02-18 00:00:00','2002-08-14 00:00:00','11120 Jasper Ave NW','Edmonton','AB','Canada','T5K 2N1','+1 (780) 428-9482','+1 (780) 428-3457','andrew@chinookcorp.com'),
(2,'Edwards','Nancy','Sales Manager',1,'1958-12-08 00:00:00','2002-05-01 00:00:00','825 8 Ave SW','Calgary','AB','Canada','T2P 2T3','+1 (403) 262-3443','+1 (403) 262-3322','nancy@chinookcorp.com'),
(3,'Peacock','Jane','Sales Support Agent',2,'1973-08-29 00:00:00','2002-04-01 00:00:00','1111 6 Ave SW','Calgary','AB','Canada','T2P 5M5','+1 (403) 262-3443','+1 (403) 262-6712','jane@chinookcorp.com'),
(4,'Park','Margaret','Sales Support Agent',2,'1947-09-19 00:00:00','2003-05-03 00:00:00','683 10 Street SW','Calgary','AB','Canada','T2P 5G3','+1 (403) 263-4423','+1 (403) 263-4289','margaret@chinookcorp.com'),
(5,'Johnson','Steve','Sales Support Agent',2,'1965-03-03 00:00:00','2003-10-17 00:00:00','7727B 41 Ave','Calgary','AB','Canada','T3B 1Y7','1 (780) 836-9987','1 (780) 836-9543','steve@chinookcorp.com'),
(6,'Mitchell','Michael','IT Manager',1,'1973-07-01 00:00:00','2003-10-17 00:00:00','5827 Bowness Road NW','Calgary','AB','Canada','T3B 0C5','+1 (403) 246-9887','+1 (403) 246-9899','michael@chinookcorp.com'),
(7,'King','Robert','IT Staff',6,'1970-05-29 00:00:00','2004-01-02 00:00:00','590 Columbia Boulevard West','Lethbridge','AB','Canada','T1K 5N8','+1 (403) 456-9986','+1 (403) 456-8485','robert@chinookcorp.com'),
(8,'Callahan','Laura','IT Staff',6,'1968-01-09 00:00:00','2004-03-04 00:00:00','923 7 ST NW','Lethbridge','AB','Canada','T1H 1Y8','+1 (403) 467-3351','+1 (403) 467-8772','laura@chinookcorp.com');

Create sample API with Employee.java pojo, EmployeeRepo.java repo, EmployeeService.java service and EmployeeController.java controller.

package com.shastry.flyway.demo.FlywayDemo.model;

import jakarta.persistence.Entity;
import jakarta.persistence.Id;
import jakarta.persistence.Table;
import lombok.Data;
import java.time.OffsetDateTime;

@Data
@Entity
@Table
public class Employee {

    @Id
    private Integer employeeId;
    private String firstName;
    private String lastName;
    private String title;
    private Integer reportsTo;
    private OffsetDateTime birthDate;
    private OffsetDateTime hireDate;
    private String address;
    private String city;
    private String state;
    private String country;
    private String postalCode;
    private String phone;
    private String fax;
    private String email;
}

package com.shastry.flyway.demo.FlywayDemo.repo;

import com.shastry.flyway.demo.FlywayDemo.model.Employee;
import org.springframework.data.jpa.repository.JpaRepository;

public interface EmployeeRepo extends JpaRepository<Employee, Integer> {
}

package com.shastry.flyway.demo.FlywayDemo.service;

import com.shastry.flyway.demo.FlywayDemo.model.Employee;
import com.shastry.flyway.demo.FlywayDemo.repo.EmployeeRepo;
import lombok.RequiredArgsConstructor;
import org.springframework.stereotype.Service;

import java.util.List;

@Service
@RequiredArgsConstructor
public class EmployeeService {

    private final EmployeeRepo employeeRepo;

    public Employee getEmployee(Integer id) {
        return employeeRepo.findById(id).orElse(null);
    }

    public List<Employee> getAllEmployees() {
        return employeeRepo.findAll();
    }
}

package com.shastry.flyway.demo.FlywayDemo.controller;

import com.shastry.flyway.demo.FlywayDemo.model.Employee;
import com.shastry.flyway.demo.FlywayDemo.service.EmployeeService;
import lombok.RequiredArgsConstructor;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

import java.util.List;

@RestController
@RequestMapping("/api/v1/employee")
@RequiredArgsConstructor
public class EmployeeController {

    private final EmployeeService employeeService;

    @GetMapping
    public List<Employee> list() {
        return employeeService.getAllEmployees();
    }

    @GetMapping("/{employeeId}")
    public Employee get(@PathVariable Integer employeeId) {
        return employeeService.getEmployee(employeeId);
    }

}

Run the Spring boot application and verify the logs to check if the flyway v1 sql is executed during server startup. If application started successfully, below log be displayed.
Below logs mentions it successfully applied DB migration script.

2024-10-27T12:42:29.014+05:30  INFO 51719 --- [FlywayDemo] [           main] org.flywaydb.core.FlywayExecutor         : Database: jdbc:hsqldb:mem:testdb;sql.syntax_pgs=true (HSQL Database Engine 2.7)
2024-10-27T12:42:29.030+05:30  INFO 51719 --- [FlywayDemo] [           main] o.f.c.i.s.JdbcTableSchemaHistory         : Schema history table "PUBLIC"."flyway_schema_history" does not exist yet
2024-10-27T12:42:29.031+05:30  INFO 51719 --- [FlywayDemo] [           main] o.f.core.internal.command.DbValidate     : Successfully validated 1 migration (execution time 00:00.007s)
2024-10-27T12:42:29.034+05:30  INFO 51719 --- [FlywayDemo] [           main] o.f.c.i.s.JdbcTableSchemaHistory         : Creating Schema History table "PUBLIC"."flyway_schema_history" ...
2024-10-27T12:42:29.045+05:30  INFO 51719 --- [FlywayDemo] [           main] o.f.core.internal.command.DbMigrate      : Current version of schema "PUBLIC": << Empty Schema >>
2024-10-27T12:42:29.052+05:30  INFO 51719 --- [FlywayDemo] [           main] o.f.core.internal.command.DbMigrate      : Migrating schema "PUBLIC" to version "1 - Initial DB Structure"
2024-10-27T12:42:29.078+05:30  INFO 51719 --- [FlywayDemo] [           main] o.f.core.internal.command.DbMigrate      : Successfully applied 1 migration to schema "PUBLIC", now at version v1 (execution time 00:00.019s)

Test the employee endpoints to verify if the SQL in the flyway is executed and application is able to access.

sumanth.shastry ~ % curl --location 'http://localhost:8080/api/v1/employee'
[{"employeeId":1,"firstName":"Andrew","lastName":"Adams","title":"General Manager","reportsTo":null,"birthDate":"1962-02-18T00:00:00Z","hireDate":"2002-08-14T00:00:00Z","address":"11120 Jasper Ave NW","city":"Edmonton","state":"AB","country":"Canada","postalCode":"T5K 2N1","phone":"+1 (780) 428-9482","fax":"+1 (780) 428-3457","email":"andrew@chinookcorp.com"},{"employeeId":2,"firstName":"Nancy","lastName":"Edwards","title":"Sales Manager","reportsTo":1,"birthDate":"1958-12-08T00:00:00Z","hireDate":"2002-05-01T00:00:00Z","address":"825 8 Ave SW","city":"Calgary","state":"AB","country":"Canada","postalCode":"T2P 2T3","phone":"+1 (403) 262-3443","fax":"+1 (403) 262-3322","email":"nancy@chinookcorp.com"},{"employeeId":3,"firstName":"Jane","lastName":"Peacock","title":"Sales Support Agent","reportsTo":2,"birthDate":"1973-08-29T00:00:00Z","hireDate":"2002-04-01T00:00:00Z","address":"1111 6 Ave SW","city":"Calgary","state":"AB","country":"Canada","postalCode":"T2P 5M5","phone":"+1 (403) 262-3443","fax":"+1 (403) 262-6712","email":"jane@chinookcorp.com"},{"employeeId":4,"firstName":"Margaret","lastName":"Park","title":"Sales Support Agent","reportsTo":2,"birthDate":"1947-09-19T00:00:00Z","hireDate":"2003-05-03T00:00:00Z","address":"683 10 Street SW","city":"Calgary","state":"AB","country":"Canada","postalCode":"T2P 5G3","phone":"+1 (403) 263-4423","fax":"+1 (403) 263-4289","email":"margaret@chinookcorp.com"},{"employeeId":5,"firstName":"Steve","lastName":"Johnson","title":"Sales Support Agent","reportsTo":2,"birthDate":"1965-03-03T00:00:00Z","hireDate":"2003-10-17T00:00:00Z","address":"7727B 41 Ave","city":"Calgary","state":"AB","country":"Canada","postalCode":"T3B 1Y7","phone":"1 (780) 836-9987","fax":"1 (780) 836-9543","email":"steve@chinookcorp.com"},{"employeeId":6,"firstName":"Michael","lastName":"Mitchell","title":"IT Manager","reportsTo":1,"birthDate":"1973-07-01T00:00:00Z","hireDate":"2003-10-17T00:00:00Z","address":"5827 Bowness Road NW","city":"Calgary","state":"AB","country":"Canada","postalCode":"T3B 0C5","phone":"+1 (403) 246-9887","fax":"+1 (403) 246-9899","email":"michael@chinookcorp.com"},{"employeeId":7,"firstName":"Robert","lastName":"King","title":"IT Staff","reportsTo":6,"birthDate":"1970-05-29T00:00:00Z","hireDate":"2004-01-02T00:00:00Z","address":"590 Columbia Boulevard West","city":"Lethbridge","state":"AB","country":"Canada","postalCode":"T1K 5N8","phone":"+1 (403) 456-9986","fax":"+1 (403) 456-8485","email":"robert@chinookcorp.com"},{"employeeId":8,"firstName":"Laura","lastName":"Callahan","title":"IT Staff","reportsTo":6,"birthDate":"1968-01-09T00:00:00Z","hireDate":"2004-03-04T00:00:00Z","address":"923 7 ST NW","city":"Lethbridge","state":"AB","country":"Canada","postalCode":"T1H 1Y8","phone":"+1 (403) 467-3351","fax":"+1 (403) 467-8772","email":"laura@chinookcorp.com"}]%

sumanth.shastry ~ % curl --location 'http://localhost:8080/api/v1/employee/1'
{"employeeId":1,"firstName":"Andrew","lastName":"Adams","title":"General Manager","reportsTo":null,"birthDate":"1962-02-18T00:00:00Z","hireDate":"2002-08-14T00:00:00Z","address":"11120 Jasper Ave NW","city":"Edmonton","state":"AB","country":"Canada","postalCode":"T5K 2N1","phone":"+1 (780) 428-9482","fax":"+1 (780) 428-3457","email":"andrew@chinookcorp.com"}%

2. Create Flyway V2 migration:

Follow the below steps to create 2nd migration file.

Create flyway migration file V2__Create_Customer_Table.sql similar to previous section.

-- Customer definition

CREATE TABLE Customer
(
    CustomerId INTEGER  NOT NULL,
    FirstName NVARCHAR(40)  NOT NULL,
    LastName NVARCHAR(20)  NOT NULL,
    Company NVARCHAR(80),
    Address NVARCHAR(70),
    City NVARCHAR(40),
    State NVARCHAR(40),
    Country NVARCHAR(40),
    PostalCode NVARCHAR(10),
    Phone NVARCHAR(24),
    Fax NVARCHAR(24),
    Email NVARCHAR(60)  NOT NULL,
    SupportRepId INTEGER,
    CONSTRAINT PK_Customer PRIMARY KEY  (CustomerId),
    FOREIGN KEY (SupportRepId) REFERENCES Employee (EmployeeId)
    ON DELETE NO ACTION ON UPDATE NO ACTION
    );

CREATE UNIQUE INDEX IPK_Customer ON Customer(CustomerId);
CREATE INDEX IFK_CustomerSupportRepId ON Customer (SupportRepId);

INSERT INTO Customer (CustomerId,FirstName,LastName,Company,Address,City,State,Country,PostalCode,Phone,Fax,Email,SupportRepId) VALUES
(1,'Luís','Gonçalves','Embraer - Empresa Brasileira de Aeronáutica S.A.','Av. Brigadeiro Faria Lima, 2170','São José dos Campos','SP','Brazil','12227-000','+55 (12) 3923-5555','+55 (12) 3923-5566','luisg@embraer.com.br',3),
(2,'Leonie','Köhler',NULL,'Theodor-Heuss-Straße 34','Stuttgart',NULL,'Germany','70174','+49 0711 2842222',NULL,'leonekohler@surfeu.de',5),
(3,'François','Tremblay',NULL,'1498 rue Bélanger','Montréal','QC','Canada','H2G 1A7','+1 (514) 721-4711',NULL,'ftremblay@gmail.com',3),
(4,'Bjørn','Hansen',NULL,'Ullevålsveien 14','Oslo',NULL,'Norway','0171','+47 22 44 22 22',NULL,'bjorn.hansen@yahoo.no',4),
(5,'František','Wichterlová','JetBrains s.r.o.','Klanova 9/506','Prague',NULL,'Czech Republic','14700','+420 2 4172 5555','+420 2 4172 5555','frantisekw@jetbrains.com',4),
(6,'Helena','Holý',NULL,'Rilská 3174/6','Prague',NULL,'Czech Republic','14300','+420 2 4177 0449',NULL,'hholy@gmail.com',5),
(7,'Astrid','Gruber',NULL,'Rotenturmstraße 4, 1010 Innere Stadt','Vienne',NULL,'Austria','1010','+43 01 5134505',NULL,'astrid.gruber@apple.at',5),
(8,'Daan','Peeters',NULL,'Grétrystraat 63','Brussels',NULL,'Belgium','1000','+32 02 219 03 03',NULL,'daan_peeters@apple.be',4),
(9,'Kara','Nielsen',NULL,'Sønder Boulevard 51','Copenhagen',NULL,'Denmark','1720','+453 3331 9991',NULL,'kara.nielsen@jubii.dk',4),
(10,'Eduardo','Martins','Woodstock Discos','Rua Dr. Falcão Filho, 155','São Paulo','SP','Brazil','01007-010','+55 (11) 3033-5446','+55 (11) 3033-4564','eduardo@woodstock.com.br',4);
INSERT INTO Customer (CustomerId,FirstName,LastName,Company,Address,City,State,Country,PostalCode,Phone,Fax,Email,SupportRepId) VALUES
(11,'Alexandre','Rocha','Banco do Brasil S.A.','Av. Paulista, 2022','São Paulo','SP','Brazil','01310-200','+55 (11) 3055-3278','+55 (11) 3055-8131','alero@uol.com.br',5),
(12,'Roberto','Almeida','Riotur','Praça Pio X, 119','Rio de Janeiro','RJ','Brazil','20040-020','+55 (21) 2271-7000','+55 (21) 2271-7070','roberto.almeida@riotur.gov.br',3),
(13,'Fernanda','Ramos',NULL,'Qe 7 Bloco G','Brasília','DF','Brazil','71020-677','+55 (61) 3363-5547','+55 (61) 3363-7855','fernadaramos4@uol.com.br',4),
(14,'Mark','Philips','Telus','8210 111 ST NW','Edmonton','AB','Canada','T6G 2C7','+1 (780) 434-4554','+1 (780) 434-5565','mphilips12@shaw.ca',5),
(15,'Jennifer','Peterson','Rogers Canada','700 W Pender Street','Vancouver','BC','Canada','V6C 1G8','+1 (604) 688-2255','+1 (604) 688-8756','jenniferp@rogers.ca',3),
(16,'Frank','Harris','Google Inc.','1600 Amphitheatre Parkway','Mountain View','CA','USA','94043-1351','+1 (650) 253-0000','+1 (650) 253-0000','fharris@google.com',4),
(17,'Jack','Smith','Microsoft Corporation','1 Microsoft Way','Redmond','WA','USA','98052-8300','+1 (425) 882-8080','+1 (425) 882-8081','jacksmith@microsoft.com',5),
(18,'Michelle','Brooks',NULL,'627 Broadway','New York','NY','USA','10012-2612','+1 (212) 221-3546','+1 (212) 221-4679','michelleb@aol.com',3),
(19,'Tim','Goyer','Apple Inc.','1 Infinite Loop','Cupertino','CA','USA','95014','+1 (408) 996-1010','+1 (408) 996-1011','tgoyer@apple.com',3),
(20,'Dan','Miller',NULL,'541 Del Medio Avenue','Mountain View','CA','USA','94040-111','+1 (650) 644-3358',NULL,'dmiller@comcast.com',4);
INSERT INTO Customer (CustomerId,FirstName,LastName,Company,Address,City,State,Country,PostalCode,Phone,Fax,Email,SupportRepId) VALUES
(21,'Kathy','Chase',NULL,'801 W 4th Street','Reno','NV','USA','89503','+1 (775) 223-7665',NULL,'kachase@hotmail.com',5),
(22,'Heather','Leacock',NULL,'120 S Orange Ave','Orlando','FL','USA','32801','+1 (407) 999-7788',NULL,'hleacock@gmail.com',4),
(23,'John','Gordon',NULL,'69 Salem Street','Boston','MA','USA','2113','+1 (617) 522-1333',NULL,'johngordon22@yahoo.com',4),
(24,'Frank','Ralston',NULL,'162 E Superior Street','Chicago','IL','USA','60611','+1 (312) 332-3232',NULL,'fralston@gmail.com',3),
(25,'Victor','Stevens',NULL,'319 N. Frances Street','Madison','WI','USA','53703','+1 (608) 257-0597',NULL,'vstevens@yahoo.com',5),
(26,'Richard','Cunningham',NULL,'2211 W Berry Street','Fort Worth','TX','USA','76110','+1 (817) 924-7272',NULL,'ricunningham@hotmail.com',4),
(27,'Patrick','Gray',NULL,'1033 N Park Ave','Tucson','AZ','USA','85719','+1 (520) 622-4200',NULL,'patrick.gray@aol.com',4),
(28,'Julia','Barnett',NULL,'302 S 700 E','Salt Lake City','UT','USA','84102','+1 (801) 531-7272',NULL,'jubarnett@gmail.com',5),
(29,'Robert','Brown',NULL,'796 Dundas Street West','Toronto','ON','Canada','M6J 1V1','+1 (416) 363-8888',NULL,'robbrown@shaw.ca',3),
(30,'Edward','Francis',NULL,'230 Elgin Street','Ottawa','ON','Canada','K2P 1L7','+1 (613) 234-3322',NULL,'edfrancis@yachoo.ca',3);
INSERT INTO Customer (CustomerId,FirstName,LastName,Company,Address,City,State,Country,PostalCode,Phone,Fax,Email,SupportRepId) VALUES
(31,'Martha','Silk',NULL,'194A Chain Lake Drive','Halifax','NS','Canada','B3S 1C5','+1 (902) 450-0450',NULL,'marthasilk@gmail.com',5),
(32,'Aaron','Mitchell',NULL,'696 Osborne Street','Winnipeg','MB','Canada','R3L 2B9','+1 (204) 452-6452',NULL,'aaronmitchell@yahoo.ca',4),
(33,'Ellie','Sullivan',NULL,'5112 48 Street','Yellowknife','NT','Canada','X1A 1N6','+1 (867) 920-2233',NULL,'ellie.sullivan@shaw.ca',3),
(34,'João','Fernandes',NULL,'Rua da Assunção 53','Lisbon',NULL,'Portugal',NULL,'+351 (213) 466-111',NULL,'jfernandes@yahoo.pt',4),
(35,'Madalena','Sampaio',NULL,'Rua dos Campeões Europeus de Viena, 4350','Porto',NULL,'Portugal',NULL,'+351 (225) 022-448',NULL,'masampaio@sapo.pt',4),
(36,'Hannah','Schneider',NULL,'Tauentzienstraße 8','Berlin',NULL,'Germany','10789','+49 030 26550280',NULL,'hannah.schneider@yahoo.de',5),
(37,'Fynn','Zimmermann',NULL,'Berger Straße 10','Frankfurt',NULL,'Germany','60316','+49 069 40598889',NULL,'fzimmermann@yahoo.de',3),
(38,'Niklas','Schröder',NULL,'Barbarossastraße 19','Berlin',NULL,'Germany','10779','+49 030 2141444',NULL,'nschroder@surfeu.de',3),
(39,'Camille','Bernard',NULL,'4, Rue Milton','Paris',NULL,'France','75009','+33 01 49 70 65 65',NULL,'camille.bernard@yahoo.fr',4),
(40,'Dominique','Lefebvre',NULL,'8, Rue Hanovre','Paris',NULL,'France','75002','+33 01 47 42 71 71',NULL,'dominiquelefebvre@gmail.com',4);
INSERT INTO Customer (CustomerId,FirstName,LastName,Company,Address,City,State,Country,PostalCode,Phone,Fax,Email,SupportRepId) VALUES
(41,'Marc','Dubois',NULL,'11, Place Bellecour','Lyon',NULL,'France','69002','+33 04 78 30 30 30',NULL,'marc.dubois@hotmail.com',5),
(42,'Wyatt','Girard',NULL,'9, Place Louis Barthou','Bordeaux',NULL,'France','33000','+33 05 56 96 96 96',NULL,'wyatt.girard@yahoo.fr',3),
(43,'Isabelle','Mercier',NULL,'68, Rue Jouvence','Dijon',NULL,'France','21000','+33 03 80 73 66 99',NULL,'isabelle_mercier@apple.fr',3),
(44,'Terhi','Hämäläinen',NULL,'Porthaninkatu 9','Helsinki',NULL,'Finland','00530','+358 09 870 2000',NULL,'terhi.hamalainen@apple.fi',3),
(45,'Ladislav','Kovács',NULL,'Erzsébet krt. 58.','Budapest',NULL,'Hungary','H-1073',NULL,NULL,'ladislav_kovacs@apple.hu',3),
(46,'Hugh','O''Reilly',NULL,'3 Chatham Street','Dublin','Dublin','Ireland',NULL,'+353 01 6792424',NULL,'hughoreilly@apple.ie',3),
(47,'Lucas','Mancini',NULL,'Via Degli Scipioni, 43','Rome','RM','Italy','00192','+39 06 39733434',NULL,'lucas.mancini@yahoo.it',5),
(48,'Johannes','Van der Berg',NULL,'Lijnbaansgracht 120bg','Amsterdam','VV','Netherlands','1016','+31 020 6223130',NULL,'johavanderberg@yahoo.nl',5),
(49,'Stanisław','Wójcik',NULL,'Ordynacka 10','Warsaw',NULL,'Poland','00-358','+48 22 828 37 39',NULL,'stanisław.wójcik@wp.pl',4),
(50,'Enrique','Muñoz',NULL,'C/ San Bernardo 85','Madrid',NULL,'Spain','28015','+34 914 454 454',NULL,'enrique_munoz@yahoo.es',5);
INSERT INTO Customer (CustomerId,FirstName,LastName,Company,Address,City,State,Country,PostalCode,Phone,Fax,Email,SupportRepId) VALUES
(51,'Joakim','Johansson',NULL,'Celsiusg. 9','Stockholm',NULL,'Sweden','11230','+46 08-651 52 52',NULL,'joakim.johansson@yahoo.se',5),
(52,'Emma','Jones',NULL,'202 Hoxton Street','London',NULL,'United Kingdom','N1 5LH','+44 020 7707 0707',NULL,'emma_jones@hotmail.com',3),
(53,'Phil','Hughes',NULL,'113 Lupus St','London',NULL,'United Kingdom','SW1V 3EN','+44 020 7976 5722',NULL,'phil.hughes@gmail.com',3),
(54,'Steve','Murray',NULL,'110 Raeburn Pl','Edinburgh ',NULL,'United Kingdom','EH4 1HH','+44 0131 315 3300',NULL,'steve.murray@yahoo.uk',5),
(55,'Mark','Taylor',NULL,'421 Bourke Street','Sidney','NSW','Australia','2010','+61 (02) 9332 3633',NULL,'mark.taylor@yahoo.au',4),
(56,'Diego','Gutiérrez',NULL,'307 Macacha Güemes','Buenos Aires',NULL,'Argentina','1106','+54 (0)11 4311 4333',NULL,'diego.gutierrez@yahoo.ar',4),
(57,'Luis','Rojas',NULL,'Calle Lira, 198','Santiago',NULL,'Chile',NULL,'+56 (0)2 635 4444',NULL,'luisrojas@yahoo.cl',5),
(58,'Manoj','Pareek',NULL,'12,Community Centre','Delhi',NULL,'India','110017','+91 0124 39883988',NULL,'manoj.pareek@rediff.com',3),
(59,'Puja','Srivastava',NULL,'3,Raj Bhavan Road','Bangalore',NULL,'India','560001','+91 080 22289999',NULL,'puja_srivastava@yahoo.in',3);

Create Customer.java pojo, CustomerRepo.java repo, CustomerService.java service and CustomerController.java controller.

package com.shastry.flyway.demo.FlywayDemo.model;

import jakarta.persistence.Entity;
import jakarta.persistence.Id;
import jakarta.persistence.Table;
import lombok.Data;

@Data
@Entity
@Table
public class Customer {

    @Id
    private Integer customerId;
    private String firstName;
    private String lastName;
    private String company;
    private String address;
    private String city;
    private String state;
    private String country;
    private String postalCode;
    private String phone;
    private String fax;
    private String email;
    private Integer supportRepId;
}

package com.shastry.flyway.demo.FlywayDemo.repo;

import com.shastry.flyway.demo.FlywayDemo.model.Customer;
import org.springframework.data.jpa.repository.JpaRepository;

public interface CustomerRepo extends JpaRepository<Customer, Integer> {
}

package com.shastry.flyway.demo.FlywayDemo.service;

import com.shastry.flyway.demo.FlywayDemo.model.Customer;
import com.shastry.flyway.demo.FlywayDemo.repo.CustomerRepo;
import lombok.RequiredArgsConstructor;
import org.springframework.stereotype.Service;

import java.util.List;

@Service
@RequiredArgsConstructor
public class CustomerService {

    private final CustomerRepo customerRepo;

    public Customer get(Integer id) {
        return customerRepo.findById(id).orElse(null);
    }

    public List<Customer> getAllCustomers() {
        return customerRepo.findAll();
    }
}

package com.shastry.flyway.demo.FlywayDemo.controller;

import com.shastry.flyway.demo.FlywayDemo.model.Customer;
import com.shastry.flyway.demo.FlywayDemo.service.CustomerService;
import lombok.RequiredArgsConstructor;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

import java.util.List;

@RestController
@RequestMapping("/api/v1/customer")
@RequiredArgsConstructor
public class CustomerController {

    private final CustomerService customerService;

    @GetMapping("/{customerId}")
    public Customer get(@PathVariable Integer customerId) {
        return customerService.get(customerId);
    }

    @GetMapping
    public List<Customer> getAll() {
        return customerService.getAllCustomers();
    }
}

Run the Spring boot application and verify the logs to check if the flyway v1 and v2 sql are executed during server startup. If application started successfully, below log be displayed.

2024-10-27T12:42:29.014+05:30  INFO 51719 --- [FlywayDemo] [           main] org.flywaydb.core.FlywayExecutor         : Database: jdbc:hsqldb:mem:testdb;sql.syntax_pgs=true (HSQL Database Engine 2.7)
2024-10-27T12:56:23.039+05:30  INFO 54805 --- [FlywayDemo] [           main] o.f.c.i.s.JdbcTableSchemaHistory         : Schema history table "PUBLIC"."flyway_schema_history" does not exist yet
2024-10-27T12:56:23.040+05:30  INFO 54805 --- [FlywayDemo] [           main] o.f.core.internal.command.DbValidate     : Successfully validated 2 migrations (execution time 00:00.006s)
2024-10-27T12:56:23.043+05:30  INFO 54805 --- [FlywayDemo] [           main] o.f.c.i.s.JdbcTableSchemaHistory         : Creating Schema History table "PUBLIC"."flyway_schema_history" ...
2024-10-27T12:56:23.054+05:30  INFO 54805 --- [FlywayDemo] [           main] o.f.core.internal.command.DbMigrate      : Current version of schema "PUBLIC": << Empty Schema >>
2024-10-27T12:56:23.077+05:30  INFO 54805 --- [FlywayDemo] [           main] o.f.core.internal.command.DbMigrate      : Migrating schema "PUBLIC" to version "1 - Initial DB Structure"
2024-10-27T12:56:23.098+05:30  INFO 54805 --- [FlywayDemo] [           main] o.f.core.internal.command.DbMigrate      : Migrating schema "PUBLIC" to version "2 - Create Customer Table"
2024-10-27T12:56:23.104+05:30  INFO 54805 --- [FlywayDemo] [           main] o.f.core.internal.command.DbMigrate      : Successfully applied 2 migrations to schema "PUBLIC", now at version v2 (execution time 00:00.004s)

Test the employee endpoints to verify if the SQL in the flyway is executed and application is able to access.

sumanth.shastry ~ % curl --location 'http://localhost:8080/api/v1/customer/1'
{"customerId":1,"firstName":"Luís","lastName":"Gonçalves","company":"Embraer - Empresa Brasileira de Aeronáutica S.A.","address":"Av. Brigadeiro Faria Lima, 2170","city":"São José dos Campos","state":"SP","country":"Brazil","postalCode":"12227-000","phone":"+55 (12) 3923-5555","fax":"+55 (12) 3923-5566","email":"luisg@embraer.com.br","supportRepId":3}%

Conclusion:

In this article, we learned a simple, easy hands-on and integrating Flyway migration with Spring boot.

Find the respective code over the Github.

Quartz Scheduler setup, integration with Spring Boot, deploy using kubernates, a simple Hands-on

sumanthshastry — Sun, 13 Apr 2025 14:41:10 +0000

This article is about how to setup and integrate quartz scheduler in a Spring Boot application with simple setup, deploy using Kubernates.

Quartz Scheduler is an open- source Scheduling framework for Java. Below are some of the main features of Quartz which will be focused in this article.

Job Schedule/Execution/Persistence
Clustering.

Prerequisites:

Below are the dependencies for this demo. Used https://start.spring.io/ to create the Spring Boot application.

JDK 17
Gradle-Groovy
Quartz Scheduler
PostgreSQL Driver.(Using it to persist Job info)
Lombok

Below is the project folder structure.

Code Setup:

Follow the below steps for code setup..

Download/Generate the Spring boot project with prerequisites mentioned above and import it to any IDE. In this case I am using IntelliJ
build.gradlefile should be similar to below.

plugins {
 id 'java'
 id 'org.springframework.boot' version '3.4.4'
 id 'io.spring.dependency-management' version '1.1.7'
}

group = 'com.shastry.learning.quartz'
version = '0.0.1-SNAPSHOT'

java {
 toolchain {
  languageVersion = JavaLanguageVersion.of(17)
 }
}

configurations {
 compileOnly {
  extendsFrom annotationProcessor
 }
}

repositories {
 mavenCentral()
}

dependencies {
 implementation 'org.springframework.boot:spring-boot-starter-quartz'
 compileOnly 'org.projectlombok:lombok'
 runtimeOnly 'org.postgresql:postgresql'
 annotationProcessor 'org.projectlombok:lombok'
 testImplementation 'org.springframework.boot:spring-boot-starter-test'
 testRuntimeOnly 'org.junit.platform:junit-platform-launcher'
}

tasks.named('test') {
 useJUnitPlatform()
}

Create couple of jobs which will be executed by Quartz. @DisallowConcurrentExecution will prevent running concurrent execution of job. Meaning only one Job of this type be executed at any time. @PersistJobDataAfterExecution will persist the Job data once execution is completed.

package com.shastry.learning.quartz.QuartzDemo.scheduled;

import lombok.extern.slf4j.Slf4j;
import org.quartz.DisallowConcurrentExecution;
import org.quartz.Job;
import org.quartz.JobExecutionContext;
import org.quartz.JobExecutionException;
import org.quartz.PersistJobDataAfterExecution;

import java.time.OffsetDateTime;

@Slf4j
@DisallowConcurrentExecution
@PersistJobDataAfterExecution
public class OneMinuteJob implements Job {

    /**
     * @param context
     * @throws JobExecutionException
     */
    @Override
    public void execute(JobExecutionContext context) throws JobExecutionException {
        log.info("Executing OneMinuteJob at {}", OffsetDateTime.now().toInstant());
    }
}

package com.shastry.learning.quartz.QuartzDemo.scheduled;

import lombok.extern.slf4j.Slf4j;
import org.quartz.DisallowConcurrentExecution;
import org.quartz.Job;
import org.quartz.JobExecutionContext;
import org.quartz.JobExecutionException;
import org.quartz.PersistJobDataAfterExecution;

import java.time.OffsetDateTime;

@Slf4j
@DisallowConcurrentExecution
@PersistJobDataAfterExecution
public class ThirtySecondJob implements Job {

    /**
     * @param context
     * @throws JobExecutionException
     */
    @Override
    public void execute(JobExecutionContext context) throws JobExecutionException {
        log.info("Executing ThirtySecondJob at {}", OffsetDateTime.now().toInstant());
    }
}

Configure AutowiringSpringBeanFactory This config ensures Quartz to create job instances using Spring’s context with required dependencies.

package com.shastry.learning.quartz.QuartzDemo.config;

import org.quartz.spi.TriggerFiredBundle;
import org.springframework.beans.factory.config.AutowireCapableBeanFactory;
import org.springframework.context.ApplicationContext;
import org.springframework.context.ApplicationContextAware;
import org.springframework.scheduling.quartz.SpringBeanJobFactory;

public final class AutowiringSpringBeanJobFactory extends SpringBeanJobFactory implements ApplicationContextAware {

    private transient AutowireCapableBeanFactory beanFactory;

    @Override
    public void setApplicationContext(final ApplicationContext context) {
        beanFactory = context.getAutowireCapableBeanFactory();
    }

    @Override
    protected Object createJobInstance(final TriggerFiredBundle bundle) throws Exception {
        final Object job = super.createJobInstance(bundle);
        beanFactory.autowireBean(job);
        return job;
    }

}

Configure Quartz Scheduler. SchedulerConfig configures Jobs, triggers and schedulers. Different type of triggers are used for different jobs

Simple Trigger: This can be used to fire jobs at regular interval of time repeatedly. The trigger can be configured to start after some fixed delay. Here 30s job is configured using SimpleTrigger.
Cron Trigger: This can be used for creating triggers based on a cron expression. Here 1min job is configured using CRON expression.

package com.shastry.learning.quartz.QuartzDemo.config;

import com.shastry.learning.quartz.QuartzDemo.scheduled.OneMinuteJob;
import com.shastry.learning.quartz.QuartzDemo.scheduled.ThirtySecondJob;
import lombok.AllArgsConstructor;
import lombok.Data;
import org.quartz.Job;
import org.quartz.JobDetail;
import org.quartz.Trigger;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.boot.autoconfigure.quartz.QuartzDataSourceScriptDatabaseInitializer;
import org.springframework.boot.autoconfigure.quartz.QuartzProperties;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.DependsOn;
import org.springframework.jdbc.datasource.DataSourceTransactionManager;
import org.springframework.scheduling.quartz.CronTriggerFactoryBean;
import org.springframework.scheduling.quartz.JobDetailFactoryBean;
import org.springframework.scheduling.quartz.SchedulerFactoryBean;
import org.springframework.scheduling.quartz.SimpleTriggerFactoryBean;
import org.springframework.stereotype.Component;

import javax.sql.DataSource;
import java.util.List;
import java.util.Properties;

@Configuration
@AllArgsConstructor
public class SchedulerConfig {

    @Component
    @Data
    @ConfigurationProperties("app.scheduling")
    public static class SchedulerConfigProperties {

        @Data
        public static class ScheduledJobTimingConfig {
            private String cron;
            private int startDelayMillis;
            private int fixedDelay;
        }

        private ScheduledJobTimingConfig oneMinuteJob;
        private ScheduledJobTimingConfig thirtySecondJob;

    }

    private SchedulerConfigProperties schedulerConfigProperties;
    private final QuartzProperties quartzProperties;

    // --------------------- Jobs -----------------------------

    @Bean("oneMinuteJob")
    public JobDetailFactoryBean oneMinuteJob() {
        return createJobDetail(OneMinuteJob.class, "One Minute Job");
    }

    @Bean("thirtySecondJob")
    public JobDetailFactoryBean thirtySecondJob() {
        return createJobDetail(ThirtySecondJob.class, "Thirty Second Job");
    }

    // --------------------- Triggers  ------------------------

    @Bean("thirtySecondJobTrigger")
    public SimpleTriggerFactoryBean thirtySecondJobTrigger(@Qualifier("thirtySecondJob") JobDetail jobDetail) {
        var trigger = new SimpleTriggerFactoryBean();
        trigger.setJobDetail(jobDetail);
        trigger.setName(jobDetail.getDescription());
        trigger.setStartDelay(schedulerConfigProperties.getThirtySecondJob().getStartDelayMillis());
        trigger.setRepeatInterval(schedulerConfigProperties.getThirtySecondJob().getFixedDelay());
        return trigger;
    }

    @Bean("oneMinuteJobTrigger")
    public CronTriggerFactoryBean oneMinuteJobTrigger(@Qualifier("oneMinuteJob") JobDetail jobDetail) {
        return createCronTrigger(jobDetail,
            schedulerConfigProperties.getOneMinuteJob().getCron(),
            schedulerConfigProperties.getOneMinuteJob().getStartDelayMillis());
    }

    @Bean
    @DependsOn("quartzDataSourceInitializer")
    public SchedulerFactoryBean schedulerFactoryBean(
        List<JobDetail> jobDetails,
        List<Trigger> triggers,
        DataSource dataSource,
        DataSourceTransactionManager transactionManager) {

        var props = new Properties();
        var factory = new SchedulerFactoryBean();

        props.putAll(quartzProperties.getProperties());
        factory.setQuartzProperties(props);
        factory.setDataSource(dataSource);
        factory.setTransactionManager(transactionManager);
        factory.setJobFactory(autowiringSpringBeanJobFactory());
        factory.setJobDetails(jobDetails.toArray(new JobDetail[0]));
        factory.setTriggers(triggers.toArray(new Trigger[0]));
        factory.setAutoStartup(true);
        factory.setOverwriteExistingJobs(true);

        return factory;
    }

    @Bean
    public QuartzDataSourceScriptDatabaseInitializer quartzDataSourceInitializer(DataSource dataSource) {
        return new QuartzDataSourceScriptDatabaseInitializer(dataSource, quartzProperties);
    }

    @Bean
    public AutowiringSpringBeanJobFactory autowiringSpringBeanJobFactory() {
        return new AutowiringSpringBeanJobFactory();
    }


    private <T extends Job> JobDetailFactoryBean createJobDetail(Class<T> jobClass, String jobName) {
        JobDetailFactoryBean jobDetailFactoryBean = new JobDetailFactoryBean();
        jobDetailFactoryBean.setJobClass(jobClass);
        jobDetailFactoryBean.setDescription(jobName);
        jobDetailFactoryBean.setDurability(true);
        return jobDetailFactoryBean;
    }

    private CronTriggerFactoryBean createCronTrigger(JobDetail jobDetail, String cron, int startDelayMillis) {
        CronTriggerFactoryBean cronTriggerFactoryBean = new CronTriggerFactoryBean();
        cronTriggerFactoryBean.setJobDetail(jobDetail);
        cronTriggerFactoryBean.setCronExpression(cron);
        cronTriggerFactoryBean.setStartDelay(startDelayMillis);
        cronTriggerFactoryBean.setName(jobDetail.getDescription());
        return cronTriggerFactoryBean;
    }
}

Create application.yml to configure db, quartz and schedulers.

spring:
  application:
    name=QuartzDemo
  datasource:
    url: jdbc:postgresql://quartz-postgres-service:5432/postgres
    username: postgres
    password: postgres
    driver-class-name: org.postgresql.Driver
  jpa:
    show-sql: false
    hibernate:
      ddl-auto: update
      dialect: org.hibernate.dialect.PostgreSQLDialect 
  quartz:
    job-store-type: jdbc
    jdbc:
      initialize-schema: always
    properties:
      org:
        quartz:
          jobStore:
            clusterCheckinInterval: 20000
            misfireThreshold: 60000
            driverDelegateClass: org.quartz.impl.jdbcjobstore.PostgreSQLDelegate #Postgre specific
          scheduler:
            instanceName: quartz-demo
            instanceId: AUTO
          threadPool:
            threadCount: 10
            threadPriority: 5
            class: org.quartz.simpl.SimpleThreadPool

app:
  scheduling:
    oneMinuteJob:
      cron: "0 0/1 * * * ? *" # Cron expression to run every minute
      start-delay-millis: 10000 # Start the job after the configured delay
    thirtySecondJob:
      fixed-delay: 30000 # Run/Repeat the job every 30 seconds
      start-delay-millis: 20000

Note:
postgresql hostnames are configured in kubernates config which will be covered in the deployment section.

Deployment:

Generate Dockerfile

# Java 17 base image for the app
FROM openjdk:17-jdk-slim-buster
LABEL authors="sumanth.shastry"

# Setup base dir for the app to run
WORKDIR /app

# Copy the app jar
COPY build/libs/QuartzDemo*.jar /app/quartz-demo.jar

ENTRYPOINT ["java", "-jar", "/app/quartz-demo.jar"]

Create Persistent Volume and claim for PostgreSQL

apiVersion: v1
kind: PersistentVolume
metadata:
  name: quartz-postgres-pv
spec:
  capacity:
    storage: 2Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/mnt/data/quartz-postgres"
---

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: quartz-postgres-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi

Create ConfigMap to configure PostgreSQL DB details

apiVersion: v1
kind: ConfigMap
metadata:
  name: quartz-postgres-config
data:
  POSTGRES_DB: "postgres"
  POSTGRES_USER: "postgres"
  POSTGRES_PASSWORD: "postgres"
Create services for PostgreSQL and spring boot app.
apiVersion: v1
kind: Service
metadata:
  name: quartz-postgres-service
spec:
  selector:
    app: quartz-postgres-service
  ports:
    - protocol: TCP
      port: 5432
      targetPort: 5432
      nodePort: 31079
  type: NodePort
---
apiVersion: v1
kind: Service
metadata:
  name: quartz-demo
spec:
  selector:
    app: quartz-demo
  ports:
    - port: 8080
      targetPort: 8080
      nodePort: 30080
      protocol: TCP
  type: NodePort

Create deployment forPostgreSQL and spring boot app.

Note: Created 2 replicas for Spring Boot app to test Load Balancing and no overlap/concurrent jobs running in separete instances.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: quartz-postgres-service
  labels:
    app: quartz-postgres-service
spec:
  replicas: 1
  selector:
    matchLabels:
      app: quartz-postgres-service
  template:
    metadata:
      name: quartz-postgres-service
      labels:
        app: quartz-postgres-service
    spec:
      containers:
        - name: quartz-postgres-service
          image: postgres:latest
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 5432
              protocol: TCP
          envFrom:
            - configMapRef:
                name: quartz-postgres-config
          volumeMounts:
            - mountPath: /var/lib/postgresql/data
              name: postgres-storage
      volumes:
        - name: postgres-storage
          persistentVolumeClaim:
              claimName: quartz-postgres-pvc
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: quartz-demo
  labels:
    app: quartz-demo
spec:
  replicas: 2
  selector:
    matchLabels:
      app: quartz-demo
  template:
    metadata:
      name: quartz-demo
      labels:
          app: quartz-demo
    spec:
      containers:
        - name: quartz-demo
          image: quartz-demo:0.0.1-SNAPSHOT
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 8080
              protocol: TCP

Create deployment script to build spring boot, create docker image create PostgreSQL using above k8s config.

#!/bin/bash

APP_NAME="quartz-demo"
VERSION="0.0.1-SNAPSHOT"
DOCKER_IMAGE="${APP_NAME}:${VERSION}"
NAMESPACE="quartz-namespace"

# Build the application
echo "Building the application..."
./gradlew clean build

# Generate Docker image
echo "Building Docker image..."
docker build -t ${DOCKER_IMAGE} .

# Clean up Kubernetes deployment
echo "Cleaning up Kubernetes deployment..."
kubectl delete -f k8s/quartz-demo-configmap.yaml -n ${NAMESPACE}
kubectl delete -f k8s/quartz-demo-service.yaml -n ${NAMESPACE}
kubectl delete -f k8s/quartz-demo-deployment.yaml -n ${NAMESPACE}
kubectl delete -f k8s/quartz-demo-pv.yaml -n ${NAMESPACE}
kubectl delete namespace ${NAMESPACE}

# Apply Kubernetes configurations
echo "Deploying application to Kubernetes..."
kubectl create namespace ${NAMESPACE}
kubectl apply -f k8s/quartz-demo-configmap.yaml -n ${NAMESPACE}
kubectl apply -f k8s/quartz-demo-pv.yaml -n ${NAMESPACE}
kubectl apply -f k8s/quartz-demo-service.yaml -n ${NAMESPACE}
kubectl apply -f k8s/quartz-demo-deployment.yaml -n ${NAMESPACE}

echo "Deployment complete."

Testing:

Once build script is executed, 2 instances would be running. Below snapshot of logs shows same job is not executed 2 instances. Jobs execution is load balanced.

Sample log:

2025-03-28T06:59:00.006Z  INFO 1 --- [-demo_Worker-10] c.s.l.q.Q.scheduled.OneMinuteJob         : Executing OneMinuteJob at 2025-03-28T06:59:00.006097128Z
2025-03-28T07:00:05.766Z  INFO 1 --- [z-demo_Worker-1] c.s.l.q.Q.scheduled.ThirtySecondJob      : Executing ThirtySecondJob at 2025-03-28T07:00:05.766212130Z

Conclusion:

In this article, we learned a simple, easy hands-on related to integrating Quartz framework with Spring boot.

Find the respective code over the GitHub.

References:

https://github.com/quartz-scheduler/quartz/blob/main/docs/introduction.adoc

Install Kafka+Zookeeper using Docker container in Windows Subsystem for Linux(WSL)

sumanthshastry — Sun, 06 Apr 2025 16:24:38 +0000

The article is about how to install Single Node multiple broker setup of Kafka and Zookeeper in Windows subsystem for Linux(WSL). Previous article has installation guide for windows. This helps to simulate installing Kafka and Zookeeper in Linux from scratch.

Installation

1. Install Windows subsystem for Linux in Windows machine

The document contains step-by-step process for installation of ubuntu https://learn.microsoft.com/en-us/windows/wsl/install
Click on search bar and type ubuntu and open the app.
Setup username and password in the ubuntu app.

C:\Users\suman>wsl --install
Ubuntu is already installed.
Launching Ubuntu...
Installing, this may take a few minutes...
Please create a default UNIX user account. The username does not need to match your Windows username.
For more information visit: https://aka.ms/wslusers
Enter new UNIX username: sumanthshastry
New password:
Retype new password:
passwd: password updated successfully
The operation completed successfully.
Installation successful!
To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.

Welcome to Ubuntu 22.04.3 LTS (GNU/Linux 5.15.133.1-microsoft-standard-WSL2 x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage


This message is shown once a day. To disable it please create the
/home/sumanthshastry/.hushlogin file.
sumanthshastry@shastry:~$

2. Install required software's

Update Advanced Package tool apt-get update

root@shastry:~# apt-get update
Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease
Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB]
Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB]
Get:4 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1205 kB]
Get:5 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB]
Get:6 http://archive.ubuntu.com/ubuntu jammy/universe amd64 Packages [14.1 MB]
Get:7 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [219 kB]
Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1476 kB]
Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [244 kB]
Get:10 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [846 kB]
Get:11 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [161 kB]
Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB]
Get:13 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.1 kB]
Get:14 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7476 B]
Get:15 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B]
Get:16 http://archive.ubuntu.com/ubuntu jammy/universe Translation-en [5652 kB]
Get:17 http://archive.ubuntu.com/ubuntu jammy/universe amd64 c-n-f Metadata [286 kB]
Get:18 http://archive.ubuntu.com/ubuntu jammy/multiverse amd64 Packages [217 kB]
Get:19 http://archive.ubuntu.com/ubuntu jammy/multiverse Translation-en [112 kB]
Get:20 http://archive.ubuntu.com/ubuntu jammy/multiverse amd64 c-n-f Metadata [8372 B]
Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1421 kB]
Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [279 kB]
Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1504 kB]
Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [247 kB]
Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1052 kB]
Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [237 kB]
Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB]
Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.1 kB]
Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.1 kB]
Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B]
Get:31 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [41.7 kB]
Get:32 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [10.5 kB]
Get:33 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B]
Get:34 http://archive.ubuntu.com/ubuntu jammy-backports/restricted amd64 c-n-f Metadata [116 B]
Get:35 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [24.3 kB]
Get:36 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB]
Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B]
Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/multiverse amd64 c-n-f Metadata [116 B]
Fetched 29.8 MB in 7s (4478 kB/s)
Reading package lists... Done
root@shastry:~#
Install docker apt-get install docker
root@shastry:~# apt-get install docker
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
  wmdocker
The following NEW packages will be installed:
  docker wmdocker
0 upgraded, 2 newly installed, 0 to remove and 79 not upgraded.
Need to get 14.3 kB of archives.
After this operation, 58.4 kB of additional disk space will be used.
Do you want to continue? [Y/n] Y
Get:1 http://archive.ubuntu.com/ubuntu jammy/universe amd64 wmdocker amd64 1.5-2 [13.0 kB]
Get:2 http://archive.ubuntu.com/ubuntu jammy/universe amd64 docker all 1.5-2 [1316 B]
Fetched 14.3 kB in 1s (28.4 kB/s)
Selecting previously unselected package wmdocker.
(Reading database ... 24208 files and directories currently installed.)
Preparing to unpack .../wmdocker_1.5-2_amd64.deb ...
Unpacking wmdocker (1.5-2) ...
Selecting previously unselected package docker.
Preparing to unpack .../archives/docker_1.5-2_all.deb ...
Unpacking docker (1.5-2) ...
Setting up wmdocker (1.5-2) ...
Setting up docker (1.5-2) ...
Processing triggers for man-db (2.10.2-1) ...
root@shastry:~#

Install docker compose apt-get install docker-compose

root@shastry:~# apt-get install docker-compose
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
  bridge-utils containerd dns-root-data dnsmasq-base docker.io pigz python3-attr python3-certifi python3-chardet python3-distutils python3-docker python3-dockerpty python3-docopt python3-dotenv python3-idna
  python3-jsonschema python3-lib2to3 python3-pyrsistent python3-requests python3-setuptools python3-texttable python3-urllib3 python3-websocket runc ubuntu-fan
Suggested packages:
  ifupdown aufs-tools btrfs-progs cgroupfs-mount | cgroup-lite debootstrap docker-doc rinse zfs-fuse | zfsutils python-attr-doc python-jsonschema-doc python3-openssl python3-socks python-requests-doc
  python-setuptools-doc
The following NEW packages will be installed:
  bridge-utils containerd dns-root-data dnsmasq-base docker-compose docker.io pigz python3-attr python3-certifi python3-chardet python3-distutils python3-docker python3-dockerpty python3-docopt
  python3-dotenv python3-idna python3-jsonschema python3-lib2to3 python3-pyrsistent python3-requests python3-setuptools python3-texttable python3-urllib3 python3-websocket runc ubuntu-fan
0 upgraded, 26 newly installed, 0 to remove and 79 not upgraded.
Need to get 71.2 MB of archives.
After this operation, 274 MB of additional disk space will be used.
Do you want to continue? [Y/n] Y
Get:1 http://archive.ubuntu.com/ubuntu jammy/universe amd64 pigz amd64 2.6-1 [63.6 kB]
Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 bridge-utils amd64 1.7-1ubuntu3 [34.4 kB]
Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 runc amd64 1.1.7-0ubuntu1~22.04.2 [4267 kB]
Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 containerd amd64 1.7.2-0ubuntu1~22.04.1 [36.0 MB]
Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 dns-root-data all 2023112702~ubuntu0.22.04.1 [5136 B]
Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 dnsmasq-base amd64 2.90-0ubuntu0.22.04.1 [374 kB]
Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-lib2to3 all 3.10.8-1~22.04 [77.6 kB]
Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-distutils all 3.10.8-1~22.04 [139 kB]
Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB]
Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-chardet all 4.0.0-1 [98.0 kB]
Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB]
Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB]
Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB]
Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB]
Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-docker all 5.0.3-1 [89.3 kB]
Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-dockerpty all 0.4.1-2 [11.1 kB]
Get:17 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-docopt all 0.6.2-4 [26.9 kB]
Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-dotenv all 0.19.2-1 [20.5 kB]
Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-attr all 21.2.0-1 [44.0 kB]
Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-setuptools all 59.6.0-1.2ubuntu0.22.04.1 [339 kB]
Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyrsistent amd64 0.18.1-1build1 [55.5 kB]
Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jsonschema all 3.2.0-0ubuntu2 [43.1 kB]
Get:23 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-texttable all 1.6.4-1 [11.4 kB]
Get:24 http://archive.ubuntu.com/ubuntu jammy/universe amd64 docker-compose all 1.29.2-1 [95.8 kB]
Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 docker.io amd64 24.0.5-0ubuntu1~22.04.1 [28.9 MB]
Get:26 http://archive.ubuntu.com/ubuntu jammy/universe amd64 ubuntu-fan all 0.12.16 [35.2 kB]
Fetched 71.2 MB in 9s (7698 kB/s)
Preconfiguring packages ...
Selecting previously unselected package pigz.
(Reading database ... 40597 files and directories currently installed.)
Preparing to unpack .../00-pigz_2.6-1_amd64.deb ...
Unpacking pigz (2.6-1) ...
Selecting previously unselected package bridge-utils.
Preparing to unpack .../01-bridge-utils_1.7-1ubuntu3_amd64.deb ...
Unpacking bridge-utils (1.7-1ubuntu3) ...
Selecting previously unselected package runc.
Preparing to unpack .../02-runc_1.1.7-0ubuntu1~22.04.2_amd64.deb ...
Unpacking runc (1.1.7-0ubuntu1~22.04.2) ...
Selecting previously unselected package containerd.
Preparing to unpack .../03-containerd_1.7.2-0ubuntu1~22.04.1_amd64.deb ...
Unpacking containerd (1.7.2-0ubuntu1~22.04.1) ...
Selecting previously unselected package dns-root-data.
Preparing to unpack .../04-dns-root-data_2023112702~ubuntu0.22.04.1_all.deb ...
Unpacking dns-root-data (2023112702~ubuntu0.22.04.1) ...
Selecting previously unselected package dnsmasq-base.
Preparing to unpack .../05-dnsmasq-base_2.90-0ubuntu0.22.04.1_amd64.deb ...
Unpacking dnsmasq-base (2.90-0ubuntu0.22.04.1) ...
Selecting previously unselected package python3-lib2to3.
Preparing to unpack .../06-python3-lib2to3_3.10.8-1~22.04_all.deb ...
Unpacking python3-lib2to3 (3.10.8-1~22.04) ...
Selecting previously unselected package python3-distutils.
Preparing to unpack .../07-python3-distutils_3.10.8-1~22.04_all.deb ...
Unpacking python3-distutils (3.10.8-1~22.04) ...
Selecting previously unselected package python3-certifi.
Preparing to unpack .../08-python3-certifi_2020.6.20-1_all.deb ...
Unpacking python3-certifi (2020.6.20-1) ...
Selecting previously unselected package python3-chardet.
Preparing to unpack .../09-python3-chardet_4.0.0-1_all.deb ...
Unpacking python3-chardet (4.0.0-1) ...
Selecting previously unselected package python3-idna.
Preparing to unpack .../10-python3-idna_3.3-1_all.deb ...
Unpacking python3-idna (3.3-1) ...
Selecting previously unselected package python3-urllib3.
Preparing to unpack .../11-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ...
Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ...
Selecting previously unselected package python3-requests.
Preparing to unpack .../12-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ...
Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ...
Selecting previously unselected package python3-websocket.
Preparing to unpack .../13-python3-websocket_1.2.3-1_all.deb ...
Unpacking python3-websocket (1.2.3-1) ...
Selecting previously unselected package python3-docker.
Preparing to unpack .../14-python3-docker_5.0.3-1_all.deb ...
Unpacking python3-docker (5.0.3-1) ...
Selecting previously unselected package python3-dockerpty.
Preparing to unpack .../15-python3-dockerpty_0.4.1-2_all.deb ...
Unpacking python3-dockerpty (0.4.1-2) ...
Selecting previously unselected package python3-docopt.
Preparing to unpack .../16-python3-docopt_0.6.2-4_all.deb ...
Unpacking python3-docopt (0.6.2-4) ...
Selecting previously unselected package python3-dotenv.
Preparing to unpack .../17-python3-dotenv_0.19.2-1_all.deb ...
Unpacking python3-dotenv (0.19.2-1) ...
Selecting previously unselected package python3-attr.
Preparing to unpack .../18-python3-attr_21.2.0-1_all.deb ...
Unpacking python3-attr (21.2.0-1) ...
Selecting previously unselected package python3-setuptools.
Preparing to unpack .../19-python3-setuptools_59.6.0-1.2ubuntu0.22.04.1_all.deb ...
Unpacking python3-setuptools (59.6.0-1.2ubuntu0.22.04.1) ...
Selecting previously unselected package python3-pyrsistent:amd64.
Preparing to unpack .../20-python3-pyrsistent_0.18.1-1build1_amd64.deb ...
Unpacking python3-pyrsistent:amd64 (0.18.1-1build1) ...
Selecting previously unselected package python3-jsonschema.
Preparing to unpack .../21-python3-jsonschema_3.2.0-0ubuntu2_all.deb ...
Unpacking python3-jsonschema (3.2.0-0ubuntu2) ...
Selecting previously unselected package python3-texttable.
Preparing to unpack .../22-python3-texttable_1.6.4-1_all.deb ...
Unpacking python3-texttable (1.6.4-1) ...
Selecting previously unselected package docker-compose.
Preparing to unpack .../23-docker-compose_1.29.2-1_all.deb ...
Unpacking docker-compose (1.29.2-1) ...
Selecting previously unselected package docker.io.
Preparing to unpack .../24-docker.io_24.0.5-0ubuntu1~22.04.1_amd64.deb ...
Unpacking docker.io (24.0.5-0ubuntu1~22.04.1) ...
Selecting previously unselected package ubuntu-fan.
Preparing to unpack .../25-ubuntu-fan_0.12.16_all.deb ...
Unpacking ubuntu-fan (0.12.16) ...
Setting up python3-dotenv (0.19.2-1) ...
Setting up python3-attr (21.2.0-1) ...
Setting up python3-texttable (1.6.4-1) ...
Setting up python3-docopt (0.6.2-4) ...
Setting up dnsmasq-base (2.90-0ubuntu0.22.04.1) ...
Setting up runc (1.1.7-0ubuntu1~22.04.2) ...
Setting up dns-root-data (2023112702~ubuntu0.22.04.1) ...
Setting up python3-chardet (4.0.0-1) ...
Setting up python3-certifi (2020.6.20-1) ...
Setting up python3-idna (3.3-1) ...
Setting up bridge-utils (1.7-1ubuntu3) ...
Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ...
Setting up python3-pyrsistent:amd64 (0.18.1-1build1) ...
Setting up pigz (2.6-1) ...
Setting up containerd (1.7.2-0ubuntu1~22.04.1) ...
Created symlink /etc/systemd/system/multi-user.target.wants/containerd.service → /lib/systemd/system/containerd.service.
Setting up python3-lib2to3 (3.10.8-1~22.04) ...
Setting up python3-websocket (1.2.3-1) ...
Setting up python3-dockerpty (0.4.1-2) ...
Setting up python3-distutils (3.10.8-1~22.04) ...
Setting up ubuntu-fan (0.12.16) ...
Created symlink /etc/systemd/system/multi-user.target.wants/ubuntu-fan.service → /lib/systemd/system/ubuntu-fan.service.
Setting up python3-setuptools (59.6.0-1.2ubuntu0.22.04.1) ...
Setting up docker.io (24.0.5-0ubuntu1~22.04.1) ...
Adding group `docker' (GID 117) ...
Done.
Created symlink /etc/systemd/system/multi-user.target.wants/docker.service → /lib/systemd/system/docker.service.
Created symlink /etc/systemd/system/sockets.target.wants/docker.socket → /lib/systemd/system/docker.socket.
Setting up python3-jsonschema (3.2.0-0ubuntu2) ...
Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ...
Setting up python3-docker (5.0.3-1) ...
Setting up docker-compose (1.29.2-1) ...
Processing triggers for dbus (1.12.20-2ubuntu4.1) ...
Processing triggers for man-db (2.10.2-1) ...
root@shastry:~#

Install OpenJDK 17 apt-get install openjdk-17-jdk

root@shastry:~# apt-get install openjdk-17-jdk
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
  adwaita-icon-theme alsa-topology-conf alsa-ucm-conf at-spi2-core ca-certificates-java dconf-gsettings-backend dconf-service fontconfig fontconfig-config fonts-dejavu-core fonts-dejavu-extra
  gsettings-desktop-schemas gtk-update-icon-cache hicolor-icon-theme humanity-icon-theme java-common libasound2 libasound2-data libatk-bridge2.0-0 libatk-wrapper-java libatk-wrapper-java-jni libatk1.0-0
  libatk1.0-data libatspi2.0-0 libavahi-client3 libavahi-common-data libavahi-common3 libcairo-gobject2 libcairo2 libcups2 libdatrie1 libdconf1 libdeflate0 libdrm-amdgpu1 libdrm-intel1 libdrm-nouveau2
  libdrm-radeon1 libfontconfig1 libfontenc1 libfreetype6 libgail-common libgail18 libgdk-pixbuf-2.0-0 libgdk-pixbuf2.0-bin libgdk-pixbuf2.0-common libgif7 libgl1 libgl1-amber-dri libgl1-mesa-dri
  libglapi-mesa libglvnd0 libglx-mesa0 libglx0 libgraphite2-3 libgtk2.0-0 libgtk2.0-bin libgtk2.0-common libharfbuzz0b libice-dev libice6 libjbig0 libjpeg-turbo8 libjpeg8 liblcms2-2 libllvm15 libnspr4
  libnss3 libpango-1.0-0 libpangocairo-1.0-0 libpangoft2-1.0-0 libpciaccess0 libpcsclite1 libpixman-1-0 libpthread-stubs0-dev librsvg2-2 librsvg2-common libsensors-config libsensors5 libsm-dev libsm6
  libthai-data libthai0 libtiff5 libwebp7 libx11-dev libx11-xcb1 libxau-dev libxaw7 libxcb-dri2-0 libxcb-dri3-0 libxcb-glx0 libxcb-present0 libxcb-randr0 libxcb-render0 libxcb-shape0 libxcb-shm0 libxcb-sync1
  libxcb-xfixes0 libxcb1-dev libxcomposite1 libxcursor1 libxdamage1 libxdmcp-dev libxfixes3 libxft2 libxi6 libxinerama1 libxkbfile1 libxmu6 libxpm4 libxrandr2 libxrender1 libxshmfence1 libxt-dev libxt6
  libxtst6 libxv1 libxxf86dga1 libxxf86vm1 openjdk-17-jdk-headless openjdk-17-jre openjdk-17-jre-headless session-migration ubuntu-mono x11-common x11-utils x11proto-dev xorg-sgml-doctools xtrans-dev
 ...
root@shastry:~#

3. Create kafka folder and docker compose file

mkdir /opt/kafka

cd /opt/kafka

4. Create Docker compose file

Copy contents of docker-compose.yml from git and save it

root@shastry:/opt/kafka# pwd
/opt/kafka
root@shastry:/opt/kafka# cat docker-compose.yml
version: '2.1'

services:
  zookeep1:
    image: zookeeper:3.4.14
    hostname: zookeep1
    ports:
      - "2181:2181"
    environment:
        ZOO_MY_ID: 1
        ZOO_PORT: 2181
        ZOO_SERVERS: server.1=zookeep1:2888:3888 server.2=zookeep2:2888:3888 server.3=zookeep3:2888:3888
    volumes:
      - ./kafka-windows-docker-data/zookeep1/data:/data
      - ./kafka-windows-docker-data/zookeep1/datalog:/datalog

  zookeep2:
    image: zookeeper:3.4.14
    hostname: zookeep2
    ports:
      - "2182:2182"
    environment:
        ZOO_MY_ID: 2
        ZOO_PORT: 2182
        ZOO_SERVERS: server.1=zookeep1:2888:3888 server.2=zookeep2:2888:3888 server.3=zookeep3:2888:3888
    volumes:
      - ./kafka-windows-docker-data/zookeep2/data:/data
      - ./kafka-windows-docker-data/zookeep2/datalog:/datalog

  zookeep3:
    image: zookeeper:3.4.14
    hostname: zookeep3
    ports:
      - "2183:2183"
    environment:
        ZOO_MY_ID: 3
        ZOO_PORT: 2183
        ZOO_SERVERS: server.1=zookeep1:2888:3888 server.2=zookeep2:2888:3888 server.3=zookeep3:2888:3888
    volumes:
      - ./kafka-windows-docker-data/zookeep3/data:/data
      - ./kafka-windows-docker-data/zookeep3/datalog:/datalog


  kafka1:
    image: confluentinc/cp-kafka:7.5.0
    user: root
    hostname: kafka1
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_LISTENERS: LISTENER_DOCKER_INTERNAL://kafka1:19092,LISTENER_DOCKER_EXTERNAL://${DOCKER_HOST_IP:-127.0.0.1}:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER_DOCKER_INTERNAL:PLAINTEXT,LISTENER_DOCKER_EXTERNAL:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_DOCKER_INTERNAL
      KAFKA_ZOOKEEPER_CONNECT: "zookeep1:2181,zookeep2:2182,zookeep3:2183"
      KAFKA_BROKER_ID: 1
      KAFKA_LOG4J_LOGGERS: "kafka.controller=INFO,kafka.producer.async.DefaultEventHandler=INFO,state.change.logger=INFO"
    volumes:
      - ./kafka-windows-docker-data/kafka1/data:/var/lib/kafka/data
    depends_on:
      - zookeep1
      - zookeep2
      - zookeep3

  kafka2:
    image: confluentinc/cp-kafka:7.5.0
    user: root
    hostname: kafka2
    ports:
      - "9093:9093"
    environment:
      KAFKA_ADVERTISED_LISTENERS: LISTENER_DOCKER_INTERNAL://kafka2:19093,LISTENER_DOCKER_EXTERNAL://${DOCKER_HOST_IP:-127.0.0.1}:9093
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER_DOCKER_INTERNAL:PLAINTEXT,LISTENER_DOCKER_EXTERNAL:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_DOCKER_INTERNAL
      KAFKA_ZOOKEEPER_CONNECT: "zookeep1:2181,zookeep2:2182,zookeep3:2183"
      KAFKA_BROKER_ID: 2
      KAFKA_LOG4J_LOGGERS: "kafka.controller=INFO,kafka.producer.async.DefaultEventHandler=INFO,state.change.logger=INFO"
    volumes:
      - ./kafka-windows-docker-data/kafka2/data:/var/lib/kafka/data
    depends_on:
      - zookeep1
      - zookeep2
      - zookeep3

  kafka3:
    image: confluentinc/cp-kafka:7.5.0
    user: root
    hostname: kafka3
    ports:
      - "9094:9094"
    environment:
      KAFKA_ADVERTISED_LISTENERS: LISTENER_DOCKER_INTERNAL://kafka3:19094,LISTENER_DOCKER_EXTERNAL://${DOCKER_HOST_IP:-127.0.0.1}:9094
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER_DOCKER_INTERNAL:PLAINTEXT,LISTENER_DOCKER_EXTERNAL:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_DOCKER_INTERNAL
      KAFKA_ZOOKEEPER_CONNECT: "zookeep1:2181,zookeep2:2182,zookeep3:2183"
      KAFKA_BROKER_ID: 3
      KAFKA_LOG4J_LOGGERS: "kafka.controller=INFO,kafka.producer.async.DefaultEventHandler=INFO,state.change.logger=INFO"
    volumes:
      - ./kafka-windows-docker-data/kafka3/data:/var/lib/kafka/data
    depends_on:
      - zookeep1
      - zookeep2
      - zookeep3
root@shastry:/opt/kafka#

5. Download kafka client

Download kafka client from http://mirror.cogentco.com/pub/apache/kafka/3.6.1/kafka_2.12-3.6.1.tgz

root@shastry:/opt/kafka# wget http://mirror.cogentco.com/pub/apache/kafka/3.6.1/kafka_2.12-3.6.1.tgz
--2024-03-03 21:03:58--  http://mirror.cogentco.com/pub/apache/kafka/3.6.1/kafka_2.12-3.6.1.tgz
Resolving mirror.cogentco.com (mirror.cogentco.com)... 204.157.3.70
Connecting to mirror.cogentco.com (mirror.cogentco.com)|204.157.3.70|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 113609072 (108M) [application/x-gzip]
Saving to: ‘kafka_2.12-3.6.1.tgz’

kafka_2.12-3.6.1.tgz                                 100%[===================================================================================================================>] 108.35M  2.14MB/s    in 89s

2024-03-03 21:05:28 (1.21 MB/s) - ‘kafka_2.12-3.6.1.tgz’ saved [113609072/113609072]

root@shastry:/opt/kafka#

untar downloaded kafka client

root@shastry:/opt/kafka# ls -ltr
total 110952
-rw-r--r-- 1 root root 113609072 Dec  5 22:23 kafka_2.12-3.6.1.tgz
-rw-r--r-- 1 root root      3611 Mar  3 21:01 docker-compose.yml
root@shastry:/opt/kafka# tar -xf kafka_2.12-3.6.1.tgz
root@shastry:/opt/kafka# ls -ltr
total 110956
drwxr-xr-x 7 root root      4096 Nov 24 15:13 kafka_2.12-3.6.1
-rw-r--r-- 1 root root 113609072 Dec  5 22:23 kafka_2.12-3.6.1.tgz
-rw-r--r-- 1 root root      3611 Mar  3 21:01 docker-compose.yml
root@shastry:/opt/kafka# mv kafka_2.12-3.6.1 kafka_cli
root@shastry:/opt/kafka# ls -ltr
total 110956
drwxr-xr-x 7 root root      4096 Nov 24 15:13 kafka_cli
-rw-r--r-- 1 root root 113609072 Dec  5 22:23 kafka_2.12-3.6.1.tgz
-rw-r--r-- 1 root root      3611 Mar  3 21:01 docker-compose.yml
root@shastry:/opt/kafka#

6. Deploy kafka and zookeeper

docker-compose up -d --build

root@shastry:/opt/kafka# docker-compose up -d --build
Creating network "kafka_default" with the default driver
Pulling zookeep1 (zookeeper:3.4.14)...
3.4.14: Pulling from library/zookeeper
69692152171a: Pull complete
ce2b89b60818: Pull complete
a3c211c6bdc9: Pull complete
4c25606a3064: Pull complete
d7d77cede5c8: Pull complete
b2a5fc6a5ffe: Pull complete
49bd81bc3707: Pull complete
b6b7d3138b59: Pull complete
Digest: sha256:3882d9493d387ba77b7b69e2a031b9396477ec29483d51ceaed645c1389182e5
Status: Downloaded newer image for zookeeper:3.4.14
Pulling kafka1 (confluentinc/cp-kafka:7.5.0)...
7.5.0: Pulling from confluentinc/cp-kafka
57168402cb72: Pull complete
6edc125f768c: Pull complete
6086b74cc48f: Pull complete
04b3375cab83: Pull complete
f80eb2db9d48: Pull complete
95d7897d98ce: Pull complete
bd3de482b201: Pull complete
0fbb79a570de: Pull complete
8592ceb925f7: Pull complete
be07a4b85081: Pull complete
210860798c03: Pull complete
Digest: sha256:fbbb6fa11b258a88b83f54d4f0bddfcffbf2279f99d66a843486e3da7bdfbf41
Status: Downloaded newer image for confluentinc/cp-kafka:7.5.0
Creating kafka_zookeep3_1 ... done
Creating kafka_zookeep2_1 ... done
Creating kafka_zookeep1_1 ... done
Creating kafka_kafka1_1   ... done
Creating kafka_kafka3_1   ... done
Creating kafka_kafka2_1   ... done
root@shastry:/opt/kafka# docker-compose ps

      Name                    Command               State                                   Ports
----------------------------------------------------------------------------------------------------------------------------------
kafka_kafka1_1     /etc/confluent/docker/run        Up      0.0.0.0:9092->9092/tcp,:::9092->9092/tcp
kafka_kafka2_1     /etc/confluent/docker/run        Up      9092/tcp, 0.0.0.0:9093->9093/tcp,:::9093->9093/tcp
kafka_kafka3_1     /etc/confluent/docker/run        Up      9092/tcp, 0.0.0.0:9094->9094/tcp,:::9094->9094/tcp
kafka_zookeep1_1   /docker-entrypoint.sh zkSe ...   Up      0.0.0.0:2181->2181/tcp,:::2181->2181/tcp, 2888/tcp, 3888/tcp
kafka_zookeep2_1   /docker-entrypoint.sh zkSe ...   Up      2181/tcp, 0.0.0.0:2182->2182/tcp,:::2182->2182/tcp, 2888/tcp, 3888/tcp
kafka_zookeep3_1   /docker-entrypoint.sh zkSe ...   Up      2181/tcp, 0.0.0.0:2183->2183/tcp,:::2183->2183/tcp, 2888/tcp, 3888/tcp
root@shastry:/opt/kafka#

7. Create kafka topics

Create new topic with topicName “test” with 3 partitions and 3 replication-factor

root@shastry:/opt/kafka/kafka_cli/bin# ./kafka-topics.sh --bootstrap-server localhost:9092 --topic test --create --partitions 3 --replication-factor 3
Created topic test.
root@shastry:/opt/kafka/kafka_cli/bin# ./kafka-topics.sh --bootstrap-server localhost:9092 --topic test1 --create --partitions 3 --replication-factor 3
Created topic test1.
root@shastry:/opt/kafka/kafka_cli/bin#

List down the topics

root@shastry:/opt/kafka/kafka_cli/bin# ./kafka-topics.sh --bootstrap-server localhost:9092 --list
test
test1
root@shastry:/opt/kafka/kafka_cli/bin#

Describe a specific topic

root@shastry:/opt/kafka/kafka_cli/bin# ./kafka-topics.sh --bootstrap-server localhost:9092 --topic test1 --describe
Topic: test1    TopicId: Lp-rUh0bTi22G_5pbSzTAA PartitionCount: 3       ReplicationFactor: 3    Configs:
        Topic: test1    Partition: 0    Leader: 1       Replicas: 1,2,3 Isr: 1,2,3
        Topic: test1    Partition: 1    Leader: 2       Replicas: 2,3,1 Isr: 2,3,1
        Topic: test1    Partition: 2    Leader: 3       Replicas: 3,1,2 Isr: 3,1,2
root@shastry:/opt/kafka/kafka_cli/bin#

8. Produce and consume messages

Produce few messages

root@shastry:/opt/kafka/kafka_cli/bin# ./kafka-console-producer.sh --broker-list localhost:9092 --topic test1
>Hello
>from
>Bharath
>

Consume messages from beginning.

root@shastry:/opt/kafka/kafka_cli/bin# ./kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test1 --from-beginning
Hello
from
Bharath

Reference Links

https://learn.microsoft.com/en-us/windows/wsl/install
http://kafka.apache.org/documentation.html
https://zookeeper.apache.org/doc/trunk/zookeeperStarted.html
https://www.conduktor.io/kafka/how-to-start-kafka-using-docker/
https://docs.confluent.io/platform/current/installation/versions-interoperability.html#zk

Install Kafka+Zookeeper using Docker container in Windows

sumanthshastry — Mon, 31 Mar 2025 16:10:18 +0000

The article is about how to install Single Node multiple broker setup of Kafka and Zookeeper. This process is faster compared to installing kafka and zookeeper directly in Windows machine as both installation and configuration of kafka brokers and zookeeper are built-in using docker.

The article is useful for development and testing in windows machine.

Software's required

Open Jdk-17 for windows https://www.oracle.com/in/java/technologies/javase/jdk11-archive-downloads.html
Docker for windows https://docs.docker.com/desktop/install/windows-install/
Kafka Client http://mirror.cogentco.com/pub/apache/kafka/3.7.0/kafka_2.12-3.7.0.tgz

Installation

1. Setup JDK 17 and configure PATH variables

Skip this Step if JAVA is already installed.
Open Windows Explorer, Right click on “This PC” and select Properties. Click on Advanced System Properties => Environment Variables.
In System Variables section click on New Button and configure JAVA_HOME = C:\jdk-17

Configure PATH variable. Select PATH and click on Edit. Add %JAVA_HOME%/bin

Verify if the configuration is successfull. Open Command prompt and execute java -version

C:\Users\suman>java -version
openjdk version "17" 2021-09-14
OpenJDK Runtime Environment (build 17+35-2724)
OpenJDK 64-Bit Server VM (build 17+35-2724, mixed mode, sharing)

2. Install Docker

Download latest docker from https://docs.docker.com/desktop/install/windows-install/
Follow normal installation steps to install docker desktop

3. Create docker-compose.yml file

Download docker-compose.yml file from Git https://github.com/sumanthshastry/kafka-windows-docker
Install kafka and zookeeper as per the configuration in docker file docker-compose up -d — build

E:\kafka\kafka-windows-docker>docker-compose up -d --build
[+] Running 25/25
 ✔ zookeep3 Pulled                                                                                                20.6s
 ✔ zookeep1 Pulled                                                                                                20.6s
 ✔ kafka1 Pulled                                                                                                  63.1s
 ✔ zookeep2 8 layers [⣿⣿⣿⣿⣿⣿⣿⣿]      0B/0B      Pulled                                                            20.6s
   ✔ 69692152171a Pull complete                                                                                   11.2s
   ✔ ce2b89b60818 Pull complete                                                                                    1.2s
   ✔ a3c211c6bdc9 Pull complete                                                                                    0.9s
   ✔ 4c25606a3064 Pull complete                                                                                   12.2s
   ✔ d7d77cede5c8 Pull complete                                                                                    2.2s
   ✔ b2a5fc6a5ffe Pull complete                                                                                    4.2s
   ✔ 49bd81bc3707 Pull complete                                                                                    9.5s
   ✔ b6b7d3138b59 Pull complete                                                                                   10.3s
 ✔ kafka3 Pulled                                                                                                  63.1s
 ✔ kafka2 11 layers [⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿]      0B/0B      Pulled                                                          63.1s
   ✔ 57168402cb72 Pull complete                                                                                   27.7s
   ✔ 6edc125f768c Pull complete                                                                                   48.4s
   ✔ 6086b74cc48f Pull complete                                                                                   16.0s
   ✔ 04b3375cab83 Pull complete                                                                                   17.2s
   ✔ f80eb2db9d48 Pull complete                                                                                   23.1s
   ✔ 95d7897d98ce Pull complete                                                                                   24.3s
   ✔ bd3de482b201 Pull complete                                                                                   25.5s
   ✔ 0fbb79a570de Pull complete                                                                                   26.8s
   ✔ 8592ceb925f7 Pull complete                                                                                   27.9s
   ✔ be07a4b85081 Pull complete                                                                                   41.1s
   ✔ 210860798c03 Pull complete                                                                                   29.0s
[+] Running 6/7
 - Network kafka-windows-docker_default       Created                                                              2.6s
 ✔ Container kafka-windows-docker-zookeep3-1  Started                                                              1.4s
 ✔ Container kafka-windows-docker-zookeep2-1  Started                                                              1.6s
 ✔ Container kafka-windows-docker-zookeep1-1  Started                                                              1.7s
 ✔ Container kafka-windows-docker-kafka3-1    Started                                                              1.5s
 ✔ Container kafka-windows-docker-kafka1-1    Started                                                              1.6s
 ✔ Container kafka-windows-docker-kafka2-1    Started                                                              1.8s

E:\kafka\kafka-windows-docker>docker-compose ps
NAME                              IMAGE                         COMMAND                  SERVICE    CREATED          STATUS          PORTS
kafka-windows-docker-kafka1-1     confluentinc/cp-kafka:7.5.0   "/etc/confluent/dock…"   kafka1     34 seconds ago   Up 31 seconds   0.0.0.0:9092->9092/tcp
kafka-windows-docker-kafka2-1     confluentinc/cp-kafka:7.5.0   "/etc/confluent/dock…"   kafka2     34 seconds ago   Up 31 seconds   9092/tcp, 0.0.0.0:9093->9093/tcp
kafka-windows-docker-kafka3-1     confluentinc/cp-kafka:7.5.0   "/etc/confluent/dock…"   kafka3     34 seconds ago   Up 32 seconds   9092/tcp, 0.0.0.0:9094->9094/tcp
kafka-windows-docker-zookeep1-1   zookeeper:3.4.14              "/docker-entrypoint.…"   zookeep1   35 seconds ago   Up 32 seconds   2888/tcp, 0.0.0.0:2181->2181/tcp, 3888/tcp
kafka-windows-docker-zookeep2-1   zookeeper:3.4.14              "/docker-entrypoint.…"   zookeep2   35 seconds ago   Up 32 seconds   2181/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:2182->2182/tcp
kafka-windows-docker-zookeep3-1   zookeeper:3.4.14              "/docker-entrypoint.…"   zookeep3   35 seconds ago   Up 32 seconds   2181/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:2183->2183/tcp

E:\kafka\kafka-windows-docker>

4. Install Kafka Client

Download Kafka Client from http://mirror.cogentco.com/pub/apache/kafka/3.7.0/kafka_2.12-3.7.0.tgz
Unzip the file and navigate to bin/windows folder.
Kafka Zookeeper compatibility matrix provides which versions are compatible

5. Create Topic

Create new topic with topicName “test” with 3 partitions and 3 replication-factor

E:\kafka_cli\bin\windows>kafka-topics.bat  --bootstrap-server localhost:9092 --topic test --create --partitions 3 --replication-factor 3

Created topic test.

E:\kafka_cli\bin\windows>kafka-topics.bat  --bootstrap-server localhost:9092 --topic test1 --create --partitions 3 --replication-factor 3
Created topic test1.

List down the topics

E:\kafka_cli\bin\windows>kafka-topics.bat --bootstrap-server localhost:9092 --list
test
test1

Describe a specific topic

E:\kafka_cli\bin\windows>kafka-topics.bat --bootstrap-server localhost:9092 --topic test --describe
Topic: test     TopicId: NT5ydxyeTSaGFYmrH0xNaA PartitionCount: 3       ReplicationFactor: 3    Configs:
        Topic: test     Partition: 0    Leader: 1       Replicas: 1,2,3 Isr: 1,2,3
        Topic: test     Partition: 1    Leader: 2       Replicas: 2,3,1 Isr: 2,3,1
        Topic: test     Partition: 2    Leader: 3       Replicas: 3,1,2 Isr: 3,1,2

6. Produce and consume messages

Produce few messages

E:\kafka_cli\bin\windows>kafka-console-producer.bat --broker-list localhost:9092 --topic test
>Hello
>from
>Bharath
>

Consume messages from beginning

E:\kafka_cli\bin\windows>kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic test --from-beginning
Hello
from
Bharath

Reference Links

http://kafka.apache.org/documentation.html
https://zookeeper.apache.org/doc/trunk/zookeeperStarted.html
https://www.conduktor.io/kafka/how-to-start-kafka-using-docker/
https://docs.confluent.io/platform/current/installation/versions-interoperability.html#zk