DEV Community: Ed Legaspi

I Got Tired of Rewriting Audit Logs in Spring Boot — So I Built nerv-audit

Ed Legaspi — Fri, 17 Apr 2026 07:18:23 +0000

Every backend system eventually hits this moment:

“Who changed this record?”
“What was the previous value?”
“When did it happen?”

Simple questions… until you actually need answers in production.

⚠️ Note: The core module is commercial, but you can explore the public modules and examples here:
👉 https://github.com/czetsuyatech/nerv-audit
👉 https://github.com/czetsuyatech/nerv-examples

The Problem

In most of my Spring Boot projects, I relied on Hibernate Envers.

It works—but in real systems, it starts to hurt:

You repeat the same setup across services
Audit queries are hard to read and maintain
Business-level audit logic gets scattered
Small mistakes become painful in production

After a few projects, I realized:

I wasn’t building features anymore—I was rebuilding audit infrastructure.

What I Actually Wanted

Not a replacement for Envers.

Just something that:

Standardizes audit handling
Reduces boilerplate
Makes queries readable
Works consistently across projects

So I Built nerv-audit

nerv-audit is a lightweight layer on top of Envers that focuses on developer experience.

Instead of wiring everything manually, you get a cleaner way to work with audit data.

Example

Without nerv-audit

You end up dealing with low-level Envers APIs and custom query logic.

With nerv-audit

service.getVerticalAudits(entity, criteria);

That’s it.

Want to see how it's implemented in a real project?
👉 https://github.com/czetsuyatech/nerv-audit
👉 https://github.com/czetsuyatech/nerv-examples

What It Improves

1. Consistent Audit Handling

Define audit behavior once, reuse everywhere.

@AuditedEntity
public class Order {
    private String status;
}

2. Cleaner Queries

No more complex Envers query construction.

You focus on:

What changed
When it changed

Not how to retrieve it.

3. Less Repetition

Across projects, the pattern is always the same.

nerv-audit abstracts that pattern so you don’t rewrite it every time.

Why I Built This

This came from real production work:

Multiple systems
Repeated audit requirements
Real incidents where audit data mattered

At some point, it made more sense to abstract the solution than keep rebuilding it.

About the Model (Transparency)

nerv-audit is not fully open-source.

There is a free tier you can use
Some advanced capabilities are part of a paid core

I chose this approach to:

Keep the project sustainable
Continue improving it over time
Focus on real-world use cases

Who This Is For

This might be useful if you:

Build Spring Boot applications
Use Hibernate Envers (or plan to)
Want a cleaner way to handle audit logs

I’d Like Your Input

How are you handling audit logs today?

Pure Envers?
Custom implementation?
Something else?

I’m especially interested in:

Pain points
Query challenges
Scaling issues

If you're working with Envers, try it out and let me know what breaks—or what works better.

🔗 Resources

Hands-On Coding: Exploring Hyperparameters for Programmers

Ed Legaspi — Thu, 07 Mar 2024 04:54:31 +0000

Introduction

In this article, we will explore different techniques for finding the optimal hyperparameter values from a given set of parameters in a grid. Particularly we will look at RandomizedSearchCV, GridSearchCV, and BayesSearchCV.

In this blog you will learn:

How to initialize the parameter grid.
How to find the optimal hyperparameters based on a given technique.
How to build a model (XGBClassifier) to use the hyperparameters.
How to score the performance of the model.

RandomizedSearchCV

param_grid = {
    "gamma": [0, 0.1, 0.2, 0.5, 1, 1.5, 2, 3, 6, 12, 20],
    "learning_rate": [0.01, 0.02, 0.03, 0.05, 0.1, 0.2, 0.3, 0.5, 0.7, 0.8],
    "max_depth": [1, 2, 3, 4, 5, 6, 8, 12],
    "n_estimators": [25, 50, 65, 80, 100, 115, 200]
}

grid_search = RandomizedSearchCV(estimator=classifier_0, param_distributions=param_grid, scoring=scoring)

GridSearchCV

param_grid = {
    "gamma": [0, 0.1, 0.2, 0.5, 1, 1.5, 2, 3, 6, 12, 20],
    "learning_rate": [0.01, 0.02, 0.03, 0.05, 0.1, 0.2, 0.3, 0.5, 0.7, 0.8],
    "max_depth": [2, 3, 4, 5, 6, 8, 12],
    "n_estimators": [25, 50, 65, 80, 100, 115, 200]
}

grid_search = GridSearchCV(estimator=classifier_0, param_grid=param_grid, scoring=scoring)

BayesSearchCV

param_bayes = {
    'gamma': Categorical(param_grid['gamma']),
    'learning_rate': Categorical(param_grid['learning_rate']),
    'max_depth': Categorical(param_grid['max_depth']),
    'n_estimators': Categorical(param_grid['n_estimators'])
}

grid_search = BayesSearchCV(estimator=classifier_0, search_spaces=param_bayes, scoring=scoring, n_jobs=-1, cv=10)

Finding the Best HyperParameters

best_model = grid_search.fit(X_train, y_train)
hyperparams = best_model.best_params_

Building and Scoring the Classifier using the HyperParameters

# Fitting the Model
ne = hyperparams['n_estimators']
lr = hyperparams['learning_rate']
md = hyperparams['max_depth']
gm = hyperparams['gamma']
print("Recommended Params >>", f"ne: {ne},", f"lr: {lr}", f"md: {md}", f"gm: {gm}")

# Build Classification Model
classifier_1 = XGBClassifier(
    base_score=0.5,
    colsample_bylevel=1,
    colsample_bynode=1,
    objective=objective,
    booster="gbtree",
    eval_metric=eval_metric_list,
    n_estimators=ne,
    learning_rate=lr,
    max_depth=md,
    gamma=gm,
    subsample=0.8,
    colsample_bytree=1,
    random_state=1
)

# Fit Model
eval_set = [(X_train, y_train)]
classifier_1.fit(
    X_train,
    y_train,
    eval_set=eval_set,
    verbose=False
)

# Get predictions for training data
train_yhat = classifier_1.predict(X_train)
print("Training Preds: \n", train_yhat[:5])

# Set K-Fold Cross Validation Levels
cv = RepeatedStratifiedKFold(n_splits=5, n_repeats=3, random_state=1)

# Training Results
train_results = cross_val_score(classifier_1, X_train, y_train, scoring=scoring, cv=cv, n_jobs=1)

# Brief Review of Training Results
print("Average Accuracy K-Fold: ", round(train_results.mean(), 2))
print("Std Deviation K-Fold: ", round(train_results.std(), 2))
print("Precision Score 0: ", round(precision_score(y_train, train_yhat, average=None)[0], 3))
print("Precision Score 1: ", round(precision_score(y_train, train_yhat, average=None)[1], 3))

Performance

Machine: Laptop
Processor: AMD Ryzen 7
OS: Windows
DataFrame Shape: (7282, 17)

Understanding How Scope Affects Values in Your Spring REST Controller

Ed Legaspi — Sat, 02 Mar 2024 03:33:59 +0000

Below we explore how a scope annotation affects an instance value in a Spring REST controller.

Each controller is annotated with scope.

@RestController
@Scope([SCOPE_VALUE])
public class XXXScopeController {}

How to Convert Vertically Stored Asset Data into Columnar Format for Cointegration Analysis

Ed Legaspi — Sat, 02 Mar 2024 01:12:16 +0000

Introduction

This piece of code fetches asset information from a table stored vertically.

Dependencies

Install the following package.

conda install pandas
conda install numpy as np
conda install mysql-connector-python
conda install sqlalchemy
conda install pymysql

Hands-on Coding

Connect to the database

def query_df(query):
    try: 
        engine_uri = f"mysql+pymysql://db_user:db_pass_123@localhost:3306/tradewise_pse"
        db_conn = create_engine(engine_uri)        
        df_result = pd.read_sql(query, db_conn)    
        return df_result

    except Exception as e:    
        print(str(e))

Fetching the Dataset

if not load_existing:
    sql_distinct_tickers = "select ticker from candlestick where event_time='2023-12-29' and ticker not like '^%%'"
    df_tickers = query_df(sql_distinct_tickers)

    df = pd.DataFrame(index=['event_time'])

    ### Get the candlesticks
    for ticker in df_tickers['ticker']:
        sql_ticker_col = "select event_time, close from candlestick where ticker='{0}'"
        df_temp = query_df(sql_ticker_col.format(ticker))
        df_temp.set_index('event_time', inplace=True)
        df_temp.rename(columns={'close': ticker}, inplace=True)        
        df = df.add(df_temp, fill_value=0)

    df.to_csv(file_name)

Load the dataset from file

df = pd.read_csv(file_name, index_col=0)
df.drop(index=df.index[-1],axis=0, inplace=True)

Drop NA

df.dropna(axis=1, inplace=True)

Print the Dataset

print(f"Shape: {df.shape}")
print(f"Null values: {df.isnull().values.any()}")
df

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6ypd0yfpe8ihrfvx58cl.png)

Learn to Incorporate Rolling Hurst Values into Your DataFrame

Ed Legaspi — Sat, 02 Mar 2024 01:08:30 +0000

The hurst function.

def hurst(ts, min_lag=1, max_lag=7):
    lags = range(min_lag, max_lag)
    tau = [np.sqrt(np.std(np.subtract(ts[lag:], ts[:-lag]))) for lag in lags]
    poly = np.polyfit(np.log(lags), np.log(tau), 1)
    return poly[0]*2.0

Adding to our DataFrame

The hurst value is computed with the last 14 close values.

df['Hurst'] = df['close'].rolling(14).apply(hurst, raw=True)
df[10:20]

Implementing Glowroot: A Hands-On Tech Review for Application Monitoring Mastery

Ed Legaspi — Mon, 05 Feb 2024 02:16:41 +0000

Introduction

In the realms of information technology and systems management, Application Performance Management (APM) involves monitoring and overseeing the performance and availability of software applications. APM aims to identify and diagnose intricate application performance issues to uphold a predefined level of service.

Glowroot is an open-source APM that facilitates a quicker resolution of application performance issues by helping us pinpoint the root causes. It supports applications running from Java 6 onwards.

Key Features

Response Time Breakdown Charts: Visualizes the breakdown of response times for better analysis.
Response Time Percentile Charts: Offers percentile charts to understand response time distribution.
SQL Capture and Aggregation: Captures and aggregates SQL queries for in-depth analysis.
Service Call Capture and Aggregation: Gathers and aggregates data on service calls for comprehensive insights.
MBean Attribute Capture and Charts: Monitors and charts MBean attributes for performance evaluation.
Configurable Alerting: Allows users to configure alerts based on specific criteria.
Historical Rollup: Provides historical data rollup at different intervals (1m, 5m, 30m, 4h) with configurable retention settings.
Full Support for Async Requests: Supports asynchronous requests that span multiple threads.
Responsive UI with Mobile Support: User-friendly and responsive interface with mobile support for accessibility.
Optional Central Collector: Offers the flexibility of an optional central collector for centralized data management.
Supports Multiple Application Servers: Wildfly, JBoss EAP, Tomcat, TomEE, Jetty, Glassfish, Payara, WebLogic, WebSphere

Central Collector

The central collector collects runtime information from the registered services and offers a GUI for easy viewing. https://github.com/glowroot/glowroot/wiki/Central-Collector-Installation

Installation

You can follow the steps above for running the GlowRoot central collector either as a standalone or as a docker image. For this section, I'll share how it can be run locally.
Here's my docker-compose file for running glowroot-central with cassandra. Override username, password, and contactPoints in glowroot-central.properties.



version: '3.8'

networks:

  tradewise-network:

services:

  cassandra:

    image: cassandra:latest

    container_name: cassandra

    restart: unless-stopped

    ports:

      - "9042:9042"

    networks:

      - tradewise-network

glowroot-central:

    image: glowroot/glowroot-central:0.14.1

    container_name: glowroot-central

    restart: unless-stopped

    volumes:

      - ./glowroot-central.properties:/usr/share/glowroot-central/glowroot-central.properties

    depends_on:

      - cassandra

    ports:

      - "4000:4000"

      - "8181:8181"

    networks:

      - tradewise-network

Instrumentation

There are 4 ways in which we can integrate GlowRoot into our services https://glowroot.org/instrumentation.html. For this exercise, we will focus on using the agent API.

Editing the pom.xml File

In our Spring Boot project's pom.xml file, add the GlowRoot agent configuration.



<plugin>

    <groupId>org.apache.maven.plugins</groupId>

    <artifactId>maven-resources-plugin</artifactId>

    <executions>

        <execution>

            <id>glowroot-plugins</id>

            <phase>validate</phase>

            <goals>

                <goal>copy-resources</goal>

            </goals>

            <configuration>

                <outputDirectory>target/glowroot-plugins/</outputDirectory>

                <resources>

                    <resource>

                        <directory>glowroot-plugins</directory>

                        <filtering>false</filtering>

                    </resource>

                </resources>

            </configuration>

        </execution>

    </executions>

</plugin>

<plugin>

    <groupId>org.apache.maven.plugins</groupId>

    <artifactId>maven-dependency-plugin</artifactId>

    <version>${maven-dependency-plugin.version}</version>

    <executions>

        <execution>

            <id>copy-glowroot-jar</id>

            <phase>prepare-package</phase>

            <goals>

                <goal>copy</goal>

            </goals>

        </execution>

    </executions>

    <configuration>

        <artifactItems>

            <artifactItem>

                <groupId>org.glowroot</groupId>

                <artifactId>glowroot-agent</artifactId>

                <version>${glowroot-agent.version}</version>

                <type>jar</type>

                <overWrite>false</overWrite>

                <outputDirectory>${project.build.directory}</outputDirectory>

                <destFileName>glowroot.jar</destFileName>

            </artifactItem>

        </artifactItems>

    </configuration>

</plugin>

Preparing the Dockerfile

We need to add the GlowRoot files in the docker image.



  
  
  Base Image


FROM eclipse-temurin:17-jdk-alpine

LABEL author=CzetsuyaTech

LABEL maintainer=CzetsuyaTech

  
  
  Configuration


WORKDIR /

RUN addgroup --system czetsuyatech && \

    adduser --system czetsuyatech --ingroup czetsuyatech && \

    mkdir -p /glowroot /glowroot/tmp /glowroot/logs /glowroot/plugins && \

    echo '{ "web": { "bindAddress": "0.0.0.0" } }' > /glowroot/admin.json && \

    chown czetsuyatech:czetsuyatech -R /glowroot && \

    chmod -R 777 /glowroot

USER czetsuyatech

ADD --chown=czetsuyatech:czetsuyatech target/glowroot.jar /glowroot

ADD --chown=czetsuyatech:czetsuyatech target/glowroot-plugins /glowroot/plugins

  
  
  Service


ADD --chown=czetsuyatech:czetsuyatech target/*.jar app.jar

  
  
  Start


ENV JAVA_JAR "/app.jar"

ENV JAVA_OTHERS "-Xshare:off"

ENV JAVA_OPTS ${JAVA_OPTS}

ENV JAVA_MEM ${JAVA_MEM}

RUN echo "exec java $JAVA_MEM $JAVA_OPTS $JAVA_OTHERS -jar $JAVA_JAR"

ENTRYPOINT exec java $JAVA_MEM $JAVA_OPTS $JAVA_OTHERS -jar $JAVA_JAR

Running the Spring Boot Service

To enable instrumentation in our instance, we need to specify the javaagent property. And to send information to the central collector we need to specify the central collector and give our instance an agent id.

JAVA_OPTS=-javaagent:glowroot/glowroot.jar -Dglowroot.collector.address=localhost:8181 -server -Dglowroot.agent.id=TradewiseAI

GUI

Usage

Transactions

Web

To assess our REST Endpoints' performance, we access the "Web" section. In the provided example:

By choosing "Response Time," we can identify which HttpRequests and JDBC Queries are taking longer in the REST Endpoints.
Opting for "Slow Traces" allows us to pinpoint the specific endpoint that consumes more time.
Selecting "Queries" reveals insights into the queries made, indicating that using the count query is more resource-intensive compared to the select query. This information aids in optimizing and refining the performance of REST Endpoints.

Background

To analyze our background performance, we navigate to the "Background" section. In the given example:

Choosing "Response Time" allows us to identify which Job and Hibernate Queries are consuming more time in the background processes.
Opting for "Slow Traces" reveals that some calls take more time, providing insights into areas that may require attention.
Selecting "Queries" and filtering by the select query, we observe that it is called more frequently and takes longer, particularly when filtered by status. This information helps in understanding and addressing potential bottlenecks in the background processes.

Startup

To assess our startup performance, we can utilize the "Startup" section. In the provided example:

By choosing "Response Time," we can identify which startup and filter init processes consume more time.
Opting for "Slow Traces" reveals the duration it takes for the context to initialize fully.
Selecting "Queries" provides insights into the database queries. Some queries will be more resource intensive, with fewer calls but longer duration. This information aids in pinpointing specific areas for optimization within the startup processes.

Errors

Web

To visualize errors in our REST endpoints, we can navigate to the "Web" section. In the following example:

Choosing "Error Messages" reveals errors of type XXXException.
Opting for "Error Traces" provides details on the errors.
Clicking on a specific error allows us to view the detailed trace, aiding in the understanding and resolution of the issue.

JVM

In our services' JVM, we can monitor and analyze memory status. This allows us to gain insights into the memory usage patterns, allocations, and overall health of the Java Virtual Machine, aiding in the effective management and optimization of our services.

MBeanTree

The MBeanTree functionality in Glowroot proves invaluable in monitoring instance creation. For instance, to track thread-related metrics and identify potential Thread Leaks, users can navigate to the java.lang section, specifically under Threading. This allows for a detailed examination of thread-related information, aiding in the identification and resolution of potential thread-related issues, such as Thread Leaks.

Recommendation

In a typical product infrastructure, Glowroot serves as our APM tool in each microservice. All microservices are instrumented to gather and transmit data to Glowroot, enhancing our understanding of system behavior. We've chosen Glowroot based on various criteria, including its license-free nature, alignment with the Java ecosystem, and straightforward instrumentation for microservices, ensuring a simple ramp-up and configuration process.

Navigating the Code: A Guide to Environment Variables, Configuration, and Feature Flags

Ed Legaspi — Thu, 14 Dec 2023 00:10:28 +0000

Determining the optimal location for storing essential information crucial for an application's functionality necessitates thoughtful planning. This guide aims to assist you in making informed decisions about where to store crucial data. Here, the term "application" encompasses any software created with code and deployed on various platforms and runtimes, irrespective of the underlying software architecture.

A fundamental principle is to avoid hard-coding any values within your application. Instead, adopt the practice of retrieving all configuration settings from a distinct storage system, separate from the deployment environment. This approach allows for seamless modifications to configuration settings even after the application has been deployed, eliminating the need for redeployment. This separation ensures greater flexibility and facilitates the adjustment of critical parameters without disrupting the operational state of your deployed application.

When considering the storage location for key/value pairs, it's crucial to evaluate the volatility of each pair—how frequently and by whom it might change. Reflect on the dynamic nature of the data and the responsibilities associated with its modification. This thoughtful approach will guide you in choosing an appropriate storage solution that aligns with the specific needs and characteristics of each key/value pair. By understanding the frequency and ownership of potential changes, you can make informed decisions that optimize the efficiency and maintainability of your data storage strategy.

Environment Variables

Environment variables, characterized by their infrequent changes, typically occur only once per deployment. Common examples include sensitive information like usernames and passwords for databases, which are often stored securely in a Vault. Additionally, environment variables encompass essential details required for the system's initialization, such as environment specifics and the connection details for the application configuration store, which hosts additional settings. By utilizing environment variables for these foundational aspects, you ensure a secure and stable system setup while centralizing critical configuration information.

Application Configuration

Application configuration, in contrast to environment variables, operates independently of deployments. It encompasses supplementary settings for the system that have the potential to change between deployments at runtime. This flexibility allows for real-time adjustments to the application's behavior and features, accommodating evolving requirements without the need for redeployment. By leveraging application configuration for these dynamic settings, you establish a modular and adaptable structure, enhancing the responsiveness of your system to changing runtime conditions.

Runtime User Settings

Runtime user settings represent the most volatile category, as users can alter any key/value pair at any given moment. Therefore, the system must be designed to accommodate this high level of dynamism. Options for handling such volatility include implementing robust real-time validation mechanisms and ensuring the integrity and security of user-initiated changes. Additionally, the system should provide an intuitive user interface for managing these settings, enabling seamless customization while maintaining overall system stability and security.

Here are three options for handling configuration updates:

Re-read Configuration All the Time: Continuously monitor and re-read the configuration, ensuring that the system remains up-to-date with any changes. This approach provides real-time responsiveness to modifications but may impose a continuous processing overhead.

Re-read Configuration at Set Intervals (e.g., Every 15 Minutes): Implement a periodic schedule to re-read the configuration at predefined intervals, such as every 15 minutes. This approach balances responsiveness with reduced processing overhead, making it suitable for scenarios where near-real-time updates are acceptable.

Receive Push Notifications When New Application Configuration Is Available: Set up a push notification mechanism to alert the system when new application configuration becomes available. This approach minimizes the need for continuous or scheduled re-reading, optimizing efficiency by updating the system only when changes occur. However, it requires a reliable notification infrastructure. Choosing among these options depends on the specific requirements of your system, considering factors such as the criticality of real-time updates, resource constraints, and the overall responsiveness desired for configuration changes.

Java XML Validation: How to Validate an XML Document against an XSD

Ed Legaspi — Sat, 01 Jul 2023 11:58:22 +0000

Introducing our Java XML Validation Project! Master the art of validating XML effortlessly against XSD in a Java environment. Gain hands-on experience, ensuring the integrity and conformity of your data.

1. Introduction

Introducing our Java XML Validation Project! Gain hands-on experience in validating XML documents against XSD with ease. This project serves as a practical guide, equipping you with the necessary knowledge and tools to ensure the integrity and conformity of your XML data in a Java environment. Discover the simplicity of our step-by-step approach, which walks you through the entire validation process. Master the art of validating XML effortlessly and unlock a new level of confidence in your data. Join us on this journey as we empower you to harness the power of XML validation in your Java projects.

2. Generating Java Classes from XSD

For this exercise, we will be using an XSD from w3schools https://www.w3schools.com/xml/schema_example.asp.

To be able to generate the Java classes from XSD, we will be using Eclipse Enterprise Edition. IntelliJ can also do it with the Jakarta plugin, but Eclipse performs better, especially when dealing with XJB files.

IntelliJ

Eclipse JAXB Classes Generator

Generated Classes

3. Dependencies

<dependency>
    <groupId>com.fasterxml.jackson.dataformat</groupId>
    <artifactId>jackson-dataformat-xml</artifactId>
</dependency>

<dependency>
    <groupId>jakarta.activation</groupId>
    <artifactId>jakarta.activation-api</artifactId>
    <version>2.1.1</version>
</dependency>
<dependency>
    <groupId>jakarta.xml.bind</groupId>
    <artifactId>jakarta.xml.bind-api</artifactId>
    <version>${jakarta.version}</version>
</dependency>
<dependency>
    <groupId>org.glassfish.jaxb</groupId>
    <artifactId>jaxb-runtime</artifactId>
    <version>${jakarta.version}</version>
    <scope>runtime</scope>
</dependency>

4. Core Classes

To run the validations and throw the appropriate exception or name of the tag where a constraint is violated, we will need the following classes.

4.1 ResourceResolver

To import an external XSD, for example, if we will be using the XML digital signature.

<xs:import namespace="http://www.w3.org/2000/09/xmldsig#" schemaLocation="xmldsig-core-schema.xsd"/>

public class ResourceResolver implements LSResourceResolver {

  private static final Logger LOGGER = LoggerFactory.getLogger(ResourceResolver.class);

  public LSInput resolveResource(String type, String namespaceURI, String publicId, String systemId, String baseURI) {

    // note: in this sample, the XSD's are expected to be in the root of the classpath
    InputStream resourceAsStream = this.getClass().getClassLoader().getResourceAsStream("xsd/" + systemId);
    return new Input(publicId, systemId, resourceAsStream);
  }

  static class Input implements LSInput {

    private String publicId;

    private String systemId;

    public String getStringData() {

      String textDataFromFile;

      try {
        BufferedReader bufferReader = new BufferedReader(new InputStreamReader(inputStream));
        StringBuilder stringBuilder = new StringBuilder();
        String eachStringLine;

        while ((eachStringLine = bufferReader.readLine()) != null) {
          stringBuilder.append(eachStringLine).append("\n");
        }
        textDataFromFile = stringBuilder.toString();

        return textDataFromFile;

      } catch (IOException e) {
        LOGGER.error("Fail reading from inputStream={}", e.getMessage());
        return null;
      }
    }
...

4.2 SaxErrorHandler

For us to be able to know the tag where a constraint exception is thrown, we need to define a custom exception that accepts an XMLStreamReader object where we can get the localName property.

public class SaxErrorHandler implements ErrorHandler {

  private XMLStreamReader reader;

  public SaxErrorHandler(XMLStreamReader reader) {
    this.reader = reader;
  }

  @Override
  public void error(SAXParseException e) {
    warning(e);
  }

  @Override
  public void fatalError(SAXParseException e) {
    warning(e);
  }

  @Override
  public void warning(SAXParseException e) {

    throw new XmlValidationException(reader.getLocalName());
  }
}

4.3 Miscellaneous Classes

We will also be needing utility classes for:

Creating a JAXBElement
Convert JAXBElement to XMLStreamReader
Defining the correct marshaller object

These classes are all available in the project.

4.4 XmlSchemaValidator Interface

And finally, the interface that pieces together all components to do the validation.

default void validateXMLSchema(String xsdPath, JAXBElement<?> xml) {

  Validator validator = null;
  try {
    SchemaFactory factory = SchemaFactory.newInstance(XMLConstants.W3C_XML_SCHEMA_NS_URI);
    factory.setProperty(XMLConstants.ACCESS_EXTERNAL_DTD, "");
    factory.setProperty(XMLConstants.ACCESS_EXTERNAL_SCHEMA, "");
    factory.setResourceResolver(new ResourceResolver());

    Schema schema = factory.newSchema(ResourceUtils.getFile(xsdPath));
    validator = schema.newValidator();
    XMLStreamReader xmlStreamReader = asStreamReader(xml);

    ErrorHandler errorHandler = new SaxErrorHandler(xmlStreamReader);
    validator.setErrorHandler(errorHandler);

    validator.validate(new StAXSource(xmlStreamReader));

  } catch (IOException | SAXException | JAXBException | XMLStreamException e) {

    var xmlEx = getXmlValidationExceptionIfPresent(e);
    throw new InvalidXmlException(xmlEx.getLocalName(), e);
  }
}

5. Source Code / Git Repository

The complete source code for this project is available for my sponsors at https://github.com/czetsuyatech/java-xml-validation-by-xsd.

Sponsorship link: https://github.com/sponsors/czetsuya

Project Screenshot

Unlocking Google Calendar API with Keycloak: A Developer's Guide

Ed Legaspi — Sat, 20 May 2023 07:52:14 +0000

1. Overview

Unlock the power of Keycloak for seamless authentication and authorization in your applications. This comprehensive guide is designed for software developers who want to integrate Keycloak into their systems to enable secure user login and token exchange with Google.

By following step-by-step instructions and best practices, you'll learn how to leverage Keycloak's robust features to facilitate seamless authentication and obtain a Google token. With this token, you can effortlessly access the Google Calendar API and unlock a wealth of functionality for your application. Enhance user experience and streamline calendar integration - start implementing Keycloak and Google integration today.

You need prior knowledge with:

Google
Keycloak
Spring Boot

2. Google

Before we can do the integration with Keycloak we need to create a project in Google Console, create a credential and obtain OAuth client id and secret.

If you haven't done so, you can create a Google console account at https://cloud.google.com/cloud-console.

2.1 Creating the Google Cloud Project

At the top left of your Google Cloud Console, click the project and click New Project.

I call this project "Lab".

2.2 Create OAuth2 Client

The next step is to create the OAuth2 client credential that will give us the client id and secret. This combination will be used when we set up the Google Identity provider in Keycloak.

With the Lab project selected, find API & Services / Credentials in the menu.

Click Create Credentials with the following inputs:

Type: OAuth client ID
Application type: Web application
Name: Lab

There are two important entries in this section:

Authorized Javascript origins

This should be the URL where the request for login will come from.

Authorized redirect URIs

This entry is a common cause of problems if not properly set up. This is where Google will redirect the user after a successful login.

Some common redirect URIs.

Amazon Cognito: https://lab.auth.ap-southeast-1.amazoncognito.com/oauth2/idpresponse
Postman: https://www.getpostman.com/oauth2/callback
Keycloak broker: http://localhost:8080/realms/czetsuyatech/broker/google/endpoint
Keycloak token: http://localhost:8080/realms/czetsuyatech/protocol/openid-connect/token

When you have a redirect issue, this is one of the first things you may want to check.

This is how my Lab client looks like.

On the same page, you should be able to see the client id and secret. Take note because we will need them later.

2.3 Setup OAuth Consent Screen

This section will ask for several application-related settings. You can enter values as you see fit, but normally I set the following.

Non-sensitive scopes:

auth-userinfo-email
auth-userinfo-profile
openid Don't forget to add a test user. We will use it to login later.

3. Keycloak

For this exercise, we need to create a new Keycloak realm and client. We must enable authentication in the client.

3.1 Client

Enabling authentication will give us access to Keycloak's client and secret, which we will need later for testing. Take note of it.

We also need to download the Adapter config of our newly created client.

The downloaded config should look like this. Save it as we will use it in our Spring Boot application later.

{
  "realm": "czetsuyatech",
  "auth-server-url": "http://localhost:8080/",
  "ssl-required": "external",
  "resource": "web-front",
  "credentials": {
    "secret": "BxYWD11qCAW6ZybPTy6b9M8ej0thTxxx"
  },
  "confidential-port": 0
}

3.2 Identity Provider

Identity providers are third-party services that allow users to log in to Keycloak. In our example, we will do an authorization as well by exchanging the Keycloak token to Google to allow us to call Google API services such as Calendar and Gmail.

So login to Keycloak as admin, click Identity Providers on the left side, Click Add Provider, and select Google.

In the Client ID and secret fields, enter the values that we saved when we were configuring the Google client earlier.

Under Advance settings, change the value of Scopes to "email profile openid https://www.googleapis.com/auth/calendar https://www.googleapis.com/auth/calendar.events https://www.googleapis.com/auth/gmail.readonly", without double quotes. This will give us permission to call list endpoints from Calendar and Gmail APIs.

3.3 Permissions

In this section, we will give our client permission to exchange tokens with the newly created Google identity provider.

On the identity provider page, click the Permissions tab. Toggle Permissions enabled to ON. Under the Permission list, click token-exchange.

Set the Decision strategy to Unanimous. Click Save.

Next, we need to create the permission scope. At the top-left of the page, click Client details. It should redirect you to the Authorization tab. Open the Policies sub-tab. Create a new policy with the following values:

Type: Client
Name: google-token-exchange
Clients: web-front (the Keycloak client we created earlier)
Logic: Positive

Go back to your Google identity provider’s permission page and set the Policies to the one we just created.

The setup should now be ready to exchange tokens.

4. Testing

To test the exchange of tokens between Keycloak and Google, I have created a Postman collection where we can:

Authenticate a user
Exchange Keycloak to Google token

Before we proceed with the testing, we need to run a Keycloak instance. I have prepared a docker-compose to do that.

We need to authenticate our user, so create a new POST request in Postman and use OAuth2 authorization. Set the URL to http://localhost:8080/realms/czetsuyatech/protocol/openid-connect/token.

Be sure to set the client id and secret to the ones we saved earlier when we created the Keycloak client.

For the truncated values:

Callback URL: https://www.getpostman.com/oauth2/callback

Auth URL: http://localhost:8080/realms/czetsuyatech/protocol/openid-connect/auth

Access Token URL: http://localhost:8080/realms/czetsuyatech/protocol/openid-connect/token

Click Get New Access Token. It should redirect us to a Google login page, use the test email address that you added earlier when creating the Google client.

The next page will show you the scopes that will be permitted for your user. We defined this when we created the Google identity provider earlier.

Click Allow.

Copy the access token and paste it on a bearerToken variable. The succeeding requests will have Authorization Type = Bearer and will use this value.

The Body of the request must have the following URL encoded parameters:

Click Send. It should return with the following response.

This is the token that we can use to access Google APIs.

Obviously, we need to automate the process if we wanted this feature in our application. Thus, I have created a Spring project to do it.

5. Spring Project

This project demonstrates the token-exchange feature available on Keycloak. With this, we can exchange Keycloak for Google’s token, which we can use to call Google APIs.

Note: This project is only available for my sponsors https://github.com/sponsors/czetsuya.

5.1 Available Features

Get the bearer token from the request and store it in a Bean that can be autowired.
Exchange Keycloak token to Google, this will allow us to call Google APIs (Calendar, Gmail).
Use Spring exchange to provide abstraction in calling Google APIs.
Endpoints to list Calendar events and Gmail messages.

The heart of the application exchanges tokens between Keycloak and Google.

@Component
public class GoogleTokenExchange implements TokenExchange {

  @Override
  public String exchangeToken(String accessToken) {

    AuthzClient authzClient = AuthzClient.create();
    Configuration keycloakConfig = authzClient.getConfiguration();
    String baseUrl = authzClient.getServerConfiguration().getTokenEndpoint();

    RestTemplate restTemplate = new RestTemplate();

    HttpHeaders headers = new HttpHeaders();
    headers.add("Content-Type", MediaType.APPLICATION_FORM_URLENCODED.toString());
    headers.add("Accept", MediaType.APPLICATION_JSON.toString());

    MultiValueMap<String, String> requestBody = new LinkedMultiValueMap<>();
    requestBody.add("client_id", keycloakConfig.getResource());
    requestBody.add("client_secret", keycloakConfig.getCredentials().get("secret").toString());
    requestBody.add("grant_type", "urn:ietf:params:oauth:grant-type:token-exchange");
    requestBody.add("subject_token_type", "urn:ietf:params:oauth:token-type:access_token");
    requestBody.add("subject_token", accessToken);
    requestBody.add("requested_issuer", "google");

    HttpEntity formEntity = new HttpEntity<>(requestBody, headers);

    ResponseEntity<AccessTokenResponse> response = restTemplate.exchange(baseUrl, HttpMethod.POST, formEntity,
        AccessTokenResponse.class);

    return response.getBody().getToken();
  }
}

5.2 Testing

If you have subscribed as my sponsor, you will be given access to my repository and proceed with this testing.

The project provides two endpoints, to test calling list calendar events and Gmail messages from Google. Matching Postman requests are also available to make testing easier.

As we have done earlier, we need to get an access token from Keycloak and set it as bearerToken environment variable inside Postman.

Get Calendar Events Response

Get Gmail Messages

6. Source Code

All the resources used in this article and demo are available on the GitHub repository https://github.com/czetsuyatech/auth-exchange.

Spring project source code
Docker/compose file
Keycloak realm
Postman collection for testing

7. References

Implementing X509 Certificate Validation in Java: A Step-by-Step Guide

Ed Legaspi — Sun, 07 May 2023 05:09:15 +0000

Unlock the power of secure communication with Java! Learn how to implement X509 certificate validation simply and straightforwardly, step-by-step.

Overview

An X.509 certificate is a digital certificate used to verify a particular entity's identity, such as a website or an individual. X.509 certificates are widely used in various applications, including secure communication protocols like HTTPS, SSL, and TLS.

An X.509 certificate contains several pieces of information, including the identity of the entity being verified, the public key that is associated with that entity, and various other pieces of metadata that provide additional information about the certificate itself.

Importance

One of the key benefits of X.509 certificates is that they allow entities to authenticate themselves to other parties securely and reliably. This can help to prevent various types of attacks, including man-in-the-middle attacks, where an attacker intercepts and modifies communication between two parties to steal sensitive information.

Overall, X.509 certificates play a critical role in modern security infrastructure and are essential to many different types of secure communication protocols.

Requirements

You must have the following installed on your computer.

Git
Maven
IDE (IntellIJ / Visual Studio)
Java 17+

Dependencies

This project uses the following dependencies.

bcpkix-jdk15on - certificate revocation check
spring-boot-starter-test (for testing)
lombok (for logging)
Certificates from https://www.digicert.com/kb/digicert-root-certificates.htm

Project Code

The certificate validation process.

To implement the validation, we will introduce a bean where we can define the:

Certificate that we are validating
The pem file that we can cross-check #1
Root certificate to check if the certificate is already revoked

Let's examine the code.
The CertificateBundle class is just a simple Java bean.

@Getter
@Setter
@Builder
public class CertificateBundle {

  // Certificate extracted from the message
  private X509Certificate certificate;

  // Certificate loaded from the application must match the certificate
  private X509Certificate confirmationCertificate;

  // RootCertificate of the certificate
  private X509Certificate rootCertificate;
}

To validate the certificate expiration, we need to extract the X509Certificate object and perform a time comparison against the value of getNotAfter(). Here, we're making sure that the current date is greater than the certificate's expiry date.

Date expiresOn = cert.getNotAfter();
Date now = new Date();

if (now.getTime() > expiresOn.getTime()) {
  throw new ExpiredCertificateException();
}

Comparing digital signature against a PEM file. In most fintech applications, there is a need to sign the message before sending it to another service. Upon receiving the message, the server needs to extract the public certificate embedded in it and compare it to a PEM file. This PEM file is a public key of the private key certificate used to sign the message.

boolean match = Arrays.equals(cert1.getPublicKey().getEncoded(),
    cert2.getPublicKey().getEncoded());

if (!match) {
  throw new MismatchCertificateException("Embedded certificate does not match the given PEM");
}

And finally, we do the revocation check. This process allows us to know whether a website's certificate is trustworthy or not. It uses the root certificate that we used for signing as the trust anchor.

try {
  List<X509Certificate> certs = Collections.singletonList(cb.getCertificate());
  CertificateFactory cf = CertificateFactory.getInstance("X509");
  CertPath cp = cf.generateCertPath(certs);

  // load the root CA cb
  X509Certificate rootCACert = cb.getRootCertificate();

  // init trusted certs
  TrustAnchor ta = new TrustAnchor(rootCACert, null);
  Set<TrustAnchor> trustedCerts = new HashSet<>();
  trustedCerts.add(ta);

  // init PKIX parameters
  PKIXParameters params = new PKIXParameters(trustedCerts);

  // load the CRL
  params.addCertStore(CertStore.getInstance("Collection",
      new CollectionCertStoreParameters(getX509Crls(cf, getCrl(cb.getCertificate())))));

  // perform validation
  CertPathValidator cpv = CertPathValidator.getInstance("PKIX");
  PKIXCertPathValidatorResult cpvResult = (PKIXCertPathValidatorResult) cpv.validate(cp,
      params);
  X509Certificate trustedCert = cpvResult.getTrustAnchor().getTrustedCert();

  if (trustedCert == null) {
    log.error("Trusted certificate not found");
    throw new CertificateException("Trusted certificate not found");

  } else {
    log.debug("Trusted CA DN = " + trustedCert.getSubjectX500Principal());
  }

} catch (InvalidAlgorithmParameterException | java.security.cert.CertificateException |
         CRLException |
         NoSuchAlgorithmException |
         CertPathValidatorException |
         IOException e) {
  throw new RevokedCertificateException(e);
}

Java-Based Implementation of Digital Signatures and X509 Certificates for Secure XML Document Processing

Ed Legaspi — Wed, 03 May 2023 03:34:57 +0000

Overview

XML Signature is a standard in the digital signature that allows the authentication and validation of data being sent as XML objects in web service communication. Java 6 introduces Java XML Digital Signature API, which allows the generation and validation of signatures in an XML message.

According to RFC 2828, a digital signature is a cryptographic value added to a data object to enable the recipient to verify its origin and integrity. The cryptographic digital signature API in JDK 6 is described in detail in a security lesson in the Java Tutorial.

An XML signature is a type of digital signature that has unique properties. It outlines a process and format for generating digital signatures in the XML format and offers various advanced features. For example, it allows for the signing of multiple pieces of data, either in binary or XML format, and it enables the use of various cryptographic signature algorithms.

An XML signature is capable of signing arbitrary data, whether it is in XML or binary format. Furthermore, it can sign only a portion or a subset of an XML document instead of the entire document. Uniform Resource Identifiers (URIs) identify the data to be signed. XML signatures can be categorized as one or more of three types based on their properties.

A detached signature is a signature that covers data that exists outside of the Signature element. This data may be located in a different document or file or present within the same document as the signature, such as a related element.
An enveloping signature is a signature that covers data that is contained within the Signature element itself. In other words, the data to be signed is enclosed within the signature element and the signature itself.
An enveloped signature is a type of signature that covers data containing the Signature element.

Java API Architecture

The JSR 105 Java Community Process program established the Java XML Digital Signature API to encompass all necessary and suggested features of the W3C's XML-Signature Syntax and Processing Recommendation. The API employs the Java Cryptography Service Provider Architecture, which enables the development of service provider implementations. Service providers create an XML mechanism that identifies the implementation's XML-parsing mechanism. In Java SE 6's implementation, Sun's service provider supports the Document Object Model (DOM) mechanism. For additional details on service providers, refer to the XML Digital Signature API overview.

XML Signature

This article will utilize an example of an enveloped XML signature, which means that it is generated over the contents of an XML document, specifically a sample puchase-order.xml.

<?xml version="1.0"?>
<PurchaseOrder PurchaseOrderNumber="99503" OrderDate="1999-10-20">
  <Address Type="Shipping">
    <Name>Ellen Adams</Name>
    <Street>123 Maple Street</Street>
    <City>Mill Valley</City>
    <State>CA</State>
    <Zip>10999</Zip>
    <Country>USA</Country>
  </Address>
  <Address Type="Billing">
    <Name>Tai Yee</Name>
    <Street>8 Oak Avenue</Street>
    <City>Old Town</City>
    <State>PA</State>
    <Zip>95819</Zip>
    <Country>USA</Country>
  </Address>
  <DeliveryNotes>Please leave packages in shed by driveway.</DeliveryNotes>
  <Items>
    <Item PartNumber="872-AA">
      <ProductName>Lawnmower</ProductName>
      <Quantity>1</Quantity>
      <USPrice>148.95</USPrice>
      <Comment>Confirm this is electric</Comment>
    </Item>
    <Item PartNumber="926-AA">
      <ProductName>Baby Monitor</ProductName>
      <Quantity>2</Quantity>
      <USPrice>39.98</USPrice>
      <ShipDate>1999-05-21</ShipDate>
    </Item>
  </Items>
</PurchaseOrder>

XML Example 1.

After applying the digital signature, our message should look like this.

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<PurchaseOrder OrderDate="1999-10-20" PurchaseOrderNumber="99503">
  <Address Type="Shipping">
    <Name>Ellen Adams</Name>
    <Street>123 Maple Street</Street>
    <City>Mill Valley</City>
    <State>CA</State>
    <Zip>10999</Zip>
    <Country>USA</Country>
  </Address>
  <Address Type="Billing">
    <Name>Tai Yee</Name>
    <Street>8 Oak Avenue</Street>
    <City>Old Town</City>
    <State>PA</State>
    <Zip>95819</Zip>
    <Country>USA</Country>
  </Address>
  <DeliveryNotes>Please leave packages in shed by driveway.</DeliveryNotes>
  <Items>
    <Item PartNumber="872-AA">
      <ProductName>Lawnmower</ProductName>
      <Quantity>1</Quantity>
      <USPrice>148.95</USPrice>
      <Comment>Confirm this is electric</Comment>
    </Item>
    <Item PartNumber="926-AA">
      <ProductName>Baby Monitor</ProductName>
      <Quantity>2</Quantity>
      <USPrice>39.98</USPrice>
      <ShipDate>1999-05-21</ShipDate>
    </Item>
  </Items>
  <Signature
    xmlns="http://www.w3.org/2000/09/xmldsig#">
    <SignedInfo>
      <CanonicalizationMethod Algorithm="http://www.w3.org/TR/2001/REC-xml-c14n-20010315"/>
      <SignatureMethod Algorithm="http://www.w3.org/2001/04/xmldsig-more#rsa-sha256"/>
      <Reference URI="">
        <Transforms>
          <Transform Algorithm="http://www.w3.org/2000/09/xmldsig#enveloped-signature"/>
        </Transforms>
        <DigestMethod Algorithm="http://www.w3.org/2001/04/xmlenc#sha256"/>
        <DigestValue>rKK35PJzAva2B92cG/cJp6J2BF5JUbKPS5ogYNJkgJo=</DigestValue>
      </Reference>
    </SignedInfo>
    <SignatureValue>dJ2yTNLTjE41e/q4GRAAHBjIJY+Ax3Qn00CyDRTtfdF1WzCUkxj9V7hpp/alwA3/NWeifHrJMr0F
      ak8g41u90RcKV/9rX3lRxNyOKApAKpTw6CrAM6PTcHzHF/NWiL/At8v51AZYm55UgjjrB3xxHvUn
      BidyBUtRuETIrikUD3hSMGD2u9prNhgAxWZ+QmRx2ga171/tnGo1B0JF+aBnup8kgq0I9Po3BSW3
      a7gwAbVlV3R/pFIrht8wfOVjyzQgFnB+SCJ9UwdSwPJqfyrWxSGE5em3hjbX4VGTWeyhS56s0lX5
      9vkCk6dw9cxIvyUsAVSTxYX5SRib2ekpIe7Oyg==
    </SignatureValue>
    <KeyInfo>
      <X509Data>
        <X509SubjectName>
          1.2.840.113549.1.9.1=#161e6564776172642e6c6567617370695f65787440612d746f2d62652e636f6d,CN=localhost,OU=EETS,O=Atobe,ST=Lisboa,C=PT
        </X509SubjectName>
        <X509Certificate>MIIDiTCCAnECFAtZBLeU3vRAzm0tXHf7pdP07psNMA0GCSqGSIb3DQEBCwUAMIGAMQswCQYDVQQG
          EwJQVDEPMA0GA1UECAwGTGlzYm9hMQ4wDAYDVQQKDAVBdG9iZTENMAsGA1UECwwERUVUUzESMBAG
          A1UEAwwJbG9jYWxob3N0MS0wKwYJKoZIhvcNAQkBFh5lZHdhcmQubGVnYXNwaV9leHRAYS10by1i
          ZS5jb20wHhcNMjMwMjIxMTAzNzM0WhcNMjQwMjIxMTAzNzM0WjCBgDELMAkGA1UEBhMCUFQxDzAN
          BgNVBAgMBkxpc2JvYTEOMAwGA1UECgwFQXRvYmUxDTALBgNVBAsMBEVFVFMxEjAQBgNVBAMMCWxv
          Y2FsaG9zdDEtMCsGCSqGSIb3DQEJARYeZWR3YXJkLmxlZ2FzcGlfZXh0QGEtdG8tYmUuY29tMIIB
          IjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAxpnILKjFkUsbaVhsI366Od6tfCFnZafsk0Vi
          KW+k2z2bckig2+OhDr88KItst7LWORFRmo6JltHx1TfWtkYhboVEWOhw0/kVPCBqNdtJiiAyrj5l
          pElgHW6Js8ECUgiiupoJeqwrDPCUbCF0MsOQX/swCbZMqQsAQrPRiNCiDviZ9V/aIlQEaDb7fDJT
          wfM4pmPSPzMXPRghv58EKNiJQIpmQw9H0LYA5iu/kvV6nYC8ROF/4giTGhPr3qYMyR1WCa5EuFOO
          NGUE8Hn8l43DMFa/hFxVECyYqw/wQQngvM0QDlli1K2FRyZYT7k/v+mqBiNYZC50hjkVtvO5Yd8m
          sQIDAQABMA0GCSqGSIb3DQEBCwUAA4IBAQAnLELZRFMZ8FbLNuSWOqmrizOo7BkiT7gyNx1t6fMb
          gadx8qnCpytbghr8O61QuzpYp+65hi9yDEDBfNDtvmT0olIhJK8tsVOu+857MEL0tNcNmavWI4qH
          N0lJ5HZRjTm0VLz86UZdWFJfyk7H+sZGOLucDVc4+BzcpsCiW1Dyy+rdEeLNjwmsLvPP2ol16Mzc
          Ij48aO2DLOut/+OaUl/4TNOrBKZq4Ve2F55RQUwvMDX0BLZlmDy3pI3XSho7KKDdNhl1/0AFJWVh
          skZwR+fAK/RbU1fB9Obc9/IoSm6KANsJRCqFbfkS2hIhT3cvQs/cmR5ZRjgcFgl5J3sMCUFX
        </X509Certificate>
      </X509Data>
    </KeyInfo>
  </Signature>
</PurchaseOrder>

Here is the code that digitally signs the XML document.

package com.czetsuyatech.dsig.xml;

import com.czetsuyatech.dsig.SignXml;
import com.czetsuyatech.dsig.exceptions.SignatureNotFoundException;
import com.czetsuyatech.dsig.exceptions.XmlSigningException;
import jakarta.xml.bind.JAXBException;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.OutputStream;
import java.security.InvalidAlgorithmParameterException;
import java.security.KeyStore;
import java.security.KeyStoreException;
import java.security.NoSuchAlgorithmException;
import java.security.UnrecoverableEntryException;
import java.security.cert.CertificateException;
import java.security.cert.X509Certificate;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
import javax.xml.crypto.MarshalException;
import javax.xml.crypto.dsig.CanonicalizationMethod;
import javax.xml.crypto.dsig.DigestMethod;
import javax.xml.crypto.dsig.Reference;
import javax.xml.crypto.dsig.SignatureMethod;
import javax.xml.crypto.dsig.SignedInfo;
import javax.xml.crypto.dsig.Transform;
import javax.xml.crypto.dsig.XMLSignature;
import javax.xml.crypto.dsig.XMLSignatureException;
import javax.xml.crypto.dsig.XMLSignatureFactory;
import javax.xml.crypto.dsig.dom.DOMSignContext;
import javax.xml.crypto.dsig.dom.DOMValidateContext;
import javax.xml.crypto.dsig.keyinfo.KeyInfo;
import javax.xml.crypto.dsig.keyinfo.KeyInfoFactory;
import javax.xml.crypto.dsig.keyinfo.X509Data;
import javax.xml.crypto.dsig.spec.C14NMethodParameterSpec;
import javax.xml.crypto.dsig.spec.TransformParameterSpec;
import javax.xml.parsers.DocumentBuilderFactory;
import javax.xml.parsers.ParserConfigurationException;
import javax.xml.transform.Transformer;
import javax.xml.transform.TransformerException;
import javax.xml.transform.TransformerFactory;
import javax.xml.transform.dom.DOMSource;
import javax.xml.transform.stream.StreamResult;
import org.w3c.dom.Document;
import org.w3c.dom.NodeList;
import org.xml.sax.SAXException;

public class DigitalSigneeImpl implements XmlTransformer {

  private final String keystoreFile;
  private final String keystoreFormat;
  private final String keystorePassword;
  private final String privateKeyPassword;
  private final String keyEntryAlias;
  private final XMLSignatureFactory signatureFactory;

  public DigitalSigneeImpl(String keystoreFile, String keystoreFormat, String keystorePassword,
      String privateKeyPassword, String keyEntryAlias) {

    this.keystoreFile = keystoreFile;
    this.keystoreFormat = keystoreFormat;
    this.keystorePassword = keystorePassword;
    this.privateKeyPassword = privateKeyPassword;
    this.keyEntryAlias = keyEntryAlias;

    // Instantiate a DOM XMLSignatureFactory object to produce the enveloped signature
    signatureFactory = XMLSignatureFactory.getInstance("DOM");
  }

  public Document sign(SignXml signXml) throws XmlSigningException {

    // STEP 1

    // Create a Reference to the document being signed by specifying an empty URI value to represent the entire
    // document. Additionally, specify the SHA256 digest algorithm and use the ENVELOPED Transform
    Reference ref;
    try {
      ref = signatureFactory.newReference("", signatureFactory.newDigestMethod(DigestMethod.SHA256, null),
          Collections.singletonList(signatureFactory.newTransform(Transform.ENVELOPED, (TransformParameterSpec) null)),
          null, null);

      // Create the SignedInfo
      SignedInfo signedInfo =
          signatureFactory.newSignedInfo(signatureFactory.newCanonicalizationMethod(CanonicalizationMethod.INCLUSIVE,
                  (C14NMethodParameterSpec) null),
              signatureFactory.newSignatureMethod(SignatureMethod.RSA_SHA256, null), Collections.singletonList(ref));

      // STEP 2

      // Load the KeyStore and retrieve the signing key and certificate
      KeyStore ks = KeyStore.getInstance(keystoreFormat);

      ks.load(getFileFromResourceAsStream(keystoreFile), keystorePassword.toCharArray());
      KeyStore.PrivateKeyEntry keyEntry =
          (KeyStore.PrivateKeyEntry) ks.getEntry(keyEntryAlias,
              new KeyStore.PasswordProtection(privateKeyPassword.toCharArray()));
      X509Certificate cert = (X509Certificate) keyEntry.getCertificate();

      // Create the KeyInfo containing the X509Data
      KeyInfoFactory kif = signatureFactory.getKeyInfoFactory();
      List<Object> x509Content = new ArrayList<>();
      x509Content.add(cert.getSubjectX500Principal().getName());
      x509Content.add(cert);
      X509Data xd = kif.newX509Data(x509Content);
      KeyInfo ki = kif.newKeyInfo(Collections.singletonList(xd));

      // STEP 3

      // Create an instance of the document that will be signed
      DocumentBuilderFactory dbf = getDocumentBuilderFactory();
      dbf.setNamespaceAware(true);
      //TO IMPL , temp change to be compilable
      Document doc = dbf.newDocumentBuilder().parse(getInputStream(signXml));

      // Initialize a DOMSignContext by providing the RSA PrivateKey and identifying the parent element of the resulting
      // XMLSignature
      DOMSignContext dsc = new DOMSignContext(keyEntry.getPrivateKey(), doc.getDocumentElement());

      // Instantiate the XMLSignature object without signing it yet
      XMLSignature signature = signatureFactory.newXMLSignature(signedInfo, ki);

      // Marshal, generate, and sign the enveloped signature
      signature.sign(dsc);

      return doc;

    } catch (NoSuchAlgorithmException | InvalidAlgorithmParameterException | ParserConfigurationException |
             MarshalException | XMLSignatureException | UnrecoverableEntryException | KeyStoreException |
             IOException | CertificateException | SAXException | JAXBException e) {
      throw new XmlSigningException(e);
    }
  }

  public boolean validate(Document doc) throws MarshalException, XMLSignatureException {

    // Find Signature element
    NodeList nl = doc.getElementsByTagNameNS(XMLSignature.XMLNS, "Signature");
    if (nl.getLength() == 0) {
      throw new SignatureNotFoundException("Signature element not found");
    }

    // Create a DOMValidateContext and specify a KeySelector and document context
    DOMValidateContext valContext = new DOMValidateContext(new X509KeySelector(), nl.item(0));
    valContext.setProperty("org.jcp.xml.dsig.secureValidation", Boolean.TRUE);

    // Unmarshal the signature
    XMLSignature signature = signatureFactory.unmarshalXMLSignature(valContext);

    // Validate the signature
    boolean result = signature.validate(valContext);

    if (!result) {
      System.out.println("Signature failed core validation");
      boolean sv = signature.getSignatureValue().validate(valContext);
      System.out.println("signature validation status: {}" + sv);
      if (!sv) {
        // Check the validation status of each Reference.
        for (Reference reference : signature.getSignedInfo().getReferences()) {
          boolean refValid = reference.validate(valContext);
          System.out.println("ref[{}] validity status: " + refValid);
        }
      } else {
        System.out.println("Signature passed core validation");
      }
    } else {
      System.out.println("Signature validation OK");
    }

    return result;
  }

  /**
   * Write an XML document root node to a file
   *
   * @param doc     XML Document
   * @param outFile output filename
   * @throws FileNotFoundException when input file is not found
   * @throws TransformerException  when an error occurred in the transformation
   */
  public void writeToFile(Document doc, String outFile) throws IOException, TransformerException {

    try (OutputStream os = new FileOutputStream(outFile)) {
      TransformerFactory tf = getTransformerFactory();
      Transformer trans = tf.newTransformer();
      trans.transform(new DOMSource(doc), new StreamResult(os));
    }
  }
}

Note that we need the following dependencies.

<dependency>
  <groupId>jakarta.activation</groupId>
  <artifactId>jakarta.activation-api</artifactId>
  <version>2.1.1</version>
</dependency>
<dependency>
  <groupId>jakarta.xml.bind</groupId>
  <artifactId>jakarta.xml.bind-api</artifactId>
  <version>${jaxb.version}</version>
</dependency>
<dependency>
  <groupId>com.sun.xml.bind</groupId>
  <artifactId>jaxb-impl</artifactId>
  <version>${jaxb.version}</version>
  <scope>runtime</scope>
</dependency>

How to Insert Records in Batch with Spring Data

Ed Legaspi — Sun, 12 Feb 2023 09:03:34 +0000

1. Introduction

When working with a REST API, we often deal with a single entity to create, update, delete, list, and get. But there are also times when we need to manage entities in batches, and this mostly happens in the back office. For example, in an ecommerce platform, promotions are governed by uploading a CSV file in the backend. And this CSV file contains a list of records that needs to be processed.

Let's see how we can process a list of promotions.

2. Solutions for Saving a List of Promotions

For the succeeding examples, let's assume that we have a BaseEntity class that is being extended by PromotionEntity.

2.1 Reading, Processing, and Saving One Promotion at a Time

Not the ideal solution, but I still see this approach in production.

BaseEntity

@Getter
@Setter
@SuperBuilder
@NoArgsConstructor
@AllArgsConstructor
@MappedSuperclass
@JsonInclude(JsonInclude.Include.NON_NULL)
public abstract class BaseEntity implements Serializable {

  @Serial
  private static final long serialVersionUID = 3986494663579679129L;

  public static final int NB_PRECISION = 23;
  public static final int NB_SCALE = 12;

  @Id
  @GeneratedValue(strategy = GenerationType.IDENTITY)
  @Column(name = "id")
  private Long id;

  @Version
  @Column(name = "version")
  private Integer version;
}

And in the service layer, we process the record one by one:

@Transactional(rollbackOn = {SQLException.class})
public void createPromotions(List<Promotion> promotions) {
promotionRepository.saveAll(promotions.stream()
        .map(service2EntityMapper::asPromotionEntity)
        .toList());
        }

Line 1 - it's always a good idea to wrap these kinds of operations (create/delete/update) in a transaction
Line 3 - we stream the list of promotions
Line 4 - we convert each promotion POJO to an entity
Line 5 - we collect the promotion entities
Line 3 - we save a list of promotion entities
Note that the saveAll(List) method does not necessarily save records in batch, as it only calls save(E) underneath.

We need the bare minimum configuration for this setup for the data source.

spring:
  application:
    name: czetsuyatech
  datasource:
    url: jdbc:h2:mem:testdb
    driverClassName: org.h2.Driver
    username: sa
    password:

Summary

It's always a good idea to wrap the create/update/delete operations in a transaction
Use stream when dealing with lists of entities
Use DTOs and POJOs when passing data from one layer to another
For table id, use GenerationType.IDENTITY, when dealing with simple requirements
I would use this approach when I need to process each record in a different transaction. In that case, I need to annotate this method with Transaction.NEVER and call another service that would actually do the data insertion. This means it's possible to only save a fraction of the list due to errors.

2.2 Reading, Processing, and Saving Promotions in Batch

This is the ideal solution, as it efficiently saves a list of records. This is what I will do when dealing with batch inserts.

BaseEntity

@Getter
@Setter
@SuperBuilder
@NoArgsConstructor
@AllArgsConstructor
@MappedSuperclass
@JsonInclude(JsonInclude.Include.NON_NULL)
public abstract class BaseEntity implements Serializable {

  @Serial
  private static final long serialVersionUID = 3986494663579679129L;

  @Id
  @GeneratedValue(generator = "ID_GENERATOR", strategy = GenerationType.AUTO)
  @Column(name = "id")
  private Long id;

  @Version
  @Column(name = "version")
  private Integer version;
}

Line 14 - we are using an id generator that we will define in the promotion entity
Line 20 - we added a version field annotated with @version for optimistic locking

PromotionEntity

@Getter
@Setter
@SuperBuilder
@NoArgsConstructor
@AllArgsConstructor
@Entity
@Table(name = "promotion")
@GenericGenerator(name = "ID_GENERATOR",
    strategy = "org.hibernate.id.enhanced.SequenceStyleGenerator",
    parameters = {@Parameter(name = "sequence_name", value = "promotion_seq")}
)
public class PromotionEntity extends BaseEntity {

  @Column(name = "code")
  private String code;

  // ...
}

Line 12 - we extend the BaseEntity
Line 8 - we create a generic sequence generator that will map to the ID_GENERATOR defined in the BaseEntity

Let's check our revised service layer. I would use a CompletableFuture that I can chain if needed.

@Transactional
public CompletableFuture<Void> createPromotions(Promotions promotions) {

log.debug("Creating promotions");

return CompletableFuture.runAsync(() -> promotionRepository.saveAll(
    promotions.getValues().stream()
        .map(service2EntityMapper::asPromotionEntity)
        .toList())
);
}

Line 1 - annotate the method with Transaction so that it can revert when an error occurred
Line 2 - our service method that returns a CompletableFuture
Line 6 - here, we wrap our saveAll method inside an async block
Line 7 - we stream to the list of promotions
Line 8 - we convert each of the POJO to an entity
Line 9 - we collect the list of entities
Line 6 - we save all the entities, again it doesn't necessarily mean that we're saving in batch

To enable batch saving, we must make changes to our configuration file.

application.xml

spring:
  application:
    name: czetsuyatech.com
  datasource:
    url: jdbc:h2:mem:testdb
    driverClassName: org.h2.Driver
    username: sa
    password:
  jpa:
    database-platform: org.hibernate.dialect.H2Dialect
    properties:
      hibernate:
        jdbc:
          batch_size: 5
        generate_statistics: true
    show-sql: false
  flyway:
    enabled: true
    locations: classpath:db/migrations
    baseline-on-migrate: true
    user: sa
    password: sa
  h2:
    console:
      enabled: true
      path: /db

Line 4 - our h2 data source
Line 14 - you need to define the batch size. Along with sequenced id, it enables batch saving
Line 15 - useful when checking that you have successfully enabled batch
Line 17 - is all about the flyway configuration

Summary

It's always a good idea to wrap the create/update/delete operations in a transaction
Use stream when dealing with lists of entities
Use DTOs and POJOs when passing data from one layer to another
Use a custom sequence generator
Define batch_size hibernate property in application XML
I would use this approach to speed up the processing of records. The downside is that each batch is managed in one transaction, which means failing one record in that batch means the whole set will not be saved.

3. Challenge

Using the above approach, try to save records in different sizes: 100, 1000, 10000, and 1000000. You should be able to see a noticeable improvement in inserts.