Akinwale Folorunsho Habibullah

Posted on Jul 11, 2023 • Edited on Jul 13, 2023

Spring Integration with Mail support full stack app.

#spring #springintegration #java #react

In this tutorial, you will learn to use Spring Integration to receive emails from Gmail, create an API endpoint using Spring MVC and use React to fetch data from the backend API. In this tutorial, we will keep track of payments into a Bank account, store the information in a MongoDB database and show a list of transactions in a React app.

Scenario

Assuming a small business owner needs his or her account department to keep track of payments made by customers or clients for goods or services without relying on any platform to do that. For privacy reasons, the business owner also cannot grant the accounts department access to the business's email.

We will create a full-stack React and Spring boot app. We will also create a Spring integration workflow that listens for new transaction notification emails and persists the transaction data in a MongoDB database. A React app will consume the transaction data and show a list of transactions to the account department staff.

Introduction to Spring Integration.

Spring Integration is a simple model for building enterprise integration solutions. Spring Integration extends the Spring programming model to support Enterprise Integration Patterns. Spring Integration provides lightweight messaging within Spring-based applications and supports external systems via declarative adapters.

We will use Spring Boot and other Spring projects like Spring MVC, Spring Data JPA, and Spring Integration.

Environment setup

I will create the Spring application using Spring initializer at https://start.spring.io. You can also start a new project using Spring Tool Suite. Spring Tool Suite is a Spring development environment and provides extensions for Eclipse IDE and Visual Studio Code.

Download your new Spring project by clicking the Generate button on the Spring initializer web page. Next is to unzip the folder and open the project in your favourite IDE of Text Editor. I currently use VSCode with Spring Tool Suite.

Here's what our pom.xml list of dependencies should look like;



<dependencies>
  <dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-integration</artifactId>
  </dependency>
  <dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-web</artifactId>
  </dependency>
  <dependency>
    <groupId>org.springframework.integration</groupId>
    <artifactId>spring-integration-http</artifactId>
  </dependency>
  <dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-devtools</artifactId>
    <scope>runtime</scope>
    <optional>true</optional>
  </dependency>
  <dependency>
    <groupId>org.projectlombok</groupId>
    <artifactId>lombok</artifactId>
    <optional>true</optional>
  </dependency>
  <dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-test</artifactId>
    <scope>test</scope>
  </dependency>
  <dependency>
    <groupId>org.springframework.integration</groupId>
    <artifactId>spring-integration-test</artifactId>
    <scope>test</scope>
  </dependency>
</dependencies>

We need to add other dependencies which do not have a Spring Boot starter project yet. We should also remove spring-integration-http because we will not need it. spring-integration-http is an endpoint adapter that enables Spring Integration to integrate with external systems using the HTTP protocol. We will replace spring-integration with spring-integration-mail dependency.



<dependencies>
  <dependency>
    <groupId>org.eclipse.angus</groupId>
    <artifactId>angus-mail</artifactId>
  </dependency>
  <dependency>
    <groupId>com.sun.mail</groupId>
    <artifactId>jakarta.mail</artifactId>
  </dependency>
  <dependency>
    <groupId>org.springframework.integration</groupId>
    <artifactId>spring-integration-mail</artifactId>
    <version>6.1.0</version>
  </dependency>
  .... list of dependencies
</dependencies>

"angus-mail" dependency is an SMTP protocol provider for Jakarta Mail, while the "jakarta-mail" dependency sends and receives emails via SMTP, POP3 and IMAP protocols.

Create Integration Flow

Let us create a basic integration flow which connects to Gmail and logs a message to the standard output saying we've received an email.
There are different ways to create integration flows in Spring Integration

via XML configuration
via Java configuration
via Java configuration with a DSL (Domain Specific Language)

I will use Java configuration with a DSL option because it is terser and easier to read.

I will create a new file, "MailIntegrationConfig.java", in com. akinwalehabib.transactiontracker package. This file will contain our basic integration flow configuration.



package com.akinwalehabib.transactiontracker;

import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.integration.dsl.IntegrationFlow;
import org.springframework.integration.dsl.Pollers;
import org.springframework.integration.mail.dsl.Mail;

@Configuration
public class MailIntegrationConfig {

  @Bean
  public IntegrationFlow mainIntegration(
    EmailProperties props
  ) {
    return IntegrationFlow
      .from(
        Mail.imapInboundAdapter(props.getImapUrl())
          .shouldDeleteMessages(false)
          .simpleContent(true)
          .autoCloseFolder(false),
        e -> e.poller(
          Pollers.fixedDelay(props.getPollRate())
        )
      )
      .handle(message -> {
        System.out.println("New message received: " + message);
      })
      .get();
  }
}

We created a class and annotated it @configuration, which indicates to Spring that it is a configuration class that will provide beans to the Spring application context.

In the mailIntegration method inside MailIntegrationConfig class, we injected EmailProperties class, which we need to create next. We will use the EmailProperties class to inject configuration properties from the application.yml file.



package com.akinwalehabib.transactiontracker;

import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.stereotype.Component;

import lombok.Data;

@Data
@ConfigurationProperties(prefix = "email")
@Component
public class EmailProperties {

  private String username;
  private String password;
  private String host;
  private String port;
  private String mailbox;
  private long pollRate = 30000;

  public String getImapUrl() {
    return String.format("imaps://%s:%s@%s:%s/%s", this.username, this.password, this.host, this.port, this.mailbox);
  }
}

We annotated EmailProperties with @data. @data is from the Lombok dependency, which generates our setter, getter, toString and hashCode, thereby saving us some keystrokes. @Component indicates to Spring that this is a component, making it a candidate for component scanning or auto-detection. @ConfigurationProperties annotation injects configuration properties from the Spring environment.

The attributes username, password, host, port, mailbox and pollRate are the details we will inject from our Spring environment, which we will add to the application.yml file.

There are different ways to provide configuration properties to Spring, but we will use the application.yml file in the "src/main/resources" folder. Go ahead and rename the "application.properties" file to "application.yml".



email:
  username: akinwalehabib
  password: somerandomstring
  host: imap.gmail.com
  port: 993
  mailbox: INBOX
  pollRate: 30000

Replace "username" with your email id without the @domainname, then add your app password. To work with Gmail, you need to create an app password and use that in place of your actual password. See instructions here: https://support.google.com/mail/answer/185833?hl=en.

Go ahead and run your application, and then send an email to the configured email address. The spring application will log a message to the terminal when an email arrives.

💡 New message received: GenericMessage [payload=org.springframework.integration.mail.AbstractMailReceiver$IntegrationMimeMessage@249c8b65, headers={closeableResource=org.springframework.integration.mail.AbstractMailReceiver$$Lambda$916/0x0000000801155be0@3b6d02de, id=64b0e7d7-70f6-9432-260d-a1fc0941874b, timestamp=1688931330198}]

Because we are only interested in keeping track of payments, we only care about transaction notification emails from the bank. In Spring Integration, we can place a Filter in an integration flow to allow or disallow messages from proceeding to the next step in the flow.
We will add a filter that checks the subject of the email. If it contains the keyword "Transaction Alert", we will pass the message to the next step in our integration flow. We can also add other checks in our filter method, like checking if the sender is our bank's notification email address etc.



....
import jakarta.mail.Message;
import jakarta.mail.MessagingException;

@Configuration
public class MailIntegrationConfig {

  private final String SUBJECT_KEYWORDS = "TRANSACTION ALERT";

  @Bean
  public IntegrationFlow mainIntegration(
    EmailProperties props
  ) {
    return IntegrationFlow
      .from(
        Mail.imapInboundAdapter(props.getImapUrl())
          .shouldDeleteMessages(false)
          .simpleContent(true)
          .autoCloseFolder(false),
        e -> e.poller(
          Pollers.fixedDelay(props.getPollRate())
        )
      )
      .<Message>filter((Message) -> {
        boolean containsKeyword = false;
        try {
          containsKeyword = Message.getSubject().toUpperCase().contains(SUBJECT_KEYWORDS);
        } catch (MessagingException e1) {
          e1.printStackTrace();
        }

        return containsKeyword;
      })
      .handle(message -> {
        System.out.println("New message received: " + message);
      })
      .get();
  }
}

The filter method uses an anonymous function which returns a boolean value. The integration flow will only pass the message to the next step in the integration flow if the returned value from the anonymous function is True.

If you send an email whose subject doesn't contain the keyword "Transaction alert", you should see the following message in your terminal.

💡 The message [GenericMessage [payload=org.springframework.integration.mail.AbstractMailReceiver$IntegrationMimeMessage@20625346, headers={closeableResource=org.springframework.integration.mail.AbstractMailReceiver$$Lambda$953/0x000000080118bc10@9bc0049, id=bdb31273-a3fd-3d6e-ef2e-4a9f98942254, timestamp=1688935506023}]] has been rejected in filter: bean 'mainIntegration.filter#0' for component 'mainIntegration.org.springframework.integration.config.ConsumerEndpointFactoryBean#0'; defined in: 'class path resource [com/akinwalehabib/transactiontracker/MailIntegrationConfig.class]'; from source: 'bean method mainIntegration’

The next step is to transform the received message into an Object. We will create a domain model and then persist it in the database using Spring Data MongoDB.

Let's add Spring data MongoDB dependency to our project and then add mongoDB configuration.



<dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-data-mongodb</artifactId>
</dependency>

Then update our application.yml file.



....
spring:
  data:
    mongodb:
      host: localhost
      port: 27017
      database: emailintegration

Create our domain model.




package com.akinwalehabib.transactiontracker;

import java.time.LocalDateTime;
import java.util.regex.Matcher;
import java.util.regex.Pattern;

import org.springframework.data.annotation.Id;
import org.springframework.data.mongodb.core.mapping.Document;

import lombok.Data;
import lombok.NoArgsConstructor;
import lombok.ToString;

@Data
@NoArgsConstructor
@ToString
@Document(collection = "emails")
public class Email {

  @Id
  private String id;
  private String email;
  private String subject;
  private Double amount;
  private String remarks;
  private LocalDateTime receiptDate;
  private String content;

  public Email (String email, String subject, String content) {
    this.email = email;
    this.subject = subject;
    this.content = content.toUpperCase();

    this.amount = retrieveAmount(content);
    this.receiptDate = retrieveReceiptDate(content);
    this.remarks = retrieveRemarks(content);
  }

  public double retrieveAmount(String content) {
    Pattern pattern = Pattern.compile("Amount : NGN [1-9]\\d?(?:,\\d{3})*(?:\\.\\d{2})");
    Matcher matcher = pattern.matcher(content);
    String amountString = "";

    if (matcher.find()) {
      String substring = matcher.group();
      String[] substringParts = substring
        .split(":");                                        
      String[] substringAmountPart = substringParts[1]
        .trim()
        .split(" ");                                        
      amountString = substringAmountPart[1].replace(",", "");
      double value = Double.parseDouble(amountString);
      return value;
    }

    return 0.0;
  }

  public String retrieveRemarks(String content) {
    Pattern pattern = Pattern.compile("Remarks :([\\s]+\\w+\\s+[\\w]+)+");
    Matcher matcher = pattern.matcher(content);
    String remarks = "";

    if (matcher.find()) {
      String substring = matcher.group();
      remarks = substring.split(":")[1]
        .trim();
    }

    return remarks;
  }

  public LocalDateTime retrieveReceiptDate(String content) {
    Pattern pattern = Pattern.compile("Time of Transaction : (\\d+-\\d+-\\d+ \\d+\\d+:\\d+)");
    Matcher matcher = pattern.matcher(content);
    LocalDateTime receiptDate = LocalDateTime.now();

    if (matcher.find()) {
      String substring = matcher.group();                                 
      String[] DateTimeParts = substring.split(":");                
      String[] DateParts = DateTimeParts[1].trim().split("-"); 

      int day = Integer.parseInt(DateParts[0]);
      int month = Integer.parseInt(DateParts[1]);
      int year = Integer.parseInt(DateParts[2].substring(0, DateParts[2].length() - 3));

      int hour = Integer.parseInt(DateParts[2]
        .substring(DateParts[2].length() - 2)
        .trim());
      int minute = Integer.parseInt(DateTimeParts[2]);
      receiptDate = LocalDateTime.of(year, month, day, hour, minute);
    }

    return receiptDate;
  }
}

We annotated the Email domain model with @data, @noArgsConstructor, and @ToString from Lombok dependency. We also added @Document(collection = "emails") annotation from "spring-data-mongodb", which specifies the database collection we will store our document into.

In the email domain model, we want to store the subject, the amount received, the transaction remarks and the receipt date. We will retrieve the amount, the transaction remarks and the receipt date from the email content, and that's why we have the retrieveAmount, retrieveRemarks, and retrieveReceiptDate methods.

We also need to create an interface that extends MongoRepository so our application can save our Email domain to the database and also retrieve information from the database.



package com.akinwalehabib.transactiontracker;

import org.springframework.data.mongodb.repository.MongoRepository;

public interface EmailRepository extends MongoRepository<Email, String>{}

We should create a Class to transform our incoming transaction notification email into an Email domain object. We'll then create a transformer in our integration flow, which uses the newly created class.



package com.akinwalehabib.transactiontracker;

import java.io.IOException;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.integration.mail.transformer.AbstractMailMessageTransformer;
import org.springframework.integration.support.AbstractIntegrationMessageBuilder;
import org.springframework.integration.support.MessageBuilder;
import org.springframework.stereotype.Component;

import com.akinwalehabib.transactiontracker.Email;

import jakarta.mail.BodyPart;
import jakarta.mail.Message;
import jakarta.mail.MessagingException;
import jakarta.mail.internet.ContentType;
import jakarta.mail.internet.InternetAddress;
import jakarta.mail.internet.MimeMultipart;

@Component
public class EmailTransformer extends AbstractMailMessageTransformer<Email> {

  private static Logger log = LoggerFactory.getLogger(EmailTransformer.class);
  // private boolean textIsHtml = false;

  @Override
  protected AbstractIntegrationMessageBuilder<Email> doTransform(Message mailMessage) {
    Email email = processPayload(mailMessage);
    return MessageBuilder.withPayload(email);
  }

  private Email processPayload(Message mailMessage) {
    try {
      String subject = mailMessage.getSubject();
      String email = ((InternetAddress) mailMessage.getFrom()[0]).getAddress();
      String content = getTextFromMessage(mailMessage);

      return parseEmail(email, subject, content);
    } catch (MessagingException e) {
      log.error("MessagingException: {}", e);
    } catch (Exception e) {
      log.error("IOException: {}", e);
    }

    return null;
  }

  private String getTextFromMessage(Message message) throws IOException, MessagingException {
    String result = "";
    if (message.isMimeType("text/plain")) {
      result = message.getContent().toString();
    } else if (message.isMimeType("multipart/*")) {
      MimeMultipart mimeMultipart = (MimeMultipart) message.getContent();
      result = getTextFromMimeMultipart(mimeMultipart);
    }
    return result;
  }

  private String getTextFromMimeMultipart(MimeMultipart mimeMultipart) throws IOException, MessagingException {
    int count = mimeMultipart.getCount();
    if (count == 0) throw new MessagingException("Multipart with no body parts not supported.");

    boolean multipartAlt = new ContentType(mimeMultipart.getContentType()).match("multipart/alternative");
    if (multipartAlt) {
      return getTextFromBodyPart(mimeMultipart.getBodyPart(count - 1));
    }

    String result = "";
    for (int i = 0; i < count; i++) {
        BodyPart bodyPart = mimeMultipart.getBodyPart(i);
        result += getTextFromBodyPart(bodyPart);
    }
    return result;
  }

  private String getTextFromBodyPart(BodyPart bodyPart) throws IOException, MessagingException {
    String result = "";
    if (bodyPart.isMimeType("text/plain")) {
      result = (String) bodyPart.getContent();
    } else if (bodyPart.isMimeType("text/html")) {
      String html = (String) bodyPart.getContent();
      result = org.jsoup.Jsoup.parse(html).text();
    } else if (bodyPart.getContent() instanceof MimeMultipart){
      result = getTextFromMimeMultipart((MimeMultipart)bodyPart.getContent());
    }

    return result;
  }

  private Email parseEmail(String senderEmailAddress, String subject, String content) {
    Email email = new Email(senderEmailAddress, subject, content);
    return email;
  }
}

The EmailTransformer is lengthy. It extends AbstractMailMessageTransformer and then overrides the AbstractIntegrationMessageBuilder method. It retrieves the subject and content from an email message and then creates an Object from the Email domain model class. The Email domain class is responsible for retrieving the amount, remarks, and receipt date from the email content passed to it.

We need to add another dependency to our application which we use to parse HTML and extract the text content in the Email Transformer class.



<dependency>
        <groupId>org.jsoup</groupId>
        <artifactId>jsoup</artifactId>
        <version>1.16.1</version>
</dependency>

Then add the transformer to our integration flow. Note how we inject the EmailTransformer class into the method, and then use in our integration flow. We also injected EmailRepository and used it in our handle’s callback function to persist the transaction details in the database.



@Bean
  public IntegrationFlow mainIntegration(
    EmailProperties props,
    EmailTransformer emailTransformer,
    EmailRepository emailRepository
  ) {
    return IntegrationFlow
      .from(
        Mail.imapInboundAdapter(props.getImapUrl())
          .shouldDeleteMessages(false)
          .simpleContent(true)
          .autoCloseFolder(false),
        e -> e.poller(
          Pollers.fixedDelay(props.getPollRate())
        )
      )
      .<Message>filter((Message) -> {
        boolean containsKeyword = false;
        try {
          containsKeyword = Message.getSubject().toUpperCase().contains(SUBJECT_KEYWORDS);
        } catch (MessagingException e1) {
          e1.printStackTrace();
        }

        return containsKeyword;
      })
      .transform(emailTransformer)
      .handle(message -> {
        Email email = (Email) message.getPayload();
        emailRepository.save(email);
      })
      .get();
  }

Here is a sample transaction notification email from my bank. The email consists of different MIMEBodyParts, and the main content is HTML.

Go ahead and run your application, then send an email to your configured email address. Ensure the email subject contains "Transaction alert". Also, ensure the email's content has the "remarks", "amount", and the "time of transaction" as shown above.

After running the application, check the database, and you should have the transaction details stored in the database.

Create API endpoint

We will create an API endpoint to expose our list of transactions stored in the MongoDB database. We already added the "spring-boot-starter-web" dependency through the Spring initializer.
Create a new file titled "EmailIntegrationAPI.java". This new file will contain our API endpoint.



package com.akinwalehabib.transactiontracker;

import java.util.List;

import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

@RestController
@RequestMapping(path = "/api/payments",
                produces = "application/json")
public class EmailIntegrationAPI {

  private EmailRepository emailRepository;

  public EmailIntegrationAPI(EmailRepository emailRepository) {
    this.emailRepository = emailRepository;
  }

  @GetMapping
  public List<Email> getPaymenets() {
    List<Email> payments = emailRepository.findAll();
    return payments;
  }
}

Run the application and visit http://localhost:8080/api/payments using your favourite browser. You should have a JSON response containing the received payment notification emails.

Create React client

Let us create a web client to interact with our newly created API. We will use "create-react-app" to quickly create a React application inside the src/frontend folder. Ensure your Spring application is running before proceeding.



npx create-react-app src/frontend && cd src/frontend

Add a "proxy" key in package.json, which helps avoid CORS issues and redirects unknown requests to configured host or port during development.



"proxy": "http://localhost:8080"

Let us install MUI component library. MUI provides a DataGrid component that is ideal for our use case.



npm install @mui/material @mui/styled-engine-sc styled-components @mui/x-data-gridnpm install @fontsource/roboto @mui/x-data-grid                                                                                                                  ─╯



npm install @fontsource/roboto @mui/x-data-grid

I will create a new component titled Payments. The Payments component will be in the src/folder/src folder. This component will fetch all payment data from our API and then populate a DataGrid in the DOM.




import React, { useEffect, useState } from 'react'
import {
  Box,
  Skeleton
} from '@mui/material';
import { DataGrid } from '@mui/x-data-grid';

const columns = [
  { field: 'id', headerName: 'ID', width: 70 },
  { field: 'amount', headerName: 'Amount', width: 130, type: 'number' },
  { field: 'receiptDate', headerName: 'Receipt Date', width: 130 },
  { field: 'remarks', headerName: 'Remarks', width: 700 }
];

function PaymentsSkeleton() {
  return (
    <>
      <Skeleton variant="rectangular" animation="wave" width={"100%"} height={"80vh"} />
    </>
  )
}

function Payments() {
  const [payments, setPayments] = useState([]);
  const [loading, setLoading] = useState(true);

  useEffect(() => {
    async function getTransactions() {
      let response = await fetch("/api/payments");
      return await response.json();
    }

    getTransactions()
      .then(payments => {
        setPayments(payments);
        setLoading(false);
      })
      .catch(err => console.error('Err: ' + err));
  }, [])

  if (loading) {
    return <PaymentsSkeleton />
  }

  return (
    <Box>
      <DataGrid
        columns={columns}
        rows={payments}
        initialState={{
          pagination: {
            paginationModel: { page: 0, pageSize: 20 },
          },
        }}
        pageSizeOptions={[ 5, 10, 15, 20, 50, 100 ]}
        checkboxSelection
      />
    </Box>
  );
}

export default Payments

Edit the App.js file in the src/frontend/src folder to display the Payment component.



import * as React from 'react'
import './App.css';
import Payments from './Payments';
import { Container, CssBaseline, Typography } from '@mui/material';

function App() {
  return (
    <>
      <CssBaseline />
      <Container>
        <Typography variant="h2" gutterBottom>
          Spring Integration - Mail
        </Typography>

        <Typography variant="subtitle1" gutterBottom>
          Payments received
        </Typography>

        <hr />

        <Payments />
      </Container>
    </>
  ) 
  ;
}

export default App;

Go to http://localhost:3000 using your browser, your webpage should look like this.

Conclusion

We have learned how to create a Spring integration flow using the Spring integration mail endpoint. We used Spring data MongoDB to persist our domain models. We also created an API with a GET endpoint. In conclusion, we developed a React and MUI-based web client to interact with our backend API.

Spring Integration is great for developing enterprise integration solutions using Enterprise Integration Patterns. Learn more at https://docs.spring.io/spring-integration/docs/current/reference/html/index.html.

I beleive that with Open Banking API, applications should be able to integrate with banking services and listen for events such as debits into the account.

Please feel free to share your thoughts, particularly challenges you’ve faced using Spring Integration.