The solution to prevent duplicate requests that I want to talk about here is that when a user manipulates an API feed or any data source, in reality, they only manipulate once, but for any reason, such as intentionally by the user or maybe by a hacker, to cause system data errors.
To prevent this from happening, we need to build a deduplication solution. In the scope of this article, I will implement deduplication based on Redis and Spring Boot 3.
The idea of doing it in sequence would be like this:
- Get some data fields in the Request Body that the user sends, the purpose is to create a Redis key. Which field to use depends on the business needs, as well as the system architecture that is responding to consider the choice.
- Build the key in some optional format, then hash it again with MD5 (using MD5 here is optional, depending on your needs). If you use it, consider using Fast MD5 for faster speed.
- Every time the user calls the API, the Redis key will be checked. If it exists, a duplicate data error will be returned. If not, the logic will continue to be processed.
- When inserting a key into Redis, an Expired Time must be configured. In the framework of the article, I set it to about 40 seconds for an easy demo.
That’s the idea, but the actual implementation will require a few more techniques, which I will mention later. Let’s build the project and test it first.
Project structure
In this project, I using Spring Boot 3.3.4, Java 17, Spring AOP
Here is the detailed code implementation of each part
package com.cafeincode.demo.aop;
import java.lang.annotation.Documented;
import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;
@Target({ElementType.METHOD})
@Retention(RetentionPolicy.RUNTIME)
@Documented
public @interface PreventDuplicateValidator {
String[] includeFieldKeys() default {};
String[] optionalValues() default {};
long expireTime() default 10_000L;
}
PreventDuplicateValidator I declare it as an annotation, here there are three data fields:
includeFieldKeys: to declare the list of fields that are needed to generate the key based on the fields in the Request Body.
optionalValues: a list of values that need to be attached to the key. I implemented this field for the purpose of flexibility in preventing duplication. You can add any data as you like.
expireTime: is the key expiration time value, the default is 10 seconds.
package com.cafeincode.demo.aop;
import com.cafeincode.demo.enums.ErrorCode;
import com.cafeincode.demo.exception.DuplicationException;
import com.cafeincode.demo.exception.HandleGlobalException;
import com.cafeincode.demo.utils.Utils;
import com.fasterxml.jackson.core.type.TypeReference;
import com.fasterxml.jackson.databind.ObjectMapper;
import java.util.Arrays;
import java.util.Collections;
import java.util.Map;
import java.util.Objects;
import java.util.stream.Collectors;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.aspectj.lang.ProceedingJoinPoint;
import org.aspectj.lang.annotation.Around;
import org.aspectj.lang.annotation.Aspect;
import org.springframework.data.redis.connection.RedisStringCommands;
import org.springframework.data.redis.core.RedisCallback;
import org.springframework.data.redis.core.RedisTemplate;
import org.springframework.data.redis.core.types.Expiration;
import org.springframework.stereotype.Component;
/**
* author: hungtv27
* email: hungtvk12@gmail.com
* blog: cafeincode.com
*/
@Aspect
@Component
@RequiredArgsConstructor
@Slf4jj
public class PreventDuplicateValidatorAspect {
private final RedisTemplate redisTemplate;
private final ObjectMapper objectMapper;
@Around(value = "@annotation(preventDuplicateValidator)", argNames = "pjp, preventDuplicateValidator")
public Object aroundAdvice(ProceedingJoinPoint pjp, PreventDuplicateValidator preventDuplicateValidator)
throws Throwable {
var includeKeys = preventDuplicateValidator.includeFieldKeys();
var optionalValues = preventDuplicateValidator.optionalValues();
var expiredTime = preventDuplicateValidator.expireTime();
if (includeKeys == null || includeKeys.length == 0) {
log.warn("[PreventDuplicateRequestAspect] ignore because includeKeys not found in annotation");
return pjp.proceed();
}
//extract request body in request body
var requestBody = Utils.extractRequestBody(pjp);
if (requestBody == null) {
log.warn(
"[PreventDuplicateRequestAspect] ignore because request body object find not found in method arguments");
return pjp.proceed();
}
//parse request body to map<String, Object>
var requestBodyMap = convertJsonToMap(requestBody);
//build key redis from: includeKeys, optionalValues, requestBodyMap
var keyRedis = buildKeyRedisByIncludeKeys(includeKeys, optionalValues, requestBodyMap);
//hash keyRedis to keyRedisMD5: this is Optional, should be using Fast MD5 hash to replace
var keyRedisMD5 = Utils.hashMD5(keyRedis);
log.info(String.format("[PreventDuplicateRequestAspect] rawKey: [%s] and generated keyRedisMD5: [%s]", keyRedis,
keyRedisMD5));
//handle logic check duplicate request by key in Redis
deduplicateRequestByRedisKey(keyRedisMD5, expiredTime);
return pjp.proceed();
}
private String buildKeyRedisByIncludeKeys(String[] includeKeys, String[] optionalValues, Map<String, Object> requestBodyMap) {
var keyWithIncludeKey = Arrays.stream(includeKeys)
.map(requestBodyMap::get)
.filter(Objects::nonNull)
.map(Object::toString)
.collect(Collectors.joining(":"));
if (optionalValues.length > 0) {
return keyWithIncludeKey + ":" + String.join(":", optionalValues);
}
return keyWithIncludeKey;
}
public void deduplicateRequestByRedisKey(String key, long expiredTime) {
var firstSet = (Boolean) redisTemplate.execute((RedisCallback<Boolean>) connection ->
connection.set(key.getBytes(), key.getBytes(), Expiration.milliseconds(expiredTime),
RedisStringCommands.SetOption.SET_IF_ABSENT));
if (firstSet != null && firstSet) {
log.info(String.format("[PreventDuplicateRequestAspect] key: %s has set successfully !!!", key));
return;
}
log.warn(String.format("[PreventDuplicateRequestAspect] key: %s has already existed !!!", key));
throw new DuplicationException(ErrorCode.ERROR_DUPLICATE.getCode(), ErrorCode.ERROR_DUPLICATE.getMessage());
}
public Map<String, Object> convertJsonToMap(Object jsonObject) {
if (jsonObject == null) {
return Collections.emptyMap();
}
try {
return objectMapper.convertValue(jsonObject, new TypeReference<>() {
});
} catch (Exception ignored) {
return Collections.emptyMap();
}
}
}
Here PreventDuplicateValidatorAspect is advice, implementing logic for annotation PreventDuplicateValidator, I use Around Advice for flexibility
The logic implementation in the above code is described in the order of steps below:
- First, we will need to extract the request body from the API.
- Parse request body into Map format.
- Build raw keys from defined data fields.
- Build MD5 key
- Check duplicate requests by key
- If the key already exists in Redis, then throw an exception.
- If the key does not exist in Redis, insert the key into Redis, add the expired time parameter, and then continue the logic of the main function through pjp.proceed()
package com.cafeincode.demo.config;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.datatype.jsr310.JavaTimeModule;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Primary;
import org.springframework.data.redis.connection.RedisConnectionFactory;
import org.springframework.data.redis.connection.RedisStandaloneConfiguration;
import org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory;
import org.springframework.data.redis.core.RedisTemplate;
@Configuration
public class BeanConfig {
@Value("${redis.host}")
private String redisHost;
@Value("${redis.port}")
private int redisPort;
@Bean(name = "objectMapper")
@Primary
public ObjectMapper objectMapper() {
ObjectMapper mapper = new ObjectMapper();
mapper.registerModule(new JavaTimeModule());
return mapper;
}
@Bean
public RedisConnectionFactory redisConnectionFactory() {
var config = new RedisStandaloneConfiguration(redisHost, redisPort);
return new LettuceConnectionFactory(config);
}
@Bean
@Primary
public RedisTemplate<Object, Object> redisTemplate(RedisConnectionFactory redisConnectionFactory) {
var template = new RedisTemplate<>();
template.setConnectionFactory(redisConnectionFactory);
return template;
}
}
BeanConfigI added bean configuration for ObjectMapper and Redis connection bean
package com.cafeincode.demo.dto;
import java.io.Serializable;
import lombok.AllArgsConstructor;
import lombok.Data;
import lombok.NoArgsConstructor;
import lombok.experimental.SuperBuilder;
@Data
@AllArgsConstructor
@NoArgsConstructor
@SuperBuilder
public class BaseResponse<T> implements Serializable {
public static final String OK_CODE = "200";
public static final String OK_MESSAGE = "Successfully";
private String code;
private String message;
private T data;
public static <T> BaseResponse<T> ofSucceeded(T data) {
BaseResponse<T> response = new BaseResponse<>();
response.code = OK_CODE;
response.message = OK_MESSAGE;
response.data = data;
return response;
}
}
BaseResponse is the response class that returns results via API. Large companies as well as standard systems define fields: code , message , and data in this class (maybe with different names, but not very important).
We can add other fields depending on usage needs, such as metadata , request_id , etc.
package com.cafeincode.demo.dto;
import java.time.Instant;
import lombok.Data;
@Data
public class ProductDto {
private String productId;
private String productName;
private String productDescription;
private String transactionId;
private Instant requestTime;
private String requestId;
}
package com.cafeincode.demo.enums;
import lombok.AllArgsConstructor;
import lombok.Getter;
@AllArgsConstructor
@Getter
public enum ErrorCode {
ERROR_DUPLICATE("CF_275", "Duplicated data, please try again later");
private final String code;
private final String message;
}
package com.cafeincode.demo.exception;
import lombok.AllArgsConstructor;
import lombok.Builder;
import lombok.Getter;
import lombok.Setter;
import org.springframework.http.HttpStatus;
@Getter
@Setter
@AllArgsConstructor
@Builder
public class DuplicationException extends RuntimeException {
private String code;
private String message;
private HttpStatus httpStatus;
public DuplicationException(String code, String message) {
this.code = code;
this.message = message;
httpStatus = HttpStatus.BAD_REQUEST;
}
}
package com.cafeincode.demo.exception;
import java.util.HashMap;
import java.util.Map;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.ControllerAdvice;
import org.springframework.web.bind.annotation.ExceptionHandler;
import org.springframework.web.servlet.mvc.method.annotation.ResponseEntityExceptionHandler;
@ControllerAdvice
public class HandleGlobalException extends ResponseEntityExceptionHandler {
@ExceptionHandler(DuplicationException.class)
private ResponseEntity<?> handleError(Exception ex) {
//TODO: you should custom more here
Map<String, String> body = new HashMap<>();
body.put("code", ((DuplicationException) ex).getCode());
body.put("message", ex.getMessage());
return new ResponseEntity<>(body, HttpStatus.BAD_REQUEST);
}
}
In this class HandleGlobalException, I will handle DuplicationException, the firing in the processing logic from PreventDuplicateValidatorAspect.
package com.cafeincode.demo.service;
import com.cafeincode.demo.dto.ProductDto;
public interface IProductService {
ProductDto createProduct(ProductDto dto);
}
package com.cafeincode.demo.service;
import com.cafeincode.demo.dto.ProductDto;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Component;
@Component
@Slf4j
@RequiredArgsConstructor
public class ProductService implements IProductService {
@Override
public ProductDto createProduct(ProductDto dto) {
//TODO: more logic here
return null;
}
}
You can add more logic if needed; I just need to return null to serve the demo purpose.
package com.cafeincode.demo.utils;
import jakarta.xml.bind.DatatypeConverter;
import java.lang.annotation.Annotation;
import java.lang.reflect.Method;
import java.security.MessageDigest;
import lombok.extern.slf4j.Slf4j;
import org.aspectj.lang.ProceedingJoinPoint;
import org.aspectj.lang.reflect.MethodSignature;
import org.springframework.web.bind.annotation.RequestBody;
@Slf4j
public class Utils {
private Utils() {
}
public static Object extractRequestBody(ProceedingJoinPoint pjp) {
try {
for (int i = 0; i < pjp.getArgs().length; i++) {
Object arg = pjp.getArgs()[i];
if (arg != null && isAnnotatedWithRequestBody(pjp, i)) {
return arg;
}
}
} catch (Exception ex) {
log.error("", ex);
}
return null;
}
private static boolean isAnnotatedWithRequestBody(ProceedingJoinPoint pjp, int paramIndex) {
var method = getMethod(pjp);
var parameterAnnotations = method.getParameterAnnotations();
for (Annotation annotation : parameterAnnotations[paramIndex]) {
if (RequestBody.class.isAssignableFrom(annotation.annotationType())) {
return true;
}
}
return false;
}
private static Method getMethod(ProceedingJoinPoint pjp) {
MethodSignature methodSignature = (MethodSignature) pjp.getSignature();
return methodSignature.getMethod();
}
public static String hashMD5(String source) {
String res = null;
try {
var messageDigest = MessageDigest.getInstance("MD5");
var mdBytes = messageDigest.digest(source.getBytes());
res = DatatypeConverter.printHexBinary(mdBytes);
} catch (Exception e) {
log.error("", e);
}
return res;
}
}
The class Utils includes logic functions to extract the request body from ProceedingJoinPoint and the MD5 hash function
redis:
host: localhost
port: 6379
spring:
application:
name: product-service
server:
port: 8888
configure application-local.yml
version: "3.2"
services:
redis:
container_name: demo-service-redis
image: redis:6.2.5
ports:
- '6379:6379'
package com.cafeincode.demo.controller;
import com.cafeincode.demo.aop.PreventDuplicateValidator;
import com.cafeincode.demo.dto.BaseResponse;
import com.cafeincode.demo.dto.ProductDto;
import com.cafeincode.demo.service.ProductService;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;
@RestController
@Slf4j
@RequestMapping("/products")
@RequiredArgsConstructor
public class ProductController {
private final ProductService productService;
@PostMapping
@PreventDuplicateValidator(
includeFieldKeys = {"productId", "transactionId"},
optionalValues = {"CAFEINCODE"},
expireTime = 40_000L)
public BaseResponse<?> createProduct(@RequestBody ProductDto request) {
return BaseResponse.ofSucceeded(productService.createProduct(request));
}
}
In this main controller section, I declare to use annotation with the parameter values above: PreventDuplicateValidator
- includeFieldKeys : markup will take two fields productIdand transactionId in the request body as input to generate key
- optionalValues : option value, I declare here CAFEINCODE
- expireTime : data lifetime in Redis cache, I set it to 40 seconds.
Okay, now let’s run the project and test it:
For MacOS and Windows, you need to turn on Docker Desktop first, then run the command docker-compose up -din Terminal.
For Ubuntu machines, you need to install Docker first, then run the above command.
I use a Macbook, and it’s already turned on, so I just need to turn it on to use
redis docker
check connection Redis
Check if the connection to Redis is ok, launch the application
config profile local, jdk
start application spring boot
You open Postman to test, I leave the request body below for you to easily copy and practice.
{
"productId": "hungtv27-test-001",
"productName": "CAFEINCODE",
"productDescription": "Threat identify buy war manage little friend south really chair",
"transactionId": "cd076846-ff28-4307-8524-3eb6e1809838",
"requestTime": 1696069378367,
"requestId": "{{$randomUUID}}"
}
Click Sendand follow the results
response when first call
validate success, init key to redis
Check the console log and see the message with the MD5 key: 6C518A2B1666005572EDFC8240A130F2does not exist in Redis, so it will be initialized successfully for the first time and set the expiration time to 40 seconds. Now I will check the data in Redis.
MD5 key in Redis
The key 6C518A2B1666005572EDFC8240A130F2has been successfully initialized in Redis. Now we will continue to call the API one more time to check the result. The expectation is to return an errorCF_275
response when call second
View the console log to see if the key 6C518A2B1666005572EDFC8240A130F2 already exists in Redis, so it will return error CF_275 to the client.
So we have completed the implementation of duplicate prevention based on Redis and Spring AOP. In this article, there are some conclusions that you need to consider as follows:
- Select appropriate parameter fields in the Request Body to be used as an input source to create a key; you should ignore time-type fields such as createTime or updateTime.
- Set the expiration time value to suit the project's business needs.
- Consider whether to hash MD5 or not. If you want to optimize performance, you can remove or choose to use Fast MD5 (I do not use it in this article).
Finally, after implementing the complete logic, all we need to do is declare annotation on the controllers that we need to use. Setting the data fields is very flexible, so we rarely need to modify anything further.
Thanks, before you go:
👏 If you have any better solutions, please comment below, we will discuss and learn from each other.
👏 Please clap for the story and follow the author, 👉👉👉 hungtv27
👏 Please share your questions or insights in the comments section below.
Originally published at https://cafeincode.com on October 1, 2024.
Top comments (0)