Consume Large JSON Response in Spring

Background

In the previous post Streaming Large JSON Response in Spring. we generated large json as response. In this article, we will consuming that response, without heap memory overflow.

Goal

In this post, the sole target is to consume that large json response and as an exercise do some basic calculation as a proof of concept about receiving the json response.

As a reference we receive following json contents from the server application

{
    "employees" : [
        {
            "empNo":10001,
            "birthDate":"02 September 1953",
            "firstName":"Georgi",
            "lastName":"Facello",
            "gender":"M",
            "hireDate":"26 June 1986"
        }
        ... ~300K employee entries
    ]
}

We will be counting following

• Number of female employee.
• Number of male employee.
• Total number of employee.

NB: The statistics done with the employee list is for demonstration purpose and no way near to real world scenario. Do not take these examples here literally, like counting the number of total, male and female employees from employee list is not the ideal way. Better calculate at database end using aggregate functions.

We will be creating a simple "Spring Web Application" using

• Spring Boot (2.7.4)
• Spring Web.
• Thymeleaf.
• Lombok and Spring Boot Devtools.

Implementation

Start with the enum and pojo declaration

Gender.java

public enum Gender {
    M,F;
}

Employee.java

@Getter
@Setter
@NoArgsConstructor
public class Employee {

    private Long empNo;

    private Date birthDate;

    private String firstName;

    private String lastName;

    private Gender gender;

    private Date hireDate;
}

EmployeeStats.java

@Getter
@Setter
@NoArgsConstructor
public class EmployeeStats {

    private Long totalEmployeeCount;

    private Long femaleEmployeeCount = 0L;

    private Long maleEmployeeCount = 0L;

    private Long reportGenerationDelayInSeconds;

    public void incrementFemaleEmployeeCount() {
        femaleEmployeeCount++;
    }

    public void incrementMaleEmployeeCount() {
        maleEmployeeCount++;
    }
}

Nothing fancy here, just a stats pojo, which will be holding all the statistics with some convenient method for doing the counting.

We will be using RestTemplate for the HTTP communication. So lets configure a bean for RestTemplate

WebConfig.java

@Configuration
public class WebConfig {

    @Bean
    public RestTemplate restTemplate() {
        RestTemplate restTemplate = new RestTemplate();
        for(HttpMessageConverter<?> converter : restTemplate.getMessageConverters()) {
            if(converter instanceof MappingJackson2HttpMessageConverter) {
                MappingJackson2HttpMessageConverter jacksonConverter = (MappingJackson2HttpMessageConverter) converter;
                ObjectMapper objectMapper = jacksonConverter.getObjectMapper();
                objectMapper.setDateFormat(new SimpleDateFormat("dd MMMM YYYY"));
            }
        }
        return restTemplate;
    }

    @Bean
    public ObjectMapper restTemplateObjectMapper(RestTemplate restTemplate) {
        for(HttpMessageConverter<?> converter : restTemplate.getMessageConverters()) {
            if(converter instanceof MappingJackson2HttpMessageConverter) {
                return ((MappingJackson2HttpMessageConverter) converter).getObjectMapper();
            }
        }

        return null;
    }
}

Important thing to notice here, we customised the date format for converter MappingJackson2HttpMessageConverter to align with the json response date format.

An extra bean of ObjectMapper is also exposed from the RestTemplate's converter. We will be needing that next, while deserialising the json response.

Receiving response chunk by chunk

Now the most lucrative part of this whole writing, doing the HTTP request and receive the response. From the previous article Streaming Large JSON Response in Spring it is obvious that, loading the whole response in a pojo will exhaust the heap memory. Here we will not receive the whole response at once, we will connect a JsonParser into the response's input stream and parse received json as per our need. Thus we will be moving along with the whole response without actually loading it fully in the memory.

EmployeeReportService.java

@Service
@RequiredArgConstruct
public class EmployeeReportService {

    private final RestTemplate restTemplate;
    private final ObjectMapper objectMapper;

    public Long fetchAllEmployee(Consumer<Employee> employeeConsumer) {

        String url = "http://localhost:8080/employees?stream=true";

        return restTemplate.execute(url, HttpMethod.GET, null, (response) -> {

            Long employeeCount = 0L;

            JsonParser jsonParser = objectMapper.getFactory().createParser(response.getBody());

            if(jsonParser.nextToken() == JsonToken.START_OBJECT) {
                if(jsonParser.nextFieldName() == "employees") {
                    if(jsonParser.nextToken() == JsonToken.START_ARRAY) {
                        while(jsonParser.nextToken() != JsonToken.END_ARRAY) {
                            Employee employee = jsonParser.readValueAs(Employee.class);
                            employeeConsumer.accept(employee);
                            employeeCount++;
                        }
                    }
                }
            }

            jsonParser.close();

            return employeeCount;
        });

    }
}

Actually what is happening here,

• We initiate the HTTP request using the RestTemplate and pass a ResponseExtractor implementation.
• The ResponseExtractor implementation starts with getting the inputStream from the response body.
• Prepared a JsonParser using the objectMapper configured before and inputStream from the response body.
• Then some trivial checking (starting of object { token check, starting the field "employees" token, starting of the array [ token), then we start reading the employee object one by one, until we find the end of the array token ].
• We pass the parsed employee object to the consumer, to process the employee as per the callee need and doing some trivial counting to return later.
• At the end close() the jsonParser.

Using the response

Now time to calculate the statistics based on the received employee one by one

EmployeeReportService.java

@Service
public class EmployeeReportService {

    .... Previous Code

    public EmployeeStats prepareEmployeeStats() {
        final Long startTime = System.currentTimeMillis();
        final EmployeeStats employeeStats = new EmployeeStats();

        final Long totalEmployeeCount = fetchAllEmployee((employee) -> {
            switch (employee.getGender()) {
                case F:
                    employeeStats.incrementFemaleEmployeeCount();
                    break;
                case M:
                    employeeStats.incrementMaleEmployeeCount();
                    break;
            }
        });

        final Long endTime = System.currentTimeMillis();

        employeeStats.setReportGenerationDelayInSeconds((endTime - startTime) / 1000);
        employeeStats.setTotalEmployeeCount(totalEmployeeCount);

        return employeeStats;
    }
}

Above method is pretty much self explained. It's using our previously declared fetchAllEmployee method to fetch all the employee and counting based on declared rule. To keep track of the required time to prepare the statistics, we also calculated the number of seconds required to generate the statistics.

Now the controller for rendering the statistics into the browser

EmployeeController.java

@Controller
@RequiredArgsConstructor
public class EmployeeController {

    private final EmployeeReportService employeeReportService;

    @GetMapping("/employee-stats")
    public String getEmployeeStats(Model model) {
        EmployeeStats employeeStats = employeeReportService.prepareEmployeeStats();
        model.addAttribute("stat", employeeStats);
        return "employee-stats";
    }
}

and the template

employee-stats.html

... Html

<table border="6">
    <thead>
        <th>Reference</th>
        <th>Value</th>
    </thead>
    <tbody>
        <tr>
            <!-- Because ladies first :) -->
            <td>Female Employee Count</td>
            <td class="value" th:text="${stat.femaleEmployeeCount}"></td>
        </tr>
        <tr>
            <td>Male Employee Count</td>
            <td class="value" th:text="${stat.maleEmployeeCount}"></td>
        </tr>
        <tr>
            <td>Total Employee Count</td>
            <td class="value" th:text="${stat.totalEmployeeCount}"></td>
        </tr>
        <tr>
            <td>Preparation Delay</td>
            <td class="value" th:text="${stat.reportGenerationDelayInSeconds + ' seconds'}"></td>
        </tr>
    </tbody>
</table>

... More Html

And the last thing is

server.port=8081

So our consumer application is running on port 8081.

Output

Both Server & Consumer application is running with jvm arg -Xmx32m to demonstrate the heap size limitation.

Clearly the response took longer than the expected because of the way we are counting these values. This example is just for demonstration purpose and to keep the scope of this writing small.

All the code above can be found into my github repository.

CSV Generation From Large JSON Response in Spring this post coveres more appropriate usage of this streaming based response consuming.