The Problem That Inspired This Article
Recently, I encountered an interesting situation. I opened a certain web resource (which I won't name) and tried to view a list of data. The page started loading... and loading... and the browser froze.
After restarting the page and opening DevTools, the picture became clear: the API endpoint was returning all records in a single request - tens of thousands of rows in a JSON response over 50 MB in size. The browser simply couldn't handle it.
Testing the request through Postman confirmed my suspicions:
GET /api/items
Response: [
{ "id": 1, "name": "Item 1" },
{ "id": 2, "name": "Item 2" },
...
{ "id": 50000, "name": "Item 50000" }
]
// 50 MB JSON, response time: 12 seconds
This incident prompted me to write an article about how to properly solve such problems. Using my open-source project finance as an example, I'll show how to implement pagination to avoid these situations from the start.
Target audience: This article is aimed at beginner and mid-level developers who want to learn how to properly design REST APIs with real-world loads in mind. If you've worked with Spring Boot but never added pagination - this article is for you.
The Problem with Non-Paginated Endpoints
When an API endpoint returns all data in a single request, serious problems arise:
For the server:
- High database load
- High memory consumption
- Slow response (seconds instead of milliseconds)
For the client:
- Long loading times
- Large network traffic
- Possible browser freezing
For the business:
- Poor user experience
- High infrastructure costs
- Scalability issues
Solution: Pagination
Pagination is a technique for splitting large datasets into smaller chunks (pages) that are loaded on demand. Instead of returning all 200,000 records at once, the API returns, for example, 20-50 records per page.
Benefits of pagination:
For the server:
- Reduced database load
- Less memory for request processing
- Faster response
For the client:
- Fast data loading
- Less network traffic
- Better UX (user sees data faster)
For the business:
- Lower infrastructure costs
- Ability to scale
- Better system performance
Practical Implementation in Spring Boot
Let's look at an example of pagination implementation in an accounting system prototype. Full code is available on GitHub.
The financial application has endpoints that can potentially return large amounts of data:
-
/api/exchange-rates- historical exchange rates (200,000+ records) -
/api/counterparties- list of counterparties (can grow to thousands of records)
Let's see how to properly implement pagination for such endpoints.
Step 1: Creating Base DTOs for Pagination
First, we need two data structures that will be reusable for all endpoints:
PageMetadata.java - page metadata:
public class PageMetadata {
private int currentPage; // Current page (0-based)
private int totalPages; // Total number of pages
private int pageSize; // Number of records per page
private long totalElements; // Total number of records
private boolean hasNext; // Whether there is a next page
private boolean hasPrevious; // Whether there is a previous page
// Constructors, getters, setters, builder
}
PageResponse.java - data wrapper:
public class PageResponse<T> {
private List<T> content; // Current page data
private PageMetadata metadata; // Pagination metadata
// Constructors, getters, setters, builder
}
Step 2: Updating the Repository
Spring Data JPA already has built-in pagination support through the Pageable interface:
public interface ExchangeRateRepository extends JpaRepository<ExternalExchangeRate, Long> {
// Spring Data automatically provides the method:
// Page<ExternalExchangeRate> findAll(Pageable pageable);
// For custom queries:
Page<ExternalExchangeRate> findByCurrencyFromIdAndCurrencyToId(
Long currencyFromId, Long currencyToId, Pageable pageable);
}
Step 3: Service Layer Implementation
@Service
public class ExternalExchangeRateService {
private final ExternalExchangeRateRepository repository;
private final ExternalExchangeRateMapper mapper;
@Transactional(readOnly = true)
public PageResponse<ExchangeRateResponseDTO> getAllExchangeRates(
int page, int size) {
// Create Pageable with sorting
Pageable pageable = PageRequest.of(page, size,
Sort.by(Sort.Direction.DESC, "exchangeDate", "id"));
// Query database with pagination
Page<ExternalExchangeRate> ratePage = repository.findAll(pageable);
// Map entity → DTO
List<ExchangeRateResponseDTO> dtos = ratePage.getContent()
.stream()
.map(mapper::toResponseDTO)
.toList();
// Collect metadata
PageMetadata metadata = PageMetadata.builder()
.currentPage(ratePage.getNumber())
.totalPages(ratePage.getTotalPages())
.pageSize(ratePage.getSize())
.totalElements(ratePage.getTotalElements())
.hasNext(ratePage.hasNext())
.hasPrevious(ratePage.hasPrevious())
.build();
// Return result
return PageResponse.<ExchangeRateResponseDTO>builder()
.content(dtos)
.metadata(metadata)
.build();
}
}
Step 4: REST Controller
@RestController
@RequestMapping("/api/exchange-rates")
public class ExternalExchangeRateController {
private final ExternalExchangeRateService service;
@GetMapping
@Operation(
summary = "Get all exchange rates",
description = """
Retrieve paginated list of exchange rates, sorted by date (newest first).
Examples:
- GET /api/exchange-rates (first page, default settings)
- GET /api/exchange-rates?page=1 (second page)
- GET /api/exchange-rates?page=0&size=100 (custom page size)
"""
)
public ResponseEntity<PageResponse<ExchangeRateResponseDTO>> getAllExchangeRates(
@Parameter(description = "Page number (0-based)")
@RequestParam(defaultValue = "0") int page,
@Parameter(description = "Items per page")
@RequestParam(defaultValue = "200") int size,
@Parameter(description = "Maximum allowed page size")
@RequestParam(defaultValue = "500") int maxSize) {
// Protection against abuse
if (size > maxSize) {
size = maxSize;
}
PageResponse<ExchangeRateResponseDTO> response =
service.getAllExchangeRates(page, size);
return ResponseEntity.ok(response);
}
}
Step 5: Database Optimization - Indexes
For efficient pagination with sorting, indexes are required:
@Entity
@Table(
name = "external_exchange_rates",
indexes = {
@Index(name = "idx_exchange_rate_date_id",
columnList = "exchange_date, id")
}
)
public class ExternalExchangeRate extends BaseEntity {
// entity fields
}
The composite index (exchange_date, id) provides:
- Fast sorting by date
- Stable sorting (via ID)
- Efficient OFFSET/LIMIT queries
API Results
💡 Live Demo: Want to see the results in practice? Test the API right now through Swagger UI - try different page and size values and see the response speed!
Example of non-paginated endpoint (like that web resource):
GET /api/items
Response Time: 8-12 seconds
Response Size: 45 MB
[
{ "id": 1, "date": "2024-01-01", "rate": 0.92 },
{ "id": 2, "date": "2024-01-02", "rate": 0.93 },
...
{ "id": 200000, "date": "2024-11-20", "rate": 0.91 }
]
Example of paginated endpoint:
GET /api/exchange-rates?page=0&size=200
Response Time: 150-300ms
Response Size: 45 KB
{
"content": [
{ "id": 200000, "date": "2024-11-20", "rate": 0.91 },
{ "id": 199999, "date": "2024-11-19", "rate": 0.90 },
...
{ "id": 199800, "date": "2024-05-15", "rate": 0.89 }
],
"metadata": {
"currentPage": 0,
"totalPages": 1000,
"pageSize": 200,
"totalElements": 200000,
"hasNext": true,
"hasPrevious": false
}
}
Improvement:
- ⚡ Speed: 40x faster (8-12s → 150-300ms)
- 📉 Response size: 1000x smaller (45 MB → 45 KB)
- 💾 Memory: 10x less (~500MB → ~50MB)
Best Practices
1. Reasonable Default Values
// For historical data (exchange rates)
defaultValue = "200"
maxSize = "500"
// For UI-oriented endpoints (counterparties)
defaultValue = "50"
maxSize = "500"
Note: This project uses larger values than standard because it matches the specifics of financial data and system needs. However, for most cases, it's recommended to base on industry standards:
- GitHub API: default 30, max 100
- Twitter API: default 20, max 200
- Stripe API: default 10, max 100
2. Protection Against Abuse
Always set maxSize:
if (size > maxSize) {
size = maxSize;
}
3. Proper Sorting
// For time series - newest first
Sort.by(Sort.Direction.DESC, "exchangeDate", "id")
// For reference data - alphabetical order
Sort.by(Sort.Direction.ASC, "name", "id")
Always add secondary sorting by id for stability.
4. Consistent Response Structure
Use a single PageResponse<T> structure for all endpoints - frontend developers will appreciate it.
Testing
Full test coverage is critical:
@Test
void getAllExchangeRates_ShouldReturnPagedRates() {
// Given
Pageable pageable = PageRequest.of(0, 200);
Page<ExternalExchangeRate> page = new PageImpl<>(rates, pageable, 1000);
when(repository.findAll(any(Pageable.class))).thenReturn(page);
// When
PageResponse<ExchangeRateResponseDTO> result = service.getAllExchangeRates(0, 200);
// Then
assertEquals(200, result.getContent().size());
assertEquals(0, result.getMetadata().getCurrentPage());
assertEquals(5, result.getMetadata().getTotalPages());
assertTrue(result.getMetadata().isHasNext());
}
Tests should verify:
- ✅ Basic pagination
- ✅ Metadata (hasNext, hasPrevious)
- ✅ Empty results
- ✅ Sorting
- ✅ MaxSize limits
View tests:
Test results: 99% line coverage, 97% branch coverage
Conclusions
That web resource that froze could have avoided the problem by simply adding pagination. Implementing pagination in Spring Boot is not technically complex, but it's critical for production systems.
Key takeaways:
- Always add pagination for endpoints that return collections
- Set reasonable defaults (50-200 records)
- Limit maxSize to protect against abuse
- Add indexes for sorting
- Test thoroughly - 99% coverage is a quality standard
Don't wait for your API to crash under load - add pagination at the design stage. Your users (and server) will thank you.
Resources
About the author:
Java Backend Developer with 19+ years of IT experience, building heavy backend financial applications.
Connect:
If you found this article helpful, please leave a reaction ❤️ and follow for more!
Top comments (0)