Author's Note: This article was originally written and published by me in English on the AWS Builder Center under the title Beyond File Storage: Using Amazon S3 as a Historical Data Layer. This version is an adaptation of the original content for Brazilian Portuguese. The original publication is available at: Beyond File Storage: Using Amazon S3 as a Historical Data Layer.
When we talk about Amazon S3, the first use case that typically comes to mind is file storage. While this is indeed one of its most well-known applications, limiting S3 to that role means overlooking a significant portion of its potential.
In addition to providing secure and scalable file storage, S3 can become a strategic component of modern data architectures, helping organizations reduce costs, retain historical information for extended periods, and make large volumes of data available for analytical queries without impacting transactional systems.
I work with the Pix product at a credit union, and we frequently face challenges related to the continuous growth in transaction volumes. In this article, I will present a scenario that is quite common among large financial institutions and demonstrate how S3 can be used to address one of the key challenges associated with storing and querying historical data.
The Challenge of Pix Growth
Pix has established itself as Brazil's primary instant payment method. With millions of people performing transactions every day, financial institutions now manage an enormous volume of information that must be stored, audited, and made available when necessary.
In a large financial institution, processing around 10 million transactions per day is not uncommon. When we analyze this volume over several years, the numbers become significant very quickly.
These figures help illustrate a challenge that grows at the same pace as Pix adoption itself: how to store years of historical data without compromising the performance of systems responsible for day-to-day operations.
Although keeping this entire volume in relational databases is technically feasible, the approach begins to present significant challenges as the dataset grows.
The Problem with Keeping Everything in the Transactional Database
Payment systems must be fast and highly available. For this reason, they typically rely on transactional databases, commonly known as OLTP (Online Transaction Processing) systems.
These databases are optimized for recording real-time operations such as transfers, payments, and balance inquiries. However, when the goal shifts to analyzing billions of historical records, the situation changes dramatically.
Assuming a five-year retention period, an institution could accumulate approximately 18 billion records. In this context, seemingly simple queries can become extremely expensive.
Consider a request to identify all transactions performed during a specific period or generate a detailed statement covering previous years. Even with proper partitioning strategies and indexing, the database would still need to process an enormous amount of information.
Some of the most common impacts include:
- Reading large volumes of historical data.
- Competition for processing and storage resources with daily transaction workloads.
- Increased latency in critical applications.
- Longer execution times for analytical queries.
Depending on the available infrastructure, queries like these may take several minutes to complete because OLTP databases were designed for transactional workloads rather than large-scale analytical processing.
This is precisely where a Data Lake-based archival strategy begins to make sense.
Entering the Data Lake World
An increasingly common approach is to keep only "hot" data in the transactional database, meaning data that is frequently accessed by applications.
In the case of Pix, this could represent the most recent 60 or 90 days of transactions.
Older records, often referred to as cold data, can be moved to Amazon S3, creating a dedicated historical storage layer.
In modern architectures known as Lakehouses, S3 typically serves as the central storage layer. The Lakehouse concept combines characteristics of both Data Lakes and Data Warehouses, enabling the storage of massive datasets with flexibility while simultaneously providing advanced analytical capabilities.
Under this model, historical data no longer competes for resources with transactional systems and can instead be consumed by specialized analytics platforms.
The architecture would look something like this:
S3, Parquet, and Databricks Working Together
After moving historical data into S3, a highly effective approach is to store it in formats such as Parquet or Delta Lake.
Parquet is a columnar storage format designed specifically for analytical workloads. Unlike traditional relational databases, which often read entire rows, a columnar format allows queries to retrieve only the columns required for a specific operation.
To further optimize query performance, historical datasets can be partitioned by year, month, or transaction date. This approach reduces the amount of data that analytical engines need to scan, improving query efficiency and reducing processing costs.
On top of this storage layer, platforms such as Databricks can execute distributed queries using Apache Spark, a processing engine designed for large-scale data workloads.
In practice, this means historical queries no longer need to run in the same environment responsible for processing daily Pix transactions.
Instead, the workload is moved to a dedicated analytical infrastructure.
This architectural shift brings several important benefits.
Workload Isolation
Analytical queries no longer compete with transactional workloads.
While the database continues processing millions of payments in real time, the analytics cluster independently processes historical data.
More Efficient Data Access
Because the data is stored in a columnar format, only the required columns are read.
If a query needs only the transaction date and transaction amount, there is no need to load dozens of additional fields.
This significantly reduces data scanning volumes.
Automatic Optimizations
Platforms such as Databricks provide caching and optimization mechanisms that accelerate recurring queries.
In addition, features such as Z-Ordering and Liquid Clustering help organize data physically, making filtering and search operations more efficient.
Cost Reduction
Another major advantage relates to storage economics.
Maintaining large historical datasets in high-performance relational databases typically requires costly infrastructure, whereas S3 offers a much more economical model for long-term retention.
As a result, organizations can balance performance and cost without sacrificing data availability.
Comparing the Two Approaches
When the two models are compared side by side, the benefits become even more apparent.
This comparison shows that we are not simply talking about a new place to store files. We are discussing an architectural shift capable of separating responsibilities and allowing each technology to do what it was designed for.
If your organization is facing this challenge, consider the following:
- Audit how much of your historical data truly needs to remain in the transactional database.
- Evaluate your current storage costs versus potential savings with S3.
- Run a pilot using a subset of data, such as the last two years, stored in Parquet and queried through Databricks.
- Measure query performance and operational costs before and after migration.
Although this article uses Databricks as an example for analytical processing, the same architectural pattern can be implemented using AWS-native services such as Amazon Athena, AWS Glue, Amazon EMR, and Amazon Redshift Spectrum, depending on the organization's requirements and analytical workloads.
Conclusion
The continued growth of Pix is creating new challenges for financial institutions. As billions of transactions are generated every year, storing and retrieving historical data requires more sophisticated strategies than simply increasing the capacity of relational databases.
In this context, Amazon S3 evolves from being much more than a file storage service. It becomes a strategic component of modern data architectures, capable of supporting long-term retention, reducing infrastructure costs, and enabling scalable access to historical information. By combining S3 with analytical technologies such as Parquet, Delta Lake, and distributed query engines, organizations can separate transactional and analytical workloads, allowing each platform to focus on what it was designed to do best.
This approach not only improves performance and operational efficiency but also provides greater flexibility for future growth. As data volumes continue to expand, keeping years of information exclusively in transactional databases becomes increasingly expensive and difficult to scale. A Data Lake or Lakehouse strategy can help organizations balance performance, governance, and cost while maintaining access to valuable historical data.
Ultimately, the value of Amazon S3 goes far beyond storing files. When used as a historical data layer, it becomes an architectural enabler for building scalable, resilient, and cost-effective solutions that can support the next generation of data-intensive applications.
Disclaimer: The figures presented in this article are hypothetical estimates intended solely to illustrate the architectural concepts discussed. They are based on assumptions and author-provided inputs and were not derived from production systems, real-world workloads, or measured benchmarks.
While the assumptions were informed by publicly available statistics published by Banco Central do Brasil (Bacen) regarding Pix, the values should not be interpreted as official figures, guaranteed projections, performance benchmarks, or results from an actual implementation. Context source: Banco Central do Brasil (Bacen) – Pix in Numbers. Accessed on October 21, 2026.About the Author
Felipe Camargos is a Software Engineer Specialist and Tech Lead at Sicredi, focused on Pix, cloud architecture, and large-scale distributed systems.
Top comments (1)
hot stays put, cold goes to parquet. clean split.