Improve Performance with 64K Memory Pages on AWS Graviton Processors

If you're running memory intensive workloads on AWS Graviton processors, you may want to investigate 64K pages. One commonly overlooked optimization is switching from the default 4K memory pages to 64K pages. This kernel configuration change can reduce memory management overhead and improve performance for some workloads.

This article explains what memory pages are and shows how to switch between 4K and 64K pages on Ubuntu 24.04.

What Are Memory Pages?

Memory pages are the standardized containers used to organize and move data around. The operating system and the CPU's Memory Management Unit (MMU) use a page table to track which virtual memory addresses map to physical RAM locations. This is how operating systems like Linux provide virtual memory that is larger than the physical memory of the system.

To speed up these lookups, CPUs use a fast cache called a Translation Lookaside Buffer (TLB). Every memory access first checks the TLB. If the mapping isn't there (a TLB miss), the CPU must perform a "page table walk" to fetch it from main memory, which can involve multiple memory accesses and add latency.

On x86 systems, 4K pages are standard. AWS Graviton processors support multiple page sizes. The Arm architecture supports 4K, 16K, and 64K pages, but Linux distributions typically support 4k and 64k. This flexibility lets you optimize for your specific workload.

NOTE: If you want to try 16K pages you can run Asahi Linux on Apple Silicon.

The Performance Impact of Page Size

Changing page size creates a cascading effect on system performance:

With 4K pages, you need more TLB entries to map the same amount of memory. This increases TLB misses and page-table-walk overhead. With 64K pages, each TLB entry covers 16x more memory, which reduces TLB pressure and can improve TLB hit rates.

Disk I/O and DMA operations often perform better with larger pages because fewer page boundaries are crossed during transfers. This means fewer interrupts and larger DMA bursts.

The kernel's page management structures scale with the number of pages. Fewer pages mean less memory spent on bookkeeping.

The drawback of larger pages is that for small, varied allocation patterns, more memory is consumed. This is known as internal fragmentation. For example, if a 2KB process is allocated a 64K page, 62KB of that page's memory is wasted.

A Note on Huge Pages vs. Base Page Size

It's important to distinguish the 64K page size on Arm from the "Transparent Huge Pages" (THP) feature in Linux. THP is a system that tries to dynamically allocate large 2MB pages on top of a standard 4K base page size. The 64K page support on Graviton changes the base page size for all memory management, which is a fundamental, system-wide setting.

Which Workloads Benefit from 64K Pages?

64K pages are most effective for applications that work with large, contiguous memory regions:

• Databases: Large buffer pools and cache systems (PostgreSQL, MySQL, Redis)
• Big Data Processing: Spark, Hadoop, and data analytics workloads
• Machine Learning: Training and inference with large datasets
• Video Processing: Encoding, transcoding, and streaming applications
• High-Performance Computing: Scientific simulations and numerical computing
• In-Memory Caching: Memcached, Redis with large datasets

Conversely, stick with 4K pages for:

• Web servers handling many small requests
• Microservices with small memory footprints
• Applications with highly fragmented memory access patterns

How to Switch to 64K Pages on AWS Graviton

You can experiment with 64K pages in a non-production environment to see if they help your applications.

Ubuntu 24.04 on AWS Graviton

Ubuntu makes this easy with a pre-built 64K kernel package.

Step 1: Check your current page size

getconf PAGESIZE
uname -r

You should see:

4096
6.14.0-1016-aws

Step 2: Install the 64K kernel

sudo apt-get update
sudo apt-get install -y linux-generic-64k

Step 3: Configure GRUB to use the 64K kernel by default

echo "GRUB_FLAVOUR_ORDER=generic-64k" | sudo tee /etc/default/grub.d/local-order.cfg
sudo update-grub
sudo reboot

Step 4: Verify the change

After reboot:

getconf PAGESIZE
uname -r

You should now see:

65536
6.8.0-48-generic-64k

To revert to 4K pages:

echo "GRUB_FLAVOUR_ORDER=generic" | sudo tee /etc/default/grub.d/local-order.cfg
sudo update-grub
sudo reboot

You can find instructions for more Linux distributions in the Arm Learning Path Explore performance gains by increasing the Linux kernel page size on Arm.

Measuring the Impact

After switching to 64K pages, benchmark your application to measure the impact. Key metrics to monitor:

• TLB miss rate: Use perf stat to measure TLB misses
• Application throughput: Measure requests/second or transactions/second
• Latency: Monitor p50, p95, and p99 latencies
• Memory usage: Check for any increase in memory consumption

Performance Gains

Performance gains depend on the specific workload characteristics, but the potential gains make this optimization worth testing for suitable workloads.

Conclusion

AWS Graviton processors offer optimization opportunities not available on x86 systems. Switching to 64K memory pages is a kernel change that can deliver performance improvements for memory-intensive workloads without making any code changes.

It is important to understand your workload characteristics and test thoroughly. If you're working with large datasets, databases, or high-throughput applications on AWS Graviton, testing 64K pages is recommended.

If you are running AWS Graviton instances, make sure to check out the AWS Graviton Technical Guide for more optimization tips.