DEV Community

Philip Stayetski
Philip Stayetski

Posted on

AES-GCM vs ChaCha20-Poly1305: A Practical Comparison for Protocol Designers

If you build anything with transport security — VPNs, tunnels, messaging protocols, or encrypted overlay networks — you've met two dominant AEAD constructions: AES-256-GCM and ChaCha20-Poly1305. Both give you authenticated encryption (confidentiality + integrity in one pass), but they take very different engineering paths to get there.

This is a practical comparison for developers choosing between them: the hardware landscape, the threat models each handles better, and the trade-offs that actually matter in production.

What they are

AES-GCM (Galois/Counter Mode) combines AES block encryption in CTR mode with a GHASH polynomial authenticator. It's been the default for TLS 1.2 and IPsec for over a decade. NIST-standard, FIPS-140-certifiable, and backed by dedicated CPU instructions (AES-NI + PCLMULQDQ) on virtually every modern x86 and ARM chip.

ChaCha20-Poly1305 is a stream cipher (ChaCha20) paired with a Poly1305 MAC. Designed by Daniel J. Bernstein, it does not use AES at all — it operates on 512-bit blocks with ARX (add-rotate-XOR) primitives. Standardised in RFC 8439, mandatory in TLS 1.3, and the default for SSH, WireGuard, and a growing share of HTTPS connections (especially on mobile).

Hardware acceleration: the real divide

The single most important practical difference is whether the CPU has AES-NI.

On x86-64 with AES-NI + PCLMULQDQ, AES-256-GCM encrypts at around 1–3 cycles per byte — extremely fast. On mobile ARM with the ARMv8 Crypto Extensions, the same. With hardware acceleration, AES-GCM is usually faster than ChaCha20-Poly1305.

Without it — on older ARM Cortex-A cores, low-power MCUs, or software-only fallback paths — AES-GCM slows down dramatically. AES's S-box substitution is expensive in pure software. ChaCha20-Poly1305, which runs entirely on ARX operations, has no such penalty. Without hardware AES, ChaCha20-Poly1305 is typically 3–5x faster.

This is why you see ChaCha20-Poly1305 everywhere in mobile-first protocols (WireGuard, TLS to mobile clients). It guarantees good performance on hardware where AES acceleration is absent or disabled.

Side-channel and security posture

Both ciphers are considered cryptographically sound, but their side-channel profiles differ.

AES-GCM has a well-known nonce reuse vulnerability: encrypting two messages with the same 96-bit nonce and key leaks the GHASH authentication key, allowing forgeries. With a 96-bit nonce, a random nonce collision begins to become likely after about 2^32 messages (the birthday bound). For high-volume protocols, this is a real concern — TLS 1.3 mitigates it with record sequence numbers, but not every protocol has that luxury.

GCM is also incrementally slower with short messages in software, because the GHASH polynomial multiplication has setup overhead. OpenSSL and BoringSSL have constant-time GCM implementations for x86, but careful implementation is required.

ChaCha20-Poly1305 is more forgiving. A 96-bit nonce is standard here too, and reuse is equally catastrophic, but the cipher's underlying structure is naturally constant-time on almost any architecture — no lookup tables, no secret-dependent branches. It also has a larger POLY1305 tag (128-bit) with better forgery bounds than GCM's GHASH, though this rarely matters in practice at standard message sizes.

The XChaCha20 variant (extended 192-bit nonce) is worth mentioning: the larger nonce eliminates random-collision risk entirely for most practical workloads, making it much safer for protocols that cannot guarantee unique nonces.

Where each shines

Choose AES-GCM when:

  • You run on x86-64 with AES-NI or ARMv8 Crypto Extensions (the majority of cloud servers, modern laptops, and recent phones)
  • You need FIPS 140 certification
  • You want maximum hardware-accelerated throughput for bulk encryption
  • Your protocol enforces unique nonces (sequence numbers, epoch counters)

Choose ChaCha20-Poly1305 when:

  • You target mixed hardware (older phones, IoT, embedded, SBCs)
  • You want constant-time properties guaranteed by construction
  • Your protocol cannot guarantee unique nonces (consider XChaCha20 for 192-bit nonces)
  • You ship mobile or browser clients where the other endpoint's CPU is unknown
  • You value implementation simplicity and want to audit the cipher yourself

What this means for protocol designers

Many modern transport protocols don't pick one — they offer both as ciphersuites. WireGuard only offers ChaCha20-Poly1305. TLS 1.3 mandates both (TLS_AES_128_GCM_SHA256 and TLS_CHACHA20_POLY1305_SHA256). QUIC and HTTP/3 use both.

The right question is not "which is better?" but "what does your deployment look like?" If you control both ends and know the hardware (a cloud-to-cloud tunnel), AES-256-GCM on AES-NI is the obvious choice. If you're building a P2P protocol where endpoints are unpredictable, ChaCha20-Poly1305 is the safer default.

A real example: Pilot Protocol

Pilot Protocol — an open-source overlay network for AI agents — is a case study in this exact trade-off. It connects agents across arbitrary infrastructure: cloud VMs, local workstations, edge devices, even machines behind NAT. It cannot assume AES-NI on every peer.

Pilot builds its encrypted UDP tunnels with X25519 key exchange and AES-256-GCM for transport payloads. The choice reflects a specific deployment reality: Pilot's daemon runs on servers and developer machines where AES-NI is nearly universal. The ciphersuites are pinned — no negotiation, no downgrade surface.

A protocol like Pilot could equally well use ChaCha20-Poly1305 as an alternative ciphersuite for ARM-based edge deployments. The important thing is that the cipher choice follows the topology and hardware constraints, not fashion or inertia.

Final take

AES-256-GCM and ChaCha20-Poly1305 are both excellent. Neither is "broken" or "wrong." The difference is in engineering trade-offs — hardware dependency, constant-time guarantees, nonce budget, and certification requirements. A well-designed protocol knows its deployment envelope and picks accordingly. The best protocols offer both and let the endpoints negotiate.

Top comments (1)

Collapse
 
circuit profile image
Rahul S

Good comparison, and leading with the hardware-acceleration point is the right call. The one thing I'd add sits right in the gap you flagged yourself — XChaCha20's 192-bit nonce kills random collisions, but the nonce reuse that actually bites protocols in the wild is deterministic: a counter that resets after a reboot, a rollback that replays old state, two instances that both think they own nonce zero. The extended nonce doesn't help any of those, because the collision isn't a birthday-bound accident, it's the same value generated twice on purpose.

That's where AES-GCM-SIV (and the SIV/nonce-misuse-resistant family generally) earns its place — under reuse it degrades to leaking only whether two plaintexts were equal, instead of handing over the GHASH key and letting anyone forge. So imo the first question for a protocol designer isn't AES-NI vs ARX, it's whether you can guarantee nonce uniqueness operationally across reboots and replicas. If you can't, a misuse-resistant mode matters more than the cipher underneath it.