Let me tell you a story that starts in 2013, peaks somewhere around 2019, and ends with me staring at a $4,200 AWS bill at 11pm on a Tuesday.
...
For further actions, you may consider blocking this person and/or reporting abuse
"Boring infrastructure is good infrastructure." Words to live by! It’s easy to get sucked into rewriting everything for the sake of hype, but your point about operational complexity is crucial. 3 weeks of engineering time for 2 people is a real cost, but with a 10x savings on a $4,200/month recurring bill, your ROI hit break-even almost immediately. Great execution.
Exactly! It’s easy to fall into the trap of 'resume-driven development,' but sometimes the most impressive engineering is the kind that just quietly works and saves the company money. Glad you appreciated the ROI breakdown—it's definitely satisfying to see the infrastructure pay for itself so quickly!
The timing of this is perfect. Seeing Cloudflare completely phase out FL1 for their Rust-based FL2 earlier this year really proved that this isn't just a niche optimization anymore—it's the new standard for edge and proxy layers. Go's GC is incredible for rapid development, but constant HTTP buffer allocations will always be its Achilles' heel at scale.
That’s a great observation about the Cloudflare transition. You hit the nail on the head—the GC overhead in Go is fantastic for velocity, but when you're dealing with high-frequency HTTP buffer allocations at the edge, that 'Achilles' heel' becomes impossible to ignore. Rust’s ownership model changes the game entirely for those layers.
"First time you see an Envoy config file you think someone is hazing you."
I felt this in my soul. Moving away from Traefik’s magic annotations into the raw, explicit world of Envoy is a rite of passage. Kudos to you and your co-engineer for surviving those week-one outages and getting it stable!
Haha, glad I’m not the only one who felt like it was hazing! Moving from the 'magic' of annotations to the explicit configuration of Envoy is definitely a rite of passage. Those first few outages were stressful, but they were the best learning experience I’ve had in a long time. Thanks for the kind words!
This is one of the most balanced "Go vs Rust" perspectives I've read. Usually, these articles devolve into tribalism, but you nailed the nuance: Go won the cloud-native revolution because of its concurrency and readability, while Rust is claiming the performance-critical substrate. Excellent breakdown of the memory lifecycle differences.
Thank you! I really wanted to avoid the 'my language is better than yours' debate. At the end of the day, both Go and Rust are phenomenal tools—it’s all about understanding the memory lifecycle and knowing where to apply them. Glad the nuance came through!
As a CFO, I love that ending, haha! "I learned a new programming language" is the ultimate engineering mic drop. Seriously though, dropping the node requirement from multiple t3.large instances down to a single t3.small while flattening the CPU spikes is a masterclass in modern cost optimization.
I’m glad a CFO perspective approves! It’s one thing to talk about performance, but when you can show a massive reduction in the AWS bill while simultaneously flattening those CPU spikes, it’s hard to argue with the results. It was definitely a fun 'mic drop' moment for the team
I’m curious about the trade-offs your team experienced during the rewrite. Specifically, how did you find the ecosystem maturity for K8s tooling in Rust (like kube-rs or custom async runtimes) compared to the battle-tested Go control-plane ecosystem? Also, how are you handling the increased complexity of the codebase for day-to-day maintenance now?
It’s always fascinating to see Rust’s memory efficiency and predictable performance (no GC pauses) yield such massive infrastructure savings when replacing Go in high-throughput network applications. The transition from Go to Rust for an ingress controller makes a ton of sense given how critical low latency and minimal resource footprints are at that layer.
While Go is usually the default for the cloud-native ecosystem due to its low concurrency overhead, your results highlight exactly where it hits its limits—garbage collection pauses and memory footprints under heavy, sustained network I/O. Dropping the GC overhead entirely by moving to Rust clearly paid off here.
Fantastic write-up! That drop from $4,200 to $390 is a massive win, and your breakdown of why it happens (Go's GC latency optimization vs. Rust's deterministic compile-time memory management for proxy allocation lifecycles) is spot on.
It’s refreshing to see a nuanced take that doesn’t just blindly bash Go, but instead highlights the right tool for the right job—Go for orchestrating the control plane, and Rust/C++ for the data plane substrate. Envoy configuration definitely feels like ritual hazing the first time around, but those flat memory metrics make the archaeology completely worth it. Thanks for sharing this!
Thanks, Tahir! I really appreciate the detailed feedback. You summarized the trade-off perfectly—using Go for the control plane and Rust for the high-performance data plane really is the 'sweet spot' for modern architecture. And yes, the archaeology of the Envoy config file is painful, but those flat memory metrics make the headache worth it every single time!
How did the team find the learning curve going from Go to Rust?
Were there any specific Rust libraries (like Axum or Tokio) that made building the new ingress easier, or did you write a lot of the low-level networking from scratch?
This is a fascinating case study! A 10x cost reduction at the ingress level is massive, especially when dealing with high-throughput K8s traffic.