DEV Community

loading...
Cover image for Cloud Run vs App Engine: a head-to-head comparison using facts and science

Cloud Run vs App Engine: a head-to-head comparison using facts and science

pcraig3 profile image Paul Craig Updated on ・6 min read

For low-traffic applications, Cloud Run is dramatically cheaper than App Engine.

Abstract

I was hosting a small web app as a side-project and looking to spend less money. I started out using Heroku, then moved to Google’s Cloud Platform. Using rigorous methods and markdown tables, I performed a science-inspired “how much does this cost?” comparison between App Engine and Cloud Run. This study finds that Cloud Run is usually the best option, although if you have money to burn are a “price insensitive consumer,” then App Engine is a bit zippier.

Introduction

Imagine you have a side-project-type web app and you’re looking to host it on Google’s Cloud Platform (GCP) but you don’t want to spend too much ca$h. Which GCP service do 4 out of 5 scientists recommend? Let’s find out.

Background

My incredible journey went basically thus: I built a small express app for upcoming Canadian holidays and wanted cheap but usable hosting. Initially, I was using Heroku’s $7/month Hobby Plan because at the end of the day month, it’s only $7. (ie, that’s like 3 coffees: ‘a coffee’ being the base unit of diminutive purchases.)

Heroku was really easy to get going with, to integrate with GitHub Actions, and to ssh into when I needed to fiddle with something. But around month five, it dawned on me that it was going to cost $7/month for the rest of my life, so I started looking for other options.

Pivoting to Google Cloud Platform (GCP)

GCP was the cloud vendor with the most bonus cash on sign-up, so I figured that was a pretty neutral and unbiased reason to pick it. However, as a hapless first-time user, there are a lot of “solutions” to choose from.

It seems like you’re not a real cloud vendor unless you can bury newcomers under an avalanche of vaguely differentiated products with abstract geometrical logos, so a straightforward question like “where do I host a basic express app?” didn’t have an obvious answer.

Cutting through the media bias with facts and logic, I was able to narrow it down by following the research methodology of googling “google cloud how do I host express app”.

The two options that popped up were:

  1. App Engine
  2. Cloud Run

Both services will run apps and I had an app to run. Seemed perfect: they anticipated me like how I anticipated Canadians are looking for information about holidays.

Methodology

By signing up, I was granted 300 (!!) GCP bucks, and as a long-time government employee I knew this meant I had to find a creative way to spend it before the end of the fiscal year. Are you thinking what I’m thinking? Let’s run a research study!

(This is where the science comes in.)

My research question was “Should I use App Engine or Cloud Run to host my fun but unprofitable app?”, and to investigate that I opted for the immersion method where I would assume the role of a developer trying to host an app on Google Cloud.

Setup

As a precursor, I needed to set up my app on both services simultaneously. For the initial setup, I used the Quickstart material provided by Google at no cost to embedded researchers like me. (Both Quicks-start are pretty easy to follow once you have the gcloud command-line tools installed.)

Overview: App Engine (AE)

On AE, my express app runs as a node process, like booting it up with npm start locally. AE is a traditional hosting platform: it runs continuously and serves requests as they come in. At the end of the month, you pay for the amount of time it was running, which is typically “the entire month”.

Overview: Cloud Run

Cloud Run runs containers, so for each release you have to build a container and push it to GCP. Unlike App Engine, Cloud Run only runs when requests come in, so you don’t pay for time spent idling.

Containerized apps are more portable but not always something you focus on during development. It’s worth noting that the Cloud Run Quickstart provides 9 example Dockerfiles depending on your language of choice. (I used the Node.js one as a basis.)

Simulating traffic

At this point in the study, I had 2 instances of my app running:

  1. In App Engine: https://hols-ae.nn.r.appspot.com/
  2. In Cloud Run: https://hols-hzlcxvebra-ue.a.run.app/

Because real applications have real traffic, I set up a ping service to send requests to each site exactly once every 47 minutes for the rest of time, just like how a Real Human Being™️ would browse.

Having completed my setup, it was time to let the experiment run its course, so I passed the time doing highly academic things like rinsing noobs at dominion.games.

Duration

2 months.

Findings

There were 2 principal findings of the study.

  1. For a low-traffic application, Cloud Run is dramatically cheaper than App Engine
  2. App Engine seems to respond slightly faster

1. Ongoing costs — Cloud Run wins ✅

Cloud Run App Engine Heroku Hobby Plan
Monthly cost $0.09 $11.29 $7.00

Wow.

App Engine runs 24/7 for the entire month whereas Cloud Run only runs when serving requests, and the difference is startling.

Previously, I had been paying $7 a month for Heroku’s Hobby Plan.

  • App Engine would cost me about 50% more
  • Cloud Run costs 99% less, oh my goodness

So basically it’s a blowout win for Cloud Run here.

2. Request latency — App Engine (usually) wins ✅

I also used some online speed test tools to measure the response times of my 2 instances. The results weren’t totally consistent, but App Engine generally responded more quickly.

Pingdom Speed test

(Results of 3 runs from São Paulo)

Cloud Run App Engine
Run 1 632 ms 471 ms
Run 2 485 ms 568 ms
Run 3 562 ms 470 ms
Average 559 ms 503 ms

Here we see App Engine responding on average 56 ms faster than Cloud Run (although in 1 case, Cloud Run was faster). The huge caveat here is that these times vary widely between runs, sometimes tripling or quadrupling depending on Who The F*ck Knows.

WebPageTest

(Results of 3 runs using “3G” download speed.)

Cloud Run App Engine
Run 1 5.217 s 5.010 s
Run 2 5.310 s 4.922 s
Run 3 5.353 s 5.089 s
Average 5.293 s 5.007 s

Again, keep in mind that these numbers shift around between runs.

Why is App Engine faster?

This isn’t totally clear to me, but I can speculate.

The one measurable difference I noticed is that that the total request size from Cloud Run was larger because it doesn’t gzip files by default.

Cloud Run App Engine
Page size 125.8 KB 119.4 KB

The Pingdom Speed Test for Cloud Run recommended I Compress components with gzip, and looking through the requests, my combined .js assets are indeed about 6 KB larger.

Downloading bigger files makes your site slower, but I don’t think that’s the whole story.

The big difference between the two services is that Cloud Run doesn’t run your container unless it’s getting requests. When a request comes in, it does 3 things:

  1. boots up the container
  2. serves the request
  3. shuts down the container

It seems likely that the extra time needed to boot up the container adds to the total request time, leading to an average slower response time from Cloud Run.

Of course, you also save a lot of money doing it this way, so the tradeoff here is whether you care more about optimizing your speed or your cost.

Findings

For me, the findings are decisive. If you’re a hobbyist developer and you want to host your fun app for next-to-free, you should definitely use Google Cloud Run.

However, if money is no object, then you can pay exponentially more per month for a marginal speed boost on App Engine.

Further reading

Discussion (10)

pic
Editor guide
Collapse
bharatsawnani profile image
bharatsawnani

Nice tests and post, but you should specify which environment you're using for GAE.

I primarily code in Java (Haven't deployed with Node.js on GAE so far) and from my experience the Standard environment works similarly to Cloud Run, as it spins up a new instance when a request is made (if there wasn't one already idle). The instance stays idle for 15 mins after that it's shutted down. Google gives you a daily free usage quota of 28 hours for instances. Hence if you tests were runnning once every 47 mins (and the requests didn't require much processing power)... then your daily cost would be 0.00$ as you wouldn't be surpassing the daily free quota.

If your tests were on the Flexible Environment then that's a whole different story as an instance has to be idle all the time and I am not so sure what machine type they start of from there. In Standard the lowest instance is an F1, whicho would have 256mb RAM, which is not much but enough for a simple app.

Collapse
samuelbfavarin profile image
Samuel Favarin

Nice post! In the future would be cool to do a benchmark to compare with a similar AWS service.

Collapse
pcraig3 profile image
Paul Craig Author

It's a good question. At work we use Fargate a lot, which I find a lot more complex than Cloud Run to set up, but it has a similar "serverless container" platform concept as CR does.

Collapse
ashishlogmaster profile image
Ashish “Logmaster” Boston

Your app-engine was setup to "autoscale" hence the instance would stay up constantly costing you $. If you changed it to "basic" auto-scaling, GAE would have auto scale down and stop the instance and costs should be similar to cloud run. Could you pls re-test with this setting so its a more fair comparison. Thanks,
cloud.google.com/appengine/docs/st...

Collapse
msl00 profile image
msl00

The linked documentation could be more clear, but it is not correct to say that "autoscaled" instances are "up constantly costing you $". You linked to the "Instance State" section, and it is saying that "autoscaled" instances will only ever show as being in the "running" state (vs the "stopped" state possible for "manual" or "basic" scaling). This is because "autoscaled" instances are shut down after some time (if no requests come in), not that the instances are running 24/7.

Collapse
dtobias profile image
Dom

Misleading post as the premise is that you pay for App Engine 24/7 which isn't true on the standard instances (predefined languages versions e.g. Go 1.12) only if you choose flex (custom versions). Otherwise you have a point but it's not clear and standard App Engine covers most use cases.

Collapse
jenil777007 profile image
Jenil Calcuttawala

Great post! App Engine standard or flex?

Collapse
_gdelgado profile image
Gio

Really appreciate the commentary on costs 👏👏

Collapse
tranquan profile image
Tran Quan

I am building the same kind of hobby app, and Google lead me here ヽ(^O^)ノ
Thank you very much.

I think I will also use Cloud Run, it's flexible and we can move the whole docker to another cloud later

Collapse
pcraig3 profile image
Paul Craig Author

Yeah, it's a great place to start! Heroku is easier to use but it costs like 100x more money.