DEV Community: Adam McCrea

Power Tools for Analyzing your Heroku logs

Adam McCrea — Tue, 17 Aug 2021 20:14:49 +0000

Your log data is a treasure-trove of information about your application, but it can be overwhelming. This post will dig into several strategies for extracting metrics and other helpful information from your logs. We’ll start with the basics of the heroku logs command, then we’ll dig into the real fun using a tool called Angle Grinder.

How to view your Heroku logs

heroku logs on its own just prints the most recent logs from your app and then exits. Generally that's not very useful. I almost always want the -t (or --tail) option to continually tail my logs. Additionally I usually want it scoped to a specific dyno process, so I’ll include -d router, -d web, or -d worker so I’m only seeing relevant logs.

Here's how I would tail my router logs:

heroku logs -t -d router

2021-07-28T16:23:07.870849+00:00 heroku[router]: at=info method=POST path="/api/REDACTED/v2/reports?dyno=web.8&pid=4" host=api.railsautoscale.com request_id=0ce66277-877c-4d4f-91c4-2c1075089b41 fwd="3.84.54.241,172.70.34.122" dyno=web.7 connect=1ms service=156ms status=204 bytes=358 protocol=https
2021-07-28T16:23:07.774247+00:00 heroku[router]: at=info method=POST path="/api/REDACTED/v2/reports?dyno=web.3&pid=81" host=api.railsautoscale.com request_id=fe46b69d-8938-4d41-a566-4c837050f6da fwd="3.85.98.203,172.69.62.61" dyno=web.14 connect=1ms service=14ms status=204 bytes=358 protocol=https
2021-07-28T16:23:07.627308+00:00 heroku[router]: at=info method=POST path="/api/REDACTED/v2/reports?dyno=web.1&pid=11" host=api.railsautoscale.com request_id=f5b69be4-8283-48f6-b683-30c051b4f51d fwd="34.232.107.232,172.70.42.94" dyno=web.11 connect=0ms service=327ms status=204 bytes=358 protocol=https
2021-07-28T16:23:07.740752+00:00 heroku[router]: at=info method=POST path="/api/REDACTED/v2/reports?dyno=web.4&pid=4" host=api.railsautoscale.com request_id=89d2da0d-e1d8-484d-99f2-bf26921ba9a5 fwd="3.249.54.29,162.158.158.123" dyno=web.11 connect=0ms service=354ms status=204 bytes=358 protocol=https
2021-07-28T16:23:07.881220+00:00 heroku[router]: at=info method=POST path="/api/REDACTED/v2/reports?dyno=web.1&pid=24539" host=api.railsautoscale.com request_id=37171239-4524-46cf-9dce-7cdda8bd4ace fwd="3.95.158.194,172.70.34.173" dyno=web.20 connect=2ms service=34ms status=204 bytes=358 protocol=https
2021-07-28T16:23:07.743590+00:00 heroku[router]: at=info method=POST path="/api/REDACTED/v2/reports?dyno=web.1&pid=50" host=api.railsautoscale.com request_id=bb063a51-8d83-4284-b26a-db31c3f4b485 fwd="54.204.194.19,172.70.34.122" dyno=web.8 connect=1ms service=19ms status=204 bytes=358 protocol=https

Scoping it further

Even with the -d filter, it’s usually way too much data to be useful, so I’ll want to filter it even more. This is where chaining with grep is useful.

Here I'll tail my worker logs, but only show the entries that include "StatWorker":

heroku logs -t -d worker | grep StatWorker

2021-07-28T16:28:11.718082+00:00 app[worker.18]: pid=4 tid=1fzk class=StatWorker jid=bae74463b8a3c977aaaac6b4 args=3061,2021-07-28T16:27:50Z measure#sidekiq.queued=1069 tag#worker=StatWorker tag#queue=default INFO: start
2021-07-28T16:28:11.763375+00:00 app[worker.18]: pid=4 tid=1fy4 class=StatWorker jid=2c4de245191802b57344bae8 args=2344,2021-07-28T16:27:50Z measure#sidekiq.queued=933 tag#worker=StatWorker tag#queue=default elapsed=0.181 INFO: done
2021-07-28T16:28:11.782176+00:00 app[worker.18]: pid=4 tid=1fy4 class=StatWorker jid=c704e40d09cf93b628d023f3 args=3000,2021-07-28T16:27:50Z measure#sidekiq.queued=1129 tag#worker=StatWorker tag#queue=default INFO: start
2021-07-28T16:28:11.809741+00:00 app[worker.18]: pid=4 tid=1fzk class=StatWorker jid=bae74463b8a3c977aaaac6b4 args=3061,2021-07-28T16:27:50Z measure#sidekiq.queued=1069 tag#worker=StatWorker tag#queue=default elapsed=0.092 INFO: done
2021-07-28T16:28:11.820178+00:00 app[worker.18]: pid=4 tid=1fzk class=StatWorker jid=7af1852ff673a9f747d8f4ad args=1920,2021-07-28T16:27:50Z measure#sidekiq.queued=1171 tag#worker=StatWorker tag#queue=default INFO: start
2021-07-28T16:28:11.719832+00:00 app[worker.20]: pid=4 tid=13c4g class=StatWorker jid=c531a39bf63886c8ce386c13 args=3826,2021-07-28T16:27:50Z measure#sidekiq.queued=922 tag#worker=StatWorker tag#queue=default elapsed=0.149 INFO: done
2021-07-28T16:28:11.721947+00:00 app[worker.20]: pid=4 tid=13c4g class=StatWorker jid=4e2884278d99fb616eb452b1 args=1546,2021-07-28T16:27:50Z measure#sidekiq.queued=1073 tag#worker=StatWorker tag#queue=default INFO: start
2021-07-28T16:28:11.761234+00:00 app[worker.20]: pid=4 tid=8rk60 class=StatWorker jid=9f54c9b8418801e4274f0fe7 args=2191,2021-07-28T16:27:50Z measure#sidekiq.queued=928 tag#worker=StatWorker tag#queue=default elapsed=0.184 INFO: done

This uses a "pipe" to send the output of heroku logs as input to the grep utility. Grep searches plain text data using regular expressions. We're not using any regexp syntax in this example, but we certainly could.

Chaining heroku logs -t with grep is how I’m most often interacting with my Heroku logs, but sometimes I need more power.

Extracting specific values

It's time to meet Angle Grinder, a real-time log analysis power tool. It can parse many formats, extract values, and even perform aggregations. Let’s just dive right in with an example.

Heroku’s router logs are formatted the same for every Heroku app, so let’s use those. Here’s the same router log output we saw above:

heroku logs -t -d router

2021-07-28T16:23:07.870849+00:00 heroku[router]: at=info method=POST path="/api/REDACTED/v2/reports?dyno=web.8&pid=4" host=api.railsautoscale.com request_id=0ce66277-877c-4d4f-91c4-2c1075089b41 fwd="3.84.54.241,172.70.34.122" dyno=web.7 connect=1ms service=156ms status=204 bytes=358 protocol=https
2021-07-28T16:23:07.774247+00:00 heroku[router]: at=info method=POST path="/api/REDACTED/v2/reports?dyno=web.3&pid=81" host=api.railsautoscale.com request_id=fe46b69d-8938-4d41-a566-4c837050f6da fwd="3.85.98.203,172.69.62.61" dyno=web.14 connect=1ms service=14ms status=204 bytes=358 protocol=https
2021-07-28T16:23:07.627308+00:00 heroku[router]: at=info method=POST path="/api/REDACTED/v2/reports?dyno=web.1&pid=11" host=api.railsautoscale.com request_id=f5b69be4-8283-48f6-b683-30c051b4f51d fwd="34.232.107.232,172.70.42.94" dyno=web.11 connect=0ms service=327ms status=204 bytes=358 protocol=https
2021-07-28T16:23:07.740752+00:00 heroku[router]: at=info method=POST path="/api/REDACTED/v2/reports?dyno=web.4&pid=4" host=api.railsautoscale.com request_id=89d2da0d-e1d8-484d-99f2-bf26921ba9a5 fwd="3.249.54.29,162.158.158.123" dyno=web.11 connect=0ms service=354ms status=204 bytes=358 protocol=https
2021-07-28T16:23:07.881220+00:00 heroku[router]: at=info method=POST path="/api/REDACTED/v2/reports?dyno=web.1&pid=24539" host=api.railsautoscale.com request_id=37171239-4524-46cf-9dce-7cdda8bd4ace fwd="3.95.158.194,172.70.34.173" dyno=web.20 connect=2ms service=34ms status=204 bytes=358 protocol=https
2021-07-28T16:23:07.743590+00:00 heroku[router]: at=info method=POST path="/api/REDACTED/v2/reports?dyno=web.1&pid=50" host=api.railsautoscale.com request_id=bb063a51-8d83-4284-b26a-db31c3f4b485 fwd="54.204.194.19,172.70.34.122" dyno=web.8 connect=1ms service=19ms status=204 bytes=358 protocol=https

You’ll notice that the metadata is all formatted as [key]=[value]. This format is called logfmt. It's embraced heavily by Heroku, and it’s one of the many formats that Angle Grinder understands.

Go follow the Angle Grinder installation instructions then come back to follow along.

Let's use Angle Grinder's logfmt parsing for Heroku router logs.

heroku logs -t -d router | agrind '* | logfmt'

[2021-07-28T16:50:11.654197+00:00=None][at=info][bytes=358][connect=1ms][dyno=web.2][fwd=54.154.157.222,162.158.159.109][heroku[router]:=None][host=api.railsautoscale.com][method=POST][path=/api/REDACTED/v2/reports?dyno=web.3&pid=79][protocol=https][request_id=d4ecf840-666b-430f-81c3-082b4e58c876][service=29ms][status=204]
[2021-07-28T16:50:11.450697+00:00=None][at=info][bytes=358][connect=4ms][dyno=web.18][fwd=34.233.123.109,172.69.62.81][heroku[router]:=None][host=api.railsautoscale.com][method=POST][path=/api/REDACTED/v2/reports?dyno=web.5&pid=242][protocol=https][request_id=1599b9f0-17ec-48ae-8341-5d917bcbd598][service=19ms][status=204]
[2021-07-28T16:50:11.644107+00:00=None][at=info][bytes=358][connect=1ms][dyno=web.16][fwd=3.84.186.26,172.70.38.19][heroku[router]:=None][host=api.railsautoscale.com][method=POST][path=/api/REDACTED/v2/reports?dyno=web.1&pid=9][protocol=https][request_id=2bef6858-458f-47f9-9e97-8c35c4d9ebef][service=12ms][status=204]

Okay, that's really not any better than before. We can see that the key-value pairs were parsed, but it's still just a wall of text.

Let's try again, this time using the --format option to only display the "connect" and "service" metrics from the router logs.

heroku logs -t -d router | agrind '* | logfmt' --format '{connect} {service}'

2ms 9ms
1ms 160ms
1ms 15ms
0ms 6ms
4ms 192ms

Now we're getting somewhere! Let's expand the formatting string to display even more data, along with some defined widths so we have control over the alignment. We'll also split the command into multiple lines to make it more readable.

heroku logs -t -d router | agrind \
  '* | logfmt' \
  --format '{dyno:<6} {connect:6} {service:7} {status:4} {method:5}  {path}'

web.3     0ms    52ms  204  POST  /api/REDACTED/v2/reports?dyno=web.1&pid=37
web.23    1ms    17ms  204  POST  /api/REDACTED/v2/reports?dyno=web.2&pid=4
web.4     1ms    40ms  204  POST  /api/REDACTED/v2/reports?dyno=web.3&pid=186
web.1     1ms    17ms  204  POST  /api/REDACTED/v2/reports?dyno=web.1&pid=86
web.13    1ms    10ms  204  POST  /api/REDACTED/v2/reports?dyno=web.21&pid=65

Beautiful! We've broken the wall of text into a nicely-aligned, human-readable stream of output. You can use this to watch your web traffic in real-time. From this we could quickly see if a particular dyno is struggling, or if a particular path is being requested more frequently than we expect.

How does Angle Grinder work?

Okay, let's peel apart that last example.

First we tail Heroku logs using the Heroku CLI, just like before:

heroku logs -t -d router

Then we pipe the output to agrind (just like we were piping to grep before). agrind takes a single argument—called the "query"—which tells Angle Grinder how to process the incoming text. Queries can get complex as we'll soon see, but this example keeps it simple:

heroku logs -t -d router | agrind '* | logfmt'

Here, the query is * | logfmt, and we enclose it in single quotes to separate it from the rest of the command. It might look a bit odd seeing the "pipe" character within the query string, so take care to not confuse it with a command line pipe.

The Angle Grinder query syntax is a composition of an initial filter with a series of operators. Here's how this query breaks down:

* – This is the initial filter, and it tells Angle Grinder to operate on every input line.
logfmt – This is an operator, and it extracts key/value fields using the logfmt syntax.

The first operator is generally a parser. Next we'll see how subsequent operators would then operate on the extracted fields.

✉️ Want notified when we write more articles like this one? Sign up for the Mastering Heroku mailing list! Less than one email per month, always relevant, and no sales pitches.

Aggregating your log data in real-time

With a steady volume of requests, what we've seen so far can still be an overwhelming amount of information. Depending on what we're looking for, we can use Angle Grinder's aggregate operators to summarize log data in real-time.

For example, here's a snapshot of 50th and 95th percentile service times by dyno:

heroku logs -t -d router | agrind \
  '* | logfmt | p50(service), p95(service), count by dyno | sort by p50 desc'

dyno         p50        p95        _count
------------------------------------------------------
web.2        57         369        622
web.3        40         287        622
web.1        36         114        598
web.7        18         306        612
web.5        15         93         641
web.6        13         41         631
web.4        9          25         576

Now our Angle Grinder query has gotten a bit more complex, with several operations chained together:

* – Operate on every input line.
logfmt – Extract fields using the logfmt parser.
p50(service), p95(service), count by dyno – Calculate three aggregates grouped by the "dyno" field. This assumes a "dyno" field was extracted via logfmt, which it was. The output of this operation is three fields: "p50", "p95", and "_count".
sort by p50 desc – Sort the aggregate results by the "p50" field.

We could change the order of these fields, rename them using "as", or query multiple fields:

heroku logs -t -d router | agrind \
  '* | logfmt | count as count, p95(connect) as connect, p95(service) as service by dyno | sort by p50 desc'

dyno          count        connect        service
--------------------------------------------------------------
web.3         618          1              84
web.1         693          1              93
web.6         623          2              62
web.4         624          1              20
web.5         674          2              561
web.2         673          2              573
web.7         621          4              191

The query is getting hard to read, so I'd split it to many lines:

heroku logs -t -d router | agrind \
  '*
    | logfmt
    | count as count,
      p95(connect) as connect,
      p95(service) as service
      by dyno
    | sort by p50 desc'

Parsing more than just logfmt

One query I find very handy is grouping by the second. This let’s me see a live second-by-second summary of how my app is performing. I'll use this when there's a production bottleneck so I can see exactly when the issue is resolved.

heroku logs -t -d router | agrind \
  '*
    | parse "T*." as time
    | logfmt
    | count as count,
      p95(connect) as connect,
      p95(service) as service
      by time
    | sort by time'

time            count        connect        service
----------------------------------------------------------------
19:20:36        354          2              1452
19:20:37        492          2              825
19:20:38        519          1              740
19:20:39        513          3              270
19:20:40        435          2              74
19:20:41        451          4              76
19:20:42        468          4              82

This query introduces a new parser called parse. For this aggregation, I needed the time string truncated to the second, and that’s not available in the logfmt data. parse gives us the power of arbitrary string matching to pull out values from anywhere in the log line.

Here an example raw router log for reference:

2021-07-28T16:23:07.870849+00:00 heroku[router]: at=info method=POST path="/api/REDACTED/v2/reports?dyno=web.8&pid=4" host=api.railsautoscale.com request_id=0ce66277-877c-4d4f-91c4-2c1075089b41 fwd="3.84.54.241,172.70.34.122" dyno=web.7 connect=1ms service=156ms status=204 bytes=358 protocol=https

Notice that the timestamp precedes the logfmt data, and it is not formatted as a logfmt key/value pair.

parse "T*." as time extracts a single field, "time", using the pattern T*.. The asterisk (*) is a wildcard, and anything it consumes is extracted as a field. So a single asterisk means a single field is extracted. The text between T and . is 16:23:07, so that becomes the value for "time".

From there, it's all what we've seen before: logfmt to parse the logfmt data, then aggregate operations, then sorting the output.

Conditional aggregates

One last example to show how we can use conditional logic in our queries.

heroku logs -t -d router | agrind \
  '*
    | parse "T*." as time
    | logfmt
    | count(status == 200) as successful,
      count(status != 200) as failure,
      p95(service) as service,
      p95(connect) as connect
      by time
    | sort by time desc'

time            successful        failure        service        connect
------------------------------------------------------------------------------------
19:41:58        189               244            1803           1
19:41:57        182               120            2076           1
19:41:56        191               163            2479           1
19:41:55        174               122            2300           1
19:41:54        176               97             2196           1

This example is taken from a different app, and it's not doing so well. Over half of the requests are failing. 😱

Still, it's pretty cool that we can use conditionals in our count aggregate using extracted logfmt fields like "status".

Your turn

Hopefully by now you've installed Angle Grinder, and you're having some fun slicing and dicing your own log data.

Our Heroku log stream can be completely overwhelming, but tools like heroku logs, grep, and especially agrind give us everything we need to get useful and actionable information from the firehose.

How to Use Heroku For Free

Adam McCrea — Fri, 21 May 2021 16:12:59 +0000

Heroku is the easiest way to deploy and host a server-rendered web app, and it doesn't have to cost an arm and a leg. In fact, for some apps you can do it for free.

This guide covers everything you need to know about using Heroku for free, including free dynos, the free Postgres database plan, and free add-ons.

How does Heroku work?

Heroku runs your app on "dynos". This is just Heroku's word for "container" or "server". For your app to run and serve requests, you need at least one web dyno running. Adding more web dynos will increase your capacity to handle a higher volume of requests.

Heroku offers six dyno types, each increasing in price and performance characteristics. At the lowest tier are the free dynos.

Running a free dyno literally costs you nothing, and it will always cost you nothing. No matter how many hours they run or what you have remaining in your free dyno quota (more on this below), you will never be charged for running free dynos.

Free dyno limitations

Naturally, "free" comes with some limitations.

Free web dynos will idle (sleep) after a period of inactivity. The next request will wake it up, but users will notice a delay while the app is started.
You have a limited number of "free dyno hours" each month. When you've exhausted your quota, your app shuts down.
Free dynos are not available for Heroku Teams.
Free dynos do not support Heroku SSL for custom domains.
Heroku metrics are not available for free dynos.

Those limitations might seem like dealbreakers, but fear not! We have workarounds for many of them.

How to prevent free dynos from sleeping/idling

Free dynos will sleep (shut down) after 30 minutes of inactivity. This is actually a feature to avoid using up your free dyno hours. This makes free dynos great for staging/test environments that don't have constant usage.

If you want your free web dynos to run 24/7, you'll need to ensure they receive at least one request every 30 minutes. The easiest way to prevent your dynos from sleeping is using an uptime monitor that pings your app all day long.

There are lots of uptime monitor services, and there's a good chance you're already using one. If you need a free one (you're reading about free dynos, right?), I've had success with free plans from Freshping and UptimeRobot.

Note that dyno sleeping *only applies to web dynos*. If you're running a "worker" dyno (asynchronous/background job processing), it will not go to sleep. It's very easy to consume your free dyno hours quota with worker dynos.

How are free dyno hours calculated?

Heroku prorates dyno usage to the second. If you run a free dyno for 30 minutes, that'll use 0.5 hours from your free dyno hours quota. If you run an app with single free dyno 24/7 for 31 days, that'll use 744 (24 x 31) of your free dyno hours.

*So how many free dyno hours do you get?*

From the official docs:

Accounts are given a base of 550 free dyno hours each month. In addition to these base hours, accounts which verify with a credit card will receive an additional 450 hours added to the monthly free dyno quota. This means you can receive a total of 1000 free dyno hours per month, if you verify your account with a credit card.

Pretty simple. Add a credit card, and you'll have 1,000 free dyno hours each month. Since a full month is at most 744 hours, this means you can run a single web dyno 24/7 for free.

Heroku trivia

In 2015-2016 there was a time when free dynos could only run 18 hours a day, but this is no longer the case. Free dynos will run 24 hours a day as long as you have sufficient hours available in your quota.

If you had a free app running when these new rules launched in 2016, you may have been grandfathered in with more than 1000 free dyno hours available to you. Lucky you!

Check your free dyno usage

So how many free dyno hours do *you* have available? Check out your Heroku billing page.

This is a great way to check up on any forgotten apps that are consuming your free dynos quota!

Free vs. Hobby vs. Standard-1x dynos

Heroku's six dyno types are generally more powerful as they get more expensive, but that's not the case for cheapest three.

Free, Hobby, and Standard-1X dynos are exactly the same hardware with the same performance characteristics. Hobby dynos only lift a few of the feature limitations of Free dynos:

Hobby dynos do not go to sleep.
Hobby dynos support Heroku SSL for custom domains.
There's no "hours quota" for Hobby dynos.

One shared limitation between Free and Hobby dynos is that you cannot scale to more than one dyno per process. If you need to scale up to two web dynos, for example, you must use standard or performance dynos.

SSL for free dynos

SSL is no longer optional these days—you must serve your web app via HTTPS. On free dynos, you have two options:

Serve your app from the default Heroku URL. Every Heroku app is available by default at [app-name].herokuapp.com, and all of these URLs include SSL support out of the box.
If you need a custom domain, you'll need to use a service like Cloudflare's Flexible SSL (also free). Your users will connect via HTTPS to Cloudflare, and Cloudflare will connect via HTTP to your Heroku app.

If neither of these options are acceptable to you, you'll need to use one of the paid dyno types.

Running a production app on free dynos

Since you can run a free dyno (which is equivalent to a standard-1x dyno) for 1,000 hours per month, *that means you can run your production app 24/7 on Heroku for free, right?*

Well, that depends.

Most production apps need at least a web process and a background worker process for things like sending email and communicating with third-party services. This requires a "worker dyno" in addition to your "web dyno", which exceeds your 1,000 hour allotment if you run them both all month. And free worker dynos do not automatically sleep like free web dynos do, so they'll be running 24/7 unless you manually scale them down.

This is where you need to decide just how "production-scale" your app needs to be right now. You could perform this work synchronously within your web process instead of backgrounding it, or you could find a way to run async jobs within your web process.

Some background work could also be performed on a periodic basis with Heroku Scheduler, which creates a "one-off dyno" and only uses dyno hours for the time it's running.

If you really do need a separate worker process (and most full-stack production apps do), then you've probably outgrown free dynos.

When to upgrade your free dynos

If you're already running your app on free dynos, how will you know when you've outgrown them? If your app is a successful product with growing traffic, you will certainly hit a point where a single free dyno is insufficient.

You can't scale free dynos horizontally—in other words, you can't add more than one free web dyno to your app. If one dyno is not enough for you, you'll need to upgrade to standard dynos.

The best way to know when you're outgrowing free dynos is by watching your metrics. Since Heroku's metrics tab is only available on paid plans, check out Librato's free add-on for similar metrics and alerting.

If you're seeing high request times and/or HTTP errors, there's a good chance you're outgrowing free dynos.

It's also possible your app is just slow. An APM tool like Scout will help you drill down into specific bottlenecks. A full tour of Scout is beyond the scope of this article, but generally speaking, high request queue time indicates a capacity issue (need more dynos).

If you're seeing lots of time spent in other parts of your stack (Ruby, Postgres, etc.) then adding more dynos will not help. You'll need to investigate your app code to resolve the performance issue.

✉️ Want notified when I write more articles like this one? Sign up for my Mastering Heroku mailing list! Very infrequent, very relevant, no sales pitches.

Heroku's free Postgres database

Whew! All that and we haven't even talked about your database yet. Much like dynos, Heroku offers multiple tiers of Postgres.

Ouch. Heroku's free Postgresis limited to 10,000 records.

Even the smallest production apps are likely going to exceed this limitation. Here are your options:

Upgrade to a paid Heroku database plan, starting at $9/month for 10,000,000 records.
Use a third-party hosted database provider with a more generous (but still quite limited) free plan, like ElephantSQL.

There's really no way around it. Your database will eventually be the most expensive layer of your stack, so be prepared to pay for it early.

Free Heroku add-ons

One of the best parts of Heroku is the add-on ecosystem, and many add-ons have excellent free plans. Here are some of my favorites:

Librato — As mentioned this earlier, Librato gives you visibility into your app metrics above and beyond what Heroku provides (and Heroku provides no metrics for free dynos).
Scout APM — Monitor and diagnose app performance issues. I install this on every app.
Logentries — You'll need a way to view your application logs. Logentries isn't the best option, but it has the most generous free plan.
Sentry — Fantastic error monitoring service with generous free plan.
Redis Cloud – Heroku has their own Redis service, but you'll get more for your money (or for free) with Redis Cloud.
AutoIdle — This one isn't a free option, but it's super useful if you've upgraded to paid dynos and you want to save money when your app isn't being used. It essentially replicates the "sleeping" behavior of free dynos. Very nice for staging/QA apps.

Putting it all together

So with all this in mind, when do Heroku's free offerings make sense for you?

Launching a side project. There's a good chance the side project won't get much traction at first, so throw it on free dynos and a free Postgres DB and see what happens! You won't have enough free hours to run worker dynos, but who cares... until you have real users and revenue, you can do everything in your web process.
Running a staging or QA app. These are perfect for free dynos because they don't need to run all the time. Your staging data might even fit within a free Postgres DB.

Beyond those use cases, you're probably going to have to pay for Heroku. The free options exist to try out the service, play with an idea, and run experiments. Once you're running a "real app", you're hopefully at a point where you can spend a few dollars on hosting.

Questions or thoughts on this article? Let's chat on Twitter.

How many Heroku dynos do you need, and which size—An opinionated guide

Adam McCrea — Mon, 17 Aug 2020 00:00:00 +0000

Heroku makes it effortless to deploy our web apps when we're just getting started, but anyone who's scaled an app on Heroku knows that there are still lots of decisions to make. Topping the list are "which dyno type should I use?" and "how many dynos do I need?"

If you're feeling overwhelmed or confused, you're not alone. It's not always clear which dyno type is best and how many we need, but by the end of this article, we'll all be experts.

If you don't care about the details and just want a recommendation, here it is:

Standard-2x dynos offer the best combination of price and performance for most apps. If you encounter memory quota warnings on 2x dynos, you should make the jump to Performance-L. For either dyno type, autoscaling is the only way to know how many dynos you should be running.

How did I land on that recommendation? Let's dig in.

Heroku dyno types
How are the dyno types different?
Free dynos
Hobby dynos
Standard-1x dynos
Heroku routing and "in-dyno concurrency"
Multiple processes on a standard-1x dyno
Standard-2x dynos
Performance-M dynos
Performance-L dynos
How many dynos do you need?
Dyno calculations in the real world
Automation is the answer
Putting it all together

Heroku dyno types

Heroku offers six "Common Runtime" dyno types. These are often referred to as dyno "sizes" since the more expensive ("larger") dynos typically offer more memory and CPU.

Heroku also offers "Private" and "Private Shield" dynos, which offer increasing levels of security compliance. The performance characteristics of these dyno types are almost identical to their Common Runtime counterparts, so we'll focus on the six Common Runtime dyno types.

How are the dyno types different?

A few important supplemental notes to the comparison chart above:

Monthly dyno cost from top to bottom: $0 (Free), $7 (Hobby), $25 (Standard-1x), $50 (Standard-2x), $250 (Perf-M), $500 (Perf-L).
Free, Hobby, and Standard-1x dynos are identical performance-wise, but Heroku imposes some feature limitations on Free and Hobby. More on these limitations below.
The two Performance-level dynos run on dedicated hardware. The four Standard dynos run on shared hardware, where we're susceptible to "noisy neighbors" and somewhat ambiguous processing power (Heroku does not explain what the "Compute" metric represents).

With that as our high-level view, let's go into each dyno type in detail.

Free dynos

Free dynos automatically shut down during periods of inactivity and can only run for a limited number of hours. As I explain in my deep dive into Heroku's free dynos, these limitations make free dynos a poor fit for production applications. Heroku provides enough free hours to run a single dyno continuously for a month, but if we need a worker dyno for background processing (most apps do), we will not have enough free dyno hours.

Free dynos are for great for demos, experimentation, and perhaps a staging app. For production apps, we need to pay.

Hobby dynos

Hobby dynos are identical in performance to Standard-1x dynos, and they don't have the limited hours and automatic shutdown constraints of Free dynos. Sounds great, right?

The catch is a limitation in how we scale our dynos. We can run multiple Hobby dynos if they're different process types (web and worker dynos, for example), but we can't run multiple dynos of the same type. This means that if we ever need to scale to multiple web dynos, Hobby dynos are not an option.

Even if a single Hobby dyno is sufficient for our current traffic, running a production app on a single dyno is risky. Remember that Free, Hobby, and Standard dynos run on shared architecture. This means a noisy neighbor can slow our app down. Or maybe an unexpected spike in traffic has saturated our dyno. There are many ways a dyno can enter a "bad state", and running on a single dyno is a single point of failure.

Later on we'll go deeper on how many dynos to run, but for now let's rule out Hobby dynos since they prevent having any redundancy in our web dynos.

Standard-1x Dynos

Standard-1x dynos are the first of the "professional" dynos—dynos that don't have any feature restrictions upon them. Feature restrictions aside, these dynos are identical to Free and Hobby dynos. I know I've said that multiple times, but it bears repeating.

I've also mentioned that Standard dynos run on shared hardware, making them susceptible to noisy neighbors—tenants running on the same hardware that are consuming more than their fair share of resources. Noisy neighbors on Standard dynos are a real thing, and they're tough to detect and mitigate.

That doesn't rule out Standard dynos altogether, though. At a fraction of the price of Performance dynos, we can run many more Standard dynos for the same cost, helping mitigate possible performance issues caused by noisy neighbors.

The real problem with Standard-1x dynos is memory. With a limit of 512 MB, many apps will exceed that quota with even a single process. Running multiple processes per dyno is critical, and we're going to take a little digression to discuss why.

Heroku routing and "in-dyno concurrency"

When a user requests a page on our web app, Heroku's router decides which of our web dynos receives the request. Ideally, the router would know how busy each dyno is, and it would give the request to the least busy dyno.

But that's not how it works.

Heroku's router uses a random routing algorithm. It doesn't care about the size of the request, the path of the request, or how busy each web dyno might be. This means our dynos will inevitably receive an unfair share of large or slow requests, at least some of the time. It also means that if a web dyno can only process a single request at a time, we've introduced a dangerous bottleneck into our system.

If we have no "in-dyno concurrency", a single slow request can cause other requests to back up inside the dyno. This is when we start to see request queue time increase. It's a combination of not running enough web dynos, and those dynos not having sufficient concurrency.

No matter how many dynos we run, it's critical that each dyno can process multiple requests concurrently. In Ruby, this means running multiple web processes—usually Puma workers. Running multiple threads doesn't cut it. Ruby threads do provide a bit of concurrency (especially when there's a lot of I/O), but due to the GVL, it's not true concurrency.

Multiple processes on a Standard-1x dyno

This an important point when considering a Standard-1x dyno, because most Rails apps will consume far more than 512 MB when running mulitple Puma workers.

An easy way to test this is to run your app for 24 hours on Standard-2x dynos (which we'll discuss next) with 2 Puma workers. Usually this is accomplished by setting WEB_CONCURRENCY to "2", and ensuring that you've uncommented the "workers" line in your Puma config. Check your memory usage on your Heroku dashboard, and you should see it start to level off after a few hours (it's normal for it to increase initially).

For most of us, this experiment will show 2 processes consuming somewhere between 500-1000 MB. Since running at least 2 processes is a must, and Standard-1x dynos are limited to 512 MB, Standard-1x is rarely a viable option.

Pro tip: One way to decrease memory usage for Ruby apps on Heroku is by using the Jemalloc buildpack.

Standard-2x dynos

Standard-2x dynos are identical with Standard-1x, with double the memory and CPU at double the price.

If you're wondering how a single Standard-2x dyno is any different than two Standard-1x dynos, it's all about the memory and concurrency. While most apps are constrained by memory to a single process in a 1x dyno, the 2x dyno opens up the possibility of running multiple processes.

Compare these scenarios:

Two 1x Dynos running 1 web process: Heroku routes requests randomly between the two dynos. Inevitably it will make bad decisions, sending requests to a busy dyno when another is available for work.
One 2x Dyno running 2 web processes: The web processes will balance the requests coming into the dyno, ensuring that an available process always gets the next request. Same cost, better concurrency.

Standard-2x dynos in-dyno concurrency possible while costing a fraction of Performance dynos. That's why I recommend them for most apps.

But not all apps can squeeze multiple processes onto a 2x dyno. Rails apps that consume 1 GB or more memory with two processes are not uncommon, and for those apps there's Performance dynos.

Performance-M dynos

Perf-M dynos are the first of two "Performance" dyno options. Looking at the cost and performance characteristics relative to other dyno types, we can see there's not a lot of value here.

Perf-M dynos have less than one fifth the memory of Performance-L dynos at half the cost.
Perf-M have only 2.5x the memory of Standard-2x dynos at 5x the cost.

Not much more to say about these. They're just a bad deal.

Performance-L dynos

Perf-L dynos cost $500/month per dyno—10 times the cost ($50/month) of Standard-2x—but we get what we pay for. With a 14 GB memory quota, we can easily run several app processes in a single dyno.

Performance dynos run on dedicated architecture, so we don't have to worry about noisy neighbors. With Standard dynos, performance differences can be noticeable between dynos, such as after restarting or deploying. Performance dynos don't have this issue. The performance is very consistent from one dyno to the next.

Still, the performance variability with Standard dynos can be mitigated by running more of them, so I only recommend Perf-L dynos for apps that are memory-constrained on 2x dynos.

Want notified when I write more articles like this one? Sign up for my Mastering Heroku mailing list! Very infrequent, very relevant, no sales pitches.

How many dynos?

Once we've selected a dyno type, the next logical question is "how many of them?" Let's run a quick calculation.

We'll assume an example app that takes 100ms to process each request. This translates to 10 requests per second without any concurrency. If we're running two processes per dyno, this takes us to 20 requests per second per dyno.

We can look at Heroku's throughput chart so see how many requests per second the app receives. Let's assume this example app maxes out at 200 requests per second.

The math is straightforward: If we can handle 20 req/s per dyno, we'll need 10 dynos to handle 200 req/s. Here's the formula:

The only problem is that this example app doesn't exist.

Dyno calculations in the real world

If we looked at a real app instead of a hypothetical app, we'd see some stark differences:

Response times are not consistent. Some endpoints are slow, some are fast.
Requests don't show up evenly. We might get a burst of 1,000 requests in one minute, then just a trickle of traffic for the next 10 minutes.
Traffic patterns change throughout the day, week, and year.

Attempting to use the calculation above in a real-world app is fraught with error. I've been there and felt the pain. We're bound to find ourselves in one of these scenarios:

We under-provision our dynos, and our app struggles to keep up. Requests are queueing because we don't have enough capacity to serve them, and our users experience slow page requests. Unsure of how to fix the problem, we crank up the dynos. Now we're in the next scenario.
We over-provision our dynos, and our app is performing just fine. Unfortunately, we're paying for extra capacity we don't need. This can be really frustrating, and it leads to the many claims of Heroku being too expensive.

So how can we determine how many dynos to run, having confidence that our app will stay fast, without paying any more than necessary?

Automation is the answer

Instead of trying to manually calculate something that's changing every second, let's make software do it for us. That's exactly what an autoscaler does: it continually calculates how many dynos are needed right now based on live metrics.

I built an autoscaler that does this better than anything out there, but this isn't a sales pitch for Rails Autoscale—it is a pitch for autoscaling in general. There's just no reason to do it the hard way.

Every production app should have autoscaling in place. Even if your production app receives little traffic and runs fine on a single dyno, that dyno is a single point of failure. Running a single dyno without autoscaling is an invitation for slowdowns at best, a production outage at worst. Think of it as a low-cost safety net.

Autoscaling on Heroku is easy and cheap. With it we avoid the painful battles of being under-provisioned, and we avoid paying for unnecessary capacity when over-provisioned.

Putting it all together

We started with six dyno types, and narrowed them down to two: Standard-2x and Performance-L.

I recommend all apps try Standard-2x dynos first, running two app processes per dyno. If the 1 GB memory quota proves insufficient for your app, then make the jump to Perf-L. (Or better yet, find out why your app is consuming so much memory!)

Regardless of dyno type, always set up autoscaling on production apps. The cost of this setup will rival any non-Heroku option, and it'll run smoothly for years with little manual intervention.

DEV Community: Adam McCrea

Power Tools for Analyzing your Heroku logs

How to view your Heroku logs

Scoping it further

Extracting specific values

How does Angle Grinder work?

Aggregating your log data in real-time

Parsing more than just logfmt

Conditional aggregates

Your turn

How to Use Heroku For Free

How does Heroku work?

Free dyno limitations

How to prevent free dynos from sleeping/idling

How are free dyno hours calculated?

Heroku trivia

Check your free dyno usage

Free vs. Hobby vs. Standard-1x dynos

SSL for free dynos

Running a production app on free dynos

When to upgrade your free dynos

Heroku's free Postgres database

Free Heroku add-ons

Putting it all together

How many Heroku dynos do you need, and which size—An opinionated guide

Table of contents

Heroku dyno types

How are the dyno types different?

Free dynos

Hobby dynos

Standard-1x Dynos

Heroku routing and "in-dyno concurrency"

Multiple processes on a Standard-1x dyno

Standard-2x dynos

Performance-M dynos

Performance-L dynos

How many dynos?

Dyno calculations in the real world

Automation is the answer

Putting it all together