DEV Community: Garen Torikian

Seven more DevOps things Heroku does (so you don't have to!)

Garen Torikian — Wed, 11 Dec 2019 16:31:20 +0000

A few weeks ago, I published a post about eight different DevOps chores that Heroku handles for you. There are, of course, many more things which Heroku takes care of, so that you can focus on building your application and not worry about your operations or infrastructure. We're going to continue our list with seven more things that Heroku does, this time focusing on networking, workflows, and integrating with other tools.

1. Easily attach external systems

You're probably already aware that Heroku provides a marketplace with hundreds of add-ons. These are (typically) third-party services that can be tightly integrated with your application which provide third-party solutions for various extensions, such as logging, databases, search, job queueing, and more.

Heroku believes that offering a choice in external dependencies in a marketplace like this is a good thing. They've vetted the list to ensure that only the highest quality add-ons are available for applications hosted on Heroku. What's more, scaling these dependencies up or down to account for changing traffic demands can be executed through a single command: heroku addons:upgrade ADDON [PLAN].

2. Automatic security updates for add-ons

Much like the safeties guaranteed for your language or framework, Heroku regularly rolls out updates for add-ons to address critical issues; for example, here's one done for Redis a while back. Heroku takes security seriously, and takes a proactive approach to patching software like this so you don't have to worry about staying on top of vulnerabilities.

3. Manages access rights and security levels

In order to establish access rights in the name of privacy, some application hosts require you to configure complicated ACLs for each and every aspect of your application and its dependencies.

Heroku provides a handful of permissions categories, within which you can group individuals or entire teams. The granularity of abilities for these permissions range from read-only access to an app, to managing the code, to full administrative ownership. They are simple enough to reason about, and flexible enough to protect users, whether they're a start-up or an enterprise-level organization.

4. Load testing for Apache Kafka

If you're making use of Apache Kafka on Heroku, you can actually trigger one of your nodes to intentionally fail. The purpose of this is to observe how the message stream behaves under an unexpected disaster. The goal is to provide peace of mind, by demonstrating that your app will suffer no performance issues or degradation as a result, since we end up automatically rerouting data to another node. Regularly invoking this sort of chaos testing can help verify that your application operates successfully in the event of a real (though unlikely!) disaster.

5. Tracking deployments for individual environments

You've coded a feature, opened up a pull request, and are looking for feedback from others on your team. An easy way to enable a frictionless discussion is to set up a pipeline with a Review App. For any new PR that's opened up, Heroku can build a Review App, which is a disposable app with a unique URL. Review Apps can quickly demonstrate the value of new changes, without requiring others to pull your branch or even have a development environment set up on their machine.

6. Network isolation with managed VPCs

For apps with sensitive access to data, you may need to limit its exposure to the public Internet. Normally, this might involve setting up a VPC, with subnets defining the range of IP addresses that your app can communicate over. For an app which requires high availability, this also involves establishing separate VPCs per geographic region.

Heroku abstracts away all of the hassle with Private Spaces. This allows you to establish secure, site-to-site connections between a Heroku app and your company's Intranet. As can be expected, this too can be established just through the command-line:

heroku spaces:create my-space-name --team my-team-name --region tokyo

7. Configure SSL

Last, but certainly not least, Heroku will secure your domain with an SSL certificate. Better yet, your apps can enable automated certificate management, to automatically provisions and renew SSL certificates over time. This guarantees that your certs are regularly kept up-to-date.

Want more?

Are you curious about any other differences between Heroku and other PaaS providers that can be highlighted in a future post? Let me know in the comments!

Eight DevOps things Heroku does (so you don't have to!)

Garen Torikian — Tue, 15 Oct 2019 16:25:13 +0000

You might know—vaguely, roughly, innately—that deploying an application to Heroku's Platform-as-a-Service would be a good choice to make. It'll save you from the struggles of setting up networks, managing infrastructure, and configuring services, freeing up time to focus on building features for your application, to deliver value to your customers. But maybe you aren't really sure how Heroku does those things. Or maybe (gulp) you don't think Heroku can help you out much.

Heroku takes care of so many DevOps-y chores for you that its simplicity tends to obfuscate the number of concerns involved in deploying and maintaining a production-grade app. So in this post, we're going to take a look at just eight ways in which Heroku makes your development life just a little bit easier.

1. Manage machine images

Your application is more than just the code you've written. It also relies on the packages you've explicitly imported, the programming language you're writing in, and the operating system your code runs on. You know what that all adds up to? Maintenance. You've got to be on the lookout to make sure that your operating system is patched from security flaws and your programming language is updated for performance improvements, constantly.

Instead of sorting through package security updates, or figuring out how to take your site down and update to the latest Ubuntu, Heroku will take care of all of the security overhead for you. Here’s a recent blog post demonstrating what the process for the end user is to receive critical security updates (hint: it’s nothing).

2. Deploy a stack to multiple environments

The problem of machine maintenance scales linearly as you add more servers to your workflow, typically by way of needing more environments. If you need to introduce a staging environment into your setup, you may already have some automation scripts to set up that infrastructure, but now you need to pay close attention to their system dependencies as well.

Deploying the entire stack of a Heroku application is as simple as a git push to a new Heroku app. Rather than spending time re-provisioning the minutiae of your servers' needs onto staging servers, you can just link your different environments and push your code out to a new environment instantaneously.

But maybe you have many environments, and you don’t want to keep entering commands to deploy the app to all of them. That’s fine; you can set up a pipeline instead. A pipeline allows you to promote your app through various environments: development, review, staging, production.

When it comes to environment variables, you can get and set these through the command line, too:

# set for production (by default)
$ heroku config:set APP_SECRET=XXX
# set for staging
$ heroku config:set APP_SECRET=YYY --remote staging

3. Administration and tuning your databases

Databases are at the heart of every application, from the very simple to the very complex. Applications rely on the presence of a database from which to store and retrieve essential information. If connections to this service stop flowing, it can completely kill your application. Tweaking the database engine itself can also eke out some performance gains, which can be critical for high-traffic sites.

Let Heroku's database administrators ensure that the installations are well-tuned, so that your engineers can work on other issues. They'll upgrade the database, automatically apply security patches, migrate you off of bad instances, and more. While your team is iterating on improvements, Heroku's DBAs will optimize your database for gains you didn't even know you needed.

Frequent maintenance of your backend isn’t just lip-service: it’s a core value Heroku prides itself on.

4. Log everything across every environment

Logs: they're not just for lumberjacks anymore! You're likely tracking various actions across the lifecycle of requests, whether that's to your web app, its worker queues, system components, or any backing services. Configuring all of those sources of information can be a challenge, and aggregating into something that can be consumed could be even harder.

Heroku's Logplex will not only slurp up information from any source you tell it to, it'll also spit it back out in a format that you expect, whether that's for long-term storage, a terminal somewhere, or any API endpoint. You can build your own pipeline, or tap into the existing add-on ecosystem for log routing.

5. Monitoring and alerting

As you might expect from its extensive and flexible logging setup, Heroku's also got a phenomenally beautiful monitoring system. Although you can choose to integrate with third-parties on this, in many cases, you won't need to. Heroku's dashboard will show you much of what you need to know, from memory health (via GC times and overall usage), to HTTP request/response times, and even the overall health of your server's interaction with its resources. There are also plenty of external integrations to choose from if you’d prefer to hook into an external system like New Relic or Graphite.

For Professional dynos, you can also set up alerts directly from the Heroku Dashboard. As you're configuring your thresholds, Heroku will tell you how many times it would've triggered notifications to your time, and advise you to adjust the metrics if they're too sensitive. There's also a native integration with PagerDuty.

6. Backups and recovery

Have you got a backup strategy for your application and its infrastructure? And are you sure it works? With dependencies that your system might need—Redis, Postgres, Kafka—you have to set up data backup and recovery strategies, configure the granularity of its retention, and make sure that you've always got enough disk space on hand to keep the backups you need.

Since Heroku's on your side, and probably more nervous about data loss than you are, they've already thought through a thousand different disaster scenarios, and are able to store and restore your data in the safest manner possible. For example, a single command through your terminal can rollback your database to a previous state; if a Kafka node fails, Heroku's automated systems will work to restore it to its normal operation.

For your Postgres database, you can even set a schedule for the backups to be taken outside of peak hours. Redis backups/forks can also be set through the command line.

7. Automatically respawn a downed server

It's not uncommon for applications to require the need of a process manager, in order to restart stalled—or completely crashed—applications. But as with backups, ensuring that this daemon is doing the right thing can be tough, especially considering that it "works" only when everything else doesn't.

In the event of the unexpected, Heroku will restart your servers for you, whether that's due to resource exhaustion, a segfault, or just a plain logic bug. While attempting to recover, the runtime also has a cool-off period of several minutes to gracefully resume normal operations.

8. Forking primary databases

If your application is highly available, it needs to be in more places than one, which means more of everything--including databases. In the average HA setup, all data writes go to a primary database, and several region-specific secondaries are distributed across the world. Ensuring the proper throughout from the primary to the secondary is a bundle of networking and latency issues, chief among them being to ensure that your application does not read data that has become stale.

For Postgres (The One True Database™️), Heroku provides the ability to fork your primary database more easily than you might believe possible. By creating a database through the command-line, and providing the --follow flag, you can create a read-only follower of your primary database. Changes from the primary are streamed in real-time, allowing for seamless migrations, data redundancy, and a hot standby.

Concentrate on your needs, not your infrastructure

When you spend less time setting up and executing on your production needs, you’ll have more time for your users’ needs. Heroku runs eight million apps at 25 billion requests per day, and the developers building those apps benefit from a DevOps workflow that's managed for them.

The PaaS Advantage

Garen Torikian — Mon, 17 Jun 2019 14:15:17 +0000

I've been thinking a lot recently about how modern applications are developed and deployed. A lot of my experience with this stems from having worked at large corporations, at early stage startups, and simply trying to make goofy projects on my own time.

When I was at larger organizations many years ago, there was a strict release workflow: you checked in your code, documentation, and assets by an agreed upon deadline—something like Tuesdays at 11pm PST. Then, all of that was bundled up and released to the world by an operations team whose sole responsibility was to make sure everything went smoothly. If there was an infrastructure issue with the deploy, a release manager worked with the operations team to solve it. For example, if there was some replication lag in production, the ops team needed to investigate which patch from all the weekly changes caused that problem and fix it. Obviously, this is unfair and not ideal. As a developer working on an application, I was ignorant of what the operations team handled. I had no clue as to the amount of stress they worked through, and how their role was to suss out from hundreds of changelists where a problem might've been introduced. Looking back, this was all completely erroneous: had the developers known what the operations team was dealing with, we would have probably agreed to more ownership and responsibility, in order to avoid introducing issues in the first place. These infrastructure problems were sometimes caused by code that didn't fully grasp how the underlying systems worked, a problem that was, in a sense, abstracted away by organizational hierarchies.

During my time working at a couple of startups, the responsibility of the code and infrastructure operating correctly was distributed amongst everyone working on the product. If, on a deploy, there was an issue with the database, or if storage space ran out, or if timeouts were increasing as a result of unexpected behavior, no one was going to save you. It was your change and yours to fix. This was an immense learning opportunity, and one that I'm definitely grateful for, but it swung too far in the opposite direction. Now, in addition to building features, I was also in charge of controlling the underlying server behavior. While developers should have a grasp as to how their code behaves in production in order to mitigate performance and resource issues, I don't think it's the best use of an application developer's time to constantly wade into these operational details.

In my personal projects, I've struggled to set up modern services that my applications have required. In order to set up a Rails site on a vanilla infrastructure-as-a-service model, one needs to be familiar with the packages available on their Ubuntu base image, an understanding of how to map their nginx server to their domain, setting up services that start when the server does, and securing all of them. This ends up being too much mental overhead. I mean, it can be done, but it takes a few hours and dozens of outdated help articles written over various years by various authors to figure it all out. And once it's set up correctly, how to handle continuous deployments is a Sisyphean task. It turns out that it's actually quite complicated to establish them. For example, which deployment tool should I even use? Is Capistrano still a good choice? I've heard good things about Mina, which is newer and something else to learn. If I decide to write my next project in Node, I'll need to learn and configure a whole new tool, like PM2. And what if, against all odds, the project takes off? Suddenly I'll need to think about the appropriate resources for scaling, load balancing requests, and maybe introducing new dependencies, like Redis for caching.

Know what you need to (and forget the rest)

In recent years, there's been a shift towards embracing DevOps as a methodology. In a nutshell, DevOps means applying long-established principles, such as writing tests or using version control, to your infrastructure setup. You can automate the resources, services, and networking essentials that your application needs, roll them back if problems are introduced, or discuss them with others in a pull request, just like any other piece of code. For example, a DevOps workflow would advocate that your changes include tests for how your code behaves for the end user and on your infrastructure.

However, there's an immense learning curve to achieving proficiency in operational management. You may find that you're spending more time tweaking configuration files than actually writing code for your application. DevOps tends to conflate the roles of developing an application with running its infrastructure; that to be a developer, you must know how to plan, build, and satisfy your user's needs, as well as know how to architect, setup, and scale a server's needs. No one would assume that a backend Ruby developer be also savvy enough in frontend CSS and JavaScript, yet that's more or less what an overemphasis on server management suggests.

I believe that DevOps is a great idea—for companies that are at an immense scale. I'm talking Google/Netflix/Facebook scale, applications that receive millions of requests a second, who host their own data centers (or at least, crucially rely on dedicated servers). A mistake or inefficiency in their operational setup could affect hundreds of thousands of people in an instant, so it makes sense for them to rely on technologies that reduce the possibility for error.

As an application developer, I don't want to think about provisioning servers and CPUs or tweaking processes and deployments. It doesn't much matter whether a project is just me or me and a group of friends or me and an entire organization. Our main concern—and frankly, our main interest—is to build something that people use.

Simple is easier

For the rest of us, there's really no need to make the responsibility of building something that people use more complicated than it already is. Deploying to a platform-as-a-service takes the burden of servicing operations and places it on experts whose sole responsibility is to keep your site up. While PaaS models tend to be pricier than DIY infrastructure-as-a-service models, the trade-off is that you won't be paying engineers to fine-tune your networks, nor will your application developers juggle their focus between features that grow your business and managing an application's underlying services.

Not only is the cognitive overload alleviated, platforms like Heroku (for complete applications), Zeit (for simpler microservices), or Netlify (for static sites) are just easier to work with. Rather than learning a new configuration language or stringing together Unix commands, you can work in a faster continuous deployment cycles through a git push without worrying about the details of getting the site online during its many iterations. If a new feature depends upon a new service, you can attach it to the app through a command-line sequence as easy as addons kafka:standard-1 and rest assured knowing that best practices are provided for you. As your site grows, you can just drag a slider on the administration page to handle any scaling issues. Even if an infrastructure problem is introduced as a result of a code change, you can temporarily adjust the infrastructure via your PaaSes UI while working on a fix in the application.

The web has become more complicated as it has grown, and over the years, communities have continued to develop tooling that satisfies their intricate needs. I think application developers would do well to take a step back and ask what problems they're trying to solve, and then pick a development strategy that solves them in the simplest way possible. It may well be that taking control of your infrastructure and operational management is essential for your application and organization, but if it's not, it's better to focus on what's important—delivering features that your users need from you.