DEV Community: Heroku

Yes! I Can Finally Run My .NET Application on Heroku!

Michael Bogan — Wed, 19 Feb 2025 19:18:36 +0000

Heroku now officially supports .NET!

.NET developers now have access to the officially supported buildpack for .NET, which means you can now deploy your .NET apps onto Heroku with just one command: git push heroku main. 🤯 Gone are the days of searching for Dockerfiles or community buildpacks. With official support, .NET developers can now run any .NET application (version 8.0 and higher) on the Heroku platform.

Being on the platform means you also get:

Simple, low friction deployment
Scaling and service management
Access to the add-on ecosystem
Security and governance features for enterprise use

Intrigued? Let’s talk about what this means for .NET developers.

Why This Matters for .NET Developers

In my experience, running an app on Heroku is pretty easy. But deploying .NET apps was an exception. You could deploy on Heroku, but there wasn’t official support. One option was to wrap your app in a Docker container. This meant creating a Dockerfile and dealing with all the maintenance that comes along with that approach. Alternatively, you could find a third-party buildpack; but that introduced another dependency into your deployment process, and you’d lose time trying to figure out which community buildpack was the right one for you.

Needing to use these workarounds was unfortunate, as Heroku’s seamless deployment is supposed to make it easy to create and prototype new apps. Now, with official buildpack support, the deployment experience for .NET developers is smoother and more reliable.

Key Benefits of .NET on Heroku

The benefits of the new update center around simplicity and scalability. It all begins with simple deployment. Just one git command… and your deployment begins. No need to start another workflow or log into another site every time; just push your code from the command line, and Heroku takes care of the rest.

Heroku’s official .NET support currently includes C#, Visual Basic, and F# projects for .NET and ASP.NET Core frameworks (version 8.0 and higher). This means that a wide variety of .NET projects are now officially supported. Want to deploy a Blazor app alongside your ASP.NET REST API? You can do that now.

Coming into the platform also means you can scale as your app grows. If you need to add another service using a different language, you can deploy that service just as easily as your original app. Or you can easily scale your dynos to match peak load requirements. This scaling extends to Heroku’s ecosystem of add-ons, making it easy for you to add value to your application with supporting services while keeping you and your team focused on your core application logic.

In addition to simple application deployment, the platform also supports more advanced CI/CD and DevOps needs. With Heroku Pipelines, you have multiple deployment environment support options and can set up review apps so code reviewers can access a live version of your app for each pull request. And all of this integrates tightly with GitHub, giving you automatic deployment triggers to streamline your dev flow.

Getting Started

Let’s do a quick walk-through on how to get started. In addition to your application and Git, you will also need the Heroku CLI installed on your local machine. Initialize the CLI with the heroku login command. This will take you to a browser to log into your Heroku account:

Once you’re logged in, navigate to your .NET application folder. In that folder, run the following commands:

~/project$ heroku create
~/project$ heroku buildpacks:add heroku/dotnet

Now, you’re ready to push your app! You just need one command to go live:

~/project$ git push heroku main

That’s it! For simpler .NET applications, this is all you need. Your application is now live at the app URL provided in the response to your heroku create command. To see it again, you can always use heroku info. Or, you can run heroku open to launch your browser at your app URL.

If you can’t find the URL, log in to the Heroku Dashboard. Find your app and click on Open app. You’ll be redirected to your app URL.

If you have a more complex application or one with multiple parts, you will need to define a Procfile, which will tell Heroku how to start up your application. Don’t be intimidated! Many Procfiles are just a couple lines. For more in-depth information, check out the Getting Started on Heroku with .NET guide.

Now we’ve got another question to tackle…

Who Should Care?

The arrival of .NET on Heroku is relevant to anyone who wants to deploy scalable .NET services and applications seamlessly.

For solo devs and startups, the platform’s low friction and scaling takes away the burden of deployment and hosting. This allows small teams to focus on building out their core application logic. These teams are also not restricted by their app’s architecture, as Heroku supports both large single-service applications as well as distributed microservice apps.

Enterprise teams are poised to benefit from this as well. .NET has historically found much of its adoption in the enterprise, and the addition of official support for .NET to Heroku means that these teams can now combine their .NET experience with the ease of deploying to the Heroku platform. Heroku’s low friction enables rapid prototyping of new applications, and Dyno Formations make it easier to manage and scale a microservice architecture. Additionally, you can get governance through Heroku Enterprise, enabling the security and controls that larger enterprises require.

Finally, .NET enthusiasts from all backgrounds and skill levels can now benefit from this new platform addition. By going with a modern PaaS, you can play around with apps and projects of all sizes, hassle-free.

Wrap-up

That’s a brief introduction to official .NET support on Heroku! It’s now easier than ever to deploy .NET applications of all sizes to Heroku. What are you going to build and deploy first? Let me know in the comments!

5 Simple Steps to Get Your Test Suite Running in Heroku CI

Michael Bogan — Wed, 26 Jun 2024 13:27:49 +0000

So, I’ve always thought about Heroku as just a place to run my code. They have a CLI. I can connect it to my GitHub repo, push my code to a Heroku remote, and bam… it’s deployed. No fuss. No mess.

But I had always run my test suite… somewhere else: locally, or with CircleCI, or in GitHub Actions. How did I not know that Heroku has CI capabilities? You mean I can run my tests there? Where have I been for the last few years?

So that’s why I didn’t know about Heroku CI…

CI is pretty awesome. You can build, test, and integrate new code changes. You get fast feedback on those code changes so that you can identify and fix issues early. Ultimately, you deliver higher-quality software.

By doing it in Heroku, I get my test suite running in an environment much closer to my staging and production deployments. And if I piece together a pipeline, I can automate the progression from passing tests to a staging deployment and then promote that staged build to production.

So, how do we get our application test suite up and running in Heroku CI? It will take you 5 steps:

Write your tests
Deploy your Heroku app
Push your code to Heroku
Create a Heroku Pipeline to use Heroku CI
Run your tests with Heroku CI

We’ll walk through these steps by testing a simple Python application. If you want to follow along, you can clone my GitHub repo.

Our python app: is it prime?

We’ve built an API in Python that listens for GET requests on a single endpoint: /prime/{number}. It expects a number as a path parameter and then returns true or false based on whether that number is a prime number. Pretty simple.

We have a modularized function in is_prime.py:

def is_prime(num):
    if num <= 1:
        return False
    if num <= 3:
        return True
    if num % 2 == 0 or num % 3 == 0:
        return False
    i = 5
    while i * i <= num:
        if num % i == 0 or num % (i + 2) == 0:
            return False
        i += 6
    return True

Then, our main.py file looks like this:

from fastapi import FastAPI, HTTPException
from is_prime import is_prime

app = FastAPI()

# Route to check if a number is a prime number
@app.get("/prime/{number}")
def check_if_prime(number: int):
    return is_prime(number)
    raise HTTPException(status_code=400, detail="Input invalid")

if __name__ == "__main__":
    import uvicorn
    uvicorn.run(app, host="localhost", port=8000)

That’s all there is to it. We can start our API locally (python main.py) and send some requests to try it out:

~$ curl http://localhost:8000/prime/91
false

~$ curl http://localhost:8000/prime/97
true

That looks pretty good. But we’d feel better with a unit test for the is_prime function. Let’s get to it.

Step #1: Write your tests

With pytest added to our Python dependencies, we’ll write a file called test_is_prime.py and put it in a subfolder called tests. We have a set of numbers that we’ll test to make sure our function determines correctly if they are prime or not. Here’s our test file:

from is_prime import is_prime

def test_1_is_not_prime():
    assert not is_prime(1)

def test_2_is_prime():
    assert is_prime(2)

def test_3_is_prime():
    assert is_prime(3)

def test_4_is_not_prime():
    assert not is_prime(4)

def test_5_is_prime():
    assert is_prime(5)

def test_991_is_prime():
    assert is_prime(991)

def test_993_is_not_prime():
    assert not is_prime(993)

def test_7873_is_prime():
    assert is_prime(7873)

def test_7802143_is_not_prime():
    assert not is_prime(7802143)

When we run pytest from the command line, here’s what we see:

~/project$ pytest
=========================== test session starts ===========================
platform linux -- Python 3.8.10, pytest-8.0.2, pluggy-1.4.0
rootdir: /home/michael/project/tests
plugins: anyio-4.3.0
collected 9 items                                                                                                                                                                                            

test_is_prime.py .........                                                                                                                                                                             [100%]

============================ 9 passed in 0.02s ============================

Our tests pass! It looks like is_prime is doing what it’s supposed to.

Step #2: Deploy your Heroku app

It’s time to wire up Heroku. Assuming you have a Heroku account and you’ve installed the Heroku CLI, creating your Heroku app is going to go pretty quickly.

Heroku will look in our project root folder for a file called requirements.txt, listing the Python dependencies our project has. This is what the file should look like:

fastapi==0.110.1
pydantic==2.7.0
uvicorn==0.29.0
pytest==8.0.2

Next, Heroku will look for a file called Procfile to determine how to start our Python application. Procfile should look like this:

web: uvicorn main:app --host=0.0.0.0 --port=${PORT}

With those files in place, let’s create our app.

~/project$ heroku login

~/project$ heroku apps:create is-it-prime

That was it? Yeah. That was it.

Step #3: Push your code to Heroku

Next, we push our project code to the git remote that the Heroku CLI set up when we created our app.

~/project$ git push heroku main
…
remote: -----> Launching...
remote:        Released v3
remote:        https://is-it-prime-2f2e4fe7adc1.herokuapp.com/ deployed to Heroku

So, that’s done. Let’s check our API.

$ curl https://is-it-prime-2f2e4fe7adc1.herokuapp.com/prime/91
false

$ curl https://is-it-prime-2f2e4fe7adc1.herokuapp.com/prime/7873
true

$ curl https://is-it-prime-2f2e4fe7adc1.herokuapp.com/prime/7802143
false

It works!

Step #4: Create a Heroku Pipeline to use Heroku CI

Now, we want to create a Heroku Pipeline with Heroku CI enabled so that we can run our tests.

We create the pipeline (called is-it-prime-pipeline), adding the app we created above to the staging phase of the pipeline.

$ heroku pipelines:create \
  --app=is-it-prime \
  --stage=staging \
  is-it-prime-pipeline

Creating is-it-prime-pipeline pipeline... done
Adding ⬢ is-it-prime to is-it-prime-pipeline pipeline as staging... done

With our pipeline created, we want to connect it to a GitHub repo so that our actions on the repo (such as new pull requests or merges) can trigger events in our pipeline (like automatically running the test suite).

$ heroku pipelines:connect is-it-prime-pipeline -r capnMB/heroku-ci-demo

Linking to repo... done

As you can see, I’m connecting my pipeline to my GitHub repo. When something like a pull request or a merge occurs in my repo, it will trigger the Heroku CI to run the test suite.

Next, we need to configure our test environment in an app.json manifest. Our file contents should look like this:

{
  "environments": {
    "test": {
      "formation": {
        "test": {
          "quantity": 1,
          "size": "standard-1x"
        }
      },
      "scripts": {
        "test": "pytest"
      }
    }
  }
}

This manifest contains the script we would use to run through our test suite. It also specifies the dyno size we (standard-1x) would want to use for our test environment. We commit this file to our repo.

Finally, in the web UI for Heroku, we navigate to the Tests page of our pipeline, and we click the Enable Heroku CI button.

After enabling Heroku CI, here’s what we see:

Step #5: Run your tests with Heroku CI

Just to demonstrate it, we can manually trigger a run of our test suite using the Heroku CLI:

$ heroku ci:run --pipeline is-it-prime-pipeline
…
-----> Running test command `pytest`...
========================= test session starts ============================
platform linux -- Python 3.12.3, pytest-8.0.2, pluggy-1.4.0
rootdir: /app
plugins: anyio-4.3.0
collected 9 items

tests/test_is_prime.py .........                                         [100%]

============================ 9 passed in 0.03s ============================

How does the test run look in our browser? We navigate to our pipeline and click Tests. There, we see our first test run in the left-side nav.

A closer inspection of our tests shows this:

Awesome. Now, let’s push some new code to a branch in our repo and watch the tests run!

We create a new branch (called new-test), adding another test case to test_is_prime.py. As soon as we push our branch to GitHub, here’s what we see at Heroku:

Heroku CI detects the pushed code and automates a new run of the test suite. Not too long after, we see the successful results:

Heroku CI for the win

If you’re using Heroku for your production environment—and you’re ready to go all in with DevOps—then using pipelines and Heroku CI may be the way to go.

Rather than using different tools and platforms for building, testing, reviewing, staging, and releasing to production… I can consolidate all these pieces in a single Heroku Pipeline. And with Heroku CI, I get automated testing with every push to my repo.

How To Build a Simple GitHub Action To Deploy a Django Application to the Cloud

Michael Bogan — Mon, 10 Jun 2024 14:25:13 +0000

Continuous integration and continuous delivery (CI/CD) capabilities are basic expectations for modern development teams who want fast feedback on their changes and rapid deployment to the cloud. In recent years, we’ve seen the growing adoption of GitHub Actions, a feature-rich CI/CD system that dovetails nicely with cloud hosting platforms such as Heroku. In this article, we’ll demonstrate the power of these tools used in combination—specifically how GitHub Actions can be used to quickly deploy a Django application to the cloud.

A Quick Introduction to Django

Django is a Python web application framework that’s been around since the early 2000s. It follows a model-view-controller (MVC) architecture and is known as the “batteries-included” web framework for Python. That’s because it has lots of capabilities, including a strong object-relational mapping (ORM) for abstracting database operations and models. It also has a rich templating system with many object-oriented design features.

Instagram, Nextdoor, and Bitbucket are examples of applications built using Django. Clearly, if Django is behind Instagram, then we know that it can scale well. (Instagram hovers around being the fourth most visited site in the world!)

Security is another built-in feature; authentication, cross-site scripting protection, and CSRF features all come out of the box and are easy to configure. Django is over 20 years old, which means it has a large dev community and documentation base—both helpful when you’re trying to figure out why something has gone awry.

Downsides to Django? Yes, there are a few, with the biggest one being a steeper learning curve than other web application frameworks. You need to know parts of everything in the system to get it to work. For example, to get a minimal “hello world” page up in your browser, you need to set up the ORM, templates, views, routes, and a few other things. Contrast that with a framework like Flask (which is, admittedly, less feature-rich), where less than 20 lines of code can get your content displayed on a web page.

Building Our Simple Django Application

If you’re not familiar with Django, their tutorial is a good place to start learning how to get a base system configured and running. For this article, I’ve created a similar system using a PostgreSQL database and a few simple models and views. But we won’t spend time describing how to set up a complete Django application. That’s what the Django tutorial is for.

My application here is different from the tutorial in that I use PostgreSQL—instead of the default SQLite—as the database engine. The trouble with SQLite (besides poor performance in a web application setting) is that it is file-based, and the file resides on the same server as the web application that uses it. Most cloud platforms assume a stateless deployment, meaning the container that holds the application is wiped clean and refreshed every deployment. So, your database should run on a separate server from the web application. PostgreSQL will provide that for us.

The source code for this mini-demo project is available in this GitHub repository.

Install Python dependencies

After you have cloned the repository, start up a virtual environment and install the Python dependencies for this project:



(venv) ~/project$ pip install -r requirements.txt

Set up Django to use PostgreSQL

To use PostgreSQL with Django, we use the following packages:

psycopg2 provides the engine drivers for Postgres.
dj-database-url helps us set up the database connection string from an environment variable (useful for local testing and cloud deployments).

In our Django app, we navigate to mysite/mysite/ and modify settings.py (around line 78) to use PostgreSQL.



DATABASES = {"default": dj_database_url.config(conn_max_age=600, ssl_require=True)}

We’ll start by testing out our application locally. So, on your local PostgreSQL instance, create a new database.



postgres=# create database django_test_db;

Assuming our PostgreSQL username is dbuser and the password is password, then our DATABASE_URL will look something like this:



postgres://dbuser:password@localhost:5432/django_test_db

From here, we need to run our database migrations to set up our tables.



(venv) ~/project$ \
  DATABASE_URL=postgres://dbuser:password@localhost:5432/django_test_db\
  python mysite/manage.py migrate

Operations to perform:
  Apply all migrations: admin, auth, contenttypes, movie_journal, sessions
Running migrations:
  Applying contenttypes.0001_initial... OK
  Applying auth.0001_initial... OK
  Applying admin.0001_initial... OK
  Applying admin.0002_logentry_remove_auto_add... OK
  Applying admin.0003_logentry_add_action_flag_choices... OK
  Applying contenttypes.0002_remove_content_type_name... OK
  Applying auth.0002_alter_permission_name_max_length... OK
  Applying auth.0003_alter_user_email_max_length... OK
  Applying auth.0004_alter_user_username_opts... OK
  Applying auth.0005_alter_user_last_login_null... OK
  Applying auth.0006_require_contenttypes_0002... OK
  Applying auth.0007_alter_validators_add_error_messages... OK
  Applying auth.0008_alter_user_username_max_length... OK
  Applying auth.0009_alter_user_last_name_max_length... OK
  Applying auth.0010_alter_group_name_max_length... OK
  Applying auth.0011_update_proxy_permissions... OK
  Applying auth.0012_alter_user_first_name_max_length... OK
  Applying movie_journal.0001_initial... OK
  Applying sessions.0001_initial... OK

Test application locally

Now that we have set up our database, we can spin up our application and test it in the browser.



(venv) ~/project$ \
  DATABASE_URL=postgres://dbuser:password@localhost:5432/django_test_db\
  python mysite/manage.py runserver

…
Django version 4.2.11, using settings 'mysite.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.

In our browser, we visit http://localhost:8000/movie-journal. This is what we see:

We’re up and running! We can go through the flow of creating a new journal entry.

Looking in our database, we see the record for our new entry.



django_test_db=# select * from movie_journal_moviejournalentry;
-[ RECORD 1 ]+-------------------------------------------------------------
id           | 1
title        | Best of the Best
imdb_link    | https://www.imdb.com/title/tt0096913/
is_positive  | t
review       | Had some great fight scenes. The plot was amazing.
release_year | 1989
created_at   | 2024-03-29 09:36:59.24143-07
updated_at   | 2024-03-29 09:36:59.241442-07

Our application is working. We’re ready to deploy. Let’s walk through how to deploy using GitHub Actions directly from our repository on commit.

The Power of GitHub Actions

Over the years, GitHub Actions has built up a large library of jobs/workflows, providing lots of reusable code and conveniences for developers.

With CI/CD, a development team can get fast feedback as soon as code changes are committed and pushed. Typical jobs found in a CI pipeline include style checkers, static analysis tools, and unit test runners. All of these help enforce good coding practices and adherence to team standards. Yes, all these tools existed before. But now, developers don’t need to worry about manually running them or waiting for them to finish.

Push your changes to the remote branch, and the job starts automatically. Go on to focus on your next coding task as GitHub runs the current jobs and displays their results as they come in. That’s the power of automation and the cloud, baby!

Plug-and-play GitHub Action workflows

You can even have GitHub create your job configuration file for you. Within your repository on GitHub, click Actions. You’ll see an entire library of templates, giving you pre-built workflows that could potentially fit your needs.

Let’s click on the Configure button for the Pylint workflow. It looks like this:



name: Pylint

on: [push]

jobs:
  build:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: ["3.8", "3.9", "3.10"]
    steps:
    - uses: actions/checkout@v3
    - name: Set up Python ${{ matrix.python-version }}
      uses: actions/setup-python@v3
      with:
        python-version: ${{ matrix.python-version }}
    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
        pip install pylint
    - name: Analysing the code with pylint
      run: |
        pylint $(git ls-files '*.py')

This configuration directs GitHub Actions to create a new workflow in your repository named Pylint. It triggers a push to any branch. It has one job, build, that runs the latest Ubuntu image. Then, it runs all the steps for each of the three different versions of Python specified.

The steps are where the nitty-gritty work is defined. In this example, the job checks out your code, sets up the Python version, installs dependencies, and then runs the linter over your code.

Let’s create our own GitHub Action workflow to deploy our application directly to Heroku.

Deploying to Heroku via a GitHub Action

Here’s the good news: it’s easy. First, sign up for a Heroku account and install the Heroku CLI.

Login, create app, and PostgreSQL add-on

With the Heroku CLI, we run the following commands to create our app and the PostgreSQL add-on:



$ heroku login

$ heroku apps:create django-github
Creating ⬢ django-github... done
https://django-github-6cbf23e36b5b.herokuapp.com/ | https://git.heroku.com/django-github.git

$ heroku addons:create heroku-postgresql:mini --app django-github
Creating heroku-postgresql:mini on ⬢ django-github... ~$0.007/hour (max $5/month)
Database has been created and is available
 ! This database is empty. If upgrading, you can transfer
 ! data from another database with pg:copy

Add Heroku app host to allowed hosts list in Django

In our Django application settings, we need to update the list of ALLOWED_HOSTS, which represent the host/domain names that your Django site can serve. We need to add the host from our newly created Heroku app. Edit mysite/mysite/settings.py, at around line 31, to add your Heroku app host. It will look similar to this:



ALLOWED_HOSTS = ["localhost", "django-github-6cbf23e36b5b.herokuapp.com"]

Don’t forget to commit this file to your repository.

Procfile and requirements.txt

Next, we need to add a Heroku-specific file called Procfile. This goes into the root folder of our repository. This file tells Heroku how to start up our app and run migrations. It should have the following contents:



web: gunicorn --pythonpath mysite mysite.wsgi:application
release: cd mysite && ./manage.py migrate --no-input

Heroku will also need your requirements.txt file so it knows which Python dependencies to install.

Get your Heroku API key

We will need our Heroku account API key. We’ll store this at GitHub so that our GitHub Action has authorization to deploy code to our Heroku app.

In your Heroku account settings, find the auto-generated API key and copy the value.

Then, in your GitHub repository settings, navigate to Secrets and variables > Actions.

On that page, click New repository secret. Supply a name for your repository secret and. Then, paste in your Heroku API key and click Add secret.

Your list of GitHub repository secrets should look like this:

Create the job configuration file

Let’s create our GitHub Action workflow. Typically, we configure CI/CD jobs with a YAML file. With GitHub Actions, this is no different.

To add an action to your repository, create a .github subfolder in your project, and then create a workflows subfolder within that one. In .github/workflows/, we’ll create a file called django.yml. Your project tree should look like this:



.
├── .git
│   └── …
├── .github
│   └── workflows
│       └── django.yml

├── mysite
│   ├── manage.py
│   ├── mysite
│   │   ├── …
│   │   └── settings.py
│   └── …
├── Procfile
└── requirements.txt

Our django.yml file has the following contents:



name: Django CI

on:
  push:
    branches: [ "main" ]

jobs:
  release:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v2
       - uses: akhileshns/heroku-deploy@v3.13.15
         with:
           heroku_api_key: ${{ secrets.HEROKU_API_KEY }}
           heroku_app_name: "<your-heroku-app-name>"
           heroku_email: "<your-heroku-email>"

This workflow builds off of the Deploy to Heroku Action in the GitHub Actions library. In fact, using that pre-built action makes our Heroku deployment simple. The only things you need to configure in this file are your Heroku app name and account email.

When we commit this file to our repo and push our main branch to GitHub, this kicks off our GitHub Action job for deploying to Heroku. In GitHub, we click the Actions tab and see the newly triggered workflow. When we click the release job in the workflow, this is what we see:

Near the bottom of the output of the deploy step, we see results from the Heroku deploy:

When we look in our Heroku app logs, we also see the successful deploy.

And finally, when we test our Heroku-deployed app in our browser, we see that it’s up and running.

Congrats! You’ve successfully deployed your Django action to Heroku via a GitHub Action!

Conclusion

In this article, we set up a simple Django application with a PostgreSQL database. Then, we walked through how to use GitHub Actions to deploy the application directly to your Heroku on commit.

Django is a feature-rich web application framework for Python. Although for some cloud platforms, it can take some time to get things configured correctly, that’s not the case when you’re deploying to Heroku with GitHub Actions. Convenient off-the-shelf tools are available in both GitHub and Heroku, and they make deploying your Django application a breeze.

Working with Heroku Logplex for Comprehensive Application Logging

Michael Bogan — Tue, 28 May 2024 15:13:48 +0000

With the complexity of modern software applications, one of the biggest challenges for developers is simply understanding how their applications behave. Understanding the behavior of your app is key to maintaining its stability, performance, and security.

This is a big reason why we do application logging: to capture and record events through an application’s lifecycle, so that we can gain valuable insights into our application. What kinds of insights? Application activity (user interactions, system events, and so on), errors and exceptions, resource usage, potential security threats, and more.

When developers can capture and analyze these logs effectively, this improves application stability and security, which, in turn, improves the user experience. It’s a win-win for everybody.

Application logging is easy—if you have the right tools. In this post, we’ll walk through using Heroku Logplex as a centralized logging solution. We’ll start by deploying a simple Python application to Heroku. Then, we’ll explore the different ways to use Logplex to view and filter our logs. Finally, we’ll show how to use Logplex to send your logs to an external service for further analysis.

Ready to dive in? Let’s start with a brief introduction to Heroku Logplex.

Introducing Heroku Logplex

Heroku Logplex is a central hub that collects, aggregates, and routes log messages from various sources across your Heroku applications. Those sources include:

Dyno logs: generated by your application running on Heroku dynos.
Heroku logs: generated by Heroku itself, such as platform events and deployments.
Custom sources: generated by external sources, such as databases or third-party services.

By consolidating logs in a single, central place, Logplex simplifies log management and analysis. You can find all your logs in one place for simplified monitoring and troubleshooting. You can perform powerful filtering and searching on your logs. And you can even route logs to different destinations for further processing and analysis.

Core components

At its heart, Heroku Logplex consists of three crucial components that work together to streamline application logging:

Log sources are the starting points where log messages originate within your Heroku environment. They are your dyno logs, Heroku logs, and custom sources which we mentioned above.
Log drains are the designated destinations for your log messages. Logplex allows you to configure drains to route your logs to various endpoints for further processing. Popular options for log drains include:

External logging services with advanced log management features, dashboards, and alerting capabilities. Examples are Datadog, Papertrail, and Sumo Logic.
Notification systems that send alerts or notifications based on specific log entries, enabling real-time monitoring and troubleshooting.
Custom destinations such as your own Syslog or web server.
Log filters are powerful tools that act as checkpoints, allowing you to refine the log messages before they reach their final destinations. Logplex allows you to filter logs based on source, log level, and even message content. By using filters, you can significantly reduce the volume of data sent to your drains, focusing only on the most relevant log entries for that specific destination.

Routing and processing

As Logplex collects log messages from all your defined sources, it passes these messages through your configured filters, potentially discarding entries that don't match the criteria. Finally, filtered messages are routed to their designated log drains for further processing or storage.

Alright, enough talk. Show me how, already!

Integrating Logplex with Your Application

Let’s walk through how to use Logplex for a simple Python application. To get started, make sure you have a Heroku account. Then, download and install the Heroku CLI.

Demo application

You can find our very simple Python script (main.py) in the GitHub repo for this demo. Our script runs an endless integer counter, starting from zero. With each iteration, it emits a log message (cycling through log levels INFO, DEBUG, ERROR, and WARN). Whenever it detects a prime number, it emits an additional CRITICAL log event to let us know. We use isprime from the sympy library to help us determine if a number is prime.

To run this Python application on your local machine, first clone the repository. Then, install the dependencies:

(venv) ~/project$ pip install -r requirements.txt

Next, start up the Python application. We use gunicorn to spin up a server that binds to a port, while our prime number logging continues to run in the background. (We do this because a Heroku deployment is designed to bind to a port, so that’s how we’ve written our application even though we’re focused on logging).

(venv) ~/project$ gunicorn -w 1 --bind localhost:8000 main:app
[2024-03-25 23:18:59 -0700] [785441] [INFO] Starting gunicorn 21.2.0
[2024-03-25 23:18:59 -0700] [785441] [INFO] Listening at: http://127.0.0.1:8000 (785441)
[2024-03-25 23:18:59 -0700] [785441] [INFO] Using worker: sync
[2024-03-25 23:18:59 -0700] [785443] [INFO] Booting worker with pid: 785443
{"timestamp": "2024-03-25T23:18:59.507828Z", "level": "INFO", "name": "root", "message": "New number", "Number": 0}
{"timestamp": "2024-03-25T23:19:00.509182Z", "level": "DEBUG", "name": "root", "message": "New number", "Number": 1}
{"timestamp": "2024-03-25T23:19:01.510634Z", "level": "ERROR", "name": "root", "message": "New number", "Number": 2}
{"timestamp": "2024-03-25T23:19:02.512100Z", "level": "CRITICAL", "name": "root", "message": "Prime found!", "Prime Number": 2}
{"timestamp": "2024-03-25T23:19:05.515133Z", "level": "WARNING", "name": "root", "message": "New number", "Number": 3}
{"timestamp": "2024-03-25T23:19:06.516567Z", "level": "CRITICAL", "name": "root", "message": "Prime found!", "Prime Number": 3}
{"timestamp": "2024-03-25T23:19:09.519082Z", "level": "INFO", "name": "root", "message": "New number", "Number": 4}

Simple enough. Now, let’s get ready to deploy it and work with logs.

Create the app

We start by logging into Heroku through the CLI.

$ heroku login

Then, we create a new Heroku app. I’ve named my app logging-primes-in-python, but you can name yours whatever you’d like.

$ heroku apps:create logging-primes-in-python
Creating ⬢ logging-primes-in-python... done
https://logging-primes-in-python-6140bfd3c044.herokuapp.com/ | https://git.heroku.com/logging-primes-in-python.git

Next, we create a Heroku remote for our GitHub repo with this Python application.

$ heroku git:remote -a logging-primes-in-python
set git remote heroku to https://git.heroku.com/logging-primes-in-python.git

A note on requirements.txt and Procfile

We need to let Heroku know what dependencies our Python application needs, and also how it should start up our application. To do this, our repository has two files: requirements.txt and Procfile.

The first file, requirements.txt, looks like this:

python-json-logger==2.0.4
pytest==8.0.2
sympy==1.12
gunicorn==21.2.0

And Procfile looks like this:

web: gunicorn -w 1 --bind 0.0.0.0:${PORT} main:app

That’s it. Our entire repository has these files:

$ tree
.
├── main.py
├── Procfile
└── requirements.txt

0 directories, 3 files

Deploy the code

Now, we’re ready to deploy our code. We run this command:

$ git push heroku main
…
remote: Building source:
remote: 
remote: -----> Building on the Heroku-22 stack
remote: -----> Determining which buildpack to use for this app
remote: -----> Python app detected
…
remote: -----> Installing requirements with pip
…
remote: -----> Launching...
remote:        Released v3
remote:        https://logging-primes-in-python-6140bfd3c044.herokuapp.com/ deployed to Heroku
remote: 
remote: Verifying deploy... done.

Verify the app is running

To verify that everything works as expected, we can dive into Logplex right away. Logplex is enabled by default for all Heroku applications.

$ heroku logs --tail -a logging-primes-in-python
…
2024-03-22T04:34:15.540260+00:00 heroku[web.1]: Starting process with command `gunicorn -w 1 --bind 0.0.0.0:${PORT} main:app`
…
2024-03-22T04:34:16.425619+00:00 app[web.1]: {"timestamp": "2024-03-22T04:34:16.425552Z", "level": "INFO", "name": "root", "message": "New number", "taskName": null, "Number": 0}
2024-03-22T04:34:17.425987+00:00 app[web.1]: {"timestamp": "2024-03-22T04:34:17.425837Z", "level": "DEBUG", "name": "root", "message": "New number", "taskName": null, "Number": 1}
2024-03-22T04:34:18.000000+00:00 app[api]: Build succeeded
2024-03-22T04:34:18.426354+00:00 app[web.1]: {"timestamp": "2024-03-22T04:34:18.426205Z", "level": "ERROR", "name": "root", "message": "New number", "taskName": null, "Number": 2}
2024-03-22T04:34:19.426700+00:00 app[web.1]: {"timestamp": "2024-03-22T04:34:19.426534Z", "level": "CRITICAL", "name": "root", "message": "Prime found!", "taskName": null, "Prime Number": 2}

We can see that logs are already being written. Heroku’s log format is following this scheme:

timestamp source[dyno]: message

Timestamp: The date and time recorded at the time the dyno or component produced the log line. The timestamp is in the format specified by RFC5424 and includes microsecond precision.
Source: All of your app’s dynos (web dynos, background workers, cron) have the source app. Meanwhile, all of Heroku’s system components (HTTP router, dyno manager) have the source heroku.
Dyno: The name of the dyno or component that wrote the log line. For example, web dyno #1 appears as web.1, and the Heroku HTTP router appears as router.
Message: The content of the log line. Logplex splits any lines generated by dynos that exceed 10,000 bytes into 10,000-byte chunks without extra trailing newlines. It submits each chunk that is submitted as a separate log line.

View and filter logs

We’ve seen the first option for examining our logs, the Heroku CLI. You can use command line arguments, such as --source and --dyno, to use filters and specify which logs to view.

To specify the number of (most recent) log entries to view, do this:

$ heroku logs --num 10

To filter down logs to a specific dyno or source, do this:

$ heroku logs --dyno web.1
$ heroku logs --source app

Of course, you can combine these filters, too:

$ heroku logs --source app --dyno web.1

The Heroku Dashboard is another place where you can look at your logs. On your app page, click More -> View logs.

Here is what we see:

If you look closely, you’ll see different sources: heroku and app.

Log Drains

Let’s demonstrate how to use a log drain. For this, we’ll use BetterStack (formerly Logtail). We created a free account. After logging in, we navigated to the Source page, and clicked Connect source.

We enter a name for our source and select Heroku as the source platform. Then, we click Create source.

After creating our source, BetterStack provides the Heroku CLI command we would use to add a log drain for sending logs to BetterStack.

Technically, this command adds an HTTPS drain that points to an HTTPS endpoint from BetterStack. We run the command in our terminal, and then we restart our application:

$ heroku drains:add \
"https://in.logs.betterstack.com:6515/events?source_token=YKGWLN7****************" \
-a logging-primes-in-python


Successfully added drain https://in.logs.betterstack.com:6515/events?source_token=YKGWLN7*****************

$ heroku restart -a logging-primes-in-python

Almost instantly, we begin to see our Heroku logs appear on the Live tail page at BetterStack.

By using a log drain to send our logs from Heroku Logplex to an external service, we can take advantage of the features from BetterStack to work with our Heroku logs. For example, we can create visualization charts and configure alerts on certain log events.

Custom drains

In our example above, we created a custom HTTPS log drain that happened to point to an endpoint from BetterStack. However, we can send our logs to any endpoint we want. We could even send our logs to another Heroku app! 🤯 Imagine building a web service on Heroku that only Heroku Logplex can make POST requests to.

Logging best practices

Before we conclude our walkthrough, let’s briefly touch on some logging best practices.

Focus on relevant events: Log only the information that’s necessary to understand and troubleshoot your application's behavior. Prioritize logging application errors, user actions, data changes, and other crucial activities.
Enrich logs with context: Include details that provide helpful context to logged events. Your future troubleshooting self will thank you. So, instead of just logging "User logged in," capture details like user ID, device information, and relevant data associated with the login event.
Embrace structured logging: Use a standardized format like JSON to make your logs machine-readable. This allows easier parsing and analysis by logging tools, saving you time in analysis.
Protect sensitive data: Never log anything that could compromise user privacy or violate data regulations. This includes passwords, credit card information, or other confidential data.
Take advantage of log levels: Use different log levels (like DEBUG, INFO, WARNING, and ERROR) to categorize log events based on their severity. This helps with issue prioritization, allowing you to focus on critical events requiring immediate attention.

Conclusion

Heroku Logplex empowers developers and operations teams with a centralized and efficient solution for application logging within the Heroku environment. While our goal in this article was to provide a basic foundation for understanding Heroku Logplex, remember that the platform offers a vast array of advanced features to explore and customize your logging based on your specific needs.
As you dig deeper into Heroku’s documentation, you’ll come across advanced functionalities like:

Customizable log processing: Leverage plugins and filters to tailor log processing workflows for specific use cases.
Real-time alerting: Configure alerts based on log patterns or events to proactively address potential issues.
Advanced log analysis tools: Integrate with external log management services for comprehensive log analysis, visualization, and anomaly detection. By understanding the core functionalities and exploring the potential of advanced features, you can leverage Heroku Logplex to create a robust and efficient logging strategy. Ultimately, good logging will go a long way in enhancing the reliability, performance, and security of your Heroku applications.

Caching RESTful API requests with Heroku’s Redis Add-on

Michael Bogan — Wed, 17 Apr 2024 13:39:30 +0000

Most software developers encounter two main problems: naming things, caching, and off-by-one errors. 🤦🏻‍♂️

In this tutorial, we’ll deal with caching. We’ll walk through how to implement RESTful request caching with an open source-licensed version of Redis. We’ll also set up and deploy this system easily with Heroku.

For this demo, we’ll build a Node.js application with the Fastify framework, and we’ll integrate caching with Redis to reduce certain types of latency.

Ready to dive in? Let’s go!

Node.js + Fastify + long-running tasks

As I’m sure our readers know, Node.js is a very popular platform for building web applications. With its support for JavaScript (or TypeScript, or both at the same time!), Node.js allows you to use the same language for both the frontend and the backend of your application. It also has a rich event loop that makes asynchronous request handling more intuitive.

The concurrency model in Node.js is very performant, able to handle upwards of 15,000 requests per second. But even then, you might still run into situations where the request latency is unacceptably high. We’ll show this with our application.

As you follow along, you can always browse the codebase for this mini demo at my GitHub repository.

Initialize the basic application

By using Fastify, you can quickly get a Node.js application up and running to handle requests. Assuming you have Node.js installed, you’ll start by initializing a new project. We’ll use npm as our package manager.

After initializing a new project, we will install our Fastify-related dependencies.

~/project$ npm i fastify fastify-cli fastify-plugin

Then, we update our package.json file to add two scripts and turn on ES module syntax. We make sure to have the following lines:

  "type": "module",
  "main": "app.js",
  "scripts": {
    "start": "fastify start -a 0.0.0.0 -l info app.js",
    "dev": "fastify start -p 8000 -w -l info -P app.js"
  },

From there, we create our first file (routes.js) with an initial route:

// routes.js

export default async function (fastify, _opts) {
  fastify.get("/api/health", async (_, reply) => {
    return reply.send({ status: "ok" });
  });
}

Then, we create our app.js file that prepares a Fastify instance and registers the routes:

// app.js
import routes from "./routes.js";

export default async (fastify, opts) => {
  fastify.register(routes);
};

These two simple files—our application and our route definitions—are all we need to get up and running with a small Fastify service that exposes one endpoint: /api/health. Our dev script in package.json is set to run the fastify-cli to start our server on localhost port 8000, which is good enough for now. We start up our server:

~/project$ npm run dev

Then, in another terminal window, we use curl to hit the endpoint:

~$ curl http://localhost:8000/api/health 
{"status":"ok"}

Add a simulated long-running process

We’re off to a good start. Next, let’s add another route to simulate a long-running process. This will help us gather some latency data. In routes.js, we add another route handler within our exported default async function:

 fastify.get("/api/user-data", async (_, reply) => {
    await sleep(5000);
    const userData = readData();
    return reply.send({ data: userData });
  });

This exposes another endpoint: /api/user-data. Here, we have a method to simulate reading a lot of data from a database (readData) and a long-running process (sleep). We define those methods in routes.js as well. They look like this:

import fs from "fs";

function readData() {
  try {
    const data = fs.readFileSync("data.txt", "utf8");
    return data;
  } catch (err) {
    console.error(err);
  }
}

function sleep(ms) {
  return new Promise((resolve) => {
    setTimeout(resolve, ms);
  });
}

With our new route in place, we restart our server (npm run dev).

Measure latency with curl

How do we measure latency? The simplest way is to use curl. Curl captures various time profiling metrics when it makes requests. We just need to format curl’s output so that we can easily see the various latency values available. To do this, we define the output we want to see with a text file (curl-format.txt):

    time_namelookup:  %{time_namelookup}s\n
        time_connect:  %{time_connect}s\n
     time_appconnect:  %{time_appconnect}s\n
    time_pretransfer:  %{time_pretransfer}s\n
       time_redirect:  %{time_redirect}s\n
  time_starttransfer:  %{time_starttransfer}s\n
  -------------------  ----------\n
          time_total:  %{time_total}s\n

With our output format defined, we can use it with our next curl call:

curl -w "@curl-format.txt" \
     -o /dev/null -s \
     "http://localhost:8000/api/user-data"

The response we receive looks like this:

  time_namelookup:  0.000028s
        time_connect:  0.000692s
     time_appconnect:  0.000000s
    time_pretransfer:  0.000772s
       time_redirect:  0.000000s
  time_starttransfer:  5.055683s
                       ----------
          time_total:  5.058479s

Well, that’s not good. Over five seconds is way too long for a transfer time (the time it takes the server to actually handle the request). Imagine if this endpoint was being hit hundreds or thousands of times per second! Your users would be frustrated, and your server may crash under the weight of continually re-doing this work.

Redis to the rescue!

Caching your responses is the first line of defense to reduce your transfer time (assuming you’ve addressed any of the poor programming practices that might be causing the latency!). So, let’s assume we’ve done everything we can do to reduce latency, but our application still needs five seconds to put this complex data together and return it to the user.

In our scenario, because the data is the same every time for every request to /api/user-data, we have a perfect candidate for caching. With caching, we’ll perform the necessary computation once, cache the result, and return the cached value for all subsequent requests.

Redis is a performant, in-memory key/value store, and it’s a common tool used for caching. To leverage it, we first install Redis on our local machine. Then, we need to add Fastify’s Redis plugin to our project:

~/project$ npm i @fastify/redis

Register the Redis plugin with Fastify

We create a file, redis.js, which configures our Redis plugin and registers it with Fastify. Our file looks like this:

// redis.js

const REDIS_URL = process.env.REDIS_URL || "redis://127.0.0.1:6379";

import fp from "fastify-plugin";
import redis from "@fastify/redis";

const parseRedisUrl = (redisUrl) => {
  const url = new URL(redisUrl);
  const password = url.password;
  return {
    host: url.hostname,
    port: url.port,
    password,
  };
};

export default fp(async (fastify) => {
  fastify.register(redis, parseRedisUrl(REDIS_URL));
});

Most of the lines in this file are dedicated to parsing a REDIS_URL value into a host, port, and password. If we have REDIS_URL set properly at runtime as an environment variable, then registering Redis with Fastify is simple. After configuring our plugin, we just need to modify app.js to use it:

// app.js

import redis from "./redis.js";
import routes from "./routes.js";

export default async (fastify, opts) => {
  fastify.register(redis);
  fastify.register(routes);
};

Now we have access to our Redis instance by referencing fastify.redis anywhere within our app.

Modify our endpoint to use caching

With Redis in the mix, let’s change our /api/user-data endpoint to use caching:

  fastify.get("/api/user-data", async (_, reply) => {
    const { redis } = fastify;

    // check if data is in cache
    const data = await redis.get("user-data", (err, val) => {
      if (val) {
        return { data: val };
      }
      return null;
    });

    if (data) {
      return reply.send(data);
    }

    // simulate a long-running task
    await sleep(5000);
    const userData = readData();


    // add data to the cache
    redis.set("user-data", userData);


    return reply.send({ data: userData });
  });

Here, you see that we’ve hardcoded in Redis a single key, user-data, and stored our data under that key. Of course, our key could be a user ID or some other value that identifies a particular type of request or state. Also, we could set a timeout value to expire our key, in the case that we expect data to change after a certain window of time.

If there is data in the cache, then we’ll return it and skip all the time-consuming work. Otherwise, do the long-running computation, add the result to the cache, and then return it to the user.

What do our transfer times look like after hitting this endpoint two more times (the first one to add the data into the cache, and the second one to retrieve it)?

   time_namelookup:  0.000023s
        time_connect:  0.000560s
     time_appconnect:  0.000000s
    time_pretransfer:  0.000729s
       time_redirect:  0.000000s
  time_starttransfer:  0.044512s
                       ----------
          time_total:  0.047479s

Much better! We’ve reduced our request times from several seconds to milliseconds. That’s a huge improvement in performance!

Redis has many more features that may be useful here, including having key/value pairs timeout after a certain amount of time; that’s a more common scenario in production environments.

Using Redis in your Heroku deployment

Up to this point, we’ve only shown how this works in a local environment. Now, let’s go one step further and deploy it all to the cloud. Fortunately, Heroku provides many options for deploying web applications and working with Redis. Let’s walk through how to get set up there.

After signing up for a Heroku account and installing their CLI tool, we’re ready to create a new app. In our case, we’ll call our app fastify-with-caching. Here are our steps:

Step 1: Login to Heroku

~/projects$ heroku login
...
Logging in... done

Step 2: Create the Heroku app

When we create our Heroku app, we’ll get back our Heroku app URL. We take note of this because we’ll use it in our subsequent curl requests.

~/project$ heroku create -a fastify-with-caching
Creating ⬢ fastify-with-caching... done
https://fastify-with-caching-3e247d11f4ad.herokuapp.com/ | https://git.heroku.com/fastify-with-caching.git

Step 3: Add the Redis add-on

We need to set up a Redis add-on that meets our application’s needs. For our demo project, it’s sufficient to create a Mini-tier Redis instance:

~/project$ heroku addons:create heroku-redis:mini -a fastify-with-caching
Creating heroku-redis:mini on ⬢ fastify-with-caching…
…
redis-transparent-98258 is being created in the background.
…

Spinning up the Redis instance may take two or three minutes. We can check the status of our instance periodically:

~/project$ heroku addons:info redis-transparent-98258
...
State:        creating

Not too long after, we see this:

State:        created

We’re just about ready to go!

When Heroku spins up our Redis add-on, it also adds our Redis credentials as config variables attached to our Heroku app. We can run the following command to see these config variables:

~/project$ heroku config -a fastify-with-caching
=== fastify-with-caching Config Vars

REDIS_TLS_URL: rediss://:p171d98f7696ab7eb2319f7b78083af749a0d0bb37622fc420e6c1205d8c4579c@ec2-18-213-142-76.compute-1.amazonaws.com:15940
REDIS_URL:     redis://:p171d98f7696ab7eb2319f7b78083af749a0d0bb37622fc420e6c1205d8c4579c@ec2-18-213-142-76.compute-1.amazonaws.com:15939

(Your credentials, of course, will be unique and different from what you see above.)

Notice that we have a REDIS_URL variable all set up for us. It’s a good thing our redis.js file is coded to properly parse an environment variable called REDIS_URL.

Step 4: Create a Heroku remote

Finally, we need to create a Heroku remote in our git repo so that we can easily deploy with git.

~/project$ heroku git:remote -a fastify-with-caching
set git remote heroku to https://git.heroku.com/fastify-with-caching.git

Step 5: Deploy!

Now, when we push our branch to our Heroku remote, Heroku will build and deploy our application.

~/project$ git push heroku main
...
remote: Building source:
remote: 
remote: -----> Building on the Heroku-22 stack
remote: -----> Determining which buildpack to use for this app
remote: -----> Node.js app detected
remote:        
remote: -----> Creating runtime environment
...
remote: -----> Compressing...
remote:        Done: 50.8M
remote: -----> Launching...
remote:        Released v4
remote:        https://fastify-with-caching-3e247d11f4ad.herokuapp.com/ deployed to Heroku
remote: 
remote: Verifying deploy... done.

Our application is up and running. It’s time to test it.

Test our deployed application

We start with a basic curl request to our /api/health endpoint:

$ curl https://fastify-with-caching-3e247d11f4ad.herokuapp.com/api/health
{"status":"ok"}

Excellent. That looks promising.

Next, let’s send our first request to the long-running process and capture the latency metrics:

$ curl \
  -w "@curl-format.txt" \
  -o /dev/null -s \
  https://fastify-with-caching-3e247d11f4ad.herokuapp.com/api/user-data

     time_namelookup:  0.035958s
        time_connect:  0.101336s
     time_appconnect:  0.249308s
    time_pretransfer:  0.249389s
       time_redirect:  0.000000s
  time_starttransfer:  5.384986s
  -------------------  ----------
          time_total:  6.554382s

When we send the same request a second time, here’s the result:

$ curl \
  -w "@curl-format.txt" \
  -o /dev/null -s \
  https://fastify-with-caching-3e247d11f4ad.herokuapp.com/api/user-data

     time_namelookup:  0.025807s
        time_connect:  0.091763s
     time_appconnect:  0.236050s
    time_pretransfer:  0.236119s
       time_redirect:  0.000000s
  time_starttransfer:  0.334859s
  -------------------  ----------
          time_total:  1.276264s

Much better! Caching allows us to bypass the long-running processes. From here, we can build out a much more robust caching mechanism for our application across all our routes and processes. We can continue to lean on Heroku and Heroku’s Redis add-on when we need to deploy our application to the cloud.

Bonus Tip: Clearing the cache for future tests

By the way, if you want to test this more than once, then you may occasionally need to delete the user-data key/value pair in Redis. You can use the Heroku CLI to access the Redis CLI for your Redis instance:

~$ heroku redis:cli -a fastify-with-caching
Connecting to redis-transparent-98258 (REDIS_TLS_URL, REDIS_URL):
ec2-18-213-142-76.compute-1.amazonaws.com:15940> DEL user-data
1

Conclusion

In this tutorial, we explored how caching can greatly improve your web service's response time in cases where identical requests would produce identical responses. We looked at how to implement this with Redis, the industry-standard caching tool. We did this all with ease within a Node.js application that leverages the Fastify framework. Lastly, we deployed our demo application to Heroku, using their built-in Heroku Data for Redis instance management to cache in the cloud.

Happy coding!

Managing Django Media & Static Files on Heroku with Bucketeer

Daniel Starner — Thu, 13 Jan 2022 18:22:54 +0000

This article will walk through how we correctly persist static & media files for a Django application hosted on Heroku. As a bonus, it will also explain how we can satisfy the additional constraint of specifying private versus public media files based on model definitions.

Before I begin, this post extends from this TestDriven.io article that was written awhile back. I frequent it often when setting up my projects, and have built some extra functionality on top of it over the years. I decided to create a more focused post that references Heroku & Bucketeer with these extra features after helping an individual on StackOverflow.

Dec 26 '21

I think it's because I turn off a PC, where I took these images

This probably is not it, because Heroku doesn't have access to the files on your computer.

When you upload a file to the Django admin, it looks at the DEFAULT_FILE_STORAGE settings configuration to determine how to…

Open Full Answer

So without further ado, let's first dive into what static & media files are and how Heroku dynos manage their filesystem?

What are Media & Static Files

If you are working with a Django project, then you inevitably have all of your Python application code written around a bunch of .py files. These are the code paths of your application, and the end-user - hopefully - never actually sees these files or their contents.

Outside of these business-logic files, it is common to serve users directly from your server's file system. For these static files, Django doesn't need to run any code for them; the framework looks up the file and returns the contents for the requesting user to view.

Some examples of static files include:

Non-templated HTML
CSS & JavaScript files to make your page look nice
User profile pictures
Generated PDFs

Media files in Django are a particular variant of static files. Media files are read from the server's file system as well. Unlike static files, though, they are usually generated files uploaded by users or generated by your application and are associated with a model's FileField or ImageField. In the examples above, user profile pictures and generated PDFs are typical examples of media files.

Django with Media & Static Files

When a new media file is uploaded to a Django web application, the framework looks at the DEFAULT_FILE_STORAGE settings configuration to determine how to store that file. By default, it uses the django.core.files.storage.FileSystemStorage class, which is what most projects start off as having configured. This implementation looks at the MEDIA_ROOT configuration that is defined in the settings.py file and copies the uploaded file contents to a deterministically-created file path under that given MEDIA_ROOT.

For example, if the MEDIA_ROOT is set as /var/www/media, all uploaded files will be copied and written to a location under /var/www/media/.

Heroku with Media & Static Files

Storing these static files on your server's disk file system is okay until you start to work with a containerization platform such as Heroku. To explain why this is the case, it helps to take a step back.

When downloading files on your personal computer, it's okay that these get written to the file system - usually under ~/Downloads or somewhere similar. This download is because you expect your computer's file system to persist across restarts and shutdowns; if you download a file and restart your computer, that downloaded file should still be there once the laptop is finished restarting.

Heroku uses containerization to execute customer workloads. One fact of this environment is that the associated file systems do not persist across restarts and reschedules. Heroku dynos are ephemeral, and they can be destroyed, restarted, and moved without any warning, which replaces the associated filesystem. This situation means that any uploaded files referenced by FileField's andImageField's are just deleted without a trace every time the dyno is restarted, moved, or scaled.

Complete Example Codebase

I will be stepping through the process of configuring the Django application for Heroku & S3-compatible storage, but feel free to reference the repository below for the complete code to browse through.

dstarner / django-heroku-static-file-example

Used in my blog post of detailing private & public static files for a Heroku-served Django application

Properly Managing Django Media & Static Files on Heroku Example

Used in my blog post of detailing private & public static files for a Heroku-served Django application.

Note: This does include a $5.00 / month Bucketeer add-on as a part of the one-click deployment.

View on GitHub

Bootstrapping Django on Heroku

This tutorial aims to help you retrofit an existing Django project with S3-compatible storage, but I'll quickly go through the steps I used to set up the example Django application. It may help those new to Django & Heroku or those who encounter bugs following the rest of the setup process.

You can view the tagged project before the storage change at commit 299bbe2.

Bootstrapped a Django project example
- Uses poetry for dependency management
- All of the Django code is under the example package, and the manage.py file is in the root. I've always found this structure cleaner than the Django apps defined in the project root.
Configured the project for Heroku
- django-heroku package to automatically configure ALLOWED_HOSTS, DATABASE_URL, and more. This reduces the headache of deploying Django on Heroku considerably
- A Procfile that runs a gunicorn process for managing the WSGI application
- An app.json is defined with some fundamental configuration values and resources defined for the project to work
- A release process definition in the Procfile and an associated scripts/release.sh script that runs staticfile collection and database migrations

Introducing Heroku's Bucketeer Add-On

Before we can start managing static and media files, the Django application needs a persistent place to store the files. Again, we can look to Heroku's extensive list of Add-Ons for s3-compatible storage. Ours of choice will be one called Bucketeer.

Heroku's Bucketeer add-on provides an AWS S3 storage bucket to upload and download files for our application. The Django application will use this configured bucket to store files uploaded by the server and download them from the S3 when a user requests the files.

If you'd like to learn more about AWS S3, the widely-popular data storage solution that Bucketeer is built upon, you can read the S3 user documentation.

It is worth mentioning that the base plan for Bucketeer - Hobbyist - is $5 per month. If you plan on spinning up the one-click example posted above, it should only cost a few cents if you proactively destroy the application when you are done using it.

Including the Bucketeer Add-On

To include the Bucketeer add-on in our application, we can configure it through the Heroku CLI, web dashboard, or via the project's app.json file. We will use the third method of including the add-on in an app.json file.

If the project does not have one already, we can create the basic structure listed below, with the critical part being the addition of the "add-ons" configuration. This array defines the "bucketeer:hobbyist" resource that our application will use, and Heroku will install the add-on into our application if it does not already exist. We also include the " as" keyword, which will preface the associated configuration variables with the term BUCKETEER. This prefacing is helpful to keep the generated configuration value names deterministic because, by default, Heroku will generate the prefix as a random color.



{
    // ... rest above
    "addons": [
        // ...other addons...
        {
            "plan": "bucketeer:hobbyist",
            "as": "BUCKETEER"
        }
    ]
}

With the required resources being defined, we can start integrating with our storage add-on.

Implementing Our Storage Solution

The django-storages package is a collection of custom, reuseable storage backends for Django. It aids immensely in saving static and media files to different cloud & storage provider options. One of the supported storage providers is S3, which our Bucketeer add-on is built on. We will leverage the S3 django-storages backend to handle different file types.

Installing `django-storages`

Begin by installing the django-storages package and the related boto3 package used to interface with AWS's S3. We will also lock our dependencies to ensure poetry and our Heroku deployment continue to work as expected.



poetry add django-storages boto3 && poetry lock

Then, just like most Django-related packages, django-storages will need to be added to the project's INSTALLED_APPS in the projects settings.py file. This will allow Django to load the appropriate code flows as the application starts up.



# example/config/settings.py
INSTALLED_APPS = [
    # ... django.X.Y apps above
    'storages',
    # ... custom project apps below
]

Implementing Static, Public & Private Storage Backends

We will return to the settings.py file later to configure the usage of django-storages, but before that can be done, we will implement three custom storage backends:

A storage backend for static files - CSS, Javascript, and publicly accessible images - that will be stored in version control - aka git - and shipped with the application
A public storage backend for dynamic media files that are not stored in version control, such as uploaded files and attachments
A private storage backend for dynamic media files that are not stored in the version control that require extra access to be viewed, such as per-user reports and potentially profile images. Files managed by this backend require an access key and will block access to those without a valid key.

We can extend from django-storages 's S3Boto3Storage storage backend to create these. The following code can be directly "copy and paste "'d into your project. The different settings attributes read in the module will be written shortly, so do not expect this code to work if you import it right now.



# FILE: example/utils/storage_backends.py

from django.conf import settings
from storages.backends.s3boto3 import S3Boto3Storage


class StaticStorage(S3Boto3Storage):
    """Used to manage static files for the web server"""
    location = settings.STATIC_LOCATION
    default_acl = settings.STATIC_DEFAULT_ACL


class PublicMediaStorage(S3Boto3Storage):
    """Used to store & serve dynamic media files with no access expiration"""
    location = settings.PUBLIC_MEDIA_LOCATION
    default_acl = settings.PUBLIC_MEDIA_DEFAULT_ACL
    file_overwrite = False


class PrivateMediaStorage(S3Boto3Storage):
    """
    Used to store & serve dynamic media files using access keys
    and short-lived expirations to ensure more privacy control
    """
    location = settings.PRIVATE_MEDIA_LOCATION
    default_acl = settings.PRIVATE_MEDIA_DEFAULT_ACL
    file_overwrite = False
    custom_domain = False

The attributes listed in each storage backend class perform the following:

location: This dictates the parent directory used in the S3 bucket for associated files. This is concatenated with the generated path provided by a FileField or ImageField 's upload_to method.
default_acl: This dictates the access policy required for reading the files. This dictates the storage backend's access control through values of None, public-read, and private. django-storages and the S3Boto3Storage parent class with translate these into object policies.
file_overwrite: In most cases, it's better not to overwrite existing files if we update a specific path. With this set to False, a unique suffix will be appended to the path to prevent naming collisions.
custom_domain: Disabled here, but you can enable it if you want to use AWS's CloudFront and django-storage to serve from it.

Configure Settings to Use the Storage Backends

With our storage backends defined, we can configure them to be used in different situations via the settings.py file. However, it is challenging to use S3 and these different cloud storage backends while in development, and I've always been a proponent of keeping all resources and files "local" to the development machine, so we will create a logic path that will:

Use the local filesystem to store static and media files for convenience. The Django server will be responsible for serving these files directly.
Use the custom S3 storage backends when an environment variable is enabled. We will use the S3_ENABLED variable to control this, enabling it in our Heroku configuration variables.

First, we will assume that you have a relatively vanilla settings.py file concerning the static- & media-related variables. For reference, a new project should have a block that looks similar to the following:



# Static files (CSS, JavaScript, Images)
# https://docs.djangoproject.com/en/4.0/howto/static-files/

STATIC_URL = 'static/'

STATIC_ROOT = BASE_DIR / 'collected-static'

We will design a slightly advanced control flow that will seamlessly handle the two cases defined above. In addition, it will provide enough control to override each part of the configuration as needed.

Since there are already default values for the static file usage, we can add default values for media file usage. These will be used when serving files locally from the server while in development mode.



STATIC_URL = '/static/'
STATIC_ROOT = BASE_DIR / 'collected-static'

MEDIA_URL = '/media/'
MEDIA_ROOT = BASE_DIR / 'collected-media'

To begin the process of including S3, let's create the controls to manage if we should serve static & media files from the local server or through the S3 storage backend. We will create three variables

S3_ENABLED: controls whether media & static files should use S3 storage by default
LOCAL_SERVE_MEDIA_FILES: controls whether media files should use S3 storage. Defaults to the negated S3_ENABLED value
LOCAL_SERVE_STATIC_FILES: controls whether static files should use S3 storage. Defaults to the negated S3_ENABLED value



from decouple import config  # import explained below

# ...STATIC and MEDIA settings here...

# The following configs determine if files get served from the server or an S3 storage
S3_ENABLED = config('S3_ENABLED', cast=bool, default=False)
LOCAL_SERVE_MEDIA_FILES = config('LOCAL_SERVE_MEDIA_FILES', cast=bool, default=not S3_ENABLED)
LOCAL_SERVE_STATIC_FILES = config('LOCAL_SERVE_STATIC_FILES', cast=bool, default=not S3_ENABLED)

if (not LOCAL_SERVE_MEDIA_FILES or not LOCAL_SERVE_STATIC_FILES) and not S3_ENABLED:
    raise ValueError('S3_ENABLED must be true if either media or static files are not served locally')

In the example above, we are using the python-decouple package to make it easier to read and cast environment variables to Python variables. I highly recommend this package when working with settings.py configurations. We also include a value check to ensure consistency across these three variables. If all three variables are defined in the environment but conflict with one another, the program will throw an error.

We can now start configuring the different configuration variables required by our file storage backends based on those control variables' value(s). We begin by including some S3 configurations required whether we are serving static, media, or both types of files.



if S3_ENABLED:
    AWS_ACCESS_KEY_ID = config('BUCKETEER_AWS_ACCESS_KEY_ID')
    AWS_SECRET_ACCESS_KEY = config('BUCKETEER_AWS_SECRET_ACCESS_KEY')
    AWS_STORAGE_BUCKET_NAME = config('BUCKETEER_BUCKET_NAME')
    AWS_S3_REGION_NAME = config('BUCKETEER_AWS_REGION')
    AWS_DEFAULT_ACL = None
    AWS_S3_SIGNATURE_VERSION = config('S3_SIGNATURE_VERSION', default='s3v4')
    AWS_S3_ENDPOINT_URL = f'https://{AWS_STORAGE_BUCKET_NAME}.s3.amazonaws.com'
    AWS_S3_OBJECT_PARAMETERS = {'CacheControl': 'max-age=86400'}

The above defines some of the variables required by the django-storages S3 backend and sets the values to environment configurations that are provided by the Bucketeer add-on. As previously mentioned, all of the add-on environment variables are prefixed with BUCKETEER_. The S3_SIGNATURE_VERSION environment variable is not required and most likely does not need to be included.

With the S3 configuration together, we can reference the LOCAL_SERVE_MEDIA_FILES and LOCAL_SERVE_STATIC_FILES control variables to override the default static and media file settings if they are desired to be served via S3.



if not LOCAL_SERVE_STATIC_FILES:
    STATIC_DEFAULT_ACL = 'public-read'
    STATIC_LOCATION = 'static'
    STATIC_URL = f'{AWS_S3_ENDPOINT_URL}/{STATIC_LOCATION}/'
    STATICFILES_STORAGE = 'example.utils.storage_backends.StaticStorage'

Notice the last line where STATICFILES_STORAGE is set to the custom Backend we created. That ensures it follows the location & ACL (Access Control List) policies that we configured initially. With this configuration, all static files will be placed under /static/ in the bucket, but feel free to update STATIC_LOCATION if desired.

We can configure a very similar situation for media files.



if not LOCAL_SERVE_MEDIA_FILES:
    PUBLIC_MEDIA_DEFAULT_ACL = 'public-read'
    PUBLIC_MEDIA_LOCATION = 'media/public'

    MEDIA_URL = f'{AWS_S3_ENDPOINT_URL}/{PUBLIC_MEDIA_LOCATION}/'
    DEFAULT_FILE_STORAGE = 'example.utils.storage_backends.PublicMediaStorage'

    PRIVATE_MEDIA_DEFAULT_ACL = 'private'
    PRIVATE_MEDIA_LOCATION = 'media/private'
    PRIVATE_FILE_STORAGE = 'example.utils.storage_backends.PrivateMediaStorage'

The big difference here is that we have configured two different storage backends for media files; one for publicly accessible objects and one for objects that require an access token. When the file is requested, this token will be generated internally by django-storages so you do not have to worry about anonymous public access.

Local Development Serving

Since we will have S3_ENABLED set to False in our local development environment, it will serve static and media files locally through the Django server instead of from S3. We will need to configure the URL routing to handle this scenario. We can configure our urls.py file to serve the appropriate files like so:



from django.conf import settings
from django.conf.urls.static import static
from django.contrib import admin
from django.urls import path


urlpatterns = [
    path('admin/', admin.site.urls),
]

if settings.LOCAL_SERVE_STATIC_FILES:
    urlpatterns += static(settings.STATIC_URL, document_root=settings.STATIC_ROOT)

if settings.LOCAL_SERVE_MEDIA_FILES:
    urlpatterns += static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT)

This will locally serve the static or media files based on the values of the LOCAL_SERVE_STATIC_FILES and LOCAL_SERVE_MEDIA_FILES settings variables we defined.

Enabling S3 Storage

We can enable these storages and our add-on in the app.json file to start using these storage backends. This will effectively disable LOCAL_SERVE_STATIC_FILES and LOCAL_SERVE_MEDIA_FILES to start serving both via S3 when deployed to Heroku.



{
  // ...rest of configs...
  "env": {
    // ...rest of envs...
    "S3_ENABLED": {
      "description": "Enable to upload & serve static and media files from S3",
      "value": "True"
    },
  }
}

Using the Private Storage

By default, Django will use the PublicMediaStorage class for uploading media files, meaning the contents will be publicly accessible to anyone with the link. However, a model can utilize the PrivateMediaStorage backend when desired, which will create short-lived access tokens that prevent the public from viewing the associated object.

The below is an example of using public and private media files on the same model.



from django.db import models

from example.utils.storage_backends import PrivateMediaStorage


class Organization(models.Model):
    """A sample Organization model with public and private file field usage
    """

    logo = models.ImageField(help_text='A publicly accessible company logo')

    expense_report = models.FileField(
        help_text='The private expense report requires a short-lived access token'
        storage=PrivateMediaStorage()  # will create private files
    )

You can see the code for this complete example at commit 265becc. This configuration will allow your project to scale efficiently using Django on Heroku using Bucketeer.

In a future post, we will discuss how to upload and set these files using vanilla Django & Django REST Framework.

As always, if you find any bugs, issues, or unclear explanations, please reach out to me so I can improve the tutorial & experience for future readers.

Take care everyone

Deploying a Kotlin App to Heroku

Michael Bogan — Wed, 13 Oct 2021 11:24:52 +0000

Since its earliest release, Java has touted itself as a "write once, run everywhere" programming language. The idea was that a programmer could develop an app in Java, have it compiled down to bytecode, and become an executable that can run on any platform, regardless of operating system or platform. It was able to do so in part by a runtime known as the Java Virtual Machine, or JVM.

To Java's credit, the JVM was (and still is!) an incredibly fine-tuned runtime that abstracted away a computer's underlying hardware. While Java as a programming language survives to this day, it is often viewed as cumbersome, stodgy, and representative of an outdated approach to implementing solutions.

In the last 10 years, more and more languages that run on the JVM have developed, but look and feel nothing like Java. One such language is Kotlin. Because of the JVM, it has no real performance advantages over regular Java. Still, its strength lies in the fact that it prioritizes legibility in a way that Java does not. Consider, for example, printing a substring in Java:

// Java
String input = "What is the answer to the Ultimate Question of Life, the Universe, and Everything? 42";
String answer = input.substring(input.indexOf("?") + 1);
System.out.println(answer);

You must first get the index of the character you want to be in the substring, add one (since strings are zero-indexed), and call System.out.println to write to stdout.

In Kotlin, this is much shorter:

// Kotlin
val input = "What is the answer to the Ultimate Question of Life, the Universe, and Everything? 42"
val answer = input.substringAfter("?")
println(answer)

Kotlin has garnered so much interest that Google even recommends it over Java for developing Android apps.

In this post, we'll take a quick look at how to develop an app in Kotlin. We'll build a simple API with a PostgreSQL database and deploy it to Heroku to see it live.

Prerequisites

Before we begin, you'll need to make sure you've got the following software installed on your machine:

An account on Heroku. This is completely free and doesn't require any payment information.
The Heroku CLI. Once your application is on Heroku, this will make managing it much easier.
You'll need to have Kotlin installed (>= 1.4).
You'll also need Gradle installed (>= 7.0).

You will also need to be a little familiar with Git and have it installed on your machine.

We’re going to be using the IntelliJ IDE for this Kotlin app. Their documentation provides some guidance on how to create a new project. Make sure you select the following options:

We want to create a Kotlin application that uses Gradle as a build system
Set the name of the project to kotlin-api
Set the JDK version to 16. If you don’t have this version installed, you can select Download JDK… from the dropdown, then choose Oracle Open JDK version 16

After the IDE sets everything up, you should have a directory structure that looks roughly like this:

kotlin-api
├── build.gradle.kts
└── src
    ├── main
    │   ├── kotlin

Our Kotlin files will be kept in src/main/kotlin, and our build logic will be in build.gradle.kts.

Getting started

Gradle is a build tool for a variety of languages. It also acts as a dependency management tool, similar to Maven. You’ll already have some lines in your build.gradle.kts file, which the IDE automatically added to be helpful. You can delete all of that, and paste in these lines instead:

plugins {
    id("java")
    id("org.jetbrains.kotlin.jvm") version "1.5.10"
    id("org.springframework.boot") version "2.4.3"

    id("io.spring.dependency-management") version "1.0.11.RELEASE"
}

group "com.example"
version "0.0.1"

repositories {
    mavenCentral()
}

dependencies {
    implementation("org.jetbrains.kotlin:kotlin-stdlib") 

    implementation("org.springframework.boot:spring-boot-starter-web")    
    implementation("org.springframework.boot:spring-boot-starter")

    developmentOnly("org.springframework.boot:spring-boot-devtools")
}

These lines specify our project's dependencies and where to find them. For example, we want to use [org.springframework.boot](https://plugins.gradle.org/plugin/org.springframework.boot) at version 2.4.3, which is why it's defined within the plugins block. We point out the repositories where the packages can be found—at mavenCentral()—and which exposed classes we want to use (implementation( "org.springframework.boot:spring-boot-starter-web")).

Let's create two small files to test our setup. Create a file called Application.kt in the src/main/kotlin folder and paste in the following:

package com.example

import org.springframework.boot.SpringApplication
import org.springframework.boot.autoconfigure.SpringBootApplication

@SpringBootApplication
open class Application

fun main(args: Array<String>) {
    SpringApplication.run(Application::class.java, *args)
}

This starts a default app using the Spring framework. The real magic happens in this next file, Controller.kt, which you should create alongside Application.kt in src/main/kotlin:

package com.example

import org.springframework.web.bind.annotation.GetMapping
import org.springframework.web.bind.annotation.PathVariable
import org.springframework.web.bind.annotation.RestController

@RestController
class GreetingController {

    @GetMapping("/{name}")
    fun get(@PathVariable name: String) = "Hello, $name"
}

Here, we define a route (@GetMapping("/{name}")), where {name} is a dynamic value. By placing this decorator over a Kotlin method (fun get, or "a function named get"), we're able to match the route to whatever behavior we want—in this case, returning a greeting with the path name for a GET request.

In order for the IDE to know how to launch our application, we need to create a run configuration. At the top of the IDE menu, click the button that says Add Configuration…. Select Add new run configuration, then choose Gradle from the list. For the Gradle project name, enter kotlin-api. In the Tasks field, type bootRun. bootRun is a Gradle task provided by the Spring framework which will compile our code and start the server. Click Ok; you should now have a green Play button in your IDE menu bar. When you click on this, the IDE will execute gradle bootRun to build this Kotlin app and start the server. When that finishes, navigate to http://localhost:8080/world. You should see a nice greeting.

Interacting with the database

Now, let's get to the (somewhat) serious stuff. Suppose we wanted to attach a database to this project. In a Maven/Java world, we'd need to update an XML file and add a reference to a JAR file. In Gradle, we can get by with just adding a few lines to our build.gradle.kts file:

dependencies {
    # ...

    implementation("com.zaxxer:HikariCP:4.0.3")
    runtimeOnly("org.postgresql:postgresql")

    # ...
}

Here, we've included HikariCP in our project, which is a popular database connection driver. We also indicate that we want to "load" the org.postgresql library during runtime. With just these two lines, we've let our configuration know that we want to interact with a PostgreSQL database. If you already have a PostgreSQL database running locally, that's great. You'll be able to continue the rest of this guide locally and see the results when browsing localhost. If you don't have PostgreSQL, don't fret—we'll show you just how easy it is to deploy this app on Heroku, which will take care of the infrastructure for you.

Head back to Controller.kt, and replace it with the contents below. This takes some of what we had from before but adds to it. We'll go over the changes shortly.

package com.example
import org.springframework.web.bind.annotation.GetMapping
import org.springframework.web.bind.annotation.PathVariable
import org.springframework.web.bind.annotation.RestController
import org.springframework.web.bind.annotation.PostMapping
import org.springframework.web.bind.annotation.RequestBody
import org.springframework.http.MediaType
import com.zaxxer.hikari.HikariConfig
import com.zaxxer.hikari.HikariDataSource
import java.net.URI
import javax.sql.DataSource

@RestController
class GreetingController {

   val dataSource = dataSource()
   val connection = dataSource.connection

   @GetMapping("/{name}")

   fun get(@PathVariable name: String) = "Hello, $name"

   @PostMapping(value = ["/add-name"], consumes = [MediaType.TEXT_PLAIN_VALUE])
   fun post(@RequestBody requestBody: String) : String {
       initDb()
       val stmt = connection.createStatement()
       stmt.executeUpdate("INSERT INTO names values('$requestBody')")
       return "Added $requestBody"
   }

   @GetMapping("/everyone")

   fun getAll() : String {
       initDb()
       val stmt = connection.createStatement()
       val rs = stmt.executeQuery("SELECT name FROM names")
       val output = ArrayList<String>()
       while (rs.next()) {
           output.add(rs.getString("name"))
       }
       val names = output.joinToString(", ")
       return "Here are the names: $names"
   }

   internal fun initDb() {
       val stmt = connection.createStatement()
       stmt.executeUpdate("CREATE TABLE IF NOT EXISTS names (name text)")
   }

   internal fun dataSource(): HikariDataSource {
       val config = HikariConfig()
       var dbUri = URI(System.getenv("DATABASE_URL") ?: "postgresql://localhost:5432/")
       var dbUserInfo =  dbUri.getUserInfo()
       var username: String?; var password: String?;
       if (dbUserInfo != null) {
           username = dbUserInfo.split(":").get(0)
           password = dbUserInfo.split(":").get(1)
       } else {
           username = System.getenv("DATABASE_USERNAME") ?: null
           password = System.getenv("DATABASE_PASSWORD") ?: null
       }
       if (username != null) {
           config.setUsername(username)
       }
       if (password != null) {
           config.setPassword(password)
       }
       val dbUrl = "jdbc:postgresql://${dbUri.getHost()}:${dbUri.getPort()}${dbUri.getPath()}"
       config.setJdbcUrl(dbUrl)
       return HikariDataSource(config)
   }
}

There's quite a lot going on here! Let's start from the bottom. We define a function called dataSource which provides a connection to our database. Because we're building a 12-Factor app, our database credentials are stored in an environment variable called DATABASE_URL. We fetch that URL and pull out the username and password from it if one exists. If not, we check another two environment variables for that information—DATABASE_USERNAME and DATABASE_PASSWORD. We then put all that information together into a format that the database connector needs. The initDb function creates a table called names, with a single text column called name. The /everyone endpoint has a @GetMapping decorator just like before. This defines a GET /everyone route, which gets all the names from the database.

Finally, we've added something rather new: a @PostMapping decorator. Here, we need to define what types of content this POST route can accept. In this case, it consumes a TEXT_PLAIN_VALUE media type (in other words, "Content-Type: text/plain"). Whatever string of information we put in the request body will be added to the database. In just a few lines, we've built a small API that we can add to and query.

If you start this server now—and if you have PostgreSQL running locally—you should be able to interact with it. Try making the following request:

$ curl -H "Content-Type: text/plain" -X POST http://localhost:8080/add-name -d 'Frank'

If you navigate to http://localhost:8080/everyone, you'll see that Frank was included.

Deploying to Heroku

It's time to see just how easy it is to get Kotlin running on Heroku. First, we need to create a file that's specific to Heroku: the Procfile. This text file defines how our application should boot and run.

Create a file named Procfile in the root level directory, right next to your build.gradle.kts file. Copy-paste the following lines into it:

web: java -jar build/libs/kotlin-api.jar --server.port=$PORT

Here, we're saying that we want Heroku to run java -jar build/libs/kotlin-api.jar. That JAR is packaged and built during the deployment process; Heroku will create it automatically for us because it knows how to execute the Gradle task to do so. We are also binding the $PORT environment variable so that Heroku knows which port the server is listening to.

Next, run the following Git commands:

$ git init
$ git add .
$ git commit -m "Preparing my first Kotlin app"

Since we have the Heroku CLI installed, we can call heroku create on the command line to generate an app:

$ heroku create
Creating app... done, ⬢ desolate-plains-67007
Created http://desolate-plains-67007.herokuapp.com/ | git@heroku.com:desolate-plains-67007.git

Your app will be assigned a random name—in this example, it's desolate-plains-67007—as well as a publicly accessible URL.

In order to provision a database, we use the heroku addons command. Calling it without arguments will let us know if any exist:

$ heroku addons
No add-ons for app desolate-plains-67007.

No add-ons exist for our app, which makes sense—we just created it! To add a PostgreSQL database, we can use the addons:create command like this:

$ heroku addons:create heroku-postgresql:hobby-dev

Heroku offers several tiers of PostgreSQL databases. hobby-dev is the free tier, so we can play around with this without paying a dime.

Your code is ready, your Heroku app is configured—you’re ready to deploy. This is the easy part! Just type out the following command:

$ git push heroku master

Your code will be pushed to Heroku. From that point on, Heroku will take over. You'll see your build logs scrolling through your terminal. This will show you what Heroku is installing on your behalf and where you are in the build process. After it’s complete, you can visit your special URL in the browser (in this case, https://desolate-plains-67007.herokuapp.com) and interact with the API on the internet!

Learning more

Kotlin's performant design and legible syntax—combined with the ease of Gradle—make it ideal for enterprises that need to rely on battle-tested Java packages. Because of its interoperability with Java, Kotlin is ideal as a transitional language; vast swaths of Java code can be converted into Kotlin while still maintaining a functional app. Deploying to Heroku is smooth, and I didn't even take advantage of the different ways to fine-tune Java and JVM-based apps for deployment.

Azure/Heroku Service Bus

Michael Bogan — Thu, 04 Feb 2021 15:29:33 +0000

Background Jobs in Heroku with Azure Service Bus

Web applications are optimized for throughput and latency to service a high number of HTTP requests as quickly as possible. For improved performance, web applications defer the CPU intensive, IO intensive, time-intensive, and scheduled processing workloads to background jobs that run independently of the user interface. These background jobs must function without intervention from the user interface and should not block a synchronous user and system interaction. Offloading slow and compute or memory-intensive activity to background jobs improves web applications' performance and throughput.

For example, consider an eCommerce web application that captures a customer’s orders and triggers the background jobs to process the orders further. The application’s background jobs work with the operational data (all orders placed by customers) and the contextual data (orders for a single customer) to update the inventory and shipping systems.

Heroku supports several queue services as add-ons such as RabbitMQ, Kafka, and IronMQ. However, you are not limited to using add-ons for integrating with cloud queue services. In this example, we will build a background job that processes messages from an Azure Service Bus queue. AWS, Azure, and GCP offer message queues as a service that you can use to extend the capabilities of your Heroku applications.

Azure Service Bus offers a rich set of features including support for At-Least-Once and At-Most-Once delivery guarantee. Azure Service Bus also offers First In, First Out (FIFO) messages for both point-to-point (queue) and publish/subscribe communication. While Heroku's application platform is simple, easy to scale, and supports low ceremony DevOps integration, Azure supports an array of enterprise grade services of Azure that can be easily integrated. For complex scenarios, you will find it easy to build applications by integrating the right services across the cloud.

Background Jobs in Heroku

Heroku allows you to compose your application from various process types such as web and worker processes. In this demo, we will deploy a simple background worker process that processes messages from a work queue. Heroku allows you to scale the processes in an application independently, which gives you the ability to scale worker instances in proportion to the workload.

Apart from the worker, a feature-rich queue is the next crucial component of an event-driven worker process. Azure Service Bus queue service allows consumer processes to lock and process messages independently, enabling you to scale the number of worker dynos and achieve high throughput. Let’s discuss the Azure Service Bus queue service in detail next.

Azure Service Bus Queues

The Azure Service Bus service includes a reliable queue service and a durable publish/subscribe messaging service, any of which you can choose based on your needs. Let’s focus on the Azure Service Bus queue service, which offers FIFO message delivery. The message receivers of an Azure Service Bus queue receive the messages in the same sequence in which they were added to the queue by the producer.

Service Bus queues act as a buffer between the producer and the consumer of the messages. During the peak load period, the producer can enqueue several additional messages to the queue, which the message consumers can keep processing at the same scale as during an average load period. You can create an Azure Service Bus queue using the Azure CLI and the Azure Portal, among other options. The Azure Service Bus SDK is available in many popular programming languages such as C#, Java, Node, and Go.

The Demo Application

I will use Go and the Azure Service Bus Go package to build a sample application to demonstrate how we can develop and deploy a background service that reads messages off a work queue, processes them, and prints the results. The following link will take you to the GitHub repository of the application.

Source Code

The application itself is straightforward. It receives messages from the configured Service Bus queue and prints the message body to the console after a small delay. The deliberate processing delay will help me demonstrate that each worker dyno instance receives a different message from the queue and can process the messages independently and thus scale out if required.

Building the Application

Start your favorite Go code editor such as VSCode, create a folder for your project, and create a module named sbworker using the following command:

go mod init tcblabs.net/sbworker

To work with Azure Service Bus, let’s install the Azure Service Bus Go package and the Godotenv package to load environment variables from a .env file. The Godotenv package makes it easier to work with applications on development machines and CI servers where several applications might run with each requiring their own set of environment variables. You can read more about this package in the README of its GitHub repository.

go get github.com/Azure/azure-service-bus-go
go get github.com/joho/godotenv

Create a file named main.go and create the main method in it as follows:

func main() {

    // Read env variables from .env file if it exists

    loadEnvFromFileIfExists()

    handler := &MessageHandler{}

    // Set background context

    ctx, cancel := context.WithCancel(context.Background())

    defer cancel()

    connStr := os.Getenv("SERVICEBUS_CONNECTION_STRING")

    qName := os.Getenv("QUEUE_NAME")

    if connStr == "" || qName == "" {

        fmt.Println("FATAL: expected environment variables SERVICEBUS_CONNECTION_STRING or QUEUE_NAME not set")

        return

    }

    // Create a client to communicate with a Service Bus Namespace.

    ns, err := servicebus.NewNamespace(servicebus.NamespaceWithConnectionString(connStr))

    if err != nil {

        fmt.Println(err)

        return

    }

    // Create queue receiver

    q, err := ns.NewQueue(qName)

    if err != nil {

        fmt.Println(err)

        return

    }

    for {

        if err = q.ReceiveOne(ctx, handler); err != nil {

            if innerErr, ok := err.(*amqp.Error); ok && innerErr.Condition == "com.microsoft:timeout" {

                fmt.Println("➰ Timeout waiting for messages. Entering next loop.")

                continue

            }

            fmt.Println(err)

            return

        }

    }

}

Let’s read the code together. The loadEnvFromFileIfExists function loads the environment variables from the .env file present in the folder. We will create and populate the .env file later. Also, remember that this feature is only for our convenience. We will use actual environment variables for configuring our application in Heroku.

Next, we instantiated the message handler that will receive and process the messages we receive from the Service Bus queue. We will discuss the message handler in detail later. Since we intend to build a background application, we created a background context with support for cancellation.

Next, we fetched the connection string for the Service Bus namespace and the name of the queue from environment variables. We then created a client to communicate with the service bus namespace, and we also created a client to communicate with the queue. A namespace is a container for all messaging components; in this case, the queue.

Finally, we started the message receiver with the ReceiveOne function. We handled the particular case of a timeout error, in which case we recurse the loop and reattach the message receiver to the queue. Note that we passed the handler object to the ReceiveOne function, which implements the Handler interface. This interface only requires defining the Handle function that is invoked whenever the receiver can lock a message for processing on the service bus. Let’s define the struct MessageHandler next.

type MessageHandler struct{}

func (mh *MessageHandler) Handle(ctx context.Context, msg *servicebus.Message) error {

    fmt.Printf("-> Received message: %s\n", string(msg.Data))

    // Processing of message simulated through delay

    time.Sleep(5 * time.Second)

    fmt.Printf("✔ Finished processing the message: %s\n", string(msg.Data))

    return msg.Complete(ctx)

}

The implementation of the function Handle is straightforward. We log the message data, wait five seconds, and mark the message as complete. Note that you must mark a message as complete after processing; otherwise it will reappear on the queue.

Finally, let’s define the loadEnvFromFileIfExists function to help us read and load environment variables from a file.

func loadEnvFromFileIfExists() {

    envFile := ".env"

    if _, err := os.Stat(envFile); err == nil {

        if err = godotenv.Load(envFile); err != nil {

            log.Fatalf("Error loading .env file")

        }

    }

}

Add a file named .env to the folder. We will add the Service Bus connection string and the name of the queue to this file shortly. The last artifact that you need to add to the project is a Procfile. A Heroku Procfile specifies the processes in your application and the commands executed by the applications on startup. Our application is of the worker process type. To start the application, we need to run the command sbworker to launch the module executable generated after Go compiles our application.

Create the Azure Service Bus Queue

Let’s spin up an Azure namespace and a queue. I prefer to use the Azure CLI, but you can also use any supported means, such as the Azure portal. The following commands will create a resource group named azsb-heroku-worker-rg, an Azure Service Bus namespace named worker-ns, and a queue named messages in the namespace.

az group create -l westus -n azsb-heroku-worker-rg

az servicebus namespace create --resource-group azsb-heroku-worker-rg --name worker-ns --location westus --sku Standard

az servicebus queue create --resource-group azsb-heroku-worker-rg --namespace-name worker-ns --name messages

Let’s now navigate to the Azure portal to fetch the connection strings of the namespace. Visit the Azure portal quickstart guide for creating Azure Service Bus namespace and queue if you face difficulty navigating through the portal.

We will create an access policy that grants only the listen permission (receive messages) to the client. Open the Service Bus namespace that you created and click on Shared access policies. In the next blade, click on the Add button, and on the next panel, provide a name for the policy and select Listen from the list of permissions. Finally, click on the Create button to finish creating the policy.

Create listen only policy

After creating the policy, click on it, and copy the connection string value from the next panel.

Copy the connection string

Let’s now apply this value to the .env file that we created earlier as follows:

SERVICEBUS_CONNECTION_STRING=&lt;connection string>

QUEUE_NAME=messages

Add a .gitignore file to the project and add the pattern .env to avoid committing this file to the Git repository.

You can try running the program on your system with the command go run main.go and debug any errors if the application fails to start. Create a GitHub repository and push the code to it. We will connect this repository to Heroku next.

Create Heroku App

Navigate to the Heroku dashboard and create an app using the Common Runtime as follows:

Create app in Heroku

After creating your app, add the config vars to it, which Heroku will surface as environment variables to our application. Click on the gears icon, and click on the Reveal Config Vars button as follows:

Show config vars

Create two config vars SERVICEBUS_CONNECTION_STRING and QUEUE_NAME and set the same value of the variables you set in the .env file earlier.

Set config vars

It is now time to connect our GitHub repository to the application. Navigate to the Deployment tab and click on the Connect to GitHub button. You will be asked to log into GitHub and grant access to Heroku to your repositories, which you must accept. Search for your repository and connect it as shown below.

Connect app to GitHub

In the expanded panel of the deployment view, select the branch you want to deploy to Heroku and click on the Enable Automatic Deploys button. Any subsequent commit to your repository now will trigger a build and deployment on Heroku.

Select the repository branch to deploy

Since we have already committed our code and do not intend to make any changes to our application, click on the Deploy Branch button to immediately kick off a deployment.

Heroku does not automatically create worker dyno instances upon the first deployment. You must use the Heroku CLI or the portal to select the type and the number of dyno instances that you require. Click on the Dynos tab and click on the Edit button, as shown below:

Edit dyno configuration

In the dyno edit view, you can select the compute configuration and the instance count of dynos. Set the count of dyno instances to 2 and click the Confirm button.

Set dyno instance count

It’s now time to run the application by submitting some messages to it from the Azure portal.

Running the Application

Launch the Azure portal in your browser and navigate to the queue that you created in the namespace. Click on the Service Bus Explorer option, which will launch the Service Bus Explorer tool that you can use to send, receive, and peek (see without lock or delete) messages in your queue. Send a few messages to your queue successively after changing the message text. Remember to keep the Content-type to Text/Plain, which is what our receiver expects.

Send messages to queue

Open the logs view of your application in the Heroku portal, as shown below:

View application logs

In the logs, you can see the two instances processing the messages independently. Also, each receiver instance is independently locking a different message to process, and hence the messages are not duplicating between them.

Worker dynos processing the queue messages

Instead of the intentional delay, you can try adding an actual operation to your application and store the result of processing in a persistent data store. You can also try to add a front end to the application that submits messages to the Service Bus, which will convert this simple background job to a complete application.

Conclusion

This article presented you with the procedure to integrate Azure Service Bus queues with Heroku worker process to build an event-driven background job. Background services are a critical component of event driven architecture which enables building microservices that are decoupled and iterate independently. Since messages placed on the Azure Service Bus are immutable, they can be treated as the source of truth of business events that can be audited.

Thanks to Rahul Rai for his kind permission to publish this article.

Using Serenade to Code by Voice

Michael Bogan — Wed, 27 Jan 2021 16:03:23 +0000

Using Serenade to Code By Voice

The emergence of voice-controlled devices like Siri, Amazon Echo, and Google Home have simplified tasks that previously required a keyboard. But we've only just begun to inhabit a world filled with devices that we control in different ways, one where we can accomplish more by what we say than what we can do. Serenade aims to tackle an interesting problem: writing code through speech.

Serenade is a voice-to-code software that's available to plug into several popular IDEs, like VS Code and IntelliJ. Their claim is to allow you to code "using natural speech," and with support for nearly a dozen languages, it's an intriguing proposition. The ability to code through voice commands is more than just a Sci-Fi fantasy. For programmers with carpal tunnel, or other more severe physical ailments, coding can continue to provide them access to the creative endeavor they enjoy—and a living.

Is Serenade actually viable? In this post, we'll put the program to the test. My colleague Garen and I will attempt to build a Node.js site, with a simple HTML and CSS frontend, using only our voice. Our site will be nothing more than a button that, when clicked, will display a random emoji. To truly immerse ourselves in the hands-free experience, we'll also attempt to deploy the app to Heroku using their CLI. Let's get started!

Set up the server

As with most new projects, we'd like to create a new directory, change into it, and install our package dependencies. Since we can only use Serenade within the IDEs they support, we must resort to macOS' Dictation feature, which is lacking and unable to comprehend programmer speak. We must explicitly spell out Unix commands, or the program won't understand them. Furthermore, there doesn't seem to be any way to enforce lowercase, which really irks the directory aesthetic. Perhaps most frustrating of all is the inability to speak filename conventions:

In order to move this along, we just typed npm i express --save-dev on the command line, created a new file called server.js, and opened up Visual Studio Code. Hands 1, Voice 0.

Serenade runs on your computer and detects your IDE, in order to integrate with its functionality. After installing and activating the app, you can pretty much get started by issuing explicit directives. In the event that Serenade didn't understand, you can choose a visual option to resolve the ambiguity. There are several tutorials available to help you begin learning the directives. Here's the first try:

The first attempt at using Serenade

There's a lot to like about how Serenade works: it knows common IDE commands like "delete word". Additionally, in general, if you speak out a sentence, it'll know when to add spaces and semicolons. However, one issue to keep in mind is the fact that Serenade has three different ways to add text:

insert, which is a general way to describe a line
add, which specifically recognizes language features like classes and functions
type, which enters raw, plain text

Knowing which one to use can be a bit confusing, but after spending some time with the program (and thoroughly reading the documentation), it is possible to move at a somewhat faster clip. However, phrases are still misunderstood from time to time:

Moving a little faster as we learn the syntax

There seems to be some confusion about where to place parentheses and arguments. On occasion, it can feel like we are fighting against the app's syntax, rather than it understanding me. Hands 2, Voice 0.

Write the HTML and CSS

Let's assume we did manage to get a basic server going:

const express = require('express');

const app = express();

const path = require('path');

app.get('/', function(req, res) {

   res.sendFile('/index.html');

});

app.listen(8080);

We'll now write a file called index.html to represent the entirety of our design. HTML is meant to be a structured language; how does it fare?

HTML for the win

Success! Writing HTML is a breeze. We was worried there would be some mistakes with tag placement, especially after adding the attributes, but the alignment and nesting are well understood. Hands 2, Voice 1.

Let's move on to the CSS:

CSS also works well

This is an example of when a really close understanding of the underlying language's grammar is really important; we never knew the bracketed CSS definitions were specifically called "rulesets." Still, at this point, we've started to understand how Serenade works, including its nifty capability to move directly to a line. Hands 2, Voice 2.

Write the client-side JavaScript

Our final HTML and CSS will end up looking like this:

<!DOCTYPE html>
<html>
    <head>
    <link rel=stylesheet type=text/css href=/main.css></link>
    <script src="main.js"></script>
    </head>
    <body>
    <h1>hello!</h1>
    <p id="container"></p>
    <button id="emoji">Emoji!</button>
    </body>
</html>

h1 {
  color: blue;
  margin-top: 1em;
}
#container {
  text-align: center;
}

To avoid executing our JavaScript immediately, we need to wait for the DOMContentLoaded event to fire before setting up any event listeners. Capitalization matters here, and we couldn't get Serenade to recognize the right spelling:

Capitalizations are a challenge

Still, either through our increased proficiency with Serenade, or the relatively straightforward nature of what to add, it was easy to add the event handling code. As programmers are not infallible, we intentionally tried to make mistakes to see if Serenade could recover. For example, we missed some indentation:

We'll call this one a draw.

Deploy to Heroku

For the final piece of our test, we need to deploy to Heroku. Heroku requires a Procfile, which is rather short for this project.

web: node index.js

Unfortunately, once again, the obstacle here is switching to the Dictation app, which simply cannot understand the commands required to push this application up to Heroku:

With a final clinch at the end: Hands 3, Voice 2.

Conclusion

Serenade is pretty fun to use. Navigating the IDE is solid, and explicitly speaking out the proper names for programming concepts makes one feel like a wise sage. However, it requires a specific understanding of the various commands, which doesn't feel like speaking English; it feels like we're speaking specific instructions to a computer, rather than a person we're pair programming with. Still, we'd recommend it as a system to augment your current two-handed coding style, rather than a full-on replacement. With practice and patience, it could be a useful tool for developers who otherwise can't code. And as a nice touch, the company provides free one-on-one training to anyone who wants to get started quickly.

Another issue that we did not note here is that, like any good programmer, we frequently relied on the internet to look up specific APIs. This means that we spent the majority of our time navigating away from the IDE (and Serenade), which tested Dictation's coping capabilities. We're not quite at the point of fully composing applications with voice (and there's a long way to go before we can deploy them), but we're getting closer.

Deploying Your First Golang Webapp

Michael Bogan — Tue, 19 Jan 2021 14:18:05 +0000

The Go programming language, often referred to as "golang", has gained a lot of well-deserved traction in the DevOps community. Many of the most popular tools such as Docker, Kubernetes, and Terraform are written in Go, but it can also be a great choice for building web applications and APIs.

Go provides you with all the speed and performance of a compiled language, but feels like coding in an interpreted language. This comes down to the great tooling that you get out of the box with a compiler, runner, test suite and code formatter provided by default. Add to that the robust and easily comprehensible approach to concurrency, to take maximum advantage of today's multi-core or multi-cpu execution environments, and it's clear why the Go community has grown so rapidly.

Go feels like it was created with specific goals in mind, leading to a deceptively simple language design that's straightforward to learn, without compromising on capabilities.

In this post, I'm going to show you how easy it is to develop a simple web application in Go, package it as a lightweight Docker image, and deploy it to Heroku. I'll also show a fairly new feature of Go: built-in package management.

Go Modules

I'm going to use Go's built-in module support for this article.

Go 1.0 was released in March 2012. Up until version 1.11 (released in August 2018), developing Go applications involved managing a GOPATH for each "workspace", analogous to java's JAVA_HOME, and all of your Go source code and any third-party libraries were stored below the GOPATH.

I always found this a bit off-putting, compared to developing code in languages like Ruby or Javascript where I could have a simpler directory structure isolating each project. In both of those languages, a single file (Gemfile for Ruby, package.json for Javascript) lists all the external libraries, and the package manager keeps track of managing and installing dependencies for me.

I'm not saying you can't manage the GOPATH environment variable to isolate projects from one another. I particularly find the package manager approach is easier.

Thankfully, Go now has excellent package management built in, so this is no longer a problem. However, you might find GOPATH mentioned in many older blog posts and articles, which can be a little confusing.

Hello, World!

Let's get started on our web application. As usual, this is going to be a very simple "Hello, World!" app, because I want to focus on the development and deployment process, and keep this article to a reasonable length.

Pre-requisites

You'll need:

A recent version of golang (I'm using 1.14.9.)
Docker
A Heroku account (The free account works fine for this example.)
The Heroku command-line client
git

Go mod init

To create our new project, we need to create a directory for it, and use the go mod init command to initialize it as a Go module.



mkdir helloworld
cd helloworld
go mod init digitalronin/helloworld

It's common practice to use your github username to keep your project names globally unique, and avoid name conflicts with any of your project dependencies, but you can use any name you like.

You'll see a go.mod file in the directory now. This is where Go will track any project dependencies. If you look at the contents of the file, they should look something like this:



module digitalronin/helloworld

go 1.14

Let's start committing our changes:



git init git add * git commit -m "Initial commit"

gin

We're going to use gin for our web application. Gin is a lightweight web framework, similar to Sinatra for Ruby, express.js for Javascript, or Flask for Python.

Create a file called hello.go containing this code:



package main

import "github.com/gin-gonic/gin"

func main() {
    r := gin.Default()

    r.GET("/hello", func(c *gin.Context) {
        c.String(200, "Hello, World!")
    })

    r.Run(":3000")
}

Let's break this down a little:



r := gin.Default()

This creates a router object, r, using the built-in defaults that come with gin.

Then, we assign a handler function to be called for any HTTP GET requests to the path /hello, and to return the string "Hello, World!" and a 200 (HTTP OK) status code:



r.GET("/hello", func(c *gin.Context) {
        c.String(200, "Hello, World!")
    })

Finally, we start our webserver and tell it to listen on port 3000:



r.Run(":3000")

To run this code, execute:



go run hello.go

You should see output like this:



go: finding module for package github.com/gin-gonic/gin
go: found github.com/gin-gonic/gin in github.com/gin-gonic/gin v1.6.3
[GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middleware already attached.

[GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
 - using env:   export GIN_MODE=release
 - using code:  gin.SetMode(gin.ReleaseMode)

[GIN-debug] GET    /hello                    --> main.main.func1 (3 handlers)
[GIN-debug] Listening and serving HTTP on :3000

Now if you visit http://localhost:3000/hello in your web browser, you should see the message "Hello, World!"

Notice that we didn't have to install gin separately, or even edit our go.mod file to declare it as a dependency. Go figures that out and makes the necessary changes for us, which is what's happening when we see these lines in the output:



go: finding module for package github.com/gin-gonic/gin
go: found github.com/gin-gonic/gin in github.com/gin-gonic/gin v1.6.3

If you look at the go.mod file, you'll see it now contains this:



module digitalronin/helloworld

go 1.14

require github.com/gin-gonic/gin v1.6.3 // indirect

You will also see a go.sum file now. This is a text file containing references to the specific versions of all the package dependencies, and their dependencies, along with a cryptographic hash of the contents of that version of the relevant module.

The go.sum file serves a similar function to package-lock.json for a Javascript project, or Gemfile.lock in a Ruby project, and you should always check it into version control along with your source code.

Let's do that now:



git add *
git commit -m "Add 'Hello world' web server"

Serving HTML and JSON

I'm not going very far into what you can build with gin, but I do want to demonstrate a little more of its functionality. In particular, sending JSON responses and serving static files.

Let's look at JSON responses first. Add the following code to your hello.go file, right after the r.GET block:



api := r.Group("/api")

api.GET("/ping", func(c *gin.Context) {
  c.JSON(200, gin.H{
    "message": "pong",
  })
})

Here we're creating a "group" of routes behind the path /api with a single path /ping which will return a JSON response.

With this code in place, run the server with go run and then hit the new API endpoint:



curl http://localhost:3000/api/ping

You should get the response:



{"message":"pong"}

Finally, let's make our webserver serve static files. Gin has an additional library for this.

Change the import block at the top of the hello.go file to this:



import (
    "github.com/gin-gonic/contrib/static"
    "github.com/gin-gonic/gin"
)

The most popular code editors have golang support packages you can install which will take care of the import declarations for you automatically, updating them for you whenever you use a new module in your code.

Then, add this line inside the main function:



r.Use(static.Serve("/", static.LocalFile("./views", true)))

The full code for our web application now looks like this:

hello.go



package main

import (
    "github.com/gin-gonic/contrib/static"
    "github.com/gin-gonic/gin"
)

func main() {
    r := gin.Default()

    r.GET("/hello", func(c *gin.Context) {
        c.String(200, "Hello, World!")
    })

    api := r.Group("/api")

    api.GET("/ping", func(c *gin.Context) {
        c.JSON(200, gin.H{
            "message": "pong",
        })
    })

    r.Use(static.Serve("/", static.LocalFile("./views", true)))

    r.Run()
}

The r.Use(static.Serve... line enables our webserver to serve any static files from the views directory, so let's add a few:



mkdir -p views/css

views/css/stylesheet.css



body {
  font-family: Arial;
}

h1 {
  color: red;
}

views/index.html



<html>
  <head>
    <link rel="stylesheet" href="/css/stylesheet.css" />
  </head>
  <body>
    <h1>Hello, World!</h1>
  </body>
</html>

Now restart the webserver using go run hello.go and visit http://localhost:3000 and you should see the styled message:

Dockerize

We've written our go web application, now let's package it up as a Docker image. We could create it as a Heroku buildpack, but one of the nice features of Go is that you can distribute your software as a single binary file. This is an area where Go really shines, and using a Docker-based Heroku deployment lets us take advantage of that. Also, this technique isn't limited to Go applications: You can use Docker-based deployment to Heroku for projects in any language. So, it's a good technique to understand.

So far, we've been running our code with the go run command. To compile it into a single, executable binary, we simply run:



go build

This will compile all our Go source code and create a single file. By default, the output file will be named according to the module name, so in our case it will be called helloworld.

We can run this:



./helloworld

And we can hit the same HTTP endpoints as before, either with curl or our web browser.

The static files are not compiled into the binary, so if you put the helloworld file in a different directory, it will not find the views directory to serve our HTML and CSS content.>

That's all we need to do to create a binary for whatever platform we're developing on (in my case, my Mac laptop). However, to run inside a Docker container (for eventual deployment to Heroku) we need to compile a binary for whatever architecture our Docker container will run on.

I'm going to use alpine linux, so let's build our binary on that OS. Create a Dockerfile with the following content:



FROM golang:1.14.9-alpine
RUN mkdir /build
ADD go.mod go.sum hello.go /build/
WORKDIR /build
RUN go build

In this image, we start with the golang base image, add our source code, and run go build to create our helloworld binary.

We can build our Docker image like this:



docker build -t helloworld .

Don't forget the trailing . at the end of that command. It tells Docker we want to use the current directory as the build context.

This creates a Docker image with our helloworld binary in it, but it also contains all the Go tools needed to compile our code, and we don't want any of that in our final image for deployment, because it makes the image unnecessarily large. It can also be a security risk to install unnecessary executables on your Docker images.

We can see the size of our Docker image like this:



$ docker images helloworld
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
helloworld          latest              9657ec1ca905        4 minutes ago       370MB

For comparison, the alpine image (a lightweight linux distribution, often used as a base for docker images) is much smaller:



$ docker images alpine
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
alpine              latest              caf27325b298        20 months ago       5.53MB

On my Mac, the helloworld binary is around 14MB, so the golang image is much bigger than it needs to be.

What we want to do is use this Dockerfile to build our helloworld binary to run on alpine linux, then copy the compiled binary into an alpine base image, without all the extra golang tools.

We can do this using a "multistage" Docker build. Change the Dockerfile to look like this:



FROM golang:1.14.9-alpine AS builder
RUN mkdir /build
ADD go.mod go.sum hello.go /build/
WORKDIR /build
RUN go build

FROM alpine
RUN adduser -S -D -H -h /app appuser
USER appuser
COPY --from=builder /build/helloworld /app/
COPY views/ /app/views
WORKDIR /app
CMD ["./helloworld"]

On the first line, we label our initial Docker image AS builder.

Later, we switch to a different base image FROM alpine and then copy the helloworld binary from our builder image like this:



COPY --from=builder /build/helloworld /app/

Build the new Docker image:



docker build -t helloworld .

Now, it's the size you would expect for a base alpine image plus our helloworld binary:



$ docker images helloworld
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
helloworld          latest              1d6d9cb64c7e        8 seconds ago       20.7MB

We can run our webserver from the Docker image like this. (If you have another version running using go run hello.go or ./helloworld, you'll need to stop that one first, to free up port 3000.)



docker run --rm -p 3000:3000 helloworld

The dockerized webserver should behave just like the go run hello.go and ./helloworld versions except that it has its own copies of the static files. So, if you change any of the files in views/ you won't see the changes until you rebuild the Docker image and restart the container.

Deploy to Heroku

Now that we have our dockerized web application, let's deploy it to Heroku. Heroku is a PaaS provider that makes it simple to deploy and host an application. You can set up and deploy your application through the Heroku UI, or through the Heroku CLI. For this example, we'll use the Heroku command-line application.

Getting PORT from an environment variable

We've hard-coded our webserver to run on port 3000, but that won't work on Heroku. Instead, we need to alter it to run on whichever port number is specified in the PORT environment variable, which Heroku will supply automatically.

To do this, alter the r.Run line near the bottom of our hello.go file, and remove the ":3000" string value so the line becomes:



r.Run()

The default behavior of gin is to run on whatever port is in the PORT environment variable (or port 8080 if nothing is specified). This is exactly the behavior Heroku needs.

Setting up our Heroku app

First, log into Heroku:



heroku login

Now, create an app:



heroku create

Tell Heroku we want to build this project using a Dockerfile, rather than a buildpack:



heroku stack:set container

To do this, we also need to create a heroku.yml file like this:



build:
  docker:
    web: Dockerfile

run:
  web: ./helloworld

The heroku.yml file is a manifest that defines our app and allows us to specify add-ons and config vars to use during app provisioning.

Next, add git and commit these files, then push to heroku to deploy:



git push heroku main

My git configuration uses main as the default branch. If your default branch is called master, then run git push heroku master instead.

You should see Heroku building the image from your Dockerfile, and pushing it to the Heroku Docker registry. Once the command completes, you can view the deployed application in your browser by running:



heroku open

Conclusion

To recap, here's a summary of what we covered today:

Creating a golang web application, using go modules and the gin web framework to serve strings, JSON, and static files
Using a multistage Dockerfile to create a lightweight Docker image
Deploying a Docker-based application to Heroku using heroku stack:set container and a heroku.yml file

I've only scratched the surface in this article on how to use Go to build web applications and APIs. I hope this gives you enough to get started on your own golang web applications.

Monitoring Postgres on Heroku

Michael Bogan — Fri, 15 Jan 2021 13:04:14 +0000

Introduction

In the vast majority of applications, the database is the source of truth. The database stores critical business records along with irreplaceable user data. So it is imperative that developers have visibility into their databases to diagnose and remedy any potential issues before they impact the business. If they don't, developers will find unexpected bills at the end of the month that they may not understand.

In this article, you will learn how to set up a Heroku Postgres database with Librato for automated monitoring. Then we'll look at a range of different metrics that are available along with best practices for how to get the most out of each metric. With the right metrics in place you can anticipate where you will need to provision extra resources and where you should keep an eye on potential issues.

Creating Our Example Application with Heroku, Librato, and Postgres

This article assumes you already have a Heroku account. If you don't, creating a free account will be sufficient to follow along, although the metrics will not actually be available unless you have a standard account plan.

Once you have an account, log into our Heroku account and clone the NodeJS Getting Started template. You can use the web interface to do all this, but we will use the CLI in this article. We will then cd into the repository and create a new Heroku application called monitoring-heroku-postgres.



heroku login

git clone https://github.com/heroku/node-js-getting-started.git

cd node-js-getting-started

heroku create -a monitoring-heroku-postgres

Set the get remote for the starter project. Then, push the project to the Heroku application that we created.



heroku git:remote -a monitoring-heroku-postgres

git push heroku main

After deploying the application, we will start a single instance of the application.



heroku ps:scale web=1

heroku open

Now that our application is set up and deployed, we will provision our database and install the pg into our project.



heroku addons:create heroku-postgresql:standard-0

npm install pg

Open index.js and include the following code to connect our application to the database:



const { Pool } = require('pg');

const pool = new Pool({

  connectionString: process.env.DATABASE_URL,

  ssl: {

    rejectUnauthorized: false

  }

});

We will create another route called /db.



.get('/db', async (req, res) => {

    try {

      const client = await pool.connect();

      const result = await client.query('SELECT * FROM test_table');

      const results = { 'results': (result) ? result.rows : null};

      res.render('pages/db', results );

      client.release();

    } catch (err) {

      console.error(err);

      res.send("Error " + err);

    }

  })

Our application is now connected to our database. Commit the changes and push them to our deployed application.



git add .

git commit -m "Install PG dependency, connect database, and create db route"

git push heroku main

Right now our database is empty. To write data to the database, we will use psql.



heroku pg:psql

create table test_table (id integer, name text);

insert into test_table values (1, 'hello database');

\q

heroku open db

Finally, we will add Librato to our project for automated monitoring. Librato is an add-on for collecting, understanding, and acting on real-time metrics. It automatically creates interactive visualizations so you can quickly understand what is happening with your database. You can access the Librato dashboard under the Resources tab.



heroku addons:create librato

The Metrics

Now let's look at our new metrics and see what insights we've gained, and how we can use this new data. We'll look at several database metrics and server metrics.

Database Metrics

db_size

db_size tells you the number of bytes contained in the database. It includes all table and index data on disk, including database bloat.

Your database will have a certain size allotted based on your plan. If your database grows past that allotted size then you will receive a warning email with directions on how to fix the issue. You may receive an enforcement date for when you will be allowed only a single database connection. Access will be restricted to READ, DELETE, and TRUNCATE until the database is back under the plan limit.

Heroku recommends setting a warning alert when your database reaches 80% of the allotted size for your plan and a critical alert when it reaches 90% of the allotted size. As you approach maximum size you can either upgrade your plan or delete data to stay within plan limits.

active-connections

active-connections tells you the number of connections established on the database. Much like db_size, Heroku Postgres enforces a connection limit. There is a hard limit of 500 connections for plans tier-3 and higher. After reaching this limit, you will not be able to create any new connections. You can set up alerts for active-connections in two different ways:

1. Sudden, large changes to the current connection count

If your baseline connection number experiences big changes it can be a sign of increased query and/or transaction run times. The alerting thresholds will depend on your application’s connection count range, assessed under normal operating conditions. +50/+100 over your normal daily maximum is a good rule of thumb.

2. Connection count approaches its hard maximum

For tier-3 plans and higher, the maximum is 500 connections. As with db_size it is recommended to set alerts at 80% and 90% usage, so 400 and 450 are good numbers to start with. If you find that you are frequently approaching your connection limit, then consider using connection pooling.

index-cache-hit-rate

index-cache-hit-rate tells you the ratio of index lookups served from the shared buffer cache, rounded to five decimal points. Heroku recommends a value of 0.99 or greater. If your index hit rate is usually less than 0.99, then it is worth investigating which queries are the most expensive. You can also upgrade your database plan for more RAM.

waiting-connections

waiting-connections tells you the number of connections waiting on a lock to be acquired. Many waiting connections can be a sign of mishandled database concurrency.

We recommend setting up an alert for any connections waiting five consecutive minutes. The pg-extras CLI plugin can help identify queries that are preventing other operations from taking place. Once those queries have been identified they can be terminated in order to resolve lock contention. Knowing which statements are causing blocks can help identify application code to optimize for reducing locks.

Server Metrics

load-avg

load-avg tells you the average system load over a period of 1, 5, and 15 minutes, divided by the number of available CPUs. A load-avg of 1.0 means that, on average, processes requested CPU resources for 100% of the timespan.

A load over 1.0 indicates that processes had to wait for CPU time in the given window. Higher values indicate more time spent by processes waiting. Values under 1.0 indicate that CPUs spent time idle during the given window. If this value is high, then this means that you will get less consistent query execution times and longer wait times.

Once the value goes over 1.0, you are over-utilizing resources. Therefore, you will want to know before the load reaches values over 1.0. Again, we recommend setting alerts at 80% (8.0) and 90% (9.0).

You can check current activity with the pg:ps command for cpu-intensive queries. If your load-avg values are consistently high then it may be worth upgrading to a larger plan. However, before upgrading it is useful to tune expensive queries to reduce the amount of processing work done on the database.

read-iops

read-iops tells you how many read IO requests are made to the main database disk partition in values of IO Operations Per Second (IOPS). Each plan has a provided Provisioned IOPS (PIOPS) max. This is the maximum total reads + writes per second that the provisioned disk volume can sustain.

You want your reads to come from memory (cache) rather than disk to increase the speed of reads. Exceeding provisioned IOPS can lead to long transaction times and high load-avg since processes need to wait on I/O to become available.

You can set up an alert for 90% of your Provisioned IOPS to identify activities or statements that require significant I/O. Before upgrading to a larger plan you can tune expensive queries to reduce the amount of data being read directly from disk.

wal-percentage-used

wal-percentage-used tells you the space left to store temporary Postgres write-ahead logs. You are at risk of completely filling up the WAL volume if the rate of WAL generation exceeds the rate of WAL archival. This will shut down the database and potentially lead to a risk of data loss.

If you reach 75% utilization, then your database connection limit will be automatically throttled by Heroku. All connections will be terminated at 95% utilization. Unlike previous metrics, we recommend setting up an alert for when this number reaches 60%.

Conclusion

We have reviewed a broad collection of different metrics that provide a range of different insights. The health of a database depends on numerous factors, which include the following:

size
number of connections
cache utilization
server load
read frequency
WAL percentage

Once a developer has visibility into these metrics, they will be better equipped to make informed decisions about provisioning and managing their data. For additional information on monitoring, I recommend you check out this article from Heroku.

What is DevSecOps (using Heroku Flow as an example)

Michael Bogan — Thu, 07 Jan 2021 13:40:40 +0000

An Overview of DevSecOps and How to Automate It

With the proliferation of agile product development models, industry experts from all levels have come to appreciate the value of incremental releases. However, there is also an expectation that each release cycle will maintain and improve the reliability and security of the product being delivered.

As a developer or engineer, your challenge is to implement security best practices without slowing down development or delaying your release dates. This article will illustrate several ways to include security practices in your development lifecycle to prevent critical issues later, and without slowing you down.

I’ll use Heroku Flow as an example flow to show how these security practices (or DevSecOps) can be integrated into your CI/CD practice, though the practices can be used in almost any common scenario.

What is DevSecOps?

DevSecOps is the philosophy of integrating security best practices early in the product development process. With DevSecOps, security is not treated as an isolated process or separate feature, but rather as an integral part of your development lifecycle. Automation helps you identify and fix security problems early, ideally before merging the application code to the main branch of the code repository.

Some examples of DevSecOps practices include scanning repositories for security vulnerabilities, early threat modeling, security design reviews, static code analysis, and code reviews.

Enter Heroku Flow

Heroku Flow provides a comprehensive CI/CD solution for Heroku-based applications. It seamlessly ties together several services (Heroku Pipelines, Review Apps, Heroku CI, and GitHub integrations) in a single view, giving engineers greater visibility to each code release— from a pull request to the production drop.

For a visual check out this Heroku Flow workflow from initial commit to production.

As the image visual shows, automated tests will run in Heroku CI when pull requests are created. Heroku CI is a cloud continuous integration tool from Heroku; it can either automatically detect the language and run default commands (e.g. npm test) or can be configured via the app.json file. CI results are available in both the pull request details in GitHub and the Heroku interface.

For a successful CI build, create a new review of the application and deploy it to a new temporary Heroku environment using Review Apps. The new environment link is available in GitHub pull request view, allowing engineers to easily check CI results or run any manual tests immediately.

After merging the pull request, the new review of the application is available in pre-production environments using Heroku Pipelines. Then, the review can be promoted to production.

Note that while some pieces of Heroku flow are included with a free account (namely pipelines and review apps) some features do incur a fee (Heroku CI).

How to Automate DevSecOps with Heroku Flow

As a comprehensive CI/CD solution integrated with GitHub, Heroku Flow offers several ways to automate your DevSecOps practices. Let's explore three common examples below:

Upgrading dependencies with security vulnerabilities safely
Identifying security bugs early
Preventing unauthorized components or libraries

Safely Upgrade Dependencies with Security Vulnerabilities

You probably already know that you should upgrade dependencies with known vulnerabilities. It can be pretty time-consuming to identify and update those dependencies. Thankfully, you can automate the majority of this work.

GitHub provides a dependency vulnerability scanner, also known as Dependabot, which can be enabled per repository in GitHub’s Security settings. By default, it will add warnings to the GitHub interface when identifying a dependency with a known vulnerability.

While it’s a useful feature, it still requires you to check the warnings and manually create pull requests to upgrade the affected dependencies and create a fixed version. Fortunately, there's a beta feature in Dependabot that automatically creates pull requests to fix known vulnerabilities.

To enable this feature, simply add a .github/dependabot.yml file to your repository:



# Basic dependabot.yml file for JavaScript application 
# check docs for other dependencies
version: 2

updates:
  - package-ecosystem: "npm"
    directory: "/"
    schedule:
      interval: "daily"

Dependabot will create pull requests with the suggested fixes, adding the GitHub code owners as default reviewers. The Dependabot documentation covers all available options.

Pull request created by Dependabot to address a known vulnerability

While the pull request will upgrade the affected library version, it's still important to verify that the application will work as expected after the upgrade.

The pull request raised by Dependabot will run CI tests and will be deployed to a new Heroku environment. Both versions are accessible from the GitHub interface.

Checks from Pull Request view in GitHub

After merging the pull request, the pipeline will run CI tests and deploy it to pre-production environments. Then, it can be promoted to production.

Heroku Pipeline view

Setting up Dependabot and Heroku Flow will automate most of the manual work required to address security vulnerabilities in libraries and dependencies.

Identify Security Bugs Early

Naturally, the ideal time to catch security bugs is before deploying to production. Many different tools can run static code analysis and identify problematic code before merging the code to the main branch.

As an example, let’s consider a simple Node.js application. Developers commonly use ESlint to enforce consistent coding styles and to catch common problems. Enabling ESlint-plugin-security will also identify common security bugs:

.eslintrc



...
"plugins": [
  "security"
],
"extends": [
  "plugin:security/recommended"
]
...

To ensure Eslint executes during CI, the app.json file is editable and points to a file in the repository:

app.json



{
  "environments": {
      "test": {
        "scripts": {
           "test": "bash ./ci.sh"
      }
    }
  }
}

In this custom script file, you can run any desired command:

ci.sh



#!/bin/bash
set -eux
#### Running unit tests
npm test
#### Searching for security problems 
npm run lint

If the lint fails, the build will be marked unsuccessful, and the deployment will not continue.

While ESlint-plugin-security is specific for JavaScript, most mature languages have static code analysis tools, like the popular Brakeman for Ruby, or Find-Sec-Bugs for Java.

Although the CI snippets in this article were shown in bash scripts, the Heroku CI supports multiple languages.

Prevent Unauthorized Components or Libraries

Some organizations heavily emphasize controlling application deployment complexity and implementing centralized controls for all applications. For example, these controls may prevent the use of Redis add-on or a specific JavaScript library.

All Heroku components for an application are defined in the app.json file as code. This opens up the possibility for pre-deployment checks. Infrastructure engineers can create a centralized script to prevent specific components from being deployed, and ensure all applications pass the same checks.

For example, let's consider the centralized script infrastructure-checks.sh shown below. It’s currently available in a public git repository "mygithubaccount/infrastructure-scripts". For this tutorial, let's say your goal is to prevent all Heroku add-ons from deploying.

infrastructure-scripts.sh



#!/bin/bash
set -eux
ADDONS=$(cat app.json | jq '.addons')
# Prevents all addons
if [[ "$ADDONS" != "null" ]]; then
  echo "Add-ons are not allowed"
  exit 1
fi

Inside the infrastructure script, you can add any desired number of checks to exclude specific addons, check environment variables, prevent certain instance types from being created, and even check for specific libraries that shouldn't be used. In short, you can do anything necessary to maintain the consistency of all environments.

For each Heroku application, the CI can be configured to download and execute infrastructure-scripts.sh from the centralized repository_: _

app.json



{
  "environments": {
    "test": {
      "scripts": {
        "test": "bash ./ci.sh"
      }
    }
  }
}

ci.sh



#!/bin/bash
set -eux
INFRA_SCRIPTS_REPOSITORY="mygithubaccount/infrastructure-scripts"
wget https://raw.githubusercontent.com/${INFRA_SCRIPTS_REPOSITORY}/master/infrastructure-checks.sh
bash ./infrastructure-checks.sh
# Add as well all commands to run all other tests 
...

The central infrastructure repository can also be private, but authentication will be required when downloading the script file.

Conclusion

Hopefully, you’ve now seen some practical examples of implementing security controls as part of your CI/CD pipeline using Heroku Flow. You should be able to implement similar controls in other CI/CD solutions as well, but those may not have tight integration with GitHub and Heroku.

More and more organizations are realizing, security shouldn’t be an afterthought, but rather part of a continuous improvement process. Implementing security controls and fixes as code in a minimally intrusive way will help you deliver code reliably and safely without slowing down delivery speed. It also ensures customers or end-users are largely protected from potential security breaches.

As part of DevSecOps, automation also means catching security vulnerabilities isn’t a reactive process where scanners and audit processes find security loopholes in live systems, but rather a proactive approach.

DEV Community: Heroku

Yes! I Can Finally Run My .NET Application on Heroku!

Why This Matters for .NET Developers

Key Benefits of .NET on Heroku

Getting Started

Who Should Care?

Wrap-up

5 Simple Steps to Get Your Test Suite Running in Heroku CI

Our python app: is it prime?

Step #1: Write your tests

Step #2: Deploy your Heroku app

Step #3: Push your code to Heroku

Step #4: Create a Heroku Pipeline to use Heroku CI

Step #5: Run your tests with Heroku CI

Heroku CI for the win

How To Build a Simple GitHub Action To Deploy a Django Application to the Cloud

A Quick Introduction to Django

Building Our Simple Django Application

Install Python dependencies

Set up Django to use PostgreSQL

Test application locally

The Power of GitHub Actions

Plug-and-play GitHub Action workflows

Deploying to Heroku via a GitHub Action

Login, create app, and PostgreSQL add-on

Add Heroku app host to allowed hosts list in Django

Procfile and requirements.txt

Get your Heroku API key

Create the job configuration file

Conclusion

Working with Heroku Logplex for Comprehensive Application Logging

Introducing Heroku Logplex

Core components

Routing and processing

Integrating Logplex with Your Application

Demo application

Create the app

A note on requirements.txt and Procfile

Deploy the code

Verify the app is running

View and filter logs

Log Drains

Custom drains

Logging best practices

Conclusion

Caching RESTful API requests with Heroku’s Redis Add-on

Node.js + Fastify + long-running tasks

Initialize the basic application

Add a simulated long-running process

Measure latency with curl

Redis to the rescue!

Register the Redis plugin with Fastify

Modify our endpoint to use caching

Using Redis in your Heroku deployment

Step 1: Login to Heroku

Step 2: Create the Heroku app

Step 3: Add the Redis add-on

Step 4: Create a Heroku remote

Step 5: Deploy!

Test our deployed application

Bonus Tip: Clearing the cache for future tests

Conclusion

Managing Django Media & Static Files on Heroku with Bucketeer

What are Media & Static Files

Django with Media & Static Files

Heroku with Media & Static Files

Complete Example Codebase

dstarner / django-heroku-static-file-example

Used in my blog post of detailing private & public static files for a Heroku-served Django application

Properly Managing Django Media & Static Files on Heroku Example

Bootstrapping Django on Heroku

Introducing Heroku's Bucketeer Add-On

Including the Bucketeer Add-On

Implementing Our Storage Solution

Installing django-storages

Implementing Static, Public & Private Storage Backends

Configure Settings to Use the Storage Backends

Local Development Serving

Enabling S3 Storage

Using the Private Storage

Installing `django-storages`