DEV Community: Matthew Hegarty

Troubleshooting AWS Batch and PrivateLink

Matthew Hegarty — Wed, 22 Apr 2020 13:47:06 +0000

AWS Batch is a powerful platform for running batch jobs, but setting up can be difficult, especially if your environment uses private subnets and you want to make use of AWS PrivateLink.

One source of frustration is that your jobs get stuck in a 'RUNNABLE' state and don't progress. Here are some pointers you can use for troubleshooting this issue.

Read through the AWS Batch Troubleshooting guide and 'Why is my AWS Batch job stuck in RUNNABLE status'. If you carefully work through these guides then they should lead you to the source of the issue.

One thing to highlight is:

Container instances need access to communicate with the Amazon ECS service endpoint. This can be through an interface VPC endpoint or through your container instances having public IP addresses.

If you are using a NATGateway, then this shouldn't be a problem, because this is a standard solution to communicating with web resources from private subnets.

However, I was configuring service access via PrivateLink, meaning that requests are not routed over the public internet. Using PrivateLink brings with it additional steps for successfully running AWS Batch jobs.

ECS Instances not being created in the ECS cluster

Your batch job goes to RUNNABLE, but you don't see any ECS cluster instances created (refer to 'troubleshooting' for more on this).

Refer to Creating the VPC Endpoints for Amazon ECS. Your containers are going to need to be able to access the following endpoints:
- com.amazonaws.<your region>.ecs-agent
- com.amazonaws.<your region>.ecs-telemetry
- com.amazonaws.<your region>.ecs
The Security Group associated with each endpoint need to be able to send 'all traffic' outbound requests - refer to the troubleshooting guide.
The Security Group associated with the endpoint(s) needs to be able to accept TCP requests on port 443.

Jobs run but then fail with 'CannotPullECRContainerError'

Refer to this guide for setting up VPC Endpoints to ECR.

The key points is that you will need two VPC Endpoints:

com.amazonaws.<your region>.ecr.dkr
com.amazonaws.<your region>.ecr.api

I found setup of AWS Batch with VPC Endpoints was not straightforward, but hopefully some of the pointers in this guide can save you time if you are stuck.

Other Resources

Postgresql: Better security for Django Applications

Matthew Hegarty — Tue, 31 Mar 2020 19:54:22 +0000

Whilst Django is intended to be database-agnostic, it supports many features of Postgresql which are not shared by other databases, and this makes Postgres a popular choice of database for Django applications.

A core principle of web application security is that the application should use the 'least privilege' model, meaning that the database permissions for the application are locked-down so that the application has only the permissions it needs to do its job, and no more.

For example, imagine that an attacker was able to exploit a web application bug and run their own SQL commands. This would be bad, but the severity could be minimised if the attacker was restricted to a certain set of operations.

In this blog post, I'm going to demonstrate how we can use Postgres schema and roles to ensure that Django application privileges are limited appropriately.

Sample Application

To demonstrate the concepts, I've created a clone of the Django Rest Framework Tutorial which runs on Docker.

Feel free to clone the repo and run with docker-compose up (Ctrl+C to stop).

Once the app is running:

Browse to http://localhost:8000/
Use admin / password123 to login.

To start from scratch, run docker-compose down -v, followed by docker-compose up.

NB The sample application includes hard-coded passwords (in docker-compose.yml) - this is just to make life easier for demonstration purposes, and you should never hard-code passwords in a real application.

Database initialisation

When the Postgres docker container starts for the first time, it runs an initialisation script, and this is where the database is created, and restrictions are applied.

Using containers in this way makes it very easy to initialise and run an application consistently. If you are not using Docker, then use another method of initialising the Postgres database. For example, an external hook which runs when the db is initialised, such as an AWS Lambda function.

Postgres schemas

Postgres has the concept of schemas, and this forms a useful basis for securing our application. A schema is effectively a namespace. If you have a table called Customer, and a schema called App, then the Customer table can be isolated to within the App schema.

By explicitly defining roles which have access to a schema, we can be confident that a database user (i.e. our Django application) is accessing only those tables and sequences which they are permitted to do so.

Postgres has a default schema, called public, and unless otherwise specified, database users can access and create objects in this public schema. This creates a potential security weakness which we will remove.

Postgres roles

A Postgres role can be created and given explicit privileges. A database user can then be assigned to the role. For example, we could create a 'read-only' role and a 'read-write' role. Permissions could be restricted according to these roles.

By explicitly defining privileges, we are clear about what given roles are permitted to do.

DB initialisation walkthrough

There are some particular nuances with defining schemas and roles for use with Django. In the rest of this post, I'll walk through the steps required to restrict user access for Django.

Specifically, our script needs to address the following:

Create a database and define schemas and roles.
Create a 'django migrator' user which has permission to create our application objects (i.e. those created when manage.py migrate is run). Additionally, the 'django migrator' user must be allowed to create databases, because this is necessary when tests are run.
Create a 'django application' user. This user will have appropriate read and write privileges on our application tables.

The initialisation script

Our script will run only when the database container is started for the first time. The script will connect to the Postgres db using the default user.
The full script is visible here.

Note that environment variables referenced here (e.g. $DB_NAME) are defined in docker-compose.yml.

Create our application database

createdb $DB_NAME

Define a heredoc to run a script through psql. The connection to psql is made using the default Postgres user.

psql -v ON_ERROR_STOP=1 --dbname "$DB_NAME" <<-EOSQL
...
EOSQL

Lockdown the 'public' schema. This prevents creation of new objects unless given permission.

REVOKE CREATE ON SCHEMA public FROM PUBLIC;

Prevent any connection to the new database unless explicitly given

REVOKE ALL ON DATABASE $DB_NAME FROM PUBLIC;

Create Migrator role. The migrator user is allowed to create databases for the purposes of running the Django test suite.

CREATE USER $MIGRATOR_DB_USER WITH PASSWORD '$MIGRATOR_DB_PASS' CREATEDB;
CREATE SCHEMA $SCHEMA AUTHORIZATION $MIGRATOR_DB_USER;

The migrator user must have access to the 'public' schema because that is where new db tables for the test db will be created.

ALTER ROLE $MIGRATOR_DB_USER SET SEARCH_PATH TO $SCHEMA, public;

Create an 'application user' role. This user must only be allowed to query the application schema. The 'search path' is defined to ensure that only our application schema is visible to the application db user.

CREATE USER $DB_USER WITH PASSWORD '$DB_PASS';
ALTER ROLE $DB_USER SET SEARCH_PATH TO $SCHEMA;

Create a role which can connect to the db and has read / write access to the application schema only.

CREATE ROLE $RW_ROLE;
GRANT CONNECT ON DATABASE $DB_NAME TO $RW_ROLE;
GRANT USAGE ON SCHEMA $SCHEMA TO $RW_ROLE;
GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA $SCHEMA TO $RW_ROLE;
GRANT USAGE ON ALL SEQUENCES IN SCHEMA $SCHEMA TO $RW_ROLE;

If you examine the GRANT statements above, you can see how it would be possible to restrict access further for other types of roles. For example, we could define a 'read-only' role which allowed only SELECT privileges.

Grant the defined read / write role to our db users.

GRANT $RW_ROLE TO $MIGRATOR_DB_USER;
GRANT $RW_ROLE TO $DB_USER;

Alter default privileges for read / write role. This is a necessary step because otherwise the role will not have access to any tables which are created later. That will include tables created in the manage.py migrate step, and that would render the application inaccessible to users!

ALTER DEFAULT PRIVILEGES FOR USER $MIGRATOR_DB_USER GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO $RW_ROLE;
ALTER DEFAULT PRIVILEGES FOR USER $MIGRATOR_DB_USER GRANT USAGE, SELECT, UPDATE ON SEQUENCES TO $RW_ROLE;

We can see the above steps in the output produced when Postgres is initialised. Once complete, it will mean that we have two users, one of whom has restricted privileges to access database objects.

Application entrypoint

If you are familiar with Docker, you will know that you can define an entrypoint script which will be run when the container starts.

We will use this script to run our migrations and start the server.

Referencing the Migrator User

Recall that we have defined our 'migrator user' to have the appropriate permissions to create tables (and other objects). We want to use this user when our migrations are run.

This is achieved by defining an additional settings_migrator.py file, which overrides the db connection parameters:

DATABASES["default"]["USER"] = 'migrator_user'  
DATABASES["default"]["PASSWORD"] = os.environ["MIGRATOR_DB_PASS"]

We must reference this overridden settings file when we run migrate (in entrypoint.sh):

./manage.py migrate --settings=tutorial.settings_migrator

We will not override settings anywhere else in the entrypoint script, which means that all other manage.py commands will run using our application user.

Note that if you exclude the --settings argument, then the migration task will fail, because it doesn't have appropriate permissions. You can try this out by removing the argument from entrypoint.sh, and running docker-compose down -v; docker-compose up --build.

A note on testing

As mentioned earlier, by default Django requires permission to create databases when running tests. This means that tests also need to be run using the --settings flag which defines the connection details for the 'migrator user'.

Conclusion

This post has shown how we can restrict database access for our Django applications using Postgres features. Two users were defined in the Postgres db. We then used one 'migrator user' to create run our migrations. The application user was given read / write privileges to the application schema only.

When building web applications, restricting access rights at the database level is an important security consideration, therefore it is worthing spending some time understanding how Postgres configuration can help.

SwaggerUI inside Django Rest Framework

Matthew Hegarty — Thu, 28 Mar 2019 11:58:06 +0000

Introduction

An API schema provides a standard definition of the details of your API, which can be rendered in interactive web pages, or can be used to generate client code.

In this post I describe how to modify a Django Rest Framework (DRF) application to serve an existing OpenAPI (aka Swagger) schema definition rendered using Swagger UI:

A sample application is available here.

Why not use a framework?

DRF does support generation of schemas using OpenAPI, however support for OpenAPI was only added in release 3.9, and it is still in its early stages. It's worth checking DRF for ongoing OpenAPI support because this is being actively developed.

If you are working with Swagger (OpenAPI v2), then you can potentially use drf-yasg to generate and serve a schema view. drf-yasg is a third-party plugin for DRF.

However, if you are using OpenAPI v3 (OpenAPI is the new name for Swagger), or have a pre-existing schema definition file, then using a third-party framework may be impossible or impractical, and the next best thing might be to render a static schema file, as I describe here.

There are other options in DRF for rendering a schema which don't involve OpenAPI (see the DRF) docs), although DRF is moving towards supporting OpenAPI.

What do we need to do?

To serve an existing API schema file from DRF, we need to do the following:

Define the schema definition file.
Configure DRF to serve the schema file.
Install Swagger UI as a static resource.
Create a Django Template to serve the UI.

These steps are described in detail below.

Install the DRF Tutorial

I have used a fork of the DRF tutorial to demonstrate the steps required.
You can clone the repo and run locally to see how it fits together.
The fork includes a schema.yaml file for the tutorial project.

You can install the clone as follows:

git clone git@github.com:matthewhegarty/rest-framework-tutorial.git
cd rest-framework-tutorial

# Create a virtualenv to isolate our package dependencies locally
virtualenv env
source env/bin/activate  
pip install -r requirements.txt

export DJANGO_SETTINGS_MODULE=tutorial.settings
python manage.py migrate

# when prompted, create a suitable password
python manage.py createsuperuser --email user@example.com --username admin

Now you can run the server with:

python manage.py runserver

Test the API

If the tutorial is installed correctly, then you should be able to browse http://localhost:8000/ and see the default API Root.

You can now login as Admin using the link at the top right hand side of the page, and log in using the credentials you created earlier.

Define the schema

First of all we need to serve the Schema definition. OpenAPI definitions can be written in either yaml or json. In this example, yaml is used.

As of DRF 3.9, you can generate a schema with:

python manage.py generateschema > schema.yml

However, the output of generateschema is only going to be a stub of your API, and you will likely need to use this as a starting point, and add the specifics of your API.

As you work on your schema, you can validate it by copying the source definition into the Swagger editor tool.

I have already created a schema for the rest-framework-tutorial application
here.

Serve the schema

Once the schema is ready, it should be checked into source control. It can now be served as part of the application.

Create a 'static' directory under the application root, in which we will put static web content to serve to clients. For example:

mkdir -p snippets/static/openapi

Move your schema.yaml into this new directory.

Update URLs

Now edit the URLs file (tutorial/urls.py), and add the following:

from django.conf.urls.static import static
from tutorial import settings

urlpatterns = [
# Leave the existing urls defined here
] + static(settings.STATIC_URL)

This is making use of Django's static file serving functionality, and you should read Django's documentation on this topic.

Restart the server if necessary, and now hitting the endpoint should download the schema file, for example:

wget http://localhost:8000/static/openapi/schema.yaml

Install Swagger UI

The next step is to install the Swagger UI distribution into our static files, so that it can be served alongside the application.

Clone the Swagger UI repo locally.

Create static directory for Swagger UI

Create another directory under your static root to serve the SwaggerUI files:

mkdir -p snippets/static/openapi/swagger-dist-ui

Now copy the contents of SwaggerUI's dist directory into the swagger-dist-ui directory you just created, for example:

cp -av ../swagger-ui/dist/* snippets/static/openapi/swagger-dist-ui

Create a Django Template for Swagger UI

Our final step is to configure Django to serve Swagger UI. To do this we need to create a template which Django can serve SwaggerUI files from.

Create a new directory for the template:

mkdir -p snippets/templates

Now move the SwaggerUI index.html into this directory:

mv snippets/static/openapi/swagger-dist-ui/index.html snippets/templates/

Define Template Directory in Config

After the above step, check that your config references the templates directory correctly. In tutorial/settings.py, add the 'templates' directory to DIRS (leave all other config as is):

# tutorial/settings.py
TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',
        'DIRS': ['templates'],
        ...
    },
]

Configure Template

Now we need to edit the index.html file so that it references static resources such as Swagger UI's js and css files.

Edit index.html:

Add the {% load static %} directive to the top of the file.
Work through the file and modify all static file references to use a
Django template directive.

For example, on line 7, change the href reference from ./swagger-ui.css to {% static "openapi/swagger-dist-ui/swagger-ui.css" %}.

Do this for all other file references in index.html.

Change the url reference in SwaggerUIBundle to reference your schema file: url: "{% static "openapi/schema.yaml" %}"

The end product should look something like this.

Add a URL reference

Last of all, we need to add a reference to the template in urls.py in order to serve the index.html file.

from django.views.generic import TemplateView

urlpatterns = [
    url('openapi/', TemplateView.as_view(template_name="index.html")),
    url(r'^', include(router.urls))
]

Test the UI

Now if we browse to http://localhost:8000/openapi/ we should see the Swagger UI rendered with our schema.yaml.

Conclusion

In this post I've covered rendering an existing schema file in Swagger UI and serving the UI as part of a Django Rest Framework project.

This approach works well in Development, but is not suitable for Production use. Refer to the Django documentation for further guidance on serving static files in Production.

Further support for OpenAPI / Swagger is planned for Django Rest Framework, so this process might be refined in future DRF releases. Follow the release notes for updates.

DEV Community: Matthew Hegarty

Troubleshooting AWS Batch and PrivateLink

ECS Instances not being created in the ECS cluster

Jobs run but then fail with 'CannotPullECRContainerError'

Other Resources

Postgresql: Better security for Django Applications

Sample Application

Database initialisation

Postgres schemas

Postgres roles

DB initialisation walkthrough

The initialisation script

Application entrypoint

Referencing the Migrator User

A note on testing

Conclusion

Further reading

SwaggerUI inside Django Rest Framework

Introduction

Why not use a framework?

What do we need to do?

Install the DRF Tutorial

Test the API

Define the schema

Serve the schema

Update URLs

Install Swagger UI

Create static directory for Swagger UI

Create a Django Template for Swagger UI

Define Template Directory in Config

Configure Template

Add a URL reference

Test the UI

Conclusion