DEV Community: Takahiro Kudo

Django with whitenoise.storage.CompressedManifestStaticFilesStorage on Heroku

Takahiro Kudo — Wed, 28 Jul 2021 13:46:41 +0000

I had been having trouble deploying the Django app to Heroku with "whitenoise.storage.CompressedManifestStaticFilesStorage."

Premise

Use docker
Django 3.2.4
whitenoise 5.3.0

Deploying flow

Push to GitHub
Build a container image from a Dockerfile in CircleCI
Push the container image to Heroku

In this time, mistakes were in the flow 2.

Mistakes

settings.py was below:

# ...

DEBUG = os.getenv('APP_DEBUG', 'true') == 'true'
APP_ENV = os.getenv('APP_ENV', 'dev')

# ...

MIDDLEWARE = [
    # ...
    # @see https://devcenter.heroku.com/articles/django-assets
    'whitenoise.middleware.WhiteNoiseMiddleware',
]

# ...

STATIC_ROOT = os.path.join(BASE_DIR, 'assets')
# STATIC_URL = '/static/'
STATIC_URL = '/assets/'
# STATICFILES_DIRS = [
#     BASE_DIR / "static",
# ]

if APP_ENV == 'prod':
    # Error!
    STATICFILES_STORAGE = 'whitenoise.storage.CompressedManifestStaticFilesStorage'
    # OK. Why?
    # STATICFILES_STORAGE = 'whitenoise.storage.CompressedStaticFilesStorage'

It was OK to run on Heroku with "whitenoise.storage.CompressedStaticFilesStorage", but Error500 raised when using "whitenoise.storage.CompressedManifestStaticFilesStorage".

Cause

The cause was in Dockerfile. APP_ENV=prod and DEBUG=false were set in Heroku, but these environment variables were not added in CircleCI. Therefore the correct container image didn't run on Heroku.

Fixed

Fixed Dockerfile as follow:

# ...

# Added lines below.
# Django: Collect static
ENV APP_ENV=prod APP_DEBUG=false
RUN rm -rf assets && python manage.py collectstatic --no-input

# ...

Deploy Docker image to Heroku with CicleCI

Takahiro Kudo — Tue, 13 Jul 2021 16:11:36 +0000

Heroku is a functional PaaS that can deploy the Docker image, and CircleCI is also useful SaaS for CI/CD.

In this post, I show an example of CircleCI config which deploys the Docker image to Heroku.

Actually, I had trouble when setting it up. 🤣

Prerequisite

Code repository and CircleCI had already connected
Created Heroku app for deploying the Docker image
Generated Heroku API Key in Manage Account view on Heroku

Set up

Environment Variable

Set environment variables on your project settings in CircleCI.

Name	Value
HEROKU_API_KEY	Your Heroku API Key
HEROKU_APP_NAME	Heroku app name for deploying the Docker image

config.yml

Modify config.yml below.

jobs:
  # Build the docker image and run tests.
  build_and_test:
    docker:
      - image: cimg/base:2021.07
    steps:
      # For using docker commands.
      - setup_remote_docker:
          version: 19.03.13

      # Build the docker image
      - checkout
      - run: docker build -t webapp .

      # Run tests.
      - run:
          command: docker run webapp python3 tests.py

  # Build the docker image and deploy it to Heroku.
  build_and_deploy:
    docker:
      - image: cimg/base:2021.07
    steps:
      # For using docker commands.
      - setup_remote_docker:
          version: 19.03.13

      # Build the docker image
      - checkout
      - run: docker build .

      # Deploy to Heroku
      - heroku/install
      - run: heroku container:login
      - heroku/push-docker-image:
          process-types: web
      - heroku/release-docker-image:
          process-types: web

orbs:
  # For using Heroku CLI commandｓ.
  heroku: circleci/heroku@1.2.6

version: 2.1
workflows:
  heroku_deploy:
    jobs:
      - build_and_test
      - build_and_deploy:
          requires:
            # Deploy after testing.
            - build_and_test

Test

Example codes are in this repository.

Refs.

Thanks!😆

Note: graphql-go Tutorial

Takahiro Kudo — Sun, 21 Mar 2021 18:05:47 +0000

I did graphql-go Tutorial and had a little trouble. Therefore I share my note.

Getting started

https://www.howtographql.com/graphql-go/1-getting-started/

validation failed: packages.Load error occured.
To fix it, setup go module.

% cd <project dir>
% go mod init
% go get github.com/99designs/gqlgen

# This module is needed.
% go get github.com/vektah/gqlparser/v2

%gqlgen init

Database

https://www.howtographql.com/graphql-go/4-database/

server.go needs go-chi.

% go get github.com/go-chi/chi

Testing query and mutation.

We can test on browser UI which URL is http://localhost:8080.

Codes

https://github.com/takakd/graphql-go-tutorial

AWS Cognito components overview

Takahiro Kudo — Sat, 20 Mar 2021 15:18:25 +0000

My note about AWS Cognito.

User Pool

It is an authentication component that stores user attributes; name, email address, etc.
A client can retrieve some tokens as JWT; ID token, Access token, and Refresh token.

ID token

ID token includes user attributes; name, email address, etc.
API Gateway and Cognito Identity Pool uses this token to authenticate.

Access token

A client requests to User Pool to edit user attributes with the access token.

Refresh token

A client requests User Pool to retrieve the new access token with the refresh token if the access token is expired.

Identity Pool

It is an authorization component that serves token to access AWS resources.
A client can retrieve STS tokens as JWT with the ID token.

Go - package is not in GOROOT

Takahiro Kudo — Tue, 02 Mar 2021 13:23:26 +0000

The below error occurred when I had changed GOPATH, made two packages, and written some codes.

% package other/pkg1 is not in GOROOT (/usr/local/go/src/other/pkg1)

The cause is a package structure. Putting it together solves it.

Error

Below is the directory structure when the error occurred.

${GOPATH}/src
|-- other
|   |-- go.mod
|   `-- pkg1
|       `-- pkg1.go
`-- prj
    |-- go.mod
    `-- main.go

prj package is unable to refer to other/pkg1 although it is in GOPATH.

Codes are below.

prj/main.go

package main

import "other/pkg1"

func main()  {
    pkg1.Func()
}

other/pkg1/pkg1.go

package pkg1

import "fmt"

func Func() {
    fmt.Println("called func in pkg1")
}

Fix

To include other in prj solves the error.

${GOPATH}/src
`-- prj
    |-- go.mod
    |-- main.go
    `-- other
        `-- pkg1
            `-- pkg1.go

prj/main.go

package main

import "prj/other/pkg1"

func main()  {
    pkg1.Func()
}

Perl - mockgen helper perl script

Takahiro Kudo — Mon, 25 Jan 2021 12:59:26 +0000

Created the helper script of mockgen because typing mockgen command takes time and effort.

Ref.
golang/mock

Shell - load env variables from .env

Takahiro Kudo — Sat, 19 Dec 2020 11:57:35 +0000

Sample codes to load environment variables from .env file in sh.

#!/usr/bin/env sh

# Load variables from .env file.
export $(cat ./.env | grep -v ^# | xargs) >/dev/null

# Refer to variables.
echo "KEY_IN_ENV=${KEY_IN_ENV}"

Here is test code, which creates test .env and echo it.

#!/usr/bin/env sh

TEST_KEY1="This value will be replaced."

# Create test .env
cat <<EOF > .env
TEST_KEY1=VALUE1
TEST_KEY2=VALUE2
EOF

# Load .env
export $(cat ./.env | grep -v ^# | xargs) >/dev/null

# Check
echo "TEST_KEY1=${TEST_KEY1}"
echo "TEST_KEY2=${TEST_KEY2}"

Go - Set structure

Takahiro Kudo — Sat, 12 Dec 2020 19:43:10 +0000

Simple Set struct in Golang.

Example

package collection_test

import (
    "fmt"
    "<Project>/collection"
)

func ExampleSet() {
    s := collection.NewSet()

    // Set values
    s.Set("value1")
    s.Set("value2")

    // Check
    fmt.Println(s.Contains("value1"))
    fmt.Println(s.Contains("value2"))

    // Remove
    s.Remove("value2")

    // Removed so does not exists.
    fmt.Println(s.Contains("value2"))

    // Output:
    // true
    // true
    // false
}

Go - aws-sdk-go/service Unit Test

Takahiro Kudo — Thu, 26 Nov 2020 17:22:42 +0000

Here is sample codes for the unit test of aws-sdk services.

In the case of DynamoDB.

package repository

import (
    "context"
    "fmt"
    "github.com/aws/aws-sdk-go/aws"
    "github.com/aws/aws-sdk-go/aws/session"
    "github.com/aws/aws-sdk-go/service/dynamodb"
    "github.com/aws/aws-sdk-go/service/dynamodb/dynamodbattribute"
    // ...
)

// SDK wrapper interface, which can be able to unit test.
type DynamoDbWrapper interface {
    // Define the same as SDK.
    Query(input *dynamodb.QueryInput) (*dynamodb.QueryOutput, error)

    // Define, if it needs more.
    // ...     
}

// Default implementation.
// Use this if the argument of "NewDynamoDbRepository" is not passed.
type DynamoDbWrapperAdapter struct {
    svc *dynamodb.DynamoDB
}

// Dispatch simply.
func (d *DynamoDbWrapperAdapter) Query(input *dynamodb.QueryInput) (*dynamodb.QueryOutput, error) {
    return d.svc.Query(input)
}

// Set wrp to null. In case unit test, set mock interface.
func NewDynamoDbRepository(wrp DynamoDbWrapper) updatetimerevent.Repository {
    if wrp == nil {
        wrp = &DynamoDbWrapperAdapter{
            svc: dynamodb.New(session.New()),
        }
    }
    return &DynamoDbRepository{
        wrp: wrp,
    }
}

// Find something.
func (r *DynamoDbRepository) FindSome(ctx context.Context, userId string) (some *SomeStruct, err error) {
    input := &dynamodb.QueryInput{
        ExpressionAttributeValues: map[string]*dynamodb.AttributeValue{
            ":userid": {
                S: aws.String(userId),
            },
        },
        KeyConditionExpression: aws.String("UserId = :userid"),
        TableName:              aws.String("DYNAMODB_TABLE"),
    }

    // Call mock method, if r.wrp is set mock.
    result, err := r.wrp.Query(input)
    if err != nil {
        return
    }
    // ...
}

Test code

t.Run("ok:Query", func(t *testing.T) {
    caseUserId := "test user"
    caseInput := &dynamodb.QueryInput{
        ExpressionAttributeValues: map[string]*dynamodb.AttributeValue{
            ":userid": {
                S: aws.String(caseUserId),
            },
        },
        KeyConditionExpression: aws.String("UserId = :userid"),
        TableName:              aws.String("DYNAMODB_TABLE"),
    }
    caseItem := &dynamodb.QueryOutput{
        Items: []map[string]*dynamodb.AttributeValue{},
    }

    ctrl := gomock.NewController(t)
    defer ctrl.Finish()

    s := NewMockDynamoDbWrapper(ctrl)
    s.EXPECT().Query(gomock.Eq(caseInput)).Return(caseItem, nil)

    repo := NewDynamoDbRepository(s)
    got, err := repo.FindSome(context.TODO(), caseUserId)
    assert.NoError(t, err)
    // assert...
})

AWS - The REST API doesn't contain any methods (Service: AmazonApiGateway...

Takahiro Kudo — Sun, 15 Nov 2020 16:07:04 +0000

The below error was raised when the CloudFormation template deploys the proxy integration API.

CREATE_FAILED  AWS::ApiGateway::Deployment  SomeDeployment The REST API doesn't contain any methods (Service:BadRequestException; Request ID:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx; Proxy: null)

The reason for this is that DependsOn attribute needs in AWS:ApiGateway:Deployment because AWS:ApiGateway:Method should be created before creating the deployment.

Should we always write DependsOn attribute?

AWS - Set static value to IntegrationResponses

Takahiro Kudo — Sun, 01 Nov 2020 15:59:49 +0000

Write the following in case setting the static value to Integration response in API Gateway.

SomeMethod:
  Type: AWS::ApiGateway::Method
  Properties:
    HttpMethod: POST
    ...
    Integration:
      Type: AWS
      IntegrationHttpMethod: POST
      ...
      IntegrationResponses:
        - StatusCode: 200
          SelectionPattern: "status:200"
          ResponseParameters:
            # OK
            method.response.header.X-STATIC-HEADER: "'1'"
            # NG
            method.response.header.X-STATIC-HEADER: 1
            # NG
            method.response.header.X-STATIC-HEADER: "1"

Reference

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-apitgateway-method-integration-integrationresponse.html#cfn-apigateway-method-integration-integrationresponse-responseparameters

Go - Prevent Slack Event API retry call

Takahiro Kudo — Thu, 22 Oct 2020 18:55:45 +0000

Prevent Slack Event API retry call if the callback returns the error response.

func handler(w http.ResponseWriter, r *http.Request) {
    // Process something...

    // Set "X-Slack-No-Retry: 1" if prevent retry.
    w.Header().Set("X-Slack-No-Retry", "1")

    w.Header().Set("Content-Type", "application/json")
    w.WriteHeader(http.StatusBadRequest)
    w.Write([]byte(""))
}

Reference

https://api.slack.com/events-api#the-events-api__field-guide__error-handling__graceful-retries__turning-retries-off