DEV Community: Jinwook Baek

Majestic Monolith Django

Jinwook Baek — Sun, 17 Apr 2022 07:34:50 +0000

Introduction

https://github.com/kokospapa8/majestic-monolith-django

This project is a starter django repo aiming to achieve majestic modular monolith architecture. Main purpose is to provide scaffolding skeleton and sample architecture for rapid prototype structure that can scale to mid-range size application. I have complied useful techniques and libraries to help build backend API server.

Inspired by Majestic monolith and Two Scoops of Django, this starter code will help you build scalable application for a small team of developers.

Majestic monolith

MicroService is everywhere and no doubt that they are the next big thing, for a company with many developers and in need for concurrent feature releases.

However, MSA needs a lot of coordination and preparation to make is work. If you are the only developer in the team or developing ina a relatively small to medium scale architecture, MSA can be overwhelming.

You can reduce cognitive load by following DDD practice. With code isolation, data isolation and some cloud architecture help, majestic monolith django(MMD) can prepare for the scale and bigger team coordination.

Example application

This repo provides sample application illustrating following usecase.

I have broken down the application into four modules(auth, user, shipping, distribution) using techniques I used for application prototyping.

Module Structure

Code

Each domain consists of following structure.

admin: django admin
apps: Django App Config
choices: Enum files used in module
docs: yasg doc
events: events emitted from django module
exceptions: Custom exceptions
models: Ordinary django models
manager: Queryset managers
serializers: DRF serialiers
selectors: Query that requires join
services: Biz logics for domain
- domain services (domain specific biz logic)
- application services (UoW)
schema: api or model schema for testing
urls: django url resolver
utils_*: utils method
views: DRF views

Infrastructure

In order to achieve modular structure in single monolith, we need to use eventbus for inter-module communications. Also we are fully utilizing lambda compute to unburden load of api servers. Python application typically utilizes celery beat for heartbeat (cron) process but it is much easier to use eventbridge schedule with lambda calling heartbeat api.

Eventbus
- In order to decouple models, mmd uses serverless event bus(Eventbridge)
SAM (lambda) for async compute
- notification
- long runnign tasks
- Heartbeat with lambda to call api

Cookie cutter

You can use cookie cutter to start the repo.

pip install cookiecutter
cookiecutter https://github.com/kokospapa8/majestic-monolith-django.git --checkout cookie-cutter

Future todo

Async support
Dependency injection
SQS for batch POST request

TL;DR

Try out my sample application!

https://github.com/kokospapa8/majestic-monolith-django/

Migrate log analysis from AWS ElasticSearch to BigQuery

Jinwook Baek — Fri, 29 Jan 2021 14:17:26 +0000

Original blog post

Introduction

I have been using AWS ElasticSearch for near-real time analysis for API servers. AWS provides built-in Elasticsearch subscription filter for CloudWatch. With less than a hour of effort I can spin up an Elasticsearch cluster to visualize and analyze server logs. However AWS Elasticsearch is not cheap in production setup, also as logs accumulate, it needs some maintenance (lifecycle policy, JVM pressure, etc). Also dev team was not fully utilizing the elastic stack as much. I have been meaning to decommission the Elasticsearch and find alternative option for several month. And finally I decided that it's time to migrate to other solution for following reason.

Amazon: NOT OK - why we had to change Elastic licensing

After some survey, I listed requirements for the substitue

Near-realtime analysis (no batch load)
Simple filter and aggregation by time range, api status, etc
Access management
Support for visualization tool (substitue Kibana)
No extra setup on server code (no splunk, fluntd, etc)

Anchoring CloudWatch logs are the inception point of all the logs(nginx, api logs, etc) There were couple of alternatives I could think about.

Vanilla CloudWatch (manually searching through logs and no visulization, query support → no need to implement anything else)
CloudWatch → S3(firehose) → Athena
CloudWatch → S3(firehose) → Redshift Spectrum
CloudWatch → Kinesis Stream → Kinesis Analysis
CloudWatch → Lambda → BigQuery

I was initially inclined to choose something from AWS since I prefer to have less moving pieces and do less coding. However I am already using BigQuery for Data warehouse. I concluded that if I was going to stash some data somewhere I might as well centralize all of them in single place. So I chose to go with lambda+Bigquery. For this post, I will be using nginx log for example, since these formats are common for most of people.

Load vs. Stream

Before going into actual setup process I must acknowledge couple things such as distinction betweenLoad and streaming. Load is when you load batch of data for once or in recurrence. In BigQuery, Load is free and it only charge for storage. However Stream incurs charge and each insert is $0.010 per 200 MB .

Prerequisite

GCP Account with billing enabled (streaming is disabled on free tier)
Nginx log on Cloudwatch
SAM environment setup + Lambda role (refer to previous blog post regarding lambda env setup) Migrate python async worker to asynchrounous Lambda

Setup

I am going to cover following components

Inspect Cloudwatch logs
BigQuery table
Lambda code
Cloudwatch subscription
Biquery + datastudio

BigQuery Table and Schema

First of all, I need to create BigQuery table for the nginx logs to be stored. In order to create correct schema, let's take a look at how the logs will be sent from CloudWatch to lambda subscription filter.

Nginx log

Default nginx log format looks like this. (My log sample has remote ip appended at the end.)

log_format combined '$remote_addr - $remote_user [$time_local] '
                    '"$request" $status $body_bytes_sent '
                    '"$http_referer" "$http_user_agent"';

//sample
172.11.1.1111 - - [28/Jan/2021:16:24:03 +0000] "GET /api/healthcheck/ HTTP/1.1" 200 2 "-" "CRAZY_USER_AGENT/2.0" "111.111.111.111"

I need to transform this log in order to insert to bigquery table in certain format. Thankfully, CloudWatch subscription filter provide filter pattern.

Filter and Pattern Syntax

If I provide following filter pattern like [internal_ip, identitiy , auth_user, date, request, status, bytes, referer, useragent, remote_addr], subscription filter automatically breaks down the log in json format as extractedFields. You can even select certain logs for this filter like [internal_ip, identity , auth_user, date, request !="**/api/healthcheck/**", status, bytes, referer, useragent, remote_addr]

Now we know that the log will come in certain format, I will create a table with following format.

id:string,
timestamp:timestamp,
remote_addr:string,
useragent:string,
referer:string,
bytes:numeric,
status:string,
request:string,
auth_user:string,
date:string,
internal_ip:string,
identity:string

If I need more fields later, I will add them. Make sure that the table is partitioned by timestamp for the query performance and also expire the table after certain time for the cost reduction. (BQ applies cold storage for untouched data after certain periods of time so you might just want to keep them)

Create BigQuery table

Let's go to GCP console and create table with following schema.

You can use cloudshell if you prefer CLI.

bq mk \
--table \
--description description \
--time_partitioning_field timestamp \
--time_partitioning_type DAY \
--label key:value, key:value \
<project_id>:<dataset>.<table> \
id:string,timestamp:timestamp,remote_addr:string,\
useragent:string,referer:string,bytes:numeric,status:string,\
request:string,auth_user:string,date:string,internal_ip:string,\
identity:string,

Once table is created, copy the table name from detail tab. <project_id>.<dataset>.<table-name>

Service Account

Before going back to AWS and create a lambda function, we need to create a service account for lambda in order to gain permission to insert rows to BQ.

Go to service account under IAM menu and click + create service account

Google Cloud Platform

service account name : lambda_bq_stream

permissions : bigquery dataeditor

you can defined custom role with more restrictive permission you only need

bigquery.tables.list
bigquery.tables.get
bigquery.tables.updateData

Once you created a service account you can download a json key click Add key -> create new key. Download and stash somewhere safe, we will be using this key later.

Json key sample

{
  "type": "service_account",
  "project_id": "****",
  "private_key_id": "****",
  "private_key": "****",
  "client_email": "****",
  "client_id": "****",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/lambda-bigquery-stream%40hangfive-26bb4.iam.gserviceaccount.com"
}

I will use SAM for lambda deployment, please refer to previous post for the SAM setup.

Migrate python async worker to asynchrounous Lambda

Lambda Fucntion

Ok, now let's focus on the code. This function will receive stream of cloudwatch logs and stream insert rows to BQ table using google bigquery SDK written in python. I will break down the code in pieces to explanation.

Library

All you need is single pip library.

google-cloud-bigquery

Authentication

Client requires authentication and there are two ways to authenticate; from file or json dictionary. There are couple options to do this.
google-auth

Json file
- upload file to s3 and load them from lambda
- package json with the lambda code → (x)

google-auth

dictionary
- environment variable → I will use this for the purpose of simplicity
- load key info from secret manager or parameter store

Environment Variable

Although it is better to manage key from secret manager, I will use env var for the simplicity of exercise. I will supply following Environment Variable from json dictionary I have downloaded from previous sections

  "project_id":
  "private_key_id":
  "private_key":
  "client_email": 
  "client_id":

Decompress and decode cloudwatch logs

When you add lambda subscription filter to CloudWatch, the CloudWatch log event will be sent in gzipped / base64 encoded data. This is an example format. (Don't try to decode the data it's broken.)

You decompress and decode the message with following code.

compressed_payload = b64decode(data)
cloudwatch_payload = zlib.decompress(compressed_payload, 16 + zlib.MAX_WBITS)
json_payload = json.loads(cloudwatch_payload)

Resulting message will be decoded as following example format. Notice logEvents for the list of logs. Notice that message is actually log and they are broken as into extractedFields according to filter supplied as [internal_ip, identity , auth_user, date, request, status, bytes, referer, useragent, remote_addr]

{
    'messageType': 'DATA_MESSAGE',
    'owner': '12341244',
    'logGroup': '/ecs/nginx',
    'logStream': 'ecs/nginx/65366ed9299f4554a2cf0dbd4c49ee08',
    'subscriptionFilters': ['nginx_filter_sample'],
    'logEvents': [{
        'id': '35945479414329140852051181308733124824760966176424001536',
        'timestamp': 1611851043287,
        'message': '172.11.1.1111 - - [28/Jan/2021:16:24:03 +0000] "GET /api/healthcheck/ HTTP/1.1" 200 2 "-" "CRAZY_USER_AGENT/2.0" "111.111.111.111"',
        'extractedFields': {
            'date': '28/Jan/2021:16:24:03 +0000',
            'request': 'GET /api/healthcheck/ HTTP/1.1',
            'referer': '-',
            'remote_addr': '-',
            'bytes': '2',
            'ip': '172.11.1.111',
            'useragent': 'ELB-HealthChecker/2.0',
            'identity': '-',
            'auth_user': '-',
            'status': '200'
        }
    }]
}

SKIP CONTROL MESSAGE

If control message is passed, we will skip insert

if json_payload['messageType'] == 'CONTROL_MESSAGE':
        return

Create payload

We will create payload for the nginx log. Notice I have devided timestamp by 1000 since BQ supports microsecond resolution for timestamp type column.

rows_to_insert = []

for row in json_payload['logEvents']:
    item = {}
    item['id'] = row['id']
    item['timestamp'] = row['timestamp'] / 1000
    if 'extractedFields' in row:
        for k, v in row['extractedFields'].items():
            item[k] = v

    rows_to_insert.append(item)

Inserting rows

Before inserting to BQ, let's take a quick look at the description of insert_rows_json method.

insert_rows_json method description Make sure to read row_ids params. It's a unique identifier in order to maintain deduplication.

"""Insert rows into a table without applying local type conversions.

        See
        https://cloud.google.com/bigquery/docs/reference/rest/v2/tabledata/insertAll

        Args:
            table (Union[ \
                google.cloud.bigquery.table.Table \
                google.cloud.bigquery.table.TableReference, \
                str \
            ]):
                The destination table for the row data, or a reference to it.
            json_rows (Sequence[Dict]):
                Row data to be inserted. Keys must match the table schema fields
                and values must be JSON-compatible representations.
            row_ids (Optional[Sequence[Optional[str]]]):
                Unique IDs, one per row being inserted. An ID can also be
                ``None``, indicating that an explicit insert ID should **not**
                be used for that row. If the argument is omitted altogether,
                unique IDs are created automatically.
            skip_invalid_rows (Optional[bool]):
                Insert all valid rows of a request, even if invalid rows exist.
                The default value is ``False``, which causes the entire request
                to fail if any invalid rows exist.
            ignore_unknown_values (Optional[bool]):
                Accept rows that contain values that do not match the schema.
                The unknown values are ignored. Default is ``False``, which
                treats unknown values as errors.
            template_suffix (Optional[str]):
                Treat ``name`` as a template table and provide a suffix.
                BigQuery will create the table ``<name> + <template_suffix>``
                based on the schema of the template table. See
                https://cloud.google.com/bigquery/streaming-data-into-bigquery#template-tables
            retry (Optional[google.api_core.retry.Retry]):
                How to retry the RPC.
            timeout (Optional[float]):
                The number of seconds to wait for the underlying HTTP transport
                before using ``retry``.

        Returns:
            Sequence[Mappings]:
                One mapping per row with insert errors: the "index" key
                identifies the row, and the "errors" key contains a list of
                the mappings describing one or more problems with the row.

        Raises:
            TypeError: if `json_rows` is not a `Sequence`.
        """

Following code will insert a row into BQ table

table_id = '<project_id>.<dataset>.<table-name>'
errors = client.insert_rows_json(table_id, rows_to_insert)  # Make an API request.
    if errors == []:
        print("New rows have been added.")
    else:
        print("Encountered errors while inserting rows: {}".format(errors))

Full Lambda Code

Now that we have covered all the parts, let's combine together and create a function.

Full Lambda Code

Now that we have covered all the parts, let's combine together and create a function.

Test the function with sample events

Let's test the code for sample event payload. As a reminder, you can invoke function with following command

sam local invoke --env-vars env.json -e events/payload.json

Once you see following success message, let's go to BigQuery table and check if data is inserted successfully.

Successfully streamed to the table!

CloudWatch Subscription filter

IT's time to attach lambda subscription on log group. Refer to following CLI command or console.

put-subscription-filter - AWS CLI 1.18.221 Command Reference

put-subscription-filter
--log-group-name <value>
--filter-name nginxToBQ
--filter-pattern [internal_ip, identity , auth_user, date, request, status, bytes, referer, useragent, remote_addr]
--destination-arn <lambdaFUNCTIONARN>

Check Lambda Log

Let's make sure everything is working as expected before setting it on cruise mode. Check cloudwatch logs for the lambda for the cloudwatch subscription filter. If you don't see any error logs, it's all set!

If you are not certain that the logs will be delivered as expected, there are couple way to get notification from abnormal errors.

Create another lambda to detect error from logs to trigger SNS notification. Refer to the post for detailed setup

https://aws.amazon.com/ko/blogs/mt/get-notified-specific-lambda-function-error-patterns-using-cloudwatch/
Setup AWS DevOps Guru with SNS notification. This ML powered devops support service will notify you when there are error on your lambda function automatically. (It's really useful!)

Setting up Amazon DevOps Guru

Data in BigQuery

After a while if everything is working as expected, you will see streams of data in your table.

It's time to make some visualization with the data we have. I will make a heat map of user request from the data. First using aggregation, create a query to get count of each valid remote addr

SELECT remote_addr ip, count(*) c
FROM `<project_id>.<dataset>.<table>` 
WHERE DATE(timestamp) = "2021-01-29" and remote_addr not like "-" 
group by 1

Then join this info with geolite2 table. (Since this query uses huge geolite2 table, it will incur cost.)

WITH source_of_ip_addresses AS (

    SELECT remote_addr ip, count(*) c
    FROM `hangfive-26bb4.cloudwatch.hangfive-nginx` 
    WHERE DATE(timestamp) = "2021-01-29" and remote_addr not  like "-" 
    group by 1
)
SELECT city_name, SUM(c) c, ST_GeogPoint(AVG(longitude), AVG(latitude)) point
FROM (
  SELECT ip, city_name, c, latitude, longitude, geoname_id
  FROM (
    SELECT *, NET.SAFE_IP_FROM_STRING(ip) & NET.IP_NET_MASK(4, mask) network_bin
    FROM source_of_ip_addresses, UNNEST(GENERATE_ARRAY(9,32)) mask
    WHERE BYTE_LENGTH(NET.SAFE_IP_FROM_STRING(ip)) = 4
  )
  JOIN `fh-bigquery.geocode.201806_geolite2_city_ipv4_locs`  
  USING (network_bin, mask)
)
WHERE city_name  IS NOT null
GROUP BY city_name, geoname_id
ORDER BY c DESC
LIMIT 5000

This query will result in count of user in each city.

Visualize

Since we don't have support from Kibana anymore, I will visualize the data with Data Studio.

Data studio

visualize with data studio or your favorite BI tool.

Conclusion

We have looked bunch of different services to get here. At the core, as long as you know how to write a lambda code, you can stream and ingest app logs to anywhere regardless cloudplatform. It's different feeling knowing that you span across other cloud platform. I hope my blog helped you and thank you for reading.

Extra

Concurrency

After letting the lambda to work for a while, my devops guru setup actaully kicked in warning me of concurrency issues. If traffic spike, lambda is bound to be throttled without conccurency configuration.

{
    'AccountId': '1111111111',
    'Region': 'us-east-1',
    'MessageType': 'NEW_RECOMMENDATION',
    'InsightId': 'AGp3byn_PgRryjAItOHdAlcAAAAAAAAAAV8rYO_CKM4aynEOZWZiTd2_OV_0bsqS',
    'Recommendations': [{
        'Name': 'Troubleshoot errors and set up automatic retries in AWS Lambda',
        'Description': 'Your Lambda function is throwing a high number of errors. To learn about common Lambda errors, their causes, and mitigation strategies, see this link.',
        'Reason': 'The Errors metric in AWS::Lambda::FunctionName breached a high threshold. ',
        'Link': 'https://docs.aws.amazon.com/lambda/latest/dg/invocation-retries.html',
        'RelatedEvents': [],
        'RelatedAnomalies': [{
            'SourceDetails': {
                'CloudWatchMetrics': [{
                    'MetricName': 'Errors',
                    'Namespace': 'AWS::Lambda::FunctionName'
                }]
            },
            'Resources': [{
                'Name': 'CloudwatchToBQ',
                'Type': 'AWS::Lambda::FunctionName'
            }]
        }]
    }, {
        'Name': 'Configure provisioned concurrency for AWS Lambda',
        'Description': 'Your Lambda function is having trouble scaling. To learn how to enable provisioned concurrency, which allows your function to scale without fluctuations in latency, see this link.',
        'Reason': 'The Duration metric in AWS::Lambda::FunctionName breached a high threshold. ',
        'Link': 'https://docs.aws.amazon.com/lambda/latest/dg/configuration-concurrency.html#configuration-concurrency-provisioned',
        'RelatedEvents': [],
        'RelatedAnomalies': [{
            'SourceDetails': {
                'CloudWatchMetrics': [{
                    'MetricName': 'Duration',
                    'Namespace': 'AWS::Lambda::FunctionName'
                }]
            },
            'Resources': [{
                'Name': 'CloudwatchToBQ',
                'Type': 'AWS::Lambda::FunctionName'
            }]
        }]
    }]
}

Managing concurrency for a Lambda function

For remedial action I have inspected lambda metrics and updated reserved concurrencies.

you should inspect your lambda metrics and update concurrency accordingly. Also if this is not enough, I recommend you to add pub/sub or queue service such as SQS to handle the traffic.

Streaming Quota

I need to remind you that Google BigQuery also has streaming quota. (It's 500000 rows per second)

Quotas and limits | BigQuery | Google Cloud

Reference

Streaming data into BigQuery | Google Cloud

Streaming insert | BigQuery | Google Cloud

google-cloud-bigquery

google-auth

Geolocation with BigQuery: De-identify 76 million IP addresses in 20 seconds | Google Cloud Blog

Export AWS/GCP cost and billing data to BigQuery for analytics

Jinwook Baek — Thu, 31 Dec 2020 01:33:42 +0000

Original post has better layout with images.

Intro

I am currently using both GCP and AWS for a certain project. While each cloud providers provides very nice tool and visualizations for their own spending (Cloud billing for GCP and Cost&Usage report for AWS respectively), I wanted to consolidate both cloud providers usage report and visualize using single BI tool. The report does not need to be realtime, I only need daily granularity on my report. (No streaming)

So I had to spin up something and these were couple options

House data to AWS using Athena+Quicksight(or other BI tool)
House data to GCP using Bigquery+datastudio(or other BI tool)
Spend more time searching for 3rd party tools

I am a big fan of BigQuery and I prefer to to ELT rather than ETL on BQ. Moreover, I was already using BigQuery for the project with holistics.io, I chose to house usage and cost data to bigquery.

Move data to BQ

GCP Cloud Billing

Setup

This is easy, cloud billing natively support bigquery export.

Enable billing for the GCP project.

I assume that you already have billing account setup. (You wouldn't need data analytics unless you are spending any moeny) But if you need to enable billing for the project, refer to the following link.

Modify a project's billing settings | Cloud Billing | Google Cloud
Create a dataset in BigQuery whthin the GCP project you desire - I named it billing_export

Creating datasets | BigQuery | Google Cloud
You need appropriate permissions to setup export
- Billing account Admin
- Bigquery Admin for the project
- resourcemanager.projects.update permission

Enable Cloud Billing export to BigQuery dataset
- Go to Billing menu in Navigation menu (https://console.cloud.google.com/billing/)
- Go to linked billing account (this should be already set, if not you need to add payment info)
- Select Billing export → Select BigQuery export tab
- Select edit setting on Daily cost detail
  - Select the project and dataset where you want the data to be sinked. (note) The BigQuery API is required to export data to BigQuery. If the project you selected doesn't have the BigQuery API enabled, you will be prompted to enable it. Click Enable BigQuery API and the API will be enabled for you

Once everything is set, you will be able to see your screen like this.

When you first enable the daily cost detail export to BigQuery, it might take a few hours to start seeing your Google Cloud cost data. Table with gcp_billing_export_v1_<some_hash> will be generated. Table will be automatically partitioned by day. (you can query partitioned data to save cost)

Schema

After a while you will notice a being exported to a table. You can click table and click preview or query.



SELECT usage_start_time, location.region, cost, currency, usage.amount, usage.unit, credits
FROM `<project>.billing_export.gcp_billing_export_v1_<hash>`
WHERE _PARTITIONTIME >= "2020-12-01 00:00:00"

If you need specific information for each column, refer to this link. It will help you understand each columns of data.

Understanding the Cloud Billing data tables in BigQuery | Google Cloud

Special note on credits nested field if you are using free tier.

That's it for GCP! now to the hard part.

AWS Cost & Usage Report

I didn't want to reinvent the wheel, therefore as all the good developers do, I googled. I found couple of great articles regarding this subject, but they seemed to be complicated and outdated. Luckly, there were already a natively supported S3 transfer service in BigQuery. It supports transfer csv and parquet.

seems too complicated.

Amazon S3 transfers | BigQuery Data Transfer Service | Google Cloud

Let's save the cost and usage report to S3 first. Go to AWS Cost and Usage Reports. You can create a report here.

aws usage and cost reports

Report name - daily_report_gzip
Include resource IDs - check
Data refresh settings - uncheck
S3 bucket - create a bucket or select a bucket
prefix - billing/daily_report_gzip
Time granularity - Daily
Report versioning - overwrite
Compression type - gzip

After creating report, it takes couple hours to show up on S3. Once report is created, you will see gz file exported under following URI.

s3://<bucket>/billing/daily_report_gzip/20201201-20210101/

data - daily_report_gzip-00001.csv.gz
metadata - daily_report_gzip-Manifest.json

Schema

We need to know the schema in order to house the data in BigQuery table. If you want to know what each fields represents, consult the document.

Data dictionary

When you open up the daily_report_gzip-Manifest.json file, you will discover how each columns are structued. Take a note on variety of types - string, OptionalBigDecimal, DateInterval and so on.

Json - manifest file example

Converting manifest to bq table schema

In order to transfer the data, the BQ table needs to have correct schema to house the csv data. BQ have different types and there are too many fields to just copy and paste. You can configure table chema with Edit as text. (ex: Field1:TYPE, Field2:TYPE) Let's use jq to extract types as bq table schema.

I also need to replace some fields type to BigQuery compatible types.

BigDecimal, OptionalBigDecimal → bignumeric
Interval → string
datetime → timestamp
OptionalString → string

Download the json file and pipe with following jq command



cat daily_report_gzip-Manifest.json | jq -jr '.columns[] | .category+"_"+.name+":"+ .type +","' | sed s/.$// | sed 's/OptionalString/String/g' | sed 's/\<Interval\>/String/g' | sed 's/OptionalBigDecimal/BigNumeric/g' | sed 's/BigDecimal/BigNumeric/g' | sed 's/DateTime/Timestamp/g'

jq -jr '.columns[] | .category+"_"+.name+":"+ .type +","'
- this will extract each items in array and struct filed as <category>_<name>:<type>. I needed category prepended because there are duplicate names
- -r for raw output
sed s/.$//
- remove last ,
sed 's/<original type>/<bq type>/g'
- replace type to bq compatible types

This command will result in following texts



identity_LineItemId:String,identity_TimeInterval:String,bill_InvoiceId:String,bill_BillingEntity:String,bill_BillType:String,bill_PayerAccountId:String,bill_BillingPeriodStartDate:Timestamp,bill_BillingPeriodEndDate:Timestamp,lineItem_UsageAccountId:String,lineItem_LineItemType:String,lineItem_UsageStartDate:Timestamp,lineItem_UsageEndDate:Timestamp,lineItem_ProductCode:String,lineItem_UsageType:String,lineItem_Operation:String,lineItem_AvailabilityZone:String,lineItem_ResourceId:String,lineItem_UsageAmount:BigNumeric,lineItem_NormalizationFactor:BigNumeric,lineItem_NormalizedUsageAmount:BigNumeric,lineItem_CurrencyCode:String,lineItem_UnblendedRate:String,lineItem_UnblendedCost:BigNumeric,lineItem_BlendedRate:String,lineItem_BlendedCost:BigNumeric,lineItem_LineItemDescription:String,lineItem_TaxType:String,lineItem_LegalEntity:String,product_ProductName:String,product_accountAssistance:String,product_alarmType:String,product_architecturalReview:String,product_architectureSupport:String,product_availability:String,product_baseProductReferenceCode:String,product_bestPractices:String,product_cacheEngine:String,product_capacitystatus:String,product_caseSeverityresponseTimes:String,product_clientLocation:String,product_clockSpeed:String,product_computeFamily:String,product_computeType:String,product_cputype:String,product_currentGeneration:String,product_customerServiceAndCommunities:String,product_databaseEngine:String,product_deploymentOption:String,product_description:String,product_dominantnondominant:String,product_durability:String,product_ecu:String,product_endpointType:String,product_engineCode:String,product_enhancedNetworkingSupported:String,product_eventType:String,product_feeDescription:String,product_fromLocation:String,product_fromLocationType:String,product_group:String,product_groupDescription:String,product_includedServices:String,product_insightstype:String,product_instanceFamily:String,product_instanceType:String,product_instanceTypeFamily:String,product_intelAvx2Available:String,product_intelAvxAvailable:String,product_intelTurboAvailable:String,product_launchSupport:String,product_licenseModel:String,product_location:String,product_locationType:String,product_logsDestination:String,product_maxIopsBurstPerformance:String,product_maxIopsvolume:String,product_maxThroughputvolume:String,product_maxVolumeSize:String,product_memory:String,product_memoryGib:String,product_memorytype:String,product_messageDeliveryFrequency:String,product_messageDeliveryOrder:String,product_minVolumeSize:String,product_networkPerformance:String,product_normalizationSizeFactor:String,product_operatingSystem:String,product_operation:String,product_operationsSupport:String,product_physicalProcessor:String,product_preInstalledSw:String,product_pricingUnit:String,product_proactiveGuidance:String,product_processorArchitecture:String,product_processorFeatures:String,product_productFamily:String,product_programmaticCaseManagement:String,product_queueType:String,product_region:String,product_requestDescription:String,product_requestType:String,product_resourcePriceGroup:String,product_routingTarget:String,product_routingType:String,product_servicecode:String,product_servicename:String,product_sku:String,product_storage:String,product_storageClass:String,product_storageMedia:String,product_storageType:String,product_technicalSupport:String,product_tenancy:String,product_thirdpartySoftwareSupport:String,product_tiertype:String,product_toLocation:String,product_toLocationType:String,product_trafficDirection:String,product_training:String,product_transferType:String,product_usagetype:String,product_vcpu:String,product_version:String,product_volumeApiName:String,product_volumeType:String,product_whoCanOpenCases:String,pricing_LeaseContractLength:String,pricing_OfferingClass:String,pricing_PurchaseOption:String,pricing_RateId:String,pricing_currency:String,pricing_publicOnDemandCost:BigNumeric,pricing_publicOnDemandRate:String,pricing_term:String,pricing_unit:String,reservation_AmortizedUpfrontCostForUsage:BigNumeric,reservation_AmortizedUpfrontFeeForBillingPeriod:BigNumeric,reservation_EffectiveCost:BigNumeric,reservation_EndTime:String,reservation_ModificationStatus:String,reservation_NormalizedUnitsPerReservation:String,reservation_NumberOfReservations:String,reservation_RecurringFeeForUsage:BigNumeric,reservation_ReservationARN:String,reservation_StartTime:String,reservation_SubscriptionId:String,reservation_TotalReservedNormalizedUnits:String,reservation_TotalReservedUnits:String,reservation_UnitsPerReservation:String,reservation_UnusedAmortizedUpfrontFeeForBillingPeriod:BigNumeric,reservation_UnusedNormalizedUnitQuantity:BigNumeric,reservation_UnusedQuantity:BigNumeric,reservation_UnusedRecurringFee:BigNumeric,reservation_UpfrontValue:BigNumeric,savingsPlan_TotalCommitmentToDate:BigNumeric,savingsPlan_SavingsPlanARN:String,savingsPlan_SavingsPlanRate:BigNumeric,savingsPlan_UsedCommitment:BigNumeric,savingsPlan_SavingsPlanEffectiveCost:BigNumeric,savingsPlan_AmortizedUpfrontCommitmentForBillingPeriod:BigNumeric,savingsPlan_RecurringCommitmentForBillingPeriod:BigNumeric

Create BQ Table

Let's create a BQ table. You should have a dataset created already if you exported a Cloud billing. If not, let's create now.

Once dataset if created, click create table.

table from - Empty table
Destination - search for a project
Project name - project you created a dataset
Dataset name - select one you created
Talbe name - aws
For Schema, toggle Edit as text and paste the schema you generated with jq command
Partion - you can skip this or select a timestap filed that you will be commonly filter with. I chose billingPeriodStartData and partition by day.

Transfer Prereq

Once the table is ready, let's create a transfer.

You need to have following permissions first.

bigquery.transfers.update
bigquery.datasets.update

These permissions are includede in predefiend bigquery.admin role.

AWS credential

Create IAM for AWS credential for transfer. You only need AmazonS3ReadOnlyAccess permission on specific bucket you created, but if you are lazy, just use AmazonS3ReadOnlyAccess AWS managed policy on all resources. If you don't know how to create a IAM user, please refer to following link.

Amazon S3 - Creating an IAM user

Configure Transfer

On bigquery console, click transfer menu on left pane. You will be redirected to a transfer configuration page. Click + Create Transfer button.

Source - Amazon S3
Display Name - aws-billing-export
Schedule options - starts now, daily
Destination
- Dataset ID - dataset you created
- Destination table - aws
- Amazon S3 URI - s3://<bucket>/billing/daily_report_gzip/*/*.gz
- Access key ID
- Secert access key
- File format - csv
  - Bq support gzip natively
- Transfer option
  - number of errors allowed - 0
  - ignore unknown values - check
  - field delimieter - ,
  - Header rows to skip - 1
- Notification
  - you can create new pub/sub and configure notification here. You can configure this later. So leave it blank for now.
Click Save!

You willing following green check mark if the transfer succeeds.

If you see error with red marker, exame the error logs on right pane. Common erros consist,

Wrong S3 URI
Wrong AWs credential
Wrong table schema
No objects

Cleaning Data

You have two tables populated with billing data! It's time do some analytics and visualizations.

Let's review fields

AWS

Data dictionary

GCP

Understanding the Cloud Billing data tables in BigQuery | Google Cloud

You can use BQ console to query and testout data for integrity to makesure the load is intact. I won't go into details with sql query.

Once Extracted and Loaded, I do Transform on Holistics.io leaving the data intact, but you can also create new view to minimize, clean and consolidate the data into single table or a view using scheduled query feature of BigQuery.

Visualize data

After inspecting the with couple queries, and I am just a newbie in sql world, I use Data Studio to inspect the data.

When you use BigQuery, You can connect data to Data Studio without extra configuration.

You can use other BI tool to connect BQ to visualize and analyze you data from here.

Extra

[ ] Check data integrity with daily and monthly roll over

Conclusion

As BQ support S3 transfer, it was a lot easier for me to move data around but still there were some work to be done.

Please let me know if you find more concrete solution to consolidate billing data in single place!

Thank you for reading here are some pictures of Kokos!

Reference

Export Cloud Billing data to BigQuery | Google Cloud

Overview of Amazon S3 transfers | BigQuery Data Transfer Service

Using runtime parameters in transfers | BigQuery Data Transfer Service

AWS CDK - use your favorite language to define cloud infrastructure

Jinwook Baek — Thu, 12 Nov 2020 04:35:18 +0000

Introduction

Infrastructure as code(IaC) is a popular term in cloud environment. I have been using CloudFormation to automate AWS infrastructure deployment. However I always had hard time writing out the configuration file since they are written in string json or yaml file. There are some best practice for writing out CloudFormation out there, I always have to validate and revalidate the stacks and had to double check whether the infrastructure was deployed as I intended to. Recently I have stumbled upon AWS CDK and I fell in love with it. Here are some plus using CDK.

Organize your project into logical modules
Use logic (if statements, for-loops, etc) when defining your infrastructure
Use object-oriented techniques to create a model of your system
Type-safety, code-completion, and open-source
Define high level abstractions, share them, and publish them to your team, company, or community
Share and reuse your infrastructure as a library
Code completion within your IDE → This was huge plug for me!

In short, AWS CDK provides high level abstraction of CloudFormation json/yaml syntax. In this blog post, I will rewrite my previous ecs-sample architecture(beta stage) in CDK. Please refer to my previous blog post

How to deploy django app to ECS Fargate Part2

Jinwook Baek ・ Jul 3 '20

#aws #docker #tutorial #webdev

for infrastructure that I will be deploying with CDK.

kokospapa8/ecs-fargate-sample-app

Before diving into writing CDK code, let's take a quick look at CDK.

CDK Concepts

There are three main components in CDK: contruct, stack and app. Everything in the AWS CDK is a construct. You can think of constructs as cloud components that can represent architectures of any complexity: a single resource, such as an S3 bucket or an SNS topic, a static website, or even a complex, multi-stack application that spans multiple AWS accounts and regions. To foster reusability, constructs can include other constructs. You compose constructs together into stacks, that you can deploy into an AWS environment, and apps, a collection of one of more stacks.

App

Written in TypeScript, JavaScript, Python, Java, or C# that uses the AWS CDK to define AWS infrastructure

Stack

Equivalent to AWS CloudFormation stacks
Contains construct
Defines one or more concrete AWS resources

Constructs

Represented as types in programming.
Three fundamental flavors
- L1 (AWS CloudFormaiton only) - directly to resource types defined by AWS CloudFormation
  - Always have names that begin with Cfn
- L2 - Encapsulate L1 modules, providing sensible defaults and best-practice security policies, supporting resources needed by the primary resource
- L3 - Patterns declare multiple resources to create entire AWS architectures for particular use cases. All the plumbing is already hooked up, and configuration is boiled down to a few important parameters

Core module

Imported into code as core or cdk
Contains constructs used by the AWS CDK itself as well as base classes for constructs, apps, resources, and other AWS CDK objects.

Supported Languages

AWS CDK is developed in one language (TypeScript) and language bindings are generated for the other languages through the use of a tool called JSII. In this post, I will be using python .

TypeScript
JavaScript
Python
Java
C#

Getting started CDK

Setup AWS Credential

aws configure
cat ~/.aws/credentials
---
[default]
aws_access_key_id=AKIAI44QH8XXXXXXXX
aws_secret_access_key=je7MtGbClwBFXXXXXXXXXXXXXXXXXXXXX

Install on Mac OSX

Let's use brew 🍺 to install on Mac OSX.

brew update
brew doctor
brew install aws-cdk
cdk --version

If you are using other OS, please refer to following link.

Getting started with the AWS CDK

Python environment

you need to have python 3.6 or later including pip and virtualenv .

Pycharm for IDE

I use pycharm for python development with AWS toolkit, I strongly recommend use IDE instead of good old vim or common text editors. Especially when you are devloping CDK, you don't need to go back and forth digging through huge list of API references.

AWS Toolkit for PyCharm

Alternatively you can use VS Code with AWS toolkit.

AWS Toolkit for Visual Studio Code

Project Setup

git clone https://github.com/kokospapa8/ecs-fargate-sample-app.git
cd config/cdk
source .env/bin/activate

pip install -r requirements.txt
# set env with CDK_DEFAULT_ACCOUNT
export CDK_DEFAULT_ACCOUNT=123456789
# set env with CDK_DEFAULT_REGION
export CDK_DEFAULT_REGION=us-east-1

cdk synth

(Alternatively) setup sample app instead of pulling github project

cdk init -a sample-app --language python

cdk synth will generate cloudformation stack file in cdk.out folder. You can just take this CloudFormation files and deploy manually. But there is a easier way to deploy with CDK. We will get into that on later part of the post

App structure

Before deploying the stack, let's review the infrastructure and review stacks one by one.

We will build following resources

VPC with public and private subnet
API EC2 instance on ASG (in public subnet)
Worker EC2 instance on ASG (in public subnet)
ALB in front of API ASG
RDS mysql (in private subnet)
ElastiCache - Redis (private subnet)

Now let's look at the file structures.

app.py - application wrapper

requirements.txt - required CDK pip libraries

ecs_sample_cdk folder consists of following stack files. sample_stack is wrapper for the whole infrastructure; alb_stack , rds_stack , redis_stack and vpc_stack are nested stack of sample_stack

Deep-dive into codes

app.py

Nothing special here. If you want to know what's going on behind the scene please refer to following document.

Apps

#!/usr/bin/env python3
import os

from aws_cdk import core
from ecs_sample_cdk.sample_stack import SampleStack

env = core.Environment(
    account=os.environ.get("CDK_DEFAULT_ACCOUNT", ""),
    region=os.environ.get("CDK_DEFAULT_REGION", "")
)

app = core.App()
SampleStack(app, "ecs-sample", env=env)

app.synth()

Sample Stack

This is a class that hold other nested stack and I have used props to pass around important resource reference such as vpc and security groups. You can also notice add_dependency method which enforces dependency between different nested stacks. Here we create VPC, ALB stacks then create RDS and Elasticache because we need vpc_id, subnets and security groups in order create these resources.

class SampleStack(core.Stack):

    def __init__(self, scope: core.Construct, id: str, env, **kwargs) -> None:
        super().__init__(scope, id, **kwargs)
        props = {'namespace': 'sample'}
        vpc_stack = VPCStack(self, f"{id}-vpc", env=env, props=props)
        props.update(vpc_stack.output_props)

        alb_stack = ALBStack(self, f"{id}-alb", env=env, props=props)
        alb_stack.add_dependency(vpc_stack)
        props.update(alb_stack.output_props)

        rds_stack = RDSStack(self, f"{id}-rds", env=env, props=props)
        rds_stack.add_dependency(vpc_stack)
        rds_stack.add_dependency(alb_stack)
        props.update(rds_stack.output_props)

        redis_stack = RedisStack(self, f"{id}-redis", env=env, props=props)
        redis_stack.add_dependency(vpc_stack)
        redis_stack.add_dependency(alb_stack)

VPC_STACK

This code produces a VPC with 2 public and private subnets each in 2 different AZs with single NAT gateway. Using L2 construct, less than 10 lines of code produces route tables, subnets, NAT gateway, IGW, VPC. It is that simple to produce produce a valid VPC for app development with less than 10 lines of codes.

Notice that I have added CfnOutput for cloudformation output for vpc id and output_props to pass around vpc resource I have created in this stack

class VPCStack(core.NestedStack):

    def __init__(self, scope: core.Construct, id: str, env, props, **kwargs) -> None:
        super().__init__(scope, id, **kwargs)

        subnets = []

        public_subnet = ec2.SubnetConfiguration(
                           cidr_mask=24,
                           name=f"{id}-public",
                           subnet_type=ec2.SubnetType.PUBLIC
                       )

        private_subnet = ec2.SubnetConfiguration(
                           cidr_mask=24,
                           name=f"{id}-private",
                           subnet_type=ec2.SubnetType.PRIVATE
                       )
        subnets.append(public_subnet)
        subnets.append(private_subnet)

        # The code that defines your stack goes here
        vpc = ec2.Vpc(self, f"{id}",
                           cidr="172.0.0.0/16",
                           enable_dns_hostnames=True,
                           enable_dns_support=True,
                           nat_gateways=1,
                           nat_gateway_provider=ec2.NatProvider.gateway(),
                           max_azs=2,
                           subnet_configuration=subnets
                           )
        #Be aware that environment-agnostic stacks will be created with access to only 2 AZs, so to use more than 2 AZs, be sure to specify the account and region on your stack

        core.CfnOutput(self, "vpcid",
                       value=vpc.vpc_id)

        # Prepares output attributes to be passed into other stacks
        # In this case, it is our VPC and subnets.
        self.output_props = props.copy()
        self.output_props['vpc'] = vpc
        self.output_props['public_subnets'] = vpc.public_subnets
        self.output_props['private_subnets'] = vpc.private_subnets

        @property
    def outputs(self):
        return self.output_props

ALB_stack

AutoScalingGroup requires KEY_PAIR_NAME which you need to create on Console manually.

Amazon EC2 key pairs and Linux instances

The stack creates following resources

EC2 instance role
API AutoScalingGroup
Worker AutoScalingGroup
Security groups
- api EC2 (public subnet)
  - allows ssh from public
- worker EC2 (private subnet)
  - allows ssh from API EC2
- redis
  - allows 6379 from EC2 security group
- rds
  - allows 3306 from EC2 security group

We need add security group on props to use them on RDS and Redis stack

class ALBStack(core.NestedStack):

    def __init__(self, scope: core.Construct, id: str, env, props, **kwargs) -> None:
        super().__init__(scope, id, **kwargs)

        #create ec2role
        #get from env or create
        role = iam.Role(self, "ecs-sample-ec2-role",
                        assumed_by=iam.ServicePrincipal('ec2.amazonaws.com'),
                        )
        role.add_managed_policy(
            iam.ManagedPolicy.from_aws_managed_policy_name("AmazonEC2ContainerRegistryPowerUser")
        )

        asg_api = autoscaling.AutoScalingGroup(
            self,
            "ecs-sample-api-asg",
            vpc=props['vpc'],
            instance_type=ec2.InstanceType.of(
                ec2.InstanceClass.BURSTABLE3, ec2.InstanceSize.MICRO
            ),
            machine_image=ec2.AmazonLinuxImage(),
            key_name=KEY_PAIR_NAME,
            vpc_subnets=ec2.SubnetSelection(subnet_type=SubnetType.PUBLIC),
            desired_capacity=1,
            max_capacity=1,
            min_capacity=1,
            role=role
            # userdata=userdata

        )

        asg_worker = autoscaling.AutoScalingGroup(
            self,
            "ecs-sample-worker-asg",
            vpc=props['vpc'],
            instance_type=ec2.InstanceType.of(
                ec2.InstanceClass.BURSTABLE3, ec2.InstanceSize.MICRO
            ),
            machine_image=ec2.AmazonLinuxImage(),
            key_name=f"ecs-sample-key",
            vpc_subnets=ec2.SubnetSelection(subnet_type=SubnetType.PRIVATE),
            desired_capacity=1,
            max_capacity=1,
            min_capacity=1,
            role=role
            # userdata=userdata

        )

        # Creates a security group for our application
        sg_api = ec2.SecurityGroup(
                self,
                id="ecs-sample-ec2-api",
                vpc=props['vpc'],
                security_group_name="ecs-sample-ec2-api"
        )
        sg_worker = ec2.SecurityGroup(
                self,
                id="ecs-sample-ec2-worker",
                vpc=props['vpc'],
                security_group_name="ecs-sample-ec2-worker"
        )

        # to access this security group for SSH
        sg_api.add_ingress_rule(
            peer=ec2.Peer.ipv4(SSH_IP),
            connection=ec2.Port.tcp(22)
        )

        # use ec2 api as bastion
        sg_worker.connections.allow_from(
                sg_api, ec2.Port.tcp(22), "Allow from ec2 api")

        asg_api.add_security_group(sg_api)
        asg_worker.add_security_group(sg_worker)

        # Creates a security group for the application load balancer
        sg_alb = ec2.SecurityGroup(
                self,
                id="ecs-sample-loadbalancer",
                vpc=props['vpc'],
                security_group_name="ecs-sample-loadbalancer"
        )

        sg_api.connections.allow_from(
                sg_alb, ec2.Port.tcp(80), "Ingress")

        # Creates an application load balance
        lb = elbv2.ApplicationLoadBalancer(
                self,
                f"{id}-ALB",
                vpc=props['vpc'],
                security_group=sg_alb,
                internet_facing=True)

        listener = lb.add_listener("Listener", port=80)
        # Adds the autoscaling group's (asg_api) instance to be registered
        # as targets on port 8080
        listener.add_targets("Target", port=80, targets=[asg_api])
        # This creates a "0.0.0.0/0" rule to allow every one to access the
        # application
        listener.connections.allow_default_port_from_any_ipv4(
                "Open to the world"
                )

        # create RDS sg
        sg_rds = ec2.SecurityGroup(
                self,
                id="ecs-sample-mysql",
                vpc=props['vpc'],
                security_group_name="ecs-sample-mysql"
        )
        sg_api.connections.allow_from(
                sg_rds, ec2.Port.tcp(3306), "allow from rds to ec2 api")
        sg_rds.connections.allow_from(
                sg_api, ec2.Port.tcp(3306), "allow from ec2 api to rds")
        sg_worker.connections.allow_from(
                sg_rds, ec2.Port.tcp(3306), "allow from rds to ec2 worker")
        sg_rds.connections.allow_from(
                sg_worker, ec2.Port.tcp(3306), "allow from ec2 worker to rds")

        # create Redis SG
        sg_redis = ec2.SecurityGroup(
                self,
                id="ecs-sample-redis",
                vpc=props['vpc'],
                security_group_name="ecs-sample-redis"
        )
        sg_api.connections.allow_from(
                sg_rds, ec2.Port.tcp(6379), "allow from redis to ec2 api")
        sg_rds.connections.allow_from(
                sg_api, ec2.Port.tcp(6379), "allow from ec2 api to redis")
        sg_worker.connections.allow_from(
                sg_rds, ec2.Port.tcp(6379), "allow from redis to ec2 worker")
        sg_rds.connections.allow_from(
                sg_worker, ec2.Port.tcp(6379), "allow from ec2 worker to redis")

RDS stack

You can either choose cluster mode or instance mode by passing cluster parameter on the stack argument
You cannot pass around plaintext credentials so CDK enforces you to use Credential This code actually produces password in SecertManager (These are L2 contruct that CDK enforces for best practice)

class Credentials · AWS CDK

class RDSStack(core.NestedStack):

    def __init__(self, scope: core.Construct, id: str, env, props, cluster=False, **kwargs) -> None:
        super().__init__(scope, id, **kwargs)

        #TEMP without ASG
        # security_groups = [ec2.SecurityGroup(
        #         self,
        #         id="ecs-sample-mysql",
        #         vpc=props['vpc'],
        #         security_group_name="ecs-sample-mysql"
        # )]

        vpc = props['vpc']
        security_groups=[props['sg_rds']]
        credential = rds.Credentials.from_username(username="admin")
        private_subnet_selections = ec2.SubnetSelection(subnet_type=ec2.SubnetType.PRIVATE)
        subnet_group = rds.SubnetGroup(self, "sample-rds-subnet-group",
                                       vpc=vpc,
                                       subnet_group_name="sample-rds-subnet-group",
                                       vpc_subnets=private_subnet_selections,
                                       description="sample-rds-subnet-group")
        self.output_props = props.copy()

        if not cluster:
            rds_instance = rds.DatabaseInstance(
                self, "RDS-instance",
                database_name="sample",
                engine=rds.DatabaseInstanceEngine.mysql(
                    version=rds.MysqlEngineVersion.VER_8_0_16
                ),
                credentials=credential,
                instance_identifier="ecs-sample-db",

                vpc=vpc,
                port=3306,
                instance_type=ec2.InstanceType.of(
                    ec2.InstanceClass.BURSTABLE3,
                    ec2.InstanceSize.MICRO,
                ),
                subnet_group=subnet_group,
                vpc_subnets=private_subnet_selections,
                removal_policy=core.RemovalPolicy.DESTROY,
                deletion_protection=False,
                security_groups=security_groups

            )
            core.CfnOutput(self, "RDS_instnace_endpoint", value=rds_instance.db_instance_endpoint_address)
            self.output_props['rds'] = rds_instance
            else:
            instance_props = rds.InstanceProps(
                vpc=vpc,
                security_groups=security_groups,
                vpc_subnets=private_subnet_selections
            )
            rds_cluster = rds.DatabaseCluster(
                self, "RDS-cluster",
                cluster_identifier="ecs-sample-db-cluster",
                instance_props=instance_props,
                engine=rds.DatabaseClusterEngine.aurora_mysql(
                    version=rds.AuroraMysqlEngineVersion.VER_2_07_1
                ),
                credentials=credential,
                default_database_name="sample",
                instances=1,
                subnet_group=subnet_group,
                removal_policy=core.RemovalPolicy.DESTROY,
                deletion_protection=False
            )
            core.CfnOutput(self, "RDS_cluster_endpoint", value=rds_cluster.cluster_endpoint.hostname)
            self.output_props['rds'] = rds_cluster

Redis Stack

It seems like Redis ReplicationGroup only supports L1 construct currently. There for I have used CfnReplicationGroup in order to create Elasticache for Redis.

redis = cache.CfnReplicationGroup(self,
                                              f"{id}-replication-group",
                                              replication_group_description=f"{id}-replication group",
                                              cache_node_type="cache.t3.micro",
                                              cache_parameter_group_name=cache_parameter_group_name,
                                              security_group_ids=[sg_redis.security_group_id],
                                              cache_subnet_group_name=subnets_group.cache_subnet_group_name,
                                              engine="redis",
                                              engine_version="5.0.4",
                                              # node_group_configuration
                                              num_node_groups=1, #shard
                                              replicas_per_node_group=1 #one replica
                                              )
        redis.add_depends_on(subnets_group)

Deploy

Let's deploy by typing two commands.

Bootstrap

This command will create a bucket for CDK to stage cloudformation files.

cdk bootstrap
#check cloudformation for CDKtoolkit

Deploy

cdk deploy

You will be able to monitor nested stacks being deployed on shell. You can also monitor detailed progress on CloudFormation console.

If you get any errors, go to CloudFormation stack on console to review events tab to look for reason for failure.

Destroy

once you have finished working with the sample you can easily destroy with single command.

cdk destroy

or just delete the stack in CloudFormation console.

Takeaway

I have reduced 900 lines of json code into python code with less than 500 lines of code. Even more, they are modular and easy to revise whenever necessary. AWS CDK enables you to boost your productivity on IaC. If you have been using CloudFormation, you should definitely try out AWS CDK.

Take advantage of IDE!

Thank you for reading!

References

AWS Cloud Development Kit (CDK) - TypeScript and Python are Now Generally Available | Amazon Web Services

Examples

aws-samples/aws-cdk-examples

aws-samples/startup-kit-templates

cdkworkshop.com

AWS CDK · AWS CDK Reference Documentation

AWS CDK for Beginners

Source

This article was originally posted on here

Migrate python async worker to asynchrounous Lambda

Jinwook Baek — Sat, 10 Oct 2020 06:44:56 +0000

Original blog post

Intro

When you build a web service, there is a time that you need to implement simple async worker for background process. I am big fan of Django and whenever I need a async worker, I used celery or rq for convenience. celery or other async framework provides a lot of cool features but most of the time all I need is a simple background process that would not interrupt user experiences.

Overtime, as paradigm shifts to cloud, I have been thinking lambda as new async worker solution due to deployment complexity, scaling and cost constraint. Here are pros and cons for using lambda instead of celery on cloud environment(non-local environment).

Advantage

FasS advantage
- scaling benefit - cost, ops, etc
- Resiliency
- faster development
No need to maintain Broker (e.g elastic search redis)
Smaller side effect on code change
Fully-managed

Disadvantage

FasS disadvantage
- limited state
Hard to reuse app code
- Django ORM or settings module
- You can still use them with django package installed but inefficient
- can't reuse code from django app
- Django package is too heavy to use on lambda (still usable but too complex to setup
Latency on startup
Json formatted Invocation parameter
- If you need to pass binary or big size paras, need to consider S3 or other medium

Limitation

Most of early stage limitation are lifted as lambda is in mature state (e.g conccurent executions, layers, etc)

AWS Lambda quotas

timeout - 15min
- If your async worker take more than 5min, you need to break up codes in to multiple lambda functions or use other distributed system such as EMR, etc.
Invocation payload
- asynchronous - 256KB

Consideration

VPC, Subnet, Security Group
- If you are accessing Resources in private VPC, you need to consider placing lambda in same subnet.
- If you are lambda needs to talk to outside world, private subnets needs NAT gateway
Maintenance

Setup

AWS recently came up with SAM(Serverless Application Model) which utilize cloudformation for easier deployment of lambda application. SAM provides functionality such as API gateway and statemachine, but we will only be using AWS::Serverless::Function resource for our purpose. You can consider using Zappa for other option.

Github

refer to this repo for details

kokospapa8/async-lambda-sample-app

Prerequisite

You need following AWS resource created in order to invoke asynchronous lambda.

DLQ

In order use asynchronous lambda, you need to provide DLQ in case lambda fails and retry. Click link for details. I will create SNS topic in this post.

Asynchronous invocation

SECRET

Let's assume that you have mysql db in public VPC or s3 and need to access them in order to process some data, you can add secrets to lambda env, but it is not a good idea to put secret in plaintext for env variable. You can use aws secret manger , ssm parameterstore or vault . I will use ssm parameterstore for this post. Refer to previous post for creating secrets.(parameter store section)

How to deploy django app to ECS Fargate part3

arn:aws:ssm:<region>:<account_id>:parameter/SAMPLE/BUCKET

IAM ROLE

You need lambda execution role with following permission and polcies attached.



Policies
- AWSLambdaExecute
Permission 
- SNS – sns:Publish
- kms - kms:Decrypt
- ssm - ssm:GetParameters, ssm:DescribeParameters, ssm:GetParameter
You need following permission if you want to place your lambda function in a designated VPC
- ec2 - ec2:CreateNetworkInterface, ec2:DeleteNetworkInterface, ec2:DescribeSecurityGroups

SAM

What is the AWS Serverless Application Model (AWS SAM)?

Install (on macOS)

Installing the AWS SAM CLI on macOS



$ aws configure
AWS Access Key ID [None]: your_access_key_id
AWS Secret Access Key [None]: your_secret_access_key
Default region name [None]: 
Default output format [None]:

$ /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"
$ brew tap aws/tap
$ brew install aws-sam-cli

Development



$ sam init

This code will create following structure



sam-app/
   ├── README.md
   ├── events/
   │   └── event.json
   ├── hello_world/
   │   ├── __init__.py
   │   ├── app.py            #Contains your AWS Lambda handler logic.
   │   └── requirements.txt  #Contains any Python dependencies the application requires, used for sam build
   ├── template.yaml         #Contains the AWS SAM template defining your application's AWS resources.
   └── tests/
       └── unit/
           ├── __init__.py
           └── test_handler.py

template.yaml

Update template file if you need more info on syntax, refer to the link.

AWS::Serverless::Function

template.yaml ```

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
sample-app
Sample SAM Template for sample-app

More info about Globals: https://github.com/awslabs/serverless-application-model/blob/master/docs/globals.rst

Globals:
Function:
Timeout: 3

Resources:
SampleAppdFunction:
Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
Properties:
CodeUri: sample_app/
DeadLetterQueue:
Type: SNS
TargetArn: arn:aws:sns:::sample-dlq
Handler: app.lambda_handler
Runtime: python3.8
Description: sample lambda
EventInvokeConfig:
MaximumEventAgeInSeconds: 60
MaximumRetryAttempts: 2
FunctionName: SampleApp
Role: arn:aws:iam:::role/sample_lambda_execution_role
Environment:
Variables:
S3_URL: ""

Outputs:
SampleAppdFunction:
Description: "Sample app Function ARN"
Value: !GetAtt SampleAppdFunction.Arn

- requirements.txt

boto3

- app.py
This sample app receives image_url as parameter and puts the image on s3bucket name provided on s3 bucket. (AWSLambdaExecute policy has s3 access permission)

import boto3
import os
import requests

def lambda_handler(event, context):
image_url = event['image_url']
S3_BUCKET_PARAM = os.environ['S3_BUCKET_PARAM']

# download

image = requests.get(image_url).content


ssm_client = boto3.client('ssm', region_name="ap-northeast-2")
response = ssm_client.get_parameter(
    Name=S3_BUCKET_PARAM,
    WithDecryption=True
)
bucket_name = response['Parameter']['Value']

s3_client = boto3.client('s3', region_name="ap-northeast-2")
response = s3_client.put_object(
    Bucket=bucket_name,
    Key="image.png",
    Body=image
)

return response


## Build

Once your logic code is ready, you can build it using following command 

```yaml


sam build -t template.yaml --region <region_name>

You can test your function with following command



$ sam local invoke
# if you need to provide environment variable or paramters, use following command
$ sam local invoke -e events/event.json --env-vars env.json



#env.json
{
  "SampleAppdFunction": {
        "S3_BUCKET_PARAM_ARN": "parameterstore_arn",
  }
}
#event.json
{
  "image_url": "https://picsum.photos/200/300"
}

Deploy

Once your function is tested locally, let's deploy the function to cloud. You have two options. Enter appropriate response to prompt



$ sam deploy --guided

Configuring SAM deploy
======================

    Looking for samconfig.toml :  Found
    Reading default arguments  :  Success

    Setting default arguments for 'sam deploy'
    =========================================
    Stack Name [sample-app]: y
    AWS Region [ap-northeast-2]: y
    #Shows you resources changes to be deployed and require a 'Y' to initiate deploy
    Confirm changes before deploy [Y/n]: y
    #SAM needs permission to be able to create roles to connect to the resources in your template
    Allow SAM CLI IAM role creation [Y/n]: y
    Save arguments to samconfig.toml [Y/n]: y

output should look like this




    Looking for resources needed for deployment: Found!

        Managed S3 bucket: aws-sam-cli-managed-default-samclisourcebucket-fdwl3buruuk1
        A different default S3 bucket can be set in samconfig.toml

    Deploying with following values
    ===============================
    Stack name                 : sample-app
    Region                     : ap-northeast-2
    Confirm changeset          : True
    Deployment s3 bucket       : aws-sam-cli-managed-default-samclisourcebucket-fdwl3buruuk1
    Capabilities               : ["CAPABILITY_IAM"]
    Parameter overrides        : {}

Initiating deployment
=====================

    Saved arguments to config file
    Running 'sam deploy' for future deployments will use the parameters saved above.
    The above parameters can be changed by modifying samconfig.toml
    Learn more about samconfig.toml syntax at 
    https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-config.html

    Deploying with following values
    ===============================
    Stack name                 : sample-app
    Region                     : ap-northeast-2
    Confirm changeset          : True
    Deployment s3 bucket       : aws-sam-cli-managed-default-samclisourcebucket-fdwl3buruuk1
    Capabilities               : ["CAPABILITY_IAM"]
    Parameter overrides        : {}

Initiating deployment
=====================

Waiting for changeset to be created..

CloudFormation stack changeset
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Operation                                                    LogicalResourceId                                            ResourceType                                               
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ Add                                                        SampleAppdFunctionEventInvokeConfig                          AWS::Lambda::EventInvokeConfig                             
+ Add                                                        SampleAppdFunction                                           AWS::Lambda::Function                                      
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Changeset created successfully. arn:aws:cloudformation:ap-northeast-2:982947632035:changeSet/samcli-deploy1602235720/1bb1b8a3-da32-4de9-9a14-49aaae3f066f


Previewing CloudFormation changeset before deployment
======================================================
Deploy this changeset? [y/N]: y

2020-10-09 18:29:04 - Waiting for stack create/update to complete

CloudFormation events from changeset
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
ResourceStatus                                ResourceType                                  LogicalResourceId                             ResourceStatusReason                        
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
CREATE_IN_PROGRESS                            AWS::Lambda::Function                         SampleAppdFunction                            -                                           
CREATE_IN_PROGRESS                            AWS::Lambda::Function                         SampleAppdFunction                            Resource creation Initiated                 
CREATE_COMPLETE                               AWS::Lambda::Function                         SampleAppdFunction                            -                                           
CREATE_IN_PROGRESS                            AWS::Lambda::EventInvokeConfig                SampleAppdFunctionEventInvokeConfig           -                                           
CREATE_IN_PROGRESS                            AWS::Lambda::EventInvokeConfig                SampleAppdFunctionEventInvokeConfig           Resource creation Initiated                 
CREATE_COMPLETE                               AWS::Lambda::EventInvokeConfig                SampleAppdFunctionEventInvokeConfig           -                                           
CREATE_COMPLETE                               AWS::CloudFormation::Stack                    sample-app                                    -                                           
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

CloudFormation outputs from deployed stack
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Outputs                                                                                                                                                                              
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Key                 SampleAppdFunction                                                                                                                                               
Description         Hello World Lambda Function ARN                                                                                                                                  
Value               arn:aws:lambda:ap-northeast-2:982947632035:function:SampleApp                                                                                                    
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Successfully created/updated stack - sample-app in ap-northeast-2

Check cloudformation and lambda console for successful deployment.

Invoke function in lambda console

Once Lambda is deployed, you can create test event in console.

You will get successful log data.

Invoke from python app

Now you need to invoke lambda function from your app code. Here are examples you can use.



# invoke_from_app.py
import boto3
import json

SETTINGS = {
    "AWS_DEFAULT_REGION": "ap-northeast-2",
    "ENV": "prod"
}
def main():
    lambda_client = boto3.client('lambda', region_name=SETTINGS['AWS_DEFAULT_REGION'])

    payload = {
        "image_url": "https://picsum.photos/200/300"
    }

    ret = lambda_client.invoke(
        FunctionName="SampleApp",
        InvocationType="DryRun" if SETTINGS['ENV'] == "test" else "Event",
        Payload=json.dumps(payload)
    )
    print(ret)

if __name__ == "__main__":
    main()

Monitoring

Celery or rq provides native or 3rd party too for monitoring such as sentry . There are some options for monitoring lambda functions but SAM application also provides minimal monitoring environment. Go to lambda service and application menu. Select Monitoring tab to dashboard and cloudwatch logs. You can also configure x-ray for tracing.

CD

Let's talk about CD pipeline, I will use conditional trigger on github workflow and run the

Github workflow

following workflow file will build and deploy to lambda. You need to set secret key with AWS credential.



on:
  push:
    branches:
      - master

name: Lambda deployment
jobs:
  deploy:
    name: Lambda deploy
    runs-on: ubuntu-latest

    steps:
    - name: Checkout
      uses: actions/checkout@v2

    - name: Configure AWS credentials
      uses: aws-actions/configure-aws-credentials@v1
      with:
        aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
        aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        aws-region: ap-northeast-2

    - name: Build, and deploy
      run: |
        /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"
        test -d ~/.linuxbrew && eval $(~/.linuxbrew/bin/brew shellenv)
        test -d /home/linuxbrew/.linuxbrew && eval $(/home/linuxbrew/.linuxbrew/bin/brew shellenv)
        test -r ~/.bash_profile && echo "eval \$($(brew --prefix)/bin/brew shellenv)" >>~/.bash_profile
        brew --version
        brew tap aws/tap
        brew install aws-sam-cli
        sam --version

        sam build
        sam deploy --stack-name sample-app --region ap-northeast-2 -t .aws-sam/build/template.yaml --capabilities CAPABILITY_IAM --no-confirm-changeset --s3-bucket aws-sam-cli-managed-default-samclisourcebucket-fdwl3buruuk1

If you want to implement canary or linear deployment using codedeploy, refer to the following link for more detail.

Deploying serverless applications gradually

Conclusion

Once lambda is deployed, I removed 20% of my app code related to RQ. I also removed broker(Elasticsearch Redis) service anymore. Moreover since I was using ECS and EKS, I can remove additional task and pod for more api tasks and pods. I am pretty happy with the result that I don't need to manage another container on production. If you have simple workers running on RQ or Celery, I recommend that you try this setup. Thank you for reading.

How I prepared for GCP Professional Certificate Test

Jinwook Baek — Thu, 10 Sep 2020 05:06:43 +0000

original blog link

Intro

Cloud environment had exponential growth over the course (AWS didn't have RDS when I started) and I wanted to get a firm graps on the subject before venturing into new job. During covid-19 pandemic, I have racked up 4 professional cloud certificates(2 AWS, 2 GCP). I am proud of my achievement. Being certified is rewarding experience, but the knowledge and exprience I gained was more rewarding. If you are interested preparing and taking the test will be great first step, if you are already experienced, this will harden your knowledge in every aspect.

Where to start

If you do not know where to start, I recommend you to start with coursera. You can get first week free and google provides 50% off on first month. (refer to last section for the link)

Coursera

Cloud Engineering with Google Cloud

Cloud Architecture with Google Cloud

Data Engineering with Google Cloud

Qwiklab

Getting hands on is essential you make your knowledge tangible by actually working on the product. Google offers 30 day free subscriptions. I think the quest badge they offer is great example of gamification done right way. Learnig through qwiklab was really fun process for me. You can synthesize your knowledge as you get hands on with what you have learned through texts and slides.

Badges

Jinwook B. | Qwiklabs

If you don't have much time you should at least go through following quests.

For Cloud Architect

Cloud Architecture: Design, Implement, and Manage | Qwiklabs
For Data Engineering

Data Engineering | Qwiklabs

Practice exams

Once you have firm knowledge it's time for practice exams. I used following courses and it helped me get a good sense how test questions will be formed and what kind of answers will be presented. They don't cover all the topics, but you can definately figure out what you are missing.

Dig deeper into each subject

By this time you are already an expert on several topics and can dive deeper into each topics. You know what you are looking for. These resource will help you get there.

gregsramblings/google-cloud-4-words

Google Cloud for AWS Professionals

sathishvj/awesome-gcp-certifications

Taking exam

I won't go into details on how to schedule and make payment. Due to COVID-19, Google offers remote proctored exam for almost all the tests. It was really convenient for me to just take exam at my home. Of course it might better for some people if your house is not as quiet or secure place for taking exam.

Couple things I want to note

You can change your test time as long as it's 24 hours prior to the exam time. (they will charge you if you change time within 24hours
Prepare for goverment photo ID (in English) - passport is gauranteed
Have a good internet connection, it took me 30 min to start exam because my wifi was choppy. I had to go through ethernet and hotspot to get stable connection. It was exhausting process and by the time I start the test, I was already beat.
Solve easier ones first. 120 min is enough time for 50 questions. Skip or mark for later if you are any doubt or don't under stand and comeback later.
I will add more if I can think about

Topics you should know

Architect

Case study
- 20% of the exam was Scenario-based - Case Studies (Mountkirk, Dress4Win and TerramEarth). They cover this on the official website and also on coursera
Administratives
- IAM
  - role
  - policy, effective policy
  - hierarchy
  - binding
  - service account
- Billing
- Audit log
- free quota
Compute
- difference between GCE, GKE, App engine, cloud Function
- app engine runtime for standard and flexible
- Basic Kubernetes concept
  - pod, services, ingress, etc
Storage
- persistent disks
- filestore
Stackdriver
Network
- VPC, subnet, firewall
- Cloud DNS, VPN, Router
- Difference between Dedicated, partner interconnect
- Types of Loadbalancers
- Direct Peering vs Carrier Peering
Deployment
Database type by scale insurance, distribution, replica consistency, multi-primary, transacrtions, joins and complex queries, latency, serverless
- Storage transfer services vs transfer appliance vs gsutil
Know which resources are global, regional, zonal services

Global, regional, and zonal resources | Compute Engine Documentation
ETC
- SRE
  - SLI, SLO, SLA
- Opex, capex, TCO

Data Engineer

Dataflow
- pardo, transform, pCollection, pipeline
- Windowing
  - session, fixed, sliding, global
- deduplication
Bigquery
- BQML
- Billing
- Permission
- query optimization
- Responsive, predictive cache
- Partition
Dataproc
- cluster resize, modification
- HDFS vs Cloudstorage
- storage connectors
Dataprep
- recipe
ML Engine
Storage

Must know storage limit, processing limit and NoOP or unmanaged service
- datastore
- cloud spanner
  - key design
- cloud sql
- bigtable
  - bigtable key design
  - performance optimization
- bigquery
Basic ML knowledge
- Overfitting/underfitting
- False positive, False Negative
- Regression, Classification, Clustering
- Supervised, unsupervised learning
- Recall, Precision, Accuracy
- Regularization
- AUC
AutoML
AppEngine
Cloud Composer
pubsub
- retention
- push,pull

Offer for Coursera and Qwiklab

Google Cloud Training

Final Word

Don't just memorize to get correct answer, do enjoy the process of research. Knowledge is power. If you have questions and curious about something dig deeper into the product and make it yours.
Qwiklab rules!
Good luck!

Gunicorn performance analysis on AWS EC2

Jinwook Baek — Mon, 27 Jul 2020 13:18:01 +0000

Introduction

There are several production level self-hosted options for running django app server behind nginx web sever using wsgi protocol, such as uWsgi, mod_wsgi and gunicorn. I have been using gunicorn for many projects for mayn years. Since then I have never doubted the performance and configuration details but recently I wanted to test the actual performance for gunicorn associated with worker count and find out optimal configuration for ECS fargate. In contrast to on-prem servers where I can grasp on actual number of physical cores, AWS only allow me to configure number of logical cores(via vCPU). I will illustrate how I have tested the performance using gunicorn, django and locust.

Standalone WSGI Containers - Flask Documentation (1.1.x)

Gunicorn

Gunicorn is Python WSGI HTTP Server for UNIX.

$ gunicorn django-sample.wsgi:application -w ${WORKER_COUNT} 
--threads ${THREAD_COUNT} -b 0.0.0.0:8000

You generally use following command to run the applicaiton. According to the design document, they recommend (2 x $num_cores) + 1 as the number of workers to start off with. The formula is based on the assumption that for a given core, one worker will be reading or writing from the socket while the other worker is processing a request.

Gunicorn - WSGI server - Gunicorn 20.0.4 documentation

EC2 vCPU

I need to determine the number of cpu cores for the server. In traditional linux server, you can determine number of core using following commands.

According to AWS, not all vCPUs are made the same. For T2 instances, 1 vCPU = 1 physical core. For all others, 1 vCPU = 1 logical core.

$ cat /proc/cpuinfo
-> cpu_cores

Optimizing CPU options

The number of vCPUs for the instance is the number of CPU cores multiplied by the threads per core. To specify a custom number of vCPUs, you must specify a valid number of CPU cores and threads per core for the instance type.

vCPU = Physical core * threads

According to this blog post there were 34% drop in multi-threaded performance on logical cores. ****

What's in a vCPU: State of Amazon EC2 in 2018 - Credera

You can configure threads per core during instance launch.

Methodology

Architecture

Github Repo

Test will run on 3 tier web architecture. Django application will run behind the nginx through gunicorn. Django app will be querying Mysql DB in private subnet.

Instances used

t3.micro - 1 CPU
- cpuinfo
t3.micro - 2 vCPU (1 cpu * 2 threads)
- cpuinfo
t3.xlarge - 2 cpu
- cpuinfo
t3.xlarge - 4 vCPU (2 cpu * 2 threads)
t3.2xlarge - 4 cpu
- cpuinfo

Locust

Locust - A modern load testing framework

I will use locust to test and analyze the performance under load. Following Locust file will spawn multiple users concurrently with simple SQL query.

from random import randint

from locust import User, TaskSet, between, task, HttpUser

class ReadPosts(TaskSet):
    @task(1)
    def read_posts(self):
        response = self.client.get("api/v1/posts/", name="list posts")
        print("Response status code:", response.status_code)
        print("Response content:", response.text)

class WebsiteUser(HttpUser):
    tasks = [ReadPosts]
    wait_time = between(1.0, 2.0)

Additional setup

Tested django app with DEBUG = False
Latency is not considered since test was executed within VPC
Used serverless aurora RDS to avoid bottleneck on DB I/O

Test procedure

Used htop to record CPU usage of each core and mem usage
Incremented number of concurrent users by 100 to determine the load to see two thresholds

Result

Full results:
Gunicorn performance test result

Observations

Single worker cannot fully utilize cpus
Recommended number of workers (2*core +1) generally perform well as expected.
There are not much performance difference in physical or logical core.
- Single threads seem to have slightly better performance
- Therefore, it is efficient to use multiple threads if you consider price and memory usage is not the issue
  - t3.micro 1 core 2 threads - $0.0104 per Hour
  - t3.large 2 cores 1 thread - $0.0832 per Hour

Questions

What would be the efficient cpu usage cap? 60% - 90%?
Any other metrics I should have count into experiments?

ECS fargate performance

Since I gained general idea how gunicorn performed on ec2 instances. I took it further to test how gunicorn performs on ECS fargate setup with regarding vCPUs allocated. Please refer to the github for task definitions I have used.

ecs-task-definition

Since I cannot manage instance in faragte setup, I needed to assign cpu units for each tasks to comare with ec2. It was impossible to run htop on fargate since I have no access to the instances. ECS provide container insight but they are not as accurate nor real-time as I would run htop inside the instances. I only recorded Fail rate according to concurrent user count.

I didn't assign cpu unit for each task, since two tasks in previous test shared cpu resources in single machine.

Task definition parameters

Results

Results does not seem to be as consistent as ec2 cores. It seems like more worker counts result better performance proportionally.

After thoughts

I was not entirely satisfied with the result since I was not able to isolate all variables.

Performance probably will depend on app behavior (mem
long running i/o app will probably will have different results
Test result would have been more accurate with CPU reousrce config on docker-compose file (https://docs.docker.com/compose/compose-file/#resources)

Next Todo

ASGI benchmark on different vCPU options
Use Asyncworker + threads options

Reference

PEP 3333 -- Python Web Server Gateway Interface v1.0.1

uWSGI vs. Gunicorn, or How to Make Python Go Faster than Node

A Performance Analysis of Python WSGI Servers: Part 2 | Blog | AppDynamics

Better performance by optimizing Gunicorn config

A Guide to ASGI in Django 3.0 and its Performance

Quick start - Locust 1.1.1 documentation

A Guide to ASGI in Django 3.0 and its Performance

What's in a vCPU: State of Amazon EC2 in 2018 - Credera

How Amazon ECS manages CPU and memory resources | Amazon Web Services

How to deploy django app to ECS Fargate part3

Jinwook Baek — Fri, 03 Jul 2020 12:40:54 +0000

This post was originally published on notion. Click here if you prefer to read in notion page which has better readability.

Introduction

This blog post illustrate development cycle using django app container. I assume readers are already somewhat familar with docker and docker-compose. Although I used django for app development, it is language-agnostic since the post is about containerized application deployment.

Walkthough is devided into three parts consisting three different environemt respectively. First part, describes the architecture of the app(api and async worker) and how they are deployed on local enviroment. Second part is how to deploy the docker containers on cloud using single ec2 instance with on staging environment. Third part, illustrate how to convert traditional ec2 deployment into ECS using fargate with github actions on prod environment.

local - run docker containers on desktop/laptop with sqlite and redis server using docker-compose

stating - run docker containers on single ec2 instance with mysql RDS and ElasticCache

prod - convert stagning setup to ECS Fargate

You can check out previous part here.

How to deploy django app to ECS Fargate Part1

Jinwook Baek ・ Jul 3 ・ 4 min read

#aws #docker #tutorial #webdev

How to deploy django app to ECS Fargate Part2

Jinwook Baek ・ Jul 3 ・ 10 min read

#aws #docker #tutorial #webdev

ECS fargate primer

ECS provides container management service that makes it easy to run, stop, and manage Docker containers on a cluster. However you still have to manage container instances with container agent running on instances. With AWS Fargate, you no longer have to provision, configure, or scale clusters of virtual machines to run containers. It's a good idea to have some understanding on following components of ECS.

Components

Cluster

Logical grouping of resources that tasks and services are running on. When using the Fargate launch type with tasks within your cluster, ECS manages your cluster resources.

Service

This enables you to run and maintain a specified number of instances of a task definition simultaneously in an Amazon ECS cluster. You need service to run them behind

Task definition

json file that describes one or more containers. Task definition is not cluster dependent. You can understand it functions similar as docker-compose file.

Task

instantiation of a task definition within a cluster. Each task that uses the Fargate launch type has its own isolation boundary and does not share the underlying kernel, CPU resources, memory resources, or elastic network interface with another task.

Task Scheduler

is responsible for placing tasks within your cluster. There are several different scheduling options available. Fargate currently only support REPLICA which places and maintains the desired number of tasks across your cluster. By default, the service scheduler spreads tasks across Availability Zones. You can use task placement strategies and constraints to customize task placement decisions.

Amazon ECS on AWS Fargate

Fargate tasks

Fargate task definitions require that you specify CPU and memory at the task level.
Fargate task definitions only support the awslogs log driver for the log configuration. This configures your Fargate tasks to send log information to Amazon CloudWatch Logs.
Put multiple containers in the same task definition if:
- Containers share a common lifecycle.
- Containers are required to be run on the same underlying host.
- You want your containers to share resources.
- Your containers share data volumes.

Walkthrough

We will use the VPC , RDB and Redis we created from part2, refer to the previous post or CF template for the setup.

Container networking is different in fargate

Let's review how container is structued in part 2. Nginx container and app container communictes through 8000 port through birdge network. And worker container will work.

We will configure our task definition like this.

If you are running these two components as two processes on a single EC2 instance, the web tier application process could communicate with the API process on the same machine by using the local loopback interface. The local loopback interface has a special IP address of 127.0.0.1 and hostname of localhost.

By making a networking request to this local interface, it bypasses the network interface hardware and instead the operating system just routes network calls from one process to the other directly. This gives the web tier a fast and efficient way to fetch information from the API tier with almost no networking latency.

We will update nginx image with new conf with above restriction

new nginx conf

# portal
server {
  listen 80;

  # all requests proxies to app
  location / {
        proxy_pass http://127.0.0.1:8000;
        proxy_set_header  X-Real-IP  $remote_addr;
        proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $host;
        proxy_redirect off;

    }

  # domain localhost
  server_name *;
}

This sends a local network request, which goes directly from one container to the other over the local loopback interface without traversing the network. This deployment strategy allows for fast and efficient communication between two tightly coupled containers.

We will build new nginx image with this new conf and push the new image to ecr repository to include in the task definition.

# make sure you have <aws_account_id>.dkr.ecr.us-east-1.amazonaws.com/nginx ECR created
$ docker build -f config/app/Docker_nginx -t  <aws_account_id>.dkr.ecr.us-east-1.amazonaws.com/nginx:latest .
$ docker push <aws_account_id>.dkr.ecr.us-east-1.amazonaws.com/nginx:latest

Create ECS execution role

create sampleECSTaskExecutionRole with following policies and trust relationships

Policies

AmazonECSTaskExecutionRolePolicy

You need to attach following policies for environment variable setup on task (explained in later step)

AWS Managed Policy AmazonSSMReadOnlyAccess
- you might want tighter restriction on resources for actual development

Inline policy for decryption

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "kms:Decrypt"
            ],
            "Resource": "*"
        }
    ]
}

make sure to add trust entities for ecs-tasks.amazonaws.com

Create Security Group for ecs cluster

Let's create secrity group for ecs cluster named - ecs-sg. I recommend copy ecs-sample-ec2 from previous post which consists inbound rule

allow 80 port to public
allow 80 to ecs-sample-lb security group
allow 6379 to ecs-sample-redis security group
allow 3306 to ecs-sample-mysql security group

You also need to allow

ecs-sample-redis and ecs-sample-mysql to allow traffic for each port from ecs-sg

Loadbalancer

Let's craete a new loadbalancer which will be used for the service who manages connection between lb and tasks.

Select ALB
name - ecs-fargate
select VPC created from previous post
listener and target group will be managed by service so you can create empty listener and target group. we will delete them on following step.

delete listener and target group you created from previous alb creation

you can also use cli to create ALB instead of using console. This way, you don't need to create empty listener and target group whic will be deleted later

$ aws elbv2 create-load-balancer --name <ecs-fargte>  \
--subnets <subnet-12345678> <subnet-23456789> --security-groups <ecs-sg>

Take a note on DNS name of this load balancer for later use.

Create Cluster

We will create empty cluster first. Goto ECS menu on console. (make sure you are in correct region) Create first cluster template

Cluster is created

Alternatively you can create cluster with following command

aws ecs create-cluster --cluster-name ecs-sample

Create task definition

Parameter store

Part of problem using fargate is that you cannot pass environment variable as we used to because we do not have ec2 access. Therefore we will be using parameter store to inject environment variables to tasks when they start.

Click to AWS Systems Manager on console menu.

Click Paramter Store on left menu. link(make sure you are on correct region)

click createk parameter

Fill in the input as following screenshot.

You will create value for following environment varibales

SECRET_KEY
REDIS_HOST
DB_PASSWORD
DB_HOST

Alternatively you can use cli to create secure strings

$ aws ssm put-parameter --name "/ecs-sample/prod/<ENV_NAME>" --type "SecureString" --value "<value>" --description "<ENV_NAME>" --region "<region>"
>
{
    "Version": 1,
    "Tier": "Standard"
}

results

Task definition

As I mentioned earlier taks definition works smilar to docker-compose. Just like staging evn, we will have two task definitions and nginx container + api container and worker container

Using task definition template, you can fill in following configurations

$ aws ecs register-task-definition --generate-cli-skeleton

sample-task-definition

Parameters

Task family – the name of the task, and each family can have multiple revisions.
IAM task role – specifies the permissions that containers in the task should have.
Network mode – determines how the networking is configured for your containers. (fargate only supports awsvpn currently)
Container definitions – specify which image to use, how much CPU and memory the container are allocated, and many more options.

refer to following link for each parameters

Task definition parameters

{
    "family": "",
    "taskRoleArn": "",
    "executionRoleArn": "",
    "networkMode": "awsvpc",
    "containerDefinitions": [
        {
            "name": "",
            "image": "",
            "repositoryCredentials": {
                "credentialsParameter": ""
            },
            "cpu": 0,
            "memory": 0,
            "memoryReservation": 0,
            "links": [
                ""
            ],
            "portMappings": [
                {
                    "containerPort": 0,
                    "hostPort": 0,
                    "protocol": "tcp"
                }
            ],
            "essential": true,
            "entryPoint": [
                ""
            ],
            "command": [
                ""
            ],
            "environment": [
                {
                    "name": "",
                    "value": ""
                }
            ],
            "environmentFiles": [
                {
                    "value": "",
                    "type": "s3"
                }
            ],
            "mountPoints": [
                {
                    "sourceVolume": "",
                    "containerPath": "",
                    "readOnly": true
                }
            ],
            "volumesFrom": [
                {
                    "sourceContainer": "",
                    "readOnly": true
                }
            ],
            "linuxParameters": {
                "capabilities": {
                    "add": [
                        ""
                    ],
                    "drop": [
                        ""
                    ]
                },
                "devices": [
                    {
                        "hostPath": "",
                        "containerPath": "",
                        "permissions": [
                            "read"
                        ]
                    }
                ],
                "initProcessEnabled": true,
                "sharedMemorySize": 0,
                "tmpfs": [
                    {
                        "containerPath": "",
                        "size": 0,
                        "mountOptions": [
                            ""
                        ]
                    }
                ],
                "maxSwap": 0,
                "swappiness": 0
            },
            "secrets": [
                {
                    "name": "",
                    "valueFrom": ""
                }
            ],
            "dependsOn": [
                {
                    "containerName": "",
                    "condition": "HEALTHY"
                }
            ],
            "startTimeout": 0,
            "stopTimeout": 0,
            "hostname": "",
            "user": "",
            "workingDirectory": "",
            "disableNetworking": true,
            "privileged": true,
            "readonlyRootFilesystem": true,
            "dnsServers": [
                ""
            ],
            "dnsSearchDomains": [
                ""
            ],
            "extraHosts": [
                {
                    "hostname": "",
                    "ipAddress": ""
                }
            ],
            "dockerSecurityOptions": [
                ""
            ],
            "interactive": true,
            "pseudoTerminal": true,
            "dockerLabels": {
                "KeyName": ""
            },
            "ulimits": [
                {
                    "name": "msgqueue",
                    "softLimit": 0,
                    "hardLimit": 0
                }
            ],
            "logConfiguration": {
                "logDriver": "awslogs",
                "options": {
                    "KeyName": ""
                },
                "secretOptions": [
                    {
                        "name": "",
                        "valueFrom": ""
                    }
                ]
            },
            "healthCheck": {
                "command": [
                    ""
                ],
                "interval": 0,
                "timeout": 0,
                "retries": 0,
                "startPeriod": 0
            },
            "systemControls": [
                {
                    "namespace": "",
                    "value": ""
                }
            ],
            "resourceRequirements": [
                {
                    "value": "",
                    "type": "GPU"
                }
            ],
            "firelensConfiguration": {
                "type": "fluentd",
                "options": {
                    "KeyName": ""
                }
            }
        }
    ],
    "volumes": [
        {
            "name": "",
            "host": {
                "sourcePath": ""
            },
            "dockerVolumeConfiguration": {
                "scope": "task",
                "autoprovision": true,
                "driver": "",
                "driverOpts": {
                    "KeyName": ""
                },
                "labels": {
                    "KeyName": ""
                }
            },
            "efsVolumeConfiguration": {
                "fileSystemId": "",
                "rootDirectory": "",
                "transitEncryption": "ENABLED",
                "transitEncryptionPort": 0,
                "authorizationConfig": {
                    "accessPointId": "",
                    "iam": "ENABLED"
                }
            }
        }
    ],
    "placementConstraints": [
        {
            "type": "memberOf",
            "expression": ""
        }
    ],
    "requiresCompatibilities": [
        "EC2"
    ],
    "cpu": "",
    "memory": "",
    "tags": [
        {
            "key": "",
            "value": ""
        }
    ],
    "pidMode": "task",
    "ipcMode": "none",
    "proxyConfiguration": {
        "type": "APPMESH",
        "containerName": "",
        "properties": [
            {
                "name": "",
                "value": ""
            }
        ]
    },
    "inferenceAccelerators": [
        {
            "deviceName": "",
            "deviceType": ""
        }
    ]
}

Let's create task definitions

$ aws ecs register-task-definition \
    --cli-input-json file://<path_to_json_file>/task-definition-api.json
$ aws ecs register-task-definition \
    --cli-input-json file://<path_to_json_file>/task-definition-worker.json

you will be able to see 2 task definitions registered in your ECS service. Inspect configuration for each task definitions.

Create services

Now you are ready to create ECS service which will fire up the tasks. Go to cluster and press create button on the services tab.

API task

1) configure service
2) configure network

Network
- we will select VPC from part 2 and select security group we created previously ecs-sg
- Choose public subnet for the ecs-sample-api task

Load Balancer

- select `Application Load Balancer` for Load blancer type*
    - Production listener port : 80
    - select `ecs-fargate` load balancer we have created in this post
    - once you select the load balance you will be able to choose `container to load blaance`. Service automatically detect ports opened. Choose `nginx:80:80`. Click add to load balancer
    - You will be able to configure the health check path. Update to `/api/healthcheck`/

Service discovery

- click service discovery
- update namespace as you like, I used `api`
- update serive discovery name `ecs-sample-api`

3) set auto scaling(optional)
- we will skip this part
4) Review

Worker task

For worker task, reapeat above step except

choose private subnet
no load balancer
no service discovery

Service

Once everything is set let's deploy new service and wait for all service status to turn to green.

Go back to service you created and go to tasks tab, if you see all service running you will see following screenshot, however if you don't see any tasks running search tasks with stopped status and inspect logs(you might need to update task definition to configure logs) for any misconfigurations.

Let's check elb and route 53 to make sure everythign work.

Service discovery and healthcheck seems to work fine.

To access the nginx from browser, type in DNS name of the load balancer to see if everything is working as we expected.

Continuous Deployment

Once everything is deployed, there is a problem if I need to update a source image. I need to repeat all steps above include

Build new docker image
Push docker image to ECR
Create new task definition with new docker image
Update service to run new task definition revision

Github action

We can automate this step with github action.

Features * GitHub Actions

Take a look at following file in the repo .github/workflows/ecs_api.yml

workflow file

on:
  push:
    branches:
      - master

name: Deploy Hangfive app to Amazon ECS

jobs:
  deploy:
    name: Deploy
    runs-on: ubuntu-latest

    steps:
    - name: Checkout
      uses: actions/checkout@v2

    - name: Configure AWS credentials
      uses: aws-actions/configure-aws-credentials@v1
      with:
        aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
        aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        aws-region: us-east-1

    - name: Login to Amazon ECR
      id: login-ecr
      uses: aws-actions/amazon-ecr-login@v1

    - name: Build, tag, and push image to Amazon ECR
      id: build-image
      env:
        ECR_REGISTRY: 982947632035.dkr.ecr.us-east-1.amazonaws.com
        ECR_REPOSITORY: hangfive
        IMAGE_TAG: ${{ github.sha }}
      run: |
        # Build a docker container and
        # push it to ECR so that it can
        # be deployed to ECS.
        docker build -f config/app/Dockerfile_app -t $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG .
        docker push $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG
        echo "::set-output name=image::$ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG"

    - name: Fill in the new image ID in the Amazon ECS task definition
      id: task-def
      uses: aws-actions/amazon-ecs-render-task-definition@v1
      with:
        task-definition: config/ecs/task-definition.json
        container-name: app
        image: ${{ steps.build-image.outputs.image }}

    - name: Deploy Amazon ECS task definition
      uses: aws-actions/amazon-ecs-deploy-task-definition@v1
      with:
        task-definition: ${{ steps.task-def.outputs.task-definition }}
        service: hangfive-prod-ecs-fargate
        cluster: hangfive-prod
        wait-for-service-stability: true

This yaml file describes following actions
- triggers on every master branch push
  
  Events that trigger workflows
- checkout the source
- AWS CLI config ( secret from github)
  - AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY of user with appropriate permission
  Creating and storing encrypted secrets
- log into ECR
- build docker image and tag with different hash everytime (for immutability)
- update task definition then deploy new task definition
Click actions tab on your repository for the progress.

Teardown

List of aws resources you need to delete
- EC2 instances, ALB, target group, autoscaling
- ECS Services, cluster, task definitions
- ECR
- SSM parameter store
- Route 53 hosted zone
- RDS
  - subnet group, cluster paramter group
- Elastic Cache
  - subnet group, parmater group
- VPC
  - security group, ncl, subnets, IGW, NAT gateway, EIP

Future todos

Logging / Monitoring setup with cloudwatch
CI - pytest integration with github action before merge

Reference

Task Networking in AWS Fargate | Amazon Web Services

Task Networking with the awsvpc Network Mode

Use bridge networks

AWS ECS의 매니지드 컨테이너 AWS 파게이트(AWS Fargate) 시작하기

Tutorial: Creating a Cluster with a Fargate Task Using the AWS CLI

Pass Secrets or Sensitive Data Securely to Containers in Amazon ECS Tasks

put-parameter - AWS CLI 1.18.89 Command Reference

Amazon ECS "Deploy Task Definition" Action for GitHub Actions - GitHub Marketplace

Pass Secrets or Sensitive Data Securely to Containers in Amazon ECS Tasks

Trigger workflow only on pull request MERGE

How to deploy django app to ECS Fargate Part2

Jinwook Baek — Fri, 03 Jul 2020 12:00:25 +0000

This post was originally published on notion. Click here if you prefer to read in notion page which has better readability.

Introduction

local - run docker containers on desktop/laptop with sqlite and redis server using docker-compose

stating - run docker containers on single ec2 instance with mysql RDS and ElasticCache

prod - convert stagning setup to ECS Fargate

For part1, click here.

How to deploy django app to ECS Fargate Part1

Jinwook Baek ・ Jul 3 ・ 4 min read

#aws #docker #tutorial #webdev

You can skip to part3 if you are already familiar with AWS architectures.

How to deploy django app to ECS Fargate part3

Jinwook Baek ・ Jul 3 ・ 12 min read

#aws #docker #tutorial #webdev

Staging Infra Setup

Before going straight to ECS deployment, we will setup up a staging env to test the application using ec2 compute instance and other AWS cloud services. If you are familar with AWS infra and confident with ecs setup, you can skip this part. However I will be using same vpc, redis and mysql for production env so it would be worth while to take a look at the setup. (you should use different vpc, redis and mysql for your actual production deployment)

The staging cloud architecture will consist following AWS services.

VPC with public and private subnet
EC2 instance (in public subnet)
ALB in front of ec2 instance
RDS mysql (in private subnet)
ElasticCache - Redis (private subnet)

Refer to the cloudformation for detailed configurations.

kokospapa8/ecs-fargate-sample-app

Preview

Network - VPC

"VpcEcsSample": {
      "Type": "AWS::EC2::VPC",
      "Properties": {
        "CidrBlock": "172.10.0.0/16",
        "InstanceTenancy": "default",
        "EnableDnsSupport": "true",
        "EnableDnsHostnames": "true",
        "Tags": [
          {
            "Key": "Name",
            "Value": "ecs-sample"
          }
        ]
      }
    },
    "EcsSamplePrivate1": {
      "Type": "AWS::EC2::Subnet",
      "Properties": {
        "CidrBlock": "172.10.11.0/24",
        "AvailabilityZone": "us-west-2a",
        "VpcId": {
          "Ref": "VpcEcsSample"
        },
        "Tags": [
          {
            "Key": "Name",
            "Value": "ecs-sample-private 1"
          }
        ]
      },
      "DependsOn": "VpcEcsSample"

    },
"EcsSamplePublic1": {
      "Type": "AWS::EC2::Subnet",
      "Properties": {
        "CidrBlock": "172.10.1.0/24",
        "AvailabilityZone": "us-west-2a",
        "VpcId": {
          "Ref": "VpcEcsSample"
        },
        "Tags": [
          {
            "Key": "Name",
            "Value": "ecs-sample-public 1"
          }
        ]
      },
      "DependsOn": "VpcEcsSample"
    },
    "IgwEcsSample": {
      "Type": "AWS::EC2::InternetGateway",
      "Properties": {
        "Tags": [
          {
            "Key": "Name",
            "Value": "ecs-sample IGW"
          }
        ]
      },
      "DependsOn": "VpcEcsSample"
    },
"NATGateway": {
        "Type": "AWS::EC2::NatGateway",
        "Properties": {
            "AllocationId": {
                "Fn::GetAtt": [
                    "EipNat",
                    "AllocationId"
                ]
            },
            "SubnetId": {
                "Ref": "EcsSamplePublic1"
            }
        },
        "DependsOn": [
            "EcsSamplePublic1",
            "EipNat",
            "IgwEcsSample",
            "GatewayAttachment"
        ]
    },

RDS

"RDSCluster" : {
      "Type": "AWS::RDS::DBCluster",
      "Properties" : {
          "MasterUsername" : {
              "Ref": "DBUsername"
          },
          "MasterUserPassword" : {
              "Ref": "DBPassword"
          },
          "DBClusterIdentifier" : "ecs-sample",
          "Engine" : "aurora",
          "EngineVersion" : "5.6.10a",
          "EngineMode" : "serverless",
          "ScalingConfiguration" : {
              "AutoPause" : true,
              "MinCapacity" : 4,
              "MaxCapacity" : 8,
              "SecondsUntilAutoPause" : 1000
          }
      }
    },

Redis

cacheecssample001": {
      "Type": "AWS::ElastiCache::CacheCluster",
      "Properties": {
        "AutoMinorVersionUpgrade": "true",
        "AZMode": "single-az",
        "CacheNodeType": "cache.t2.micro",
        "Engine": "redis",
        "EngineVersion": "5.0.6",
        "NumCacheNodes": "1",
        "PreferredAvailabilityZone": "us-west-1b",
        "PreferredMaintenanceWindow": "thu:02:30-thu:03:30",
        "CacheSubnetGroupName": {
          "Ref": "cachesubnetecssampleredissubnetgroup"
        },
        "VpcSecurityGroupIds": [
          {
            "Fn::GetAtt": [
              "sgecssampleredis",
              "GroupId"
            ]
          }
        ]
      }
    },

This template is preconfigured to us-west-2 region.

Migrate docker images to ECR

Before setting up the infrastructure I will move docker images to elastic container registry. If you want to still use docker hub or other private docker image repository, you can skip this part

ecr setup

Getting started with Amazon ECR using the AWS CLI

log into ecr

$ eval $(aws ecr get-login-password --region us-east-2 --no-include-email)
> Login Succeeded

Create repository with cli

$ aws ecr create-repository --repository-name ecs-sample-nginx --region us-east-2
$ aws ecr create-repository --repository-name ecs-sample-api --region us-east-2
$ aws ecr create-repository --repository-name ecs-sample-worker --region us-east-2

or create repository on the console

Build docker image if you have not build the image from previoud post

```bash
$ docker build -f config/app/Docker_base -t ecs-fargate-sample_base:latest .
$ docker build -f config/app/Docker_app -t ecs-fargate-sample_app:latest .
$ docker build -f config/app/Docker_worker -f ecs-fargate-sample_app:latest .
```

Tag and push images

```bash
$ docker tag ecs-fargate-sample_app:base <aws_account_id>.dkr.ecr.us-east-1.amazonaws.com/ecs-sample-api:base
$ docker push <aws_account_id>.dkr.ecr.us-east-1.amazonaws.com/ecs-sample-api:base
$ docker tag ecs-fargate-sample_app:latest <aws_account_id>.dkr.ecr.us-east-1.amazonaws.com/ecs-sample-api:latest
$ docker push <aws_account_id>.dkr.ecr.us-east-1.amazonaws.com/ecs-sample-api:latest
$ docker tag ecs-fargate-sample_worker:base <aws_account_id>.dkr.ecr.us-east-1.amazonaws.com/ecs-sample-worker:latest
$ docker push <aws_account_id>.dkr.ecr.us-east-1.amazonaws.com/ecs-sample-worker:latest
```

Check the console for the images

IAM role

You need access to ECR from your ec2 instances, instead of embedding AWS accesskey in the instance I will attach an iam role.

ecs-sample-ec2_role (optional)
- I will give permission to read from ECR
Amazon ECR Managed Policies

Network

VPC

I have created VPC with CIDR 172.10.0.0/16. refer to the images attached for reference. For production I advise you to make more subnets in multi-AZ for better availability. For the sake of the post, I have just used two available zones (serverless mysql needs at least two AZs)

Public subnet

CIDR - 172.10.1.0/24, 172.10.2.0/24

Components

- ec2 instance for `api`
- NAT GATEWAY for `worker` instance

Private subnet

CIDR - 172.10.11.0/24, 172.10.12.0/24

Components

- mysql
- elastic cache
- ec2 instance for `worker`

Security group

I have added minial security measure for the sake of the post. This is just sample you should not use this for production.

Setup

ecs-sample-VPC
- Opened all ports to public (not safe)
ecs-sample-loadbalancer
- 80 and 443 to public
- 80 with ec2-api instance
ecs-sample-mysql
- 3306 with ec2
ecs-sample-ec2-api and ecs-sample-ec2-worker (they should be different but I just used single sg for both ec2, should block 80 access for particular ec2 instance)
- 3306 mysql
- 6379 redis,
- 80 for ALB and public ip access
- 22 for ssh
ecs-sample-redis
- 6379 with ec2 ### ALB

Created Alb with simple setup attached to public subnet. We don't have instance yet so just created target group with no instance attached yet.

configuration
- name: ecs-sanple-lb
- Listener: port 80
- Security group: ecs-sample-lb
- VPC and subnet - public subnet created above

Database

RDS - Mysql

I will use serverless mysql instead of traditional mysql instance to minimize cost. (serverless mysql only support mysql engine version up to 5.6)

configuration
- database - sample
- username - admin
- network - VPC and private subnet created above

Take a note on your mysql endpoint and password, will be using it later for env variable setup.

Elasticcache - Redis

Take a note on your primary endpoint, will be using it later for env variable setup.

Compute instances

We will start two ec2 instances. One for api in public subnet and another for worker in private sunet.

EC2 instance - API

configurations
- AMI : Ubuntu image 20.04 LTS
- Network: public subnet
- auto-assign ip : enabled

Ec2 instance - worker

This instance is basically same with api instance but it needs to in private subnet. I advise making AMI from api instance and fire up on private instance. Since there will be no public ip attached, you need to tunnel through api or bastion instance to connect the instance with private ip. (You can use following method to connect directly since we have NAT gateway deployed)

Deployment

Once the instance is up and running, we are ready to deploy docker containers.

Let's ssh into to ec2 instance.

ssh -i "/path/to/keyfile/" ubuntu@<public_ip>

Docker setup

Next, you need to install docker engine and docker-compose. refer to following link

install steps

Install Docker Engine on Ubuntu

Install Docker Compose

Post-installation steps for Linux

# docker engine
$ sudo apt-get update
$ sudo apt-get install \
    apt-transport-https \
    ca-certificates \
    curl \
    gnupg-agent \
    software-properties-common
$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
$ sudo add-apt-repository \
   "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
   $(lsb_release -cs) \
   stable"
$ sudo apt-get update
$ sudo apt-get install docker-ce docker-ce-cli containerd.io
# docker-compose
$ sudo curl -L  \ 
"https://github.com/docker/compose/releases/download/1.26.0/docker-compose-$(uname -s)-$(uname -m)" \ 
-o /usr/local/bin/docker-compose

Environment variables

You need to set couple env for staging since we are using redis and mysql.

Please take a closer look at settings/staging.py for DB_USER and DB_NAME. If they are different, update the file accordingly.

export SECRET_KEY=""
export DB_HOST=""
export DB_PASSWORD=""
export REDIS_HOST=""

Run containers

Check out sources from git and run the docker-compose

$ git clone https://github.com/kokospapa8/ecs-fargate-sample-app.git
$ cd ecs-fargate-sample-app
$ docker-compose -f docker-compose-staging-api.yml up --build
# add -d option for detach mode I explicitly left it out to see access logs
# if you get error running docker-compose or docker, you probably don't have previlige to run docker as user, 
# refer to https://docs.docker.com/engine/install/linux-postinstall/

# instead of building the whole images you can pull the image from ECR - https://docs.docker.com/compose/reference/pull/
# in order to pull the image from repository make sure you add images filed to each docker-compose file - https://docs.docker.com/compose/compose-file/#image
# 
$ docker-compose -f docker-compose-staging-api.yml pull
$ docker-compose -f docker-compose-staging-api.yml up

Once docker containers are up and running and no error shown, access public ip of your app instance.

Attach instace behind ALB

Attach your instance to ALB's target group then you will start seeing ALB's health check on the log.

For worker instance

Follow same step as previous step on docker setup or create another AMI image from api instance and start the instance in private subnet.

$ git clone https://github.com/kokospapa8/ecs-fargate-sample-app.git
$ cd ecs-fargate-sample-app
$ docker-compose -f docker-compose-staging-worker.yml up --build

Caveats

Everything checked out; in order to use this setup in production level, there are couple problems we need to address.

Scaling services
Updating sources for further development

Scaling services

We can create launch config and attach it to autoscaling group for automatic scaling.

What Is AWS Auto Scaling?

We need to prepare couple things for this to work

Each instances must have environment variable injected on startup. There are couple ways to do this
Pull new source
Run docker-compose on start

There are various way to accomplish this, you can search for other methods

Aws Ec2 run script program at startup

Updating sources for further development

Configuration changes - environment
pulling new src from git

Brute force

ssh into each ec2 instance

> ssh -i "/path/to/keyfile/" ubuntu@<public_ip>
$ git pull
$ docker-compose -f docker-compose-staging-api.yml up --build

Obviously this is not a great idea if you have multiple instances of ec2 running. Also ip address change if instances are running behind autoscaling group.

Fabric

You can use fabric to alleviate such painful process. It's library designed to execute shell commands remotely over SSH.

Fabric

I have provided sample fab file for accessing ec2 instances in public subnets.

def _get_ec2_instances():
    instances = []
    connection = connect_to_region(
        region_name = "ap-northeast-2", #TODO beta env
        aws_access_key_id=os.getenv("AWS_ACCESS_KEY_ID"),
        aws_secret_access_key=os.getenv("AWS_SECRET_ACCESS_KEY")
    )

    try:
        reservations = connection.get_all_reservations(filters= {'tag:Name':'ecs-sample-api','tag:env':'staging'})
        for r in reservations:
            for instance in r.instances:
                instances.append(instance)

    except boto.exception.EC2ResponseError as e:
        print(e)
        return

    instances=filter(None, instances)
    return instances

kokospapa8/ecs-fargate-sample-app

When you run fabfile, it uses boto library to traverse ec2 instance with tag named as ecs-sample-api. Therefore you need to add ecs-sample-api for Name tag for each ec2 instances.

# you need to install fab and boto library
$ pip install fabric, boto
#edit fabfile.py 
# SSH_KEY= "PATH/TO/YOUR/SSH_KEY"
# set env for AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY with ec2 describe policies

$ fab --list
$ fab gitpull
$ fab docker-restart

Wrap up

We are done for staging environment setup. We now have a docker containers running on cloud evironment. However there are couple concerns I woud like to address.

Accessing each instances manually to update the source is not secure job
I would like to address immutability to container deployment.
I would like to fully utilize my compute node for containers.

Let's move to part3

How to deploy django app to ECS Fargate part3

Jinwook Baek ・ Jul 3 ・ 12 min read

#aws #docker #tutorial #webdev

Reference

Save yourself a lot of pain (and money) by choosing your AWS Region wisely

VPC with public and private subnets (NAT)

Fabric

Securely Connect to Linux Instances Running in a Private Amazon VPC | Amazon Web Services

What Is Amazon EC2 Auto Scaling?

How to deploy django app to ECS Fargate Part1

Jinwook Baek — Fri, 03 Jul 2020 06:43:22 +0000

This post was originally published on notion. Click here if you prefer to read in notion page which has better readability.

Introduction

You can skip to next part if you are already familiar with docker and AWS architectures.

How to deploy django app to ECS Fargate Part2

Jinwook Baek ・ Jul 3 ・ 10 min read

#aws #docker #tutorial #webdev

Setup

First we need to prepare application which is development ready.

This application consist 3 docker containers

nginx - web server
app - Api server
worker- Async worker

app and worker share same base images, main difference is that app serves http request behind nginx web server using gunicorn and wsgi. Throughout this walkthough nginx and app container will share a common lifecycle.

worker runs asyncronously using redis queue as broker using Django-RQ.

Download source

You can download the app source code here.

github: kokospapa8/ecs-fargate-sample-app

Docker Containers

APP

Base app image

Since app and worker both use same base image and pip takes quite some time to build image, I have uploaded base image in the dockerhub repo.

Docker Hub

# Creating image based on official python3 image
FROM python:3.6

# Your contacts, so people blame you afterwards
MAINTAINER Jinwook Baek <kokos.papa8@gmail.com>

# Sets dumping log messages directly to stream instead of buffering
ENV PYTHONUNBUFFERED 1

# Creating and putting configurations
RUN mkdir /config
ADD config/app /config/

# Installing all python dependencies
RUN pip install -r /config/requirements.txt

APP

FROM kokospapa8/django-sample:base

# Installing all python dependencies
RUN pip install -r /config/requirements.txt

# Open port 8000 to outside world
EXPOSE 8000

# When container starts, this script will be executed.
# Note that it is NOT executed during building
CMD ["sh", "/config/django_app.sh"]

# Creating and putting application inside container
# and setting it to working directory (meaning it is going to be default)
RUN mkdir /ecs-sample
WORKDIR /ecs-sample
ADD ecs-sample /ecs-sample/

Worker

FROM kokospapa8/django-sample:base

# Installing all python dependencies
RUN pip install -r /config/requirements.txt

# When container starts, this script will be executed.
# Note that it is NOT executed during building
CMD ["sh", "/config/django_worker.sh"]

# Creating and putting application inside container
# and setting it to working directory (meaning it is going to be default)
RUN mkdir /ecs-sample
WORKDIR /ecs-sample
ADD ecs-sample /ecs-sample/

Nginx config

# app_local.conf
server {
  listen 80;

  # all requests proxies to app
  location / {
        proxy_pass http://app:8000;
    }

  # domain localhost
  server_name localhost;
}

Enviroment

As you can see from the source code you need to set some sensitive data into env

# settings/secrets.py
SECRET_KEY = get_env_variable("SECRET_KEY")
                        ......
REDIS_HOST = get_env_variable("REDIS_HOST")

----

# export SECRET_KEY = '' #only SECRET_KEY is needed to be set for local dev

Docker compose

docker-compose-local.yml

Just for local environment, we are using sqlite and redis docker container to mimic cloud enviroment. We will be using RDS-mysql and ElasticCache-redis on staging and production environment.

# File structure version
version: '3'

services:
  # Our django application
  # Build from remote dockerfile
  # Connect local app folder with image folder, so changes will be pushed to image instantly
  # Open port 8000
  app:
    build:
      context: .
      dockerfile: config/app/Dockerfile_app
    hostname: app
    volumes:
      - ./ecs-sample:/ecs-sample
    expose:
      - "8000"
    environment:
      - SECRET_KEY
      - ENV=dev
      - DJANGO_SETTINGS_MODULE=settings.local
      - REDIS_HOST=redis
    depends_on:
      - redis
      - worker

  worker:
    build:
      context: .
      dockerfile: config/app/Dockerfile_worker
    hostname: worker
    volumes:
      - ./ecs-sample:/ecs-sample
    environment:
      - SECRET_KEY
      - ENV=dev
      - DJANGO_SETTINGS_MODULE=settings.local
      - REDIS_HOST=redis
    depends_on:
      - redis

  redis:
    image: redis:5.0.5
    expose:
      - "6379"
    restart: always

  nginx:
    image: nginx
    hostname: nginx
    ports:
      - "80:80"
    volumes:
      - ./config/nginx/app_local.conf:/etc/nginx/conf.d/app_local.conf
    depends_on:
      - app

Let's run the containers

$ docker-compose -f docker-compose-local.yml up --build

Check result

Type in following url to see if you are getting correct response.

http://localhost/api/healthcheck/
http://localhost/api/v1/posts/
http://localhost/admin/posts/
http://localhost/django-rq/ - to check async worker

Moving on

How to deploy django app to ECS Fargate Part2

Jinwook Baek ・ Jul 3 ・ 10 min read

#aws #docker #tutorial #webdev

Reference

Overview of Docker Compose

How To Set Up Django with Postgres, Nginx, and Gunicorn on Ubuntu 16.04 | DigitalOcean

Dockerfile reference

Dockerizing Django with Postgres, Gunicorn, and Nginx